In the ever-evolving world of technology, human-machine interfaces (HMIs) have played a crucial role in creating seamless interactions between humans and machines. One of the fascinating aspects of human communication is the use of finger motion to convey nonverbal information, such as expression and emphasis. Harnessing this intuitive ability, researchers have made significant strides in integrating finger motion with sensing technology to create advanced HMIs that hold great potential for various applications, including augmented and virtual reality systems.

However, the traditional methods of detecting and recognizing finger motion in free space have often relied on 3D depth cameras, infrared cameras, and inertial measurement units. These techniques, while effective, suffer from limitations in processing efficiency, comfortability, and portability. Additionally, complex algorithms are required to distinguish finger motion from the background, making them time and energy-intensive. Addressing these challenges, we have proposed cutting-edge approaches based on the integration of sensors and artificial synapses for innovative human motion perception artificial intelligence systems.

The ultrathin skin-conformable electronics integrated with titanium oxide (TiO₂)-based artificial synapse (TOAS) and organic proximity sensors (OPS) that can be rolled, folded, and conformably attached to any surface, including human skin. The total thickness of TOAS is under approximately 55 nm, confirming its conformability with the ultrathin substrate. Even after being peeled off from the glass substrate, the TOAS device maintains similar switching characteristics and nonvolatile analog conductance states, indicating its self-standing operation. The highly flexible TOAS device adheres to elastomer surfaces and can be compressed or released through elastomer deformation. The practical and robust application of TOAS devices on finger skin requires stable periodicity under repeated input pulses and mechanical deformations. Verification of TOAS operation under skin-like bending and stretching tests was performed by evaluating postsynaptic currents (PSCs) for LTP/LTD functions while changing strain S (%). The TOAS device demonstrated stable synaptic operations with slight deviations under different strains, and even after 1200 bending cycles at 20% strain, it exhibited acceptable synaptic behavior and I-V switching characteristics (Fig. 1).

Fig. 1. TOAS device setup, imaging, and initial characterization

a, Schematic of flexible 8 × 8 TOAS device array. b, Optical image (left) and schematic representation of the cross-sectional image (right) of freestanding TOAS device array after peeling-off. c, Cross-sectional TEM image of selected TOAS memristor cell. The scale bar is 10 nm. d, Representative I-V switching curves of TOAS device before (left) and after (right) peeling-off. e, Schematic of compression/release tests. f, Images of TOAS device under strain S of 0%–60%. g, Sequential LTP/LTD transitions while changing strain S from 0% to 40%.

To digitize real-time finger movements in 3D free space, we used an OPS that converts movement information into electrical signals in the presence of light. Ultra-thin OPS was attached to the top and side of the index finger, and LEDs were fixed on the x-axis (green) and y-axis (red). We drew numbers by moving our fingers to the OPS in the area following the Euler trace. For example, when the number 3 was plotted, the field time-resolved V_out showed a variation of tens of millivolts along the x-axis (green) and y-axis. To verify the applicability of the integrated OPS-TOAS array, a 2 × 1 crossbar array was constructed using two integrated OPS-TOAS devices that sum the PSC along a single row based on each output signal. OPS generates V_out based on light intensity (I_L) and TOAS devices generate PSC based on applied input voltage. As a result, the OPS-TOAS array allowed modulation of the PSC level according to the conductance of individual TOAS devices for compatible applied voltages related to the input I_L to the OPS (Fig. 2).

Fig. 2. Characterization of OPS and principles of OPS–TOAS devices.

a, Scheme of V_out variation according to D_L between the light source and OPS. b, Normalized V_out according to D_L for freestanding OPS under red and green LED illumination. c, Photograph of OPSs attached to the top (x-axis) and side (y-axis) of the index finger. d, In-situ time-resolved ∆V_X and ∆V_Y along x- and y-axes while drawing digit 3. e, Time-dependent 3D trajectory obtained from ∆V_X and ∆V_Y. f, Schematic of 2 ´ 1 OPS–TOAS crossbar array. g, PSC sums from 2 ´ 1 OPS–TOAS array for three states of the two TOAS devices.

An integrated OPS-TOAS array-based neuromorphic system showcases its capacity for learning and finger-writing recognition in 3D free space. The approach involves training an 88 TOAS array, established via OPS signal processing, to recognize number patterns based on finger movement trajectories. By translating hand movements into OPS voltage signals, which are converted into synaptic weight maps in the TOAS array, the system effectively simulates human motion perception akin to cognitive science processes. The procedure comprises three stages: 1) physical action of finger-writing number patterns in free space while wearing the integrated OPS-TOAS device under light exposure, 2) real-time digitalization of OPS optical detection and time-series Vout conversion (i.e., optoelectronic signals) of input neurons (V_n) as image reshaping process, and 3) TOAS device updating synaptic weights (w) modulation using determined image feature for learning and recognition. The results showed that finger-written random number images based on OPS-attached finger motions could be evaluated and recognized based on acquired w values. All finger-writing actions for numbers 0 to 9 were successfully reproduced as 1616 matrix images with w values determined by the synaptic weight conductance map of TOAS devices (Fig. 3).

Fig. 3. Finger-writing motion recognition and learning capability.

a, Schematic of finger-writing recognition neuromorphic system in 3D free space under exposure to light based on the integration of OPS–TOAS device. b, Finger-writing motion for digit 3 in 3D free space, and c, Recognition accuracy according to strain S and number of repeated cycles for TOAS array

In conclusion, this novel real-time finger motion recognition system represents a significant advancement in soft human-machine interfaces. By integrating ultrathin TOAS and OPS devices, it offers efficient and accurate finger motion recognition without restricting finger movement. Its potential applications span from augmented and virtual reality systems to various interactive interfaces, paving the way for more intuitive and seamless human-machine interactions.

For more information, please refer to our article published in Nature Electronics:

Cho, H., Lee, I., Jang, J., Kim, J. H., Lee, H., Park, S., & Wang, G. (2023). Real-time finger motion recognition using skin-conformable electronics. Nature Electronics, 1-11.

https://doi.org/10.1038/s41928-023-01012-z