Dynamic machine vision technology requires past motion recognition and future trajectory prediction for intelligent sensing and on-site decision-making. Current image sensing and machine vision technologies accomplish this by analyzing massive frame-by-frame image sequences in multiple hardware blocks and complex software algorithms, engendering redundant data flows, high energy consumption, and latency.

Photomemristors, or optoelectronic memristors, originally proposed for photosensing, processing, and memory functions [1], are ideal candidates for dynamic machine vision tasks. In recent years, photomemristors have been studied in neuromorphic vision and processing systems for static image classification [2-6] and human action recognition [6]. However, motion recognition and prediction within a compact dynamic sensing system, which is crucial for dynamic machine vision technology, has been elusive until very recently. In our recent work published in Nature Communications, we reported recurrent photomemristor networks consisting of a retinomorphic photomemristor array operating as a dynamic vision reservoir and readout networks for postprocessing (Fig. 1a).

In the retinomorphic photomemristor-reservoir computing system (Fig. 1b), the inherent dynamic memory (Fig. 1c-e) of the photomemristor networks stores spatiotemporal information of a frame-by-frame visual sequence as hidden states (h) in the last frame (Fig. 1a). The dynamic photomemristor networks reservoir, containing all the past spatiotemporal visual information, is used for dynamic processing tasks through the training of readout networks.

Fig. 1: Retinomorphic photomemristor-reservoir computing system and dynamic memory states of the photomemristor.

To demonstrate the spatiotemporal processing capability of the retinomorphic photomemristor-reservoir computing system, we implemented the classification of videos playing English words ending with the same letter but with different spatiotemporal dynamics for language learning. An accuracy of 91.3 % was achieved, which is much higher than the 36.2 % accuracy obtained when the system was operated as a conventional photosensor. Moreover, our retinomorphic photomemristor-reservoir computing system shows memory-dependent dynamic recognition behavior (100 % with higher memory states), which well resembles memory-dependent perception in the brain, enabling intelligent sensors with tunable attention.

We also realized the most crucial dynamic machine vision task—motion recognition and trajectory prediction—with our retinomorphic photomemristor-reservoir computing system using classification and inherent memory association by the readout networks. Additionally, to emulate crossmodal prediction using a single compressed frame, we associated vision motion perception with audio inputs through crossmodal learning, providing a promising multimodal neuromorphic platform for in-sensor dynamic machine vision.

The demonstrated recurrent photomemristor networks hold great potential for urgent dynamic machine vision applications requiring accurate on-site motion perception and prediction.

References:

1. Tan, H., Liu, G., Zhu, X., Yang, H., Chen, B., Chen, X., Shang, J., Lu, W. D., Wu, Y. & Li, R.-W. An optoelectronic resistive switching memory with integrated demodulating and arithmetic functions. Adv. Mater. 27, 2797–2803 (2015).
2. Tan, H., Tao, Q., Pande, I., Majumdar, S., Liu, F., Zhou, Y., Person, P. O. Å., Rosen, J. & van Dijken, S. Tactile sensory coding and learning with bioinspired optoelectronic spiking afferent nerves. Nat. Commun. 11, 1369 (2020).
3. Tan, H., Zhou, Y., Tao, Q., Rosen, J. & van Dijken, S. Bioinspired multisensory neural network with crossmodal integration and recognition. Nat. Commun. 12, 1120 (2021).
4. Zhou, F., Zhou, Z., Chen, J., Choy, T. H., Wang, J., Zhang, N., Lin, Z., Yu, S., Kang, J., Wong, H.-S. P. & Chai, Y. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).
5. Meng, Y., Li, F., Lan, C., Bu, X., Kang, X., Wei, R., Yip, S., Li, D., Wang, F., Takahashi, T., Hosomi, T., Nagashima, K., Yanagida, T. & Ho, J. C. Artificial visual systems enabled by quasi–two-dimensional electron gases in oxide superlattice nanowires. Sci. Adv. 6, eabc6389 (2020).
6. Sun, Y., Li, Q., Zhu, X., Liao, C., Wang, Y., Li, Z., Liu, S., Xu, H. & Wang, W. In-sensor reservoir computing based on optoelectronic synapse. Adv. Intell. Syst. 5, 2200196 (2023).