Photomemristors as dynamic machine vision sensor

In-photomemristor motion perception recognizes the past and predicts the future path of objects.
Photomemristors as dynamic machine vision sensor
Like

Share this post

Choose a social network to share with, or copy the shortened URL to share elsewhere

This is a representation of how your post may appear on social media. The actual post will vary between social networks

Dynamic machine vision technology requires past motion recognition and future trajectory prediction for intelligent sensing and on-site decision-making. Current image sensing and machine vision technologies accomplish this by analyzing massive frame-by-frame image sequences in multiple hardware blocks and complex software algorithms, engendering redundant data flows, high energy consumption, and latency.

Photomemristors, or optoelectronic memristors, originally proposed for photosensing, processing, and memory functions [1], are ideal candidates for dynamic machine vision tasks. In recent years, photomemristors have been studied in neuromorphic vision and processing systems for static image classification [2-6] and human action recognition [6]. However, motion recognition and prediction within a compact dynamic sensing system, which is crucial for dynamic machine vision technology, has been elusive until very recently. In our recent work published in Nature Communications, we reported recurrent photomemristor networks consisting of a retinomorphic photomemristor array operating as a dynamic vision reservoir and readout networks for postprocessing (Fig. 1a).

In the retinomorphic photomemristor-reservoir computing system (Fig. 1b), the inherent dynamic memory (Fig. 1c-e) of the photomemristor networks stores spatiotemporal information of a frame-by-frame visual sequence as hidden states (h) in the last frame (Fig. 1a). The dynamic photomemristor networks reservoir, containing all the past spatiotemporal visual information, is used for dynamic processing tasks through the training of readout networks.

Fig. 1: Retinomorphic photomemristor-reservoir computing system and dynamic memory states of the photomemristor.

To demonstrate the spatiotemporal processing capability of the retinomorphic photomemristor-reservoir computing system, we implemented the classification of videos playing English words ending with the same letter but with different spatiotemporal dynamics for language learning. An accuracy of 91.3 % was achieved, which is much higher than the 36.2 % accuracy obtained when the system was operated as a conventional photosensor. Moreover, our retinomorphic photomemristor-reservoir computing system shows memory-dependent dynamic recognition behavior (100 % with higher memory states), which well resembles memory-dependent perception in the brain, enabling intelligent sensors with tunable attention.

We also realized the most crucial dynamic machine vision task—motion recognition and trajectory prediction—with our retinomorphic photomemristor-reservoir computing system using classification and inherent memory association by the readout networks. Additionally, to emulate crossmodal prediction using a single compressed frame, we associated vision motion perception with audio inputs through crossmodal learning, providing a promising multimodal neuromorphic platform for in-sensor dynamic machine vision.

The demonstrated recurrent photomemristor networks hold great potential for urgent dynamic machine vision applications requiring accurate on-site motion perception and prediction.

References:

  1. Tan, H., Liu, G., Zhu, X., Yang, H., Chen, B., Chen, X., Shang, J., Lu, W. D., Wu, Y. & Li, R.-W. An optoelectronic resistive switching memory with integrated demodulating and arithmetic functions. Adv. Mater. 27, 2797–2803 (2015).
  2. Tan, H., Tao, Q., Pande, I., Majumdar, S., Liu, F., Zhou, Y., Person, P. O. Å., Rosen, J. & van Dijken, S. Tactile sensory coding and learning with bioinspired optoelectronic spiking afferent nerves. Nat. Commun. 11, 1369 (2020).
  3. Tan, H., Zhou, Y., Tao, Q., Rosen, J. & van Dijken, S. Bioinspired multisensory neural network with crossmodal integration and recognition. Nat. Commun. 12, 1120 (2021).
  4. Zhou, F., Zhou, Z., Chen, J., Choy, T. H., Wang, J., Zhang, N., Lin, Z., Yu, S., Kang, J., Wong, H.-S. P. & Chai, Y. Optoelectronic resistive random access memory for neuromorphic vision sensors. Nat. Nanotechnol. 14, 776–782 (2019).
  5. Meng, Y., Li, F., Lan, C., Bu, X., Kang, X., Wei, R., Yip, S., Li, D., Wang, F., Takahashi, T., Hosomi, T., Nagashima, K., Yanagida, T. & Ho, J. C. Artificial visual systems enabled by quasi–two-dimensional electron gases in oxide superlattice nanowires. Sci. Adv. 6, eabc6389 (2020).
  6. Sun, Y., Li, Q., Zhu, X., Liao, C., Wang, Y., Li, Z., Liu, S., Xu, H. & Wang, W. In-sensor reservoir computing based on optoelectronic synapse. Adv. Intell. Syst. 5, 2200196 (2023).

Please sign in or register for FREE

If you are a registered user on Research Communities by Springer Nature, please sign in

Subscribe to the Topic

Electrical and Electronic Engineering
Technology and Engineering > Electrical and Electronic Engineering

Related Collections

With collections, you can get published faster and increase your visibility.

Cancer and aging

This cross-journal Collection invites original research that explicitly explores the role of aging in cancer and vice versa, from the bench to the bedside.

Publishing Model: Hybrid

Deadline: Jul 31, 2024

Applied Sciences

This collection highlights research and commentary in applied science. The range of topics is large, spanning all scientific disciplines, with the unifying factor being the goal to turn scientific knowledge into positive benefits for society.

Publishing Model: Open Access

Deadline: Ongoing