Bioinspired: A hardware implementation replicating human visual pathways.

A human visual pathway-replicated hardware is constructed with split floating gate 2D tungsten diselenide crossbar arrays and related peripheral circuits, achiving functions of color-blindness processing, shape recognition, and self-driven motion tracking.
Published in Physics
Like

Share this post

Choose a social network to share with, or copy the URL to share elsewhere

This is a representation of how your post may appear on social media. The actual post will vary between social networks

Natural selection, survival of the fittest. Biological evolution contains infinite wisdom and inspires the development of science and technology. The human visual system dominated by the actual neural circuits of visual pathways which consists of the retina, visual cortex and their connective neural circuits, realizes powerful visual functions through multi-level signal processing. In recent years, more and more new optoelectronic sensor devices have been developed, inspired by the connectomics of biological visual systems. Previous 2D materials-based artificial visual hardware designs almost mimic architecture and mechanism of the retina [Nat. Electron. 5, 84-91 (2022); Sci. Adv. 6, eaba6173 (2020); Nat. Nanotechnol. 17, 27-32 (2022); Sci. Adv. 9, eadi4083 (2023)]. However, they disregard the replication of visual pathways in hardware design, making it challenging to combine the basic functionalities in one hardware to enable more complex and efficient functions. The human visual system is dominated by visual pathways, which include the retina, visual cortex, and connectomics between them. Following the anatomical structure and specific connectivity of them, the P and M pathways are constructed to process static [color and shape] and dynamic [direction and motion] information (Fig. 1a). Therefore, it’s necessary to design a general hardware architecture to replicate actual neural circuits of visual pathways for more powerful functions. In this work, we designed a general hardware architecture with the crossbar array and related connective peripheral circuits to replicate the neural circuits of visual pathways.

At the device level, we put forward a kind of split floating gate (SFG) 2D tungsten diselenide (WSe2) device, which is modulated by floating gate voltage pulse with two operation modes. The 10 × 10 crossbar array is fabricated based on the unit device (Fig. 1b). On the one hand, two adjacent regions of the device channel are reconfigurable electrostatic doped by the SFGs, which constitutes the p-n or n-p photodiode with the gate voltage linearly modulated responsivity. Under this photovoltaic diode mode, reconfigurable positive/negative optical responsivity of the array is used to convolute with input light, mimicking center-surround antagonism receptive fields (CSRFs) of the retina. The sensing operation of the array under photovoltaic effect is self-powered with nearly zero standby power consumption, and the programming energy for floating gate is below 1 pJ/spike, promising ultra-low power consumption. On the other hand, when one gate is fixed and the other modulates the transistor conductance, the device serves as the unit weight of the neural network. Under this bipolar transistor mode, the reconfigurable conductivity of the array is used to calculate matrix multiplication by Kirchhoff’s laws, mimicking neural network of the visual cortex.

At the circuit level, according to the neural connection structure of different visual pathways, we designed the peripheral circuits and fabricated the related Printed Circuit Board (PCB) to construct the human visual pathway-replicated hardware (Schematic in Fig. 1c). Integrating with related necessary peripheral circuits for current–voltage transition, programming and control, multi-functions consisting of color processing, shape recognition, and motion tracking are realized. The specific implementation scheme and results are as follows:

1. Color processing: The CSRFs of different colors in the visual pathway converge to perform single/double color opponency (SO/DO) process and generate color information through specific connection states in the lateral geniculate nucleus (LGN), primary visual cortex (V1), area V4, and the inferotemporal (IT) cortex. We construct the retina–LGN–V1–V4–IT circuit to demonstrate the function of color processing. The cause of red–green color blindness (Daltonism) is explained with the hardware based on the visual pathway-replicated design.

2. Shape recognition: The CSRFs in the retina–LGN are integrated into the orientation-selective CSRF in V1 in accordance with different spatial distributions. The contour information of each point is summarized through V2/V4 with a sparse connection, realizing shape classification in IT. We construct the retina–LGN–V1–V2–V4–IT circuit to demonstrate the function of shape recognition. The effective shape classification with a recognition rate of >95% is demonstrated in experiment within a double-layer sparse neural network which is implemented by the visual cortex of the hardware, promising low-power applications by a 61.1% reduction in device usage and only 0.9 nJ programming energy per operation.

3. Motion tracking: The M pathway of the human visual system implements motion tracking based on the Barlow–Levick model with length-dependent signal transmission delay of axons. The CSRF-based direction selector in the retina and visual cortex only produces signal superposition and activation to the motion stimulus that is parallel to the given movement direction, and controls the eye movement to track moving objects. By integrating the crossbar array with related peripheral circuits to replicate the M pathway, the self-driven motion tracking is realized with an adjustable speed controlled by delay time.

Fig. 1 | Human visual pathways and working mechanisms of the human visual pathway-replicated hardware. a, Human visual pathways, characteristics of the center-surround receptive field (CSRF), and functions of the human visual system.  LGN: lateral geniculate nucleus; V1: primary visual cortex; V2: secondary visual cortex; V4: extrastriate cortex; MT: middle temporal cortex; IT: inferotemporal cortex. b, Microscope image of a 10 × 10 tungsten diselenide (WSe2) split floating gate (SFG) diode/transistor array mimicking the retina and visual cortex. Insert: Schematic of a unit device, exhibiting reconfigurable responsivity and conductivity under the photovoltaic diode and bipolar transistor modes. The array under the photovoltaic diode and bipolar transistor modes performs photoresponse convolution and matrix multiplication to mimic the retina and visual cortex, respectively. c, Flowchart of the human visual pathway-replicated hardware that consists of the SFG array and connective peripheral circuits for color process, shape recognition, and motion tracking. The shadings in the background represent the parts of peripheral circuits. SO: single color opponency receptive field; DO: double color opponency receptive field.

Overall, our scheme of the visual pathway-replicated hardware may inspire neuroscience in turn and promote progress in the fields of driverless technology and intelligent robots. Furthermore, the complete reproduction of visual pathways will help develop brain-computer interface devices that are more adapted to neural structures, and help blind or color-blindness patients regain normal vision.

For more information, please refer to our recent publication in Nature Communications, “Multifunctional human visual pathway-replicated hardware based on 2D materials” (https://doi.org/10.1038/s41467-024-52982-3).

Please sign in or register for FREE

If you are a registered user on Research Communities by Springer Nature, please sign in

Follow the Topic

Visual system
Physical Sciences > Physics and Astronomy > Biophysics > Sensory Systems > Visual system
Two-dimensional Electronic Properties
Physical Sciences > Physics and Astronomy > Condensed Matter Physics > Two-dimensional Materials > Two-dimensional Electronic Properties
Optoelectronic Devices
Physical Sciences > Physics and Astronomy > Optics and Photonics > Optoelectronic Devices

Related Collections

With collections, you can get published faster and increase your visibility.

Biology of rare genetic disorders

This cross-journal Collection between Nature Communications, Communications Biology, npj Genomic Medicine and Scientific Reports brings together research articles that provide new insights into the biology of rare genetic disorders, also known as Mendelian or monogenic disorders.

Publishing Model: Open Access

Deadline: Jan 31, 2025

Carbon dioxide removal, capture and storage

In this cross-journal Collection, we bring together studies that address novel and existing carbon dioxide removal and carbon capture and storage methods and their potential for up-scaling, including critical questions of timing, location, and cost. We also welcome articles on methodologies that measure and verify the climate and environmental impact and explore public perceptions.

Publishing Model: Open Access

Deadline: Mar 22, 2025