Natural selection, survival of the fittest. Biological evolution contains infinite wisdom and inspires the development of science and technology. The human visual system dominated by the actual neural circuits of visual pathways which consists of the retina, visual cortex and their connective neural circuits, realizes powerful visual functions through multi-level signal processing. In recent years, more and more new optoelectronic sensor devices have been developed, inspired by the connectomics of biological visual systems. Previous 2D materials-based artificial visual hardware designs almost mimic architecture and mechanism of the retina [Nat. Electron. 5, 84-91 (2022); Sci. Adv. 6, eaba6173 (2020); Nat. Nanotechnol. 17, 27-32 (2022); Sci. Adv. 9, eadi4083 (2023)]. However, they disregard the replication of visual pathways in hardware design, making it challenging to combine the basic functionalities in one hardware to enable more complex and efficient functions. The human visual system is dominated by visual pathways, which include the retina, visual cortex, and connectomics between them. Following the anatomical structure and specific connectivity of them, the P and M pathways are constructed to process static [color and shape] and dynamic [direction and motion] information (Fig. 1a). Therefore, it’s necessary to design a general hardware architecture to replicate actual neural circuits of visual pathways for more powerful functions. In this work, we designed a general hardware architecture with the crossbar array and related connective peripheral circuits to replicate the neural circuits of visual pathways.
At the device level, we put forward a kind of split floating gate (SFG) 2D tungsten diselenide (WSe2) device, which is modulated by floating gate voltage pulse with two operation modes. The 10 × 10 crossbar array is fabricated based on the unit device (Fig. 1b). On the one hand, two adjacent regions of the device channel are reconfigurable electrostatic doped by the SFGs, which constitutes the p-n or n-p photodiode with the gate voltage linearly modulated responsivity. Under this photovoltaic diode mode, reconfigurable positive/negative optical responsivity of the array is used to convolute with input light, mimicking center-surround antagonism receptive fields (CSRFs) of the retina. The sensing operation of the array under photovoltaic effect is self-powered with nearly zero standby power consumption, and the programming energy for floating gate is below 1 pJ/spike, promising ultra-low power consumption. On the other hand, when one gate is fixed and the other modulates the transistor conductance, the device serves as the unit weight of the neural network. Under this bipolar transistor mode, the reconfigurable conductivity of the array is used to calculate matrix multiplication by Kirchhoff’s laws, mimicking neural network of the visual cortex.
At the circuit level, according to the neural connection structure of different visual pathways, we designed the peripheral circuits and fabricated the related Printed Circuit Board (PCB) to construct the human visual pathway-replicated hardware (Schematic in Fig. 1c). Integrating with related necessary peripheral circuits for current–voltage transition, programming and control, multi-functions consisting of color processing, shape recognition, and motion tracking are realized. The specific implementation scheme and results are as follows:
1. Color processing: The CSRFs of different colors in the visual pathway converge to perform single/double color opponency (SO/DO) process and generate color information through specific connection states in the lateral geniculate nucleus (LGN), primary visual cortex (V1), area V4, and the inferotemporal (IT) cortex. We construct the retina–LGN–V1–V4–IT circuit to demonstrate the function of color processing. The cause of red–green color blindness (Daltonism) is explained with the hardware based on the visual pathway-replicated design.
2. Shape recognition: The CSRFs in the retina–LGN are integrated into the orientation-selective CSRF in V1 in accordance with different spatial distributions. The contour information of each point is summarized through V2/V4 with a sparse connection, realizing shape classification in IT. We construct the retina–LGN–V1–V2–V4–IT circuit to demonstrate the function of shape recognition. The effective shape classification with a recognition rate of >95% is demonstrated in experiment within a double-layer sparse neural network which is implemented by the visual cortex of the hardware, promising low-power applications by a 61.1% reduction in device usage and only 0.9 nJ programming energy per operation.
3. Motion tracking: The M pathway of the human visual system implements motion tracking based on the Barlow–Levick model with length-dependent signal transmission delay of axons. The CSRF-based direction selector in the retina and visual cortex only produces signal superposition and activation to the motion stimulus that is parallel to the given movement direction, and controls the eye movement to track moving objects. By integrating the crossbar array with related peripheral circuits to replicate the M pathway, the self-driven motion tracking is realized with an adjustable speed controlled by delay time.
Overall, our scheme of the visual pathway-replicated hardware may inspire neuroscience in turn and promote progress in the fields of driverless technology and intelligent robots. Furthermore, the complete reproduction of visual pathways will help develop brain-computer interface devices that are more adapted to neural structures, and help blind or color-blindness patients regain normal vision.
For more information, please refer to our recent publication in Nature Communications, “Multifunctional human visual pathway-replicated hardware based on 2D materials” (https://doi.org/10.1038/s41467-024-52982-3).
Please sign in or register for FREE
If you are a registered user on Research Communities by Springer Nature, please sign in