Machine vision is developing rapidly and is widely used in autonomous vehicles, drones, and surveillance systems. In challenging lighting conditions such as high-glare conditions, foggy environments, or smoky conditions, machine vision based on conventional image sensors captures low-quality images, making it difficult to extract visual features effectively. Although post-imaging algorithms can be used to improve the quality of captured images, they require additional power and latency. However, additional power consumption can seriously impact the endurance capacity of power-sensitive applications, such as drones. In high-stakes situations like autonomous vehicles, having minimal latency is extremely important because even a small delay or mistake can lead to serious outcomes. Biological vision systems are efficient in processing visual information, in part due to the preprocessing capability of the sensory terminal (retina), which helps save time and power. Thus, there has been a demand for the design of novel biomimetic vision sensors with in-sensor or near-sensor computing capability. Particularly, bioinspired in-sensor light intensity adaptation has been shown to be an effective method for improving imaging quality. However, the in-sensor spectral adaptation of biological eyes is rarely discussed.

In the natural world, the Pacific salmons have developed a spectrally adaptive vision to inhabit environments with variable spectral conditions (Fig. 1a). This spectral adaptation function originates from two kinds of photosensitive visual pigments — vitamin A₁-based visual pigments (VP₁) and vitamin A₂-based visual pigments (VP₂) — with tunable ratios. VP₁ and VP₂ are more sensitive to shorter and longer wavelengths, respectively. By converting vitamin A₁ into vitamin A₂ through an enzyme mechanism, the retina can dynamically shift its spectral sensitivity to match the spectra in the surrounding environment (Fig. 1b). Specifically, the spectral adaptation can increase the contrast of the perceived images, and allow Pacific salmons to clearly see the surroundings in the spectrally distinctive environments (Fig. 1c).

Fig. 1| Spectral adaptation behavior in the vision system of Pacific salmons. a, Pacific salmons habit spectrally variable environments, from turbid inland streams to clear open seas, in different stages of their life cycles. b, Pacific salmons can adapt the spectral sensitivity to shorter or longer wavelength light according to the spectral environments. c, The vision systems of the Pacific salmon vision system can perceive clearer images through spectral adaptation.

 In our recent research, published in Nature Electronics, we took inspiration from the vision systems of Pacific salmon to design a spectra-adapted vision sensor based on an array of back-to-back photodiodes (Fig. 2a). The back-to-back photodiodes composed of switchable junctions — between surface shallow and bottom deep junctions by changing bias voltages — with different spectra sensitivity. The back-to-back photodiodes can be adjusted to match either the broadband visible spectrum or narrowband near-infrared spectrum (Fig. 2b). Afterward, we conducted a test using an 8 × 8 vision sensor array to show how the sensor array adapts to different light spectrums (Figs. 2c-f). The target digit “5” carried by visible and near-infrared spectrum is projected on the array, as well as a zigzag pattern formed by visible light to represent interference light (Fig. 2c). By switching the voltage V_drive from +2 V to -2 V to perform the in-sensor near-infrared-spectrum adaptation within 48 μs, the vision sensor array fastly avoids the zigzag interference and captures the real digit “5” (iii) instead of the digit resembling “6” (i) (Fig. 2d). The other situation is the target digit “9” carried by visible light and a uniform near-infrared light is projected on the array (Fig. 2e). By switching the voltage V_drive from -2 V to +2 V to perform the in-sensor visible-spectrum adaptation within 48 μs, the vision sensor array fastly captures the real digit “9” (iii) instead of displaying no digit (i) (Fig. 2f).

Fig. 2| Bias-switchable in-sensor spectral adaptation. a, Cross-sectional schematic and SEM image of the device structure. Scale bar is 200 nm. b, EQE-wavelength curve of the back-to-back photodiode under different bias voltages. c,d Illustration of an 8 × 8 spectra-adapted vision sensor array in the NIR-spectrum adaptation test. e,f Illustration of an 8 × 8 spectra-adapted vision sensor array in the Vis-spectrum adaptation test.

   Finally, we demonstrate the application of our vision sensor in the machine vision system of an autonomous vehicle. The vision system needs to identify objects on the road, even when exposed to high-beam glares from incoming cars, which are defined as type-I features (Fig. 3a(i). The visible-light glare results in a high-illuminance background, and decreases the contrast of the targeting objects. To reduce glare interference and obtain clear images, we can utilize a narrowband spectral response in the NIR region. In the other case, the machine vision system needs to recognize the information from the traffic sign with white digits and a red background, which is defined as type-II features (Fig. 3a(ii)). To capture the information on the traffic sign, we can use a broadband spectral response in the visible spectrum region. The captured features in both broadband vision and narrowband NIR vision were fed to the pre-trained neural network for recognition. (Fig. 3b). Fig. 3c summarizes the accuracy in simultaneously classifying all 10 classes in the 2 types of spectrally distinctive features. The spectra-adapted vision sensors can effectively capture two types of spectrally distinctive features, resulting in a recognition accuracy of 90%. In contrast, the conventional vision sensors, broadband or narrowband NIR, show poor performance.

Fig. 3| Spectra-adapted vision sensor for imaging and classifying spectrally distinctive features. a, Schematic illustration of the application scenario of the spectra-adapted vision sensors. b, Operational scheme of the spectra-adapted vision system. Panels (i) and (ii) demonstrate the imaging of typical type-I and type-II features by the spectra-adapted vision sensor, and panel (iii) shows the architecture of the two-channel artificial neural network. c, Recognition accuracies of the 10 classes in the two types of spectrally distinctive features.

This work reports a spectra-adapted vision sensor that can collect information in broad spectra and resist the interference of visible light, which helps machine vision systems improve recognition accuracy in high-glare, smoky, and foggy environments. For additional details, please refer to our most recent article published in Nature Electronics: “Bioinspired in-sensor spectral adaptation for perceiving spectrally distinctive features” (https://doi.org/10.1038/s41928-024-01208-x).