Six years ago, when we began this project, we did not yet have the term light-to-token. Vision Transformer (ViT) had only just appeared, and large AI models had not yet transformed the field. At the time, our starting point was biological vision. We were interested in bringing the efficient information-processing strategies of the retina into artificial vision hardware, with the goal of accelerating convolutional neural networks, which were then the dominant paradigm in computer vision. Looking back today, however, we realize that our work was never truly driven by a particular neural network architecture. Instead, it was motivated by a simple question that remained unchanged over the past six years: If a sensor observes the physical world, what should it output?
At that time, bio-inspired vision hardware was developing rapidly. Researchers were exploring ways to process static images, dynamic scenes, and different optical modalities—including light intensity, polarization, and wavelength—directly at the sensor level, hoping to extract useful information before the data reached the neural network. However, to process real images, a single device or a small processing unit was never enough; we needed a much larger system.
Our early device designs were based on mechanically exfoliated van der Waals heterostructures, which offered rich optoelectronic functionalities but were difficult to scale into large arrays. We therefore continuously simplified the device architecture while preserving its essential functions, and eventually developed a 32 × 32 photosensitive memory array based on monolayer MoS2 floating-gate transistors, together with the peripheral circuits and control system required to operate it. This transition took nearly three years and gradually transformed an individual device concept into a complete hardware platform capable of receiving and processing full images.
We initially believed that making more devices, improving yield, and integrating them into a larger array would remain the biggest challenge. Instead, we soon discovered that the real difficulties only emerged once the entire system started to operate. Problems that were invisible in a single device or a small array became unavoidable in the complete system. Device yield directly affected the overall computational accuracy. Peripheral control circuits had to be carefully matched to device characteristics. The optical projection system imposed practical constraints on the array design, while the timing of the control software determined programming efficiency. Even electrical relationships that appeared straightforward in small arrays no longer behaved as expected once the system grew larger.
One example was current summation. The principle appears straightforward: currents contributed by multiple devices should add along a shared line according to Kirchhoff’s current law. In a small array, this relationship worked much as expected. In a larger array, however, the result was influenced by device non-uniformity, parasitic effects, line resistance, imperfect contacts, and leakages throughout the system. This distinction took time to understand. The challenge was not to find a new physical law, but to identify which aspects of the real system prevented a familiar law from producing the simple result we expected.
These challenges did not belong to any single discipline; they only emerged when every part of the system had to work together. This experience also changed the way we thought about the value of scale. A larger array was not valuable simply because it contained more pixels. Instead, it served as a physical test of whether the idea remained valid after the assumptions of a small demonstration were replaced by the constraints of a real platform. The completed system was therefore the result not of a single isolated advance, but of sustained collaboration among researchers working on materials, devices, circuits, optics, control, and computation. Through the tremendous efforts of our team and our collaborators in Lance Li's group, we eventually built the complete platform.
When the system had become sufficiently mature, we returned to the question that had motivated us from the very beginning: What form should visual information take when it leaves the sensor and enters an intelligent system? During the same period, however, ViT began to change how visual information was organized and processed in modern vision models. Instead of treating an image only as a grid of pixels or a hierarchy of convolutional feature maps, ViT first divides the image into patches and represents each patch as an embedded token. These tokens then become the basic units passed into the Transformer encoder. This also changed how we understood the boundary between a sensor and an AI model. We realized that the question itself had never changed—only the answer had gradually become clearer. That was the point at which we began to describe the work as light-to-token.
In retrospect, the idea of matching the output of a sensor to the system that receives it has long existed in different forms. In biological vision, the retina transforms optical stimuli into patterns of neural activity that can be communicated through the optic nerve and processed by the brain. A CMOS camera converts incident light into electrical signals that are ultimately organized into digital images, typically represented as RGB pixel values, because that is a form that digital computers can readily store and process. For future physical AI systems, the appropriate interface may once again be different. If modern AI models operate on tokens, it is natural to ask whether future vision hardware could produce those tokens directly, rather than first generating a conventional image and then reconstructing the model input through a sequence of digital operations.
We do not regard this as meaning that every sensor should output tokens, or that images and task-specific features will no longer be useful. Different systems will require different representations. What this work demonstrates is a further possibility: the representation required by an AI model can begin to form at the same physical location where visual information is first detected. We hope that this way of thinking may offer a useful direction for future physical AI hardware: not only moving computation closer to the sensor, but reconsidering what information the sensor should provide to the intelligent system in the first place.