Why ?

In this era of digitization, deep learning has emerged as a pivotal tool in scientific research.   However, despite its remarkable achievements across various domains, its application in physics remains relatively limited.   Our team's research endeavors to explore the integration of deep learning with principles of physics to address real-world challenges.We chose liquid crystal cameras as our research subject due to their extensive applicability.   With the capability to manipulate multidimensional optical fields (wavefronts, polarization, etc.), liquid crystal cameras hold immense potential in image capture and processing.   Nevertheless, traditional liquid crystal cameras encounter limitations such as the inability to adaptively modulate for different scenes.   Hence, we aim to enhance the performance of liquid crystal cameras using physics-informed deep learning methods to tackle these challenges.

How did we approach it ?

Our research methodology revolves around two main aspects: physics inspiration and deep learning.Firstly, we delved into the working principles of liquid crystal cameras and unearthed rich insights into their physics.   Leveraging these physical principles, we devised a novel data-driven diffraction guidance method aimed at optimizing the camera's imaging performance.
Secondly, we integrated deep learning techniques into the design of liquid crystal cameras.   By training physical neural networks, we achieved real-time perception of targets in complex scenes.   This data-driven approach enabled us to continuously enhance the camera's performance and adapt to different shooting environments.   Specifically, we combined diffraction techniques with the electrical addressing architecture of liquid crystal cameras, harnessing the parallel processing capability of light propagation and the dynamic adjustment mechanism of liquid crystals.   Additionally, by utilizing deep learning, we trained optical neural networks to accurately predict the ideal arrangement of liquid crystal molecules.   This combination of physics principles and data-driven approaches facilitated real-time tracking of targets in complex backgrounds.

The results

Our research yielded promising outcomes.   The physics-inspired deep learning liquid crystal camera we developed not only demonstrated outstanding performance in complex scenes but also exhibited high real-time capability and stability.  Most importantly, our study sets a new paradigm for applying deep learning techniques in the field of physics.

Brief Introduction

An overview of a proposed data-driven ToI tracking architecture is illustrated in Fig. 1. Scenes captured by the designed LC-based camera are utilized for training a 3-layer optical neural network in Fig. 1a. This training process establishes a fundamental mapping between the input scenes containing the hidden ToI and an appropriate electrical reconfiguration of the LC molecules. An error backpropagation process is employed based on a gradient descent strategy to switch the physical state of the LC layer and capture the desired ToI. The optimized deep diffractive phase surfaces are then used to enable all-optical inference through layer-to-layer phase modulation, which is a collaboration by multilayer diffraction to vote for the best answer, so we call it deep diffractive voting. This process predicts the optimal reorientation of the LC molecules, facilitating the efficient manipulation of target lightwaves to track the intended ToI.

In Fig. 2, we present experimental confirmation of learning-based diffractive guidance for LC molecule alignment, leading to a comprehensive rendering of the ToI. Figure 2a–h showcases the input scenes and their corresponding output distributions. Figure 2i visually represents the LC mode switching. Finally, the resulting confusion matrices are depicted in Fig. 2j.