A physics-informed deep learning liquid crystal camera with data-driven diffractive guidance

We build an integrated camera capable of tracking objects of interest. By using optical computing to arrange molecules in the liquid crystal mask for enhanced distinction between the target and background.
Published in Physics and Computational Sciences
A physics-informed deep learning liquid crystal camera with data-driven diffractive guidance

Share this post

Choose a social network to share with, or copy the shortened URL to share elsewhere

This is a representation of how your post may appear on social media. The actual post will vary between social networks

Why ?

In this era of digitization, deep learning has emerged as a pivotal tool in scientific research.   However, despite its remarkable achievements across various domains, its application in physics remains relatively limited.   Our team's research endeavors to explore the integration of deep learning with principles of physics to address real-world challenges.We chose liquid crystal cameras as our research subject due to their extensive applicability.   With the capability to manipulate multidimensional optical fields (wavefronts, polarization, etc.), liquid crystal cameras hold immense potential in image capture and processing.   Nevertheless, traditional liquid crystal cameras encounter limitations such as the inability to adaptively modulate for different scenes.   Hence, we aim to enhance the performance of liquid crystal cameras using physics-informed deep learning methods to tackle these challenges.

How did we approach it ?

Our research methodology revolves around two main aspects: physics inspiration and deep learning.Firstly, we delved into the working principles of liquid crystal cameras and unearthed rich insights into their physics.   Leveraging these physical principles, we devised a novel data-driven diffraction guidance method aimed at optimizing the camera's imaging performance.
Secondly, we integrated deep learning techniques into the design of liquid crystal cameras.   By training physical neural networks, we achieved real-time perception of targets in complex scenes.   This data-driven approach enabled us to continuously enhance the camera's performance and adapt to different shooting environments.   Specifically, we combined diffraction techniques with the electrical addressing architecture of liquid crystal cameras, harnessing the parallel processing capability of light propagation and the dynamic adjustment mechanism of liquid crystals.   Additionally, by utilizing deep learning, we trained optical neural networks to accurately predict the ideal arrangement of liquid crystal molecules.   This combination of physics principles and data-driven approaches facilitated real-time tracking of targets in complex backgrounds.

The results

Our research yielded promising outcomes.   The physics-inspired deep learning liquid crystal camera we developed not only demonstrated outstanding performance in complex scenes but also exhibited high real-time capability and stability.  Most importantly, our study sets a new paradigm for applying deep learning techniques in the field of physics.

Brief Introduction

An overview of a proposed data-driven ToI tracking architecture is illustrated in Fig. 1. Scenes captured by the designed LC-based camera are utilized for training a 3-layer optical neural network in Fig. 1a. This training process establishes a fundamental mapping between the input scenes containing the hidden ToI and an appropriate electrical reconfiguration of the LC molecules. An error backpropagation process is employed based on a gradient descent strategy to switch the physical state of the LC layer and capture the desired ToI. The optimized deep diffractive phase surfaces are then used to enable all-optical inference through layer-to-layer phase modulation, which is a collaboration by multilayer diffraction to vote for the best answer, so we call it deep diffractive voting. This process predicts the optimal reorientation of the LC molecules, facilitating the efficient manipulation of target lightwaves to track the intended ToI.

 Training and predicting principle of the proposed target tracking model. b Trained phase distributions of optical neural networks using 1452 original scenes. c Three-dimensional construction of optical computing with alternating current (AC) signal feedback for target tracking.

In Fig. 2, we present experimental confirmation of learning-based diffractive guidance for LC molecule alignment, leading to a comprehensive rendering of the ToI. Figure 2a–h showcases the input scenes and their corresponding output distributions. Figure 2i visually represents the LC mode switching. Finally, the resulting confusion matrices are depicted in Fig. 2j.

Testing scene shown in (a) and (b) are related to actual prediction outputs shown in (c–f) according to single-channel training (SCT) and multi-channel training (MCT) method, where those in (g) and (h) depict the energy distribution percentage (EDP) of the concentration points in output intensity distributions. i Typical depiction of the functioned liquid crystal layer switched for scenes (a) and (b). j Confusion matrices correspond to single channel training and multi-channel training method.

Please sign in or register for FREE

If you are a registered user on Research Communities by Springer Nature, please sign in

Subscribe to the Topic

Computer Imaging, Vision, Pattern Recognition and Graphics
Mathematics and Computing > Computer Science > Computer Imaging, Vision, Pattern Recognition and Graphics
Computer Vision
Mathematics and Computing > Computer Science > Computer Imaging, Vision, Pattern Recognition and Graphics > Computer Vision
Photonics and Optical Engineering
Technology and Engineering > Biological and Physical Engineering > Photonics and Optical Engineering

Related Collections

With collections, you can get published faster and increase your visibility.

Cleaner Flight

This joint collection from Communications Engineering, Communications Earth and Environment and Scientific Reports aims to offer insights into the environmental impacts of aviation, along with technological design, operational and infrastructure solutions to mitigate aviation’s future environmental footprint.

Publishing Model: Open Access

Deadline: Apr 16, 2024

Thermal Engineering for Sustainability

This collection will publish papers on the topic of thermal engineering with a particular focus on applications related to sustainability.

Publishing Model: Open Access

Deadline: Mar 30, 2024