Holographic 3D display has been a research hotspot in recent years, and its properties of no vergence-accommodation conflict and no visual fatigue have prompted researchers to develop a strong research interest in it. However, most of the reports on holographic 3D display are centered around virtual 3D scenes. In addition, holographic 3D display relies on the illumination of coherent light, which inevitably introduces speckle noise in the reconstruction process. Currently, real-time generation and high-fidelity reconstruction of the hologram of a real 3D scene are still difficult to achieve.

In this work, a holographic camera for real-time acquisition and real-time hologram generation of the real 3D scene is proposed. Figure 1 shows the concept of the holographic camera. At the acquisition end, a liquid camera based on the elastic membrane liquid lens is designed, which can take a fast zoom within 15 ms driven by a voice coil motor. At the calculation end, a real-time hologram generation method based on the focus stack network (FS-Net) is proposed. To solve the problem that the blur of the holographic reconstructed image does not match the blur of the real 3D scene, an aberration compensation method based on Zernike phase is used in the FS-Net. The time for the FS-Net to calculate the hologram is only 13 ms after the focus stack captured by the liquid camera is fed into the FS-Net. With the proposed holographic camera, the peak signal-to-noise ratio of the holographic reconstructed image can reach 40 dB, and the time from scene capture to holographic display is only 28 ms.

Figure 1 Concept of the proposed holographic camera. 

Figure 2 shows the structure and function of the elastic membrane liquid lens, which is the core device of the holographic camera. The elastic membrane liquid lens consists of a shell, two pieces of window glass, a press plate, a liquid cavity, a magnet, a coil, an elastic membrane, filled liquid and several ventilation holes. To make the response speed of the elastic membrane liquid lens fast enough, we use a high refractive index filled liquid material and a special elastic membrane control assembly to shorten the zoom response time of the elastic membrane liquid lens to 15 ms. When different driving currents are applied, the face shape of the elastic membrane liquid lens changes under the effect of ampere force, which in turn has the function of focusing and defocusing.

Figure 2 Structure and function of the elastic membrane liquid lens.

Figure 3 shows the flow of training and generating holograms based on the FS-Net. We take the focus stack of the real 3D scene as the input to FS-Net, and then introduce the focus stack renderer and learnable Zernike polynomials in the hologram training process. Firstly, we process the all-in-focus image and depth map of the real 3D scene into the focus stack using the focus stack renderer. Next, the complex amplitude distribution of the focus stack is output using the untrained FS-Net. Then, the complex amplitude distribution is back-propagated using a physical diffraction model to obtain the layered holographic reconstructed images. Subsequently, several binary masks are calculated based on the number of layers and the depth map, and the inner products of the binary masks with the focus stack and the layered holographic reconstructed images are calculated, respectively, which in turn yield the target focus and defocus images, and the reconstructed focus and defocus images. Finally, the loss function between the target focus and defocus images and the reconstructed focus and defocus images are calculated and the parameters in the learnable Zernike polynomials and FS-Net are optimized. The above steps are repeated and the training of the FS-Net stops when the network iterates to a fixed number of rounds. We input the focus stack captured by the liquid camera into the trained FS-Net to obtain the complex amplitude distribution of the focus stack. After compensating for complex amplitude distribution using the Zernike phase, a high-fidelity hologram of the focus stack can be obtained using the double-phase encoding method.

Figure 3 Flow of training and generating holograms based on the FS-Net.

In general, we break through the bottlenecks of the current holographic 3D display,  namely the difficulty of capturing real 3D scene, the low quality of holographic reconstructed images and the incorrect defocus blur. The holographic camera we developed is easy to operate. We believe that the real-time holographic camera can further advance the development of holographic technology and provide new research ideas for 3D display, 3D measurement and other fields.