Sensitive measurement of magnetic fields in the Earth-field range is of great significance for various real-world applications, including geological survey, bio-medicine, fundamental physics experiments, and magnetic navigation. Alkali-metal atomic magnetometers are outstanding candidates for such missions because of their high sensitivity. However, in the Earth-field range, the Nonlinear Zeeman (NLZ) effect produces splittings and asymmetries of magnetic-resonance resonances, leading to a well-known bottleneck limiting the sensitivity and accuracy of atomic magnetometry. For decades, researchers have devoted considerable effort to suppress this effect, but we wondered: could this apparent bottleneck actually be an opportunity in disguise?

By revisiting the NLZ effect, we were surprised to find that the Hamiltonian of NLZ effect shares the same form as the one-axis twisting (OAT) Hamiltonian, which is well-known for generating spin-squeezed states (SSS). Through the NLZ effect, the electronic spin and nuclear spin within the atom become entangled, resulting in intrinsic spin squeezing under geomagnetic conditions. This naturally raises another question: why hasn’t the squeezing property been observed before?

By further studying the spin dynamics under the NLZ effect, we find that the squeezing axis of this NLZ-induced SSS is not aligned with the measurement axis traditionally used in atomic magnetometers. Furthermore, the squeezing degree and axis oscillate over time, making it challenging to directly observe or utilize for magnetic sensing. In conventional methods for generating SSS, control fields are turned off at the optimal squeezing point, leaving the squeezed state ready for subsequent measurement. However, in Earth-field-range magnetometers, the Earth’s magnetic field cannot be simply turned off as it serves a dual role. It is both the control field that generates spin squeezing and the field to be measured. To harness the quantum advantage of NLZ-induced SSS, it is therefore essential to find a way to lock the squeezing axis to the measurement axis at the optimal squeezing degree.

Building on our prior success in using spin locking and dynamic decoupling (DD) techniques to suppress the NLZ effect for coherent spin states (CSS), we extended these ideas to the more complex task of locking SSS. Compared to CSS, SSS involve more coherence terms in the density matrix that require careful handling. To meet this challenge, we turned to machine learning to design optimized rotation pulse sequences for the DD protocol. This approach proved remarkably successful: with the machine-learning-assisted DD sequence, the SSS maintained a fidelity as high as 99.99% in the Earth’s magnetic field. This result opens up new possibilities for leveraging spin squeezing in practical Earth-field magnetometry, potentially transforming the limitations imposed by NLZ effects into a quantum advantage.