In 2020, my collaborator Bharath Ramsundar introduced ChemBERTa, an attempt to use language models for molecular representation learning. Initially, I was skeptical with this approach, but, Bharath demonstrated that their strength lies in scalability. As model size and data grow, model performance improves consistently. That insight laid the conceptual foundation for what we now call scientific foundation models which is making big leaps across various domains. After our group moved to the University of Michigan, I launched the Center for Foundation Models and AI Agents for Science and in this article, we will discuss our latest work, CLOUD, a foundation model for crystalline materials.

My PhD student Changwen Xu joined that year and started to develop a large language model for crystal representation learning. Rather than relying on existing formats like SMILES, commonly used for molecules, or CIF files that explicitly encode full 3D geometry, he took a different perspective. He recognized what is fundamentally missing from most string representations of crystals: symmetry. This observation led to a shift in how we represent crystal structures. Instead of building from atomic coordinates upward, we took a top-down view: starting from the global symmetry of the crystal, then specifying the symmetry-equivalent sites, namely Wyckoff positions, and finally the atomic composition. In this formulation, the 3D atomic coordinates are not explicitly encoded, deliberately so, because they are already constrained by symmetry. The result is a representation we call Symmetry-Consistent Ordered Parameter Encoding (SCOPE), which captures the essential physics of crystalline materials in a sequence format suitable for language models. With this representation in place, we developed a scientific foundation model for crystals, CLOUD, based on the BERT architecture widely used in language models. CLOUD is pre-trained on approximately 6.3 million crystal structures collected from open-sourced materials databases. Through masked language modeling, the model learns the statistical “grammar” of crystalline materials: how symmetry and composition co-occur across millions of known compounds. This large-scale pre-training allows the model to learn universal features of crystal structures that can later be transferred to many downstream property prediction tasks.

After building CLOUD and running a series of benchmarks, the model worked well. It performed competitively across many materials property prediction tasks. Furthermore, it demonstrates strong scalability, suggesting that further improvements could be achieved by scaling the model and the data. Like many other models, CLOUD could take a crystal structure as input and output a predicted property. However, at the end of the day, it was still a mapping from structure to number—a very sophisticated black box.

This became particularly clear when we started looking at phonon-related properties such as the internal energy U and the constant-volume heat capacity Cv. Initially, our goal was simply to benchmark CLOUD on these datasets and demonstrate that the model could capture long-range structural effects. The datasets commonly used to train machine learning models only provide values at a single temperature, typically 300 K. If the model simply learns to predict the phonon-related properties from structures alone, something feels off. The temperature dependence—the physics itself—is missing from the picture.

There is a well-established theory for describing phonon-related properties: the Debye model, which was proposed by Peter Debye, who won the 1936 Nobel Prize in Chemistry. The model beautifully connects the microscopic vibrations of atoms in a crystal lattice to macroscopic thermodynamic quantities. The temperature dependence of phonon-related properties is well captured, along with another input variable, the Debye temperature Θ, which is a characteristic property dependent solely on the crystal structure itself. In principle, then, all we need is an ML model that can predict Θ from the structure. In order to realize this, another challenge arises: Debye temperature data are not readily available for training, and more importantly, Θ itself is not the physical quantity we ultimately care about. Ideally, we would like to train our model with U and Cv labels while handling the temperature dependence with a physics model. 

At this point, the path forward became clear: integrating our foundation model CLOUD with differentiable physics. As long as the physical equations can be implemented in a differentiable form, the loss computed from the target properties can be back-propagated through the physics model to update the machine learning model. In our framework, the output from CLOUD will be taken as the Debye temperature and then passed to the Debye model to predict the target, and the model will be trained end-to-end. We make the Debye model differentiable by using the Gauss-Legendre quadrature. This is our CLOUD-DEBYE framework, an example to showcase the power of integrating the scientific foundation model with differentiable physics. Rather than replacing physical laws, machine learning works alongside them, learning the intrinsic material properties, while physical laws govern how those properties translate into observable properties.

The work represents an exciting new frontier - merging two ideas we have been working on.  The fusion of differentiable physics and foundation models will open up new avenues for materials discovery, design and optimization.