At the beginning of my computational biology postdoc, I was living in a space between fields.

I knew microbiome research. I had worked with environmental data, microbial communities, and the strange richness of biological systems that cannot be understood by looking at one organism at a time. Microbiomes were appearing everywhere: in precision medicine and public health, in food safety and agriculture, in oceans and soils, in climate change, and in places as remote as deep-sea environments. Wherever there was life, there were communities. Wherever there were communities, there was the question of distance.

How different are two microbial worlds?

For microbiome researchers, UniFrac is one of the most beautiful answers to that question. It is not simply a formula. It is a way of saying that two communities should be compared through their evolutionary history. In that sense, UniFrac is almost like Euclidean distance for biological community studies. Euclidean distance gives geometry a way to measure space. UniFrac gives microbial ecology a way to measure evolutionary separation.

But the world had changed around this distance.

The microbiome field had become enormous. The Earth Microbiome Project, the American Gut Project, Qiita, the large resources from the Knight Lab, and many other efforts had turned microbiome science into a data-rich field. We were no longer thinking only about hundreds of samples. We were thinking about hundreds of thousands, millions, and eventually much more. The number of taxa, strains, and genomes was growing too. The tree was getting larger. The sample collections were getting larger. The simple act of comparing everything to everything was becoming a wall.

I was also carrying another background with me. My Ph.D. training was in sketching algorithms, MinHash, weighted sketching, randomized methods, and the mathematics of approximation. That training had changed the way I looked at data. I had learned to ask a different kind of question: not only “can we compute this exactly?” but “what is the smallest random object that still preserves the truth we need?”

For a while, I did not know where that training belonged in my postdoc. I was looking for the right problem, the one where microbiome science and sketching theory would not just sit next to each other, but actually become the same thing.

Then one night, I was studying the UniFrac equations.

I began rewriting them branch by branch. Not as a software engineer trying to optimize loops, and not only as a microbiome researcher reading a familiar metric, but as someone trained to recognize sets, weights, collisions, and estimators hiding inside equations.

And then it appeared.

Both unweighted and weighted UniFrac could be expressed through weighted Jaccard. For unweighted UniFrac, the branch lengths become weights on presence and absence over the phylogenetic tree. For weighted UniFrac, the connection appears through a Bray-Curtis transformation. The details mattered, but the feeling was immediate: UniFrac was already speaking the language of sketching. It had simply been written in the language of phylogenetic distance.

That was the spark.

If UniFrac was weighted Jaccard over tree branches, then weighted MinHash was not just a convenient approximation. It was the natural algorithm. The thing I had been trained to see was suddenly inside one of the most fundamental distances in microbiome research.

I mentioned the idea to Yuhan and Daniel, and both of them liked it. Rob saw the larger shape of it: this was a connection between two worlds that are usually separate, phylogenetic beta diversity and sketching algorithms. Manuel helped make sure the mathematics was right, that the proof was not only elegant but correct.

Then came the less romantic part, which in science is often the real romance.

I had to build it.

The equation had arrived in one night, but the software required months of focus. Many of the pieces I needed did not exist in Rust. I had to build supporting crates and infrastructure: succinct tree representation with succparen, weighted MinHash, DartMinHash, TreeMinHash, fast sample parsing, compressed sketches, parallel sketching, streaming computation, and GPU acceleration for the rate-limiting Hamming comparison step.

Rust mattered deeply. My interest in Rust was never only about taste. It was about speed, safety, control, and the feeling that large biological computation should not be fragile. Rust made it easier to parallelize tree parsing, sample parsing, sketching, and distance computation without losing confidence in the code. Life is short. Scientific life is even shorter. Researchers should not spend days waiting for something that better mathematics and better systems programming can make practical.

DartUniFrac became more than a faster implementation. It changed the shape of the computation. Exact UniFrac must repeatedly touch the tree across pairs of samples. DartUniFrac turns each sample into a fixed-length sketch. After that, the problem becomes comparing compact signatures by integer Hamming similarity. The tree, once too large to carry into every comparison, is distilled into sketches.

This also changed the answer to the ultimate question: can we be even faster?

Yes, up to a point. But beyond that point, the wall is no longer bad implementation. It is information theory, estimator variance, and hardware memory bandwidth. DartMinHash and ERS are already close to the current theoretical limits for unbiased weighted Jaccard estimation in sparse and dense regimes. Once the sketches exist, the bottleneck is mostly moving fixed-length integers through memory as fast as hardware allows. To go fundamentally beyond that, we may need new mathematics, new theory, or a different question.

That is part of why I like the project. It does not only say, “Here is faster software.” It says, “Here is where the problem changes form.”

The streaming-friendly PCoA was another step in that direction. A massive distance matrix should not always have to exist as one giant object in memory. If sketches can be stored, merged, queried, and compared block by block, then ordination and downstream analysis can also become more streaming-friendly. That opens a path toward fine-scale time series, fine spatial microbiome maps, and expandable computation where new samples can be added without starting over.

This is the future I see. Microbiome data will become more continuous, more spatial, more temporal, and more deeply connected to health, ecology, food systems, public infrastructure, and the changing planet. Under the deep learning and LLM framework, computational methods will not merely support biology. They will define what biological patterns can be learned at scale. Fast, accurate, sketchable distances can become training signals for future models of microbial ecology and evolution.

DartUniFrac began with uncertainty in a postdoc, then one late-night equation, then months of concentrated implementation. It became software, benchmarks, and finally a paper.

For me, the heart of the story is simple:

I did not make UniFrac less biological. I found the sketchable structure already inside it.

See paper here: 

https://www.nature.com/articles/s41587-026-03260-8