The analysis of 16S rRNA gene sequencing data involves several critical steps, including preprocessing, dereplication, chimera removal, and ultimately, clustering or denoising to infer biological sequences. To accurately assess the performance of each of these steps and ensure reliable results, the use of a complex mock community with a validated ground truth is essential for proper benchmarking. While large volumes of publicly available data exist and offer the advantage of being derived from real samples—unlike simulated data, which relies on prior assumptions—these datasets present a significant limitation: the true composition of the microbial communities is often unknown. This lack of a definitive ground truth poses a major challenge for comparative analyses, as it hampers our ability to rigorously evaluate the accuracy and effectiveness of clustering and denoising algorithms.
The mock community presented in this study comprises 235 bacterial strains representing 197 distinct species, providing a valuable and rigorous resource for the bioinformatics community. It offers an ideal framework for developers aiming to optimize their algorithms, as well as for analysts seeking to critically assess and benchmark existing 16S rRNA analysis pipelines. Notably, this same mock community has also been previously characterized at the shotgun metagenomic level by Gleb Goussarov, facilitating accurate metagenomic binning (see publication: https://link.springer.com/article/10.1186/s40793-022-00403-7). This dual availability at both amplicon and shotgun levels further enhances its utility as a comprehensive benchmarking standard for diverse microbial analysis workflow
In this study, we leveraged the complex mock community to conduct a head-to-head comparison of clustering and denoising approaches—specifically, Operational Taxonomic Units (OTUs) and Amplicon Sequence Variants (ASVs). This direct comparison allowed us to systematically highlight the strengths and limitations of each method. We believe that the robust design of our benchmarking framework, combined with the utilization of this complex mock community, provides a solid foundation for evaluating 16S rRNA analysis algorithms. Moreover, this framework offers a scalable model that could be extended to encompass entire pipeline comparisons in future studies.
Our comprehensive benchmarking framework, along with all datasets, detailed analyses, and key insights, is freely accessible https://environmentalmicrobiome.biomedcentral.com/articles/10.1186/s40793-025-00705-6.Additionally, the mock community dataset—available under accession number PRJNA975486—serves as a valuable resource for both bioinformatics algorithm development and rigorous performance evaluation.