The study conceived from a newly established genomics laboratory

Single-cell RNA sequencing (scRNA-seq) is finding increasing applications in cancer mutation detection, molecular diagnostics, and design of personalized medicine therapies. This technology is evolving rapidly, with many different platforms and protocols available. However, there has been no consensus on the best bioinformatics methods to integrate diverse scRNA-seq data from different platforms and labs, and increasing recognition that analytical reproducibility and measurement assurance are critical to realize the full potential of these methods¹. In the summer of 2017, as an outgrowth of the US Food and Drug Administration (FDA)-led Sequencing Quality Control (SEQC-2)², the idea of benchmarking the “reproducibility” of different scRNA-seq technologies and data was conceived at the newly-established Center for Genomics directed by Wang at Loma Linda University (LLU), which was one of the major testing sites in the SEQC-2 consortium phase-2 study. We later became aware that two other groups under the Human Cell Atlas (HCA) consortium, Holger’s lab at CNAG and the Levin lab at Broad had initiated their studies in early 2017, with a somewhat different primary goal of benchmarking wet-bench scRNA-seq protocols^3,4.

The intention of determining the scRNA-seq detection sensitivity with spiked in cells

For our multi-center scRNA-seq benchmarking study, we chose the same two well-characterized dissimilar cell lines used for the FDA SEQC-2 multi-center and multi-platform WGS & WES study². The very first set of scRNA-seq data (10x and C1) for the two cell lines generated at LLU through a training were presented at the MAQC Society annual meeting in February 2018, in Shanghai, China. We finalized the study design in March 2018, and recruited three more sites to our study: the National Cancer Institute (NCI) on 10x, the US FDA using the Fluidigm C1_HT, and Takara Bio USA (TBU) on the ICELL8 platform. One intriguing thing worthy to recall was our intention of determining the scRNA-seq detection sensitivity, i.e., by spiking in some cancer cells into the B cells using the 10x Genomics platform. We initially set to spike 1% of breast cancer cells into B cells in separate single-cell captures and library constructions to test this capability. However, we had to change to spiking-in 5-10% (10x) cancer cells into B cells instead, because of the large variations in cell counting techniques, which made it impractical to carry out a 1% spike-in. This change, nevertheless, turned our study design into a robust one, which was proved later to be critical for us in comparing different bioinformatics methods in integrating scRNA-seq data from different centers/platforms.

Catching a moving target—fluid and fast-growing bioinformatics methods for scRNA-seq data

Data collection was astonishingly quick and smooth; we completed almost all the single-cell captures and sequencing by April 2018 at all sites. Thanks to weekly Zoom conference calls, which sometimes went for 2-3 hours, we simultaneously achieved a solid understanding of the data integration and other bioinformatic methods available at the time. This was facilitated by the publication of two batch-effect correction and data integration methods, MNN and CCA, in Nature Biotechnology^5,6 at about the same time. We were amazed that, compared with CCA (which overcorrected the batch effect), MNN was able to group and separate two cell types clearly when applying these methods to our multi-platform cross-center scRNA-seq datasets (see inserted movie). We found that data from mixtures of cell types were critical for MNN to group the cell types correctly, a discovery also reported by Ritchie’s lab at the University of Melbourne in Australia⁷. The year of 2018 seemed a golden year for single-cell sequencing data analysis, with many more algorithms developed, including BBKNN, Harmony, and Scanorama, which we also evaluated.

Establishing sustainable reference samples, reference dataset and reference methods

Our benchmark study has produced well-characterized reference materials, 20 openly available scRNA-seq datasets⁸, and detailed methods. We not only evaluated the influence of technology platform, but also investigated the effects of sample composition and bioinformatic methods. Previous studies using heterogeneous mixtures containing different cell types provided useful insights into bioinformatics methods, but no non-mixture cells/data were captured^3,4,7. However, studies restricted to mixtures of cell types cannot examine the ability of a method to eliminate variability due purely to technical factors, important to evaluating a method’s capacity to group similar cells together, and also—independently—assess the ability of a method to separate dissimilar cells correctly. We found that analyzing both mixtures of dissimilar cells in various proportions and un-mixed samples of the two distinct cell lines provided important additional insights. For example, the widely used Seurat v3 method excelled at grouping similar cells together, but it over-corrected—completely failing to separate B cells from breast cancer cells-when large proportions of two dissimilar cell types were analyzed. Also, it may be difficult to replicate the cellular sample used in Mereu et al. & Ding et al. (which focused primarily on wet-bench protocols, not bioinformatics methods)^3,4 for confirmatory or further exploratory evaluations. In contrast, our study was not only empowered by a mixology design but was also further strengthened by the inclusion of non-mixture sample/data captured separately. Moreover, the reference cell lines we used are commercially available for future benchmarking studies. Another unique advantage of our standard reference samples is the availability of massive cross-platform deep WGS and WES data² which will be a valuable resource for benchmarking future single-cell WGS or proteomics technologies. Our findings offer practical guidance for selecting the combination of technology platform and bioinformatic methods best suited to the scientific question addressed.

This blog reflects the views of the contributors/authors for this post, and should not be construed to represent FDA’s views or policies.

Our paper: https://doi.org/10.1038/s41587-020-00748-9

1. Plant, A.L., Locascio, L.E., May, W.E. & Gallagher, P.D. Improved reproducibility by assuring confidence in measurements in biomedical research. Nat. methods 11, 895-898 (2014).
2. Xiao, W. et al. Towards best practice in cancer mutation detection with whole-genome and whole-exome sequencing. Nat. Biotechnol. 39, 2021 (in press)
3. Mereu, E. et al. Benchmarking single-cell RNA-sequencing protocols for cell atlas projects. Nat. Biotechnol. 38, 747–755 (2020).
4. Ding, J. et al. Systematic comparison of single-cell and single-nucleus RNA-sequencing methods. Nat. Biotechnol. 38, 737-746 (2020).
5. Haghverdi, L., Lun, A. T. L., Morgan, M. D. & Marioni, J. C. Batch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors. Nat. Biotechnol. 36, 421–427 (2018).
6. Butler, A., Hoffman, P., Smibert, P., Papalexi, E. & Satija, R. Integrating single-cell transcriptomic data across different conditions, technologies, and species. Nat. Biotechnol. 36, 411–420 (2018).
7. Tian, L. et al. Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments. Nat. Methods 16, 479-487 (2019).
8. Chen, X. et al. A multi-center cross-platform single-cell RNA sequencing reference dataset. Preprint at bioRxiv https://doi.org/10.1101/2020.09.20.305474 (2020).