In this Behind the Paper post, we look at how we identified and corrected an error while publishing our paper, A framework for assessing 16S rRNA marker-gene survey data analysis methods using mixtures (Olson et al., 2020). During peer-review, we discovered an error in our sample labels when generating a figure requested by a reviewer. Diligent documentation, developing our analysis in a reproducible manner, and the peer review process allowed us to quickly identify, verify, and correct the error. We hope that the lessons learned here will help other scientists prevent, identify, and correct similar errors.

We described our initial mixture and sequencing experimental design in a study protocol document. We developed the initial protocol to minimize the impact of potential sample cross-contamination and propagation of pipetting biases (Fig. 1A). However, the execution of the protocol was not practical. We revised the protocol based on feedback from the co-authors performing the laboratory experiments. Unfortunately, the sample sheet was not updated to reflect these changes. Due to our collaborator’s diligent documentation of the laboratory procedure, we have photographic evidence of the actual sample layout (Fig 1B).

A reviewer requested a PCA analysis of all the samples and titrations to show that the titrations were behaving as expected (Fig. 2A). We generated similar figures during our initial exploratory data analysis but only on individual samples and the expected titration trends were observed. When we plotted the titration series for all five individuals on a single plot, we identified an error in our sample sheet. The plot revealed that the unmixed post-exposure samples were incorrectly grouping for four of the five individuals. From the plot, we hypothesized that we assigned the unmixed post-exposure samples to the wrong individuals.  Our hypothesized sample IDs were consistent with the plate layout photo taken in the lab. (Fig. 1B). We corrected the sample IDs and re-generated the PCA analysis, resulting in the expected trend with samples grouping by individual and titration (Fig. 2B). By developing our analysis pipeline in a reproducible manner under version control we quickly reproduced the study results and evaluated the impact of the error on the study conclusions. 

Figure 1: Titration series (A) experimental design plate layout and (B) image of the titration series plate layout for the titration series taken after sample mixing and prior to the initial 16S rRNA PCR.

Figure 2: PCA of benchmarking dataset with color indicating titration and shape is individual or subject for the (A) uncorrected sample labels and (B) corrected sample labels. Before sample label correction the unmixed pre-treatment samples, titration 20, are grouped with the incorrect individual for four of the five individuals, E01JH0038 was not effected by the sample sheet error. Whereas, after sample correction the pre-treatment samples were correctly grouped.

It is our hope that documenting and reporting this error will help prevent other researchers from making these same mistakes. While the peer-reviewer process succeeded in identifying the error prior to publication, diligent documentation and a reproducible analysis pipeline allowed us to quickly correct the error and re-run the full analysis pipeline. Let this experience serve as a reminder that mistakes happen in science, and so, it is important to consider preventative measures as a part of any initial study design, perform analyses like PCA that can identify potential errors, and include sample label verification in the protocol.

Reference

Olson, N.D., Kumar, M.S., Li, S. et al. A framework for assessing 16S rRNA marker-gene survey data analysis methods using mixtures.. Microbiome 8, 35 (2020). https://doi.org/10.1186/s40168...