Building a robust tool for processing metagenomes, one error at a time

Bioinformatics analysis is a central piece of modern life science research, including when working on the microbiome. Our recent paper in Microbiome presents a set of microbiome analysis pipelines for metagenomics studies.
Published in Microbiology
Building a robust tool for processing metagenomes, one error at a time

Share this post

Choose a social network to share with, or copy the shortened URL to share elsewhere

This is a representation of how your post may appear on social media. The actual post will vary between social networks

Read the paper

BioMed Central
BioMed Central BioMed Central

NG-meta-profiler: fast processing of metagenomes using NGLess, a domain-specific language - Microbiome

Background Shotgun metagenomes contain a sample of all the genomic material in an environment, allowing for the characterization of a microbial community. In order to understand these communities, bioinformatics methods are crucial. A common first step in processing metagenomes is to compute abundance estimates of different taxonomic or functional groups from the raw sequencing data. Given the breadth of the field, computational solutions need to be flexible and extensible, enabling the combination of different tools into a larger pipeline. Results We present NGLess and NG-meta-profiler. NGLess is a domain specific language for describing next-generation sequence processing pipelines. It was developed with the goal of enabling user-friendly computational reproducibility. It provides built-in support for many common operations on sequencing data and is extensible with external tools with configuration files. Using this framework, we developed NG-meta-profiler, a fast profiler for metagenomes which performs sequence preprocessing, mapping to bundled databases, filtering of the mapping results, and profiling (taxonomic and functional). It is significantly faster than either MOCAT2 or htseq-count and (as it builds on NGLess) its results are perfectly reproducible. Conclusions NG-meta-profiler is a high-performance solution for metagenomics processing built on NGLess. It can be used as-is to execute standard analyses or serve as the starting point for customization in a perfectly reproducible fashion. NGLess and NG-meta-profiler are open source software (under the liberal MIT license) and can be downloaded from or installed through bioconda.

The goal of NG-meta-profiler is to take metagenomes as inputs and produces estimates of either taxonomic or functional abundances. At a high-level, we use a well-validated approach to metagenomic analysis: use a pre-annotated gene catalogue as a reference against which to align metagenomic reads [1-3]. The big innovation is in the way the tool was designed and is presented to the users.

What do we want from our computational tools?

On the one hand, we want standard pipelines that embody best practices and enable comparisons between different studies and datasets, we want them to be easy to install and use by individuals without a strong background in bioinformatics. On the other hand, we want the flexibility to adapt to the specificity of each dataset and expert bioinformaticians will want to tweak and combine tools to address different contexts and requirements.

We propose a solution to these seemingly conflicting goals by presenting a set of standard pipelines that can be easily adapted while maintaining reproducibility. The central concept behind the work is a novel programming language for processing NGS data called NGLess which was used to write these standard pipelines.

NGLess cartoon

Cartoon view of how NGLess is used for building metagenomics profilers

NG-meta-profiler and NGLess are just now being published as a journal article, but the tools themselves have been in development for several years and were always available as open-source software. In fact, the initial intuition that this could be a good idea first came to me back in 2012! Looking back at my mail archives, I found an email from me to co-author Ana Teresa Freitas from September 21st 2012 with some initial thoughts on the idea as a follow-up to a conversation we had earlier.

Then, the project proceeded through three phases:

  1. Proof-of-concept (2013-2014).

  2. Internal use, maturation (2015-2016).

  3. Release to wider public (2017-present).

The initial goal was to design a simple programming language and write a piece of software that would solve some toy problems. This work was the bulk of Paulo Monteiro’s MS Thesis, which he submitted at the end of 2014.

At this point, most of the ideas were in place and we had proved that the concept could work to process data, but the tool was slow and would often produce hard-to-understand error messages. After improving the performance, we started using the tool internally for our own projects (e.g., [4]). The NGLess interpreter became more robust with time and started to produce better error messages. This was a slow process as behind each little improvement there was often a confusing debugging session. For example, a malformed FastQ file could cause an “Out of memory” error. Naturally, we would then attempt to run the analysis in a large-memory machine before realizing what was going on.

Bad error message
The type of error we worked hard to get rid off.

Around 2017, we started encouraging others to use the tools and receiving feedback. We realized that the way the software was built caused issues with some types of HPC clusters (even though it had run fine on our systems). We received further reports of confusing error messages and improved those. We realized that there were some inconsistencies that confused new users and streamlined the interface.

Individually, all of these were minor changes, but collectively these hundreds of small improvements make all the difference between the finicky proof-of-concept we had in 2014 and a tool we could proudly recommend to others. So, in 2018, we finally decided it was time to submit it as a manuscript to a journal, and you can now read about it at Microbiome.


[1] Kultima JR, Coelho LP, Forslund K, Huerta-Cepas J, Li SS, Driessen M, Voigt AY, Zeller G, Sunagawa S, Bork P. MOCAT2: a metagenomic assembly, annotation and profiling framework. Bioinformatics. 2016 Apr 8;32(16):2520-3.

[2] Huerta-Cepas J, Forslund K, Coelho LP, Szklarczyk D, Jensen LJ, von Mering C, Bork P. Fast genome-wide functional annotation through orthology assignment by eggNOG-mapper. Molecular biology and evolution. 2017 Aug 1;34(8):2115-22.

[3] Ugarte A, Vicedomini R, Bernardes J, Carbone A. A multi-source domain annotation pipeline for quantitative metagenomic and metatranscriptomic functional profiling. Microbiome. 2018 Dec;6(1):149.

[4] Schmidt TS, Hayward MR, Coelho LP, Li SS, Costea PI, Voigt AY, Wirbel J, Maistrenko OM, Alves RJ, Bergsten E, de Beaufort C. Extensive transmission of microbes along the gastrointestinal tract. eLife. 2019 Feb 12;8:e42693.

Please sign in or register for FREE

If you are a registered user on Research Communities by Springer Nature, please sign in

Follow the Topic

Life Sciences > Biological Sciences > Microbiology
  • Microbiome Microbiome

    This journal hopes to integrate researchers with common scientific objectives across a broad cross-section of sub-disciplines within microbial ecology. It covers studies of microbiomes colonizing humans, animals, plants or the environment, both built and natural or manipulated, as in agriculture.

Related Collections

With collections, you can get published faster and increase your visibility.

Retroviruses & the Microbiome

The focus of this issue is to move beyond associations to mechanisms and functional interactions, including both replication-competent and endogenous retroviruses, human and non-human hosts, and implications for homeostasis, disease and/or therapeutics.

Publishing Model: Open Access

Deadline: Ongoing

Food security and sustainability from a microbiology perspective

This collection includes various aspects of maintaining or enhancing food production, sustainability and security from a microbiology perspective.

Publishing Model: Open Access

Deadline: Ongoing