The reproducabilty crisis and gravitation towards concensus

To truly transform a research field, a discovery must be reproducible. Decades after unveiling the noxious influence of ocean acidification on marine life, panic ensued over its reproducibility and credibility. We assess whether researchers of ocean acidification can regain confidence and verve.
Published in Ecology & Evolution
The reproducabilty crisis and gravitation towards concensus
Like

History does not need to repeat itself perfectly to offer lessons to the present. Scientific discovery has been shown to be essential for human progress, driving innovation and addressing critical challenges that shape our society, culture, and our future. Yet, a most striking feature of research is how little new discoveries are truly transformative. Many discoveries wither and disappear over time.

 In our field, the marine science community was galvanised by the demonstration of how ocean acidification impairs shell-building life, which has profound implications for life on the planet. This field was so transformational to marine science that a journal editor observed that ocean acidification “…has probably been the most-studied single topic in marine science in recent times”.  Yet, with the episodic activity and accompanied spectacle of probes into the reproducibility of other researchers’ findings, the field of ocean acidification lost confidence.

 Up front, we acknowledge the controversy that erupted over a set of failures to reproduce key behaviours of tropical fish to ocean acidification. We also appreciate the successful execution of an Editor-in-Chief to “Applying organized scepticism to ocean acidification research”. Now, we believe that sufficient time has passed to allow the field to look objectively into its knowledge base and rebuild confidence for the future. We did so by assessing one of the largest databases in ocean acidification research – the calcification of shell-building life - the subject that ignited the transformation of marine science.

With some trepidation towards the labour-intensive effort ahead, we assembled an exhaustive meta-analysis of calcification studies over the last 23 years. Unsurprisingly, the pioneering studies were often marked by large negative effects of ocean acidification, but the size of these effects declined over the years. Essentially, the initiation of this field was based on reproducible effects. We acknowledge notable exceptions that seem to highlight potential insights into less studied processes, such as compensatory dynamics. Yet, the key point is that the negative effects remain and are frequently observed to this day. Whilst our paper discusses the possible reasons for the decline in effect sizes, which are generic to most research disciplines, it is inescapable to concede that ocean acidification can impose harmful effects on shell-building life. We believe that researchers of ocean acidification can have confidence in their research community; researchers have been contributing robust data. 

 Some may find that the last statement is not sufficient – that we are waving away an underlying problem. So, let us focus on the issue causing alarm and anxiety of a field in crisis. In our case, the results from the pioneers appeared impressive, but subsequent attempts by other researchers to replicate the same findings become less impressive as they assess how the results reproduce in their own system. This is a repeated pattern in science; the failure to reproduce the initially impressive effects of pioneering studies and eventually declines is known as the “decline effect”. The decline effect is frequently experienced in many disciplines, such as medicine, neuroscience and genomics. The causes are so diverse which makes generalisations hard. One feature they have in common is how their natural complexity buffers or boosts the measured response whose detectability varies with observational or experimental method. In medical research, initial clinical trials for drug development often overestimate the efficacy of treatments and then realistic estimates ensue as testing intensifies. Neuroscience research often involves novel analysis, yet early exciting findings diminish as methods improve. In genomics research, pioneering studies can link genetic variants with diseases, but their associations weaken as more variants are tested. Biological and ecological studies are particularly subject to the decline effect because the myriad of species in an ecosystem has infinite ways to interact under continuously varying environmental conditions in time and space. Yet fundamentally, if the phenomenon is indeed general, it should be detected regardless of context and method.

 It is commonly agreed that to be truly transformative, a discovery must not only be reproducible, but also be of sizable importance. In other words, the phenomena should not only be frequently observable, but also be sufficiently large that it is hard to ignore in the research domain. It will not only account substantially for the behaviour of a natural system, but also be repeatedly observable across widely varying contexts. Yet, frequently, pioneering discoveries cannot be replicated. As time passes, discoveries lacking empirical support wither and eventually disappear, while those gaining empirical support become transformative and are assimilated into a central theory. It is replication across the research communities that distinguishes widespread acceptance or rejection of the discovery within a scientific community. It is hard to overstate the interpretive value provided by replication. 

In the case of ocean acidification on calcification, the discoveries of the past 23 years have been reproducible. We argue that they may be reliably incorporated into theories of change or adjustment to future climate (e.g. resistance, resilience and adaptation), including their underpinning mechanisms that cope or eventually fail (e.g. compensatory dynamics, epigenetics and gene selection). Our vision is that as time passes, research into ocean acidification will be increasingly incorporated into research on ocean warming, heatwaves and other forms of chemical and physical change to our oceans. Such combinations might be our future reality, but they remain substantially understudied. Their combinations are likely to have context-dependencies that will initially confuse the field as various combinations and contexts are likely to suppress or exaggerate responses. Initially, they too will bedevil reproducibility.

 This history is likely to repeat itself in our lifetimes. We will see the gravitational pull of the decline effect across many new discoveries. Yet, when new decline effects are revealed, we need not lose confidence that something is amiss. On rare occasions that might be true, but panic is unwarranted. The failure to reproduce large effects in pioneering studies may not represent a crisis, but most often a normal part of theory development. Over time, discoveries that are difficult to reproduce or reproduce over a narrow range of contexts will be surpassed. It is only through our collective replication that well-supported discoveries gain widespread acceptance.

Science is done by people and is a social enterprise. Yet, like most human endeavours, it is subject to imperfections inherent in human nature. Humans are prone to confirmation bias and tend to seek information that confirms their existing beliefs or knowledge. We have a conservative side. This is why creativity and originality are needed to continually challenge our current understanding. Scientific discovery is disruptive because it confronts us to revisit what we already know. We cannot afford to discourage creativity and originality. Without them, we would merely be repeating existing experiments and theories. Indeed, openness to unfamiliar ideas and methods is a valued trait of a scientist. Yet, it is the scientific method that remains one of the most powerful tools for acquiring reliable knowledge of the natural world.

 We believe that the “growing pains” of maturation of ocean acidification research is not so distinctive from other disciplines, their toil, setbacks and legacies. What seems timeless is the potency of the research communities to collectively trial a new discovery in a way that creates a process of replication. Indeed, the global potency of researchers employing unique and shared approaches within their specific study systems facilitates the replication required to achieve global validation.

 Replication is a powerful force.

Please sign in or register for FREE

If you are a registered user on Research Communities by Springer Nature, please sign in