The Social Sciences Replication Project in Nature Human Behaviour is here: https://go.nature.com/2C4M5VC

Replication studies are both valuable and necessary for the progression of science, as their results provide empirical evidence for those conclusions that can – and should– be used as the foundation for future research. Though this sentiment is shared by many in the academic community, replication studies are often both hard to publish and controversial. This may be due to a lack of consensus as to the definition of replication: What does a “successful” versus “failed” replication really mean?

Recently, Nature Human Behaviour published a paper that described the results of direct replications of 21 experimental studies in the social sciences that were published in the journals Nature and Science between 2010 and 2015. When I first read the paper, I was struck by the fact that there are different indicators of replicability, and that a different number of papers “successfully” replicated depending on the indicator used. The variability, however, was not huge (it ranged from 57% - 67%) but did show that the criteria that are set to determine whether a study is replicable or not will necessarily influence the likelihood of whether we conclude that results replicate. Put another way, the variability in the number of studies that did or did not successfully replicate show us that how we define what constitutes a successful replication will, in part determine whether or not a study is considered replicable. This points to the importance of defining replication in a way that helps research move forward, rather than pointing to failures in the scientific method. 

To me, the point of replication is to determine whether or not the results are robust. However, as I discovered when I read the Correspondences associated with the Social Science Replication Project paper, there are many other factors that could have influenced whether or not the results replicated. For example, the analyses presented by David Rand suggest that prior experience could have influenced the current results and Anuj Shah and colleagues suggest that it may be the specific experiment, not the general conclusions, that do not replicate. These points and those raised in the other associated Correspondences point to the importance of considering replication within the broader context surrounding each original study and factors that may have changed over time.

Working  with the social science replication team and the authors of the associated correspondences has been an incredible learning experience, as it has shown me that replication is not a binary term and that, to move forward, we have to interpret the results of replication studies in context. Even more importantly, perhaps, it has shown me the incredible opportunities for future research that these replication studies can spark: Are there common features shared by those studies that replicate? For example, do they all use large and representative samples? And are their commonalities shared by those studies that do not replicate? For example, are they more grounded in a specific time period and unlikely to generalize to a new context?

Taking replication forward means considering the results of replication studies such as the one we recently published alongside the voices of the original authors and the relevant literature. This was possible at Nature Human Behaviour because we have professional editors who are passionate about taking science forward and have the time to invest in truly understanding the subtleties of replication studies. We hope that, like us, the field sees them as an opportunity to move science forward rather than to expose its failures.