Since the Enlightenment, advances in science have improved the world in innumerable ways: from Jonas Salk’s polio vaccine to Norman Borlaug’s Green Revolution. Dedicated researchers and technicians have worked tirelessly and sometimes slowly, applying the scientific method to address society’s most pressing problems. Such advances require careful attention to detail and application of the scientific method: a systematic approach to epistemology that uses observable, testable, repeatable, and falsifiable experimentation to understand how nature behaves.
A new study shows that in the social sciences at least, that method has too often been misapplied—or maybe even ignored. The Social Sciences Replication Project—the joint effort of 24 professors, graduate students, and research fellows—was established because of increasing concerns about reproducibility in social science research.
The project’s participants describe their work, which is now complete:
“We will replicate 21 experimental studies in the social sciences published in Nature and Science in 2010-2015. These papers were objectively chosen because they were published in these two high-profile journals in this time period, they share a common structure in testing a treatment effect within or between subjects, they test at least one clear hypothesis with a statistically significant finding, and they were performed using students or accessible convenience samples. We plan to conduct the replications between September 2016 and September 2017.”
The replicators performed all their research in an open and transparent way, intentionally increasing the size of each study to ensure it was methodologically sound. The results were startling. The replicators were only able to reproduce the results of 13 out of 21 studies—62 percent.
Ed Wong, a science reporter at the Atlantic, pointed out a disturbing similarity between the studies that didn’t replicate: they were all sensational. This suggests that some scientists, at least, are pursuing headlines instead of the advancement of knowledge.
“[S]everal of the studies that didn’t replicate have another quality in common: newsworthiness. They reported cute, attention-grabbing, whoa-if-true results that conform to the biases of at least some parts of society. One purportedly showed that reading literary fiction improves our ability to understand other people’s beliefs and desires. Another said that thinking analytically weakens belief in religion. Yet another said that people who think about computers are worse at recalling old information—a phenomenon that the authors billed as ‘the Google effect.’ All of these were widely covered in the media.”
The Social Science Replication project serves an important function. It reminds us that the advancement of knowledge depends upon a sound and consistent process, painstaking attention to detail, and commitment to the search for truth.