MJgraphics/Shutterstock.com

Two more classic psychology studies just failed the reproducibility test

The scientific method in action.

FIONA MACDONALD
26 SEP 2016
 

For years now, researchers have been warning about a reproducibility crisis in science - the realisation that a lot of seminal papers, particularly in psychology, don't actually hold up when scientists take the time to try to reproduce the results.

Now, two more key papers in the psychology have failed the reproducibility test, serving as an important reminder that many of the scientific 'facts' we've come to believe aren't necessarily as steadfast as we thought.

 

To be fair, just because findings can't be reproduced, it doesn't automatically mean they're wrong. Replication is an important part of the scientific method that helps us nut out what's really going on - it could be that the new researchers did something differently, or that the trend is more subtle than originally thought.

But the problem is that, for decades now, the importance of replicating results has been largely overlooked, with researchers usually choosing to chase a 'new' discovery rather than fact-checking an old one - thanks to the pressure to publish exciting and novel findings in order to secure jobs.

As John Oliver said earlier this year: "There's no Nobel prize for fact-checking."

That's brought us to the 'crisis' we're in now, where most papers that are published can't be replicated. Last year, the University of Virginia led a new Reproducibility Project that repeated 100 experiments... with only one-third of them successfully being replicated - although this study has since been criticised for havings its own replication errors.

The two latest examples are widely cited papers from 1988 and 1998.

The 1988 study concluded that our facial expressions can influence our mood - so the more we smile, the happier we'll be, and vice versa.

 

The 1998 study, led by Roy Baumestier from Case Western University, provided evidence for something called ego depletion, which is the idea that our willpower can be worn down over time.

The latter assumption has been the basis of a huge amount of follow-on psychological studies, but now Martin Hagger from Curtin University in Australia has led researchers from 24 labs in an attempt to recreate the seminal paper, and found no evidence that the effect exists.

His results have been accepted for publication in the journal Perspectives in Psychological Science in the coming weeks.

The facial expression replication attempt follows much the same trend.

In the original paper, researchers from Germany asked participants to read The Far Side comics by artist Gary Larson, with either a pen held between their teeth (forcing them into a smile) or between their lips (replicating a pout).

The team found that people who smiled found the comics funnier than those who were pouting, leading the researchers to conclude that changing our facial expression can change our moods, something known as the facial feedback hypothesis.

But when a team of researchers at the University of Amsterdam in the Netherlands conducted the same experiment - even using the same '80s comics - they failed to replicate the findings "in a statistically compelling fashion".

"Overall, the results were inconsistent with the original result," the team conclude in Perspectives in Psychological Science - a separate paper to the ego depletion replication, but also due to be published in a few weeks.

Again, that doesn't necessarily mean that the original result wasn't accurate - nine out of the 17 Dutch labs that attempted to recreate the experiment actually reported a similar result to the 1988 study. But the remaining eight labs didn't, and when the results were combined, the effect disappeared.  

"[T]his does not mean the entire facial feedback hypothesis is dead in the water," writes Christian Jarrett for the British Psychological Society's Research Digest.

"Many diverse studies have supported the hypothesis, including research involving participants who have undergone botox treatment, which affects their facial muscles."

The results could be due to a number of other variables - like, maybe people today don't find The Far Side funny anymore. And the Dutch study also used psychology students, many of whom would have been familiar with the 1988 paper, which could have skewed the results.

Only more investigation will help us know for sure.

But in the meantime, all this hype over the reproducibility crisis in the media lately can only be a good thing for the state of science. 

"It shows how much effort and attention has gone towards improving the accuracy of the knowledge produced," John Ioannidis, a Stanford University researcher who led a 2005 reproducibility study, told Olivia Goldhill at Quartz.

"Psychology is a discipline that has always been very strong methodologically and was at the forefront at describing various biases and better methods. Now they are again taking the lead in improving their replication record."

One positive that's already emerged is a discussion about pre-registering trials, which would stop researchers tweaking their results after they've been collected to get a more exciting results.

And hopefully, the more people talk and think about replicating results, the better the public will get at thinking critically about the science news they read.

"Science isn’t about truth and falsity, it’s about reducing uncertainty," Brian Nosek, the researcher behind the Reproducibility Project, told Quartz.

"Really, this whole project is science on science: researchers doing what science is supposed to do, which is be skeptical of our own process, procedure, methods, and look for ways to improve." 

More From ScienceAlert

Giraffes are heading towards a "silent extinction", as populations plummet
54 minutes ago