There’s a hoo-har in psychology right now about replication. Spurred on by some high profile fraud cases, awareness of the structural biases surrounding publication and perennial rumblings about statistical malpractice, many are asking if the effects reported in the literature are real. There are some laudable projects aimed at improving best practice in science – journals of null results, pre-registration for experiments, the Center for Open Science (see previous link), but it occurs to me that all of this ignores an important bit of context. At the risk of stating the obvious: you need to build in support for replications only to the extent that these do not happen as part of normal practice.
Cumulative science inherently supports replication. For most of science, what counts on news is based on what has been done before – not just in an abstract theoretical sense, but in the sense that it relies on those results being true to make the experiments work. Since I’m a psychologist, and my greatest expertise is in my own work, I’ll give you an example from this recent paper. It’s a study of action learning, but we use a stimulus control technique from colour psychophysics (and by ‘we’, I really mean Martin, who did all the hard stuff). As part of preparing the experiment we replicated some results using stimuli of this type. Only because this work had been done (thanks Petroc!) could we design our experiment; and if this work didn’t replicate, we would have found out in the course of preparing for our study of action learning. Previously in my career I’ve had occasion to do direct replications, and I’ve almost always found the effect reported. I haven’t agreed with the interpretation of why the effect happens, or I’ve found that my beliefs about the effect from just reading the literature were wrong, but the effect has been there.
It is important that replication is possible, but I’ve been bemused that there has been such a noise about creating space for additional formal replications. It makes me wonder what people believe about psychology. If a field was one where news was made by collecting isolated interesting phenomena, then I there would be more need for structures to support formal replication. Should I take the reverse lesson from this – the extent to which people call for structures to support formal replication is evidence of the lack of cumulative science in psychology?
2 replies on “Surely the hoo-har about replication could only concern a non-cumulative science?”
You’re right, but I suppose in defence of psychology, the problem is not so much that nothing in psychology builds cumulatively on other stuff (you cite one example & there are others) – the problem is just that today a great many psychologists are more interested in doing ‘novel’ experiments, than ones that build on others – perhaps because they get more cited…
But, if your replication just didn’t work? You do it, the results show nothing, or nothing that is interpretable. Maybe do it again (just in case). Or a variant thereof. Then nothing.
Shrug and move on? Or undertake some work on showing that the effect isn’t there so you can publish? (I’ve seen two of those).
Then, someone else, who don’t know your failed attempt at replicating (because the scientific community is too large for everybody to be up on the gossip), undertakes the work only to, well, not replicate. Into the filedrawer it goes.
Wasted time, and no way of finding out that this is a wobbly effect that perhaps is a type I. Meanwhile, the reported results figures in arguments for something else, that (perhaps) works.
(I’m just going through an advanced book on Social Cognition, in a teaching capacity, and finding Stapel articles that are now known fakes. They are not the only ones cited as evidence for some phenomenon, but they are there).
I have a modest filedrawer on cognition and emotion things. Some of it is hard to interpret, hence in the file-drawer. My first set of five experiments basically suggests that emotional states do not facilitate perception of emotion congruent clear facial expressions, but that is nothing that can be published. (Subsequent work suggests it works on ambiguous faces).
From what I understand, there are file-drawers around who look at attention to threatening stuff that doesn’t really find anyting. (I have those, although it shows something, but I can’t for my life put it together into a manuscript, and by now it is pushing 10 years old). Meanwhile, my intro psych book thought that attention to threat was so sexy my undergrads think this is how it works.
Just, to give examples from my meager experience.