I am teaching a course, where I ask students to critically review papers reporting psychology experiments. It is making me question how you try and teach a skill as fundamental as critical thinking. I am trying to do it by example, walking the students through my reading of each paper, showing them what I do and consider when looking at it, and then providing them with a model answer (which is identical in structure to the final coursework I will require of them).
Reading a few practice questions students have handed in, I’m struck that there is a habit I want my students to acquire, which they haven’t quite got yet, and which I don’t even know the name of. Which I why I write this here, to ask you – Kind Interwebs – if you know what it is I am about to talk about and how I can convey it best to people taking my course.
This habit is that of asking the skeptical follow up questions to every proposal they make. So, for example, most undergraduate psychology students will have up their sleeve a well rehearsed list of possible flaws in experiments. Things like: was the sample representative of the population? Are there confounds in the experiment which prevent you cleanly inferring the causation?
It is nice, but inadequate, to write a review of a paper listing flaws like this. Doing so does not constitute a useful or interesting critical review (or, on my course, a gradeworthy one).
What I would like to encourage is my students to go the extra mile and, having spotted a potential flaw, assess it for plausibility, and consider what it would mean if the flaw was a significant one (i.e. how does it limit our interpretation of the current experiment, and what does it mean for future possible experiments?).
Here’s a concrete example: one paper I am teaching is one of my own, that looked at how students’ use of a course wiki predicted final exam score. A student suggested that because students knew their wiki use was being monitored, demand effects may have played a role (demand effects are a classic psychology experiment confound: participants distort their behaviour according to what they think you want to find). Now this is fair enough, but there are a number of follow up questions.
What is required for this to be the case? That students were able and willing to alter their final exam score based on their wiki use, but not because of it, perhaps. This seems implausible
What is implied if it is the case? If the demand effect did hold, would it even mean that the wiki use wasn’t effective? For example, we might decide that since students can’t easily score higher on exams at a whim, even an effect via demand was an effect worth having
How could I test if it is the case? Demand effects may hold, but how could we tell if they do hold?
What if it isn’t the case? What are the differences between the two situations Imagine two worlds with and without demand effects. What are the crucial differences between them, and what implications do these differences have for our experiment interpretation or further research? If there are no major differences, maybe we don’t need to worry about demand effects.
I pick demand effects because I wanted to use a specific example, but my aim is to encourage students to deploy these questions about every possible flaw or improvement that they suggest. My question today, though, is is there a general principle which students could follow to guide them in asking these kind of skeptical follow up questions? It seems like there isn’t anything too domain specific about this, so even if you aren’t an expert in psychology experiments you could semi-independently develop this skill of probing the logical structure of claims about an interpretation. It also seems that, cognitively, such thinking puts a heavy demand on your working memory, since it consists of layers and iterations of hypotheticals and counter-factuals. This makes it extra hard, is there any way to make it easier?
If there is no general principle, it may be that me (and my students) are stuck with going through worked examples. I’m seeking short cuts up the mountain.