I always promised myself that I’d never do any research involving questionnaires.
Well, times change and we’ve all done things for money which we might not have done otherwise. So I’ve been running these huge postal-questionnaire surveys and gathering hundreds of thousands of data points and wondering what sense can be made of the morass of information.
Why the previous distaste for questionnaires? Well, true to the behaviourist-roots which i share with all experimental psychologists, I don’t have a lot of faith that people’s answers to questionnaire questions bare much relation to the thing that we, asking the questions, are interested in. The vagaries of personal intepretation, context, ambiguities in wording, differences in perspective between researcher and respondent add so much noise – why should i believe that the average response on a particular question reflects anything more than the willingness of the average respondent to tick that part of the response scale on that question?
(by the way, this is common, useful and potentially unhealthy aspect of the experimental psychologist’s trade: a complete distrust of people’s professed desires and beliefs. Just because they said they’ll vote Labour / choose to do that job for that reason / are a kind and conscientious person / etc you don’t actually believe them do you??).
Anyway, does that mean that my 200,000 data points are a load of junk? It means that i think that most of the survey data reported in the news is a load of junk. 75% of people think this. 2 in 3 people think that. etc etc. Junk. So, why not my stuff?
Well, it’s all about statistics and differences. Admittedly the point someone marks on a questionnaire may bare little relation to the thing the question refers to, but we can demonstrate that there is consistency in how people answer certain questions. Further than this, there are systemmatic differences in how different groups of people answer questions. By looking at differences, we can stop worrying about the reponse to the questionnaire as an indicator of wider meaning, and focus on the existence of differences between different people’s responses as indicators of wider meaning. Sure, if someone asks “How worried are you about water pollution” then my response is pretty meaningless, whether I indicate 1 (not at all) or 7 (extremely). If I ask 200 people, then the average response is still pretty meaningless. But if I find that the 100 Guardian readers give a statistically higher response than 100 Telegraph readers then that says something about the world. Anyway, maybe this was obvious to the social psychologists all along, but if it was they never told me.