I always promised myself that I’d never do any research involving questionnaires.
Well, times change and we’ve all done things for money which we might not have done otherwise. So I’ve been running these huge postal-questionnaire surveys and gathering hundreds of thousands of data points and wondering what sense can be made of the morass of information.
Why the previous distaste for questionnaires? Well, true to the behaviourist-roots which i share with all experimental psychologists, I don’t have a lot of faith that people’s answers to questionnaire questions bare much relation to the thing that we, asking the questions, are interested in. The vagaries of personal intepretation, context, ambiguities in wording, differences in perspective between researcher and respondent add so much noise – why should i believe that the average response on a particular question reflects anything more than the willingness of the average respondent to tick that part of the response scale on that question?
(by the way, this is common, useful and potentially unhealthy aspect of the experimental psychologist’s trade: a complete distrust of people’s professed desires and beliefs. Just because they said they’ll vote Labour / choose to do that job for that reason / are a kind and conscientious person / etc you don’t actually believe them do you??).
Anyway, does that mean that my 200,000 data points are a load of junk? It means that i think that most of the survey data reported in the news is a load of junk. 75% of people think this. 2 in 3 people think that. etc etc. Junk. So, why not my stuff?
Well, it’s all about statistics and differences. Admittedly the point someone marks on a questionnaire may bare little relation to the thing the question refers to, but we can demonstrate that there is consistency in how people answer certain questions. Further than this, there are systemmatic differences in how different groups of people answer questions. By looking at differences, we can stop worrying about the reponse to the questionnaire as an indicator of wider meaning, and focus on the existence of differences between different people’s responses as indicators of wider meaning. Sure, if someone asks “How worried are you about water pollution” then my response is pretty meaningless, whether I indicate 1 (not at all) or 7 (extremely). If I ask 200 people, then the average response is still pretty meaningless. But if I find that the 100 Guardian readers give a statistically higher response than 100 Telegraph readers then that says something about the world. Anyway, maybe this was obvious to the social psychologists all along, but if it was they never told me.
9 replies on “questionnaire data”
Some economists say that preferences can only be revealed by action.
Great spirits meet !
Absolutely, looking at behaviours rather than declarations simplifies the whole business of understanding humans considerably.
Once at a discussion someone tried to pull this trick on me: they insisted that what people did was a good indication of what they really wanted to do (fair enough) – he called it ‘revealled preference’. And then he said that because lots of people did choose to live in london they were actually really happy with their choice (despite what they said). Now i agree that living in london was obviously, after all considerations, the best resolution of the conflicting demands that life and their desires placed on them, but to say that because they choose it they were happy about it is another thing altogether. You can be threatened and coerced into something, and you choice represents what you really want to do (because you want to avoid punishment) but that doesn’t mean you are happy with (or should put up with) the factors that are coercing you!
how else have you degraded yourself for money tom?!
but seriously, i’ve been wandering through the British Social Attitudes survey as a vast expanse of data. Ultimately, my statistical bias is that we can understand more about how responses change over time – rather than performing complex regressions and etc within one wave… of course, i would say that, and the problem with time series is that concepts and therefore questions change in meaning over time – so life is basically messy.
I am not an economist, but as you point out, preferences and happiness are not the same thing.
However, if you are acting under coercion (i.e., you are forced at gun-point to do something) that hasn’t got anything to do with preferences either…
Will
I think we’re agreeing here – comparing responses over time gives the same leverage that comparing responses between groups does. The individual data points are meaningless (i.e. we cannot divine their meaning) but the differences aren’t.
And, yes, life is messy. A peripheral issue that worries me is how much relation the data we extract by averaging bears to the individuals from which it came. We might find a consistent pattern across many people, but no single individual might embody that pattern. Like they say, most people have an above average number of legs
Hubert – coercion does involve preferences. If i’m threatened with a gun it involves my preference not to get shot. Less extreme coercion will involve preferences in less trivial ways. If prefer not to live in london because it smells, but i also prefer to have the kind of job that you can only get in london then those two preferences must clash
(i’m using coercion in the non-agenic sense here; just referring to coercion by circumstances)
I’m sure you will be unsurprised with my reluctance to accept your definition of coercion, which I feel blurs the line between violent behaviour against you by another person and the fateful realization that you can’t have everything your way.
For instance, I would very much like to be a better aikidoka than I can ever be. I have to settle for being the aikidoka I am. Am I “coerced by circumstances” ?
As for preferences, we are talking about different things (psychology vs. micro-economics), I think.
Yes, i did conflate agenic coercion and coercion by circumstances – it was deliberate. From the perspective of the ‘done to’ it doesn’t much matter. And lots of our circumstances *are* constructions of human agency, directly or indirectly (and of course some aren’t). My conflation blurs the line, but then i contest that the line *is* blurred already (and anyone who tells you otherwise is trying to politically hoodwink you)
The line *is* only blurred when we fail to distinguish between what is within our (i.e. my and the other guy’s) means of action and what isn’t, in other words, when we delude ourselves.
Do you remember the conversation we once had with this bloke who insisted that we are determined because we are limited by our human nature ? We can’t hop from planet to planet, read the human soul, that sort of thing. This is the same thing, I think.
Anyway, we drifted away from the initial idea, which is that acts speak louder than words.
hello. the beginning of this conversations reminds me of this: http://www.azlyrics.com/lyrics/anidifranco/joyfulgirl.html
which seems to suggest that we don’t really know if we want to do what we do, but we do it anyway.
Also it reminds me that there are things that I want to do, but somehow never manage to get round to doing them, even though it’s possible – maybe i’m being coerced by my own lethargy…