Wednesday, 15 September 2010

The weakness with statistics

I was wondering the other day whether statistics have any real value other than providing points of arguing. For instance you could claim that the government should invest more in X than Y since your survey says that twice as many people like X than Y, case proven! I have already blogged about the problem with the lack of ability to say "no" in surveys. In the example of the Olympics, saying that 500,000 people signed a petition asking for the Olympics is impressive unless you have the chance to report that 45 million didn't want it.
Another thing that has come up recently however is the weakness of survey questions. An article I read on the BBC news site was talking about how the "don't knows" might not be people who don't care as much as people who can't answer the question because they are not sure quite what it means.
Consider the question, "Do you regard yourself as belonging to any particular religion?" (taken from the National Centre for Social Research). A typical type of survey question whose data will end up somewhere important. But the answer is not necessarily straight-forward. If you are an atheist, the answer is simple, if you are Muslim or even Catholic, the answer is probably also straight-forward. But think about a bog-standard Christian. Firstly I don't call my faith a religion because religion to me is not synonymous with faith and is not what I practice (even though I call myself a Christian). Also, even if I do consider my Christianity a religion or understand that the question probably does not make the distinction that I do, do I belong to a particular religion? How particular? I am a Christian but do not call myself Anglican or Evangelical so should I answer "No" because I am not particular?
The problem for statistics involving large numbers of people is that if 30% of Christians say, "No" to the above question, the figures might suggest that the number of Christians is only 70% of the actual figure and then this figure is used to derive all sorts of "scientific" conclusions.
No doubt some people are supposed to be experts in these things but I would suggest any self-respecting information expert reject attempting to make any conclusions based on questions that are hard to decipher or at least provide a box labelled, "I am not sure I understand the question" which allows us to distinguish "I don't care" from "I'm not sure".

No comments: