OPINION: Polls – how come we are getting so much inconsistency?

Polls are simply samples of the population as a whole which are used to deduce how the population as a whole will behave.  Factors like poll size are governed by the laws of statistics, and are very well known and applied by all the major polling organisations.  Statistics allows uncertainty bounds to be placed on the reliability of poll results relative to the actual behaviour or opinion of the population as a whole.

Polls are particularly used by political parties, both to help develop strategy for being elected and as one input to the development of policy.

However, poll results seem to have become very unreliable in recent years.  A particular example was the publication of two political polls recently which gave diametrically opposed results well beyond statistical uncertainty bounds. One poll gave Labour a clear majority over National and the other poll showed a clear majority for National over Labour.

So what’s going on?   Part of the problem is technical.  The cheapest way to run a poll is to phone poll a sample of people over their (published) land lines.  But increasingly people are ditching their landlines and relying on mobile phones only. In this situation I reckon the only way to get a fair result is to apply a correction factor which represents differences between those who do or don’t have land lines.  This presupposes that some prior research is done which determines whether there are significant behaviour /opinion differences between those who do or don’t have land line, and what those differences are.

But I think the major problem is that of human behaviour. In this respect people tend to fall into one of the following categories:

• Those who answer poll questions as honestly as possible
• Those who give the answer they think is expected of them rather than giving a more honest answer. I think this particularly applies when polling very difficult or controversial environmental or social issues. In many situations what people actually do is often quite different from what they say they would do, and taking account of this is very difficult.
• Those who deliberately try to skew the poll in a particular direction by giving an answer which does that, but does not necessarily represent their private opinion.

I suspect that even the professional pollsters have difficulty in taking account of this human behaviour factor, because it probably varies in significance depending on the subject of the poll.

A classic approach to this problem is to design the poll so that people answer a set of questions, some of which are designed to detect any voting biases and some of which are directed at the issue which is the subject of the poll.  If the poll is well-designed it may be difficult to spot the strategy underlying the whole set of questions.  The problem with this approach is that it is time consuming and many people will shy away from making the necessary commitment of time.   Conversely, people who are prepared to make the commitment of time may have an “axe to grind”, and the poll results will be biased accordingly.

I think the result of all of the above factors is that polls based simply on random land line phone calls will be increasingly unreliable.    Credible polls will need to be carried out within a defined structure, intended to eliminate the kinds of bias discussed above.  I also think that one off polls will no longer be enough – polls should be repeated, preferably using an alternative methodology, so that there is some check on reproducibility.   Doing this will increase the cost of polling which will probably means less frequent polls. But that may be the price that has to be paid for more reliable and useful results.

By Bas Walker

This is another of Bas Walker’s posts on GrownUps.  Please look out for his articles, containing his Beachside Ponderings.