At first glance it would seem that there is little of interest in the latest TNS-BMRB poll of referendum vote intentions released today. It looks like yet another poll that shows how difficult it is proving to be for either side to change the balance of opinion. But once we take a closer look, the poll adds new fuel to the continuing debate about why there are such big differences between the polls, and on what might be the best way of securing an accurate gauge of public opinion on the referendum.
The poll’s headline figures are Yes, 25%, No, 44%, Don’t Knows 31%. That represents no change on the Yes figure reported in TNS BMRB’s poll of a month ago, a three point drop in the No vote and a three point increase in the Don’t Knows. If we take the Don’t Knows out of the calculation, this points to a Yes vote of 36%, No, 64%, an increase of one point in the Yes vote as compared with the previous reading, but three points below the average Yes vote in all the polls that have been conducted since July. TNS BMRB thus remain one of the less optimistic pollsters so far as the prospects of the Yes side are concerned.
But TNS BMRB are entirely alone in one respect – in recording such a high level of Don’t Knows. At 31% their reported figure is almost twice the average in all other recent polls (16%).
This much higher level of Don’t Knows first emerged in last month’s TNS BMRB poll – when they made a change to the wording of their referendum voting intention question. The company now introduces its question by stating that the referendum will be held in the autumn of 2014, and then asks people how they ‘intend to vote’ in the referendum. Previously respondents had been asked how they would vote if the referendum were to be held ‘tomorrow’. And perhaps we should not be surprised to learn that people find it more difficult to say what they will do in a year’s time than to indicate what they would do now.
Moreover, not only is TNS BMRB’s current practice different from what it was previously, it is also different from that of most other pollsters, who either simply do not refer to the date of the referendum at all, or else ask people what they would do if the referendum were to be held tomorrow. We thus might well be tempted to conclude that through their change of wording TNS BMRB have revealed that the race is much more open than we might hitherto have thought.
However, TNS BMRB are not entirely alone in asking people what they think they will do next year. ICM also adopt a similar approach, yet at 19% the proportion of Don’t Knows in their most recent poll was not particularly remarkable.
There is in fact a subtle difference between the wording used by ICM and that by TNS BMRB. TNS BMRB ask people how ‘do you intend to vote’; ICM ask how ‘do you think you will vote’. The latter wording perhaps makes it easier for those who remain less than certain about they will do but still have an idea of how they will eventually mark their ballot to report their inclination to a pollster. These folk may well be open to persuasion, but perhaps we should not presume that all those who respond to TNS BMRB’s question by saying, ‘Don’t Know’ are necessarily entirely open minded in their views.
Meanwhile, TNS BMRB have made a notable change to the way in which they have analysed their data. The company’s previous poll came in for a certain amount of criticism because more of its respondents said they had voted Labour in 2011 than said they voted SNP, thereby raising some doubts about the representativeness of the sample. In contrast, as previous readers of this blog will be aware, a number of pollsters have opted to weight their data by how people said that they voted in the last Scottish Parliament election in 2011 in order to try and correct any such apparent problem.
TNS BMRB have now joined their ranks. But in so doing they have adopted a somewhat different approach. One of the features of virtually any poll is that fewer people say they did not vote at the last election than actually did so according to the official result. In part this will be because the official figures exaggerate somewhat the level of abstention (some people will have died, others will be registered twice, etc.), in part because people who do not vote are reluctant to talk to pollsters, and in part because people may be reluctant to admit they did not vote. Given this complex of possible reasons – and given the greater likelihood that those who say they did not vote last time will not do so next time either – pollsters who weight by past vote typically do not attempt to weight their reported levels of abstention so that it matches the official outcome of the last election.
However, TNS BMRB have weighted their poll such that between them, the combined total of people who say they did not vote in 2011 or cannot remember what they did in 2011 matches the near 50% official abstention rate in that year. Whether those who said they cannot remember what they did in 2011 should be treated in this way is certainly debatable. But in any event TNS BMRB’s approach means that while the balance of people saying they voted Labour and SNP in 2011 is now in line with the actual result, at the same time both sets of voters have been downweighted as compared with the number actually interviewed. (At the same time TNS BMRB have in fact mistakenly transposed the weights that were applied to 2011 Conservative and Liberal Democrat voters though this has not made a significant difference to the headline referendum figures in the poll.)
Unsurprisingly, those who say they did not vote in 2011, who are now heavily upweighted, are not markedly more likely than Scots in general to say they will vote Yes (or No). So it comes as little surprise to hear that when TNS BMRB apply their new weighting method to their poll of last month, it made relatively little difference to the overall result. The Yes vote was only increased by one point while the No vote was reduced by two – rather less of an adjustment than perhaps some of TNS BMRB’s critics might have anticipated. Mind you, if we take that adjustment into account it means today’s poll in fact points to a one point drop in both the Yes and the No vote. Despite the confusion of the pollsters, it seems there really is no change happening at all!
About the author
John Curtice is Professor of Politics at Strathclyde University, Senior Research Fellow at ScotCen and at 'UK in a Changing Europe', and Chief Commentator on the What Scotland Thinks website.
