After the excitement generated during the last fortnight by some highly divergent polls, today we have two polls that tell much the same story. Some valuable light is shed too on some of the methodological debates that have been occasioned by the recent big differences between some of the polls.
First Panelbase have undertaken the latest in their regular polls for The Sunday Times. It puts Yes on 37% and No on 47%. The figures are virtually identical to those in their previous Sunday Times poll in July (Yes 37%, No 46%). At the same time they stand in stark contrast to the same company’s poll for the SNP published a fortnight ago that put the Yes side one point ahead – but in which the referendum voting intention question was placed after two others that appeared likely to cue voters into saying Yes.
Second, ICM have entered the fray for the first time with a poll for Scotland on Sunday. This puts Yes on 32% and No on 49%. These figures are remarkably close to the average of all the referendum vote intention estimates published so far this year, that is Yes 32%, No 51%. So a first reading though this poll is, it too can be regarded as implying that so far neither side has made any demonstrable progress.
Still, these findings also mean that Panelbase are continuing to produce findings that are more favourable to the Yes side than any other pollster, suggesting that the race is potentially quite close. Once the Don’t Knows and Won’t Says are excluded from their figures their poll puts to a referendum outcome of Yes 44%, No 56%. ICM’s figures point to a margin of Yes 40%, No 60%.
Of course the Don’t Knows may not necessarily split in the same way as the rest of the population has so far. Indeed, both of today’s polls suggest they will not (as indeed have previous Ipsos MORI polls). When Panelbase asked their Don’t Knows what they would do if they were standing in the polling station now, as many as 59% said they would vote Yes while only 34% indicated they would vote now. If that is indeed what they were to do then, on my calculations, Panelbase’s estimate of the Yes vote increases from 44% to 47%. However, when ICM asked their Don’t Knows which way they thought they were likely to vote next year, Yes were ahead by a more modest 21% to 19%, enough only to take their estimate of the Yes vote up from 40% to 41%.
Interestingly, Panelbase also report today on the results of an experiment designed to cast some light on the efficacy of their methods. To date Panelbase have been the only company to weight their data such that the proportion of people who say they voted for each of the parties in the 2011 election more or less matches the actual outcome of the last Holyrood contest. In contrast YouGov have (amongst other things) weighted their data to match the result of the 2010 Westminster contest, while other pollsters have not weighted their data to match the result of either contest.
This time Panelbase asked their respondents not only how they voted in 2010 but also in 2011. And they find that their respondents’ recall of how they voted in 2011 matched the actual result much more closely than did their memory of how they voted in 2010. It would appear that voters find it easier to recall the more recent and more Scottish contest in which far more voters backed the SNP, some of who it appears have now forgotten that they did not do the same in 2010.
If voters are indeed inaccurate in recalling what they did in 2010 then weighting a poll to match what happened in 2010 could make it less rather than more accurate. And Panelbase report if they had gone down that road they would have put Yes at just 31% while No would have been on 53%. Both these figures are six points adrift of the figures that Panelbase have actually reported.
However, Panelbase are no longer alone in opting to weight by Holyrood vote intention. For in their poll today ICM too have weighted their data by Holyrood voting intention, having discovered, much like Panelbase, that voters appear better able to recall what they did on that occasion accurately than they can what they did in 2010. Indeed 22% of ICM’s respondents admitted that they could not remember what they did in 2010, whereas only 6% said the same about 2011.
Meanwhile, although not so favourable to the Yes side than Panelbase’s figures, ICM’s are more favourable than those of some other recent polls. Part of the reason for such difference as there is between ICM and Panelbase lies in Panelbase’s decision to exclude those who say they are unlikely to vote (who, relatively few though they may be, are more inclined to be No voters) but also partly in the fact that Panelbase’s sample contains slightly fewer No voters in the first place.
So where does this put the other pollsters? It would certainly seem to underline the doubts that some have expressed about the most recent TNS BMRB poll which reported a 25% Yes but also had a sample in which more people said they voted Labour in 2011 than said they voted SNP. It might also seem even more obviously to raise questions about YouGov’s decision to weight their data according to how people said they voted in 2010.
Here though we need to be careful before jumping to conclusions. Unlike other pollsters YouGov are not collecting information now on how their respondents voted in 2010. Rather they either did so either immediately after the 2010 election, or if they have signed up to YouGov’s panel of potential respondents since 2010 when they applied to become members. So YouGov’s data on how their respondents voted in 2010 may be more accurate than that obtained by Panelbase or ICM – though it would be interesting to see how well their method does succeed in replicating the actual 2011 outcome. The methodological debate is likely to continue for a while yet.
About the author
John Curtice is Professor of Politics at Strathclyde University, Senior Research Fellow at ScotCen and at 'UK in a Changing Europe', and Chief Commentator on the What Scotland Thinks website.
