Much excitement was caused this week by a poll from BMG Research for The Herald which was reported by the newspaper under the headline ‘Support for independence surges on hard Brexit vow’. There was then even more of a flutter when in an interview for UK Business Insider I suggested that the three-point increase in support for independence that the poll had identified – thereby putting the level at 49% – might be accounted for by ‘sampling error’. Some seem to think I was suggesting that there was something wrong with the poll. If so, they are mistaken.
Regular readers of this blog will know that we repeatedly caution that the figures reported by any poll are subject to chance variation. Polls attempt to estimate the balance of opinion amongst people in general by interviewing just a thousand or so of them in particular. Although every pollster does their best to make sure that each and every one of their samples is representative of the whole population, inevitably there is always some purely random chance that a poll is not exactly spot on in this regard. As a result, support for a party or a proposition as registered by a poll can go up or down a bit between one poll and the next even when the level of support amongst the population as a whole has not changed at all. This is what is meant by ‘sampling error’, which, of course, is not the same thing at all as ‘sampling bias’.
A three-point difference between two polls that each contain around a thousand respondents is certainly too small for us to rule out the possibility that it has simply been generated by this chance variation. Given that sudden and sharp changes in public attitudes occur relatively rarely, this can make polling a somewhat frustrating business. Small changes are often what we are trying to detect, and when, as in the case of independence the balance of opinion is already quite even, such changes can, of course, be politically highly significant. But because of the possibility of chance variation, one poll is rarely sufficient evidence for us to conclude that a such a small change has actually occurred.
What we can do in these circumstances is to see whether the polls are consistently finding that attitudes have shifted. If a number of polls all show much the same shift, small though it may be, we can then be reasonably sure that the apparent change is not simply due to chance variation. Trouble is, in this instance, such evidence as we do have is not consistent. One other poll, by Panelbase for The Sunday Times, has been conducted since Theresa May gave her speech at Lancaster House outlining the UK government’s negotiating stance on Brexit, and at which point it became clear that her stance is very different from the one that the Scottish Government has proposed. That poll reported that, after excluding Don’t Knows, support for independence has dropped by one point – from 47% to 46% – since September of last year.
More broadly, we might note that eleven polls of how people would vote in indyref2 have been conducted since the beginning of July last year. On average, once Don’t Knows are left to one side, these polls have put support for Yes on 47%. This also happens to be the average level of support for Yes registered in the twelve polls that were conducted in the first half of 2016, before the EU referendum. Against that backdrop we should not get too excited about the odd poll that suggests that support for independence has fallen back to, say, 45%, or the occasional one that indicates it might have increased to 49%. That is just the kind of variation that ‘sampling error’ can produce.
So, to date at least, there is insufficient evidence to draw the conclusion that the disagreement over Brexit between the UK and the Scottish governments has altered the balance of opinion on independence. That is not to say that the BMG poll is ‘wrong’, but merely that there is room for disagreement about the interpretation that was placed on it in The Herald. But, of course, newspapers are never going to say, ‘Our poll says support for independence has gone up, but we are not sure whether you should believe it!’.
About the author
John Curtice is Professor of Politics at Strathclyde University, Senior Research Fellow at ScotCen and at 'UK in a Changing Europe', and Chief Commentator on the What Scotland Thinks website.
