We have often drawn attention in this blog to differences in the way on which polls of referendum vote intention are conducted and how far these differences might help account for the discrepancies in their results. Indeed, an extended discussion of the methodology of the polls of referendum vote intention will appear in the next number of the academic journal, Scottish Affairs.
If questions of methodology require us to adopt a critical approach to what the polls are telling us, this is even more clearly the case when we move the focus of attention to subjects other than referendum voting intention. Two considerations in particular need always to be borne in mind. First, how precisely have the survey questions been worded? Second, what questions have been selected for inclusion? Might other questions have been asked that might have cast the results of the survey in a somewhat different light?
The importance of both these considerations has been illustrated by two sets of polls that have been published in the last week or so. The first set was conducted by YouGov for the Better Together campaign and comprised separate polls of each of Scotland, England and Wales that focused on how people think welfare benefits and public services should be funded. The second individual poll, administered to 6,000 people across Great Britain by Populus for the Financial Times, covered what people hoped and expected the referendum outcome would be, together with their attitudes towards the prospect of sharing the pound with an independent Scotland.
In both cases careful attention needs to be paid to the way in which some of the key questions were worded. Meanwhile, so far as the Better Together exercise at least is concerned, it is also instructive to consider what respondents were not asked as well as what they were.
One of the key findings of more than one Scottish Social Attitudes (SSA) survey in recent years has been that people in Scotland are reluctant for the government old age pension to be different on the two sides of the border. The survey question reads:
What about the old age pension paid out by the government? Should this…
..always be the same in Scotland as it is in England
or, is it OK for it to be different in Scotland – either higher or lower – than it is in England?
In the most recent survey, just 37% said it was OK for the pension to be different, while as many as 58% stated that it should always be the same.
Understandably this finding is regarded by the No campaign as ‘helpful’ to its side of the independence argument. It thus sought to replicate and expand on it in its in its own poll. As well as asking once again about the prospect of having different pensions, the Better Together poll also asked about ‘welfare payments’ and corporation tax.
In Scotland the poll secured even more emphatic results than did SSA. As many as 79% said the pension should be the same throughout the UK, while only 15% felt it was OK for it to be different. Similar results were obtained for welfare payments and corporation tax – and in the parallel polls conducted in England and Wales.
However, the question asked in the Better Together poll was a little different from that asked on SSA. In the case of pensions, for example, it read:
Should the old age pension always be the same throughout the UK, or is it OK to be different?
There is no explanation here as to how the pension might be ‘different’. Respondents could perhaps have thought it referred to the prospect of a different pension for men than for women (which so far as entitlement is concerned of course historically there has been but this is now being phased out), or for the better-off than the less well-off (ditto SERPS and the State Second Pension), rather than a different pension in Scotland than in England. Because the way in which the pension might be different is not made clear, we cannot be sure what respondents took the question to mean.
Meanwhile, given the move towards a single state pension, we might surmise that the prospect of these other kinds of possible differences is not popular. Thus any potential misunderstanding in people’s minds may have served to make it less likely that respondents would say it was OK for the pension to be ‘different’, and thereby helped produce results that were even more ‘helpful’ to Better Together than those obtained by SSA.
Meanwhile, as we noted earlier, one of the subjects covered by the Populus/Financial Times poll was whether an independent Scotland should be able to share the pound. It reported that only 17% of people in England and Wales were willing to back a monetary union with an independent Scotland, while no less than 68% were opposed. (In Scotland, where just over 500 people were interviewed, the figures were almost the reverse, with 61% backing the idea and 22% opposed.) This would seem to be game, set and match for the claim that public opinion in the rest of the UK would not be willing to tolerate a monetary union.
However, this is the full wording of the question that was asked:
Alex Salmond and the Scottish nationalists say that the Pound belongs just as much to Scotland as to the rest of UK and, therefore, that if Scotland votes to become an independent country, they will continue to use the Pound as their currency, retaining the Bank of England as Scotland’s Central Bank and lender of last resort. This would mean that in the event of a financial crisis affecting an independent Scotland, the Bank of England would step in and taxpayers in England, Wales and Northern Ireland would have to bear some of the cost. For this reason the Conservative, Labour and Liberal Democrat parties have said they couldn’t agree to let an independent Scotland use the Pound and retain the Bank of England as its Central Bank. If Scotland votes to become an independent country, would you support or oppose Scotland continuing to use the Pound as its currency and retaining the Bank of England as its Central Bank and lender of last resort?
Yes, it was indeed that long (and hopefully you are still with us)!
In truth, the warning bells should always be ringing if a poll question is more than a couple of sentences long and starts to give respondents a detailed explanation of the relevant arguments. It is often an implicit recognition that in fact many respondents have simply not thought about the issue and do not in fact have a clear view either way. Nevertheless, if one wants to secure an adequate reflection of what the public does and does not think about a subject, it is better to uncover the fact that there is indeed a high proportion of ‘don’t knows’ rather than attempt to give them a civics lesson. After all, one of the problems with attempting to convey such a lesson is that it might be thought that it fails to convey a fair summary of the arguments on the two sides – and one can see why it might be thought that this particular example fails to satisfy that particular criterion.
But, finally, let us return to the Better Together exercise and the issue of question selection. In recent weeks both Labour and the Conservatives have come forward with proposals to give the Scottish Parliament greater power over and responsibility for income tax, while the Liberal Democrats proposed such ideas a while ago. One of the implications of all three sets of proposals – and indeed the provisions of the 2012 Scotland Act that are already in the course of being implemented – is that the basic rate of income tax might be different in Scotland than it is in England.
However, according to the evidence collected by SSA, such a prospect is almost as unattractive to voters in Scotland as is that of the old age pension being different on both sides of the border. In much the same vein as on pensions, the survey has asked:
Thinking about the basic rate of income tax, should this…
…always be the same in Scotland as it is in England
or, is it OK for it to be different in Scotland – either higher or lower – than it is in England?
In the most recent survey in 2013, only 41% felt that it was OK for the basic rate to be different while as many as 52% believe it should always be the same.
However, a question on this subject did not make its way into the Better Together poll. Rather, the poll appeared content only to cover issues such as pensions, welfare payments and corporation tax where there is a marked reluctance amongst the three unionist parties to see power and responsibility transferred from London to Edinburgh. In being selective in this way the poll clearly ran the risk of giving less than a complete picture.
Leaving aside the apparent imperfections of question wording in this poll, there does indeed appear to be reluctance amongst many people in Scotland to embrace the idea that tax and benefit rates may be different on the two sides of the border (even though at the same time a majority say that the decision about such rates should be devolved!). However, that observation not only raises awkward questions for the independence camp, but also contains a potential sting in the tail for those shiny new proposals for more devolution too.
Alas for politicians, voters do not always fit snugly into the ideological boxes that have been so carefully prepared for them.
About the author
John Curtice is Professor of Politics at Strathclyde University, Senior Research Fellow at ScotCen and at 'UK in a Changing Europe', and Chief Commentator on the What Scotland Thinks website.
