The week is only half over, yet it has already been one of the more dramatic so far as referendum polling is concerned. Two polls have claimed that Leave now enjoy a record or near record lead, while two others contained some evidence that Leave may have gained some ground. Perhaps this is the first week in which the balance of opinion has genuinely moved?
The first poll to put Leave on a high came from the internet polling company, YouGov on Monday (for Good Morning Britain). Once Don’t Knows are put to one side it put Leave on 52%, Remain on 48%. It was the first time the company had reckoned Leave were that far ahead since the beginning of February. This was followed just hours later by the publication of another online reading, this time from ICM. This put Leave even further ahead by 53% to 47%. Never before had ICM put Leave that far ahead.
Meanwhile a couple of other polls at around the same time could also be read as providing evidence of a swing to Leave. The first was an online poll from Opinium released on Sunday. Now, as it had happened, Opinium had decided to adopt a methodological change, the detail of which we discuss below. Here we should simply note that the effect of the change was to turn what would otherwise have been a 4 point swing to Leave since the company’s previous poll, enough to put Leave ahead by 52% to 48% (as the poll was reported by The Observer), into just a 1 point movement, leaving Remain still narrowly ahead.
The second came from a phone poll conducted by ORB and published on Tuesday. As we have discussed previously, in reporting ORB’s polls the Daily Telegraph tends to focus on the figures for those who say they are certain to vote, rather than those for all respondents. As it happens, the figures in ORB’s latest poll for those who said they are certain to vote were particularly dramatic. They put Remain and Leave on 50% each, representing a three point swing to Leave and a much better result for Leave than that obtained by most phone polls.
All in all this looks like a substantial body of evidence supporting the idea that Leave have made some progress. However, there are some caveats and cautions to take on board too.
First of all, a second poll from YouGov that was released on Tuesday (in The Times) and conducted a couple of days later than the poll that appeared on Monday, failed to replicate the swing to Leave in that earlier poll. Remain were credited with 51%, Leave with 49%, well in line with the readings of many another YouGov poll.
Second, while ORB’s figures for those certain to vote put the two sides neck and neck, those for all respondents told a very different story. They put Remain on 57% and Leave on 43%, very much in line with the figures in many another phone poll and actually representing a two point swing to Remain as compared with the company’s previous poll last week. While it is quite common for polls (both internet and phone) to find that Leave supporters are more likely than Remain voters to say they will make it to the polls, the gap between the two in this poll was unusually big (69% of Leave supporters said they would vote, compared with just 54% of Remain supporters). There must be a suspicion that the chance variation to which all polls are subject may have helped on this instance to exaggerate the difference in the relative propensity of the two sets of supporters to participate in the ballot on June 23rd.
Third, we need to consider the possible implications of Opnium’s decision to change their weighting strategy. This change consisted (primarily) of weighting their data by some indicators of social conservatism and national identity. Both of these characteristics have been shown to be correlated with attitudes towards the EU, while some previous experimental work undertaken by Populus and Number Cruncher Politics has suggested that internet polls may be obtaining samples that are too socially conservative. Opinium appear to have decided that this insight has some validity and their weighting seems designed to correct what they now accept may be samples that are too socially conservative. If they are right, and their insight is also true of other company’s internet polls, then of course it implies that some of the leads for Leave obtained in other polls this week may exaggerate its strength.
So how might we make sense of this apparently contradictory evidence? Well, it is worth bearing in mind that one of the reasons why some commentators have been inclined to believe that there has been a swing to Leave during the last fortnight is that the Leave side has found it easier to influence the media agenda since the onset of ‘purdah’ on May 27th. No longer has the UK government been able to command the agenda by successively publishing papers written and researched by the civil service and warning of the allegedly dire consequences of leaving.
If the onset of purdah has made a difference then we should be able to discern a difference between the results of all of the polls conducted since May 27th and those undertaken beforehand. Eleven internet polls of referendum voting intention were conducted between the beginning of May and the onset of purdah (one of which is a poll by TNS that was only released on Monday). On average those polls put Remain on 50%, Leave 50%, exactly in line with the average position in the internet polls ever since last September. In the half dozen internet polls conducted since May 27th, Leave has averaged 51%, Remain 49%. So maybe, there has been just a scintilla of movement in the direction of Leave. However, we have to bear in mind that as just three phone polls have been conducted since May 27th, including the remarkable ICM poll that put Leave ahead, we have too little evidence from this kind of polling to confirm this impression.
A couple of other points to bear in mind from this week’s polling. First, Opinium are not the only company in recent weeks to have made methodological tweaks that have been reported as making the results more favourable to Remain. So also have ICM and YouGov in their internet polls and ComRes in their phone polls. Given the divergence between internet and phone polls in this referendum and the difficulties that the polls had in estimating Conservative and Labour support in last year’s general election, it is not surprising that the pollsters should be constantly trying to improve their methods. However, they may need to be wary of any temptation to tweak their polls so that they all ‘herd’ in the direction of what they think the result will be. In any event, the fact that Leave have edged ahead in the most recent internet polls despite these methodological tweaks lends a little greater weight to the suggestion that there might have been a scintilla of movement in its favour.
Second, as previous readers will be aware, we have argued here that, given it is one of the key demographic divides in this referendum, pollsters should be paying greater attention to the composition of their samples in terms of their respondents’ educational attainment. A couple of this week’s polls have provided information on this feature of their samples. In the second of the polls that they published this week, YouGov reported that (after weighting, though the weighting did not have a substantial effect) 27% of their sample had a degree and another 8% some form of higher education other than degree. These figures are not too dissimilar to the nearest equivalent figures in the most recent British Social Attitudes survey (conducted face to face and using random probability sampling), which estimates that 24% of all adults are graduates while those with some other form of higher education constitute another 11%. This similarity provides some initial assurance that internet polls do not necessarily contain too few graduates, and that their tendency to report a lower vote for Remain is not occasioned by any such under-representation.
TNS, meanwhile, (in their poll that was released on Monday but was conducted a while ago) reported that 28% of their sample were graduates, initially suggesting at least that its samples too do not suffer from a deficit of graduates. However, their question on educational qualifications seemingly did not differentiate between the wide range of other qualifications that people might have, raising some doubt, perhaps, about how those with other forms of higher education might have classified themselves. Meanwhile, at 10%, the proportion without any qualifications in TNS’s sample is markedly lower than the 18% to be found in the most recent BSA – but given that this is one of the groups that is least likely to vote for Remain any underrepresentation there might be of this group certainly cannot help account for that this poll put Leave ahead by 51% to 49%. Hopefully, future polls will also tell us more about the educational background of their respondents.