One important feature of opinion polls is that the estimates they produce can vary from one reading to the next when in reality public opinion amongst the population as a whole has not changed at all. This is because all polling is based on interviewing a sample of a thousand or so people and drawing inferences from this as to the position amongst the population as a whole. Even if such polls are conducted perfectly, their estimate of how many people hold a particular view – such as whether they wish to remain in or leave the EU – can easily vary by up to three percentage points on either side of the true value in the population as a whole purely by chance. So if one poll estimates that, say, 48% wish to remain in the EU and another then says that 52% wish to do so, it could well be the case that in reality nothing has changed at all, and that opinion remains more or less evenly balanced at around 50% support for both sides.
So how can we get a better grip on whether the balance of opinion really has changed? One approach is to build on the fact that while one poll can say 48% and another 52% purely by chance, random variation is much less likely to be the explanation if, say, four or five polls previously said the figure in question was 48% but then the following four or five all say it is 52%. In those circumstances we would conclude that the balance of opinion really has changed.
This is the insight on which our poll of polls is based. It shows the average level of support across the six most recently conducted polls. We recalculate this figure every time as new poll of referendum voting intentions is released, thereby providing a constantly updated picture (i.e., a moving average) of how opinion does or does not shift as the campaign progresses.
The result is a less erratic picture than the one painted by individual polls. You can see this by comparing the trend over time in our poll of polls with the one we find when looking at individual polls of referendum vote intentions.
Of course, not even these poll of polls figures should be taken uncritically. Even when we average across as many as six polls, our estimate will still be subject to some random fluctuation. So not too much should be made of a one point shift up and down in our estimate, or at least not until the new estimate is confirmed after several more polls have been released.
At the same time, polls are not necessarily conducted perfectly, a fact of which we were all made aware immediately after May’s UK general election, when collectively they all underestimated Conservative and overestimated Labour support. A poll of polls cannot insulate us against the risk of collective failure by the polling industry.
Meanwhile, less dramatically but no less importantly, the polling companies may systematically disagree with each other about the level of support enjoyed by ‘leave’ and ‘remain’. In that event, changes in which companies’ polls contribute to the poll of polls may induce a degree of artificial instability. We thus always give details of which companies conducted the polls that contribute to our latest poll of polls, while, where necessary, we will warn in our Commentary section if there is a risk that a change in whose polls are included may have induced a change in the estimates.
We also have to bear in mind that one option open to respondents to any poll is to say that they ‘don’t know’ how they will vote. However, this is not a choice that is open to them on polling day. Thus, in order to provide the clearest possible indication as to the referendum outcome to which the polls are pointing, in calculating the poll of polls we leave aside those who say ‘don’t know’. Thus our estimate of the percentage who will vote ‘Leave’ or vote ‘Remain’ is based on just those who state either choice. That, of course means that the two percentages will always add up to 100%.
Finally, bear in mind that while any set of six polls may well have all been conducted at much the same time, equally it is possible that quite a few weeks may have elapsed between when the first and the last was conducted. The poll of polls is thus inevitably less up to date the longer the period over which the polls on which it is based were conducted. You will thus see that we also always provide details of the dates between which the interviewing (fieldwork) for the polls included in the poll of polls took place.