Tag Archives: 2016

Opinion Polling in the EU Referendum: Challenges and Lessons

Thursday 8th December, 1300-1700, Royal Statistical Society, London

Please join us for this event, jointly organised by the National Centre for Research Methods, British Polling Council and Market Research Society.

Register for this free event at: Eventbrite

DESCRIPTION

This event is jointly organised by the National Centre for Research Methods, British Polling Council and Market Research Society.

The referendum on Britain’s membership of the European Union presented the opinion polls with a formidable challenge. Support for Remain and Leave crossed party lines. Rather than a debate between left and right, opinions reflected a division between social liberals and social conservatives. Even if the polls had overcome the difficulties that had beset them in 2015 – and that appeared to be still a work in progress – there was no guarantee that methods that had been honed to estimate party support in a general election would work effectively in this seemingly very different environment.

As a result, the campaign was marked by a lively debate about polling methodology, and significant methodological experimentation and adaptation by polling companies. In the event the final polls correctly indicated that the result would be close, but for the most part, incorrectly indicated that Remain would be the most likely winner.

With speakers from the polling companies and members of the BPC/MRS inquiry into the performance of the polls in the 2015 election, this seminar features presentations of how the polling companies set about their task and independent evaluations of the methodology that they used. Its aim is to identify the key lessons to be learned from the referendum for the future of opinion polling.

AGENDA

13.00Registration opens
13.30Welcome from Jane Frost (Market Research Society)
13.35Introduction from the Chair: Sharon Witherspoon MBE (Academy of Social Sciences)
13.40Polling in the EU Referendum: an overview, John Curtice (British Polling Council & University of Strathclyde)
14.05The challenges of polling by phone in the EU Referendum, Ben Page (Ipsos-MORI)
14.20The challenges of polling via the internet in the EU Referendum, Adrian Drummond (Opinium)
14.35Discussion and Q&A
15.00Tea & Coffee
15.20Sampling and mode of interview, Patrick Sturgis(NCRM, University of Southampton
15.45Treatment of don’t knows and turnout weighting, Stephen Fisher (University of Oxford)
16.10The effect of methodological adjustments during the campaign, Will Jennings ( University of Southampton)
16.35Discussion and Q&A
17.00Close

Performance of the polls in the EU referendum

Seven member companies issued ‘final’ polls of voting intentions in the EU referendum. While no company forecast the eventual result exactly, in three cases the result was within the poll’s margin of error of plus or minus three points. In one case Leave were correctly estimated to be ahead. In the four remaining cases, however, support for Remain was clearly overestimated. This is obviously a disappointing result for the pollsters, and for the BPC, especially because every single poll, even those within sampling error, overstated the Remain vote share.

Polling of voting intentions in the referendum was a unique event, and the lessons of this experience will not necessarily be transferable to polling of future general elections. Consequently, the BPC does not believe that a full-scale independent inquiry, similar to that set up after the 2015 General Election, would justify the large amount of time involved. However, the Council will be asking all its members to look carefully at the methods that they used in the referendum and to report their findings, and will consider inviting an external reviewer to consider the methods that were used.

BPC President John Curtice commented, ‘Polling of voting intentions in this referendum was always going to be a difficult exercise. Yet their central message – that this looked like a close referendum that neither side could be sure of winning – proved prescient. Nevertheless, there will be an obligation on members to try and establish why there was a tendency to overestimate support for Remain, and the Council will report later this year on why that appears to have been the case.’

The following table shows the results of all polls conducted by BPC members that had at least some fieldwork within the last 4 days of the campaign.

Remain Leave error on remain Method Sample Size Fieldwork
ORB 54 46 +6 telephone 877 14-19 June
Survation 51 49 +2 telephone 1003 20-June
ComRes 54 46 +6 telephone 1032 17-22 June
Opinium 49 51 +1 online 3011 20-22 June
YouGov 51 49 +3 online 3766 20-23 June
Ipsos MORI 52 48 +4 telephone 1592 21-22 June
Populus 55 45 +7 online 4740 21-22 June
RESULT 48 52
Average error 4.3

As well as the pre-election polls there were three published on-the-day polls, by BMG, YouGov and Ipsos MORI. As they combine people who are reporting their actual vote and others who are reporting their voting intention they are treated separately from the pre-election polls above. The YouGov poll interviewed 4,772 people online and produced a result of Remain 52% Leave 48%. Ipsos MORI interviewed 546 people by telephone and produced a result of Remain 54% Leave 46%. This is a smaller sample size than their standard published polls as it was not intended as a standalone on-the-day poll, and only presented as a day-by-day comparison drawing on their final poll. BMG interviewed 2,000 people online with a declared result of Remain 46% Leave 41% and Don’t know 13%.

The ORB and Ipsos MORI polls covered only Great Britain. All the other polls above covered the whole of the UK.

British Polling Council member TNS-BMRB also conducted a poll within the fieldwork period above. However, they took the decision not to remove and reallocate in any way those who said they were undecided, or that they would not vote, and so no calculation of error is possible. For the record, their final poll was conducted online from 16-22 June, with a sample size of 2,320, and the results were Remain 41% Leave 43% Undecided/would not vote 16%

In the interests of completeness, British Polling Council member BMG Research conducted two polls, one telephone and one online, that finished over a week before polling day, and which are therefore not included in the table above. Fieldwork was conducted from 10-15 June, with sample sizes of 1,043 (telephone) and 1,468 (online). The results were Remain 53% Leave 47% (telephone) and Remain 44% Leave 56% (online).

British Polling Council member LucidTalk conducted a poll solely in Northern Ireland. Fieldwork was conducted online from 15-17 June, with a prediction of Remain 57% Leave 43%. As the actual Northern Ireland result was Remain 56% Leave 44% this represents an error of only 1% in its estimate of the vote.

Performance of the Opinion Polls in the 2016 Local and Devolved Elections

The following tables compare the results of opinion polls of voting intentions for the elections that were held on May 5th in London, Scotland and Wales with the final outcome. A poll is included if its fieldwork was conducted wholly or mostly in the seven days before polling day (May 5th), and it was the final poll published by the pollster concerned. (This is a slightly longer period than is normally used in BPC reports on the performance of the polls in general elections — the period has been lengthened because there were very few polls conducted in the last few days.)

It should be noted that all of these polls were conducted online, with the exception of the Survation poll in Scotland, which was conducted by telephone.

London Mayor 1st Preference Vote

Company / Fieldwork Dates Con Lab LibDem UKIP Green Other
Opinium 26.4–1.5.16 35 48 4 5 5 3
TNS 26.4–3.5.16 33 45 7 5 4 5
ComRes 28.4–3.5.16 36 45 6 4 6 3
YouGov 2–4.5.16 32 43 6 7 7 5
RESULT 35 44 5 4 6 7

London Mayor After Redistribution of 2nd Preferences

Company / Fieldwork Dates Con Lab
Opinium 26.4–1.5.16 43 57
TNS 26.4–3.5.16 43 57
ComRes 28.4–3.5.16 44 56
YouGov 2–4.5.16 43 57
RESULT 43 57

London Assembly — Constituency Vote

Company / Fieldwork Dates Con Lab LibDem UKIP Green Other
YouGov 2–4.5.16 30 44 7 11 7 1
RESULT 31 42 8 8 9 3

London Assembly — List Vote

Company / Fieldwork Dates Con Lab LibDem UKIP Green Other
YouGov 2–4.5.16 29 39 8 11 9 5
RESULT 29 40 6 8 8 3

Scotland — Constituency Vote

Company / Fieldwork Dates Con Lab LibDem SNP Green Other
Survation 1–2.5.16 19 21 7 49 5
YouGov 2–4.5.16 19 22 7 48 4
RESULT 22 23 8 47 1

Scotland — List Vote

Company / Fieldwork Dates Con Lab LibDem SNP Green Other
Survation 1–2.5.16 20 19 8 43 7 4
YouGov 2–4.5.16 20 19 6 41 9 5
RESULT 23 19 5 42 7 4

Wales — Constituency Vote

Company / Fieldwork Dates Con Lab LibDem PC UKIP Other
YouGov 2–4.5.16 21 33 8 19 16 4
RESULT 21 35 8 21 12 3

Wales — List Vote

Company / Fieldwork Dates Con Lab LibDem PC UKIP Other
YouGov 5.5.16 20 31 6 20 16 8
RESULT 19 31 6 21 13 3

YouGov also conducted on polling day an exercise in which it recontacted a sample of those who it had interviewed previously in order to ascertain how they had or intended to vote. This produced the same result for the Constituency vote, and very similar figures for the list vote: Con 19%, Lab 30% LibDem 6%, PC 21%, UKIP 16% and other 8%.