All posts by admin

BPC Comments on PRCA enquiry into political polling and predictions

The British Polling Council welcomes the interest of the Public Relations and Communications Association (PRCA) in the work of the polling industry, with its announcement that it is to hold an enquiry that, according to PR Week, ‘will address trust and confidence in political predictions following the May 2015 General Election, the UK referendum result in June, and the US Presidential Election result earlier this month’.

The BPC would like to highlight the extensive investigations that have and continue to be made into the performance of the polls. The evidence and conclusions of these reviews will, the BPC believes, provide a valuable source of material to assist the PRCA in its enquiry. These reviews set out the challenges facing the opinion polling industry, covering both what they are getting right as well as the premise as to whether (and where) the ‘pollsters got it wrong’ in the Brexit referendum and the US Presidential election.

Following the undoubted underestimation of the Conservative vote and over-estimate of the Labour vote at the 2015 general election, the BPC, along with the Market Research Society (MRS), sponsored a wholly independent inquiry under the chairmanship of Prof. Patrick Sturgis. That inquiry published an extensive 115 page report and made a series of recommendations that the BPC is currently implementing. Meanwhile, the BPC and MRS are sponsoring a public seminar on the performance of the polls in the EU referendum that will be held at the Royal Statistical Society on 8th December.

On the other side of the Atlantic, the American Association of Public Opinion Research (AAPOR) announced last spring that, in line with its practice after previous presidential elections, a panel under the chairmanship of Courtney Kennedy will analyse and report on the performance of the polls in the US Presidential election. The Panel is due to report by May 2017.

Pollsters in the UK are far from complacent about their performance in recent ballots. However, it is to be hoped that the PRCA’s enquiry will add to rather than duplicate the existing attempts that are being made to assess and improve the methodology of the polling industry. The BPC would certainly welcome an opportunity to advise the enquiry of the efforts that have been made to date.

It is, of course, the case that most of the final polls conducted by BPC members pointed to a victory for Remain rather than for Leave. But in most cases they also suggested that the outcome would be very close. Meanwhile, of the 34 polls of voting intentions conducted during the four week official campaign period between 27 May and 22 June, 17 put Leave ahead, 14 Remain while three suggested the two sides were tied. The evidence from the polls did not point unambiguously to a Remain victory or the simple assertion that the pollsters ‘got it wrong’.

Meanwhile in the United States it is all too often forgotten that, as the nationwide polls anticipated, Hillary Clinton won more votes than Donald Trump. At present she leads in the nationwide vote by around two points. This compares with an average lead for Ms Clinton in the final polls as calculated by Real Clear Politics of 3.2 points. If, when the final tally is available, the nationwide polls are shown to have overestimated the Democrat candidate’s lead by just over one point, this will be lower than the average error in polls of presidential elections between 1968 and 2012 of two points.

While the record of the polls in both the EU referendum and the US Presidential election was certainly not perfect there appears to be a widespread misperception that in both cases the polls pointed to a clear winner. They did not.

We therefore hope that the findings of these reviews into the opinion polls will provide a firm evidence base for the PCRA as it starts its enquiry and the BPC welcomes the opportunity for more dialogue as the review takes shape – including the question of how the findings of opinion polls can be better communicated to the public at large.

Opinion Polling in the EU Referendum: Challenges and Lessons

Thursday 8th December, 1300-1700, Royal Statistical Society, London

Please join us for this event, jointly organised by the National Centre for Research Methods, British Polling Council and Market Research Society.

Register for this free event at: Eventbrite

DESCRIPTION

This event is jointly organised by the National Centre for Research Methods, British Polling Council and Market Research Society.

The referendum on Britain’s membership of the European Union presented the opinion polls with a formidable challenge. Support for Remain and Leave crossed party lines. Rather than a debate between left and right, opinions reflected a division between social liberals and social conservatives. Even if the polls had overcome the difficulties that had beset them in 2015 – and that appeared to be still a work in progress – there was no guarantee that methods that had been honed to estimate party support in a general election would work effectively in this seemingly very different environment.

As a result, the campaign was marked by a lively debate about polling methodology, and significant methodological experimentation and adaptation by polling companies. In the event the final polls correctly indicated that the result would be close, but for the most part, incorrectly indicated that Remain would be the most likely winner.

With speakers from the polling companies and members of the BPC/MRS inquiry into the performance of the polls in the 2015 election, this seminar features presentations of how the polling companies set about their task and independent evaluations of the methodology that they used. Its aim is to identify the key lessons to be learned from the referendum for the future of opinion polling.

AGENDA

13.00Registration opens
13.30Welcome from Jane Frost (Market Research Society)
13.35Introduction from the Chair: Sharon Witherspoon MBE (Academy of Social Sciences)
13.40Polling in the EU Referendum: an overview, John Curtice (British Polling Council & University of Strathclyde)
14.05The challenges of polling by phone in the EU Referendum, Ben Page (Ipsos-MORI)
14.20The challenges of polling via the internet in the EU Referendum, Adrian Drummond (Opinium)
14.35Discussion and Q&A
15.00Tea & Coffee
15.20Sampling and mode of interview, Patrick Sturgis(NCRM, University of Southampton
15.45Treatment of don’t knows and turnout weighting, Stephen Fisher (University of Oxford)
16.10The effect of methodological adjustments during the campaign, Will Jennings ( University of Southampton)
16.35Discussion and Q&A
17.00Close

Performance of the polls in the EU referendum

Seven member companies issued ‘final’ polls of voting intentions in the EU referendum. While no company forecast the eventual result exactly, in three cases the result was within the poll’s margin of error of plus or minus three points. In one case Leave were correctly estimated to be ahead. In the four remaining cases, however, support for Remain was clearly overestimated. This is obviously a disappointing result for the pollsters, and for the BPC, especially because every single poll, even those within sampling error, overstated the Remain vote share.

Polling of voting intentions in the referendum was a unique event, and the lessons of this experience will not necessarily be transferable to polling of future general elections. Consequently, the BPC does not believe that a full-scale independent inquiry, similar to that set up after the 2015 General Election, would justify the large amount of time involved. However, the Council will be asking all its members to look carefully at the methods that they used in the referendum and to report their findings, and will consider inviting an external reviewer to consider the methods that were used.

BPC President John Curtice commented, ‘Polling of voting intentions in this referendum was always going to be a difficult exercise. Yet their central message – that this looked like a close referendum that neither side could be sure of winning – proved prescient. Nevertheless, there will be an obligation on members to try and establish why there was a tendency to overestimate support for Remain, and the Council will report later this year on why that appears to have been the case.’

The following table shows the results of all polls conducted by BPC members that had at least some fieldwork within the last 4 days of the campaign.

Remain Leave error on remain Method Sample Size Fieldwork
ORB 54 46 +6 telephone 877 14-19 June
Survation 51 49 +2 telephone 1003 20-June
ComRes 54 46 +6 telephone 1032 17-22 June
Opinium 49 51 +1 online 3011 20-22 June
YouGov 51 49 +3 online 3766 20-23 June
Ipsos MORI 52 48 +4 telephone 1592 21-22 June
Populus 55 45 +7 online 4740 21-22 June
RESULT 48 52
Average error 4.3

As well as the pre-election polls there were three published on-the-day polls, by BMG, YouGov and Ipsos MORI. As they combine people who are reporting their actual vote and others who are reporting their voting intention they are treated separately from the pre-election polls above. The YouGov poll interviewed 4,772 people online and produced a result of Remain 52% Leave 48%. Ipsos MORI interviewed 546 people by telephone and produced a result of Remain 54% Leave 46%. This is a smaller sample size than their standard published polls as it was not intended as a standalone on-the-day poll, and only presented as a day-by-day comparison drawing on their final poll. BMG interviewed 2,000 people online with a declared result of Remain 46% Leave 41% and Don’t know 13%.

The ORB and Ipsos MORI polls covered only Great Britain. All the other polls above covered the whole of the UK.

British Polling Council member TNS-BMRB also conducted a poll within the fieldwork period above. However, they took the decision not to remove and reallocate in any way those who said they were undecided, or that they would not vote, and so no calculation of error is possible. For the record, their final poll was conducted online from 16-22 June, with a sample size of 2,320, and the results were Remain 41% Leave 43% Undecided/would not vote 16%

In the interests of completeness, British Polling Council member BMG Research conducted two polls, one telephone and one online, that finished over a week before polling day, and which are therefore not included in the table above. Fieldwork was conducted from 10-15 June, with sample sizes of 1,043 (telephone) and 1,468 (online). The results were Remain 53% Leave 47% (telephone) and Remain 44% Leave 56% (online).

British Polling Council member LucidTalk conducted a poll solely in Northern Ireland. Fieldwork was conducted online from 15-17 June, with a prediction of Remain 57% Leave 43%. As the actual Northern Ireland result was Remain 56% Leave 44% this represents an error of only 1% in its estimate of the vote.

Performance of the Opinion Polls in the 2016 Local and Devolved Elections

The following tables compare the results of opinion polls of voting intentions for the elections that were held on May 5th in London, Scotland and Wales with the final outcome. A poll is included if its fieldwork was conducted wholly or mostly in the seven days before polling day (May 5th), and it was the final poll published by the pollster concerned. (This is a slightly longer period than is normally used in BPC reports on the performance of the polls in general elections — the period has been lengthened because there were very few polls conducted in the last few days.)

It should be noted that all of these polls were conducted online, with the exception of the Survation poll in Scotland, which was conducted by telephone.

London Mayor 1st Preference Vote

Company / Fieldwork Dates Con Lab LibDem UKIP Green Other
Opinium 26.4–1.5.16 35 48 4 5 5 3
TNS 26.4–3.5.16 33 45 7 5 4 5
ComRes 28.4–3.5.16 36 45 6 4 6 3
YouGov 2–4.5.16 32 43 6 7 7 5
RESULT 35 44 5 4 6 7

London Mayor After Redistribution of 2nd Preferences

Company / Fieldwork Dates Con Lab
Opinium 26.4–1.5.16 43 57
TNS 26.4–3.5.16 43 57
ComRes 28.4–3.5.16 44 56
YouGov 2–4.5.16 43 57
RESULT 43 57

London Assembly — Constituency Vote

Company / Fieldwork Dates Con Lab LibDem UKIP Green Other
YouGov 2–4.5.16 30 44 7 11 7 1
RESULT 31 42 8 8 9 3

London Assembly — List Vote

Company / Fieldwork Dates Con Lab LibDem UKIP Green Other
YouGov 2–4.5.16 29 39 8 11 9 5
RESULT 29 40 6 8 8 3

Scotland — Constituency Vote

Company / Fieldwork Dates Con Lab LibDem SNP Green Other
Survation 1–2.5.16 19 21 7 49 5
YouGov 2–4.5.16 19 22 7 48 4
RESULT 22 23 8 47 1

Scotland — List Vote

Company / Fieldwork Dates Con Lab LibDem SNP Green Other
Survation 1–2.5.16 20 19 8 43 7 4
YouGov 2–4.5.16 20 19 6 41 9 5
RESULT 23 19 5 42 7 4

Wales — Constituency Vote

Company / Fieldwork Dates Con Lab LibDem PC UKIP Other
YouGov 2–4.5.16 21 33 8 19 16 4
RESULT 21 35 8 21 12 3

Wales — List Vote

Company / Fieldwork Dates Con Lab LibDem PC UKIP Other
YouGov 5.5.16 20 31 6 20 16 8
RESULT 19 31 6 21 13 3

YouGov also conducted on polling day an exercise in which it recontacted a sample of those who it had interviewed previously in order to ascertain how they had or intended to vote. This produced the same result for the Constituency vote, and very similar figures for the list vote: Con 19%, Lab 30% LibDem 6%, PC 21%, UKIP 16% and other 8%.

BPC Inquiry Report

The British Polling Council welcomes the publication today of the report of the Independent Inquiry into the performance of the polls in the May 2015 general election. It is deeply grateful to the Chair, Prof. Patrick Sturgis, and his colleagues for their forensic and thorough analysis of why the polls underestimated Conservative and overestimated Labour performance at that election.

The principal objective of the Council is to promote transparency in the reporting of opinion polls, and its members are required to adhere to a set of rules designed to achieve that objective. The Inquiry’s report makes a series of recommendations for changes to those rules.

The Council has resolved that so far as some of those recommendations are concerned, the necessary changes to its rules should be made with immediate effect. Other of the Inquiry’s recommendations require preparatory work be undertaken before they can be implemented, and the Council has agreed that that work should be put in train in the expectation that the consequent rule changes can be introduced early in 2017.

Specifically, the Council has agreed to implement immediately rule changes that will (i) require greater transparency about how polls have been weighted, (ii) specify what changes, if any, have been made since a company’s previous published poll in how the data have been weighted or otherwise adjusted, and (iii) place an obligation on members to supply to any inquiry or committee that has been established by the BPC the micro data set for any poll in which that inquiry or committee has an interest.

Meanwhile, the Council has agreed that work should be undertaken to develop (i) an industry-wide method for calculating the confidence limits associated with a poll’s estimate of a party’s share of the vote, and (ii) an industry-wide approach to calculating the statistical significance of the change in a party’s estimated vote share since a company’s previously published poll. Rules that will require members to publish these calculations for any poll of vote intentions will be introduced once this work is completed. At the same time the Council will also consider how best to respond to the Inquiry’s recommendation that members should register with the BPC the fact that they are undertaking a poll.

The Inquiry also makes a series of recommendations for changing the way in which polls are conducted. It will be for individual member companies to decide how best to take these forward. At the same time, however, the Council is aware of the need to show that due note has been taken of those recommendations and to make clear what changes have as a result been made to the way in which polls are conducted. The Council has thus also resolved that (assuming the next election is held in May 2020) it will issue a report in the second half of 2019 that describes the methodology that is being used its members in the run up to the next election and how this methodology has changed since 2015.

Prof. John Curtice, President of the British Polling Council, said, ‘The Inquiry has undertaken what was an important but demanding task in a timely and professional fashion. I am confident that all those with an interest in understanding the difficulties that beset the polls in 2015 will find its report an illuminating and profitable read. The Council now wishes to ensure that its work is put to best use so that the transparency and accuracy of opinion polls is enhanced in future.’

Notes to Editors

  1. The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results.
  2. The Inquirys Report is embargoed until 0001 hours on 31 March, and will be available online at National Centre for Research Methods (NCRM)
  3. For further information, please contact a member of the BPC’s Management Committee:
    • Simon Atkinson: 07791-680 770
    • Nick Moon: 07770-564 664
    • John Curtice 07710-348 755

Minutes of the Annual General Meeting 2015

The Meeting was held at GfK NOP on 23 March 2015

Present:

  • John Curtice, President
  • Nick Moon, GfK NOP, Secretary
  • Simon Atkinson, Ipsos MORI
  • Anthony Wells, YouGov
  • Johnny Heald, ORB
  • Damian Lyons Lowe, Survation
  • Tom Mludzinski, ComRes
  • Laurence Stellings, Populus
  • Ivor Knox, Panelbase

Apologies:

  • Bill White, Lucid Talk
  • Harris Interactive
  • Mindmetre
  • TNS BMRB

Minutes:

The minutes of the last meeting were agreed.

Officers’ Report:

Mindmetre were approved as new members during this year.

There were two complaints during the year, but only one case where the Executive Committee asked someone to provide information they had not already provided. In the other case clarification from the member showed that the poll was not covered by BPC rules.

John Curtice reported that he had repeated the earlier invitation to Lord Ashcroft to join the BPC, but had not had a reply. There was then a discussion about how much input members had to have to a poll conducted for Lord Ashcroft before they were obliged to make details available under BPC rules. It was agreed that Nick Moon would circulate a note about this, though in most cases the Rule 2.7 outlined below would take Ashcroft polls out of BPC rules.

John Curtice also reported that he had written to the Yes and No campaigns in Scotland, drawing their attention to BPC rules.
An application for membership had been received during the year, which led to a discussion about what exactly constituted a “polling organisation” as in the BPC Rules. It was agreed that there were two conditions that must be fulfilled by an organisation wishing to join the BPC:

  • must be operationally independent – eg not itself a proselytising organisation
  • must conduct polls for more than one client

One impact of this decision is that Lord Ashcroft would not be eligible to be a BPC member.

Financial Report:

At the start of 2012 BPC had £7,089 in the bank. During the year there was membership income of £500 and bank charges of £60. The balance at the end of 2012 was £7,529. However there is cheque for £240 for the website that has not been cashed. Also, invoices for the reduced 2013 membership fees will be sent out with the 2014 invoices, and if all these are paid there will be an additional £350 of income. This means that the balance sheet at the end of the year is £7,646, although Chairman’s expenses have not yet been submitted for the 2013 AGM.

Sub-committee on disclosure:

Nick Moon and Simon Atkinson have both been on the sub-committee since its formation, but as they are now Officers they need to be replaced. It was agreed that Martin Boon and Andrew Cooper would be invited to join. Martin was present and agreed, and Nick Moon will write to Andrew.

Election of Officers:

John Curtice was re-elected unopposed as President
Nick Moon was re-elected unopposed as Secretary
Simon Atkinson was elected unopposed as member of the Management Committee

Motions

Nick Moon tabled four amendments to the BPC rules:

5.3. The President will take the chair at the AGM and any EGM. In the absence of the President, members present at the meeting will nominate a chairman for that meeting. Members may also appoint the President, in advance of the meeting, to cast their vote by Proxy according to their instructions.

5.7 subject to the agreement of two thirds of members, amendments to the Objects and Rules may be made via an email vote if there is agreed to be sufficient urgency

2.6.2. The rules of disclosure apply to all polls published in the UK by BPC members, irrespective of where they are conducted. The rules of disclosure do not apply where the survey organisation has no responsibility for the design of the survey or of the analysis, and all weighting of the data is according to client instruction

2.7. Any member of the BPC responsible for conducting a private survey that has entered the public domain, with the BPC member identified by the client, will acknowledge having conducted the survey.

All were passed unanimously.

In the case of 2.7 it was agreed that it would be good practice for members to say “no comment” rather than deny having conducted the poll

Any Other Business:

John Curtice reminded members about the date when the 28 day rule for elections would come into effect, and also said that Sunday polls should ideally be on the pollster’s website on the Sunday.

Nick Moon reported that Companies House were disputing our right to use the words “British” and ”Council” in our name. Members need to write to Companies House to say that the BPC includes all relevant practitioners, and that there was no competing body.

There was some discussion as to whether BPC should hold a post-mortem conference after the election. It was agreed that we would ask for a slot at the Political Communications s conference on 17/18 June.

It was agreed that the BPC table on the accuracy of the polls would define a final poll as a poll that was conducted entirely before the polls open, and included at least some fieldwork on the Tuesday

It was agreed that the Management Committee would decide whether a separate report on any on-the-day or hybrid polls would be necessary.

British Polling Council Welcomes Unveiling of the Provisional Findings of Polling Inquiry

The British Polling Council (BPC) welcomes the unveiling today of the provisional findings of the Polling Inquiry on why the polls overestimated Labour and underestimated Conservative support in last May’s general election. Though established shortly after the last election by the BPC in collaboration with the Market Research Society, the Inquiry has been working wholly independently of its sponsors under the chairmanship of Prof. Patrick Sturgis of the University of Southampton. The Council is deeply grateful to Prof. Sturgis and his colleagues for the work they have undertaken so far and looks forward to receiving their final report in March.

Prof. John Curtice, President of the British Polling Council, said, ‘Today’s unveiling will provide the polling companies and everyone else with an interest in its work to hear and respond to the Inquiry’s initial conclusions. It hopes that this process will prove helpful to the work of the Inquiry and ensure that its final conclusions are based on the strongest possible body of evidence.’

Notes to Editors.

  1. The British Polling Council (BPC) is an association of polling organisations that publish polls. The Council promotes standards of disclosure that are designed to provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Most of the companies that conducted polls of voting intention at the 2015 UK general election are members.
  2. The Polling Inquiry is presenting its provisional conclusions about the performance of the polls at a meeting held at the Royal Statistical Society this afternoon. Further details are to be found at the NCRM — The Inquiry into the Failure of the 2015 Pre-election Polls:.
  3. The terms of reference of the Polling Inquiry can be found at NCRM — BPC/MRS Inquiry into the Performance of the Opinion Polls at the 2015 General Election, and its membership at NCRM — Panel Membership. The Inquiry is due to submit its final written report to the BPC and the Market Research Society at the beginning of March.

British Polling Council Announces Findings in Respect of Complaint by Mr Dominic Cummings

The British Polling Council (BPC) is today announcing its findings in respect of a complaint from Mr Dominic Cummings of Vote Leave about a survey of members of the CBI conducted by YouGov in June and July 2013.

The BPC’s principal aim is to ensure that its member companies are as transparent as possible about the methods that they use to conduct their polls. It does not adjudicate on the merits of the methodologies that are used in polls — and consequently it has not done so in this case. Our findings do not represent in any way a judgement on the merits of the methodology used by YouGov in the CBI poll.

Under its Rules, any complaint to the Council is in the first instance considered by the officers. If they determine that the Rules may not have been fully observed, the matter is drawn to the attention of the member company in question. Should the company agree to take any necessary action, the matter is then considered to have been resolved.

As indicated in a press release issued by the BPC on 2 November, because his initial view on the complaint had inadvertently become publicly known, the Secretary of the BPC, Mr Nick Moon, took no part in the handling of this complaint. It has been handled throughout by the President of the BPC, Prof. John Curtice, and the BPC Committee Member, Mr. Simon Atkinson.

The officers took the view that the complaint made two points that were relevant to the Council’s Rules.

  • That YouGov had failed to make clear whether and how the data in the survey had been weighted.
  • That YouGov had not provided an adequate explanation of the sampling procedures that had been used to conduct the survey.

On the first, the officers took the view that there were not sufficient grounds to uphold the complaint. On the second the officers noted that YouGov had supplied details of the sampling procedures to Vote Leave on request, but that this information was not publicly available on its web site. While YouGov’s website does provide a full description of how they conduct their regular polls amongst members of the general public, that statement did not cover this particular survey.

YouGov have corrected this omission and appended to the detailed tabulations of the CBI poll that are available on its website an explanation of the sampling procedures that were used to conduct the poll. Accordingly, the matter is now considered to be resolved.

Notes to Editors

  1. The BPC does not usually issue a press statement in respect of complaints that are made to it. It has done so exceptionally on this occasion because of the media publicity and interest that surrounded the original lodging of Mr Cummings’ complaint.
  2. The Objects and Rules of the British Polling Council can be found at Objects And Rules
  3. The detailed tabulations for the CBI poll can be found at YouGov / CBI Survey Results
  4. The methodological statement that has been appended to these tables reads as follows:

This was a survey of members of the CBI.

A list of members was given to YouGov by the CBI. All were invited to complete the survey, both by sending a letter and a paper questionnaire to their business address and by sending an online link to the survey by email.

To maximise the response rate the survey was left open for completion for a period of six weeks (from 13th June to 30th July 2013). During that time non-respondents were sent two reminder emails and contacted by telephone to encourage their participation.

The final number of responses was 415.

Statement re Complaint by Mr Dominic Cummings

The BPC today received a complaint from Mr Dominic Cummings about a poll conducted by YouGov for the CBI.

On receipt, any complaint is initially examined by the three principal officers of the British Polling Council, Prof. John Curtice (President), Mr Nick Moon (secretary) and Mr Simon Atkinson (committee member).

Unfortunately in expressing an initial provisional view about the complaint to his fellow officers Mr Moon copied Mr Cummings in to the relevant email and in so doing used some unnecessarily offensive language. For the latter Mr Moon has apologised unreservedly to Mr Cummings.

However, Mr Cummings appears to have taken the email as the determination of the BPC on his complaint and chosen to make Mr Moon’s provisional views public knowledge. In this Mr Cummings was clearly mistaken. The BPC has not made any determination on the matter, and apologises to YouGov and the CBI for its role in any unnecessary embarrassment that has been caused.

In view of this incident, the initial determination on Mr Cummings complaint will now be made by Prof. Curtice and Mr Atkinson alone.

@BritPollingCncl