The British Polling Council welcomes the publication today of the report of the Independent Inquiry into the performance of the polls in the May 2015 general election. It is deeply grateful to the Chair, Prof. Patrick Sturgis, and his colleagues for their forensic and thorough analysis of why the polls underestimated Conservative and overestimated Labour performance at that election.
The principal objective of the Council is to promote transparency in the reporting of opinion polls, and its members are required to adhere to a set of rules designed to achieve that objective. The Inquiry’s report makes a series of recommendations for changes to those rules.
The Council has resolved that so far as some of those recommendations are concerned, the necessary changes to its rules should be made with immediate effect. Other of the Inquiry’s recommendations require preparatory work be undertaken before they can be implemented, and the Council has agreed that that work should be put in train in the expectation that the consequent rule changes can be introduced early in 2017.
Specifically, the Council has agreed to implement immediately rule changes that will (i) require greater transparency about how polls have been weighted, (ii) specify what changes, if any, have been made since a company’s previous published poll in how the data have been weighted or otherwise adjusted, and (iii) place an obligation on members to supply to any inquiry or committee that has been established by the BPC the micro data set for any poll in which that inquiry or committee has an interest.
Meanwhile, the Council has agreed that work should be undertaken to develop (i) an industry-wide method for calculating the confidence limits associated with a poll’s estimate of a party’s share of the vote, and (ii) an industry-wide approach to calculating the statistical significance of the change in a party’s estimated vote share since a company’s previously published poll. Rules that will require members to publish these calculations for any poll of vote intentions will be introduced once this work is completed. At the same time the Council will also consider how best to respond to the Inquiry’s recommendation that members should register with the BPC the fact that they are undertaking a poll.
The Inquiry also makes a series of recommendations for changing the way in which polls are conducted. It will be for individual member companies to decide how best to take these forward. At the same time, however, the Council is aware of the need to show that due note has been taken of those recommendations and to make clear what changes have as a result been made to the way in which polls are conducted. The Council has thus also resolved that (assuming the next election is held in May 2020) it will issue a report in the second half of 2019 that describes the methodology that is being used its members in the run up to the next election and how this methodology has changed since 2015.
Prof. John Curtice, President of the British Polling Council, said, ‘The Inquiry has undertaken what was an important but demanding task in a timely and professional fashion. I am confident that all those with an interest in understanding the difficulties that beset the polls in 2015 will find its report an illuminating and profitable read. The Council now wishes to ensure that its work is put to best use so that the transparency and accuracy of opinion polls is enhanced in future.’
Notes to Editors
- The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results.
- The Inquirys Report is embargoed until 0001 hours on 31 March, and will be available online at National Centre for Research Methods (NCRM)
- For further information, please contact a member of the BPC’s Management Committee:
- Simon Atkinson: 07791-680 770
- Nick Moon: 07770-564 664
- John Curtice 07710-348 755
The British Polling Council (BPC) welcomes the unveiling today of the provisional findings of the Polling Inquiry on why the polls overestimated Labour and underestimated Conservative support in last May’s general election. Though established shortly after the last election by the BPC in collaboration with the Market Research Society, the Inquiry has been working wholly independently of its sponsors under the chairmanship of Prof. Patrick Sturgis of the University of Southampton. The Council is deeply grateful to Prof. Sturgis and his colleagues for the work they have undertaken so far and looks forward to receiving their final report in March.
Prof. John Curtice, President of the British Polling Council, said, ‘Today’s unveiling will provide the polling companies and everyone else with an interest in its work to hear and respond to the Inquiry’s initial conclusions. It hopes that this process will prove helpful to the work of the Inquiry and ensure that its final conclusions are based on the strongest possible body of evidence.’
Notes to Editors.
- The British Polling Council (BPC) is an association of polling organisations that publish polls. The Council promotes standards of disclosure that are designed to provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Most of the companies that conducted polls of voting intention at the 2015 UK general election are members.
- The Polling Inquiry is presenting its provisional conclusions about the performance of the polls at a meeting held at the Royal Statistical Society this afternoon. Further details are to be found at the NCRM — The Inquiry into the Failure of the 2015 Pre-election Polls:.
- The terms of reference of the Polling Inquiry can be found at NCRM — BPC/MRS Inquiry into the Performance of the Opinion Polls at the 2015 General Election, and its membership at NCRM — Panel Membership. The Inquiry is due to submit its final written report to the BPC and the Market Research Society at the beginning of March.
The British Polling Council (BPC) is today announcing its findings in respect of a complaint from Mr Dominic Cummings of Vote Leave about a survey of members of the CBI conducted by YouGov in June and July 2013.
The BPC’s principal aim is to ensure that its member companies are as transparent as possible about the methods that they use to conduct their polls. It does not adjudicate on the merits of the methodologies that are used in polls — and consequently it has not done so in this case. Our findings do not represent in any way a judgement on the merits of the methodology used by YouGov in the CBI poll.
Under its Rules, any complaint to the Council is in the first instance considered by the officers. If they determine that the Rules may not have been fully observed, the matter is drawn to the attention of the member company in question. Should the company agree to take any necessary action, the matter is then considered to have been resolved.
As indicated in a press release issued by the BPC on 2 November, because his initial view on the complaint had inadvertently become publicly known, the Secretary of the BPC, Mr Nick Moon, took no part in the handling of this complaint. It has been handled throughout by the President of the BPC, Prof. John Curtice, and the BPC Committee Member, Mr. Simon Atkinson.
The officers took the view that the complaint made two points that were relevant to the Council’s Rules.
- That YouGov had failed to make clear whether and how the data in the survey had been weighted.
- That YouGov had not provided an adequate explanation of the sampling procedures that had been used to conduct the survey.
On the first, the officers took the view that there were not sufficient grounds to uphold the complaint. On the second the officers noted that YouGov had supplied details of the sampling procedures to Vote Leave on request, but that this information was not publicly available on its web site. While YouGov’s website does provide a full description of how they conduct their regular polls amongst members of the general public, that statement did not cover this particular survey.
YouGov have corrected this omission and appended to the detailed tabulations of the CBI poll that are available on its website an explanation of the sampling procedures that were used to conduct the poll. Accordingly, the matter is now considered to be resolved.
Notes to Editors
- The BPC does not usually issue a press statement in respect of complaints that are made to it. It has done so exceptionally on this occasion because of the media publicity and interest that surrounded the original lodging of Mr Cummings’ complaint.
- The Objects and Rules of the British Polling Council can be found at Objects And Rules
- The detailed tabulations for the CBI poll can be found at YouGov / CBI Survey Results
- The methodological statement that has been appended to these tables reads as follows:
This was a survey of members of the CBI.
A list of members was given to YouGov by the CBI. All were invited to complete the survey, both by sending a letter and a paper questionnaire to their business address and by sending an online link to the survey by email.
To maximise the response rate the survey was left open for completion for a period of six weeks (from 13th June to 30th July 2013). During that time non-respondents were sent two reminder emails and contacted by telephone to encourage their participation.
The final number of responses was 415.
The BPC today received a complaint from Mr Dominic Cummings about a poll conducted by YouGov for the CBI.
On receipt, any complaint is initially examined by the three principal officers of the British Polling Council, Prof. John Curtice (President), Mr Nick Moon (secretary) and Mr Simon Atkinson (committee member).
Unfortunately in expressing an initial provisional view about the complaint to his fellow officers Mr Moon copied Mr Cummings in to the relevant email and in so doing used some unnecessarily offensive language. For the latter Mr Moon has apologised unreservedly to Mr Cummings.
However, Mr Cummings appears to have taken the email as the determination of the BPC on his complaint and chosen to make Mr Moon’s provisional views public knowledge. In this Mr Cummings was clearly mistaken. The BPC has not made any determination on the matter, and apologises to YouGov and the CBI for its role in any unnecessary embarrassment that has been caused.
In view of this incident, the initial determination on Mr Cummings complaint will now be made by Prof. Curtice and Mr Atkinson alone.
The British Polling Council (BPC) urges the House of Lords to reject the private members bill on the regulation of opinion polls that is being presented today by Lord Foulkes.
The Bill proposes that an authority be established that would regulate polls of voting intentions for all elections and referendums in the United Kingdom. The authority would be empowered to specify approved ways for selecting who should be interviewed, how the questions in polls should be worded and to ban the publication of voting intention polls during an election campaign.
Who is interviewed by a poll and how the questions it asks are phrased are important issues. How any poll has addressed them should always be clearly stated, as the rules of the BPC require. But they are not issues that are susceptible to straightforwardly ‘right’ or ‘wrong’ answers. Professional researchers can and do disagree about how polls should be conducted and how they should be worded. They regularly experiment and test alternative and new ways of doing polls and asking questions in order to improve their methods. Imposing regulatory standards would put at risk the experimentation and competition that are essential to improving the ways in which polls are conducted.
Banning the publication of polls during an election campaign would not mean that polls were not conducted. It would simply mean that access to their results would be confined to those who could afford to pay for polls, such as the banks and the political parties, or who knew where to find the results on an overseas website. Only the ordinary voter, who is meant to be central to the democratic process, would be left out of the loop. It could potentially open the way for politicians to claim that their private polling showed them ahead, regardless of what their polling actually showed, or indeed whether it existed at all.
Doubtless many people feel that in underestimating the Conservative vote and overestimating Labour’s, the polls provided unhelpful misinformation during the recent general election campaign. That is why the BPC has established an independent inquiry into why the polls were wrong and how their conduct might be improved in future. Indeed, the first meeting of that inquiry, at which BPC members will be presenting – in public – their initial findings as to what went wrong, is being held today.
Professor John Curtice, President of the British Polling Council, said, ‘What is needed now is a critical and open appraisal of where the polls went wrong, not the heavy hand of regulation that in attempting to impose common standards would make it more likely that the polls all get it wrong again in future. As any economic forecaster knows too well, forecasting how people will behave is always a difficult enterprise. No-one has yet suggested that, despite their many errors, economic forecasting should be regulated, and it is not clear why attempting to anticipate how people will vote should be treated any differently.’
Notes for Editors:
- The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Website: www.britishpollingcouncil.org. Twitter: @BritPollingCncl
- The first meeting of the inquiry is at The Royal Statistical Society, 12 Erroll St, London EC1Y 8LX at 1.30 pm on 19 June. Anyone who wishes to attend should register at BPC/MRS Polling Inquiry meetings
- Members of the inquiry are Dr. Nick Baker, Group CEO, Quadrangle Research Group Ltd; Dr. Mario Callegaro, Senior Survey Research Scientist, Google UK ; Dr. Stephen Fisher, Associate Professor of Political Sociology, University of Oxford, who runs the Electionsetc website; Dr. Jouni Kuha, Associate Professor of Statistics, London School of Economics and lead statistician for the BBC/ITV/Sky exit poll; Prof. Jane Green, Professor of Political Science, University of Manchester and Co-Director of the 2015 British Election Study; Prof. Will Jennings, Professor of Political Science and Public Policy, University of Southampton, and a member of the Polling Observatory team; Dr Ben Lauderdale, Associate Professor in Research Methodology, London School of Economics and one of the team behind the electionforecast.co.uk website; Dr. Patten Smith, Research Director, Research Methods Centre, Ipsos MORI and Chair of the Social Research Association.
- Details of Lord Foulkes’ bill can be found at: Regulation of Political Opinion Polling Bill [HL] 2015-16
For further information, please contact a member of the BPC Management Committee:
Simon Atkinson: 07791 680770
Nick Moon: 07770 564664
John Curtice 07710 348755
The British Polling Council (BPC) publishes today further details of the Inquiry into the performance of the opinion polls that it has established in collaboration with the Market Research Society (MRS).
Under the chairmanship of Prof. Patrick Sturgis, Director of the National Centre for Research Methods at the University of Southampton, the Inquiry is charged with the task of establishing the degree of inaccuracy in the polls, the reasons for the inaccuracies it identifies, and whether the findings and conduct of the polls were adequately communicated to the general public. Due to report by 1 March next year, the Inquiry will seek and welcomes submissions from all interested parties, and is empowered both to make recommendations about the future practice of polling and, where appropriate, for changes in the rules of the BPC. The BPC and MRS are committed to publishing the Inquiry’s report in full.
Eight people with professional expertise and experience in conducting and analyzing survey and polling data, have agreed to serve (unpaid) as members of the Inquiry. None of them were directly involved in conducting published polls during the election campaign. They are as follows:
- Dr. Nick Baker, Managing Director, Quadrangle Research
- Dr. Mario Callegaro, Senior Survey Research Scientist, Google UK
- Dr. Stephen Fisher, Associate Professor of Political Sociology, University of Oxford, who runs the Electionsetc website
- Dr. Jouni Kuha, Associate Professor of Statistics, London School of Economics and lead statistician for the BBC/ITV/Sky exit poll
- Prof. Jane Green, Professor of Political Science, University of Manchester and Co-Director of the 2015 British Election Study
- Prof. Will Jennings, Professor of Political Science and Public Policy, University of Southampton, and a member of the Polling Observatory team.
- Dr Ben Lauderdale, Associate Professor in Research Methodology, London School of Economics and one of the team behind the election forecast website.
- Dr. Patten Smith, Research Director, Research Methods Centre, Ipsos MORI and Chair of the Social Research Association.
Information about the work of the Inquiry will be available via a website launched today at National Centre for Research Methods (NCRM). As a first step the Inquiry is inviting written submissions, which can be uploaded via the website. A public meeting will be held during the afternoon of 19 June at the Royal Statistical Society, London, where there will be an opportunity to discuss the work of the Inquiry. The event will be free to attend but registration will be required. Registration will open, via the NCRM website, on Tuesday 26 May. Further information and updates about the conduct of the inquiry will be made available on the website thereafter.
Prof. John Curtice, President of the British Polling Council, said, ‘The polls clearly gave the public a misleading impression of the likely outcome of the 2015 election and this shaped the reporting of the campaign. The Council is committed to ensuring that there should a thorough and transparent investigation into what apparently went wrong, and how both the conduct and the reporting of the polls might be improved in future. We are deeply grateful to Prof. Sturgis and the members of the Inquiry, all of whom have substantial professional expertise in the methodology and analysis of surveys, for agreeing to conduct this Inquiry.’
Jane Frost CBE, MRS’ Chief Executive said, “As the world’s leading research association, we are actively supporting the British Polling Council in its investigation. We continue to support all of our accredited members in ensuring standards are met. Market research is a UK success story, the UK is a world leader in this sector, contributing over £3.6bn to the UK economy. We continue to learn, adapt and innovate.”
For further information:
Notes to Editors
- The full details of the Terms of the Reference of the Inquiry are appended to this release.
- The British Polling Council (BPC) is an association of polling organisations that publish polls. The Council promotes standards of disclosure that are designed to provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Most of the companies that conducted polls of voting intention at the 2015 UK general election are members. Further details can be found at http://www.britishpollingcouncil.org/.
- The Market Research Society (MRS) is the world’s leading professional research association, training and regulating the research sector in the UK. The research sector is a major UK industry worth a conservative £3.6bn (GVA) per annum.
- The original announcement of the establishment of the Inquiry can be found at General Election: 7 May 2015
BPC/MRS Inquiry into the Performance of the Opinion Polls at the 2015 General Election.
Terms of Reference
- To assess the accuracy of the published opinion polls (both national and sub-national) at the 2015 general election.
- To evaluate whether any inaccuracies identified might be part of a pattern evident at previous elections.
- To investigate the causes of any inaccuracies that are identified. Potential causes to be considered will include (but not necessarily be limited to): the possible impact of late changes in vote preferences, sampling methods, interview mode, weighting and filtering, population coverage, item refusal, differential availability and willingness to participate, question order and wording.
- To assess whether the analysis or reporting of polls was influenced by a reluctance to be out of line with the published figures of other polls.
- To consult and seek relevant evidence from all appropriate stakeholders, including but not exclusively, polling organisations that are members of the BPC.
- To assess whether adequate information was provided and communicated to interested commentators and the public about how polls were conducted and what their results meant.
- To make, as it sees fit, recommendations for improving how opinion polls are conducted and published in future.
- To make recommendations, if necessary, for changing the rules and obligations of BPC membership.
- To submit a report to the BPC and MRS by 1 March 2016, with a view to its publication by BPC and MRS as soon as possible thereafter.
The final opinion polls before the election were clearly not as accurate as we would like, and the fact that all the pollsters underestimated the Conservative lead over Labour suggests that the methods that were used should be subject to careful, independent investigation.
The British Polling Council, supported by the Market Research Society, is therefore setting up an independent enquiry to look into the possible causes of this apparent bias, and to make recommendations for future polling.
We are pleased to announce that Professor Patrick Sturgis, who is Professor of Research Methodology and Director of the ESRC National Centre for Research Methods, has agreed to chair the enquiry, and will take the lead in setting its terms of reference. The membership of the enquiry will be announced in due course.
The headline results for the final opinion polls are set out below:
|Con||Lab||Lib Dem||UKIP||Green||Other||Method||Sample Size||Fieldwork|
|Ipsos MORI||36||35||8||11||5||5||telephone||1186||May 5-6|
Following a request from ICM, the Officers of the British Polling Council have considered how its rules of disclosure should apply in the case of a private poll of four Liberal Democrat held constituencies, some details of which were published in The Guardian this morning.
Rule 2.6 of the council states that, ‘Organisations conducting privately commissioned surveys have the right to maintain the confidentiality of survey findings’ However, it goes on to state that, ‘in the event the results of a privately commissioned poll are made public by the organisation [its employees or agents] that commissioned the survey, such results will be deemed to have entered the public domain’ and that consequently the council’s rules of disclosure apply. Those rules require, inter alia, that details of the survey including computer tabulations of the results, methodology and the client commissioning the survey be published on the company’s website.
BPC’s officers welcome the fact that, following the leak of the results, ICM have immediately published details of the survey including computer tabulations of the results, and in so doing have also indicated that the client was a ‘member of the Liberal Democrats’. The publication of this information should allow anyone who wishes to do so to undertake a critical scrutiny of the research, which is the purpose of rules to which all BPC members aim to adhere. The rules are intended to enable any reader of poll to reach a reasonable judgment on the likely quality or reliability of a poll. The name of the client is included in the rules for disclosure because it may be considered germane by readers of the poll if the poll was commissioned by an organization that campaigns on one side of an issue.
It should, however, be noted that the requirement upon BPC members to publish the details of results of private polls, including of the commissioning client, only applies when those results have been published by the client or by someone acting on their behalf. To the best of our knowledge the source of the leak to The Guardian is not known, including to ICM itself, and thus it has not been established that the source was either the commissioner or someone acting on their behalf. Unless these circumstances change, the officers take the view that ICM are not obliged by BPC’s rules to disclose any further information about the poll.
8th May 2010
The table below compares the final estimates of the outcome of the General Election made by companies that are members of the British Polling Council (BPC) with the actual result across Great Britain as a whole.
While not proving as accurate as the 2005 polls, which were the most accurate predictions ever made of the outcome of a British general election, the polls nevertheless told the main story of the 2010 election — that the Conservatives had established a clear lead. All but one of the nine pollsters came within 2% of the Conservative share, and five were within 1%.
The tendency at past elections for polls to overestimate Labour came to an abrupt end, with every pollster underestimating the Labour share of the vote, though all but one were within 3%. However, every pollster overestimated the Liberal Democrat share of the vote.
NOTE. The table includes the final poll conducted by each company where that poll was conducted either wholly or partly on or after Monday 3rd May. Average error is the average of the difference between the poll result and the actual result across all four estimates.
- Nick Moon (GfK NOP) 020 7890 9830
- Andrew Cooper (Populus) 020 7253 9465
2nd April 2010
Britain’s opinion pollsters announce today that they have committed themselves to releasing speedily the technical details of all voting intention polls published during the course of the general election campaign. In the event of a May 6th election, members of the British Polling Council, who between them comprise nearly all the companies who undertake opinion polls in the UK, have agreed they will place the key details of all polls published on or after Thursday 8th April on their web sites within at most 18 hours of their initial publication. These details will include:
The precise wording of the questions asked
The sample size, how and when interviews were conducted, and how the data have been weighted to ensure they are representative.
Computer tables showing the breakdown of the results for key demographic groups.
John Curtice, President of the British Polling Council, said, ‘Opinion polls often play an important role in shaping the mood and rhythm of an election campaign, sometimes controversially so. It is therefore important that they should be as open and transparent as possible about how their results have been obtained. This commitment will ensure that anyone who is interested in or has concerns about a particular poll will be able to ascertain quickly and easily for themselves just exactly how it was conducted.’
Notes to Editors.
- The British Polling Council (BPC) was founded in 2004 with the primary object of promoting the disclosure of the technical details of the results of published opinion polls so that consumers can be make an informed judgement as to the reliability and validity of their results. Further details can be found at www.britishpollingcouncil.org
- Under its rules of disclosure, BPC members are usually required to make the technical details of polls available within 2 working days of initial publication. This new 18 hour commitment, which will operate during the 4 weeks immediately prior to polling day, will ensure these details are available more quickly at a time when polls are the subject of particular media interest and commentary.
- The membership of the Council currently comprises:
- Angus Reid
- Cello MRUK
- Dods Polling
- Harris Interactive
- Ipsos MORI
- Marketing Means
- Opinion Research Business
- TNS System Three
- For further details contact John Curtice on 07710-348 755, Andrew Cooper on 07500-858626, or Nick Moon on 07770-564 664.