Tag Archives: 2015

BPC Inquiry Report

The British Polling Council welcomes the publication today of the report of the Independent Inquiry into the performance of the polls in the May 2015 general election. It is deeply grateful to the Chair, Prof. Patrick Sturgis, and his colleagues for their forensic and thorough analysis of why the polls underestimated Conservative and overestimated Labour performance at that election.

The principal objective of the Council is to promote transparency in the reporting of opinion polls, and its members are required to adhere to a set of rules designed to achieve that objective. The Inquiry’s report makes a series of recommendations for changes to those rules.

The Council has resolved that so far as some of those recommendations are concerned, the necessary changes to its rules should be made with immediate effect. Other of the Inquiry’s recommendations require preparatory work be undertaken before they can be implemented, and the Council has agreed that that work should be put in train in the expectation that the consequent rule changes can be introduced early in 2017.

Specifically, the Council has agreed to implement immediately rule changes that will (i) require greater transparency about how polls have been weighted, (ii) specify what changes, if any, have been made since a company’s previous published poll in how the data have been weighted or otherwise adjusted, and (iii) place an obligation on members to supply to any inquiry or committee that has been established by the BPC the micro data set for any poll in which that inquiry or committee has an interest.

Meanwhile, the Council has agreed that work should be undertaken to develop (i) an industry-wide method for calculating the confidence limits associated with a poll’s estimate of a party’s share of the vote, and (ii) an industry-wide approach to calculating the statistical significance of the change in a party’s estimated vote share since a company’s previously published poll. Rules that will require members to publish these calculations for any poll of vote intentions will be introduced once this work is completed. At the same time the Council will also consider how best to respond to the Inquiry’s recommendation that members should register with the BPC the fact that they are undertaking a poll.

The Inquiry also makes a series of recommendations for changing the way in which polls are conducted. It will be for individual member companies to decide how best to take these forward. At the same time, however, the Council is aware of the need to show that due note has been taken of those recommendations and to make clear what changes have as a result been made to the way in which polls are conducted. The Council has thus also resolved that (assuming the next election is held in May 2020) it will issue a report in the second half of 2019 that describes the methodology that is being used its members in the run up to the next election and how this methodology has changed since 2015.

Prof. John Curtice, President of the British Polling Council, said, ‘The Inquiry has undertaken what was an important but demanding task in a timely and professional fashion. I am confident that all those with an interest in understanding the difficulties that beset the polls in 2015 will find its report an illuminating and profitable read. The Council now wishes to ensure that its work is put to best use so that the transparency and accuracy of opinion polls is enhanced in future.’

Notes to Editors

  1. The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results.
  2. The Inquirys Report is embargoed until 0001 hours on 31 March, and will be available online at National Centre for Research Methods (NCRM)
  3. For further information, please contact a member of the BPC’s Management Committee:
    • Simon Atkinson: 07791-680 770
    • Nick Moon: 07770-564 664
    • John Curtice 07710-348 755

Minutes of the Annual General Meeting 2015

The Meeting was held at GfK NOP on 23 March 2015

Present:

  • John Curtice, President
  • Nick Moon, GfK NOP, Secretary
  • Simon Atkinson, Ipsos MORI
  • Anthony Wells, YouGov
  • Johnny Heald, ORB
  • Damian Lyons Lowe, Survation
  • Tom Mludzinski, ComRes
  • Laurence Stellings, Populus
  • Ivor Knox, Panelbase

Apologies:

  • Bill White, Lucid Talk
  • Harris Interactive
  • Mindmetre
  • TNS BMRB

Minutes:

The minutes of the last meeting were agreed.

Officers’ Report:

Mindmetre were approved as new members during this year.

There were two complaints during the year, but only one case where the Executive Committee asked someone to provide information they had not already provided. In the other case clarification from the member showed that the poll was not covered by BPC rules.

John Curtice reported that he had repeated the earlier invitation to Lord Ashcroft to join the BPC, but had not had a reply. There was then a discussion about how much input members had to have to a poll conducted for Lord Ashcroft before they were obliged to make details available under BPC rules. It was agreed that Nick Moon would circulate a note about this, though in most cases the Rule 2.7 outlined below would take Ashcroft polls out of BPC rules.

John Curtice also reported that he had written to the Yes and No campaigns in Scotland, drawing their attention to BPC rules.
An application for membership had been received during the year, which led to a discussion about what exactly constituted a “polling organisation” as in the BPC Rules. It was agreed that there were two conditions that must be fulfilled by an organisation wishing to join the BPC:

  • must be operationally independent – eg not itself a proselytising organisation
  • must conduct polls for more than one client

One impact of this decision is that Lord Ashcroft would not be eligible to be a BPC member.

Financial Report:

At the start of 2012 BPC had £7,089 in the bank. During the year there was membership income of £500 and bank charges of £60. The balance at the end of 2012 was £7,529. However there is cheque for £240 for the website that has not been cashed. Also, invoices for the reduced 2013 membership fees will be sent out with the 2014 invoices, and if all these are paid there will be an additional £350 of income. This means that the balance sheet at the end of the year is £7,646, although Chairman’s expenses have not yet been submitted for the 2013 AGM.

Sub-committee on disclosure:

Nick Moon and Simon Atkinson have both been on the sub-committee since its formation, but as they are now Officers they need to be replaced. It was agreed that Martin Boon and Andrew Cooper would be invited to join. Martin was present and agreed, and Nick Moon will write to Andrew.

Election of Officers:

John Curtice was re-elected unopposed as President
Nick Moon was re-elected unopposed as Secretary
Simon Atkinson was elected unopposed as member of the Management Committee

Motions

Nick Moon tabled four amendments to the BPC rules:

5.3. The President will take the chair at the AGM and any EGM. In the absence of the President, members present at the meeting will nominate a chairman for that meeting. Members may also appoint the President, in advance of the meeting, to cast their vote by Proxy according to their instructions.

5.7 subject to the agreement of two thirds of members, amendments to the Objects and Rules may be made via an email vote if there is agreed to be sufficient urgency

2.6.2. The rules of disclosure apply to all polls published in the UK by BPC members, irrespective of where they are conducted. The rules of disclosure do not apply where the survey organisation has no responsibility for the design of the survey or of the analysis, and all weighting of the data is according to client instruction

2.7. Any member of the BPC responsible for conducting a private survey that has entered the public domain, with the BPC member identified by the client, will acknowledge having conducted the survey.

All were passed unanimously.

In the case of 2.7 it was agreed that it would be good practice for members to say “no comment” rather than deny having conducted the poll

Any Other Business:

John Curtice reminded members about the date when the 28 day rule for elections would come into effect, and also said that Sunday polls should ideally be on the pollster’s website on the Sunday.

Nick Moon reported that Companies House were disputing our right to use the words “British” and ”Council” in our name. Members need to write to Companies House to say that the BPC includes all relevant practitioners, and that there was no competing body.

There was some discussion as to whether BPC should hold a post-mortem conference after the election. It was agreed that we would ask for a slot at the Political Communications s conference on 17/18 June.

It was agreed that the BPC table on the accuracy of the polls would define a final poll as a poll that was conducted entirely before the polls open, and included at least some fieldwork on the Tuesday

It was agreed that the Management Committee would decide whether a separate report on any on-the-day or hybrid polls would be necessary.

Details of Opinion Poll Inquiry Announced

The British Polling Council (BPC) publishes today further details of the Inquiry into the performance of the opinion polls that it has established in collaboration with the Market Research Society (MRS).

​Under the chairmanship of Prof. Patrick Sturgis, Director of the National Centre for Research Methods at the University of Southampton, the Inquiry is charged with the task of establishing the degree of inaccuracy in the polls, the reasons for the inaccuracies it identifies, and whether the findings and conduct of the polls were adequately communicated to the general public. Due to report by 1 March next year, the Inquiry will seek and welcomes submissions from all interested parties, and is empowered both to make recommendations about the future practice of polling and, where appropriate, for changes in the rules of the BPC. The BPC and MRS are committed to publishing the Inquiry’s report in full.

​Eight people with professional expertise and experience in conducting and analyzing survey and polling data, have agreed to serve (unpaid) as members of the Inquiry. None of them were directly involved in conducting published polls during the election campaign. They are as follows:

  • ​Dr. Nick Baker, Managing Director, Quadrangle Research
  • ​Dr. Mario Callegaro, Senior Survey Research Scientist, Google UK
  • Dr. Stephen Fisher, Associate Professor of Political Sociology, University of Oxford, who runs the Electionsetc website
  • Dr. Jouni Kuha, Associate Professor of Statistics, London School of Economics and lead statistician for the BBC/ITV/Sky exit poll
  • ​Prof. Jane Green, Professor of Political Science, University of Manchester and Co-Director of the 2015 British Election Study
  • ​Prof. Will Jennings, Professor of Political Science and Public Policy, University of Southampton, and a member of the Polling Observatory team.
  • ​Dr Ben Lauderdale, Associate Professor in Research Methodology, London School of Economics and one of the team behind the election forecast website.
  • ​Dr. Patten Smith, Research Director, Research Methods Centre, Ipsos MORI and Chair of the Social Research Association.

​Information about the work of the Inquiry will be available via a website launched today at National Centre for Research Methods (NCRM). As a first step the Inquiry is inviting written submissions, which can be uploaded via the website. A public meeting will be held during the afternoon of 19 June at the Royal Statistical Society, London, where there will be an opportunity to discuss the work of the Inquiry. The event will be free to attend but registration will be required. Registration will open, via the NCRM website, on Tuesday 26 May. Further information and updates about the conduct of the inquiry will be made available on the website thereafter.

​Prof. John Curtice, President of the British Polling Council, said, ‘The polls clearly gave the public a misleading impression of the likely outcome of the 2015 election and this shaped the reporting of the campaign. The Council is committed to ensuring that there should a thorough and transparent investigation into what apparently went wrong, and how both the conduct and the reporting of the polls might be improved in future. We are deeply grateful to Prof. Sturgis and the members of the Inquiry, all of whom have substantial professional expertise in the methodology and analysis of surveys, for agreeing to conduct this Inquiry.’

Jane Frost CBE, MRS’ Chief Executive said, “As the world’s leading research association, we are actively supporting the British Polling Council in its investigation. We continue to support all of our accredited members in ensuring standards are met. Market research is a UK success story, the UK is a world leader in this sector, contributing over £3.6bn to the UK economy. We continue to learn, adapt and innovate.”

For further information:

Notes to Editors

  1. The full details of the Terms of the Reference of the Inquiry are appended to this release.
  2. The British Polling Council (BPC) is an association of polling organisations that publish polls. The Council promotes standards of disclosure that are designed to provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Most of the companies that conducted polls of voting intention at the 2015 UK general election are members. Further details can be found at http://www.britishpollingcouncil.org/.
  3. The Market Research Society (MRS) is the world’s leading professional research association, training and regulating the research sector in the UK. The research sector is a major UK industry worth a conservative £3.6bn (GVA) per annum.
  4. The original announcement of the establishment of the Inquiry can be found at General Election: 7 May 2015

BPC/MRS Inquiry into the Performance of the Opinion Polls at the 2015 General Election.

Terms of Reference

  1. To assess the accuracy of the published opinion polls (both national and sub-national) at the 2015 general election.
  2. To evaluate whether any inaccuracies identified might be part of a pattern evident at previous elections.
  3. To investigate the causes of any inaccuracies that are identified. Potential causes to be considered will include (but not necessarily be limited to): the possible impact of late changes in vote preferences, sampling methods, interview mode, weighting and filtering, population coverage, item refusal, differential availability and willingness to participate, question order and wording.
  4. To assess whether the analysis or reporting of polls was influenced by a reluctance to be out of line with the published figures of other polls.
  5. To consult and seek relevant evidence from all appropriate stakeholders, including but not exclusively, polling organisations that are members of the BPC.
  6. To assess whether adequate information was provided and communicated to interested commentators and the public about how polls were conducted and what their results meant.
  7. To make, as it sees fit, recommendations for improving how opinion polls are conducted and published in future.
  8. To make recommendations, if necessary, for changing the rules and obligations of BPC membership.
  9. To submit a report to the BPC and MRS by 1 March 2016, with a view to its publication by BPC and MRS as soon as possible thereafter.

General Election: 7 May 2015

The final opinion polls before the election were clearly not as accurate as we would like, and the fact that all the pollsters underestimated the Conservative lead over Labour suggests that the methods that were used should be subject to careful, independent investigation.

The British Polling Council, supported by the Market Research Society, is therefore setting up an independent enquiry to look into the possible causes of this apparent bias, and to make recommendations for future polling.

We are pleased to announce that Professor Patrick Sturgis, who is Professor of Research Methodology and Director of the ESRC National Centre for Research Methods, has agreed to chair the enquiry, and will take the lead in setting its terms of reference. The membership of the enquiry will be announced in due course.

The headline results for the final opinion polls are set out below:

Con Lab Lib Dem UKIP Green Other Method Sample Size Fieldwork
% % % % % % n
Opinium 35 34 8 12 6 5 online 2960 May 4-5
Survation 31 31 10 16 5 7 online 4088 May 4-6
Ipsos MORI 36 35 8 11 5 5 telephone 1186 May 5-6
ICM 34 35 9 11 4 7 telephone 2023 May 3-6
ComRes 35 34 9 12 4 6 telephone 2015 May 3-5
Populus 33 33 10 14 5 6 online 3917 May 5-6
YouGov 34 34 10 12 4 6 online 10307 May 4-6
Panelbase 31 33 8 16 5 7 online 3019 May 4-6
Average 33.6 33.6 9 13 4.8 6.1
Result 37.8 31.2 8.1 12.9 3.8 6.3
Difference -4.2 2.4 0.9 0.1 1 -0.2

Reading The Polls: Election 2015 and The British Polling Council

This will almost undoubtedly prove to be the most polled election campaign ever. After all, YouGov in particular have been polling almost every day throughout the course of the last five years, and they are not suddenly going to stop doing so now. There are nine other companies who are also all polling on a regular basis. Meanwhile the apparent closeness of the election race will encourage newspapers to spend as much money as they can on their own exclusive polls in the hope of being the news organisation that first breaks the news that the deadlock has finally been broken (if it ever is).

But polling is far from being a straightforward enterprise. Those who undertake polls are attempting to provide an accurate measure of the nation’s political pulse at a time when people have busier lives than ever, when many are increasingly reluctant to answer any kind of survey, and when no less than three insurgent political parties are enjoying unprecedented levels of support. There are evidently plenty of potential pitfalls to avoid.

At the same time, it is clear that the polls have influence. In recent weeks there has been much discussion of who might be willing to do a deal with whom in the event of a hung parliament, all of it predicated on the evidence of the polls that Conservative and Labour are neck and neck and that the SNP might displace the Liberal Democrats as the third largest party in the Commons. Without this evidence the subject matter and tone of the election campaign could well have been very different indeed.

In these circumstances it is clearly important that polls are subject to critical scrutiny. We should be able to satisfy ourselves that numbers that prove to be so influential but which are collected in what would seem quite difficult circumstances are indeed as robust and reliable as can reasonably be expected.

Making this possible is the key objective of the British Polling Council (BPC). Nearly all of the companies and organisations that conduct political opinion polls in the UK are members of the Council. In joining the body they have agreed to abide by a set of rules that demand a high level of transparency about how they go about their business.
Each member is expected to post on its website a description of how it conducts its polls and how it weights or otherwise adjusts or models the raw data it collects in order to arrive at its estimates of the balance of voting intentions. At the same time the details of each poll, including full details of the question asked and detailed tabulations of how the answers given vary by people’s demographic and political characteristics should be posted within three days of the poll being published – and indeed during an election campaign ideally within 18 hours of publication. In practice most polling companies typically publish these details very shortly indeed after initial publication.

Not that this means that all polls have to be published. Anybody has the right to commission a poll from a BPC member and keep the results to themselves. But if they do not want the details of their polling to be published, they do have to keep the results to themselves. If, for example, a commissioner starts to leak results to one or more journalists (perhaps selectively) then the BPC member becomes obligated to publish full details of the polling that has been leaked. If the results of a ‘private’ poll have been put into the public domain then they should be capable of being scrutinized in exactly the same way as a poll that was originally intended for public consumption.

However, conducting a poll is a multi-stage operation. At its most basic it requires the capacity to contact and interview successfully a representative body of voters (these days typically either by telephone or via the internet), to collate the results and to weight the data to a standard demographic scheme so that it has, for example, the correct proportion of men and women younger and older people etc. But it also requires an ability to identify a suitable sample design, to craft suitable questions and to undertake more complex weighting and filtering of the data than simply making sure it has the right proportion of men and women.

Not all of these stages are necessarily conducted by the same organisation. In particular a polling company (or indeed other organisation such as a university or a government department) may not have the capacity to undertake the fieldwork for a poll and thus opts to sub-contract it to a polling company that does. The job of the sub-contractor is simply to conduct the interviews and tabulate the results according to the specification of the contractor. In these circumstances the BPC decided some time ago that the body that should be regarded as responsible for the poll is the company or organisation that designed and commissioned the fieldwork, not the firm that did the interviewing.

This issue of who is regarded as responsible for a poll has arisen on a couple of occasions recently. One of the most active pollsters in recent years has, of course, been Lord Ashcroft, operating under the banner ‘Lord Ashcroft Polls’. Lord Ashcroft Polls does not have the ability to conduct its own fieldwork and thus sub-contracts this part of its polling to a number of companies, many of them BPC members. However, Lord Ashcroft Polls is responsible for the design, weighting and question wording of its polls, and thus it is the body that is ultimately responsible for its results. As it happens, Lord Ashcroft Polls is not a member of the BPC (and as an organisation that does not do work for multiple clients is not eligible to be a member), but as it happens it publishes full details of its polls in much the same way as a BPC member would be expected to do.

At the same time the Liberal Democrat Party has been undertaking quite a lot of polling in constituencies that it currently holds, seemingly with a view to establishing in which ones they might have a chance of winning again. Here too the party has been responsible for the design, and wording of the polling and for the weighting of the data, but has sub-contracted the fieldwork to a BPC member, in this case Survation. In recent weeks the Liberal Democrats have given journalists sight of some of their data, and in so doing apparently gave the impression that the polling was Survation’s responsibility. That, however, was not the case as Survation subsequently made clear in a statement its own web site. To date the Liberal Democrats, who are not BPC members, have published full details of one of their constituency polls, though not as yet the remainder.

BPC members will be making full details of their published polls available as quickly as possible throughout the election campaign so that everyone can come to their own view as to whether they believe the results are robust and reliable or not. But inevitably members can only do so for those polls for which they are themselves responsible. If someone claims their poll was conducted by a BPC member, do please check the claim out. It may not be true.

John Curtice is President of the British Polling Council and Professor of Politics, Strathclyde University