Category Archives: Press Releases

House of Lords Committee on Political Polling and Digital Media

Update: 12 January 2021

The latest report of the House of Lords Liaison Committee: “The Politics of Polling: an update” was published on 21 December 2020, and can be accessed here: https://committees.parliament.uk/publications/4106/documents/40691/default/

On 28th October, British Polling Council President Professor Sir John Curtice gave oral evidence to the House of Lords Select Committee on Political Polling and Digital Media.  You can watch proceedings of the Liaison Committee here on at the “Parliament Live” TV channel.

More details on the work on the Committee can be found at The politics of polling – Summary.

The British Polling Council’s response to the Committee’s 2018 report can be found at British Polling Council Welcomes Lords Committee Report on Polling.

A Guide for Journalists to the Reporting of Opinion Polls?

The British Polling Council is today publishing A Quick Guide for Journalists to the Use and Reporting of Opinion Polls.

This publication has been designed to provide practical guidance for anyone unfamiliar with polls who finds themselves charged with interpreting and writing up an opinion poll in today’s media environment.

It forms part of the British Polling Council’s response to the recommendations of the House of Lords Select Committee on Political Polling and Digital Media.

The document has been developed as a “guided tour” of the key points to bear in mind when working with opinion poll data. Its contents include:

  • An overview on how polls are conducted, including what to look out for when judging whether the sample is representative.
  • Advice on how to evaluate the questions covered in an opinion poll. For example: are they written in everyday language? Do the questions lead the respondent?
  • Guidance on how to interpret the results of polls, including key dos and don’ts when it comes to looking at sub-samples (such as differences by age) or describing changes over time.

The resource has been developed to complement the existing materials available to practitioners and users of opinion polls, including this detailed guide developed by the press regulator IMPRESS and the Market Research Society.

The Quick Guide for Journalists is available on the BPC website (Opinion Polls: Guidance for Journalists), alongside a video briefing with the President of the British Polling Council, Professor Sir John Curtice.

Commenting on the launch of the Guide, Professor Curtice says: “In its report, the House of Lords committee expressed a number of concerns about how the media report polls. Our guide is intended to help address some of these concerns by providing a quick five-minute accessible introduction to polls. It describes both the strengths and the limits of polls, outlines five key questions that should be asked of any poll, and identifies the major potential pitfalls to avoid in writing a poll story. We hope that it will help journalists in newsdesks up and down the country report polls in a way that their audience finds both interesting and informative.”

The Performance of the Polls in the 2019 General Election

After being widely criticised for their performance in the 2015 and 2017 general elections, the polls have proven more accurate in the 2019 election than in any contest since 2005.

As the table shows, on average the final polls underestimated the Conservative vote by just 1.4 points and overestimated Labour’s by only 0.5 points.

These patterns are consistent with the long-term trend for the polls to over-state Labour and under-state Tory support, but the size of the errors were so small that this did not prevent the polls correctly predicting the overall outcome of the election.

For all the other parties, the average error across the pollsters was under one point. Indeed for the LibDems, Greens, and Others the average error was a remarkably small 0.1 of a point.

 CONLABLDBrexitGREENOTHERMETHODSAMPLE SIZEFIELDWORK
Qriously433012348Online2222Dec 5-8
ICM423612324Online2011Dec 6-9
YouGov433412335Online105,612Dec 4-10
Savanta ComRes413612326Online1732Dec 9-10
BMG413214345Online1660Dec 6-11
Ipsos MORI443312236Telephone2213Dec 9-11
Kantar 443213335Online2815Dec 9-11
Deltapoll453510334Online1818Dec 9-11
Survation45349336Telephone2395Dec 10-11
Panelbase433411435Online3174Dec 10-11
Opinium453312226Online3005Dec 10-11
Average43.333.511.72.92.95.5   
Result44.733.011.82.12.85.6   
Difference-1.40.5-0.10.80.1-0.1   

There was also a Survation poll in Scotland conducted online from Dec 10-11 with 1,004 respondents.

  Poll Result
SNP 43 45.0
Con 28 25.1
Lab 20 18.6
Lib Dem 7 9.5
Other 2 1.7

Principal Changes in the Conduct and Reporting of Polls in the 2019 General Election

This document briefly summarises the main differences between the way in which member companies are conducting their polls in the latter stages of the 2019 election and the way they did so in 2017. Details of the changes that companies made in 2017 are available at How Have The Polls Changed Since 2015?. For further details and analysis of the methodology of the polls see BPC Inquiry Report.

John Curtice

BMG

The company only published one Britain-wide poll in 2017; otherwise it conducted private polls for the Labour Party.

That one published poll was a mixed-mode online and telephone poll. The company’s polls in this campaign are likely to be conducted exclusively using an online methodology.

As in 2017 data are weighted in accordance with the known levels of electoral registration among key demographic groups. However, in contrast to 2017 the data are no longer weighted by reported probability of voting or whether or not the respondent voted at the last election.

Respondents are being invited to state a vote intention from only among the parties standing in their constituency.

Deltapoll

This is the first general election to be covered by the company. Details of its methodology can be found at Deltapoll Methodology.

ICM

Data are no longer weighted according to a pre-determined level of turnout by demographic group

A 0-10 turnout scale is now used to weight the data rather than a 1-10 scale.

The voting preference of those who say they do not know how they would vote or who refuse to say how they would do so are no longer imputed on the basis of their past vote.

Data are weighted by 2016 EU referendum vote as well as 2017 general election vote.

Respondents are being invited to state a vote intention from only among the parties standing in their constituency.

Ipsos MORI

Data are no longer being weighted to match the ratios of turnout by age and tenure as recorded by the 2015 British Election Study.

Child in household is no longer used as a weight as it was in 2017, while ethnicity has been added as a weight.

Respondents who say they will vote for the Brexit Party, Liberal Democrats, Greens or Plaid Cymru are invited to state for which party they would vote if their preferred party is not standing in their constituency – and are allocated to that party if their preferred party is not standing locally.

Kantar

Weighting of the data by probability of voting is based on the respondent’s stated likelihood of voting, their age, and whether they voted in 2017, modelled using data collected by the company after the 2017 election (rather than 2015).

Weighting of the data by past vote takes into account of estimates of differential recall error by party and, in particular, a higher tendency for those who vote Labour in 2017 to fail to report that they have done so.

In similar vein to what was done in respect of those saying they were voting UKIP or Green in 2017, respondents are invited to state for which party they would vote if their preferred party is not standing in their constituency – and are allocated to that party if their preferred party is not standing locally.

Opinium

Respondents are being invited to state a vote intention from only among the parties standing in their constituency.

ORB

The company has so far only conducted one poll of vote intentions, at the very beginning of the campaign.

Panelbase

Respondents are invited to state for which party they would vote if their preferred party is not standing in their constituency – and are allocated to that party if their preferred party is not standing locally.

Savanta ComRes

Data are no longer being weighted by the voter turnout model that was used in 2017

Future polls will invite respondents to choose from among the parties standing in their constituency.

Survation

Polls are now only being conducted by phone, using the same approach as in 2017.

Data are weighted, inter alia, by 2019 European Parliament election vote.

As in 2017, respondents are being invited to state a vote intention from only among the named candidates standing in their constituency.

YouGov

There has been a slight reduction in the downweight that is applied to those who did not vote at the last election.

Respondents are being invited to state a vote intention from only among the candidates standing in their constituency.

The Performance of the Polls in the 2019 European Election

All polling of elections is subject to many potential sources of error. The historical record at general elections indicates that there is a one in ten chance that an individual poll will over or underestimate a party’s support by more than four points.

These European Elections presented particular challenges. Not only did pollsters have to face the challenge of an election in which turnout was likely to be low (and in the event stood at 36.7%), but also of estimating the level of support for two new parties, one of which went on to come top of the poll.

The table below compares the estimates of BPC members’ polls whose fieldwork ended no earlier than three days before polling with that of the actual result — though in two instances fieldwork commenced more than a week before polling day.

It reveals that on average the polls overestimated support for Labour and underestimated that for the Liberal Democrats and the Greens. However, two polls produced estimates for Labour and the Liberal Democrats that were within a point of the eventual outcome, while one was spot on in its estimate for the Greens.

There was also a less marked tendency to overestimate support for the Conservatives and the Brexit Party, and in both these instances one poll was spot on in its (integer) estimate of party support.

The detailed council by council election results showed that turnout increased more in Remain-voting areas than in Leave-voting ones, suggesting that those who voted Remain were more likely to have voted than did those who voted Leave. Meanwhile, polling conducted by Opinium on polling day suggests that those who switched from voting Labour in 2017 to either the Liberal Democrats or the Greens were more likely than most voters to have made up their minds about how to vote in the last week of the campaign. These patterns may help to explain at least some of the average error in the polls on this occasion.

 ConLabLDBrexitGreenUKIPChange UKOtherMETHODSAMPLE SIZEFIELDWORK
Opinium121715387234Online2005May 17-20
YouGov713193712346Online3864May 19-21
Panelbase122515307335Online2033May 14-21
Kantar132415278454Online2316May 14-21
Ipsos MORI915203510336Telephone1527May 20-22
BMG Research121817358245Online1601May 20-22
Survation142412327436Online2029May 22
Average11.319.416.133.48.43.03.65.1 
Result9.114.120.331.612.13.33.46.1 
Difference2.25.3-4.21.8-3.7-0.30.2-1.0 

Note: Table is based on the outcome in Great Britain. The polls conducted by Opinium and Survation were conducted across the UK as a whole, but the figures for those polls quoted here are for respondents in Great Britain only.

Polling Rules and UK Financial Markets

On 28th September, Rt Hon Nicky Morgan MP, Chair of the Treasury Committee, wrote to Professor Sir John Curtice, President of the British Polling Council. The letter focuses on the use of private polling data, particularly during the recent referendum campaigns, and follows on from media coverage during the summer. Further details, including a link to Mrs Morgan’s letter, can be found on the Treasury Committee’s web page: Change polling rules to protect integrity of UK financial markets.

Following a consultation with its membership, the British Polling Council has responded to Mrs Morgan. The contents are set out in this letter to the Rt Hon Nicky Morgan MP (pdf).

British Polling Council Introduces New Rule on Uncertainty Attached to Polls

The British Polling Council announces today that it has introduced a new requirement on its members when reporting estimates of vote intention. This requirement is an obligation to publish a statement of the level of uncertainty that has historically been associated with polls of voting intention.

The statement reads as follows:

‘All polls are subject to a wide range of potential sources of error. On the basis of the historical record of the polls at recent general elections, there is a 9 in 10 chance that the true value of a party’s support lies within 4 points of the estimates provided by this poll, and a 2 in 3 chance that they lie within 2 points.’

The statement has been developed following an analysis of the performance of the final polls at every general election since 2001. That analysis revealed that 90% of the estimates of a party’s share of the vote in the final polls were within four points of the eventual true result, while two-thirds were within two points.

Hitherto, it has often been noted that, according to statistical theory, there is a 95% chance that in a poll of a thousand people the true value of a party’s share of the vote lies within three points of the poll’s estimate. However, this statement refers solely to the error that may be occasioned by the chance variation to which all sample surveys are subject. It does not take into account any of the other many possible sources of uncertainty in a poll. The new BPC statement attempts to reflect all of the sources of uncertainty that practical experience suggests is associated with polling in Britain.

The statement is intended to discourage the media and others from reporting small changes in a party’s support – or variation between pollsters – as firm evidence of a significant change in its popularity.

The new BPC rule has been introduced in response to recommendation 11 of the Inquiry into the 2015 British general election polls that was chaired by Prof. Patrick Sturgis. This recommendation read that:

‘BPC members should provide confidence (or credible) intervals for each separately listed party in their headline share of the vote’.

The introduction of this requirement means the BPC has now responded to all of the recommendations that were made to it by Prof. Sturgis and his colleagues.

Notes to Editors

  1. The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Through full disclosure the Council aims to encourage the highest professional standards in public opinion polling and to advance the understanding, among politicians, the media and general public, of how polls are conducted and how to interpret poll results.
  2. For further details of the Council see https://www.britishpollingcouncil.org/
  3. The report of the Inquiry into the Performance of the Polls in the 2015 Election is available at the National Centre for Research Methods (NCRM) pdf

British Polling Council Welcomes Lords Committee Report on Polling

The British Polling Council welcomes the publication today (Tuesday) of the report of the House of Lords Select Committee on Political Polling and Digital Media. The Council is grateful to the Committee for its careful consideration of the role that opinion polls play in our democracy and of the challenges that currently face the conduct and reporting of polls.

The Council particularly welcomes the Committee’s conclusion that is ‘not convinced of the case for introducing a ban on the undertaking and publication of voting intention polls in the run-up to elections’ together with the Committee’s rejection of the statutory regulation of polls. In so doing, however, the Council acknowledges that this means the polling industry itself has a responsibility to promote best practice in the conduct and reporting of polls, and it is grateful to the committee for the various specific recommendations that it makes for the future work of the Council.

Most immediately, the Council is happy to accept that it should both revise its guidance to journalists on the reporting of polls and should work with other relevant organisations to develop a suitable programme of training for journalists on this subject. It is also happy to affirm its intention to continue its current practice of undertaking a post-mortem on the conduct of the polls after each election or major referendum and reporting its findings; following a public seminar held in February a report on the 2017 election is currently in preparation.

More broadly, the Committee’s recommendations envisage that the Council should adopt a wider remit and fulfill a larger role. At present, the Council is run on a purely voluntary basis with limited resources. The Council will now consider how best to resource its activities in light of the Committees report.

Professor Sir John Curtice, President of the British Polling Council, said, ‘Today’s report is a welcome contribution to a considered, informed discussion of the conduct and reporting of opinion polls in Britain. The Council welcomes the fact that the Committee recognised the weight of evidence put before it that a ban on the publication of polls would neither be desirable nor effective. At the same time, the Council accepts that the polling industry has a duty to promote high standards in the conduct and reporting of polls and will now consider how it, in collaboration with other bodies, can enhance the considerable efforts that it already makes to achieve that objective.’

For further details or to interview Prof. Curtice contact Simon Atkinson (07791 680770; simon.atkinson@ipsos.com) or Aalia Khan (07824 597435; aalia.khan@ipsos.com).

NOTES TO EDITORS

  1. The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Through full disclosure the Council aims to encourage the highest professional standards in public opinion polling and to advance the understanding, among politicians, the media and general public, of how polls are conducted and how to interpret poll results.
  2. For further details of the Council see www.britishpollingcouncil.org
  3. Details of the House of Lords Select Committee on Political Polling and Digital Media are to be found at Select Committee on Political Polling and Digital Media.

The Polls in the 2017 Election: A BPC Conference

The British Polling Council is pleased to host a special conference on 20th February 2018 to review and debate the performance of the opinion polls at the 2017 General Election.

The conference will be held at the Royal Statistical Society. Tickets are free of charge but registration is essential via Eventbrite.

Programme

Chair: Sharon Witherspoon, Academy of Social Sciences

2-2.30 Introduction & The Challenges Facing The Polls in 2017: John Curtice (BPC)

2.30-3.15 How We Performed 1: Presentations by ComRes (Jansev Jamal), Survation (Damian Lyons-Lowe) and YouGov (Anthony Wells)

3.15 Break

3.30 A New Approach? The Use of Mixed Level Regression and Post-Stratification in the 2017 Election: Ben Lauderdale (LSE)

4.00 How We Performed 2: Presentations by Kantar (Luke Taylor) & Ipsos MORI (Gideon Skinner)

4.30 An Independent Reaction and Q&A: Steve Fisher (Oxford)

Britsh Polling Council Announces Findings in Respect of Complaints about Audience Recruitment

The British Polling Council (BPC) has received two complaints, one from the Conservative Party and one from an individual member of that party, concerning the recruitment of the audience for the BBC programme, BBC Election Debate Live with Mishal Hussein, which was broadcast on 31 May 2017. The audience for this programme was recruited by ComRes for the BBC with a view to ensuring that it was ‘representative’.

Both complainants had sought further information from ComRes about how they had recruited the audience. In doing so the Conservative Party in particular cited section 2 of the BPC’s rules on disclosure. In pursuance of BPC’s aim of ensuring transparency in the conduct of opinion polls, these rules specify details that BPC members should routinely make available in respect of published polls. The complainants argued that these rules applied to an audience recruitment exercise such as that conducted by ComRes, and that thus ComRes were duty bound to supply the information they sought.

It has never previously been suggested that audience recruitment exercises do fall within BPC’s remit and that thus its rules on disclosure should apply to them. However, in view of the submissions it has received, the officers of the BPC have carefully considered whether audience recruitment exercises do fall within the Council’s remit as specified in its current Objects and Rules. They have determined that they do not. This is because such exercises do not generate published quantitative data (as defined in those Objects and Rules) that summarise the collective views of a representative sample of voters (or other sub-group), and it is to such data that the Council’s Rules on Disclosure are intended to apply.

We are grateful to the two complaints for bringing this issue to the Council’s attention and we hope that this statement helps clarify matters for those considering any complaint to the Council in future about a member’s alleged failure to follow its rules on transparency.