All posts by admin

House of Lords Committee on Political Polling and Digital Media

On 28th October, British Polling Council President Professor Sir John Curtice gave oral evidence to the House of Lords Select Committee on Political Polling and Digital Media.  You can watch proceedings of the Liaison Committee here on at the “Parliament Live” TV channel.

More details on the work on the Committee can be found at The politics of polling – Summary.

The British Polling Council’s response to the Committee’s 2018 report can be found at British Polling Council Welcomes Lords Committee Report on Polling.

A Guide for Journalists to the Reporting of Opinion Polls?

The British Polling Council is today publishing A Quick Guide for Journalists to the Use and Reporting of Opinion Polls.

This publication has been designed to provide practical guidance for anyone unfamiliar with polls who finds themselves charged with interpreting and writing up an opinion poll in today’s media environment.

It forms part of the British Polling Council’s response to the recommendations of the House of Lords Select Committee on Political Polling and Digital Media.

The document has been developed as a “guided tour” of the key points to bear in mind when working with opinion poll data. Its contents include:

  • An overview on how polls are conducted, including what to look out for when judging whether the sample is representative.
  • Advice on how to evaluate the questions covered in an opinion poll. For example: are they written in everyday language? Do the questions lead the respondent?
  • Guidance on how to interpret the results of polls, including key dos and don’ts when it comes to looking at sub-samples (such as differences by age) or describing changes over time.

The resource has been developed to complement the existing materials available to practitioners and users of opinion polls, including this detailed guide developed by the press regulator IMPRESS and the Market Research Society.

The Quick Guide for Journalists is available on the BPC website (Opinion Polls: Guidance for Journalists), alongside a video briefing with the President of the British Polling Council, Professor Sir John Curtice.

Commenting on the launch of the Guide, Professor Curtice says: “In its report, the House of Lords committee expressed a number of concerns about how the media report polls. Our guide is intended to help address some of these concerns by providing a quick five-minute accessible introduction to polls. It describes both the strengths and the limits of polls, outlines five key questions that should be asked of any poll, and identifies the major potential pitfalls to avoid in writing a poll story. We hope that it will help journalists in newsdesks up and down the country report polls in a way that their audience finds both interesting and informative.”

Minutes of the Annual General Meeting 2020

The Meeting was held at ORB on 16 March 2020

Present in person

  • Nick Moon, Secretary/Treasurer
  • Simon Atkinson, Management Committee
  • Johnny Heald, ORB
  • Anthony Wells, YouGov
  • Bill White, LucidTalk
  • Owen Thomas Populus
  • Chris Hopkins, Savanta ComRes
  • Harry Carr, Demos

Dialling in

  • Professor Sir John Curtice, President 
  • Luke Taylor, Kantar Public 
  • James Crouch, Opinium 
  • Michael Thrasher, Sky Data 
  • Abraham Mueller, qriously 
  • Martin Boon, Deltapoll 
  • Damian Lyons Lowe, Survation

1. Apologies for absence

Robert Struthers, BMG; James Kanagasooriam, Hanbury Strategy; Seb Wilde, Public First; Gideon Skinner, Ipsos MORI; Gregor Jackson, ICM.

2. Minutes of the last AGM

The minutes were approved.
There were no matters arising that were not already on the agenda.

3. Officers’ reports

Financial report

During 2019 we received £2,950 in subscription fees, of which £800 related to late paid 2018 invoices. There is still £1,400 in subscription fees due for 2019: Savanta ComRes, Harris Interactive, Ipsos MORI, YouGov (all £200), Panelbase (£150), Opinium, Survation, Lindsell Marketing, BMG (all £100).

Expenses during the year were £14.70. However we have not received an invoice for website hosting and running, and we still owe an unknown amount for President’s expenses .

The bank balance at the end of 2019 was £14,881.04.

With the current membership annual subscription income is £3,500.

Disclosure issues dealt with by the officers

Other than the usual small number of complaints about matters not within the BPC’s remit, there were no complaints about disclosure that required action from the Officers.

Membership applications dealt with by the officers

Two new members were approved during the year: Demos and Omnisis. For completeness two more have been approved in 2020 to date: Portland Communications and Redfield and Wilton Strategies. We currently have 26 members.

The issue was raised of whether members should be using membership of the BPC as a badge of quality, and it was agreed that we would send out a note making it clear that this was not something that should be claimed or implied.

4. Appointment of the officers

  • President
    NM nominated JC, SA seconded, elected nem con
  • Secretary
    DLL nominated NM, JH seconded, elected nem con
  • Management Committee member
    NM nominated SA, AW seconded, elected nem con

5. President’s report 

Report had been distributed to members. It was noted that the management of the BPC should be more proactive, and that the Officers will hold a virtual meeting every 2-3 months to discuss whether there are any issues the BPC needs to address.
There were no comments other than on matters covered elsewhere in the agenda.
It was agreed that a summary of the Political Communications event would appear in due course on the BPC website.

6. Proposed changes to the Rules

NM proposed, and SA seconded, the following changes to the BPC rules:

Delete Rule 2.3

Add a new bullet point to 2.5 as follows:

In the case of a poll of voting intentions for an election (including any election that has to be called), the following BPC statement on uncertainty.

All polls are subject to a wide range of potential sources of error. On the basis of the historical record of the polls at recent general elections, there is a 9 in 10 chance that the true value of a party’s support lies within 4 points of the estimates provided by this poll, and a 2 in 3 chance that they lie within 2 points.

Approved but agreed that we would check that the calculations still apply after the 2019 general election.

Add a new bullet point to 2.5 as follows:

In the case of polling data that has been used by a polling organisation to model vote intentions for an election (including any election that has yet to be called) in individual constituencies, a statement of which party is estimated to be first in each seat and an indication of the estimated probability (within a ten-point band) of that party winning.

After discussion this was amended to:

In the case of polling data that has been used by a polling organisation to model vote intentions for an election (including any election that has yet to be called) in individual constituencies, an indication (for parties with a realistic chance of winning) of either (i) each party’s estimated probability of winning each seat, or (ii) the estimated vote shares for each party in each constituency.

At Rule 2.5 bullet 2 insert ‘or modelled’ after ‘weighted’

At Rule 2.5 bullet 3 amend ‘weighted’ to ‘weighted/modelled’

Reorder the bullets at 2.5 so that those on ‘e-mail address’ and ‘BPC website’ appear as the penultimate and final items respectively.

All approved.

7. House of Lords Committee

The President’s Report covered this in some detail and JC explained the various points in his report dealing with the actions suggested by the Committee and his proposals for dealing with them.
JC noted that George Foulkes has tabled his Bill on the regulation of opinion polls again, exactly as he did two years ago, although it is again not expected to progress far.
It was noted that we are still awaiting replies from the two bodies the President contacted about possible collaboration on training of journalists: the Broadcast Journalism Training Council, and the National Council for the Training of Journalists.
It was again agreed that the BPC would not and could not get involved in assessing the quality of polls, and would not get into the business of reviewing media coverage of polls.
JC will respond formally to House of Lords and will copy to members.

8. Financial Conduct Authority update

The proposed new guidance for members was discussed and approved.

9. Journalists’ Guide to Opinion Polls

The proposed new Guide for Journalists was approved as a concept, and members were asked to raise points of detail directly with JC.
DLL suggested that the guide should refer specifically to what the BPC rules require.
NM suggested it should also be made clear that BPC membership is about transparency not data quality.

10. Website

SA introduced his brief paper on recent changes to modernise and improve the BPC website. Members were asked to check their own details on the members page on the site.
It was agreed that SA would investigate in more detail the costs of improving the website still further.
There was also a suggestion from a member of the public that during elections the BPC website could form a reliable source for information on all polls conducted by members. This would involve a members-only section of the website, password protected, where members could add the voting intention figures from their most recent poll into a simple template.
HC pointed out that Britain Elects already provides a similar service.
The feeling among members was that this would still not solve the problem of fake polls being posted on Wikipedia.
It was agreed that the BPC should not proceed with this suggestion, but that the Members page should have a hot link to a page on their website that either contained their latest poll, or made it easy to find their latest poll.

11. Any Other Business

JC thanked everyone for attending and also proposed a vote of thanks to ORB for hosting both the physical and virtual elements.

The Performance of the Polls in the 2019 General Election

After being widely criticised for their performance in the 2015 and 2017 general elections, the polls have proven more accurate in the 2019 election than in any contest since 2005.

As the table shows, on average the final polls underestimated the Conservative vote by just 1.4 points and overestimated Labour’s by only 0.5 points.

These patterns are consistent with the long-term trend for the polls to over-state Labour and under-state Tory support, but the size of the errors were so small that this did not prevent the polls correctly predicting the overall outcome of the election.

For all the other parties, the average error across the pollsters was under one point. Indeed for the LibDems, Greens, and Others the average error was a remarkably small 0.1 of a point.

 CONLABLDBrexitGREENOTHERMETHODSAMPLE SIZEFIELDWORK
Qriously433012348Online2222Dec 5-8
ICM423612324Online2011Dec 6-9
YouGov433412335Online105,612Dec 4-10
Savanta ComRes413612326Online1732Dec 9-10
BMG413214345Online1660Dec 6-11
Ipsos MORI443312236Telephone2213Dec 9-11
Kantar 443213335Online2815Dec 9-11
Deltapoll453510334Online1818Dec 9-11
Survation45349336Telephone2395Dec 10-11
Panelbase433411435Online3174Dec 10-11
Opinium453312226Online3005Dec 10-11
Average43.333.511.72.92.95.5   
Result44.733.011.82.12.85.6   
Difference-1.40.5-0.10.80.1-0.1   

There was also a Survation poll in Scotland conducted online from Dec 10-11 with 1,004 respondents.

  Poll Result
SNP 43 45.0
Con 28 25.1
Lab 20 18.6
Lib Dem 7 9.5
Other 2 1.7

Principal Changes in the Conduct and Reporting of Polls in the 2019 General Election

This document briefly summarises the main differences between the way in which member companies are conducting their polls in the latter stages of the 2019 election and the way they did so in 2017. Details of the changes that companies made in 2017 are available at How Have The Polls Changed Since 2015?. For further details and analysis of the methodology of the polls see BPC Inquiry Report.

John Curtice

BMG

The company only published one Britain-wide poll in 2017; otherwise it conducted private polls for the Labour Party.

That one published poll was a mixed-mode online and telephone poll. The company’s polls in this campaign are likely to be conducted exclusively using an online methodology.

As in 2017 data are weighted in accordance with the known levels of electoral registration among key demographic groups. However, in contrast to 2017 the data are no longer weighted by reported probability of voting or whether or not the respondent voted at the last election.

Respondents are being invited to state a vote intention from only among the parties standing in their constituency.

Deltapoll

This is the first general election to be covered by the company. Details of its methodology can be found at Deltapoll Methodology.

ICM

Data are no longer weighted according to a pre-determined level of turnout by demographic group

A 0-10 turnout scale is now used to weight the data rather than a 1-10 scale.

The voting preference of those who say they do not know how they would vote or who refuse to say how they would do so are no longer imputed on the basis of their past vote.

Data are weighted by 2016 EU referendum vote as well as 2017 general election vote.

Respondents are being invited to state a vote intention from only among the parties standing in their constituency.

Ipsos MORI

Data are no longer being weighted to match the ratios of turnout by age and tenure as recorded by the 2015 British Election Study.

Child in household is no longer used as a weight as it was in 2017, while ethnicity has been added as a weight.

Respondents who say they will vote for the Brexit Party, Liberal Democrats, Greens or Plaid Cymru are invited to state for which party they would vote if their preferred party is not standing in their constituency – and are allocated to that party if their preferred party is not standing locally.

Kantar

Weighting of the data by probability of voting is based on the respondent’s stated likelihood of voting, their age, and whether they voted in 2017, modelled using data collected by the company after the 2017 election (rather than 2015).

Weighting of the data by past vote takes into account of estimates of differential recall error by party and, in particular, a higher tendency for those who vote Labour in 2017 to fail to report that they have done so.

In similar vein to what was done in respect of those saying they were voting UKIP or Green in 2017, respondents are invited to state for which party they would vote if their preferred party is not standing in their constituency – and are allocated to that party if their preferred party is not standing locally.

Opinium

Respondents are being invited to state a vote intention from only among the parties standing in their constituency.

ORB

The company has so far only conducted one poll of vote intentions, at the very beginning of the campaign.

Panelbase

Respondents are invited to state for which party they would vote if their preferred party is not standing in their constituency – and are allocated to that party if their preferred party is not standing locally.

Savanta ComRes

Data are no longer being weighted by the voter turnout model that was used in 2017

Future polls will invite respondents to choose from among the parties standing in their constituency.

Survation

Polls are now only being conducted by phone, using the same approach as in 2017.

Data are weighted, inter alia, by 2019 European Parliament election vote.

As in 2017, respondents are being invited to state a vote intention from only among the named candidates standing in their constituency.

YouGov

There has been a slight reduction in the downweight that is applied to those who did not vote at the last election.

Respondents are being invited to state a vote intention from only among the candidates standing in their constituency.

Interpreting Polls and Election Data

IMPRESS, the UK’s independent press regulator, and the Market Research Society have published a consultation on guidance for journalists reporting on opinion polls.

Interpreting polls and election data – guidance for media and journalists

Comments and feedback on this draft guidance are invited until 20 November.

The MRS web page also provides access to a range of other resources to support journalists and the media in their reporting of opinion poll data.

Minutes of the Annual General Meeting 2019

The Meeting was held at ORB on 5 March 2019

Present

  • Sir John Curtice, President
  • Nick Moon, Secretary/Treasurer
  • Simon Atkinson, Management Committee
  • Erica Harrison, ORB
  • Ivor Knox, Panelbase
  • Anthony Wells, YouGov
  • Gideon Skinner, Ipsos MORI
  • Adam Drummond, Opinium
  • Luke Taylor, Kantar Public
  • Damian Lyons Lowe, Survation
  • Gregor Jackson, ICM
  • Owen Thomas, Populus
  • Seb Wilde, Public First

1. Apologies for absence

There were no apologies for absence.

2. Minutes of the last AGM

The minutes were approved.

No matters arising not already on the agenda.

3. Officers’ reports

Financial report 

During 2018 we received £4,650 in subscription fees, covering both 2017 and 2018 subs as 2017 invoices were sent out with 2018 ones. There is still £500 in subscription fees due for 2018: both years from Panelbase and one year from Ipsos MORI.

Expenses during the year were £1,180.09, the two main components being £748 to the RSS for a meeting room and £353.99 for the website hosting and running.

The bank balance at the end of 2018 was £12,000.74.

With the current membership annual subscription income is £2,950.

Disclosure issues dealt with by the officers

Other than the usual small number of complaints about matters not within the BPC’s remit, there were no complaints about disclosure that required action from the Officers.

Membership applications dealt with by the officers

A record four new members were approved during the year: qriously, Hanbury Strategy, Deltapoll and Public First.

4. President’s report

Report had been distributed to members. There were no comments other than on matters covered elsewhere in the agenda.

5. Appointment of the officers

  • President
    NM nominated JC, SA seconded, elected nem con.
  • Secretary
    JC nominated NM, GJ seconded, elected nem con.
  • Management Committee member
    JC nominated SA, JT seconded, elected nem con.

6. House of Lords Committee

The President’s Report covered this in some detail and JC explained the various points in his report dealing with the actions suggested by the Committee and his proposals for dealing with them.

  1. Meeting agreed that we would have a section on the website on how to make a complaint, and also making clear what we can and can’t adjudicate on.
  2. Meeting agreed that we would write a spec for the things to be covered in a revised Journalists’ Guide. Once this is agreed by the membership we will either commission a professional writer to write it, or write it using internal resources. JT raised the point that it might have more credibility if it were written by a journalist. GS suggested Full Facts. The Statistics Ambassador for the RSS has volunteered to help, and he will be sent the outline for comment at an early stage. It was agreed that in future the BPC will have a link to each member’s methodology statement on their own website.
  3. The recommendation for a coordinated programme of training for journalists on polling could involve the BPC in considerable expense. One approach might be to approach University Schools of Journalism to offer lectures on polling. JT is already doing this in many places and is happy to do more. This only impacts on the journalists of tomorrow. GS suggested approaching The Lobby for training in opinion polls, though it was accepted that take-up may be low.
  4. The recommendation about polls declaring far more information about the amount and details of funding would raise very serious issues of client and member confidentiality and it was agreed that it would be inappropriate for the BPC to impose such a condition on members.
  5. We do already.
  6. It was accepted that members would not agree in any detail exactly what would constitute good or bad progress. JT suggested a very broad brush top-level guide to principles, and it was agreed that this might sit within the Journalists’ Guide.
  7. LT made the point that this would involve the BPC making pronouncements on things outside our remit, and might raise issues of why we do this but don’t police such things. It was agreed that this lay outside the remit of the BPC.
  8. See 7. Though the Journalists’ Guide, once revised, would perform a useful function here and we would offer to assist Ofcom etc if they wanted help in interpreting it further, while being mindful of the need to avoid going beyond our remit.
  9. See 7
  10. See 7

7. Treasury Select Committee

JC said Morgan Committee was closed as far as we were concerned at this stage. GJ raised the question of whether the fact that our response may have been seen as unsatisfactory by committee means there might be some come-back. JC felt this would depend on what came out of the discussions with the FCA.

8. Financial Conduct Authority

JC reported that whatever FCA decide it will be looking forward not looking back. They do not feel members have done anything wrong.

9. Proposed change to Rule 2.7

NM proposed that rule 2.7 should be changed from: 

However, in the event the results of a privately commissioned poll are made public by the organisation [its employees or agents] that commissioned the survey, such results will be deemed to have entered the public domain and procedures outlined above will be followed in respect of those findings.

to:

However, in the event the results of a privately commissioned poll are made public as a result of actions by the organisation [its employees or agents] that commissioned the survey, such results will be deemed to have entered the public domain and procedures outlined above will be followed in respect of those findings.

The amendment was agreed subject to organisations being able to object within 28 days of receiving the AGM minutes.

10. Any Other Business

JC urged members who have not done so to read the proposed revisions to the MRS Code of Conduct and if they feel there are any issues that might require a response from the BPC as opposed to individual company responses.

It was agreed that we probably can’t afford a complete redesign of our website, but that SA would obtain costs for updating it and making it more user-friendly.

JC raised the issue of making it possible to attend future AGMs by video link or at least phone.

JC also proposed a vote of thanks to ORB for hosting.

Polling Rules and UK Financial Markets

On 28th September, Rt Hon Nicky Morgan MP, Chair of the Treasury Committee, wrote to Professor Sir John Curtice, President of the British Polling Council. The letter focuses on the use of private polling data, particularly during the recent referendum campaigns, and follows on from media coverage during the summer. Further details, including a link to Mrs Morgan’s letter, can be found on the Treasury Committee’s web page: Change polling rules to protect integrity of UK financial markets.

Following a consultation with its membership, the British Polling Council has responded to Mrs Morgan. The contents are set out in this letter to the Rt Hon Nicky Morgan MP (pdf).

British Polling Council Introduces New Rule on Uncertainty Attached to Polls

The British Polling Council announces today that it has introduced a new requirement on its members when reporting estimates of vote intention. This requirement is an obligation to publish a statement of the level of uncertainty that has historically been associated with polls of voting intention.

The statement reads as follows:

‘All polls are subject to a wide range of potential sources of error. On the basis of the historical record of the polls at recent general elections, there is a 9 in 10 chance that the true value of a party’s support lies within 4 points of the estimates provided by this poll, and a 2 in 3 chance that they lie within 2 points.’

The statement has been developed following an analysis of the performance of the final polls at every general election since 2001. That analysis revealed that 90% of the estimates of a party’s share of the vote in the final polls were within four points of the eventual true result, while two-thirds were within two points.

Hitherto, it has often been noted that, according to statistical theory, there is a 95% chance that in a poll of a thousand people the true value of a party’s share of the vote lies within three points of the poll’s estimate. However, this statement refers solely to the error that may be occasioned by the chance variation to which all sample surveys are subject. It does not take into account any of the other many possible sources of uncertainty in a poll. The new BPC statement attempts to reflect all of the sources of uncertainty that practical experience suggests is associated with polling in Britain.

The statement is intended to discourage the media and others from reporting small changes in a party’s support – or variation between pollsters – as firm evidence of a significant change in its popularity.

The new BPC rule has been introduced in response to recommendation 11 of the Inquiry into the 2015 British general election polls that was chaired by Prof. Patrick Sturgis. This recommendation read that:

‘BPC members should provide confidence (or credible) intervals for each separately listed party in their headline share of the vote’.

The introduction of this requirement means the BPC has now responded to all of the recommendations that were made to it by Prof. Sturgis and his colleagues.

Notes to Editors

  1. The British Polling Council (BPC) is an association of polling organisations that publish polls. The objectives of the Council are to ensure standards of disclosure that provide consumers of survey results that enter the public domain with an adequate basis for judging the reliability and validity of the results. Through full disclosure the Council aims to encourage the highest professional standards in public opinion polling and to advance the understanding, among politicians, the media and general public, of how polls are conducted and how to interpret poll results.
  2. For further details of the Council see http://www.britishpollingcouncil.org/
  3. The report of the Inquiry into the Performance of the Polls in the 2015 Election is available at the National Centre for Research Methods (NCRM) pdf