Why the Polls Miss the Mark (Part 3)

For the next few weeks, we’re shifting the focus of our blog from examining legislation to discussing issues related to public opinion polling.

With the 2018 mid-terms approaching it’s important to know about the pitfalls polls face and why their findings shouldn’t be taken at face value. Our founder has written an extensively researched book on this subject, due to be released next month, which explains both the issues in detail and what it would take to truly address them. This series consists of sections from that book.

According to polls conducted in the first few months of 2017, only about a third of Americans trust the media, which declined further in the year afterwards according to a report by the American Press Institute in June 2018, with substantial variation by political party identification.

A March 2017 McClatchy Marist poll indicated that of all political identifications, only “strong Democrats” placed at least “a good amount” of trust in polls.

Due to the issues affecting poll reliability, of course, we cannot be certain how accurate these numbers are. When segments of the population lose trust in polling, this can lead to a substantial bias in favor of those who continue to respond, which partially explains the decline in poll quality.

If strongly partisan Democrats are far more likely to respond to an opinion poll than strongly partisan Republicans—which is arguably the case since these same polls indicate 52% of strong Democrats trust polls compared to 27% of strong Republicans—the results of those polls are likely to contain bias. The effect is comparable to Literary Digest’s oversampling of Republicans in 1936 by drawing respondents from populations made up of voters who tended to be more Republican than the overall electorate.

That this disparate impact comes at the same time as the rise in narrowcast media, which allows individuals to curate and filter which information makes its way into their consciousness, makes obtaining participatory buy-in from study population members much more difficult than it has been in the past. People are becoming far more accustomed to actively filtering what information they take in. Everything from ad blockers to phone call filters have allowed confirmation bias, “the seeking or interpreting of evidence in ways that are partial to existing beliefs,” to flourish in our daily lives.

According to polls conducted in September 2018, a 68% majority of Americans get their news from social media, and 43% specifically did so from Facebook. These polls also report that for those Americans who get their news from Facebook, over 60% of them indicate the fact that news was shared from within their network is a major factor in how trustworthy they consider the information to be.

These findings are supported by research from the American Press Institute which indicates that people believe that their personal sources of information are fairer and more accurate than other information sources. If a majority of Americans favor information from within their network, this poses problems for researchers and pollsters, who are unlikely to be “in-network” for the entire populations they need to study.

A substantial level of individual confirmation bias necessarily increases the magnitude of self-selection bias present in research findings. Potential respondents who believe their responses will be misinterpreted, or that they will not be adequately represented, or that the results of the research will be used to further a partisan agenda to which they are opposed will be far less predisposed to participate even when they are appropriately included in the sample.

This post is an excerpt from our founder’s book Data in Decline: Why Polling and Social Research Miss the Mark, to be released October 2018, partially reformatted for this content medium

Part 2 of this series can be found here