Why the Polls Miss the Mark (Part 1)

For the next few weeks, we’re shifting the focus of our blog from examining legislation to discussing issues related to public opinion polling.

With the 2018 mid-terms approaching it’s important to know about the pitfalls polls face and why their findings shouldn’t be taken at face value. Our founder has written an extensively researched book on this subject, due to be released next month, which explains both the issues in detail and what it would take to truly address them. This series consists of sections from that book.

Best practices in the industry were built on assumptions that revolved around 20th century lifestyle habits, only some of which have been reexamined.

The way people now respond to attempts by unknown parties to initiate communication creates an obstacle for pollsters, one which traditional approaches to survey research are ill-equipped to address. One noteworthy technological shift has been the swift rise of mobile communication technologies, particularly the widespread adoption of cellular network operated devices in first-world nations.

As recently as 2003, nearly 95% of U.S. households had landlines. When nearly everyone had a landline telephone and regularly answered calls, landlines were a generally reliable way to sample every desired population. This is no longer the case; as of the end of 2016, fewer than half of Americans had landlines.

The rise of wireless-only households means that calling landlines creates a sampling bias. According to the Centers for Disease Control and Prevention, at the start of 2018, 72% of renters lived in wireless-only households while only 44.6% of homeowners fall into this category.

This uneven distribution of homes that still have landlines means homeowners are around twice as likely to be represented in polls conducted through landline calls. Yet, while we now live in an era when over 90% of Americans had a cell phone, 25% of calls made by researchers at Pew are still to landline telephones in 2018, as they did in 2016 when wireless adoption rates were similar. It would still be difficult to reach a representative ratio of homeowners and renters by only calling cellular devices, as 82.9% of renters have wireless devices compared with 63.8% of homeowners, but the sample would be less unbalanced along this axis.

If pollsters call wireless phones, their survey will sample three homeowners for every four renters, while a landline survey will sample eight homeowners for every four renters. Theoretically, the 25:75 landline-to-wireless sampling ratio has the effect of balancing out the sampling bias in each medium, but aggregating together two separate samples collected from two different populations is a scientifically questionable approach.

Given that there are now multiple types of voice calling including IP phones and services such as Skype or Google Voice, the continued reliance on telephone surveys necessitates dubious practices to achieve samples with are mathematically in line with what other research indicates they should be.

Even though some researchers have acknowledged the need to reach respondents digitally, doing so has posed problems which have confounded them. The disproportionate distribution of respondents leads to weighting—giving higher value to some responses over others—in an attempt to correct for any bias that might be created by this sampling error.

This post is an excerpt from our founder’s book Data in Decline: Why Polling and Social Research Miss the Mark, to be released October 2018, partially reformatted for this content medium