Why the Polls Miss the Mark (Part 2)

For the next few weeks, we’re shifting the focus of our blog from examining legislation to discussing issues related to public opinion polling.

With the 2018 mid-terms approaching it’s important to know about the pitfalls polls face and why their findings shouldn’t be taken at face value. Our founder has written an extensively researched book on this subject, due to be released next month, which explains both the issues in detail and what it would take to truly address them. This series consists of sections from that book.


In theory, weighting allows researchers to compensate for issues which present themselves during the sampling process.

As an example, let us consider a survey of ten respondents that asks for their favorite color. Eight respondents answer with red, and the other two choose green. In this case, the findings would indicate that red is the favorite color of 80% of the population.

However, if examining the data shows that eight respondents in the sample were female and the other two male, while the population is an even 50/50 split, research analysts would likely assign weights of .625 (50% / 80%) to the female responses and 2.5 (50% / 20%) to male responses in order to bring their representation in the sample into alignment with their prevalence in the study population.

Favorite Color

Sex

Weight

Red

Female

.625

Red

Female

.625

Green

Male

2.5

Red

Female

.625

Red

Female

.625

Red

Female

.625

Red

Female

.625

Green

Male

2.5

Red

Female

.625

Red

Female

.625

As a result, if the females in the sample all preferred red while the males all preferred green, the weighted study findings would show that half the population likes green while the other half likes red.

There are several problems with this approach. Among other things, it increases the standard error of the response findings.

Weighting, like other forms of data adjustments or changes, can help to hide the issues which work their way into the inputs and minimize their impact on the outputs. However, as with most ways in which researchers compensate for problems, they simply reduce the visibility of problems while doing little to solve them.

This practice gives the illusion that problems have been addressed or even eliminated, although this is far from the truth. In efforts with smaller samples, weighting may even serve to exacerbate errors and other forms of bias by assigning them greater value.

By its nature, weighting entails a lot of assumptions which do not necessarily hold up to scrutiny. It embeds the presumption that researchers somehow know the actual proportion of various groups within the study population such that deviations from those proportions within the sample can be detected.

It also presupposes that each group behaves as a block with identical characteristics, rather than as individuals. This could be described as a form of scientific stereotyping, since it suggests that individuals have behaviors identical to the groups to which researchers have assigned them.

In our example above, males comprised 20% of the sample group and 50% of the overall population, meaning that males were substantially underrepresented. Therefore, their responses were given a weighting factor of 2.5 to compensate. This is not uncommon, in magnitude or in frequency of occurrence; weighting factors as high as 8 were used for many years, though they have generally been limited to 4 since the mid-1980s.

These are not minor adjustments that are being made!

In such cases, major revisions to the sample data are undertaken to give the findings the appearance of proportional representation, but researchers are increasing the levels of error in the data by doing so. It is a well-intentioned technique designed to correct for some inevitable errors, but it can cause more problems than it prevents.


This post is an excerpt from our founder’s book Data in Decline: Why Polling and Social Research Miss the Mark, to be released October 2018, partially reformatted for this content medium

Part 1 of this series can be found here