Identifying Biased Polls (Part One)

Polls are everywhere, from political campaigns to market research, and they often play a significant role in shaping public opinion, politics, economy, and even private lives. However, not all polls are created equal, and some can be manipulated to influence the results and ultimately the decisions based on them. This is why it’s important to learn how to recognize manipulation in polls, especially in the context of political elections, where the stakes are high and the impact is far-reaching. The manipulation of election polls has been a topic of great concern in recent years, with various tactics employed to sway public opinion and influence the outcome. However, the manipulation of polls can occur in many other areas, from advertising to public opinion surveys, and it’s important to be aware of the different methods used and how to spot them. In this article, we will explore the different types of manipulative methods used in polls, with a focus on how to identify and dismiss biased polls. 

There are many manipulative methods used in biased polls and surveys, and they can vary in their subtlety and effectiveness. Some of the most common methods include leading questions, limited response options, loaded language, order effects, and sampling bias. Leading questions are designed to guide the respondent to a particular answer, while limited response options restrict the possible answers that can be given. Loaded language uses emotionally charged words to sway the respondent’s opinion, while order effects occur when the order of the questions or responses influences the answer given. Sampling bias can occur when the sample of respondents is not representative of the population being studied. These manipulative methods can be used to influence public opinion and shape decision-making, making it important to be able to recognize them in order to make informed choices.

LEADING QUESTIONS  Designed to influence a respondent’s answer by guiding them to a particular response. Questions that suggest a certain answer or point of view can manipulate responses. It could be a sinister practice or not, depending on what the goal is.

Leading questions can be useful and ethically valid on the other hand in certain contexts, such as market research. In brand preference research, for example, a neutral question might be “When choosing a soft drink, which factors are most important to you?” allowing respondents to consider multiple factors and not just Pepsi or Coca Cola. But if we need to examine preferences between these two brands, we need exactly a leading question such as “When choosing a soft drink, do you prefer the refreshing taste of Coca-Cola or the sweetness of Pepsi?”. Similarly, in advertising research, a neutral question such as “What are your thoughts on this ad?” allows respondents to provide their own feedback without being prompted to consider a specific action which is already implied in a leading question “How likely are you to purchase this product after seeing this ad?” In product development research, a neutral question like “What features would you like to see in a new mobile app?” allows respondents to offer their own ideas and preferences, rather than being prompted to consider a specific feature, but if we need to test exactly this we might ask: “How excited would you be to use a new mobile app that helps you organize your to-do list?”

While leading questions can be useful in certain contexts, such as market research, in polls and surveys they can also be used to manipulate even election results and public opinion. Questions that are phrased in a biased or leading way can influence the responses of the people being surveyed. For example, a poll question about gun control that starts with the statement “Given that the Second Amendment guarantees the right to bear arms…” is likely to produce different results than a question that simply asks “What is your opinion on gun control?”

Here are more examples of leading questions, along with how the unbiased versions of these questions should be asked:

Leading question: ”Do you agree that we need to increase taxes to stimulate the economy?” assumes that increasing taxes is the best way to stimulate the economy and may lead respondents to agree with this position.

Unbiased version of the same question might be: “What do you think are the best ways to stimulate the economy?”. This question allows respondents to consider a range of factors beyond just increasing taxes, and provide their own opinions and ideas on how to stimulate the economy.

One real-life example of a leading question was used in a 2014 poll conducted by Fox News in the United States, where the question asked, “Do you believe the Obama administration knowingly lied about the attack in Benghazi to help the President during his re-election campaign?” This question assumes that the Obama administration did, in fact, lie about the attack in Benghazi, which is a highly contested issue. By framing the question this way, the poll was able to sway the opinion of respondents towards a particular viewpoint.

In Europe, there have been instances of leading questions being used in political polls as well. In a 2016 Brexit poll conducted by Ipsos MORI, the question “Should the United Kingdom remain a member of the European Union or leave the European Union?” was criticized for being a leading question, as it placed the focus on remaining a member rather than leaving. An unbiased version of the question could have been, “What is your opinion on the United Kingdom’s membership in the European Union?”

LIMITED RESPONSE OPTIONS By limiting the response options to a few choices, pollsters can steer the results in a certain direction. For example, “Which candidate do you support: A or B?” may leave out other candidates who could be viable options.

“Are you in favor of stricter immigration policies, or do you think we should continue with the current policies?” Is the biased version of: “What do you think about the current immigration policies in the country?” The biased poll presents only two response options, one of which assumes that stricter immigration policies are needed. The neutral version allows respondents to express their opinion without being led towards one particular response.

Biased poll: “Do you support the death penalty for murderers?”

Neutral poll: “What punishment do you think is appropriate for someone convicted of murder?” with response options such as “life imprisonment without parole,” “life imprisonment with the possibility of parole,” and “death penalty.”

Biased poll: “Do you think all undocumented immigrants should be deported?”

Neutral poll: “What do you think should happen to undocumented immigrants who are currently in the U.S.?” with response options such as “offer a pathway to citizenship,” “grant legal residency but not citizenship,” and “deportation for those with criminal records.”

A biased limited response option poll and a leading question on the topic of the war in Ukraine could be:

“Which of the following is the most effective way to support Ukraine in its fight against Russian aggression: 

• • • • • send military aid and troops, 
        • impose economic sanctions on Russia, 
        • provide diplomatic support and negotiations with Russia?“

This type of limited response option poll may bias respondents towards thinking that these are the only viable options, without giving them the opportunity to express alternative solutions or nuanced perspectives on the issue.

A neutral limited response option poll might be:

“What actions do you believe should be taken to resolve the conflict in Ukraine?”

• • • • • Diplomatic negotiations between Ukraine and Russia
        • Deployment of international peacekeeping forces
        • Implementation of a ceasefire agreement
        • Send military aid and troops
        • Impose economic sanctions on Russia
        • Other (please specify)

This poll provides respondents with a limited set of options while still allowing them to express their own thoughts and opinions on the conflict in Ukraine. It does not push any specific agenda or viewpoint and does not attempt to sway respondents towards a particular position.

Continues here: Recognizing Manipulation: A Guide to Identifying Biased Polls (Part Two)