• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • AMERICAN DEMOCRACY
  • WORLD DEMOCRACY
  • POLITICAL ART
  • more
    • election technology
    • money politics
    • political dissidents
    • THIRD PARTY
      • third party central
      • green party
      • justice party
      • libertarian party
    • voting methods
  • DC INFO
    • author central
    • about
    • advertise with DC
    • contact
    • privacy policy

Democracy Chronicles

I Question Whether This is Truly a Representative Sample

by Sean T. Stevens - January 20, 2012

FacebookTweetLinkedInPin

Scrabble to Decide Alaska Elections.jpg

This post is going to focus on something that has been bugging me ever since I read this column by Roger Simon, the chief political columnist for Politico.  Simon essentially demonstrates that he knows very little about survey methodology and analysis.  This lack of knowledge is not a big deal if you are the average person but, in my opinion, it is a big deal if you are the chief political columnist for a website like Politico (which currently gets more hits in a minute than Democracy Chronicles gets in a month).  My annoyance is not only with Simon’s column, it was simply the tipping point for my frustration and provides to introduce the larger issue – that, unfortunately, this lack of knowledge appears to be fairly widespread among members of the media who tend to overgeneralize polling data and rarely provide much-needed critical analysis.

Consider the following example.  This past weekend Reuters/Ipsos released a poll showing a 21-point advantage for Mitt Romney in South Carolina.  Many in the media reported this poll (e.g., here, here, here, here, and here; there are plenty more) and a good number failed to report some basic facts about the methodology that, as Nate Silver addressed, are quite important:

“The Reuters/Ipsos poll was conducted online from January 10-13 with a sample of 995 South Carolina registered voters. It included 398 Republicans and 380 Democrats.”

Check out Nate’s article for all the details, but in a nutshell: the poll was conducted online and not in one of the more traditional formats (phone or face-to-face interview); it sampled registered voters and not likely primary voters; and the number of Republicans sampled is much smaller compared to most of the other polls conducted in South Carolina.  I want to focus on two of these three issues: 1) the sampling of registered voters versus likely voters and; 2) the broader category of sample size and the idea of a representative sample, to make a few broader points about how the media generally fails to analyze polling data effectively. This failure ultimately leads to a column like the one by Roger Simon.

The distinction between a registered voter and a likely voter is often blurred by the media and it is also particularly important for a low-turnout election such as a primary.  Simply put not all registered voters actually vote.  Polling data based on likely voters is therefore far more accurate at predicting election results, and is even more valuable in low-turnout elections.  Yet, while the media often reports how people were surveyed, whether they are registered or likely, and the margin of error, they often do so in a fashion that treats all polls equal.  It is time to acknowledge that not all polls are equal.

Identifying inferior polls and explaining why they are inferior is the job of those analyzing and reporting on the polling data.  Why discuss and possibly even focus on inferior polling data?  Is it possible for a report on a poll to include some kind of basic analysis at the end of it that recommends how much one should weigh the data?  My reason for suggesting the media is largely uninformed on this topic is because such analysis is often absent from reports and discussions of polling data.  A basic level statistics course covers all the topics necessary to perform such an analysis, and I personally do not think such knowledge is too much to expect of someone who is analyzing and reporting on polling data.

This brings us full circle to Richard Simon.  In his critique of polling data he proudly proclaims:

“I have never been called by a political pollster and don’t know anybody who has, but I know some pollsters, who assure me they don’t make the numbers up, and I believe them.”

He goes on to critique a finding in a Washington Post-ABC News poll that Gingrich and Romney were equally favored by Republicans and Republican-leaning independents:

“We are a nation of nearly 313 million people. So how many people did the pollsters actually speak to? If you  have extremely good eyes, you can find the answer in tiny type at the bottom of a chart: The Post-ABC poll was conducted by phone “among a random sample of 1,005 adults.  That represents 0.0003 percent of the nation at large. (The number of Republicans and Republican-leaning independents was an even smaller sample of 395 people).”

As he is reaching his conclusion he returns to critiquing the Washington Post-ABC News poll, this time doubting the polling data that suggests most Americans are optimistic about their personal finances:

 “Is this how “most Americans” — based on a survey of 1,005 of them — really feel?”

When one considers all of these comments it is strongly suggests that the person making them has little to no knowledge of the methodologies pollsters employ to obtain a representative sample, a concept relevant not only to public opinion data but scientific research in general.  I am sure Roger Simon is thankful for antibiotics when he gets the flu and I am also sure he has not questioned the sample sizes used in the drug trials for the medication he is prescribed.  What is downright humorous however is that Simon makes a good point – that the poll only sampled 395 Republicans and Republican- leaning independents – but fails to explain the reason his point is important.  There is no mention of what one might consider a good representative sample for such a poll and/or how to obtain such a sample would be helpful.  I would assume it is difficult to offer such analysis if one is unaware of the methodologies pollsters use to obtain their samples

Ultimately, Simon’s article is somewhat of a shame because he is able to identify what is wrong with the poll he chooses to criticize but unable to correctly explain why the factors he has identified are flawed.  He instead tried to make a broader argument about how he just doubts the polling industry at large – which again betrays a lack of knowledge of sampling and provides yet another example of the media overgeneralizing one event and trying to apply it across the board.

Our analysis of polling data can be more sophisticated and nuanced.  The ideas behind random sampling and how to obtain a representative sample are not exceedingly complex – they just are not something many people think about on a day to day basis.  The next post at The Rabbit Hole will offer an introduction to the topic in an attempt to broaden awareness on the topic. Understanding such key concepts as random sampling and representative samples will make it easier to engage in discussions about the topics this blog intends to discuss.  Expect that post in a few days…

 

FacebookTweetLinkedInPin

Filed Under: DC Authors Tagged With: Polling

About Sean T. Stevens

Welcome to The Rabbit Hole, a blog that analyzes politics in the United States from an interdisciplinary perspective. Sean T. Stevens research focuses on how social attitudes and beliefs are assessed, how the psychological and societal foundations of morality relate to (and possibly somewhat determine) an individual’s political ideology, and how an individual’s political ideology influences their information-processing and decision making.

Some highlighted Democracy Chronicles topics

Africa American Corruption American Local Elections American State Elections Asia Capitalism and Big Business Celebrity Politics China Democracy Charity Democracy Protests Democrats Dictatorships Education Election History Election Methods Election Security Election Transparency Europe Internet and Democracy Journalism and Free Speech Middle East Minority Voting Rights Money Politics New York City and State Elections Political Artwork Political Dissidents Political Lobbying Redistricting Republicans Russia Socialism and Labor Social Media and Democracy South America Spying and Privacy Supreme Court Third Party Voter Access Voter ID Voter Registration Voter Suppression Voter Turnout Voting Technology Women Voting Rights Worldwide Worldwide Corruption

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Home | DC AUTHORS | I Question Whether This is Truly a Representative Sample

Primary Sidebar

Advertise button

The Christian Evangelical Church

By Jack Jones January 22, 2023

I am not against Big Business having a voice, just not all the voices, and especially not an impersonator pretending to speak for God.

No One In Monterey County Is Good Enough To Serve In Legislature

By Joe Mathews January 9, 2023

California’s “democratic reforms” have left a place as… [key] as Monterey County without any state representation from one of its own.

democracy chronicles newsletter

DC AUTHORS

In America, the Joe-mocracy Rules

By Joe Mathews December 27, 2022

A republic? A democracy? No, our country is an avuncular autocracy run by old guys named joe.

Property Rights, Indiana-Style

By Andrew Straw December 24, 2022

Indiana’s justices have replaced constitutional property rights with ad hominem politics. Replacing them starts with better governors.

To The American Oligarchs: Lay Off Us

By Jack Jones December 21, 2022

No matter how many jobs are ruthlessly pulled out from under us, we are still going to do what it takes to put food on the table.

Indiana’s Irrational Ballot Access System

By Andrew Straw December 4, 2022

Opposing the potential 2024 retention of the Chief Justice Loretta Rush will be key to preventing disability discrimination bad blood in Indiana courts.

Federal Judge Strikes Down Biden’s Loan Forgiveness program

By Jack Jones November 24, 2022

What is unlawful is the Texas federal judge’s decision to strike down President Biden’s loan forgiveness program, not the program itself.

DeSantis Battles Trump in Florida Steel Cage Match

By Steve Schneider November 20, 2022

We take you now to the much-anticipated DeSantis-Trump steel cage match, in which “DeSanctimonious” has promised to “kick Trump’s ass.”

Why Gambling Can’t Really Lose In California

By Joe Mathews November 2, 2022

When state voters approved an expansion of casino gaming, gaming interests assured us that gambling here would be governed by strict limits.

MORE FROM OUR AUTHORS

VISIT OUR POLITICAL ART SECTION:

dc political art

DEMOCRACY CULTURE

Girl Scouts Earn Democracy Badge At Workshop In Salina

Girl Scouts Earn Democracy Badge At Workshop In Salina

January 26, 2023

The workshop assisted Girl Scouts in grades K–10 in earning their Democracy Badge through learning activities on democracy.

Study: Media Can Reduce Polarization By Telling Personal Stories

Study: Media Can Reduce Polarization By Telling Personal Stories

January 26, 2023

Sharing personal experiences and pairing them with facts reduces political dehumanization and increases political tolerance.

Study: 2020 Election Resulted In Increased Anxiety And Depression

Study: 2020 Election Resulted In Increased Anxiety And Depression

January 7, 2023

A review of 2020 Household Pulse Survey data reveals that as an election nears, people in [America] report more depression and anxiety.

Key Iran Labor Sectors Launch Major Strikes

Charlie Hebdo Caricatures Iran’s Mullahs

January 7, 2023

The satirical weekly is publishing a special issue on Wednesday, January 4, mocking Supreme Leader Ali Khamenei in support of the protests…

First Impressions Are Strongly Influenced By Political Partisanship

First Impressions Are Strongly Influenced By Political Partisanship

December 22, 2022

How we perceive strangers or the impressions we have about them, particularly their faces, is influenced by political partisanship.

MORE CULTURE

VISIT OUR US DEMOCRACY SECTION:

American Democracy