OCTOBER 10, 2019 – Here at Clout Research, we always advocate for the quality of data that goes into survey research. If your data sample is collected well and is processed and weighted according to best practices, your survey will more properly represent the larger population you are trying to measure.
These are the keys to accuracy – a proper data sample, and a good understanding of the larger population the survey seeks to measure. Not every poll measures up.
The screaming headlines today from the latest Fox News poll declare that 51% of Americans believe President Trump should be impeached and removed from office, while 44% think he should not be impeached. This is a surprising report both because it purports to show a majority of Americans would like to see the President removed from his office, and because the poll was commissioned by Fox News, a largely conservative news outlet that is often supportive of the President.
Trump, unsurprisingly, has attacked the news, saying the poll “sucks”.
In this case, Trump is correct.
We have no reason to believe that the two polling companies that team up to conduct surveys for Fox News – the Democratic firm of Beacon Research and Republican pollster Shaw & Company Research – mishandled the data. And there is no reason to think that the company that conducted the actual fieldwork, Braun Research Inc., did anything wrong with its fieldwork.
But the survey results are still flawed in several ways. Here is why:
1. The survey prefaces the impeachment question with leading negative questions. Before it asks about impeachment, the poll asks whether respondents think the “Trump Administration is more corrupt than previous administrations, less corrupt, or is the Trump Administration about as corrupt as previous administrations?”
What? The question just assumes all respondents believe the Trump Administration is corrupt – that it is just a matter of degree. This is a very poorly worded question that hints (screams?) of pollster bias.
Then, just before the impeachment question, the same sort of bias rears its head with this question: “How troubling do you find the situation surrounding President Trump’s dealings with the Ukranian President . . . ?” Now, on this question, at least the survey allows an answer selection of “Not at all” troubling, but the damage is done in the question in its assumption that there is something “troubling” about this call. There is no good excuse for this. How badly does this taint the impeachment question data that closely follows? It is impossible to tell, which is the problem – pollsters are supposed to work diligently to eliminate variables that influence the outcome of their surveys, and the structure of these questions instead injects more uncertainty into the survey findings.
2. The sample is slanted significantly toward Democrats. This survey includes a mix of 48% Democrats, 40% Republicans, and just 12% independents. This might not be a bad distribution for an election sample if the overall sample was of likely voters, but that is not what this survey is. This is a registered voter survey. Registered voters are much, much less engaged in following politics, and are much less likely to have chosen a party affiliation. That independent, or unaffiliated, voters represent so small a part of this survey sample is a huge error. The monthly track of party affiliation done by Gallup show that in September, 2019, just 31% considered themselves Democrat, which is 17 points lower than what is included in this survey. Gallup shows that 29% consider themselves Republican, which is 11 points lower than the Fox News polls show would be accurate. Gallup also says that 38% nationwide consider themselves to be independent voters, 26% MORE than is included in this survey sample. Obviously, with such discrepancies, the Fox News impeachment poll is way off the mark
It is clear that these Fox pollsters did not weight their sample for partisan affiliation. Apparently they simply took whatever sample answered the phone, and that was that. It may be a fine practice if you are surveying how people feel about their toothpaste, but it is not the right way to handle a survey about politics. Everyone pretty much brushes their teeth the same way and holds views on the subject that are not influenced by anything other than how they feel about, well, toothpaste. Demographic weighting may not be needed on such a subject. But there are very different views on politics held by Americans depending on their party affiliation, and you absolutely have to account for that in your sample by weighting it to make sure it closely represents the general partisan makeup of likely voters across the nation.
Weighting political polls is also critical because it is well-known that Democrats are much more likely to participate in political polls than are Republicans and independent voters. If you want to be accurate, you have to counter that cultural phenomenon by weighting your sample.
Back of the envelope calculations, using the Gallup partisan figures to weight this Fox poll, show that when this Fox News poll sample is balanced for party affiliation, just 46% believe Trump should be impeached, while 46% said they oppose impeachment and removal. Of course, this is still higher than any President would want to see, but given that Democrats have been pushing loud and long for Trump’s impeachment since he was elected, this is not surprising.
Without that magic “51% majority” favoring impeachment, this poll does not generate the screaming headlines nationwide and quickly fades away.
This is the big difference between media pollsters and private political pollsters. Media pollsters like the Fox News polling teams are out to produce headlines, while private political pollsters are out to produce real-world numbers on which real political decisions can be made – with confidence.
The big winner in publishing surveys like this are the publishers themselves. They get screaming headlines and lots of attention and clicks. The losers are the American public and the polling industry at large, as the former continues to get fed a steady diet of misleading survey results that have little basis in reality, and the latter gets tagged as nothing more than another purveyor of Fake News.
Fritz Wenzel, Pollster and Founder