Never Trust a Clean Partisan Story
Never Trust a Clean Partisan Story
When we see something that confirms our biases, it's a good time to give the old "share" button a break
No Grace Allowed
The Lineage of a Lie
The Big Problem - Simpsons Paradox
How Shall We Then Live?
Looney Tunes: Sahara Hare
No Grace Allowed
I need to start taking some kind of emotion-dampening medication because I still get worked up over very stupid things.
The latest stupid thing I got worked up over is the story.
I’m so tired of COVID stories written by very dishonest people, but this one is astounding in not only how dishonest it is but how wretchedly unkind it is.
I have hoped from the beginning of this crisis that this would be the one event in which we could transcend partisan bias, work together, show grace and kindness toward each other. I wasn’t under the delusion that this could extend permanently or to every topic, but I was hopeful that we could develop some kind of social taboo against using COVID data or news in the service of partisan attacks. Those people will always exist, but maybe we can make them be quiet about it through the magic of social shaming.
My hope that we could get people on board with this is largely gone and I am left with the comfort that at least the people around me seem to continue holding to this ideal and we will continue to act this way regardless of how the larger culture around us shifts in ways that we find repulsive.
In conclusion:
Now let’s deal with this specific horrible data story.
The Lineage of a Lie
The RawStory tweet linked above goes to this story. That story, all by itself, is ridiculous because the charts they use only show COVID cases and deaths during the summer of last year.
That is easy enough to point out and everyone seems to get why that is absurd. But in conversations with the author of that story, she protested that this pattern remained in place for the rest of 2020 and pointed to this CNN story to bolster her claim. In turn, the CNN story points to this study from the American Journal of Preventative Medicine.
There are so many problems with that study, I’m at a loss of where to begin. But let’s start with the thesis, which is that Republican governors made worse choices than Democratic governors.
This thesis is, on its face, ridiculous and should never have been approved for study. It is lazy, partisan, and divisive. They authors claim that Democratic governors issued longer durations of stay-at-home orders and were more likely to issue mask mandates. If that’s true, then it is their responsibility to put together a study that looks at the impact of those mitigation strategies. Tell me how mask mandates or stay-at-home orders work. Show me the results of these mitigations and then we can talk about why governors should implement them. Researchers shouldn’t be using “red vs blue” as a proxy indicator for these mitigations; they should look at the actual mitigations. The only real reason to boil these policies down to “red vs blue” is either partisanship or laziness or both.
Next, let’s look at their methods. They chose to use the COVID Tracking Project for all their case and death data. This is a fine choice except for the part where it is well known among everyone who has been working with the data that the COVID Tracking Project reported New York state’s figures instead of the CDC’s. This was a perfectly reasonable decision when it was initially made, but it’s also well known that NY state undercounted deaths by over 8,000 due to the New York health department never really getting on board with reporting “probable” COVID deaths.
This needs to be noted and or at least mentioned in the study. The authors seem fully unaware that this was even a thing.
The Big Problem - Simpsons Paradox
The biggest problem with this study is the fact that they made what is an elementary statistics error and it went all the way to publication and no one caught it.
The authors took the per capita COVID case and death numbers among the “red states” and “blue states” and ran an analysis on them. In doing this, they gave North Dakota the same weight as Texas and Hawaii the same weight as New York despite the obvious population differences. Their chart is tiny and unreadable, so I’ve roughly duplicated their work here.
At first glance, this looks like the authors at least have their data correct. It looks like, after the initial wave, states with red governors had consistently higher patterns of cases and deaths from the summer all the way through the winter surge.
However, what we’re seeing here is due to the fact that the authors weighted the death rates for small and rural states with the same weight that they applied to high population states. This is a statistics error that is so common it has its own name: Simpson’s Paradox. It is when you take the average of the averages instead of calculating the overall average based on the properly weighted data.
When we weigh the red states and blue states by population, we get this.
Once properly weighted for population and taking into account the regional differences of the northeastern wave and the southern border wave, there is basically no difference in deaths related to this very lazy “red vs blue” rubric.
How Should We Then Live?
Who should we blame for this kind of atrocious data work? Blame starts with a terrible hypothesis that should never have been approved given the clear laziness and potential for partisan hackery. The authors should each be personally embarrassed for even proposing this. It moves then to advisors, peer reviewers, and the American Journal of Preventative Medicine. At every step of the way, someone should have said “This seems too easy, this seems too clean. This tells a story that I personally think is obviously true so I need to be suspicious of it, let’s get a critical eye on this thing.”
It’s clear no critical eye was applied.
Blame also goes to CNN for barfing out a story based on this ridiculous study. Their article was the antithesis of critical journalism, nothing but copying the results from one place to another without even bothering to think about it for even a moment.
From there, the blame goes to the Raw Story. I almost have to admire the intentional partisanship they show. At least they looked at the data close enough to understand that they can’t just chart the data as a whole because the red/blue difference wasn’t stark enough for a layman to see in the chart. The data as a whole didn’t make “team red” look bad enough, so they had to truncate it and only show the worst couple of months that they could find. It’s deeply dishonest, but at least it shows that they understood the data enough to intentionally manipulate their reporting, which is something CNN did not seem to realize.
That’s how I would apportion the blame. But that doesn’t get to the real question: how should *we* then live?
We cannot go through every bad study in this kind of detail. We can’t hunt down every bad piece of data, every biased researcher, we can’t rebut every dumb article. We have lives to live and they aren’t well spent running on this treadmill of nonsense1.
I feel better when I takes these kinds of stories and studies apart piece by piece to figure out what happened. No one in this process thought they were doing an evil thing, they just don’t have anyone on their “team” who recognized the trap they’ve fallen into and puts a loving paw on their hand to say “stop”.
Then I recognize that I can fall into this trap too. I see a story that confirms my bias and I don’t think about it, I just share it. And that diminishes all my thoughtful work, all my careful analysis, all the things I’m proud of and want to be known for.
The “solution” is not so much a solution as a habit. It is the practice of simply dismissing out of hand stories that present a clean partisan story in which one side are the obvious bad guys and the other side the obvious heroes. Very few things are that clean. Any story that proposes a nice tidy package of partisanship should just be ignored. That’s hard to do, I have to train myself to do it. I incline so instinctively toward “Ingroup good, Outgroup bad” when instead I should be seeking out careful thinkers who will tell me when they get something wrong and abandoning outlets and writers who don’t dig into topics, doublecheck their biases, and hold skepticism in their minds when they see an “easy win”.
Looney Tunes: Sahara Hare (1955)
I’m watching this one because Chuck Jones said it had the funniest camel ever animated. I was a little disappointed that there were only 3 camel gags in the whole thing (all funny though).
In this short, Bugs takes a trip to Miami beach and ends up in the Sahara desert instead (“Must be low tide”). Sam finds Bugs trespassing on his desert and proceeds to to try chasing him off.
The quality of Bugs / Yosemite Sam shorts all comes down to the gag-fest at the end as Sam grows increasingly frantic about getting back at Bugs. These gags are quite good and offer a solid variety of jokes, which keeps the audience engaged all the way through. There’s even a good 4th-wall-breaking meta joke about how Looney Tunes can be thematically repetitive. Overall a solid Sam short.
Yes, I recognize the irony of this statement having just written 1000 words about this stupid story.
Excellent report
I have a historian friend who likes to post about a phenomena called mood affiliation; it would seem to be related to the problem you've pointed out here. The overall problem is determining what the preferred outcome of your study/story is before you start your work. A person will tend to accept or reject evidence based on whether it supports their preconception (or mood).