Is The Vaccine Roll-Out a Success or a Failure?

What metrics tells us how well we're doing?

I’ve updated the data set available to subscribers to include some early vaccination data. I’m now pulling state-by-state CDC data on vaccinations in order to be able to gauge where states are and the velocity at which they are managing to administer vaccines. This data isn’t super interesting yet because I only have 10 days of data, but this will become more interesting in the next few months.

• What Are We Even Measuring For Success?

• Answer The Damn Question: Success or Failure?

• Looney Tunes - Ballot Box Bunny

What Are We Even Measuring For Success?

When it comes to trying to vaccinate an entire country of 330+ million people, there are not a many people with a experience to whom we can turn.

Rapid mass vaccinations were common in the smallpox eradication era, but that was a scenario in which extremely heat-stable vaccines that cost pennies to produce had been stockpiled for years. Public health officials all participated in the vaccinations and the most common process was to literally line up every person until they were all vaccinated or you ran out of vaccine.

But if we’ve never done something like this before, how do we know what metrics indicate that things are going well or poorly?

I would like to avoid the most common method, which is to put all states in a list and then look at that list and pick the high states and say “those are good” and the low states and say “those are bad”. The problem with this approach is that each state has its own challenges both in distribution, in population homogeneity, and in reporting efficiency. Some state will always be at the top and some state will always be at the bottom. That does not mean that the #1 state can’t improve and it doesn’t mean that the 50th state is obviously failing. There is a lot of nuance in the numbers and multiple metrics that we need to be aware of.

One metric used by Bloomberg’s vaccination tracking page is “% of shots used”. This is one of those really fascinating examples of how data is presented really colors how people interpret the data. The only way to get a “best-to-worst” ranking is to click on the “% shots used” column to sort it. So that is what people do and then they make their determinations about who is doing better.

If you look at things through that lens, Texas is kicking butt having used 55% of their allotted vaccines and New York is near the bottom, using only 43% of their vaccines. Except that when you dig into the numbers you realize that a big part of the reason is because New York ordered a lot more vaccine for their population than Texas did. These two states have similar rates of vaccination, it’s just that Texas is running out faster because they don’t have as much vaccine on hand.

Another metric could be vaccine wastage (the amount of vaccine that needs to be destroyed for expiration and spoilage reasons). I think we could all agree that high rates of vaccine wastage would be big red flag that things are going poorly. However, I have not seen any state reports that detail large scale vaccine wastage. I’m mostly seeing isolated reports of vaccines being tossed but the numbers are always in the hundreds. That’s still bad! But it’s not “everyone panic” bad. It’s more “we need to be more flexible” bad.

The last metric that we might turn to in order to determine if we are doing well is the “vaccinations per 100K” which helpfully normalizes the vaccines administered so that we can see what percentage of the population has been vaccinated. The “winner” so far on that metric has been West Virginia, which has vaccinated 6% of their population. This is the metric I will rely on in my monthly COVID report (which will start including vaccination data this month). It’s not the perfect metric, but it’s the most helpful one if we decide that we’re going to rely on one metric and only one metric. I will, as always, try to include all relevant caveats so that we are not just staring at charts.

Answer the Damn Question: Success or Failure?

With all that in mind, I want to propose a somewhat impish theory of gauging vaccine success.

Sure, we’d love to have everyone in the country vaccinated by June, but is that realistic? A friend proposed to me that, if we meet the metrics proposed by the incoming Biden administration, we should consider it a success. After all, the Biden administration is entering with the vision of solving this crisis and so their targets need to demonstrate what they consider to be a realistic but optimistic drive toward that goal.

I like that idea in part because the media has described Biden’s target of 100 million vaccines by his 100th day in office as “ambitious” and described the vaccine rollout to date as slow and disastrous. So we can take what has happened right now and call it “slow” and then we can look at incoming President Biden’s targets as “ambitious”.

The problem is that what is happening right now is we are vaccinating at exactly the rate we need to vaccinate in order to hit the Biden target. The math behind this:

We currently have about 13 million people vaccinated. We’re vaccinating at a rate of 900K per day (based on an average of the last 7 days). If we vaccinate at the same rate we are now with absolutely zero improvements, we will have 107 million people vaccinated by Joe Biden’s 100th day in office.

At that point, I cynically suspect there will be the declaration of success, that the Biden administration turned around this disastrous rollout when, in fact, things just kind of moved forward as they have been moving forward heretofore.

With this in mind, I think we’re doing fairly well. We’re currently hitting the metric that the incoming administration says we need to hit in order to say that we’re being successful. Maybe they’re bullshitting us, but I don’t think that’s the case. I think they are looking realistically at the problem and then adjusting their targets so that they can declare success because they see we are already on that path.

To this end, please inoculate yourself from hyperbolic news that laments the crushing defeat of American innovation at the hands of an irrational and unfeeling bureaucracy of dunces. This is simply not true. All states that I’ve dug into in detail are looking at the problem and working within their own context to resolve it as best they can. I think that, 2 months from now, we’ll see that certain policies were more successful than others, but by that time I also expect to see the bad policies course-corrected.

I don’t see a lot of value in pitting the 10th best state against the 10th worst state and making sweeping statements about the value of their policies if they are only a few percentage points different in their vaccinated population. There will be differences but each state really does just want this thing to die so let’s keep showing grace and rooting for every vaccinated individual who gets to not fear this awful disease.

Looney Tunes: Ballot Box Bunny

Bugs Bunny manages to make enemies with almost everyone he meets but his most entertaining enemy is Yosemite Sam. There is something wonderful about the outlandish villiany Sam exhibits that can then be countered by Bugs’ wicked wit.

In a run for mayor, for reasons beyond comprehension, Sam decides to make a major plank of his platform the eradication of rabbits. This drives Bugs to run for mayor to counter Sam’s promise of two rabbits in every pot. The rest of the short is a set of gags about local politicians and the lengths to which they go to get votes. Nearly every scene ends with Sam getting blown up.

It’s funny and cute, but a little more same-y than I remembered from my childhood. The word play joke at the end is entirely worth it though.