A Long One About The Florida Data Scandal

Hoo boy, I didn’t intend for this to get so long. If you’re interested in the real story of what happened to the fired Florida data scientist, you may want to skip down to “The Saga of Manipulated Data” although every part before that is leading up to it.

In this issue:

• “Back To Normal” Twitter - People are reverting back to their partisan tribes and I hate it

• Breaking Bad Data - Lot of states have gotten data stuff wrong… and that’s ok

• Florida Deserves The Benefit of the Doubt

• The Saga of Manipulated Data

• Rebekah Jones Is Human And Truth Is Bad For Clicks

• Disney Classics: The Wise Little Hen

“Back To Normal” Twitter Is the Worst Twitter

I’m glancing through Twitter right now and things seem to be getting more and more back to normal. That makes me sad. “Normal” on Twitter is a partisan morass of accusations and the assumption of ill intent. It felt like we got a little bit of a break from this for a few months. Unfortunately, as people wear down, they seem to be returning to the comfort food of blaming the outgroup for all the ills of the world. But the world has substantially more ills than it had 3 months ago.

We’re still in uncharted territory and we all still need grace. We need to give grace to the people who are trying to figure out the best way to navigate this world of numbers, infection surfaces, masks, unemployment, stay-at-home orders, homeschooling, and multi-phase plans.

That grace might take the form of withholding judgement from the person not wearing a mask. A cancer-surviving friend of mine is worried b/c her doctor doesn’t want her wearing a mask at work since consecutive mask use above 2 hours raises the risk of bacterial contamination. Or the woman whose autistic son freaks out over breathing issues in a mask but still has a job at the grocery store (for as long as he can tolerate customers yelling at him to put a mask on).

Or it might mean swallowing our pride and just wearing a damn mask for no reason other than courtesy to our neighbors.

We’re not going to make it through a shared sacrifice and struggle if our go-to form of recreation is slinging judgement and anger to everyone who takes a contrary position. Grace returns grace. People can recognize when we are being genuine, honest, and kind and it helps them do the same. We need to give the benefit of the doubt because we need it ourselves.

This sounds too much like a sermon to me, but it was really just meant as an introduction to talking about errors in data.

Breaking Bad Data

There are a *lot* of data problems that we’ve seen surrounding the COVID-19 crisis. They take a lot of forms, they spring from a lot of errors. Some of these errors are accidents, some involve incompetence, some are policy driven.

I’ve been studying COVID-19 data pretty closely since early March. I watch to see who is reporting, how frequently, which counties or states do well or poorly, where things fail or fall through the cracks. There are dozens of examples of poor data reporting and management.

pictured: the look on your webmasters face when you tell him to add a live data dashboard

None of these data errors are malicious. No one is trying to trick us. No one is trying to intentionally under-count or intentionally over-count. No one is trying to hide cases or deaths. Everyone is doing the best they can with what they have, but problems abound. A few examples:

• problems with California’s data as tens of thousands of tests were left in “pending” status for weeks.

• problems with Washington when their health department performed a dashboard update at the end of March and they stopped reporting on all data for nearly a week at a crucial point of this crisis

• Washington and Massachusetts both retroactively updated their data series with inconsistent dates, shifting from reporting a case on the date when the result comes in to reporting it retroactively to the date the test was administered (if that information was available, in many cases it is not).

• Virginia, Texas, Pennsylvania, and Maine reported both viral and antibody tests in the same test count (which is *super* bad since those tests tell us very different things). But then… so did the CDC!

As far as I can tell, in ALL these instances, the states and institutions that have made these mistakes and stumbled in data gathering or reporting have worked tirelessly to course correct. They’re doing the best they can under a media microscope that notices every time they change a metric or push up new code or try to make improvements. Data teams that were used to doing casual updates that no one noticed once a month are now doing updates every day or multiple times a day. That’s really hard!

Real-time data reporting is complex and hard to do and, while we should expect the best out of our Departments of Health and national health institutions, we need to be generous as they work out the problems. No one expected such a heavy burden to fall on them so quickly and they are all doing the best they can with something they had to flash-learn in the last 3 months.

Florida Deserves The Benefit Of the Doubt

All of the above is true generally, but it’s even more true for Florida given how cruelly Florida and its governor have been treated in the press and among those who take a “nuke first, ask questions later” stance.

For reasons that remain somewhat opaque to me, many people have seemingly decided to go though Florida’s data looking for the worst possible interpretation for any data discrepancy. And with every story, not only did the press get it wrong, but they could have gotten it right if they only asked thoughtful questions based with a foundation in careful reporting and generosity of spirit.

I’ve spent a lot of time debunking the various ugliness coming from the press, but let’s review some of the greatest hits:

Florida officials withheld medical examiners data!

Except they didn’t. The Department of Health has been trying to review and collate the data before they publish the official numbers. The rationale was that without this review, the press would look at any death that involved COVID-like symptoms and dishonestly say that the COVID death count was higher than the official count.

I don’t know why they would suspect the press to be so unfair to them except for the fact that this is exactly what the press the did.

Florida is hiding their death count by not counting non-residents!

Florida is required by the CDC to tally COVID deaths by the primary residence of the deceased. They have tried to increase the transparency of this by adding “Positive Non-Residents” to their dashboard so that, even if they don’t include those individuals in their death numbers, citizens can see what percentage of cases are non-residents (it’s about 2.5%).

Florida is failing to report likely COVID deaths!

This is untrue. Even more insulting, the author of that piece references the fact that the CDC’s excess death data shows that COVID deaths are likely under-counted and says this is probably true of Florida. But that very specific thing is very specifically not true in Florida. In fact, Florida’s excess death data shows the opposite of New York’s. It shows that, excluding COVID deaths, excess deaths are lower than normal.

The orange line below is the high-end marker for expected excess deaths. The blue are official COVID deaths. The surge of green above the yellow line are most likely COVID deaths that are unreported. As you can see, this is a problem in New York, New Jersey, Michigan, and a few other hard-hit states.

But not Florida.

The author took a data point from one state, said that it was probably happening in Florida too so we can’t trust their data, and then never checked to see if his statement was correct.

It is not. That this could be published without a fact check on a key point like this is a disgrace.

Florida is doing good work in both managing this crisis and in handling the enormous data task set before them. They deserve better press than they are getting.

The Saga Of The Manipulated Data

But the latest and biggest story in the “Florida is lying about their data” narrative came this week with the breathless report that Florida ousted their top data scientist because she refused to manipulate the data.

I want to pause here and say that I have struggled on what to say about this because I’m watching this story unfold from several different angles. I’ve struggled with whether or not to use this woman’s name because she has effectively gone into hiding. She is being called by reporters hoping to finally pin Governor DeSantis to the wall for corruption. On the other side, she’s under a microscope of scrutiny, one that we should all wish to avoid and that few of us could manage to escape unscathed. I don’t want to be any part of a life-destroying dog pile.

I’ve decided to use her name because 1) it’s already out there and 2) I hope that people read this and see her as a human dealing with a lot of things who ended up in a really tough place.

Rebekah Jones is not the important part of this, she is merely an unwilling player. The larger problem is how deceptively and carelessly this has been reported.

If you can stomach it, watch this video.

Watch how the the initial reporter and the MSNBC host fumble around with trying to even describe what they think happened. They are certain that there has been data malfeasance and it’s definitely political and it’s certainly corrupt… but when they try to describe in detail exactly what has happened, they stumble around, tripping over their words, checking their notes, completely uncertain about any of the details of what exactly happened to the data.

What did happen? Some people say that Jones was asked to remove the data download from the website. This is not true, the data has always been available and is available right now.

Some people say “they” (it’s always an amorphous “they” and never a specific person) wanted her to remove a vital column of data, the “Event Date” which shows then the person first exhibited COVID symptoms. The idea is that this information is being suppressed because it would reveal that Florida should have acted sooner to save lives. But… that column of data is still there.

This is not a case of the data re-appearing now that the story blew up. It’s always been there. None of the things that the Florida Department of Health has been accused of happened. At all.

But something happened, right? What was it?

Rebekah Jones Is Human And Truth Is Bad For Clicks

The truth of it is incredibly boring.

The truth is that Jones made a mistake in allowing an auto-export of a PDF before the data had been reviewed by Florida’s epidemiological team for accuracy. The director, seeing that the unchecked data had gone out into the wild, requested that it be removed until it could be verified. This message is relayed to Jones through her boss and she, anxious that this data removal would impact people checking the dashboard, argued that they should keep it as it is. She was overruled and complied with the request, disabling the auto-export.

The dashboard used the old exported data for approximately 80 minutes before the data was verified and the auto-export was re-enabled. UPDATE: From further conversations, it looks like the dashboard did go down & did not fallback to older data, which is why Jones was so upset about it.

Dr. Carina Blackmore is the director for the Division of Disease Control and Health Protection and the State Epidemiologist. She is a researcher and public health manager with 27 years of work in environmental epidemiology, toxicology, and public health oversight. When she gives an order to turn off auto-export in order to perform a data review, she should get the auto-export turned off for the data review. She should not get her request rebuffed by a dashboard operator with an email that says “this is the wrong call”. That is not appropriate workplace behavior.

And so Jones was re-assigned. This frustrated her because she loved being on the dashboard. She loved answering questions and helping people sort through the data. She felt like people were relying on her. So she sent an intemperate email out telling the users of the state’s data portal, a small group of data nerds and researchers (including your humble author). The entire unredacted email is at the bottom of this issue, but the part that has been most reported is:

As a word of caution, I would not expect the new team to continue the same level of accessibility and transparency that I made central to the process during the first two months. After all, my commitment to both is largely (arguably entirely) the reason I am no longer managing it.

This email looked and sounded like she was being reassigned for her commitment to transparency, which is very likely how she felt when she sent it.

She sent a much more reckless email to CBS-12 where she claimed that a shadowy figure within the department was forcing her to “manually change data to drum up support”. This accusation was likely the reason she was fired.

At first, I was sure Jones was talking about Dr Blackmore herself in this email, but I eventually discovered that the person Jones is accusing of trying to censor the data is her supervisor, Dr Shamarial Roberson, a chronic disease epidemiologist and Deputy Secretary for Health, who has stated that it is “patently false to say that the Department of Health has manipulated any data.”

That line, “I refused to manually change data to drum up support for the plan to reopen”, is simply untrue. And it’s likely that this email is what infuriated her former team and her former boss and everyone in her organization.

If I may speak personally for a moment, I deeply wish she had not sent this. I believe it was sent in anger without proper consideration for the consequences. But it is sadly the case that often the worst moments of our lives come to define us.

Far less prominent in the reporting (if it is mentioned at all) is the follow-up email she sent when her supervisor asked her why she sent such an poorly worded email to such a vital group of researchers (also provided at the bottom of this issue).

“umh… uh oh?” is the subject of this email. In it she explains that she really just wanted to help direct people to the right place for their questions. She wonders if she shouldn’t have said what she said and is anxious that she has caused problems for the team she used to work with and that she still cares for deeply.

She ends her email:

I really don’t want to be a story, but this lady has called me like three times

Please help me.

There Is No Conspiracy

This story has caused thousands of people to claim that Florida is faking their numbers. The Sun-Sentinel all but accuses Governor DeSantis personally of “rigging COVID-19 data” which is a flat-out lie and they know it. That narrative is now set for people and there is no going back from it. In pursuit of this narrative, the press (especially the Florida press) has repeatedly misinformed, fumbled basic data concepts, and omitted vital information. They have taken any small accident in reporting, any decision to protect patient privacy, any after-the-fact data correction and spun it into a grand conspiracy.

None of this is true. In fact, the opposite is true. The Florida Department of Health has worked heroically to deliver the best data they can with the most transparency possible as quickly as they can.

I would caution that if you ever see a story on the malicious hiding of data (from Florida or any state health department) just assume that the person writing it has no goddamn idea what they are talking about. Assume they have no interest in bringing you accurate information and that you will need to seek it from other sources.

When it comes to data, assume that the data you’re seeing is coming from dedicated people who are working hard. Assume that we’re all under stress and our jobs are weighing on us. That’s the only way to be fair and the best way to find truth. Start by assuming that people are doing their best and then, if something went wrong, think hard about how that could have gone wrong in a system where people are trying to make the best decisions.

Show to this woman the grace you wish someone would show to you when you make a mistake at work. And then show that grace to the rest of Florida Department of Health and to everyone working on the hardest, most consequential, most intense data project they’ve ever had the misfortune of falling into.

Be skeptical of those who don’t show that grace. They may not intend to cause so much damage to public trust, but that is what they have done and that damage is irreversible. Trust is destroyed quickly and heals slowly. They should know that they are not acting as agents of truth, but agents of destruction.

Disney Classics: The Wise Little Hen

Watch The Wise Little Hen on Disney+

Taken all by itself, this is a fun little fable of a cartoon. Early in the world of animation (this was released in 1934), there was something of an arms race among studios to make cartoons out of classic fables and moral tales. They were stable, uncontroversial, enjoyable, widely loved, and easy to translate so that they could be distributed worldwide.

But The Wise Little Hen was actually designed as something of a trial run for Clarence Nash to play Donald Duck. Walt Disney liked Nash’s duck voice, but was hesitant to include him among the Mickey Mouse cast of characters. So Walt brought Nash in to voice Donald Duck for this short.

It worked like a charm. The Disney animators managed to get a good feel for the Donald Duck character with this short and Nash signed on for what would ultimately be a 50-year career playing Donald Duck.

This short is a lovely little piece. It’s simple but gorgeously animated and superbly scored. I love how, as the corn sprouts from the ground, the young shoots are scored with the delicate tingle of bells, and as they grow more robust, the fuller sound of the oboe gives us a heartier sense of scale. It really is astonishingly good.

Attached Documents:

The First Email

“Hey all –I've gotten a lot of emails from everyone during the last eight days ever since the dashboard went down, the data was hidden, and the functionality essentially crashed, so to clear up the confusion, I'm sending this final notice to the group.

For reasons beyond my division's control, as of late in the day on May 5, my office (the DOH-GIS office) is no longer managing the COVID-19 Dashboard. I am no longer involved in the publication of data, fixing errors, answering questions, etc., in any shape or form. I helped them get it back running a few times but I have no knowledge about their plans, what data they are now restricting, what data will be added and when, or any of that. I understand, appreciate, and even share your concern about all the dramatic changes that have occurred and those that are yet to come. However, I cannot provide any insight now or going forward.

As a word of caution, I would not expect the new team to continue the same level of accessibility and transparency that I made central to the process during the first two months. After all, my commitment to both is largely (arguably entirely) the reason I am no longer managing it.

They are making a lot of changes. I would advise being diligent in your respective uses of this data. I know many of you have broken API links and map layers. I've listed the contacts for getting that information below.

The primary contacts going forward are listed below.

Anything related to EPI: Thomas.Troelstrup@FLHealth.gov or Scott.Pritchard@FLHealth.gov

Anything related to the technical aspects of the dashboard: Jessica.Joiner@FlHealth.gov

It was great working with you guys.

Good luck, and stay safe.

The Second Email