An uncontroversial statement: COVID misinformation has been a massive problem all year and isn’t really improving.

A more uncomfortable statement: it’s not just QAnon tinfoilers. The number of major outlets and public experts not guilty of this is maddeningly low, and many of the most serial offenders are mainstays on highbrow lists of supposedly credible sources.

Today’s story centers on Reuters and Harvard-trained epidemiologist Eric Feigl-Ding. But it’s not really about them specifically. It’s about experts and institutions more generally. Why do they keep getting these things so wrong? And if we can’t trust them to meaningfully understand what they write about, who can we trust?

TL;DR

(The story in three bullets for the time-starved.)

Dr. Eric and Reuters both claimed that an Oxford/Lancet study found that 20% of COVID survivors go on to develop a new psychiatric illness (i.e., no prior diagnosis), which is 3.4x higher than what the study actually found (5.8%).
It’s not just that they looked at the wrong number. It’s that even the lower number (which likely isn’t right either) doesn’t mean what they seem to think it does, which is something we’re depending on them to understand.

While it’s intuitively true that getting COVID is worse for one’s mental health than not getting COVID, the real risk level here seems more “yeah that’s about what I’d expect given how sick some folks get and how long symptoms can linger”, not “omg it’s russian roulette but the bullet is an anxiety disorder”.

Subscriber Note: This post was supposed to be about Benford’s Law and how it relates to the latest (wildly incorrect) election fraud claims. Stay tuned for that.

New Reader Note: We reward corrections. See something? Say something. It helps.

Ok, main event time.

Let’s open with the top two tweets from the thread that stole my day:

As for the linked Reuters article:

Before we even get into the underlying study, I want to emphasize that this isn’t just a misreading of the data. It’s an absurd misreading of the data.

This was my first-instinct comment when sharing the above to a groupchat:

(*so grossly)

In like spirit, an English teacher friend in the chat, having read only the excerpt, reacted with:

Some claims are just facially incredible. While this doesn’t necessarily mean they’re wrong, if you’re, say, a public health communicator, the “yuge if true” element should probably prompt you to, like, do a quick once-over before you hit publish.

Anyway, I couldn’t help myself from digging in, and here’s what I found on the study’s first page:

The incidence of any psychiatric diagnosis in the 14 to 90 days after COVID-19 diagnosis was 18.1% (95% CI 17.6–18.6), including 5.8% (5.2–6.4) that were a first diagnosis.

So we can see the mistake here. 18.1% of these COVID survivors went on to receive a psychiatric diagnosis (we’ll get to what that means), where that number includes those who already had such a diagnosis prior. Remove the already-ill and the number is 5.8%.

But, again, it’s not just that Eric/Reuters confused the numbers and thus considerably overstated the problem. It’s that they took the numbers at face value without any discernible attempt to flesh out what the data is really saying.

Some questions a cautious expert might ask:

How many of those people would still have received these psychiatric diagnoses regardless of their COVID infection?
How sure are we about those 5.8% never having had a previous psych diagnosis?
What does “diagnosis” even mean here anyway?

Eric and Reuters never got there. But we can, so let’s take those in that order.

(In the spirit of fairness, Eric’s 6th and 7th tweets kinda address that first question by virtue of including graphs/data that could very well set an attentive reader in the right direction. But the interpretive work was all delegated there.)

I. Baselines

Ok, so 18.1% of these survivors went on to receive some psychiatric diagnosis within 90 days (which in over 2/3rds of cases was not their first). That’s still pretty high!

But bear in mind that the number of people with psychiatric illnesses is also pretty high! And anxiety in particular (the largest driver of that 18.1%) is widespread pretty much everywhere. Something like a third of us will be affected by it in our lifetimes.

Both illnesses thus being prevalent, our base expectation ought to be that a good number of people will get COVID and then go see a medical professional about anxiety where those two things are totally unconnected.

How often might this happen? And how can we guess when the data doesn’t tell us explicitly?

Well one smart approach is doing what the Oxford researchers did, which was looking at how often people received psychiatric diagnoses after non-COVID health incidents.

(The x-axis here is days after the health event kicked off.)

Ok so for respiratory infection survivors it’s a fair amount lower (14.1% vs. 18.1%). But that’s also pretty high! And there’s no obvious causation here. All things being even, a respiratory infection is not going to make you all that more anxious or depressed than you were prior (especially weeks or months later). Thus most of the correlation is coming from other things altogether (like the general prevalence of mental illness).

The study also looked at psychiatric diagnoses following health events like large bone fractures (12.7%) and gallstones (15.1%). Again, still notably lower than 18.1%. But the gap here is less “COVID IS A 5G TOWER OF ANXIETY” and more “huh, well that’s not ideal”. And given that COVID symptoms can be quite affecting / frightening / enduring, this gap isn’t exactly surprising.

Anyway, there’s more.

(Note: Yes, sure, the gap is larger when we remove patients with prior diagnoses, just as we’d expect it to be. People with a history of anxiety are more likely to have anxiety again in the future, etc. Strip those folks out and you have something closer to apples to apples. The figure for “first-time psych diagnosis came after COVID” is 5.8%, vs. just 3.4% for “after a respiratory infection”, which is a 70% gap. But as we’ll get to next, that 5.8% figure is very likely overstated, meaning the gap would also be overstated.)

II. Illness Histories

There are ways in which large studies are better, and there are ways in which they’re worse.

One disadvantage here: telling if someone had a prior psychiatric diagnosis is actually a pretty difficult data-acquisition problem, and it gets harder when you’re dealing with sample sizes beyond what can be verified with patient interviews.

But our researchers ultimately took the best route they could for the study overall, which was to consult a very large joint dataset covering some 70m Americans. Per page 21 here, they then narrowed focus to 12.6m records, all of which they indexed by “is there a relevant ICD code in this record to indicate a prior psychiatric diagnosis”.

But while 12.6m is quite a lot for study purposes, it’s also quite low relative to the adult US population (~260m). And this leaves us with a specific vulnerability: of those who received both a COVID diagnosis and a subsequent psych diagnosis from included-in-this-dataset healthcare clinics, some would have already had prior psychiatric diagnoses from not-included-in-this-dataset clinics. Ergo some supposed first-timers are actually mislabelled, thus skewing the results.

To their credit, the researchers do acknowledge this problem in the opening of their appendix:

A general limitation of EHR data is that a patient may be seen in different HCOs for different parts of their care and i[f] one HCO is not part of the federated network then part of their medical records may not be available.

Of course, this limitation doesn’t make the study bad. It’s more an interpretive thing where we need to be mindful of the likely skew, especially in light of the other main thing the study found: people diagnosed with a psychiatric issue in 2019 were far more likely to get COVID in 2020 (they say by 65% vs baseline).

This might seem like a “ok and?” thing at first, so let’s unpack why this matters. When we assume that comparing “got a first-time psych diagnosis after x health event” cohorts will give us useful data, we’re assuming that people in these cohorts are more or less equally unlikely to have a hidden prior psych diagnosis. Thus apples to apples. But if people with a past psych diagnosis are considerably more susceptible to one of those health events (i.e., COVID), we’d expect to see a resulting tilt.

(Just imagine a set of 1,000 people in which you know 100 have lied about disliking White Claw, and that you randomly assign them into groups of 250 where each group gets a different drink. You’d expect roughly 25 denialists per group. But in our case the assignment isn’t random. People got sorted in a way that was influenced by that hidden trait, which is a bit like giving those 100 people some say on which drink group they joined. You’d expect to end up with considerably more secret stans in the White Claw group than any of the rest, which would meaningfully skew any resulting data.)

While I found the study pretty good overall, I’m a bit miffed that they didn’t acknowledge this skew more openly. When one of your core findings seems likely to heavily influence the other, you should probably say that explicitly. Because we can’t trust that our public experts/communicators are going to catch it.

III. Diagnosis

When determining if someone had a prior psych diagnosis, the researchers (quite reasonably) relied on ICD-10 codes (specifically F20-48).

What are those you ask? Whenever you go see a health professional, there’s at least one reason driving your visit (even if that reason is just “it’s time for my annual checkup”). Said professional will then log said reason(s) using the most relevant ICD code(s), which makes billing/insurance easier and gives us really valuable global data about health trends and incident/disease prevalence.

But are incidents of these codes really “diagnoses”? Many in this study received repeated ICD-F codes for the same thing across different visits. If you’ve already been treated for anxiety, is your second visit about it really a re-diagnosis? Well, that depends on your frame of reference. From an ICD perspective, yes, in that both visits related to x diagnosis. But I wouldn’t return from that second visit and tell my partner “guess who just got diagnosed with anxiety again”.

To use another real-world example, consider why people often use qualifiers like “clinically-diagnosed” in speaking of a mental illness. They’re not communicating technical information. All professional diagnoses are clinical. The force is usually to establish a conceptual difference between chronic/severe cases and more situational incidents of “this is what I’m going through right now”.

Unhelpfully, we have just the one word to cover both meanings. You might be diagnosed based on a high-concern score on a DSM-derived symptom test conducted by a psychiatrist, but you might just as well be diagnosed conversationally by your family doctor in the form of “yeah sounds like you’re experiencing acute stress right now”. While a precise ICD code could help us differentiate, those logging the codes vary in their time, familiarity, and incentives. Thus sometimes “close enough” wins. (Much of the specificity we have was lost in the study’s aggregation process anyway. The best I could find was table 10 in the appendix, which labels ~45% of these first-time diagnoses as a general/unspecified version of “other anxiety disorder”.)

So what are we left with? A vague sense of “well these people all visited some health professional who saw fit to invoke some ICD-F code to describe the visit”.

That ambiguity in mind, consider the Reuters headline again:

That’s a scary premise! Imagine that you’re mentally healthy in some normal sense. Would you read that headline and think “ah yes if I get symptomatic COVID and I’m unlucky enough in my symptoms then sure yes I’m also more likely to experience some amount of mental cracking”, or would you interpret it to mean that surviving COVID would still leave you a 20% chance of becoming mentally ill?

The overstated numbers aside, that interpretive difference is meaningful.

Anyway, let’s wrap up.

Prevalence Among COVID Survivors

It’s intuitive that people with a disease known for violent and lingering symptoms stand a good chance of having a go with their mental health after. Not news really.

But how badly will it go? We don’t know. The data tells us almost nothing about severity, or duration, or anything really. And though it tells us something about likelihood, it’s not very clear there either. If people with a history of psychiatric illness are substantially more likely to get COVID, and if the limitations of our data mean that we’re likely to miss some of these prior diagnoses, then we’d expect for some of these COVID survivors to get “rediagnosed” with psychiatric illnesses they already had. It’s true just tautologically. The results follow the premise.

Of course, I don’t mean to downplay what we do know. COVID can have brutal and enduring effects on even young and healthy people, and many survivors remain in physical and mental hells. No one should take this virus casually. But if we want people to make good risk decisions, we need to give them the fairest views of the data we can. Special alarmism does not seem the right response to this particular study, as overstating the dangers seems likely to just further damage expert credibility when the claims are seen to be obviously untrue in actual experience.

Prevalence Among COVID Experts

Consider the following:

From Dr. Eric’s Federation of American Scientists profile: “[Eric] is part of FAS’s work to stop COVID misinformation, and communication with lay public.” [sic]

From Reuters’s Trust Principles: “the integrity, independence, and freedom from bias of Thomson Reuters shall at all times be fully preserved”.

I suspect that Eric and Reuters got this wrong because they prioritized speed, and I suspect that they prioritized speed because of bad reward structures. Getting in fast and early to win clickshare is easy money. But it happens to also be almost entirely incompatible with “stopping COVID misinformation” and “preserving integrity”.

So, yeah, I do think that the persistence of these broken dynamics is indicative of a mental illness: cognitive dissonance.

We know this way of doing things doesn’t work. We know it can’t work. We have watched each and every so-called credible institution fail in the same way on the same subjects, at catastrophic costs to public knowledge and institutional trust.

Yet on we go, refusing to fully confront our own reality, imagining that just voting out Trump is somehow going to fix what ails us.

——

PS

One of my reviewers, a postdoc, pointed out that the problem isn’t just speed. It’s a general deference to what study promoters wish to say. While we can forgive a bit of collegiality, the public communicator’s main currency is clarity, and clarity requires an accurate understanding of the facts. Sometimes that means pushing back, and always it means reading carefully and with a skeptical view. Speed is part of why so many fail here, but it’s not the only reason.

(To any folks who might object “but you can’t seriously suggest that communicators need to be able to challenge the science they’re reporting on”, I wonder if you think their work is done when they’ve spelled names correctly and parroted primary findings without confusing any big numbers? Because we already have bots that can do that, and they’re better at it. If the point of a journalist isn’t to establish the truth of what they report, then I genuinely have no idea what the point of a journalist might be.)

Psychiatric Illness Prevalence in COVID Surv…

Create your profile

Only paying subscribers can comment on this post

Check your email