Gaming Measurement of Vaccine Efficacy
Gaming Measurement of Vaccine Efficacy
The Chloroquine Wars Part LXIX
Biostatistics Professor and data science blogger Jeffrey Morris writes,
We (meaning those of us who are paying attention to the vaccine studies) are all aware of the Kaplan-Meier curves that show that antibody buildup seems to ramp up to effective levels at around 10 days after vaccination. And yet, clever as the hivemind may be, none of us has yet devised a way to vaccinate people in such a way that the first 14 days don't actually happen.
The problem is that when discussing the most important aspects of medical intervention (such as risk-benefit analysis), starting measurement of efficacy at that point makes no sense. But it may not be clear why it doesn't make sense to some people, and bias may get in the way. In order to sort out such temptations to reset basic modeling principles (like to pretend away the first 14 days because maximal efficacy doesn't yet take place), the experienced mathematician or statistician often starts with the extremal principle in order to assess the difference. That is to say that it should be a professional habit to examine a case in the neighborhood of the worst possible scenarios, and then to consider the spectrum of results in between the relevant boundaries.
The Morris Therapy
Since Jeffrey seems to feel okay with simply declaring (as he has in numerous emails we've shared) something like, "I'll be fine with including the risks when they're published in a peer reviewed journal" (you know...like the Surgisphere paper), while feeling just fine about performing computationswithout asterisks pointing out that his calculations would look wildly different depending on that suppressed (autopsies) data, I'll honor him with the naming of this edge-case.
Suppose that tomorrow it is announced that in a Wuhan laboratory---located somewhere between the French-designed Wuhan Institute of Virology and the fabled wet market---that an NIH funded project results in highly valuable intellectual property in the form of the Morris Therapy. After rigorous mandatory testing of the Morris Therapy on Uyghurs, including the high risk elderly, pregnant women, and also children, the Morris Therapy demonstrates 100% efficacy in preventing COVID-19...after day 13.
NIAID Princeling, Dr. Anthony Fauci quackly announces an EUA both for the Morris Therapy, and also swift approval of the as of yet unavailable COMorrisY Therapy. Nobody is reachable at any governmental organization who can explain which of these has indemnity, and whether citizens pay for those liabilities by giving up their children as they drop them off at school where they are to be treated using Morris therapeutics of unexplained composition by swiftly-trained gym teachers.
During the first two weeks, 80 million Americans jump at the opportunity to receive Morris Therapy. However, 79.2 million of those Americans seem to be...well...missing. With doctors and morgue owners tight-lipped, the CDC reports that indeed, the Morris Therapy shows 100% efficacy in preventing COVID-19 after the first 13 days. Scientists and other people who pay attention to things rush to post videos on YouTube, Twitter, and Facebook pointing out that there are nearly a million reports of death in the Highly Profitable Therapy Adverse Events Reporting System (HPTAERS). Fact checkers [just doing their job] point out that causality of those deaths HPTAERS hasn't been proved, and censor all those reports. University faculty petitioned to silence, reprimand, or simply harass their colleagues who participated in those reports, labeling them with the pejorative "anti-Morrisers". No autopsies are performed on the 79.2 million Americans who died during the first 13 days after a dose of Morris Therapy. One brave nurse who spoke up about Morrising patients to death was fired without an investigation.
What Just Happened?
What happened is that 99% of the people who received the Morris Therapy dropped dead during the first few days. Even worse, many were taken to the hospital, entered into a database as non-Morrised, and as hospital beds filled up, media outlets declared a "pandemic of the un-Morrised". All the social pressure makes it hard to talk about the problem. Meanwhile, nobody does autopsies on the bodies that might reveal clues to the deaths, such as the presence of spike-Morris protein in organs all over the body. All the 79.2 million deaths are presumed to be COVID-19 deaths, and the media dedicates itself to a 48 hour marathon of fear porn, lamenting that poorer nations are not receiving their equitable share of limited supplies of Morris Therapy.
Meanwhile, health officials and all those devoted to the success of the Morris Therapy as the final solution to the COVID-19 pandemic…"correctly" point out none of the 800,000 survivors of the Morris Therapy have COVID-19 (though nobody really wants to talk about disease etiology as it might link Morris Therapy as a Type II COVID-19). In the end, the entire population of the SARS-CoV-2 virus decided as a "species" (if we can call a virus a species), that invading the U.S. just wasn't worth it at all, deciding instead to go live amongst the flies, minks, bats, and pangolins.
Yes, The Morris Therapy is an Extreme Case
But it also displays the point: if you're computing without respect to what happens during the early period---regardless of expectations of efficacy during that time (that passing of time that cannot be avoided)---then you are not examining the sum total of the effects of the therapy.
And that's just plain dumb.
It's so dumb that I've found it extremely hard to tell the difference between those in or around public health who are dumb and those who are simply of bad faith. I'll leave that up to readers to judge.
The Statistician's Solution
A statistician should never be satisfied with an important, policy-driving computation that puts such a blunt thumb on one side of the scale. The ethical goal should always be to match numbers most meaningfully to reality. It is fine to define something like an interval efficacy number, but to the extent that we do, we should communicate clearly to the public that (1) this does not express the efficacy on the superset of time from the outset of treatment, and (2) that no efficacy numbers (even 100%) can possibly tell us whether a therapy has a positive risk-benefit. The responsible thing to do is to downplay such an efficacy number, and demand that the work be done to achieve meaningful data that would allow for the computations necessary for an even modestly meaningful risk-benefit analysis.
Excellent. Here is another way to put it: Let’s take a contained set of folks aka population of a city with 1,200,000 inhabitants. 1% of all die every year that is to say that every month you have 1,000 leaving us, sadly. Now, we take a very mild poison and administer it only to those who have 30 days to live. What could happen? Well, instead of losing 33 people per day, every day, we lost all 1,000 in the first 15 days while there were no deaths in the second 15 days. An objective measure, alone the total 30-day timeline, of the poison would be to say that it is only marginally murderous as it has not caused any additional deaths that month. But if we start looking starting on the 16th, we can come to a completely opposite conclusion, i.e. that the poison, instead of killing people quicker, actually saves everyone 100% of the time.
Now, let’s consider the Pfizer (and Moderna, and AZ, and JJ) trials. You give folks shot; something happens. In fact, “whatever happens” seems to affect the vaccinate folk in much greater measure that the unvaccinated bit. Then you give them another shot and then wait 14 days. During these 14 days, something happens. In fact, “whatever happens” seem to affect vaccinated slightly more than unvaccinated. Upon the completion of the 14-day period, something happens. In fact, “whatever happens” seems to affect the vaccinated folk less than the unvaccinated group.
You stop the trial and voila – We Have 95%!!! A Miracle Is Born!!
This reminds me of when my mother goes gambling. She always tells us how much she "won", but never mentions how much it cost her to get to those winnings...