:format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/66701021/GettyImages_1210841766.0.jpg)
The question of how many people in the United States currently have, or have already had, the novel coronavirus is one of the most urgent that scientists need to answer.
It’s agonizing to not have this information because it could stop more people being infected and help policymakers figure out when it’s safe to reopen the economy.
Recently, two research groups in California from Stanford University and the University of Southern California (with some overlap between them) took a stab at estimating just how much the virus has spread in two populations: residents of Santa Clara County and Los Angeles County.
Their results are preliminary and haven’t been peer-reviewed by other scientists or published in a journal. But they suggest there were vastly more infections than the number of confirmed cases in those areas, perhaps as many as 85 times more.
If their estimates are correct, that suggests the virus is far more prevalent in the population than previously thought, but also that the disease is far less deadly. However, other scientists noted there were major flaws with these studies and quickly picked apart the findings on Twitter.
What a weird turn to see John Ioannidis pushing one of sloppiest studies in the deluge of Covid-19 papers. If he weren't an author I would expect the Stanford serostudy to show up in one of his talks as a particularly potent cocktail of bad research practices.
— alex rubinsteyn (@iskander) April 19, 2020
Critiques ranged from how the researchers recruited participants to the accuracy of the tests they used.
While a finding that Covid-19 isn’t as dangerous as once thought may be reassuring, the evidence presented by these studies may do more to obscure than clarify the situation.
Health officials are still struggling to control the pandemic, and there is immense pressure for information needed to make critical decisions. But for a survey of the spread of the disease to be useful, it needs to be conducted on a larger scale and with more rigor.
The Covid-19 antibody surveys found that the virus is more common and less dangerous than previously thought
One of the biggest problems with figuring out how many people have been infected with Covid-19 is how stealthy this virus can be. It spreads easily between people, and many can be infected without showing any symptoms. So estimating the prevalence of the virus in a population based just on hospitalizations, deaths, and positive tests vastly underestimates its spread. As of April 24, the US has reported more than 895,000 confirmed cases of Covid-19.
The researchers behind the two California studies wanted to fill in this blank. They tested people for antibodies to SARS-CoV-2, the virus that causes Covid-19, by screening blood serum, the liquid part of blood that’s left behind after clotting. Such serology tests can reveal who was previously infected with the virus, even if the virus is no longer present in their body. The tests also could give scientists information on possible immunity to the virus.
In Santa Clara County (population 1.9 million), researchers conducted serology tests in 3,300 participants; 2,718 adults and 612 children. Based on the number of people who tested positive for antibodies to the virus, the researchers concluded that between 2.49 percent and 4.1 percent of Santa Clara residents had been infected — between 48,000 and 81,000 people.
That’s 50 to 85 times the number of confirmed cases at the time of the study. Scientists expect more people have been infected with the virus than have been confirmed by tests, due to low levels of testing and many people only having mild cases. But this number is much higher than previous estimates, and other researchers have been critical of the result.
Meanwhile, the antibody survey in Los Angeles County (population 10 million) didn’t specify the number of people tested but concluded that between 2.8 percent and 5.6 percent of adults in the county carry antibodies to the virus. There were 7,994 confirmed cases of Covid-19 in the county at the start of the study, so the projection indicated that 28 to 55 times more people may have been infected than previous testing had confirmed.
The California studies are preliminary, but they still have significant problems that overshadow their findings
The Santa Clara study is a preprint report, meaning that it hasn’t been peer-reviewed. The study, like many on Covid-19, was hosted on a preprint website so that other scientists could take a look without waiting months for the review process to run its course. That can be helpful in a rapidly moving crisis like the Covid-19 pandemic. But the methods, data analysis, and results haven’t been critiqued by other researchers and could have significant flaws that would have been corrected in a published paper.
The LA County results, meanwhile, haven’t even been written up as a paper (they were announced in a press release), and researchers say they are still testing more people.
Questions started to emerge about the conclusions of both studies shortly after they were posted.
Addendum: Folks commenting that people self-selected into the survey because they had been sick and wanted confirmation. Agreed, this is what I meant by “consent bias.” This is expected w/ volunteer studies. How to adjust/avoid it? Household samples are great but take longer.
— Natalie E. Dean, PhD (@nataliexdean) April 18, 2020
For one, they didn’t follow best practices for study design. In the Santa Clara study, researchers acknowledged these limitations. They recruited their participants through a Facebook ad, so it wasn’t a random sample of the population, and they tested people at a drive-through site, so participants had to have access to a car. The sample overrepresented younger white women and underrepresented Hispanic and Asian people. The study also didn’t control for age. “The overall effects of such biases is hard to ascertain,” authors wrote.
As a result, they likely sampled people who were motivated to be tested, oftentimes people who thought they may have been infected. That may have boosted the prevalence rate of antibodies in the sample.
The Los Angeles study recruited participants through a marketing firm from a database that is representative of the population.
The tests themselves also pose a problem. Right now, there are only four serology tests for Covid-19 that have been approved by the Food and Drug Administration, and even those were only under emergency use guidance (which means they didn’t go through the full validation process).
The antibody test used in these studies, from a company called Premier Biotech, was not FDA-approved. Researchers in the Santa Clara study reported it to be 80.3 percent sensitive to SARS-CoV-2 antibodies and 99.5 percent specific to them.
The problem is that even seemingly accurate tests don’t do well when the thing they are looking for is scarce. Right now, there’s still only a tiny fraction of the population that has ever been infected with SARS-CoV-2, so false positives end up as a disproportionately large share of the total number of positive results.
“So if you have 1 percent of your population infected and you have a test that’s only 99 percent specific, that means that when you find a positive, 50 percent of the time will be a real positive and 50 percent of the time it won’t be,” said Deborah Birx, the White House Covid-19 response coordinator, at a press conference on April 20.
6/7 In a low risk population, false positives quickly outnumber true positives that the results are very difficult to interpret. Even with a positive test, you're more likely than not to be negative. This all has implications for the use of antibody tests for "immune passports."
— Natalie E. Dean, PhD (@nataliexdean) April 23, 2020
Other researchers said they found problems in the calculations conducted in the study.
I have been corresponding with the authors of the well-known Santa Clara County COVID-19 preprint, and I am alarmed at their sloppy behavior. The confidence interval calculation in their preprint made demonstrable math errors - *not* just questionable methodological choices.
— Will Fithian (@wfithian) April 21, 2020
Andrew Gelman, a statistics and political science professor at Columbia University, criticized the findings on his blog. “I think the authors of the above-linked paper owe us all an apology,” he wrote. “We wasted time and effort discussing this paper whose main selling point was some numbers that were essentially the product of a statistical error.”
The estimates reported in these surveys don’t line up with real-world data on deaths
Based on these studies’ conclusions, the deadliness of Covid-19 should be way lower than has been actually seen. The low-end estimates of the infection fatality rate — that is, the percentage of people who were infected who later died from the virus — were barely above the percentage of the total population killed by the disease in Covid-19 hot spots. (The infection fatality rate is distinct from the case fatality rate. The latter focuses on fatalities based on reported cases, whereas the former is based on all infections, including those that resulted in no symptoms and went unreported.)
The first study would put the Santa Clara infection fatality rate for Covid-19 between 0.12 percent and 0.2 percent. For the Los Angeles study, that would put the infection fatality rate roughly between 0.13 percent and 0.27 percent. By comparison, the World Health Organization estimates seasonal influenza has an infection mortality rate “well below” 0.1 percent.
The estimates from these serology surveys don’t comport with the actual results seen in Covid-19 hot spots like Italy and New York.
In Lombardy, Italy, for instance, more than 12,500 people have been killed out of a population of 10 million, meaning 0.12 percent of the total number of people there have died of Covid-19. If one were to limit the denominator to just the number of people infected, the fatality rate would be much higher.
Similarly, New York City has already seen 0.12 percent of its total population die due to Covid-19. It stands to reason the infection fatality rate is much higher, since only a fraction of the population has been infected.
Across the board, one would expect the infection fatality rate from Covid-19 to be higher than the population fatality rate, so the estimates based on the California studies seem suspiciously low.
Why we need to be careful about using hasty results
It would be handy to know how many people have had this coronavirus and might have some immunity to it — we’re in a unique moment where huge public health decisions need to be made swiftly about when to lift social distancing measures. So it’s important to make sure the studies on these critical topics are robust.
The studies didn’t lay out the implications for the response to Covid-19. But one of the authors, Stanford University epidemiologist John Ioannidis, has previously suggested that a lack of information about the Covid-19 pandemic may be causing officials to overreact with “draconian countermeasures” if the fatality rate is much lower than previously thought.
“It’s like an elephant being attacked by a house cat,” Ioannidis wrote in a March 17 editorial for Stat News. “Frustrated and trying to avoid the cat, the elephant accidentally jumps off a cliff and dies.”
But if they’re wrong, people’s health and lives are at risk. And they might very well be wrong.
Even if the coronavirus’s infection fatality rate is in fact this low, that will hardly matter if social distancing restrictions end too soon, because the fatality rate is also related to access to health care. With more person-to-person contact, the number of infections would grow, sending more people to hospitals. If there aren’t enough beds and equipment for this surge, the fatality rate will rise as a share of the number of people infected.
And the debate over the specific value of the infection fatality rate is tangential for health workers on the ground who are on the front lines of this pandemic. Covid-19 is clearly dangerous and deadly. Already some of the top health care systems — in Northern Italy and New York City — have been overwhelmed by Covid-19 patients. Yet the vast majority of people, even in pandemic hot spots, remain vulnerable to the virus.
Whether seroprevalence is 2% or 3% is mostly of academic interest. What’s clear is that the notion that we all got infected and didn’t know it and now we are near herd immunity is completely unrealistic. Our communities are still almost completely susceptible. 2/
— Caitlin Rivers, PhD (@cmyeaton) April 22, 2020
To be clear: The conclusions of these surveys could be pointing in the right direction. Covid-19 could be way more prevalent and far less deadly than anyone has realized. But the evidence presented by the Santa Clara and Los Angeles studies isn’t enough to make a decision about relaxing social distancing or other pandemic control measures.
Serological surveys are critical to the public health response to Covid-19. However, weak studies do little to satiate the desperate need for information about the pandemic. Assessing the spread of the virus within a population is an essential step, but it needs much more rigorous research to yield anything worthwhile.
Such studies could be strengthened by sampling larger numbers of people within a population and truly randomizing who is tested. The problem of false positives and negatives can be countered by repeated testing of participants.
Conducting a study like this would be more time-consuming and expensive. And one study alone won’t cut it; multiple serological surveys are needed around the world to illuminate the true extent of the disease.
To this end, the National Institutes of Health and the Centers for Disease Control and Prevention have their own large serological surveys underway for Covid-19. NIH is recruiting up to 10,000 volunteers across the US. The CDC is also conducting a national survey, as well as targeting Covid-19 hot spots and people at higher risk of exposure to the virus.
There is no definite timeline for when these studies will wrap, but they stand to yield a sharper picture of the spread of the new coronavirus. And from there, health officials can start to focus on how to let things go back to normal.
