Why unreliable tests are flooding the coronavirus conversation

As disagreements pile up over easing COVID-19 restrictions and reopening businesses, scientists are racing to gauge how many people have truly been infected. Among other reasons, measuring the pandemic’s spread can clarify how deadly the disease is and how many people are presumably immune to it, which is crucial for anticipating future waves of the outbreak.

The cornerstone of this process is antibody testing, which looks for proteins made by the body’s immune system to combat invading pathogens. Antibodies can identify cases where people have caught COVID-19 even if they didn’t show any symptoms. Early antibody results suggest a substantial portion of cases have been missed by genetic tests for COVID-19, the first line of diagnostics for the disease.

The reported numbers vary widely—21 percent in hard-hit New York City versus roughly 6 percent in mostly spared Geneva, Switzerland—which in part is expected. But some of the differences are a consequence of rushing to publish results, and in some cases, troubles with the tests themselves.

Experts say that COVID-19 testing has now entered a “Wild West” phase. Inaccurate, unauthorized tests for the coronavirus are flooding the market due emergency deregulation of oversight. Compounding that, flaws in collecting data and analyzing results can produce misleading estimates—many of which would normally be caught by the peer-review process, a safeguard built into academic publishing. But in the mad dash to make sense of this pandemic, many research teams are instead leaning on press releases or unreviewed articles called preprints to spread the word.

“It’s very difficult to really figure out what’s going on,” says Samuel Scarpino, who heads Northeastern University’s Emerging Epidemics Lab. “What we’re facing here are some numbers that are being reported without the context necessary to interpret them.”

One notable example is a recent survey for COVID-19 antibodies among residents of Santa Clara County, California. On April 17, Stanford University researchers reported in a preprint that, of the 3,300 people they’d screened, 1.5 percent were positive. Ultimately, the team claimed that the county actually had 48,000 to 81,000 cases of COVID-19, which seemed implausibly high given only 50 deaths and about 1,500 cases were recorded at the time.

This funky math has been widely criticized, and because the study entered the public sphere without peer-review, it launched a controversial debate over whether COVID-19 is less deadly than epidemiologists have said. Since then, the Stanford team has updated its report, dropping its estimate to a range seen by other studies.

In general, this first wave of antibody testing “ends up being a balance between a quick and dirty approach, and something that’s more rigorous and representative, but will take longer,” says Natalie Dean, a biostatistician at the University of Florida who has spotted problems among other antibody surveys. “So we’re seeing the fast ones coming out the earliest.”

A survey’s results are fundamentally limited by the quality of the test used to detect COVID-19.

The problem is, many tests on the market are demonstrably not up to snuff. In the U.K., officials paid $20 million for antibody testing kits from China that didn’t work. The Spanish government had to trash 640,000 antibody testing kits that ended up having an accurate detection rate of just 30 percent. And a recent evaluation of 12 widely used antibody test kits, which declared upfront its funding by Anthem Blue Cross Blue Shield, the Chan Zuckerberg Biohub, and others, concluded that only two had reported accurate results 99 percent of the time—the benchmark for reliability set by the U.S. Food and Drug Administration on May 4.

Prior to this date, the U.S. was flooded with more than 150 unvalidated tests because the FDA had allowed companies to sell these products without seeking Emergency Use Authorization. The FDA shifted its policy after Congress opened an investigation into this runaway testing market.

“Testing is important, but how it’s done is really concerning to me,” says Michael Osterholm, an epidemiologist and director of the University of Minnesota’s Center for Infectious Disease Research and Policy. “People who are saying they’re going to use antibody testing as a way to reopen—that’s kind of like the person that looks down the barrel of a gun and pulls the trigger to see if it works.”

Osterholm worries that mediocre diagnostic tests will miss true COVID-19 infections, delivering what’s called a “false negative” and potentially prompting an infectious person to unknowingly spread the disease. As well, he says, faulty tests detecting COVID-19 antibodies, which should only be present if a person has been exposed to the virus, might deliver a “false positive” result, in which unrelated antibodies are flagged in a person who hasn’t yet been infected. If cities or states fall for shoddy tests with unreliable, misleading results, Osterholm says, it could endanger good decision-making on both personal and governmental levels.

“It’s the Wild West out there,” he says. “The FDA could do their job, and they should do their job. It’s that simple: Do your damn job.”

Even when the tests are reliable and accurate, it’s crucial to gather representative participants. In theory, this is easy—but in practice, it’s almost impossible to eliminate bias without testing 100 percent of a population.

For example, screening folks at a drive-through testing center preferentially captures data from the portion of a population with cars. The same goes for randomly dialing phone numbers and asking people to show up for testing at a local hospital—an approach that relies on transportation, or the ability to leave work. And in New York City, where scientists are randomly selecting people for testing in grocery stores, they are surveying a percentage of the population that’s at higher risk of infection simply by being out and about.

Although some experts questioned the tests used by the Santa Clara study, the team drew stronger criticism because of a sample population that skewed white, wealthy, and female. One argument suggests that this demographic might have a lower incidence of COVID-19, given how the disease hits harder among impoverished communities.

Highlighting the study’s confusing design, a contradictory argument notes that participants volunteered for the study based on a Facebook ad, and critics argue that such strategies are more likely to attract folks who already thought they’d been sick—potentially leading to a higher percentage of people with COVID-19 antibodies.

What’s more, the team did not disclose all their funders in the preprint version of the study, which would have normally been required under peer review. One of the funders is the founder of JetBlue Airways, who has voiced a strong desire to reopen the economy.

The team tried to correct for their biased sample, but their initial statistical analyses were flawed, says Andrew Gelman, a statistician at Columbia University who wasn’t involved with the study.

“They made some mistakes, and it was annoying to me, and I think it annoyed other people too,” Gelman says, noting that he probably went overboard in his public criticism. “I don’t think there are bad guys in this system, it’s just difficult.”

“I would do a much more careful job selecting a sample … and get something that’s as close to representative of our county as I possibly could,” he says. “We’ve done a host of exercises to try to understand the direction of the bias.”

Still, the Stanford group’s takeaway—that 1.5 to 2.8 percent of Santa Clara’s population had COVID-19 antibodies—is pretty much in line with surveys done where infections aren’t ballooning.

“It all seems to be giving a consistent message, which is that unless the area has been pretty hard hit, we’re seeing numbers in the single digits,” Dean says.

Recently, she notes, Swedish scientists retracted a study claiming that 11 percent of the Stockholm population had been exposed to COVID-19. That team tested samples at a blood bank for antibodies—and then discovered that their sample contained blood from donors who knew they’d been infected and were hoping their antibodies could be therapeutic.