During the pandemic, Covid-19 tests have provided a rich source of media coverage. Most of us now know a bit about how these tests work, and that they can generate errors that lead to wrong and harmful decisions. Tests have to be used on the right samples at the right time, else more errors can be made, and there are important differences between “have I got it?” viral swab tests, and “have I had it?” antibody blood tests.
Despite this, the government keeps getting testing – particularly antibody tests – wrong. In March, the UK bought two million unproven finger-prick tests from China, which Boris Johnson described as having “the potential to be a total game-changer”. In April, an Oxford study reported that the tests were too inaccurate for public use. In May, a new laboratory antibody test manufactured by Roche was widely reported as “100% accurate” but Public Health England showed that the test failed to identify 16% of samples from people who had Covid-19. By then, the government had signed a contract to make the Roche test initially available to NHS and care workers. And in June, many public sales of laboratory antibody tests were stopped after the Medicines and Healthcare products Regulatory Agency (MHRA) found they were using finger-prick samples rather than the larger venous blood samples that were required.
So why do the diagnostics industry, the media and government keep falling for these tests? First, there are the obvious reasons: politicians and journalists love headlines that boost morale and give the impression of activity and progress. Covid-19 tests are particularly prone to misleading coverage because the regulatory process for tests is weak, inadequate and largely based on self-declaration rather than scientific scrutiny. Tests can obtain official CE-IVD approval and go to market after only limited laboratory studies, which may be wrongly used to suggest the tests are accurate in clinical settings. And not all companies appear to have the knowledge, or the will, to undertake rigorous clinical evaluations.
The latest news of a “game-changing” home antibody test arrived at the end of July. According to reports, “secret human trials” from the government-backed UK Rapid Test Consortium (UK-RTC) showed the test, manufactured by Abingdon Health, was “99.40% accurate”. The Oxford Regius professor of medicine, Sir John Bell, (who was involved in creating the consortium) described the test as “truly amazing”, and the government promised to purchase millions to send to the UK public free of charge. For the first time, a Covid-19 antibody test appeared to have met the MHRA target of less than 2% false negatives and false positives.
With this new test, will the run of monthly “game-changing” stories about antibody tests come to an end? It seems unlikely. Despite optimistic coverage, the results from the Abingdon test seem too good to be true. When it comes to evaluating Covid-19 antibody tests, a golden rule has been broken. Tests must be evaluated in individuals who represent the people who will use them in the future. For a “have I had it?” test, this means people known to have had or not had Covid-19. People infected with the virus don’t all have the same antibody responses: some have high responses, others low responses and some have no response at all. A performance study will only give the right answer when it evaluates a test in a group of individuals with this diverse mixture of responses.
In the case of the Abingdon test, this doesn’t appear to have happened. The manufacturer’s website gives little information about the “secret studies” by the UK-RTC. To evaluate the test’s ability to detect Covid-19, an unstated number of people were recruited who had symptoms or a positive swab test for Covid-19. Crucially, they were then tested with a different laboratory antibody test, the EUROIMMUN Anti-SARS-CoV-2 ELISA (IgG), and anyone who didn’t show antibodies on that test was excluded, leaving 203 people. These exclusions are precisely the problem: the sample of 203 people doesn’t represent the diversity of people who will use the Abingdon test in the real world, but a sample who have already tested positive on a different antibody test.
In other words, this study of the Abingdon test looks as if it was biased from the start towards people with high antibody levels, which are easier to detect. This is because the EUROIMMUN test misses many Covid-19 cases – particularly among people with no or low antibody responses; indeed, PHE estimates it misses 28%. Working on the basis of this 28% figure, this would mean that around 79 true Covid-19 cases could have been excluded in the evaluation of the Abingdon test. These excluded people are more likely to be missed by other tests, including the Abingdon test, because all tests are more likely to fail when people have no or low antibodies. And it’s the people who are most likely to want an antibody test – those who stayed at home with mild infections – whose low antibody levels are most likely to be missed.
It’s impossible to know from this data whether UK-RTC has discovered a game-changing test. Further evaluations of the Abingdon test are apparently ongoing, but no details are available so we have no way of scrutinising them. Such secrecy destroys trust. Restoring scientific credibility will require urgent publication of the protocols and reports for these studies.
And if we’ve learned anything from this pantomime, it’s that the current regulations aren’t fit for purpose: they disregard the importance of robust evidence about clinical accuracy and fail to protect the public from the harm that can be caused by poor, inadequately evaluated tests.
•Jon Deeks is Professor of Biostatistics and leads the Biostatistics, Evidence Synthesis and Test Evaluation Research Group in the Institute of Applied Health Research, University of Birmingham
The Guardian
