A study reports that Covishield is 63% effective against Delta. The devil is in the details.
Representative: A health worker prepares a dose of AstraZeneca COVID-19 vaccine at a clinic in Bangkok, June 21, 2021. Photo: Reuters / Athit Perawongmetha
A preprint paper a new study, published on July 16, 2021, by a team of scientists led by Dr Pramod Kumar Garg of the Translational Health Science and Technology Institute, Faridabad, reported that two doses of the Covishield COVID-19 vaccine were 63% effective against the delta variant of the new coronavirus.
What was this study and how reliable are the results? Let’s find out.
India’s COVID-19 vaccination campaign heavily relies on Covishield, the vaccine manufactured by the Serum Institute of India, Pune, under license from the Anglo-Swedish pharmaceutical company AstraZeneca. Covishield and AstraZeneca vaccine are the same recombinant vaccine, named ChAdOx-1.
So far, we have taken at face value the claim that the results of major foreign clinical trials (such as those published in december 2020 and February 2021) using the ChAdOx-1 vaccine also apply to Covishield. Technically, there’s no reason they can’t – but as the virus evolves, it’s prudent to continue to make sure the two vaccines are equivalent, as well as to measure their effectiveness against them. new variants.
The July 16 preprinted document from a new Indian study is a welcome development on this front. This is a complex document because it describes two distinct research elements. This article focuses only on the case-control study that reports the efficacy of the vaccine in the field. The same article also describes the action of antibodies generated by Covishield against the delta variant. But this article will not comment on that part of the study, to keep it simple.
How vaccines are tested
Phase 3 vaccine trials are randomized controlled trials (RCTs) – the gold standard for clinical evaluation techniques, and based on data from such a trial, they calculate the effectiveness of a vaccine. Dr. Garg’s study, however, was not an RCT, but a “negative test case-control study”.
In an RCT, researchers start with participants who did not have COVID-19, divide them into two groups at random. One group receives the vaccine, the other receives a placebo. Researchers then follow up with both groups, recording the number of COVID-19 infections that occur in each group.
Because participants were assigned to the placebo and vaccine groups randomly, the differences in the “attack rate” of the virus between the two groups can be attributed to the vaccine. If the attack rate in the vaccinated group is, say, 20% of the attack rate in the placebo group, the vaccine can be said to have prevented 80% of the infections that would have occurred without the vaccine. The 20% in this example is the ratio of the risk of contracting COVID-19 between the two groups – and the 80% is the effectiveness of the vaccine.
RCTs of this type are expensive, time consuming, and once vaccines are approved – even if only for emergency use – are difficult to repeat. This is where case-control studies can help.
Here, researchers start with a group of people who are grouped into “cases” and “non-cases”, or “controls.” They then interview the subjects in each group to determine if they have received the vaccine in the past, in order to get an idea of the vaccine’s effectiveness.
For example, let’s say we had 1,000 people with COVID-19, 50 of whom had received the vaccine, leaving 950 unvaccinated. Thus, the probability that a person in the “case” group received the vaccine is 50 in 950, or 1 in 19. The chances that people in the “case” and “control” groups will be vaccinated are compared to obtain the vaccine. odds ratio (OR).
Dr Garg and his team studied people who had been tested for COVID-19 with the RT-PCR test at one of two centers in Faridabad, Haryana, between April 1 and May 31, 2021. During this During this period, a second wave of COVID-19 was in full swing in India and the pace of vaccinations was only accelerating.
Dr. Garg’s team selected 3,687 people who tested positive on RT-PCR tests and 5,163 people tested negative. After leaving out those who did not respond and those who had to be excluded for other reasons, the two groups fell to 2,766 positives and 2,377 negatives.
Then, the team went back in time1 and asked the participants if they had received the vaccine and, if so, how many doses they had received.
This is the real attraction of the case-control method: since we start from the result – “case” or “non-case” – and go back in time, it offers a relatively faster means of assess the effects of the vaccine on a population. Such assessments could proceed even faster if all of this information was readily available in a clinical database.
The data from the study are summarized in the table below. The figures are taken from Table 2 of the document.
|‘Case’ (RT-PCR + ve)||‘Controls’ (RT-PCR -ve)|
|Vaccinated – two doses||85||168|
|Chances of vaccination||85/2294 = 0.0370||168/1813 = 0.0926|
|Odds ratio||0.0370 / 0.0926 = 0.3995|
Note that 0.3995 is this author’s estimate of the odds ratio. In the paper, the researchers presented an adjusted odds ratio – taking into account differences in risk of exposure to age, gender and COVID-19, although they did not provide details on how they arrived at their final figure of 0.37. But it is not far from 0.3995.
Interpretation of numbers
It is difficult to intuitively understand the odds ratio. Mathematically speaking, it is do not the same as the relative risk or the risk ratio calculated on the basis of a prospective clinical trial. Instead, and generally speaking, it measures the strength of a causal link between exposure the protective effects of the vaccine and the result, who in this case “catches COVID-19”.
When an outcome is infrequent or rare, then the OR becomes a reasonable approximation of relative risk. The OR here is less than 1, which means that the vaccine has a protective effect.
The efficacy of a vaccine is 1 minus the relative risk, and its efficacy is 1 minus the OR. On this basis, Dr Garg and his team arrived at an estimated efficacy of 0.63 – or 63% – for two doses of Covishield.
Now first: The researchers say their study examines the effects of Covishield against delta variant infections based on sequencing only 150 samples from the cohort of 2,766 participants (“case” group). Of these, 121 (80.7%; 95% CI 73-87) samples contained the delta variant or variants of its lineage. 17 others could not be attributed to any lineage. Despite the paucity of genome sequencing activity in India, it is not unreasonable to conclude that most of 2,766 people in this study had COVID-19 caused Probably by the delta variant.
Second: researchers presented data on mild and severe COVID-19. Of the 84 participants with severe illness, only one had been vaccinated. However, they contrast these odds, 1 in 83, not with the “controls” but with mild cases (with the odds of being vaccinated 1 in 26), to arrive at an OR of 0.19.
This translates into an estimated efficiency of 81.5%. But due to the small number of people with severe COVID-19, this estimate comes with a very wide 95% confidence interval 2 – from 9.9% to 99%, which makes it totally unreliable! In all cases, the OR for critical illness should have been calculated relative to the odds in the control group. This gives an OR of 0.13, with a similar giant confidence interval of 0.018 to 0.94.
Third: In any case-control study, the greatest source of bias lies in the identification of controls. Here the authors used a standard WHO Excel Tool to calculate the ideal sample size, based on standard assumptions, of 2,237 “cases” and an equal number of “controls”. In the final analysis, the authors included 2,766 “cases” and 2,377 “controls” – both comparable to the estimated ideals.
However, that is not the number they started with. Remember that they contacted a total of 8,850 people.
|‘Case’ (RT-PCR + ve)||‘Controls’ (RT-PCR -ve)|
|Included in the final analysis||2,766||2,377|
The document itself presents this information in a somewhat confusing manner.
It is worrying that non-respondents and exclusions were 25% among the “cases” originally contacted and double that number – 54% – among the “witnesses”. The researchers write in their article that the exclusions occurred for “reasons mentioned in the supplemental material,” but this author was unable to locate such material on the preprint repository website.
Whenever a large portion of participants enroll but drop out before the final analysis, there is reasonable doubt as to the integrity and conclusions of a study. We have to assume that the participants who dropped out or were excluded are similar to those who were included in the final analysis, so that the conclusions remain valid. But in this case, the loss of follow-up was almost twice as great in the “control” group than in the “case” group. This means that we have to take a leap of faith to assume that this unbalanced loss rate did not affect the direction and size of the estimated OR.
To compare, consider the May 2021 study by Public Health England. Here, vaccination data was available for 90% of the nearly 175,000 people who underwent RT-PCR tests. The researchers also had data on 44,950 people who tested positive (“cases”) and 112,340 people tested negative (“controls”). This study was carried out entirely by linking the files of the participants through databases of RT-PCR tests and vaccinations. India does not have such systematic and comprehensive health related databases. However, the researchers did some good case-control studies to our advantage.
To repeat, case-control studies are useful when a disease or an outcome of interest is rare. They offer useful information on potential causal links between important factors such as vaccination and disease risk. But the key to these studies is to correctly select the “controls” and ensure that the data is complete. When he loses half of the “checks” he started with, the reliability of the conclusions becomes questionable.
Dr Jammi Nagaraj Rao is a public health physician, independent researcher and epidemiologist in the United Kingdom.