Why Observational Studies shouldn't be used to Assess Respiratory Virus Interventions - Part 1

Why use poor evidence when better quality evidence is available?

This post was first published on 10 February 2023.

When we first wrote the protocol of the Cochrane review “Physical interventions to interrupt or reduce the spread of respiratory viruses” in 2006, our criteria for the types of studies to be included were as follows: “We will consider individual-level or cluster randomised or quasi-randomised controlled trials. To enhance public health relevance, we will also consider observational studies (cohort and case-control designs), and any other comparative design, provided some attempt has been made to control for confounding.“ 

Published in 2007, the first version included 14 RCTs. This paucity of randomised evidence meant the first three versions included non-randomised observational evidence, provided they compared whatever intervention they were assessed with, either standard practice or another intervention. The lack of a comparator would have meant that no inference could be drawn from the study.

So, for example, in the 2010 version, we included seven case-control studies (all carried out during or after the SARS-1 epidemic), and we reported: 

• Studies reported that disinfection of living quarters was highly effective in preventing the spread of SARS (OR 0.30, 95% CI 0.23 to 0.39); 

• Handwashing for a minimum of 11 times daily prevented most cases (OR 0.45, 95% CI 0.36 to 0.57), based on all six studies; 

• Simple mask wearing was highly effective (OR 0.32, 95% CI 0.25 to 0.40), based on five studies; 

• Two studies found N95 respirator wearing even more effective (OR 0.09, 95% CI 0.03 to 0.30); glove wearing was effective (OR 0.43, 95% CI 0.29 to 0.65) 

• Gown wearing was also effective (OR 0.23, 95% CI .14 to 0.37); 

• All means combined (handwashing, masks, gloves, and gowns) achieved very high effectiveness (OR 0.09, 95% CI 0.02 to 0.35).

The term OR stands for the odds ratio. It is a measure of the strength of association (in this case) between, say, handwashing and mask use and SARS-CoV-1 positivity. 

An OR of 0.09 means that if you use all means combined, you will have a 91% decreased risk of infection. The same sort of effect sizes were reported in the pandemic. For example, the Lancet reported a systematic review on physical distancing that reported that virus transmission with physical distancing of 1 m or more was less by 82% (OR =0.18) compared with a distance of less than 1 m.

If you believe these results, virus transmission should have been halted within days.

Case-control studies are retrospective by design. Because of this and their observational nature, they are vulnerable to several biases, which, to some extent, can be controlled for if they are known, but cannot if they are unknown. This is vital to understanding the importance of randomisation, which reduces the potential for confounding by ensuring comparable groups for known and unknown confounding variables. 

The results of this early review were deceptive: they produced results that were too good to be true. A large body of literature shows that observational studies, compared to randomised trials, produce distorted results because of the play of confounders. 

The bottom line is that no observational study can exclude alternative explanations for the results observed. By the time we came to the 2020 update, there was much more randomised evidence (67 trials) to base our decisions on - analysing weaker designs made no sense. 

As expected, randomised designs provided more conservative results across all interventions (there was no change in the results from previous review versions for these randomised designs).

However, all versions and their content are still available, so the evolution of our thinking can be followed. Our concentration on higher-quality evidence, which appears to be the sensible thing to do, attracted some offensive comments from people who probably don't know much about the history of the review and the evolution of our thinking.

Dr. Hugh Gnomy 😷💉 or 🤮☠️@HughGnomy

@MaryanneDemasi Something tells me Tom Jefferson and Dr. Mengele would've been good friends.

5:18 AM · Feb 7, 2023

John Kriby@johnkriby1

@MaryanneDemasi And in 2011, before he became an anti-mitigation activist, Jefferson's Cochrane review favored all the low-cost measures reviewed, which included masks. Funny that! cochranelibrary.com/cdsr/doi/10.10…

9:31 PM · Feb 6, 2023

As we will explain in the next post, respiratory virus circulation and clinical manifestations pose specific problems for designing studies that provide a fair assessment of physical interventions.

In the midst of this complex epidemiological context, we will examine the evidence the UK Government used to formulate and maintain its policy on mask use. Lord Leicester asked the question, and Lord Markham replied, indicating a rapid review carried out by the UKHSA as the source of knowledge. We shall examine this in our next post.

Two old geezers wrote this post, having read far too much poor-quality evidence.