Why Observational Studies shouldn’t be used to assess Respiratory Virus Interventions - Part 2
The Long Read
This post was first published on 10 February 2023
We have seen how past observational studies of the SARS-CoV-1 epidemic gave implausible effect estimates while randomised trials gave more conservative estimates. However, there are other reasons specific to respiratory viral transmission why observational designs should be used cautiously for informing policy.
Let’s try and put these in the context of the UK Health Security Agency (UKHSA) review provided by Lord Markham as the basis for government living with covid advice that includes ‘wearing a face covering in crowded and enclosed spaces.’
The review consists of 25 studies, two of which are trials included in the 2023 Cochrane review. Here, we set out a series of issues on the type and quality of evidence required to assess the effectiveness of respiratory virus interventions.
Use the right study design to answer the question.
Previous reviews by the UKHSA included ecological, descriptive, and laboratory studies. In the latest version, these are excluded, and the review reports focuses ‘on higher-level evidence such as interventional studies, cohort studies, case-control studies and cross-sectional studies.’
Most of the evidence is rated as low quality (18 studies), with only one rated as high-quality. However, it is not just that the review includes evidence at high risk of bias, which may undermine the conclusions. At times, it includes the wrong evidence to inform whether the intervention works.
As an example, the review included seven cross-sectional surveys. While this design can be useful for assessing the prevalence of mask-wearing or respiratory symptoms, it is not acceptable for assessing effectiveness. This is the case for studies that lack a comparator: a point known for some time but largely ignored.
Induction, the method of Francis Bacon, is based on comparison and cumulative knowledge. When that is accumulated, you can draw inferences, so long as you remember that the inference can never be 100% accurate. Furthermore, in their Lancet overview of clinical research, Grimes and Schulz point out that ‘descriptive studies do not have a comparison group. Thus, in this type of study, investigators cannot examine associations, a fact often forgotten or ignored.’
A total of ten studies had no comparator. Because they were inappropriate designs to study associations, they should be excluded.
The included studies did not have a prior protocol
The UKHSA included 23 observational studies that do not appear to have protocols. This indicates haste in getting studies “out there” in journals while the going was good - when covid was daily headline news. But it also means that crucial details, such as detailed case definitions, are omitted, and “PCR positivity” is considered equivalent to contagiousness, something we know is nonsense.
‘Every clinical trial should be based on a protocol—a document that details the study rationale, proposed methods, organisation, and ethical considerations. Trial investigators and staff use protocols to document plans for study conduct at all stages, from participant recruitment to results dissemination. Funding agencies, research ethics committees/institutional review boards, regulatory agencies, medical journals, systematic reviewers, and other groups rely on protocols to appraise the conduct and reporting of clinical trials.’
SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials.
All observational studies in the UKHSA review failed to provide details of the PCR test used and how the authors decided a positive result was an active case. In other words, they were describing hot air. They had failed to carry out the most basic of checks: prove that a “case of Covid” was such: an active case capable of infecting third parties.
Here is what the UK’s Office for National Statistics study, considered the one high-quality study, describes as a positive case:
"Samples were called positive if either N or ORF1ab, or both, were detected. The S gene alone was not considered a reliable positive but could accompany other genes (that is, one, two or three gene positives)."
This is despite the ONS knowing that identifying single genes by PCR is not equivalent to an active or contagious case. To be infectious, virions need to be whole and in sufficient concentrations. Finding single genes in unknown concentrations is meaningless. Therefore, the ONS study's assessment of the association is also rendered meaningless.
With a peer-reviewed protocol, mistakes would be identified or at least caveats inserted in the interpretation. All protocols of randomised trials should be registered in open registers and assigned a unique number.
Don't take our word for it; take the NHS Health Research Authority’s word, which insists on it.‘“
All research should be registered in a publicly accessible database. For clinical trials, it is a condition of a favourable ethics opinion. It is good practice for all other studies.’
In studies, wiggle room for doing what you want should be virtually nil unless you have some good reason that you also need to justify publicly. Without a protocol, it’s impossible to assess the study changes that lead to the final published results.
This does not account for fluctuations in transmission.
Without a protocol, the time frame for observation is left up to researchers. You can shift the window of analysis to a time window that is the most favourable to the intervention under study, and no one would know.
This is critical to the result you obtain. If you place the analysis of your intervention at the epidemic peak, when positivity is at its height or just about to subside, and ensure follow-up isn’t too long, then a spurious cause-and-effect relationship will occur.
This is visible in the consistent sinusoidal trend previously observed in the results of the surveillance of SARS-CoV-2 data. Shift your observation window, and you can obtain the “right answer”, whether this is intentional or not. Eventually, there will be an upturn in the number of infections; therefore, if the length of follow-up is too short (as many studies in the review are), you will likely arrive at the wrong conclusion.
Source: Weekly National Influenza and COVID-19 Surveillance Report, Week 5.
Not accounting for multiple co-interventions
When you have little control over the data fed into a study, the relationship of each variable is difficult to assess and, in some cases, impossible. This is often the case with retrospective data.
Perhaps the worst example is in Manny et al., one of the cross-sectional studies, in which parents were asked to remember their child’s face covering use up to 9 months prior. - Some of us can’t remember what we were doing yesterday, but as an example of the problem with recall, can you remember what you did last May?
In a survey conducted at US Summer camps, one participant responded for each camp, or in some instances, multiple camps. Multiple varied interventions, or not, could have been undertaken that were not accounted for.
In the UKHSA review, it is unclear what the intervention entails, what it is combined with, and who is responsible for each component. Many studies have “bolt-on” parts, meaning the heterogeneity between studies is very high. That tells you that you are comparing apples with pears if, indeed, the design was comparative.
This latitude, this freedom of doing what you want and analysing things the way you want, is denied in randomised designs that follow protocols. The lack of effect of interventions observed in the Cochrane review is striking as it occurs across various settings and against different agents, known or unknown. In the Cochrane review, heterogeneity is low, indicating an appropriate comparison of apples with apples.
Certainty is distorting the study design and the chosen intervention.
Modes of transmission of respiratory viruses are 100 per cent clear only to ideologues. Those who have studied transmission know that the evidence in favour of one particular mode is never overwhelming, as respiratory viruses are capricious and difficult to pin down and likely transmit in many different ways, as the experiments in the 1960s have shown.
If you are certain that transmission is by aerosol, you will assess or favour the most draconian face barriers. Imagine the surprise when randomised evidence suggests no difference, and transmission takes place regardless. By then, you have committed yourself to “proving” something (something no real scientist would do), and the results may not be what you want. That is why there should be no sides in scientific disputes, only the advancement of knowledge.
Supporters of observational studies make a big deal of the lack of external validity of randomised designs. External validity is the extent to which you can generalise study results to other situations and settings. What you get instead is a focus on laboratory experiments, but these lack credibility because of the artificial setting, which lowers the external validity craved.
Randomised designs are also criticised for the lack of full compliance with mask-wearing or routine hand washing - If only everyone wore their ‘bloody masks or washed their hands.’
However, compliance is an issue for all interventions. People prescribed self-administered medications typically take less than half the prescribed doses. What matters is the real-world effects based on what people actually do, not what you'd like them to do. A protocol is therefore essential because it sets out the specific nature of the intervention - what was done - which importantly allows replication of the intervention.
If you do not know what was done to whom, how do you know what to do when setting an effective policy?
Blinding of the data
Blinding, also known as masking, is the ignorance of whether you are looking at exposed or unexposed data. It is essential for minimising bias, particularly in a case-control study.
Consider a case-control study that examines the prevalence of wearing a specific type of face mask among individuals who tested positive for SARS-CoV-2 versus those who tested negative. A researcher analysing the data should not know whether the dataset they are looking at comes from those who tested positive (cases) or not (controls), and whether they used or did not use the intervention being tested. If they are in the know, especially in a polarised world and with government sponsorship, this risks introducing bias in the analysis and their conclusions. None of the case controls cited in our two posts mentioned analyst blinding, so we can assume none were blinded.
The primacy of the study question
What dictates the choice of study design is not dogma or deeply held beliefs: it is the study question.
To answer with a high degree of probability (note, not certainty) the question of whether physical interventions delay or stop transmission of respiratory viruses, the correct design is a randomised controlled trial, as the design minimises the effects of bias and helps exclude alternative explanations for the findings.
Observational studies do not do that: if the study question is, what are the long-term (defined as three years or more) harms of a handwashing routine, then a randomised trial is not the right answer. You will need a cohort or case-control study.
This holds true for the types of studies you will include in a systematic review. In the first case, you will privilege RCTs, and in the second, observational studies. It is common sense that when there is a higher level of evidence from studies of appropriate designs, you will privilege their inclusion in a review compared to those of lower quality, as happened in the Cochrane review over time.
However, if you are investigating viral transmission, especially during a pandemic, a small cohort or a case series is realistically the best you can hope for, which is precisely what we have included in our corpus of evidence on SARS-CoV-2 transmission. Not a whiff of an RCT.
The importance of an accurate history
Acute Respiratory Infections are fleeting and subtle. Symptoms can range from no symptoms all the way to pneumonia and respiratory failure. However, the vast majority present with well-known symptoms such as sore throat and malaise, lasting hours or a few days. Observational studies, especially if retrospective, without a protocol and using self-reported assessments, cannot be relied on to provide a truthful history.
We had direct experience with this when we did our systematic review of transmission of SARS-CoV-2 from pre- and asymptomatic individuals. Several of the studies we identified did not specify how symptoms were elicited, while the authors of others, when going back to the original participants, found that they did remember a “tickly throat” for a few hours. In one famous case, symptoms were concealed, which became clear only months afterwards.
In properly designed randomised trials, detailed administered questionnaires and planned follow-up make the adjudication of a case more accurate. In the absence of such methods and with self-reported outcomes after the event, it's often unclear what is being reported and whether it represents the truth.
The need for better evidence
Ultimately, good science is being destroyed by political expediency and ideology.
There can be no better proof of this than CDC’s Director Dr Rachelle Walensky's statement that trials would have been unethical because equipoise was no longer present, thanks to “so many studies that demonstrated time and time again in the height of Covid transmission that masks were working in preventing transmission.”
The studies Dr Walensky was referring to are likely to be the same studies included in the UKHSA review.
Effective Policy should be based on high-quality evidence. The mistakes of the pandemic and the price that now has to be paid have highlighted the problems with policies based on low-quality evidence, and even more so when the policy was based on the wrong evidence.
This post was written by two old geezers who spend a lot of time reading and discussing ineffective research.
Prof
In your breakdown of the NHS series - did you do one on Primary Care?
What GP services etc should look like?
as a GP I expect you have an excellent insight into this babe