Share Trust the Evidence

L’Antica Birreria Peroni, founded in 1906 in central Rome, is one of the city’s most iconic historic beer halls. Closely tied to the Peroni brewery, it became a gathering place for locals, workers, and visitors seeking simple, hearty food and quality beer.

Over time, it has preserved its traditional character rather than chasing modern trends. The ambience is warm and nostalgic, with dark wood interiors, vintage photographs, and a lively, communal feel. It’s often busy and a bit noisy, but that’s part of its charm—an authentic Roman atmosphere where conversation, history, and beer flow together naturally, and two old geezers can be found having an Ale.

Due to a lack of resources (a problem not experienced by Baroness Hallett and her apparatus), we took one look at a randomised controlled trial cited to support the now defunct Oxford AstraZeneca vaccine (now known as VaxZevria).

Share

We chose this because we have already painstakingly dissected the regulatory evidence for PfizerBioNTech’s Comirnaty in 46 posts and follow-ups. For once, we thought we would look at a publication (in this case, the illustrious Lancet) rather than bore our readers with endless regulatory bumf. “Truth at last, peer-reviewed gold dust!” We hear you scream.

Read more

A counterfactual model asks a simple but unanswerable question: what would have happened if things had been different? For example: “Had we locked down one, two or three weeks earlier, we would have saved X number of lives.” Such claims sound authoritative. In reality, they are built not on observation, but on layers of assumption.

Read more

The Covid Inquiry’s Module 4 report, with its sweeping claims about lives saved, has now reached that moment. And it is not the academics or officials raising the sharpest questions, but the TTE readers—GPs, analysts, and ordinary citizens—who have begun to pick apart the foundations.

Share

Take the central claim: that vaccination saved vast numbers of lives. As docross, a retired GP, points out, the Inquiry itself admits that “directly measuring” this impact is difficult because the counterfactual—what would have happened without vaccines—was never observed. Yet there is a comparator, however imperfect: a substantial unvaccinated population. In the UK, this may have been as high as 20–25 per cent. Why, then, are we not shown detailed mortality data stratified by vaccination status, adjusted for age and risk? Why, asks docross, is such data withheld—reportedly on grounds of “commercial sensitivity”?

This is not a trivial omission. In evidence-based medicine, when direct observation is possible, one does not leap immediately to modelling. Yet that is precisely what has happened. Instead of presenting clear, adjusted mortality outcomes, the Inquiry leans heavily on estimates derived from models that depend on assumptions about infection rates, vaccine effectiveness, and behaviour in a hypothetical world without vaccination. As Richard Boughton puts it succinctly: “assumptions all the way down.”

The problem is that the modelling is presented with a confidence that belies its fragility. Several readers independently arrive at the same concern: that the “lives saved” figures are constructed backwards from assumptions rather than derived from observed data. Helen Mcardle highlights a striking example from Scotland. To justify the claimed number of lives saved, the underlying model would imply a 2021 mortality rate almost double that in 2020—a scenario so implausible that it exposes the estimate’s sensitivity to its assumptions.

Others point to what is missing. The models, notes Bilbo Baggins, assume fixed infection fatality rates, no waning immunity, and sustained infection-blocking effects—assumptions that have not held consistently in the real world. They also fail to incorporate potential harms or adverse events, effectively modelling only one side of the ledger. The result, he argues, is not a balanced assessment but a “counterfactual fantasy”—a simulation that risks creating the illusion of certainty.

Even more basic questions remain unanswered. As Dan Newell puts it: Where are the observed deaths? If the aim is to estimate deaths avoided, surely the starting point is to show deaths that actually occurred, broken down transparently by risk group and vaccination status. Without this, the exercise begins to resemble what one reader calls “grade school math”—a conclusion presented without the working.

There is also a deeper unease about transparency. Several commenters note that data releases have become less complete over time. Stephen Feldman points to earlier reports that included age-standardised comparisons, which were later discontinued. Whether justified or not, such changes erode confidence. In public health, trust depends not only on the strength of conclusions but on the openness of the process that leads to them.

The point is that the Inquiry’s methods—as presented—do not allow us to judge the scale of any effect with confidence. Instead, we are asked to accept large, precise numbers derived from models whose assumptions are only lightly interrogated.

Perhaps the most telling comment comes from Jo, who asks a deceptively simple question: if the vaccines had such a dramatic impact, why is that not clearly visible in the mortality trends of highly vaccinated countries? It is the sort of question that should be addressed head-on, not sidestepped with ever more elaborate modelling.

What emerges from these reader critiques is not a coordinated campaign, but something more interesting: a collective insistence on basic standards of evidence. Show us the data, explain the assumptions and distinguish between what is observed and what is inferred. These are not unreasonable demands; they are the foundations of good science.

And when large claims rest on opaque methods and untested assumptions, confidence inevitably gives way to doubt. Not because the public is misinformed, but because it is asking the right questions.

The two old geezers who wrote this post would like to thank our readers for their comments.

The file INQ000474703_0023-0024 paras 4.36-4.39, which formed the Inquiries summary for Module 4 on lives saved, is available here. It was authored by Professor Daniel Prieto-Alhambra, who reports:

Share

Estimates published by Public Health England and Cambridge University’s MRC Biostatistics Unit suggest that 127,500 deaths had been prevented in England as a result of the Covid-19 vaccination programme up to 24 September 2021 (UKHSA, 2021a)

Here we look at the PHE & MRC’s Biostatistics Unit report (UKHSA, 2021a) on its evidence to support number of deaths averted: COVID-19 vaccine surveillance report: 30 September 2021 (week 39)

Read more

Here we now turn to the evidence from the Inquiry that ‘numerous studies have been conducted to understand the impact of COVID-19 vaccination on COVID-related and all-cause mortality.’

Share Trust the Evidence

So, how many is “numerous”: 5? 50? 500? And what kinds of studies? Observational? RCTs? Meta-analyses?

An official inquiry is expected to be precise and transparent about evidence, distinguish between types and quality of studies and avoid language that could be seen as vague or open to interpretation, especially on contentious topics

So while it’s vague and non-evidentiary, we can look at the citations: ‘numerous’ refers to FOUR studies.

• UKHSA, 2021a. COVID-19 vaccine surveillance report: 30 September 2021 (week 39)

• Mesle et al., 2021. Estimated number of deaths directly averted in people 60 years and older as a result of COVID-19 vaccination in the WHO European Region, December 2020 to November 2021.

• Mesle et al., 2024 Estimated number of lives directly saved by COVID-19 vaccination programmes in the WHO European Region from December, 2020, to March, 2023: a retrospective surveillance study.

• Watson et al., 2022. Global impact of the first year of COVID-19 vaccination: a mathematical modelling study.

Let’s be clear: Four studies is not “numerous” in any meaningful scientific sense. More importantly, the issue isn’t just the number, but the type of evidence:

• UKHSA (2021) → surveillance report (descriptive, not a primary analytic study).

• Meslé et al. (2021, 2024) → modelling / retrospective estimation studies.

• Watson et al. (2022) → global modelling study.

So, we effectively have: 1 surveillance report, 3 modelling/estimation studies and zero primary clinical studies (e.g. RCTs, large cohort mortality analyses) cited here.

Read more

We’ve also warned about the Inquiry’s causal statements about the certainty of 23,000 deaths attributed to “la…

Read more

To enhance your enjoyment, the two old geezers decided to let you into the backstory of the post. We discussed it on a warm afternoon in Rome over a beer or two in a dive up to our lowly standards (“Live in Rome” post is forthcoming).

It goes like this.

A very well-known broadsheet (the Daily Planet) expressed interest in publishing the post after a few edits to the original draft. The edits were good and made the text flow better - so far, so good. The edited text came to 1015 words.

Share

We had to delay the post by 1 day to avoid infringing on the broadsheet’s “exclusive”. OK.

All set? No. After a few hours and further time contortions, Jake (not his real name) came back with this:

“Apologies, my boss feels the piece is too technical for us so doesn’t want to run it. I’ve only just been told. So do of course do what you wish with it on Substack or take it to another publication as well.”

Read more

Read more

A new Institute for Public Policy Research (IPPR) report has landed at a politically convenient moment. Its headline claim—that there is “no evidence” social health insurance (SHI) systems outperform…

Read more

We originally identified 26 studies examining …

Read more

Read more

Share

This lack of curiosity is very …

Read more

The Great …

Read more

Read more

First, the strategy explicitly says, “It is not, however, …

Read more

Read more

Read more

The sums retrieved so far for the poor taxpayer are …

Read more

Read more