The Story of Influenza Antivirals: Part 16

Writing the protocol without cribsheets

Writing a protocol is a must for any systematic review (or indeed any serious research study) for reasons we explained when we analysed the disgraceful evidence provided by the UKHSA as a prop for the mask mandates.

Cochrane protocols are fairly standard, but we were embarking on a voyage into the unknown. The evidence for our review came from various sources, and we were unsure whether it was complete. However, we knew approximately 60 percent of the randomised trial data for treatment were not visible, and we had in our possession the module ones from Roche and fragments of a Roche submission to NICE that had been leaked to us by Daisy - the codename we gave the leaker.   

We needed a unique source for reviewing the material to eliminate the risk of duplication or version problems. This had to be our own protected website which was run by one of us. Anything to be reviewed must come from this website and, if annotated, go back there with the extraction sheets and any other by-product.

Only one person was to update the documents, and any requests for updating were copied to the others and to Freddyflu, our electronic dead letter box to which we copied all emails relevant to the project (a trick we learned working on the 2009 review). 

The first task was an evidence framework, which in a standard review would be based on searches of electronic databases of publications.

But we had decided these were unreliable, and therefore we came up with a new procedure in four parts: 

• A Table of Contents to identify the trial identifier known to regulators (TOC)

• A trial directory: the TOCE (E for evidence). 

• A narrative diary to track who was doing what and provide a replicable path for anyone wanting to reproduce our work.

• A central repository of all our work, files and communications

We also split the review in two: Phase 1 consisted of slotting the material at our disposal into a CONSORT-modified format, adapted to the quantity of detail now available. CONSORT stands for Consolidated Standards of Reporting Trials: A checklist for reporting randomised trials based on publications.

But our modified format allowed us to recreate the trial and assess whether it was missing crucial parts. The use of CONSORT gave us what we needed: a guide to putting the piece of a massive puzzle together, as the basic elements of a trial were listed in the statement. Other bits, like the statistical analysis plan (SAP), were unknown, but we could “hang them off” or reconstruct them as they became available, and we now understood where they fitted.

In the next step, trials meeting our new inclusion criteria and considered modified CONSORT-compliant would proceed to Phase 2, where we would assess the likelihood of bias from multiple angles. 

Those trials, which were incomplete, including all the ghostly traces we had encountered up to then, were to remain at Phase 1 in the hope of future data integration.

The modified CONSORT template was piloted, and each trial was extracted in double, and if it passed Phase 2, its risk of bias (RoB) was assessed. 

At the same time, we looked into the possibility of reconstructing at least part of a Clinical Study Report from the relevant parts of the FDA Medical Officer Review (MORs). Still, we swiftly gave up the idea as there were too many gaps, as you would expect from what were essentially reviewers’ comments.

This procedure may now seem absurdly complicated, but we were unsure what to expect as no one had done this before. Nowadays, with one click, you can download a complete tabular view of past, present and planned trials. This is partly thanks to our work and others we shall meet as we go on.

You can catch up with the details of the January 2011 protocol and see the modified CONSORT reconstruction template for unpublished clinical study reports in Appendix 3 of the protocol.

We considered this modified approach would provide patients, clinicians and policy‐makers with the most transparent and independent information about NIs for influenza

It was necessary to provide a complete and unbiased view of their performance. The question was how much of the data would find its way into phase 2 of our review and therefore answer Hayashi’s question.