Evaluating the trustworthiness and characteristics of COVID-19-related trial preprints

A recent study posted to the medRxiv* preprint server investigated the impact and trustworthiness of trial preprints on coronavirus disease 2019 (COVID-19)-related decision-making.

The COVID-19 pandemic saw the adoption of preprint servers by the scientific community, which allowed researchers to disseminate their study findings before they were published in peer-reviewed journals. However, the skepticism that preprints may contain crucial errors that may be absent in published studies, has made the scientific community more cautious regarding the adoption of such preprints.

About the study

The researchers of the present study assessed the difference between the characteristics of preprints and published trial reports. They also evaluated key sections of the studies,  namely the methods and results and their impacts on COVID-19 policy and decision making.

The team used the search strategy of systematic reviews and network meta-analyses (SRNMAs) that included conducting daily searches in the COVID-19 database of the World Health Organization (WHO). This database is a comprehensive collection of published reports and preprint literature related to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), sourced from across the globe.

Articles eligible for the study were preprints and peer-reviewed studies which reported trials that included randomized patients having suspected, possible, or confirmed COVID-19 to receive either drug treatments, antiviral antibodies, cellular therapies, a placebo, or standard clinical care. Trials that randomized healthy cohorts that were either exposed or unexposed to SARS-CoV-2 to prophylactic drugs, placebo, or standard clinical care were also included in the present study.

Pairs of trained and calibrated reviewers worked independently using a pilot-tested form. They obtained data related to differences in key methods and results of preprints and published trial studies. The team of reviewers also compared the features and risk of bias of trials associated with preprints, trials initially posted as a preprint and subsequently published, and trials that were published.

The team evaluated the median time required for a preprint report to be published in a journal. They also assessed the source of funding, the number of centers and participants involved, the intensity of care, severity of disease studied, the statistical significance of primary or secondary outcomes, and the risk of bias.

Results

The study results showed that among the 356 eligible trials, 101 were preprints, 250 were journal publications, and 74 were initially available as preprints and were subsequently published.

In comparison to trials published without preprints, preprints and trials that were available as preprints before being published had a higher chance of being registered. On the other hand, trials that were only available as preprints had higher odds of reporting interim results, describing drug therapies than either antiviral antibodies, cellular therapies, or prophylaxis, and receiving industry funding. Furthermore, trials that were first posted as preprints before being subsequently published were more likely to receive government funding.

One-fourth and almost half of the total preprints eligible for the study were published in peer-reviewed journals three and six months post the initial availability of the preprint, respectively. A year after the initial availability, one-third of the total preprints remained unpublished. Moreover, preprints that were funded by the government, studied inpatients, or reported on severe patients, were published faster than the ones that did not receive any government funding, studied outpatients, or included patients suffering from mild or moderate disease, respectively.

Approximately 56.8% of the total preprints were found to have one or more discrepancies in their key methods or results as compared to their subsequent published trial report. The most common discrepancy reported in the key methods was the characterization of allocation concealment and in the key results were differences in the outcome data between publications and preprints.

The study also reported other differences in key methods including the reporting of one or more statistics that were not reported in the preprint, the reporting of the interim results in the preprint, and the inclusion of a protocol and/or statistical analysis plan in the published report.

Conclusion

The study findings showed that there was no convincing evidence stating that preprint trial reports provide results that are less reliable or less trustworthy than published trial reports. Moreover, the study results showed that preprints could affect meta-analyses results and encourage users to consider data from preprints for rapid decision-making in contexts that are subject to peer reviews.  

*Important notice

medRxiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.

Written by

Bhavana Kunkalikar

Bhavana Kunkalikar is a medical writer based in Goa, India. Her academic background is in Pharmaceutical sciences and she holds a Bachelor's degree in Pharmacy. Her educational background allowed her to foster an interest in anatomical and physiological sciences. Her college project work based on ‘The manifestations and causes of sickle cell anemia’ formed the stepping stone to a life-long fascination with human pathophysiology.