Data transparency: “Nothing has changed since Tamiflu”

Data withheld

Both studies used data from a healthcare analytics company called Surgisphere. After several concerns were raised with respect to the veracity of the data, the study authors announced an independent third party peer review.67 But Surgisphere refused to transfer the full dataset and associated information, saying it would violate confidentiality requirements and agreements with clients, leading the authors to request the retraction of both studies.

The Lancet and NEJM are signatories to the Wellcome agreement on data sharing for covid-19 studies, which calls for research findings to be openly accessible and to give clear statements regarding the availability of underlying data.8 But an open letter to the study authors and the Lancet’s editor, Richard Horton, pointed out that its paper had no statement on data and code sharing availability and no ethics review.9

Robert Kiley, head of open research at Wellcome, told The BMJ, “In the case of articles which contain private and sensitive information, the data availability statement should still indicate how the data could be accessed, typically by making a request to the appropriate data access committee. We encourage all publishers to require a data availability statement for all research articles and to make this a mandatory part of the submission process.”

So is the Wellcome statement worth the paper it’s written on? Henry Drysdale, a clinical researcher with the DataLab at Oxford University, whose academic work has focused on research integrity, believes it does have some merit as it articulates an idea of best practice and provides a standard to which medical journals can be held.

“However, the statement focuses entirely on the sharing and dissemination of research and does not address information governance or research integrity standards. In the context of major concerns over the integrity of reporting and use of results for high profile covid trials, commitments to these standards are urgently needed,” he told The BMJ.

He believes that the concerns over the Lancet trial are not so much about the editorial process or reporting but about generating and collating data, with some querying the truthfulness of the data. “It’s possible that data sharing standards have been, to a greater or lesser extent, compromised through drives to produce and disseminate covid research quickly (as encouraged by the Wellcome statement). However, it’s very difficult to say with so little information about the sources of data and data collection processes for these trials,” he said.

Deluge of research

The covid-19 pandemic has created an urgent need for scientific evidence to help politicians, doctors, researchers, and the general public understand the evolving situation and know what treatments work. This has resulted in a deluge of new research, much of it published without peer review on preprint servers. “There is a headlong rush to publish in an emergency, and that is toxic,” Jefferson said.

Tracey Brown, director of the charity Sense about Science, told The BMJ, “The urgency of sharing emerging research and data on the covid-19 crisis has created huge dilemmas over quality. Rapid publication and early sharing of results is clearly warranted, but it means that the research community must also double down on communicating the status and reliability of results and their limitations.

“Open data and research evidence is useless without this information. Researchers must avoid hype and police the discussion of the research assiduously, especially as individual papers are now being seized on to push different policy responses.”

She added, “But some of the issues we are seeing are not new. Putting your name to a data analysis when you have not seen the data is research fraud, and always has been, crisis or no crisis.”

Elizabeth Loder, The BMJ’s head of research, acknowledged that it was difficult for peer reviewers or journal editors to detect deliberate, carefully orchestrated fabrication of data.

“In the case of the Surgisphere database, there was a high level of interest in the NEJM and Lancet papers because of the pandemic. This led to rapid identification and speedy retraction of the articles and underscores the value of having many people involved in evaluating and inspecting research both before and after it is published,” she said.

Loder believes that The BMJ’s open peer review process and its commitment to sharing data and the posting of signed peer review reports alongside published research papers allow for a level of public scrutiny that is valuable.

She added, “We are considering a variety of steps we could take to prevent and detect potential fraud. In addition to solutions at the level of individual journals, however, I believe that cooperation among journals and public policy initiatives also may be needed.”