How to avoid "fatal defects"?, Survey shows that one quarter of clinical trial data is falsified by participants | authors | data

The state of research fraud in the medical field is shocking. Some researchers have found that in certain fields, at least a quarter of clinical trials may have issues, or even be completely fabricated.

British epidemiologist Ian Roberts even asserted, "If you search for all randomized trials on a certain topic, about one-third of the trials are fabricated."

They urged all parties to strengthen the review.

Fake practices are widespread

John Carlisle is an anesthesiologist working at the National Health Service in England and also the editor of the journal Anesthesiology. He is known for his ability to detect suspicious data in medical experiments.

In 2017, he decided to carefully search for all the manuscripts of randomized controlled trials that he had processed. In the following three years, he carefully reviewed over 500 studies. In October 2020, Carlisle reported an astonishing statistic:

For over 150 trials, Carlisle obtained anonymous individual participant data. By studying IPD spreadsheets, he determined that 44% of trials contained at least flawed data, such as impossible statistical data, incorrect calculations, or duplicate numbers or graphs. He judged that 26% of the papers had common problems, making them completely unreliable - either because the authors were incompetent or because they fabricated the data.

Carlisle believes that without IPD, even experienced detectives cannot detect hidden flaws. But it is this crucial data that journal editors usually do not request, and reviewers cannot see it.

He wrote in the report, "I believe journals should assume that all submitted papers may have defects, and editors should review individual patient data before publishing randomized controlled trials."

Carlisle referred to some experiments as "zombie" experiments because they looked like real research, but upon closer examination, it was found that they only had hollow shells. When he was serving as a reviewer, he rejected all studies that included "zombie" experiments. But three years have passed, and most of the research has already been published in other journals.

In fact, apart from Carlisle's research in the field of anesthesia, many scientists, doctors, and data detectives have believed that false or unreliable experiments are very common for many years. They searched for randomized controlled trials in multiple medical fields, such as women's health and pain research, and found that a large amount of data was not statistically valid.

This issue is to some extent an extension of the infamous paper factory problem: over the past decade, many journals in various fields have published tens of thousands of fake papers, some of which are produced by paper factories.

Falsifying or producing unreliable randomized controlled data is a particularly dangerous threat. They are not only related to medical interventions, but may also serve as evidence for evaluating clinical treatment - medical guidelines often reference such assessments, and these guidelines are important references for doctors when deciding how to treat patients.

Ben Mol, an obstetrics and gynecology expert at Monash University in Australia, stated that up to 20% to 30% of randomized controlled trials included in women's health system evaluations are suspicious.

However, some experts doubt whether this issue is as serious as the most shocking example suggests. Expert in Fetal and Maternal Medicine at the University of Liverpool, UK Ž Arko Alfirevi ć He said, "We must recognize that some people in the medical field have provided a large amount of suspicious data, but many scholars in the academic community believe that this is alarmist."

Fake data affects clinical practice

Randomized controlled trials are the gold standard in the field of medical research, but if the trial data is falsified, it may seriously mislead medical practice.

Japanese bone health researcher Yoshihiro Sato fabricated data in dozens of trials of drugs or supplements for preventing fractures. Sato passed away in 2016. The Retraction Watch website shows that 113 of his papers have been withdrawn. Researchers found that the 27 papers that Sato had withdrawn were cited in 88 systematic reviews and clinical guidelines, some of which provided references for the treatment of osteoporosis in Japan.

Alison Avenell, a medical researcher at the University of Aberdeen in the UK, said that if Sato's trial is excluded, about half of the findings in the review may change. "It may lead to people not receiving more effective fracture prevention and treatment."

Researchers are concerned that in certain fields, a large number of randomized controlled trials from different research teams may be unreliable.

In 2018, researchers published a review exploring whether administering steroids to cesarean section mothers can help reduce infant respiratory problems. Mol stated that steroids are beneficial to the lungs of infants, but may cause damage to the brain. In premature birth, the benefits usually outweigh the harm, but the situation is not very clear when used in late pregnancy.

In 2018, obstetrics expert Alexandros Sotiriadis from the University of Aristotle in Thessaloniki, Greece, led an analysis of evidence of steroid use in late pregnancy cesarean section women. They ultimately chose four randomized controlled trials: a study from the UK in 2005 with over 940 participants; Three Egyptian trials were conducted between 2015 and 2018, with an additional 3000 people entering the evidence pool. The conclusion drawn is that steroids may reduce the incidence of respiratory problems - subsequently cited in over 200 literature and some clinical guidelines.

However, in January 2021, Mol and others conducted more in-depth research on these papers. He pointed out that the end date of the same experiment is inconsistent in different papers. Meanwhile, Mol also questioned other data. He wrote a letter to the author, but did not receive a satisfactory response. One of the authors told Mol that he lost data while moving.

In December 2021, the Sotiriadis team updated its review to exclude all studies except those in the UK. They were made under the guidance of the non-profit civil academic group Cochrane, whose research integrity team suggested that authors should attempt to identify "problematic" or "untrustworthy" experiments and exclude them from the review.

The Sotiriadis team stated that due to only one experiment remaining, there is "insufficient data" to draw definitive conclusions about steroids.

Overall, Mol and his colleagues have pointed out issues with over 800 published medical research papers, of which at least 500 are related to randomized controlled trials. So far, this work has led to over 80 papers being withdrawn and over 50 papers being expressed with attention.

How to determine if it is trustworthy?

Sotiriadis believes that Mol is a pioneer in detecting and combating data forgery, but he denies that the updates were made based on Mol's work. Sotiriadis stated that his team followed a screening protocol aimed at checking "credibility". This protocol provides a detailed list of standards that authors should follow to check the credibility of randomized controlled trials. For example, whether the experiment is prospective registration; Is there no abnormal statistical data in this study, such as narrow or wide distribution of average values for participant height, weight, or other characteristics, as well as other warning signals. If the aforementioned check is not passed, the reviewer will be instructed to contact the original study author, and if the response is insufficient, the study will be excluded.

For Sotiriadis, the advantage of this plan is that it avoids him having to declare the experiment defective or fraudulent. They just didn't pass the credibility test.

This leaves another controversial topic: Is credibility checks unfair to authors of randomized controlled trials at certain times? What exactly can be checked to identify untrustworthy research?

In 2021, Lisa Bero, Senior Research Integrity Editor at Cochrane, pointed out that there is currently no effective and universally recognized method for credibility screening. "Classifying true research as problematic research may lead to incorrect conclusions, damage to authors' reputation, legal disputes, and ethical issues."

Currently, there are multiple credibility screening protocols at play. For example, in 2020 Avenell and others published a checklist called "REAPPRAISED", which is a checklist for journal editors.

At Alfirevi ć It seems that the credibility check used by reviewers is not important, as long as they take measures to scrutinize randomized controlled trials more carefully.

Request to provide original participant data

Carlisle stated that journals should require authors to share original participant data. Many researchers who are concerned about medical trial fraud agree with Carlisle's viewpoint.

Jack Wilkinson, a health researcher at the University of Manchester, UK, said: "It is a good policy to ask for the sharing of raw data. We have always believed in research, but our position is naive." However, this suggestion runs counter to the current practice of most medical journals.

In 2016, the International Medical Journal Editorial Board proposed mandatory data sharing for randomized controlled trials. However, this proposal has been opposed, with many believing that sharing data would lead to privacy breaches for trial participants and issues with the availability of data archiving resources.

Therefore, in the updated guidelines in 2017, ICMJE only encourages data sharing and requires clarification on whether and where data is shared. Christina Wee, Secretary General of ICMJE, stated that the feasibility of mandatory sharing of raw participant data is insufficient, but the committee may reconsider its approach in the future.

Many publishers of medical journals have stated that, according to ICMJE's recommendations, they do not require trial authors to provide original participant data. However, some journals have taken the lead by explicitly requiring authors to provide original participant data. "Most authors provide data when they are told they need it," Carlisle said.

Wilkinson stated that reviewing raw data is a time-consuming task, which greatly increases the burden on reviewers. In addition to requiring data, research integrity experts also suggest that journal editors can accelerate the decision-making process.

In April 2023, the UK Parliament released a report on repeatability and research integrity. The report points out that when the academic community raises questions, the time for publishers to publish corrected or withdrawn research reports should not exceed two months.

Avenell said, "If a journal really retracts a paper, authors should be asked to correct their work, but this rarely happens." For example, in 2022, Avenell's team released a report stating that they had repeatedly sent emails to 88 authors and journal editors who cited the Sato withdrawal experiment, informing them that their research review included the retracted study, but received little response. Currently, only 11 research reviews have been updated, indicating that authors and editors typically do not care about correcting reviews. This made Avenell's team feel frustrated, but they were not surprised.

However, a long-standing question is why so many suspicious randomized controlled trials have emerged? Based on his own investigation experience, Mol attributed the responsibility to a lack of supervision, a superficial evaluation method that drives academic development based on the number of published papers, and a lack of strict inspections by institutions and journals for misconduct.

"When these things are constantly being produced, it's like extinguishing a wildfire, but ultimately it ends in failure," Carlisle said. "The solution must start from the source."