Scientific development relies on knowledge/research credibility and validity. Experiment replication (redoing the experiment and obtaining the same results) and reproducibility (the ability to efficiently redo the experiment in another lab) ensure reliability in biomedical research. Unfortunately, there is plenty of evidence that several scientific fields, such as neurodegenerative disease and cancer biology, face a replication/reproducibility crisis when it comes to these pre-clinical experiments.
Academic and industry-led studies confirm replication/ reproducibility crisis.
Large-scale replication studies investigate replication/reproducibility problems and solutions, with the most recently one published in December 2021. According to this study, the researchers who worked in the Reproducibility Project: Cancer Biology attempted to repeat 193 experiments from 53 high-impact papers. Eventually, they managed to repeat only 50 experiments from 23 papers. The findings of this study remind us of an industry example. In 2012, Amgen researchers failed to reproduce preclinical findings from the oncology space in 47 of 53 papers in high-profile journals.
How does replication/ reproducibility crisis affect drug development?
The consequences of this replication / reproducibility crisis in the preclinical stage go beyond academia and affect the pharma industry. While in academia the inability to replicate and reproduce a study costs the researcher’s time, effort, and sometimes integrity, in the pharma industry, it can severely and negatively impact drug development.
Drug development, especially phase I clinical trials, is based on preclinical trial data. A flawed preclinical study that lacks reproducibility raises potential drug safety and efficacy concerns. As the drug development process takes 10-15 years and there is an average cost of USD 1.5 bn/drug, an irreproducibility rate, for example, around 50%, would result in significant waste of resources.
Preclinical studies issues associated with drug and clinical trial failures
Below we discuss examples of drug and clinical trial failures because of weaknesses in preclinical studies.
Drug failures
Let’s first look at zolpidem, a sleeping aid drug sold under the commercial name Ambien. Women under medication with zolpidem started sleepwalking, sleep driving, etc. Eventually, researchers found that women metabolized it at half the rate of men. Theoretically, this dose sex-specific finding should have been noticed in preclinical studies. The last FDA update concerning dose recommendations was in 2017. Other examples where preclinical data were not in accordance to clinical and the trials failed are the case of Infinity Pharmaceuticals, Inc and Saridegib (IPI-926) as well as the case of Astrazeneca and the neuroprotective drug NXY-059.
Trial failures
Moreover, according to clinicaltrials.gov, in many cases the withdrawal or termination of clinical trials for any disease/condition are related to issues in their preclinical studies. For example, this trial with title “Pharmacokinetics of SSP-004184 in the Treatment of Chronic Iron Overload Requiring Chelation Therapy”, was terminated due to the inability to draw conclusions from the data; in this case, an evaluation of nonclinical findings from experiments in rats was held. At the same time, other trials were terminated due to, among other reasons, toxicity and errors in preclinical immunostaining techniques and readouts.
The case of the glioma cell line U87MG is another example of misleading preclinical research that could have negatively affected drug development, as in 2016 turned out that U87MG has an unknown tumor origin.
An additional example which shows that defective preclinical research affects clinical trials, is related to pancreatic cancer. A study on 30 randomized phase III clinical trials in pancreatic cancer, showed that 1) only 13% of them were successful and 2)in the majority of the trials, there were insufficient in vivo preclinical data, as well as insufficient evidence that the drug “hits” the target.
Four ways to improve preclinical research in the lab
Inattention and mistakes in experimental design, animal model choice, statistical analysis, and data availability are common and can lead to replication/reproducibility problems. Below we discuss where to focus in order to design and conduct successful preclinical trials.
1) Experimental design
A well-established protocol is mandatory. Especially for in vivo studies, protocols should be according to specific guidelines, such as the ARRIVE 2.0 guidelines, in order to use the most efficient model organism (more about this in #2) which represents the human disease, sample size, sample sex, etc.
2) Appropriate animal models for in vivo preclinical trials
Since humans and mice do not respond the same to the same treatment, choosing the appropriate animal model matters. We expect variability during the translation process from preclinical to clinical research. Usually, pharmacokinetics and toxicity are very different in the preclinical and clinical stages. For these reasons, detailed cellular and molecular screening with high-throughput technologies is needed. Also in the design of an in vivo study is important to give attention to inclusion/exclusion criteria, randomization, blinding, and sample size
3) Statistical literacy
It is essential to have good statistical training (randomization, selecting the appropriate statistical test, type, and distribution of data, multiple hypothesis testing, etc.) to improve validity.
4) Data availability
Replication problems would be minimized, if we could achieve better data management. For data, metadata and infrastructures, it is essential to follow the FAIR guidelines which are based on four pillars: Findability, Accessibility, Interoperability and Reuse.
Preclinical research in advanced therapies. Improving the odds of drug development success.
Clinical translation of advanced therapy products can also be significantly affected by preclinical research variability and irreproducibility.
The complexity of advanced therapies, the need for deep biological knowledge, the interdisciplinary approach needed to develop them, the necessary long-term follow-ups, and the high cost of materials and protocols involved, render the preclinical evaluation really challenging. And hence, more prone to experimental carelessness and mistakes.
Potentially due to the complexity of advanced therapies and their translation into products, Advanced Therapy Medicinal Products (ATMPs), in most cases, lead to an unsuccessful long-term performance in the market. More specifically, since 2019, 5 out of 14 approved ATMPs were withdrawn after approval according to this comparative study.
In advanced therapies like cell and gene therapies, where efficacy and safety vary to a great degree, preclinical studies should be very well established and follow specific EMA and FDA guidelines.
For these products, manufacturing and safety are very challenging. So, it is important that the design and execution of preclinical evaluation studies go beyond the four areas we described above and also focus on the following:
- election and design of gene transfer system
- cell and tissue specificity
- cell markers, gene expression
- viral replication and persistence
- immune reaction
- off-target effects
- insertional mutagenesis, engraftment
- pharmacokinetics and toxicity studies
Increased awareness around reproducibility/ replication crisis. The role of research transparency.
Due to the large number of non-replicated studies, we see increased awareness around the problem of replication/reproducibility crisis in preclinical research and its effect on clinical research. Over the last few years, initiatives and guidelines aspire to end this crisis. Research transparency is an important measure to limit replication/reproducibility problems.
We advocate that research data and protocols should be publicly available (as much as possible) and communicate science and that journals should also publish unsuccessful experiments.
Do you want to stay updated on the latest healthcare challenges and solutions in pharma and drug development?
Contact us and gain best-in-class market insights. Learn how we help you make informed decisions.
#reproducibility #replicationcrisis #preclinical #preclinicalresearch #biotech #drugdiscovery #drugevaluation #drugdevelopment #safetypharmacology #advancedtherapies
Sources:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8651289/
https://www.taconic.com/taconic-insights/quality/replication-crisis-preclinical-research.html
https://www.sciencedirect.com/science/article/pii/S2211383522000521
Ten Points to Improve Reproducibility and Translation of Animal Research – PMC (nih.gov)
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5461896.1/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6436641/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8322039/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7327881/