It’s intentional; not a result of negligence, limitations or chance.
The crisis is not some rare ailment; it is a direct inevitable result of today’s academic design. Honest and reproducible research is an outlier and will go on becoming rarer day by day.
One of the objective markers of the deteriorated research culture is the reproducibility crisis. Some studies indicate that over 70 % of published results in biology are not reproducible, meaning useless in effect. The frequency of such useless publications is estimated to be between 50 % to 89% by different studies and these include publications from the most prestigious journals. The trustability of published science is perhaps a record-low in the history of science, deteriorating further and looks like headed to achieve zero sooner or later. Only the ignorance of reality among people is what maintains the residual trust. If people come to know what actually goes on in academia and science publishing, they would have demanded shut down of all academia by now.
I asked AI the causes of the crisis and it said the following,
According to meta-analyses (such as a study in PLoS Biology):
- 36.1% – Subpar or misidentified biological reagents, cell lines, and unspecific antibodies.
- 27.6% – Flawed study or experimental design.
- 25.5% – Data analysis and reporting errors / statistical misuse.
- 10.8% – Poor or non-standardized laboratory protocols.
I think they are completely missing on a more fundamental cause of irreproducibility. We discovered it in our exploration of clinical trials recently when two students who worked with me during their internship or just out of interest, explored some possibilities using public domain clinical trial data.
The advantage of clinical trials is that they have to be pre-registered. The objectives and entire design need a mandatory pre-registration with one or more of the regulating bodies. This information becomes public immediately on the respective regulating agency’s portal. The most commonly used pre-registration portal is USA’s clinicaltrials.gov. Others include https://www.clinicaltrialsregister.eu/ of the European Union, https://ctri.nic.in/ of India and https://www.chictr.org.cn/ of China. Because of the mandatory pre-registration a record of how many trials were initiated, how many completed, what were the outcomes and how many resulted in publications is created and is available in the public domain.
The results are revealing. The clinicaltrials.gov is the most user-friendly website and it revealed that between half to three forth of the clinical trial results are never posted on the website. India’s condition is worse. Chine has kept posting voluntary and there is no provision to post data on the website. The authors may only give links to the published papers or any other data repository if they have used. In effect in clinicaltrials.gov we are referring to the best transparency record which is of 50 to 75 % failure. This is not failure to complete in clinical trial. We are only talking about trials that were declared as successfully completed, but data were not made public even after one full year of completion. Further many of these trials have published papers in prestigious journals including the Lancet, and still not posted their data. Having published implies that their data organization is complete and analysis progressed adequately. If they do not post their data in spite of this, what can we infer? We also tested the possibility that trials thought to be “unimportant” for some reason and not used clinically would have failed to post the data. But this possibility did not stand the test. After eliminating alternative possibilities, we reach the conclusion that this is deliberate data hiding.
A little deeper exploration by another student revealed that even among the trials having published and posted the data there are internal inconsistencies and contradictions. So the claims they make in the published paper are actually not supported by data across trials of the same drug. For example in SELECT trial for Semaglutide, coronary artery disease is significantly less in treated but in Sustain 6 and FLOW trials of the same drug the pattern is reversed, although not individually significant. Similarly in SELECT trial acute kidney injury appears to be significantly protected by treatment. But in FLOW, acute kidney injury has greater incidence in treated than in the control. But they cherry pick on the favorable results alone and do not mention the contradictions anywhere. If the hidden data becomes public, I am sure many more such inconsistencies and contradictions will be revealed. Perhaps their claims of success would collapse entirely. And this is the reason they want to hide data.
What happens with clinical trials also happens with experimental biology, just that the data are not made public. Most biological experiments are not pre-registered. It is also not practical to do this. The most common story is the following. A researcher has a pre-decided conclusion perceived to be exciting, making a high impact paper and perhaps a news headline. If the experimental results are compatible, the experiment is termed successful. If not, it is said to have “failed”. Failed experiments are repeated until they are “successful”. I have heard this from several PhD students from different parts of the world. Their mentors insist on doing this. They strive hard for getting the “desired” results and use any means to get there. If they are there, which could be by chance too, they immediately declare the results and most often never repeat the experiment to confirm reproducibility. If and when they do, they often fail to reproduce their own results. So, the problem of cherry picking is there with laboratory research too, just that it is not on record, unlike clinical trials.
Why does this happen? The strongest reason is because the researchers have other objectives, particularly individual career objectives prioritized over scientific objectives. They want their thesis to be completed “successfully”, they want publications, they want funding, more reputation, career progress, promotions, positions, awards, profits if it is an industry and so on. These interests have completely hijacked scientific interests, mainly because we tend to “evaluate” researchers based on their “tangible” output.
In brief it is the concept of “evaluation” of a researcher or a team of researchers based on their tangible output that has corrupted the field. Is their any better way of evaluating? The answer is NO, there isn’t. The concept of evaluating a researcher is itself wrong. This concept comes from a corporate model. Corporates are interested in recruiting the most talented people and extract maximum work output from them by a carrot and stick approach. It works here because the output is in terms of profit, which can have pre-decided targets and success is tangible. The reason this does not work in science is that science cannot have pre-decided targets. It’s supposed to be a pursuit of the unknown. The output cannot have real tangible measures. There are surprises out there so pre-deciding targets is itself misleading. The strategy of recruiting “talent” and motivating it by rewards is itself wrong.
The other problem in evaluation of research is more fundamental. Often in history, big breakthroughs were ahead of their time. Others understood the concepts and their importance decades later. What is the guarantee that in today’s academia the evaluators understand the research of someone? The researchers can be ahead of their time, and such researchers most likely to be punished or eliminated from the system. Evaluating research is a formula to shot down every revolutionary concept.
Academia needs to run on a different model. Not the corporate one. There are people with innate tendency to question, investigate, pursue things out of curiosity. They need not be the most “intelligent”. They will still churn out better science than the most intelligent people working under wrong incentives. They are always in a minority in the society but they have existed all the time and will continue to exist. The strategy of academia should be to hunt for such people rather than hunt for “talent” and once selected, give them ample freedom and resources to ask their own questions, pursue ideas, handle problems, design new experiments, explore new lines of work and encourage, recruit and train more such people. They need not be the best paid people but they just need to have carefree life. They will be attracted to academia despite modest salaries because they perceive the freedom of investigation itself as the reward. No other reward based on “outputs” is necessary. Such rewards are bound to corrupt the system.
This needs a fundamental change. But the change will not happen with today’s academic design because of the question who will “evaluate” this concept unless it is published in an elite journal? And elites will not entertain alternatives because the prevalent system has given them the “success”, so it is the best system for them!! Who would listen to the “failed”?
