Academics: Mend your house first!!

Behavioural and Brain Science is a journal that publishes theme article along with invited commentary from multiple individuals in the field. For those who believe in impact factors, the IF of this journal was 29.3 for 2022. Last year an article by John et al (2023) was accepted by the journal, published online with a call for commentary. The article was about what they called proxy failure, which is not a new phenomenon but the authors articulated different aspects of it quite well. Often it is necessary to quantify the success of something and the further path is decided by this measurement. When the goal itself is difficult to measure, some proxy is used to reflect the progress in reaching the goal. This might work initially but often the proxy becomes more important than the goal itself and then shortcuts to the proxy evolve that may sideline the goal. The system then is likely to fail or derail because of the proxy. The authors illustrated this with several examples from biological, social and economic sector.

What struck me immediately was that the biggest example of proxy failure is research under Universities and Institutes that are supposed to support research. The original article had only a passing mention of academia. I wrote a commentary on this article focusing on proxy failure in academia, which was accepted and is now published. Since the original article had a word limit, I am giving below a little more elaborate and non-technical version of the article. The original with cited references is available at  https://doi.org/10.1017/S0140525X23002984.  

A very well known example of proxy is exam scores. They are supposed to reflect the understanding of a subject. But typically the proxy becomes the goal and all education is oriented towards scoring higher in the exams. The same happens in research. Research papers are to be written whenever something new and exciting is to be shared with others. But today published papers has become a proxy to one’s “success” in research. Getting jobs, promotions and all depends upon how many papers one publishes and where. So inevitably publishing papers is prestigious journals has become the goal of research. In education there is much awareness, realization and thinking so that there are individuals and institutions specifically focusing on education beyond exam centered coaching. But this level of thinking is absent in research and hardly anyone focuses on addressing this problem.

 I feel it is necessary to deal elaborately with proxy failure in academia for two reasons. One is that proxy failure has reached unprecedented and unparalleled levels in academia leading to bad incentives. So much so that we can easily identify consequences of proxy failure far ahead of what the authors describe in various other fields. The authors describe three major consequences of proxy failure namely proxy trademill (An arms-race between agent and regulator to hack and counter-hack a proxy), proxy cascade (In a nested hierarchical system, a higher-level proxy constrains divergence of lower-level proxies) and proxy appropriation (goals at one level are served by harnessing proxy failure at other levels). At least three more advanced levels are observed in academia that might be difficult to find in other fields.

Proxy complimentarity: In this, more than one types of actors benefit in different ways from a proxy and therefore they reinforce each others’ dependence on the proxy resulting in a rapidly deteriorating vicious cycle. Since prestige of a journal is decided by the proxy namely citations of its papers and the impressiveness of the CV of a researcher is decided by the impact factors of the journals, the two selfish motives complement each other in citation manipulation. Citation manipulation has become common because it is a natural behavioural consequence of a system relying on proxies and not only because some researchers are unethical. It is extremely common and inevitable that reviewers pressurize the authors to cite their papers and the authors agree in return of paper acceptance. The fact that this is a common practice is revealed by data in published systematic studies. Institutions and funding agencies are benefited by the citation based proxies since bibliographic indices lead to a pretense of evaluation saving the cost of in depth reading of a candidate’s research. Reading has a high cost, but a selection committee can (and mostly does) make a decision without reading a candidates work, thanks to the proxies. Such mutually convenient positive feedback cycles can potentially drive rapid deterioration of the goal. This is becoming the norm so rapidly that now nobody even thinks there is anything wrong in evaluating someone without actually reading their work.

Proxy exploitation: This is another inevitable phenomenon in which apart from the agents in the game optimizing their own cost-benefits, a party external to the field achieves selfish goals by exploiting  prevalent proxies in the field. In academic publishing profit making publishers of journals thrive almost entirely on journal prestige as the proxy. Editorial boards appear to strive more for journal prestige than the soundness and transparency of science. This was evident in the eLife open peer review debate. The members of the editorial board who opposed the change in editorial norms, never said open peer reviews would be bad for science. They said it will reduce the prestige of the journal, which for them was obviously more valuable than the progress of science itself. More prestigious journals often have higher author charges and thereby make larger profits with little contribution to the original goals. That’s why the journal appears to prestige matter more than the progress of science.

Predatory proxy: This might be the most advanced and disastrous form of proxy failure where the proxy devours the goal itself.  The authors of the original article described the process of proxy appropriation, where the higher level goal does a corrective hacking of lower level proxies. For example, the marketing team might use the number of customers contacted as a proxy of their effort and this proxy can be bloated easily. But in business, the higher level player directly monitors the goal of profit making and accordingly controls proxies at lower level. This does not work in academia since the higher level organizations themselves do not have an objective perspective of the goal. The goal of progress of science is not directly measurable. As a result not only the proxies are used to evaluate individual researcher, they might often be confused with the progress of science itself. Here clearly the proxy has replaced the goal itself.

In many fields of science highly complex work involving huge amounts of data and sophisticated methods of analysis are being published in prestigious journals adding little real insights to the field. For example in diseases like type 2 diabetes, in spite of huge amount of research being published and funds being allocated, there is no success in preventing, curing, reducing mortality or even addressing the accumulating anomalies in the underlying theory. All that we have are false claims of success of any new drug, which get exposed when anyone looks at raw data. A number of papers exposing all this fraud are already published. Nevertheless large numbers of papers continue to get published, huge amount of funding is allotted which by itself is viewed as “success” in the field. Researchers publishing in high impact journals get huge respect and funding although the disease keeps on increasing in prevalence and the society has not benefited by the research by even a bit.

Failure of achieving the goal in not a crime in science, but quite often the failure is disguised as “success” and researchers receive life time “achievement” awards. Such awards have been given for diabetes researchers. No scientist receiving any such awards appears to have admitted that they have actually failed to “achieve” the real goals. Efforts of a researcher, failed by this definition, should still be appreciated but it should not be called “success” or “achievement” just because they published papers in prestigious journals. The worst outcome of proxy failure in academia is the failure to identify research failure as failure. Many other fields including theoretical particle physics or string theory have received similar criticism. Much intellectual luxury is getting published without adding any useful insights in the field. It is published in high prestige journals and therefore is called success although it contributes nothing useful or insightful.

In the last few years many papers have demonstrated that the creativity and disruptive nature of research has declined substantially. Interestingly this decline is evident even when it is measured by proxies. The three outcomes of proxy failure are most likely to be the reason for this decline in real scientific progress. Simultaneously the frequency of research misconduct, data fabrication, reproducibility crisis, paper mills, predatory journals, citation manipulations, peer review biases and paper retractions are alarming and are on the rise. The blame for this cannot be thrust on some individuals indulging in malpractice. This is the path the system is bound to take by the principles of human behaviour.  The structure and working of academia pretends that human behaviour does not exists, there are only ideals and goals. An academic system that ignores human behaviour can never work because the epistemology engine runs entirely on the human fuel.

Interestingly, many researchers are working today on aspects of human behaviour, behavioural economics, behaviour informed system design or behavior based policy. This is a thriving field. Even noble prizes have been given in behavioural economics, for example. All this is potentially relevant to academia but researches in these fields avoid talking about the design of academic systems. The academic system is the nearest, most accessible and most relevant system to be studied. This is the second important reason why studying proxy failure in academia needs to be prioritized. However, research addressing behavioral aspects of academia is scanty and fragmentary and not yet even close to addressing the haunting questions at a system level. What academia have at the most done is having an office for monitoring research ethics, which hardly appears to prevent misconduct. Unless researchers address the issues of behaviour based system design in their own field and come out with sound solutions; unless they redesign their own systems to make them behaviorally sound and little prone to proxy failure, unless they are able to minimize flaws and make the system work smoothly towards the goals, why should other fields follow their advice to redesign their systems? When I read anything about behaviour based policy, the natural first reaction of a citizen like me working outside mainstream academia is “Researchers, mend your house first!!”