Tag: academia

“Nature” on citizen science vs the nature of citizen science:

The 3rd October 2024 issue of Nature has an editorial on citizen science (https://www.nature.com/articles/d41586-024-03182-y). It has some brilliant and successful examples of involving volunteers outside formal academia in doing exciting science. But unwritten in the article are the limits of citizen science as perceived by academia. On the other hand, I have examples which go much beyond what Nature Editors see. Whether to call them successful or not the readers can decide by the end of this article.

In all examples that the Nature Editorial describes, volunteers have been used as free or cheap skilled labor in studies that mainstream academics designed; for kind of work that needed more manual inputs; where AI was not yet reliable; hiring full time researchers was being unaffordable; they could save time and money by involving volunteers.

In contrast I have examples where citizens’ thinking has contributed to concept development; to design and conduct of experiments, where the problem identification itself is done by citizens; where novel solutions are perceived, worked out and experimentally implemented by people formally unqualified for research; where citizens have detected serious errors of academics or even exposed deliberate fraud by scientists. I would certainly say that this is far superior and the right kind of use of collective intelligence of people. What citizens can’t do is the formalism of articulating, writing and undergoing the rituals needed to publish papers where academics may need to help. But in several respects citizens are better than academics in pursuing science.

I have described in an earlier blog article the work that we did with a group of farmers during 2017-2020 (https://milindwatve.in/2020/05/19/need-to-liberate-science-my-reflections-on-the-scb-award/). This started with a problem faced by the farmer community itself, to which some of us could think of a possible solution. Thereafter farmers themselves needed to understand the concept, design a working protocol based on it, taking it to an experimental implementation stage and maintain their own data honestly. Then back to trained researchers who analyzed the data, developed the necessary mathematics etc. By the time this was done I had decided to quit academia and my other students involved in this work also had quit for different reasons. The entire team was outside academia when the major chunk of work was done and we could do it better because we were free from the institutional rituals. This piece of work received an international award ultimately. Here right from problem identification farmers, including illiterate ones, were involved in every step except the formal mathematics, statistical analysis and the publication rituals.

In January 2024, I posted on social media that anyone interested in working with me on a range of questions (including the ones they themselves have) may contact me. The response was so large that I couldn’t handle so many people. I requested that someone from the group should take the responsibility of coordinating the group so that maximum use of so many interested minds can be made. This system did not take shape as desired because of many unfortunate problems coincidently faced by all the volunteering coordinators themselves. But a few volunteers continued to work and a number of interesting themes progressed. They ranged from problems in philosophy and methods of science to identifying, studying and handling problems faced by people.

One of the major patterns in this model of citizen science involves correcting the mistakes of scientists writing in big journals, some of which we suspect were intentional misleading attempts. For example, we came across a paper in The Lancet Diabetes and Endocrinology (TLDE) which was a follow up of an interesting clinical trial in which using diet alone they had claimed substantial remission of type 2 diabetes in one year. Their definition of diabetes remission was glucose control and freedom from glucose lowering medicines. After 5 year follow up they claimed that the group under the diet intervention who achieved remission by the above definition had significantly low frequency of diabetic complications. When we looked at their raw data, it certainly did not support their conclusion. They had reached this conclusion by twisting data and cherry picking on the results. Peer reviews never look at such things if it is coming from one of the mainstream universities. This is not a baseless accusation, there is published data showing the lop-sided behaviour of peer reviewers.

The true peer reviewers need to be the readers. But in academia nobody has time to read beyond the name of the journal, title and at the most abstract. The conclusions written at the end of the abstract are taken as final by everyone, even when they are inconsistent with the data inside. This is quite common with the bigger journals of medicine. The reason academics are not interested in challenging such things is that it takes a long time and painstaking efforts by the end of which they are not going to get a good original publication. The goal of science has completely changed in academia and the individual value of publishing papers in big journals has completely replaced the value of developing insights in the field. Since anyone in academics cannot do the job of preventing misleading inferences, citizens have to do it. Citizens can do what academics can’t because number of papers and journal impact factors don’t shape their career anyway. Citizen science should focus on doing things that people in academia cannot or may not. That is the true strength of citizen science. Since people in academia seem to be least bothered about the increasing fraudulent science, citizens outside academia will have to do this.

In this case, after redoing statistical analysis ourselves, we wrote a letter to the editor of TLDE, who responded after a long time saying that the issues you raised appear to be important and she will send the letter to the authors to respond. Then nothing happened for a long time again.  On sending reminders the editor responded saying that our letter was sent to a reviewer (no mention of what the authors’ response was) and based on the reviewer’s views it was rejected. The strange thing was that the reviewer’s comments were not included in the editor’s reply. After insisting on seeing the reviewer’s comments they were made available. And amazingly (or perhaps not surprisingly) the reviewer had done even more selective cherry picking on our issues. He/she gave some sort of “explanawaytions” to some of them. For example we had raised an issue that when you do a large number of statistical tests some are bound to turn out individually significant by chance alone. Therefore just showing that you got significance in some of them is not enough. This is a well known problem in statistics and there are solutions suggested. The reviewer said something to the effect that the solutions suggested are not satisfactory for us and hence we pretend that the problem does not exist!! The reviewer completely ignored the issues for which he/she did not have any answer. So the reviewer was worse than the authors. Then we published our comments on Pubpeer (https://pubpeer.com/publications/BB3FA543038FF3DF3F83B449F8E5AA) to which the authors never responded. This entire correspondence with TLDE can be accessed here (https://drive.google.com/file/d/16zjYPeKcz0JEnlrjSXP4p1QUimdBEPFy/view?usp=sharing). The absence of author response and the fully entertaining reviewer response makes it clear that the illogical statistics was intended to mislead and not an oversight.

Two more fights are underway and I will write about them as soon as they land up here or there. Either the paper needs to be retracted/corrected or our comments published along with the paper. But this will be detrimental to the journal as well as author reputation, so it is very unlikely. A more likely response will be that they will simply reject our comments or do nothing about anything. In either case I will make the entire correspondence public. In recent years a large number of papers are being retracted (over 10,000 in 2023, perhaps much more in 2024). A large number of them are because of image manipulation. But that is because the technique of detecting image manipulation is there now. I suspect a much greater number needs to be retracted for screwing up statistics with intentional misleading, or simply to get the paper accepted. Who will expose this? In my view this is beyond the capacity and motivation of academics and therefore this should be a major objective of citizen science.

I have no doubt that many people outside academia can acquire the skill-set to do so. All that is needed is common sense about numbers. Technical knowledge about statistical tools is optional. Most of the problems in these papers were the kind of misuse of statistics that a teacher like me tells the first year students not to do. In the quality of data analysis the scientists publishing in big journals are inferior to our first year students. I have seen many more examples of this earlier.

Detecting frauds in statistics is not difficult, but the further path is. The system of science publishing has systematically made the further path difficult. In a recent case, a paper had fudged data and had reached misleading conclusions in very obvious ways. The peer reviewers should have detected it very easily, but they failed. When a group of volunteers pointed out the mistakes, reanalyzed the raw data showing that the inferences were wrong; the editors said – submit your comments through the submission process and the submission process includes a $200 submission fee. I am sure the journal did not pay the earlier reviewers anything. And when someone else did a thorough peer review, he/she is being penalized for doing a thorough job!! This is how science publishing works.

In a nut shell, many in academia are corrupt and citizen scientists are likely to do much better science. But academics know this and therefore hurdles are purposely being created so that their monopoly can be maintained. The entire system is moving towards a kind of neo-Brahmanism where common man is tactfully kept away from contributing to knowledge creation. Multiple rituals are created to keep people away effectively. The rituals in science publishing are increasing for the same purpose. I am sure this was the way brahminical domination gradually took over in India. Now the entire world of science is moving in the same direction. Confidential Peer review and authors charges are the two tools being effectively used for monopolization. There is a need that citizens become aware and prevent this right at this stage. I see tomorrow’s science safer and sound at the hands of citizens than with academia. This is the true value and true potential of citizen science. Since academia is actively engaged in suppressing this kind of citizen science, we the science loving common people need to take efforts to keep it alive.

On redesigning academia

I am not only a critic of academia, I have also been working constructively to design an alternative system that is based on the foundations of human behaviour. Behaviour based policy and system design in a relatively novel concept but many academics have started talking about it and certain behaviour based system designs are implemented on a pilot scale and some even in real life. Interestingly none seems to have thought about behaviour based design of academic systems. I made an attempt in a document that I have opened up for everyone here (https://drive.google.com/file/d/1G7Ugv0Wo4gONBQsoTaX_-ggUgTH4ju6A/view?usp=sharing ). This is for sharing, with or without credit. Plagiarism or any version of it is also welcome, all that matters to me is that it is shared widely and read with interest. It is not necessary to agree with everything. In fact a wide and open minded debate on every possible platform is most welcome. I only expect that the debate is not only based on opinions and anecdotes. I have tried to support my arguments with data, whenever possible. The debate also needs to go on the same lines. Fortunately today there are many published studies that are useful for this purpose. Wherever there are gaps in data, let that also surface so that someone may be stimulated to collect data and provide better evidence.

My perspective is mainly from India, so the document addresses the problems of Indian academia mainly. But most of it is applicable for any non-mainstream science country and much of it is applicable to the mainstream as well.

The document first describes the serious flaws, malpractices, misconducts, bad incentives, imbalances and unfairness in the academia as of today. Science appears to have been monopolized by a handful of power centers and its dissemination throughout the world is prevented by the design of the science support systems themselves. The ideal structure of science support systems should be such that good science can be done and published from any corner of the world. The prevalent structure of academia is far removed from this ideal. You have to be a part of the publication mafia (not my words) in order to get published in a prestigious journal. There is published evidence that in academia, most decisions are made without reading the contents of scientific papers/proposals. There is published evidence that peer reviews are inherently biased, flawed and favor the imbalance of power. This is taking the field of science rapidly away from diversity towards more of a stereotyped system and career path.

The document then tries to go to the behavioural roots of these. This is not a conspiracy. It is an effect of having a system in place that is easily drifted from the collective goal towards personal selfish goals. There is an underappreciated but clear and direct conflict between what is good for science and what is good for a successful career.

Having diagnosed the causes, the document then suggests an alternative system that is based on the principles of human behaviour. If a system is designed for some ideology and expects people to mold themselves with the ideology against their nature, the system is bound to fail. A system that eliminates or minimizes the difference between individual optimum and collective optimum is a robust system. A system that coerces individuals to accept ethical norms that conflict their personal gains is a badly designed system. A system that works smoothly towards the intended goal when every individual behaves selfishly is a well designed system. The system I suggest here would minimize, if not completely eliminate the biases, imbalances and defects and facilitate a good and equitable science culture globally.

Why did I write this, being in no delusion that it will bring about any change? To quote from the document itself, “But I cannot imagine myself not writing this when I can clearly see a flawed system, when I know I can diagnose what is wrong and can also see alternative design that is behaviourally sound and correctly incentivised. I have nothing to achieve by spending time and energy on something that will not even be noticed by the mainstream. But I made a statement earlier that there is a mindset that will study, investigate and innovate without any incentive, without any output, returns or rewards. This effort is a demonstration that yes, such a mindset exists and academia need to take efforts to select such minds rather than select “intelligence” and incentivise it with rewards for proxies of success which is bound to corrupt the entire system.” Read and debate on any platform that you like. Feel free to criticize, but only after reading it carefully.

Academics: Mend your house first!!

Behavioural and Brain Science is a journal that publishes theme article along with invited commentary from multiple individuals in the field. For those who believe in impact factors, the IF of this journal was 29.3 for 2022. Last year an article by John et al (2023) was accepted by the journal, published online with a call for commentary. The article was about what they called proxy failure, which is not a new phenomenon but the authors articulated different aspects of it quite well. Often it is necessary to quantify the success of something and the further path is decided by this measurement. When the goal itself is difficult to measure, some proxy is used to reflect the progress in reaching the goal. This might work initially but often the proxy becomes more important than the goal itself and then shortcuts to the proxy evolve that may sideline the goal. The system then is likely to fail or derail because of the proxy. The authors illustrated this with several examples from biological, social and economic sector.

What struck me immediately was that the biggest example of proxy failure is research under Universities and Institutes that are supposed to support research. The original article had only a passing mention of academia. I wrote a commentary on this article focusing on proxy failure in academia, which was accepted and is now published. Since the original article had a word limit, I am giving below a little more elaborate and non-technical version of the article. The original with cited references is available at  https://doi.org/10.1017/S0140525X23002984.  

A very well known example of proxy is exam scores. They are supposed to reflect the understanding of a subject. But typically the proxy becomes the goal and all education is oriented towards scoring higher in the exams. The same happens in research. Research papers are to be written whenever something new and exciting is to be shared with others. But today published papers has become a proxy to one’s “success” in research. Getting jobs, promotions and all depends upon how many papers one publishes and where. So inevitably publishing papers is prestigious journals has become the goal of research. In education there is much awareness, realization and thinking so that there are individuals and institutions specifically focusing on education beyond exam centered coaching. But this level of thinking is absent in research and hardly anyone focuses on addressing this problem.

 I feel it is necessary to deal elaborately with proxy failure in academia for two reasons. One is that proxy failure has reached unprecedented and unparalleled levels in academia leading to bad incentives. So much so that we can easily identify consequences of proxy failure far ahead of what the authors describe in various other fields. The authors describe three major consequences of proxy failure namely proxy trademill (An arms-race between agent and regulator to hack and counter-hack a proxy), proxy cascade (In a nested hierarchical system, a higher-level proxy constrains divergence of lower-level proxies) and proxy appropriation (goals at one level are served by harnessing proxy failure at other levels). At least three more advanced levels are observed in academia that might be difficult to find in other fields.

Proxy complimentarity: In this, more than one types of actors benefit in different ways from a proxy and therefore they reinforce each others’ dependence on the proxy resulting in a rapidly deteriorating vicious cycle. Since prestige of a journal is decided by the proxy namely citations of its papers and the impressiveness of the CV of a researcher is decided by the impact factors of the journals, the two selfish motives complement each other in citation manipulation. Citation manipulation has become common because it is a natural behavioural consequence of a system relying on proxies and not only because some researchers are unethical. It is extremely common and inevitable that reviewers pressurize the authors to cite their papers and the authors agree in return of paper acceptance. The fact that this is a common practice is revealed by data in published systematic studies. Institutions and funding agencies are benefited by the citation based proxies since bibliographic indices lead to a pretense of evaluation saving the cost of in depth reading of a candidate’s research. Reading has a high cost, but a selection committee can (and mostly does) make a decision without reading a candidates work, thanks to the proxies. Such mutually convenient positive feedback cycles can potentially drive rapid deterioration of the goal. This is becoming the norm so rapidly that now nobody even thinks there is anything wrong in evaluating someone without actually reading their work.

Proxy exploitation: This is another inevitable phenomenon in which apart from the agents in the game optimizing their own cost-benefits, a party external to the field achieves selfish goals by exploiting  prevalent proxies in the field. In academic publishing profit making publishers of journals thrive almost entirely on journal prestige as the proxy. Editorial boards appear to strive more for journal prestige than the soundness and transparency of science. This was evident in the eLife open peer review debate. The members of the editorial board who opposed the change in editorial norms, never said open peer reviews would be bad for science. They said it will reduce the prestige of the journal, which for them was obviously more valuable than the progress of science itself. More prestigious journals often have higher author charges and thereby make larger profits with little contribution to the original goals. That’s why the journal appears to prestige matter more than the progress of science.

Predatory proxy: This might be the most advanced and disastrous form of proxy failure where the proxy devours the goal itself.  The authors of the original article described the process of proxy appropriation, where the higher level goal does a corrective hacking of lower level proxies. For example, the marketing team might use the number of customers contacted as a proxy of their effort and this proxy can be bloated easily. But in business, the higher level player directly monitors the goal of profit making and accordingly controls proxies at lower level. This does not work in academia since the higher level organizations themselves do not have an objective perspective of the goal. The goal of progress of science is not directly measurable. As a result not only the proxies are used to evaluate individual researcher, they might often be confused with the progress of science itself. Here clearly the proxy has replaced the goal itself.

In many fields of science highly complex work involving huge amounts of data and sophisticated methods of analysis are being published in prestigious journals adding little real insights to the field. For example in diseases like type 2 diabetes, in spite of huge amount of research being published and funds being allocated, there is no success in preventing, curing, reducing mortality or even addressing the accumulating anomalies in the underlying theory. All that we have are false claims of success of any new drug, which get exposed when anyone looks at raw data. A number of papers exposing all this fraud are already published. Nevertheless large numbers of papers continue to get published, huge amount of funding is allotted which by itself is viewed as “success” in the field. Researchers publishing in high impact journals get huge respect and funding although the disease keeps on increasing in prevalence and the society has not benefited by the research by even a bit.

Failure of achieving the goal in not a crime in science, but quite often the failure is disguised as “success” and researchers receive life time “achievement” awards. Such awards have been given for diabetes researchers. No scientist receiving any such awards appears to have admitted that they have actually failed to “achieve” the real goals. Efforts of a researcher, failed by this definition, should still be appreciated but it should not be called “success” or “achievement” just because they published papers in prestigious journals. The worst outcome of proxy failure in academia is the failure to identify research failure as failure. Many other fields including theoretical particle physics or string theory have received similar criticism. Much intellectual luxury is getting published without adding any useful insights in the field. It is published in high prestige journals and therefore is called success although it contributes nothing useful or insightful.

In the last few years many papers have demonstrated that the creativity and disruptive nature of research has declined substantially. Interestingly this decline is evident even when it is measured by proxies. The three outcomes of proxy failure are most likely to be the reason for this decline in real scientific progress. Simultaneously the frequency of research misconduct, data fabrication, reproducibility crisis, paper mills, predatory journals, citation manipulations, peer review biases and paper retractions are alarming and are on the rise. The blame for this cannot be thrust on some individuals indulging in malpractice. This is the path the system is bound to take by the principles of human behaviour.  The structure and working of academia pretends that human behaviour does not exists, there are only ideals and goals. An academic system that ignores human behaviour can never work because the epistemology engine runs entirely on the human fuel.

Interestingly, many researchers are working today on aspects of human behaviour, behavioural economics, behaviour informed system design or behavior based policy. This is a thriving field. Even noble prizes have been given in behavioural economics, for example. All this is potentially relevant to academia but researches in these fields avoid talking about the design of academic systems. The academic system is the nearest, most accessible and most relevant system to be studied. This is the second important reason why studying proxy failure in academia needs to be prioritized. However, research addressing behavioral aspects of academia is scanty and fragmentary and not yet even close to addressing the haunting questions at a system level. What academia have at the most done is having an office for monitoring research ethics, which hardly appears to prevent misconduct. Unless researchers address the issues of behaviour based system design in their own field and come out with sound solutions; unless they redesign their own systems to make them behaviorally sound and little prone to proxy failure, unless they are able to minimize flaws and make the system work smoothly towards the goals, why should other fields follow their advice to redesign their systems? When I read anything about behaviour based policy, the natural first reaction of a citizen like me working outside mainstream academia is “Researchers, mend your house first!!”

A reason to welcome AI in science publishing:

More and more concern is being raised about the problems in academia that are rapidly expanding both qualitatively and quantitatively. Hardly anyone will disagree that there is a reproducibility crisis, increasing frequency of frauds and misconducts at every level. The burden of APCs is destroying the level playing field (if there was any) so that only the rich can publish in prestigious journals.  The bibliographic indices have almost taken away the need to read anything, because the importance of any piece of work is gauged by the journal impact factor; the performance of any researcher by the number of papers. So nobody reads research papers anymore. Citing them does not need reading them anyway. The rapidly changing picture in academia is a perfect case of proxy failure where proxies have completely devoured the goal of research. Now asking novel questions, getting new insights and solving society’s problems is no more the goal of research. Publishing papers in prestigious journals and grabbing more and more grants is. With this a downfall of science is bound to happen, and actual downfall that has already begun is also well demonstrated by many published studies.

An additional serious concern now is AI. Of late so many researchers are using AI to write papers whose apparent quality of presentation is often better than what researchers themselves could have written. At present AI tools have many obvious flaws and they get caught red handed quite often. Incidences in which hallucinating AI cited references that did not exist have come to light. But soon AI will evolve to be better and then it will make it harder to detect flaws. In response to the first wave of AI generated papers, some journals banned them, but soon implementing such things will become impossible. An arms race of smarter frauds and smarter whistleblowers is not exactly going to be good for science.

Who will benefit the most from the more refined AI tools? Certainly the people involved in research misconduct because the frauds will become increasingly smarter and more difficult to detect as AI gets smarter.

And precisely for this reason I would welcome the use of AI in science publishing because it can get us out of the mess created by us over the last few decades. The mess has been created by the ‘publish or perish’ narratives that nurtured bad incentives. The set of bad incentives gave rise to the journal prestige rat race, the citation manipulation practices, the predatory journals as well as the prestigious robber journals exploiting the researchers’ desperation to publish. AI will help us come out of this mess not by smarter detection of fraud and misconduct, but by enhancing it and making it more and more immune to detection.

It is already become a common knowledge that many papers are being written with substantial contribution from AI. There is yet to be any example where a deep and disruptive insight is contributed by AI. The limiting factor in AI generated science is still going to be scientists who will decide to accept or reject what AI output is saying. If AI gives an output that goes against the current beliefs and opinions in the field, it is most likely to be rejected saying that sometime AI throws up junk (which might be true at times, but who knows) and we need not take every output as true. So AI will make normal routine and ritualistic science more rapid. It will give more easily and efficiently what people in a field are already expecting. But I doubt whether it will be of any help in a Kuhnian crisis.

But AI will tremendously help those who want to strengthen their CVs deceptively by increasing their number of publications, and blowing up citations. This trend will increase so sharply that it will soon collapse under its own weight. As writing papers become easier, their value in a CV will fall rapidly. Institutions will have to find alternative means to “evaluate” their candidates and employees. The clerical evaluation based on some numbers and indices will become so obviously ridiculous that it will have to give way to curious and critical evaluation which is not quantifiable and which cannot be done without efforts and expertise. Critical thinking, disruptive ideas, reproducibility will have to come to the front seat and replace bibliographic indices. Nothing can be more welcome than this. If the importance of number papers published and citation based indices vanishes, most bad incentives for fraudulent practices will vanish in no time.  Paper mills and peer review mills will collapse. There can no more be profiteering either by predatory or by robber journals. The edema of science will vanish because it will be no more confused with growth. So let AI disturb, disrupt and destroy the mainstream science publishing systems and that is my only hope to save science from its rapidly declining trustworthiness.

This will certainly happen but not very smoothly. There will be a decade or more of utter crisis and chaos, which has started already. The entire credibility of academia will be at stake. People will cross question the very existence of so many people in academia drawing fat salaries and contributing nothing to real insights.  If academics are prudent, they should start the process of rethinking sooner to shorten the period of crisis and chaos. Vested interests of publishers and power centers in academia will not allow this to happen so easily. There attempt will be to keep things entangled in the arms race. But there still are sensible people in science, aren’t they? The only thing sensible is to allow the current system of science publishing to get crushed under the weight of the AI assisted edema and then make a fresh beginning where HUMANS and not some computed indices make value judgments, identify, attach and appreciate real and insightful contributions to science and make a community of innately curious and investigative minds with no added incentives and rewards. Then let them take help of AI for anything. When the human rat race has vanished, AI will be tremendously useful for its positive contributions.