Author: Milind Watve

I will try to explain why I think transparency in the review process is a solution to most of the problems associated with scientific publishing. But first of all why are peer-reviews necessary or are they; and if they are, what purpose do they serve?

Personally I feel peer reviews are certainly desirable if not “necessary” per se. Certainly, good science was being published prior to 1960s when only a few journals were practicing anonymous peer reviews. But the community has changed substantially and publishing has changed even more. Earlier, subscribers paid for a journal and authors did not pay for publishing. Today the trend is that the authors pay and readers enjoy free access. This is not only a technical change. It reflects the changing community, the changing market forces in scientific publishing. So what worked in the early 20^th century may not work today. Analysing this change is worth a whole thesis, but for the time being, I will only pen down my personal ‘belief’ that peer review has become indispensable in the context of today’s scientific publishing.

What are peer reviews intended to do and what do they actually do? I believe the original idea of peer review was to supplement the thinking of one research group by others in the field, who may have a somewhat different vision. It is possible and natural that researchers are often carried away by their own hypothesis or that they might have adopted a narrow view. This is likely to lead to partial blinding and inability to view the other side of the problem, if any. Peers from the same field but with different points of view can complement or at least point out other possibilities, any flaws or paradoxes that might be arising along with the new finding being reported.  Addressing the concerns of such peer reviewers, the manuscript quality can improve substantially. Such a peer review would be extremely helpful in any field of science.

However, this is hardly the purpose that today’s peer reviews serve. They are mainly used to decide acceptance or rejection of manuscripts. Often the accept/reject decision is taken first and then elaborate justifications are sought for the decision. Often this is the main purpose and content of the review report. This is partly because most journals are flooded with submissions and are looking for convenient tools to get away from this overburden. So the original purpose of peer review is completely lost.

This, in itself, is a substantial degradation of the peer review tradition, but let us accept this as inevitable and see whether peer reviews serve at least this function reasonably well. This is a valid question and should potentially be testable. But it cannot be tested in the present scenario because the data are not available for testing. So first of all, simply in order to convince the scientific community that peer reviews really serve the purpose, all peer review data should be made available, which means the peer reviews should be transparent to everyone.

There are more reasons for insisting on transparency. Although good journals can be assumed to choose reviewers carefully, there is no guarantee that ultimately these individuals actually review the manuscript. It is just too common all over the world that leading researchers find themselves too busy to give time to a seemingly ‘unproductive’ work and they ask their students, even undergraduates to do the reviews. This practice is common throughout the world, but no data on it are available. If they have to do it themselves, they can at the best devote limited time to it. This often results into irresponsible reviews. Good researchers would not like to be called irresponsible, but that is unlikely to ever happen because hardly anyone knows who has written the review report. More responsible reviewers will not take a review commitment unless they can devote sufficient time to it. Here lies the major problem. The result of their responsible behaviour is that there are more irresponsible reviewers in the field. If the reviewer remains anonymous, the blame of irresponsible reviews goes on the editor. The editor presumably would like the review process to be more responsible, but for an editor, finding a reviewer is the foremost problem. Since most responsible researchers will not commit unless they have sufficient time, which they never have, the editor has hardly any choice. A transparent review will expose irresponsibility, if any, and I think that may be sufficient to bring in more responsibility in the review process.

Third is the problem of journal quality. The best way to judge a journal’s quality is to look at the rigor and quality of reviews. But since this information remains hidden, other indices such as impact factors get an upper hand. There has been serious criticism of impact factors and other numerical indices, but since the important remains hidden, we make the easily visible important. If all peer reviews are made publicly available, the quality of the editorial process will become transparent and grading of journals can be based on what really matters in scientific publishing.

This will naturally drive away the menace of predatory journals. The definition of a predatory journal is not that they extract money from authors. Many good journals also do it. The definition depends upon absence of or having a fake review system. If reviews are transparent, they will be compelled to undertake serious reviews and if they do so they will no more be “predatory”. So transparency of the review process is necessary and sufficient to eliminate predatory journals.

Authors of rejected papers are often cribbing about unfair review. This might be often, if not always, true. There is a feel that manuscripts from authors from less known organizations or countries are more likely to be returned without review or face review by second grade reviewers. There is no way to test this since no data are available. Reviews can be unfair, biased or irresponsible. Even if a reviewer recommends rejection, the comments should be useful in improving the manuscript. My own impression based on the comments received on our manuscripts is that about 20 % of the comments are really useful in improving the quality of the manuscript, either for the same journal or for resubmission elsewhere; about 50 % are irrelevant to the central argument of the manuscript and attack some peripheral features of the manuscript or attack what has not been said in the manuscript; about 30% are factually wrong or unsupported. But my sample size is small, or my opinion might be biased. If the reviews are transparent, let readers decide whether reviews serve their main purpose or not, and for which journal the proportion of useful reviews is better.

But on top of all, I think science will benefit by studying the behaviour of different players in science and scientific publishing. Science is a human activity and all elements of human nature are very much there. Science does not progress by theorems, hypotheses and evidence alone. Certain components of the human mind and human social behaviour drive the progress of science. There are evolved psychological traits and diverse cultural traditions that decide how science progresses. What people in a field are ready to accept, what they reject and what they prefer to ignore are complex phenomena, not very clearly understood as of now. Studying these aspects of behaviour should have been an essential part of understanding science. However, the main source of data for such studies remains unavailable due to the unnecessary and religious confidentiality of the editorial process. It is utmost necessary to break this barrier.

What are the hurdles in bringing in transparency? One is the power structure in the scientific community. Transparency challenges the power of the powerful and we can expect extreme resistance to transparency from the politically powerful lobby in science. For an open minded, well intended true scientist, I am unable to imagine any reason not to support transparency of the editorial process. Anonymity at the most, is understandable, and sufficient to minimize personal conflicts potentially arising from review reports. But there is absolutely no reason why the reports should not be made public. The other hurdle would be from authors lacking confidence. If they feel their manuscript has weaknesses which are exposed by the peer review, they could be reluctant to publish them. But certainly any honest editor or reviewer, and any confident but open minded author would certainly support transparent peer review systems.

How to publish peer reviews?

Scientific publishing has changed substantially by the emerging pre-prints culture. Pre-print services also allow uploading revisions. Now just one more step is to include the review reports and authors responses along with the revisions independent of acceptance or rejection. My experience with BioRxiv is that they objected to disclosing the journal name, but published the comments and our responses that we uploaded. So a new path to publish peer reviews is opened up, although with some constraints. If you have posted your pre-print you can always post the review reports you receive. In case this doesn’t work, authors can do so on their own blogs. This is what my attempt here is. I have no idea at the moment whether and how it will click. What I mean by ‘click’ is a few more researchers feel like doing so. I presume authors confident about the quality of their work would respond positively, others will not. But in case the trend clicks, I have no doubt, it will revolutionize scientific publishing.

Links for a few more reviews of our manuscripts here:

1. Inferring causality from correlations: This is an age old problem with a lot of philosophical discussions, but little usable sound methods helping actual causal inference. We hit upon what we though was a possible major break-through. We had usable methods to infer causality from correlations provided there were three or more intercorrelated variables, not two. There was simple but sound mathematical basis for all the methods. The only problem was that this was coming from a biology lab, not a mathematical statisticians’ group. We got five rejections in a row before publishing finally in PLOS ONE (https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0204755). All the editor’s responses and the original manuscripts communicated are available Here. Interestingly some found it too complex and some thought the mathematics was too simple to publish. Nowhere, in the reasons for rejection, our central argument that causality can be inferred from correlations using a set of methods was challenged.
2. We had communicated an article to Behaviour and Brain Science. It was rejected saying that we just published another article in a similar field and we do not publish many articles in the same field. Our article had no overlap with the central argument of the earlier published article, just that it was in the same field. Incidentally a few weeks later another article on the same subject was accepted, and that came to me for comments. I pointed out that the reason that you gave for rejecting our paper is not true since your journal does publish many articles in the same field. So give some other, more logical excuse for rejecting our manuscript!! This was followed by a series of emails in which a number of secrets of the editorial process were revealed, but the editor raised strong objections for making these emails public. So I am posting Here only the rejection letter and my response to it.

Over a decade ago, I met a highly successful US based scientist of Indian origin. He routinely published, and publishes even now, dozens of papers in the most prestigious flagship journals. I have read much of his work with interest and have seen that he has consistently delivered excellent publications on the cutting edge of science. I asked him whether he thought he could do the same quality of research if he worked in India. Without having to think for a moment, he answered frankly, “At least in my field, I have no doubt, I can do the same quality work in India. What I am not so sure is that I would be able to publish it the same way!” He added further, “Publication is a mafia. If I am not in the circle, I won’t be able to publish. It is not enough to have good quality work, you need to have connections in the influential circles.”

Several years later I had a conversation with a celebrated scientist, a Noble Laureate himself. We were together for three days and he listened to my work and my ideas with interest. Then he said, “With a mind like yours, I won’t be surprised if you publish in Nature, Science, Lancet, PNAS quite frequently.” My spontaneous response was, “Try publishing from India and you will know.” He did not blink, only nodded. He did not need more than a moment to imagine and agree fully and instantaneously.

Everyone seems to know how the world of scientific publishing works. I won’t use the word ‘mafia’ myself. It is better left to discussions over coffee with successful elite scientists. But like every researcher I know that scientific publishing is a field loaded with so many weird problems, only a few of them surface once in a while and receive some attention, discussion and debate. Being a student of behaviour, I see many fascinating behavioural phenomena happening here, which make wonderful research problems. But the existing system of scientific publication does not permit any research of this kind. This is mainly due to the sacred confidentiality of the editorial process. The field of meta-science or the science of science is an extremely weak, almost non-existent effort. The handful of low ranking researchers in this field do publish something realistic and insightful once in a while, but which largely remains ignored by the scientific community and therefore has hardly made any impact on main stream science so far. Although ultimately it aught to make an impact, at present they are constrained by unavailability of data, which is largely due to the unnecessary confidentiality of the editorial process.

Most working scientists are fully aware of the flaws in the peer review system but they consider them ‘necessary evil’. Most seem to think that there is no better alternative and therefore let the system be what it is. But what if the evil is growing exponentially? Perhaps it is, and that is being felt as well. One cannot show the trend using any data, because all data remain hidden. There is nothing more ironic in the field of science. Entire science is based on availability of data and the main pillars of science are interested in hiding all data. Today science is moving rapidly towards a culture of open access, open source; where most journals now insist that authors make their raw data public in some form or the other. But the same journals are not ready to make the review process transparent and accessible to the public.

What really goes on under the carpet of confidentiality  is left to anybody’s imagination. We only need to know that scientists are humans too and everything that goes on in the human world, does happen in the world of science too. Just that it remains hidden. Some journals now publish the reviewers’ comments along with the paper, but this is limited to accepted papers alone. All rejection related correspondence always remains hidden and there lies the real problem.

Interestingly, unlike the popular belief among students, a confidential review process is not a time tested tradition. It is fairly recent in the history of science. Out of the 300 papers published by Einstein, only one was subject to anonymous peer review and the comments were quite negative. Einstein himself made some nasty comments on it and withdrew his manuscript never to publish again in any journal with confidential peer review. Nature considered peer reviews as mandatory only by 1967 and Lancet by 1976. Mandatory peer review exists only for one or two generations of researchers. This is not long enough to say that it is a time tested system. The system has remained stable only because all possible ways of challenging it are blocked. Unless journals make the review data and particularly rejection data available to any meta-science researcher, the system cannot be claimed to be fair. Whenever, the peer review system has been tested, it is found to be highly flawed. There are only a handful of attempts to test it and they are published as well (for example see 1-6). So the most important activity in science, that of publishing stands on demonstrated unscientific principles.

The confidentiality of peer review has given rise to an entirely different problem, that of the so called “predatory journals”. Predatory journal is a big trap created by editorial confidentiality of the mainstream scientists. At present there are problems even in defining a predatory journal. There is no way to demonstrate that these journals publish without good quality peer reviews. They can always escape saying that we follow rigorous peer reviews but they are confidential so we cannot disclose. There is no use blaming such journals because this ghost is created by the mainstream journals themselves.

Can scientific publishing be made scientific? Can it be brought out from its religious fervour? We can certainly answer this positively and the way to do so is also very clear and simple. Bringing transparency in the editorial correspondence is the only and ”simple” way out. The reason I put the word simple within quotes is that it is conceptually simple, but not simple to execute because it will change the power structure in the field and so the currently powerful people will be reluctant to give up.

There is some justification for confidentiality and some of it’s reasons are genuine, but not intractable. The review process can be made public without affecting the anonymity of reviewers, or keeping anonymity optional for them. Many journals already have this option and this takes care of almost all potential problems arising out of transparency.

It’s not that things are not changing. Many journals are exploring different options for peer review. Some tried double blind peer review. What could be a sound looking idea, is actually not behaviourally sound. If a reviewer receives a manuscript from unknown authors, by human nature, the first reaction is to guess who could the authors be. The focus of the reviewer then goes on picking up indirect cues by which the author identity could be guessed. This guesswork does not help in eliminating biases, it might make it even more weird. So double blinding cannot be said to be a successful alternative. But there are more practical alternatives and there are attempts to try them out.

Last month we had a pleasant surprise. On communicating a manuscript to PLOS Biology, we got a request from the editors for our consent for an open public appeal on BiorXivs and on twitter for comments on our manuscript. We gladly consented. There were hardly any comments in the public domain, apart from brief ones showing positive interest, but based on three comments confidentially received by the journal the editors rejected our MS. I wrote back saying in the spirit of open peer review that we have consented for, the comments and our responses to them should be posted along with the pre-print. After a series of emails, two of the three reviewers and the editors consented and we posted the comments and our responses on BiorXiv (see the comments section of https://www.biorxiv.org/content/10.1101/553016v1 ). I think making the peer review public is a significant historically important development. Now BiorXiv accepts posting reviewer comments and author responses for a rejected manuscript. They declined to disclose the name of the journal. But the journal itself had tweeted that they are reviewing our manuscript, therefore BiorXiv’s stand of not disclosing the journal name is meaningless anyway. I feel the rejection was a blessing in disguise and a first hand experience of this new tradition is a greater reward.

On this background, I have made one more decision. I will also make the comments received on my other manuscripts public on my own blog along with our responses to it whenever appropriate. I see no reason for not disclosing the journal or the editor’s name. Reviewers mostly remain anonymous anyway. This activity will be independent of acceptance or rejection. I will also write back to the editor asking them to inform the reviewers that we have published your comments, and our responses to it, if any, and you are welcome to react on it further. I am sure that although a small step, this will potentially be a turning point. If a number of authors start making the reviews public, the reviewers will have to be more responsible for the comments they write. This increased sense of responsibility, by itself can reduce the problem substantially.

Is it illegal to do so? I see no reason. On at least half a dozen times, after getting a rejection, I asked the editors would they mind if I make all the editorial correspondence public? On this I received an amazing diversity of responses, including sheer panic, desperate defence of editor’s rights to rejection (which I had not challenged in any case). I further asked many of them whether they would take a legal action if I make them public, and under which law? For this they did not know whether there was any legal protection to the confidentiality of the editorial process. But without worrying about legality, I have decided to do two things in this blog. One is that I will publish my analysis of optimization of behavioural strategies of the different players in the scientific publishing game in a series of articles. This is a fairly large volume of work that could potentially make a thesis. But I will restrict myself to a series of (non-peer-reviewed) articles. The other is that I will start publishing the comments that we received for manuscripts communicated from our lab, one by one on this blog. This will apply to previous years too. I will encourage other researchers, particularly from the younger generation to watch this and decide whether they too would like to make the reviews public. If a critical mass (which need not be too large) of researchers start doing so, at the best it will bring in a revolution in scientific publishing in no time.  At the least, it would provide excellent research material for researchers in history and philosophy of science and meta-science.

See these links to the editorial correspondence and peer reviews (along with follow up correspondence in some cases) of rejected manuscripts from our lab during the last few years.

1. Rejection by Diabetes Care for our manuscript “How much variance in insulin resistance is explained by obesity?”. After receiving the rejection letter, I asked the editor, whether I can quote some comments from the rejection letter in any future article. I received a very long response. In this letter the editor actually agreed completely with our main argument. The long letter did not say anything about not allowing me to quote. So I am publishing the entire correspondence Here.
2. We wrote a response letter to an article published in the Lancet, which the editors declined to publish. Then we wrote to the authors directly that for us publishing was not important but we had raised a question and would like to know your response to the question, which may be in the form of a personal email to us. The authors did not respond to this.
3. A network model of type 2 diabetes, that had somewhat surprising and non-conformist outcomes, which was ultimately published by PLOS ONE, was rejected first by a number of leading journals. In none of the rejections, out central arguments were challenged. The reasons for rejection are very interesting and worth reading. Through subsequent submissions, the manuscript did improve a bit, but the comments received were hardly of any use towards improvement. You will find the editorial correspondence with Cell, Cell Metabolism, PLoS Biology, PLoS Computational Biology, Biology Direct and PLoS One here.

I will keep on publishing more editorial correspondence about rejections as well as acceptance one by one on this blog.

1. Phillips J. S. 2011 Expert bias in peer review. Curr. Med. Res.
   Opin. 27, 2229–2233.
2. Lee et al 2013 Bias in peer review. J. Am. Soc. Ino. Sci. Tech. 64, 2–17.
3. Haffar et al 2019 Peer review bias: a critical review. Myo Clinic Proc. 94, 670-676.
4. Tomkins et al 2017 Reviewer bias in single versus double blind peer review. Proc. Natl. Acad. Sci. USA. 114, 12708-12713.
5. Kuehn B. M. 2017 Peer review: rooting out bias. eLife 2017;6:e32014
6. Tancock C. 2018 When reviewing goes wrong. The ugly side of peer review. Elsevier Connect March 23, 2018.