Should peer review stop being anonymous?

Light Switch

Should peer review stop being anonymous?

Prominent researchers can take the gamble, but junior scientists risk retribution

Dan Samorodnitsky

Biochemistry

Carnegie Mellon University

Like anything else, science is built on power structures. Influential scientists can ruin other peoples' careers if they want. A senior scientist can influence who gets hired for a job, who gets tenure, or – perhaps most importantly – who gets research grants. To be awarded a grant, you need to show, among other things, that you and your lab have been producing papers. To produce papers, run a lab, and be a successful scientist, you need grant money. Yes, it's a circle.

Papers and grants are subject to criticisms, a process called "peer review." When a paper gets submitted to a journal, the journal solicits other scientists in the same general field to give comments and essentially dictate whether to accept or reject the paper.

Usually, these comments are anonymous. They can range from a most satisfying rubber-stamping, to helpful, genuine constructive criticism, to oddly vicious and nonsensical comments. This is frequently a frustrating experience, and people can be left feeling wronged or even targeted.

That's why reviews are usually anonymous: to protect reviewers and give them space to leave honest feedback. The fear is that a younger scientist, someone still trying to get papers, grant money, or tenure could negatively review the work of an older, more established person. Not out of spite, but because older scientists are just as capable of terrible work as anyone else. Then, out of spite, using positions given to senior researchers like grant and tenure review committees or editing a journal, an older scientist could shoot down grant proposals, reject papers, or decline to award that younger scientist tenure (not being awarded tenure is usually the equivalent of being fired). Anonymity allows space for constructive criticism. And that doesn't just protect them from retribution over poor reviews, but from sexism or racism as well.

Open access

Laëtitia Buscaylet / Flickr

Kay Tye is a professor in MIT's department of Brain and Cognitive Sciences. She's a big-name scientist, with piles of citations. She works in a hot field at one of the most prestigious, grant-awarded universities on the planet. She has tenure. Recently, she tweeted that she was going to start signing her name on her reviews "to promote transparency, accountability and fairness."



That's very noble! Science is horribly opaque in many ways. The same anonymity that protects junior scientists can leave paper authors wondering why their work got negative reviews or how to navigate a politically tense field. Tye cited Leslie Vosshall, another neuroscientist, who signs her views because she "decided to be ethical."

Cons of coming forward

But many people reacting to Tye's announcement didn't see a noble act but, rather, an empty gesture. Some people said that knowing the names of your reviewers isn't actually useful information. Others continued to worry about bullying and revenge-taking. Still others worried about unconscious transactions, where a positive impression left by a signed review, even by someone the authors have never met, could result in a positive review in exchange if the roles get reversed.

Fear of retribution can't be dismissed. I'm a straight, white man and I would never sign a review. I would only stand to lose from doing it (what do I care about appearing ethical? I can't pay rent with ethics). If someone like me is afraid, imagine the consequences a woman, a person of color, an LGBTQ+ person, or another marginalized minority might face for criticizing a vindictive author. Even Tye herself admitted she did not recommend the move for younger scientists.

The craziest thing is that young scientists have been shown to give the best reviews, so encouraging them to actually do them should be a priority. Improving the peer view process is a great idea – it can feel like a kangaroo court, where faceless judges pass whimsically harsh sentences. But I doubt signed reviews will help much.

Judge, jury, peer reviewer

Paul Wright / Flickr

This debate goes back a long way, and there are other ways peer review is being improved. Preprinting, where an unedited, unreviewed version of a paper is simply deposited online where anyone can read it, has been a staple in math, physics, and astronomy for decades and is becoming mainstream in biology as well. This helps circumvent the inherent issues in reviewing: no matter what happens in review, the data gets out there.

One particularly interesting approach is at the journal eLife. There, responses are not the usual handful of sometimes contradictory opinions. Instead, reviewers are allowed to communicate with each other in a chat room, and they all come to a single, unified opinion (yes, like a jury). The British Medical Journal, for the last 20 years, has had partially open review, where reviewers names are published if the paper is accepted, but not if it's rejected. The American Naturalist, in contrast, has gone the other way. There, review can be double-blind, where both the paper's authors and its reviewers are anonymous.

There's no such thing as a professional science critic, no Michiko Kakutani whose whole job is to opine on research. Until there is – New York Times, please answer my calls – peer review will have to do. It should be productive and helpful, and not grounds for back-stabbing and revenge.

Comment Peer Commentary

We ask other scientists from our Consortium to respond to articles with commentary from their expert perspective.

Josh Peters

Biological Engineering

Massachusetts Institute of Technology

Seems like the biggest takeaway for me is a lack of convincing evidence all-around.

It seems like there are also more cons not to come forward as an author based on this study by Nature on their journals. Double-blind reviewed studies are less likely to be accepted. Some have suggested that choosing double-blind review points to “something being hidden.” If you consider this study replicating the NIH review process, which showed essentially zero agreement among reviewers, the system seems overdue for a change.

But this other article just popped up on Twitter: former director of the Danish National Research Foundation proposes evidence that, in contrast, researchers do enjoy anonymous submission.

Although it seems most young researchers are frustrated with the system, I doubt most, including myself, would go against the status quo considering the risks you mentioned in the article.

But, I’m glad to read about some senior researchers making moves to change the culture. With the hierarchical structure you explained, I think the culture change needed to further promote evidence-based, unbiased review in science will come from the top.

Cassie Freund

Ecology

Wake Forest University

I’m in the double blind camp, personally.

I did not find the NIH study that Josh mentioned too surprising. Funding often comes down to luck. 

Dan Samorodnitsky

Biochemistry

Carnegie Mellon University

I obviously have no answers, but I’m with Cassie: I think double-blind is the way to go. I mean, if you were doing a study on whether there’s bias in paper and grant acceptances, you would double-blind it. The notion that requesting it amounts to “something being hidden” makes me want to go kick pebbles down the road and stare off at a sunset. You’d think that a scientist would see that something they already do in the lab to protect against bias would be really helpful in another arena, to protect against bias. You are the problem, buddy. You’re already showing bias.*

*not you, Josh

Devang Mehta

Genomics

University of Alberta

I personally think signed reviews are the way forward. Anonymity, like you say, provides cover for the kind of nastiness and nonsensical criticism that no one (junior or very senior) would ever publicly attach their name to. Conversely, I think signed reviews will encourage more politeness all around –constructive reviews that stick to the science rather than politics or personal attacks.

One way to shield junior and minority scientists while still promoting polite disagreement rather than political ‘paper-killing’ could be to keep both authors and reviewers anonymous during the review process with both identities and review reports published along with the final manuscript. eLife’s approach is one step towards this and I like it a lot.

Cassie Freund responds:

That’s reasonable too. I’ve never submitted anything to eLife so don’t really know about their system. Editors should be playing a role here too, though, especially in monitoring reviewer feedback to junior scientists just starting to publish. They should filter out comments that are unnecessarily personal/destructive/vindictive, and reviewers who submit those should not be asked to review again. I think some editors do this on their own already.