Prescriptive reviews are a scourge

Standard

How do you spot an academic? Listen for gripes about manuscript reviews.

We all get bad reviews. I’m not talking about critical reviews — we all get those too. I mean: We all get reviews of bad quality.

Some reviews are not written with the care, diligence or maturity that we expect. Even if you’re blessed to avoid these most of the time, you surely know what I mean. Bad quality reviews happen. They can happen a lot.

How do you write a good review? There is a lot of help out there! This guide from the British Ecological Society is top notch. Also it’s worth checking out what Dynamic Ecology and Arthropod Ecology have to say about writing good reviews. I don’t think I can add much to the general genre of “what to put in a good review.”

Instead, I want to point out what a good review shouldn’t have in it. People often have put in the hard to work assemble the elements of a good review, and then spoil it by adding in stuff that gets in the way. What do I mean? Well, some reviewers don’t really understand the distinct roles of reviewers and editors. Or, maybe they understand that role, but want to assume an editorial role when they’ve been merely asked to review a manuscript.

First things first: Reviewers do not decide whether papers are accepted or rejected. That’s what editors do.

I often hear people say something like, “That manuscript I reviewed was so horrible and I rejected it.” That might be rhetorical shorthand for, “I recommended rejection.” Yes, this might be merely semantic. But I happen to think that this reflects, at least to some small extent, how some reviewers approach their responsibility with the wrong mindset.

Should reviewers be recommending rejection, revision, or acceptance? It depends on what the journal asks for. Some instructions to reviewers specifically ask for this kind of recommendation, but other journals are clear that they do not want reviewers to make a recommendation. Others ask you to make confidential recommendation to the editor, but not to the authors. I’m surprised how often — both as an author and as an editor — that reviewers don’t follow these instructions.

Editors solicit reviews to find out if the science is sound, whether claims are supported by the results, and whether the finding is novel or exciting. There are other things that editors will want to know, too.

Here is one thing that editors and authors don’t want: reviews that explain how the authors didn’t do the experiments that the reviewers wish they had done.

It’s not the place of a reviewer to prescribe additional experiments.

It’s not the place of the reviewer to prescribe that the authors test a theory that the reviewer wanted to test instead of the theory the author wanted to test.

It’s not the place of the reviewer to say that the author needs to make a different argument in the discussion.

Reviewers that don’t like how a paper is structured shouldn’t be telling the authors to restructure the paper.

Reviewers who think the introduction is too long shouldn’t tell the authors to cut it, and reviewers who think the methods are missing some important details shouldn’t tell the authors they need to put it in.

If a reviewer disagrees with an argument, it’s not the job of the reviews to tell the author to change the argument. Instead, the reviewer should explain how they disagree in the review and leave it to the editor to tell the author whether to change the argument. Why does this matter? Because:

A review should be a review, not an instruction manual. Leave the directions for repairs and improvements to the editor. It’s the job of the reviewers to tell editors and authors what they think, not what they prescribe.

Is this just a semantics, claiming that instructions for changing a paper are radically different from a set of opinions and facts? Not at all! I think it really changes the quality of a review, and modifies editor’s latitude to make nuanced recommendations to the authors.

When reviewers fill their reviews with a list of prescribed changes, then this places the editor in an awkward position. Editors craft letters to the authors and tell the authors what to do. These usually are one of these:

  • “Revise the manuscript in line with the concerns of the reviewer.”
  • “Be sure to focus on changing X and Y, and explain if you choose to not make further modifications.”
  • “Address the concerns of the reviewer in the cover letter and make changes where you feel they are necessary.”

That’s in editorial code. Let me translate those three sets of instructions:

  • “Do everything the reviewer says.”
  • “Do these one or two things the reviewer says, and some of the other changes might matter, and some might not, but I’m not telling you which.”
  • “You and I know the review is off the mark, so don’t make the changes but be sure to explain it in the cover letter to keep everybody happy.”

You know how it’s hard to find the time to write reviews? It’s the same for editors, too. We recognize that editing is a major responsibility but also it’s a volunteer role as a service for the profession. We need to treat authors fairly and make quality decisions, but also face a number of professional tradeoffs.

Editors often make decisions that piss people off, but we can’t make a habit of soliciting reviews and then ignoring them. In most journals, it’s standard form to include the reviewer on the outcome of the editorial decision as a courtesy.

It’s the job of the editor to tell the authors what changes need to happen.

That job gets a lot harder when the reviewer starts doing that job instead of the editor.

When we receive reviews that have an unsolicited detailed set of prescriptive instructions from a reviewer, that can put the editor in an awkward position. We can’t just write back to the authors and say, “Ignore everything that Reviewer 2 said.” It’s rare to get a review that is so hideous. On the other hand, it’s common to get a review that raises some very important points that need to be addressed, but the same review also raises other points that don’t matter, or might even be off the mark a bit.

It’s not kosher to edit manuscript reviews before they get back to the author, unless they contain obvious bias. So when we return the manuscript to the authors for revision, then there’s the delicate task of letting the authors know how much we agree with the reviewer, without showing any obvious disrespect for the time and effort that the reviewer put in.

On the other hand, life is a lot easier for the editor when a reviewer submits a genuinely good review. You know, a review that doesn’t contain any prescriptions but instead provides a comprehensive evaluation of strengths, weaknesses, and a set of opinions about what would improve the manuscript. With reviews like these, the editor can craft a set of instructions to the authors that prescribes what revisions are necessary, and what ideas need to be addressed, for a revised submission.

When the reviews don’t have prescriptions for specific changes, then the editor can identify what matters in the reviews and pass them on to the authors, without having to dance around the fact that the authors can ignore some of the things in the review.

Of course, editors are supposed to take charge and be leaders — and not let the feelings of reviewers get in the way of appropriate decisions. And that’s what I do. I’ll say “Be sure to address all of the concerns of Reviewer 1 and address concerns X and Y of reviewer 2.” Which I realize might piss off Reviewer 2 when they see it. But that’s a lot better than making a bad decision.

Sometimes when I get a manuscript back, the editor is acting more like an administrative assistant than an editor. They get the reviews, look at them, and pass them along to the authors and say, “make these changes.” If the reviews are negative, the paper gets rejected. If the reviews are mostly positive, then the editor ask for revisions. This is not what good editorial decisions look like.

As an author, what I usually think is: “What changes does the editor really want me to make? Some of these changes are simple, and some are difficult. Some of these I’m fine with, but I think some of them harm the paper. Some of these sell the paper to a broader audience, buts change my ideas in ways I don’t want them changed.” And what do I get from the editor? “Make the changes, do as the reviewers say.”

That then leads to a guessing game with the authors. Authors go ahead and do the revisions. But not all of them. The ones that they don’t want to do, they explain and rebut in the cover letter. But the problem is, some of these revisions might be deal breakers for the editor and some are whims of the reviewer that the editor doesn’t care about. What the authors want is the editorial judgment of the editor. What authors too often get is a transferral of editorial responsibility from the editor to the reviewer, and then the author needs to guess about whether or not the reviewer is going to get the manuscript in review again. This is not the way to run a journal.

Reviewers who provide prescriptive reviews are enabling the transfer of the editorial’s responsibilities to the reviewer. Moreover, reviewers who write prescriptive reviews make it harder for editors to clearly specify which changes need to be made.

What does a well-constructed review look like?

Let’s say as a reviewer, you don’t like a table, and think the information belongs in the body of the text instead. A good review says, “I had trouble following the manuscript and think the information in the table is better presented in the body of the text.” A bad review says, “The authors should delete Table 1 and move the information into the body of the text.”

Let’s say you don’t like the name of a certain hypothesis because it overlooks a piece of literature that you appreciate (or maybe one that you’ve written). A good review says, “This manuscript describes a new Very Interesting Mechanism Hypothesis, though in many ways resembles the Mechanistically Intriguing Hypothesis found in Smith 2012. Readers may track the literature more readily if these hypotheses were given a consistent name.” A bad review says, “The authors need to rename the Very Interesting Mechanism Hypothesis to the Mechanistically Intriguing Hypothesis and also need to cite Smith 2012 (which of course I had nothing to do with).”

Let’s say you think a major argument in the conclusion isn’t supported by the evidence. Then that’s what you should say! And you should explain it. But you shouldn’t tell the authors to fix it. That’s the editor’s frickin’ job.

Some journals are really good about making sure that editors can most readily wield editorial judgment over the caprices of reviewers. But I’ve dealt with some other journals that seem to be happy with editors that have decisions that run just a couple sentences, instructing the authors to follow reviewer instructions. Those are journals I’m not planning to revisit, because when I make revisions, I’d like to know the opinion of the editor first.

What would make the review-and-revise process work better for all parties?

As reviewers, we can write actual reviews, and avoid writing instructions to authors. We can let editors know our frank opinions but give them the professional latitude to make their own mind which of these changes need to happen.

As editors, we can exercise a heavy hand with reviews so that authors do not fuss about changes that will not improve the manuscript, spare the authors the rhetorical gymnastics required to refute obviously useless comments from reviewers, and not be afraid to write decision letters that specifically tell authors to overlook prescriptive remarks from reviews that do not need further attention.

I have yet to serve as an Editor-In-Chief. I think it’s the role of the EIC to set the tone for the journal, revise instructions to reviewers as appropriate, and provide feedback to subject editors about how they are using reviews in their recommendations. Ultimately, the direction of a journal is steered by the EIC. If the editorial process consistently involves overly prescriptive reviews that are passed on to authors without guidance from editors, then the buck stops at the EIC. (I haven’t had a bad experience yet with my EICs, though I don’t have as much editorial board experience as some of you all.)

As you might imagine, this post was inspired by revisions I just finished. I suppose I should have the wisdom to wait to find out whether this third round of revisions is going back to the reviewer. And whether the editor will have any opinion about anything other than what the reviewer thinks.

By the way, if you have somehow have navigated the internet without seeing this relevant clip, this made the rounds a while ago:

8 thoughts on “Prescriptive reviews are a scourge

  1. Hmm…,going to mostly disagree with you on this one Terry. It sounds like the reviews that inspired this post were pretty frustrating for you (you have my sympathies). But if so, I think you’ve overreacted to them a bit.

    Yes, absolutely, it is possible for reviewers to step over the line and try to do the job of editors. Most obviously when reviewers put their decision recommendation in the comments to the author (which every editor rightly hates). And yes, absolutely, reviewers need to fairly evaluate the paper, rather than effectively saying “It would’ve been much better if you’d done a totally different study and written a totally different paper.”

    But I think your advice goes too far in the other direction. As a reviewer, explaining my review often means that I have to be prescriptive. I can’t explain why I think that some bit of the ms should change without also explaining what I think should be done instead. And if I think an ms would work much better if reframed to address a different question, or to focus on only a subset of the questions the authors set out to address, I think I’m doing authors a favor by saying that (and at least in some cases I presume the authors agree that I’ve done them a favor, as they’ve taken my advice even when they didn’t have to). As an editor at Oikos, I did not just want reviewers telling me what was wrong with an ms. I also very much wanted their advice on how to fix it, if it was fixable. And I didn’t just want them giving that advice to me in their private comments to me, I wanted them giving it to the authors as well. Reviewers’ confidential comments to the editor ought to reflect and summarize their comments to the author. I feel the same as an author. The more information I have about where reviewers are coming from, the better I can address their concerns in a revision.

    It seems like a lot of your concern is with the precise phrasing of reviewer comments rather than their substance–e.g., telling authors that something about their paper “has” to change, rather than recommending that it change. Personally, as both an author, and an editor back when I was an editor, I’m not bothered by such phrasing and don’t read it as reviewers overstepping their authority. I just read it as emphasis. Saying that something “has” to change is just a compact way of saying “my recommendation to the authors and to the editor, in the strongest possible terms, is that this really should be changed.”

    I agree with you 100% that a good editor won’t just pass the reviews plus a decision on to the author. A good editor’s decision letter will make clear, at least in broad terms, what the most serious problems with the ms are and what changes are most important to make for the ms to be published. Particularly if the reviews disagree with one another, or if the editor disagrees with some or all of the reviewers. A good editor also won’t care much about the reviewers’ accept/revise/reject recommendations, and certainly won’t just come to a decision by counting the votes, but instead will use the reviewers’ comments to inform his or her own accept/reject/revise decision. And I agree that good editors aren’t as common as they should be, although in my experience they’re pretty common (which reflects positively on the journals I tend to submit to). But I disagree that the way in which reviews are written has any role in causing editors to effectively abdicate decision-making to referees.

  2. Reiteration and clairification:

    I explain that, in my view, this is not merely semantics:

    Is this just a semantics, claiming that instructions for changing a paper are radically different from a set of opinions and facts? Not at all! I think it really changes the quality of a review, and modifies editor’s latitude to make nuanced recommendations to the authors.

    In my opinion, saying “this needs to be changed” is not adequately professional shorthand for “I feel this is a very important change that should happen.” This puts the editors and authors in a situation where they both need to deal with the demands of a person who is in a defined role that is not typically empowered to make demands. At least, as an editor myself, I don’t want my reviewers making demands for changes. I want that to be left to me as an editor. If a reviewer feels strongly about something, then they can explain they feel strongly about it. And then I’ll send a few ibuprofen for the sore muscles from the extra typing.

    I do realize that submission rates are going up as there are so many people submitting so many papers, and it’s getting harder and harder to find reviewers. As a result, I think it has evolved that reviewers really truly are seeing themselves as the actual gatekeepers of the journal and not merely consultants advising the editors. So this post is just pushing back against this phenomenon as it is becoming a new norm.

  3. “This puts the editors and authors in a situation where they both need to deal with the demands of a person who is in a defined role that is not typically empowered to make demands. ”

    Maybe this is where we disagree most then. As an editor, I always felt free to disagree with some or even all the reviewers, no matter how they phrased things. I never saw reviewers as making demands on me as an editor, and if they had I’d have simply ignored them. And so there were times when I’d more or less tell the authors in a decision letter “ignore comment X of reviewer Y, I don’t agree with it” (but phrased in such a way that reviewer Y would hopefully not be offended by my decision or think I didn’t take their review seriously). I don’t think I was unique in this attitude, I think it’s pretty widely shared among the editors at the journals to which I submit. (Note also that I think it’s fairly rare for editors to find themselves disagreeing with reviewers to the extent that they would want to contradict reviewers in the decision letter. More commonly, when the decision letter is brief and just says “revise in response to the reviews”, it’s because the editor mostly agrees with the reviews.) And I haven’t sensed a recent shift in the culture, towards a new norm in which reviewers both are the real decision-makers and see themselves as such. But I don’t know of any survey data on any of this. So it could be that your experiences, or mine (or both!), are somewhat atypical.

  4. I once wrote a review in which I said something like “the data in figure x do not support claim y; instead, the authors should do experiment z.” I immediately heard back from the editor who said that I was not the author’s mentor and shouldn’t tell him what experiments to do – it’s enough to say that he needs to do additional work. I was surprised, but since then I’ve been careful to avoid making detailed prescriptive comments. Still, I can’t help but think this type of thing is very editor-specific.

  5. Terry – I think you are going a bit far here. The vast majority of reviews I get, even though very critical in many cases, result in a better paper in the end.

    I often include highly prescriptive comments in my reviews. My comments typically fall into four categories; this should signal to the author and editor which prescriptive comments need to be responded to.

    Fatal flaws in experimental design or conclusions not supported by evidence. These are near the beginning and flagged as “The authors must….”. If I think more evidence (new experiment) I will tell the authors exactly what I think is needed. If these are not taken seriously normally I recommend rejection (to the editor) on a 2nd round of review.
    Strong suggestions that I try to flag as “the authors should…”. These are things that if done will improve / broaden the papers appeal and impact in some way. While I don’t expect the authors to follow all of the suggestions in a review, I will be somewhat annoyed (wasted time on my part) if all are ignored.
    Minor suggestions.
    New directions. These are usually very clearly flagged as ideas / helpful suggestions for future work, and not requirements for inclusion in the present paper. I think it is one of the value-addeds of the peer review process (and a sign that it is working well) when as an author or reviewer it is possible to exchange ideas in this format.

You can leave a comment anonymously, just don't give your name or email.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s