Portable peer review and the manuscript cascade


I really hope portable peer review picks up speed.

It’s normal for people to shoot high with submissions. Start with a journal that feels like a little stretch, and then work one’s way down the tiers of impressiveness.

I do it myself, sometimes, though this game gets weary and seems rather wasteful of everybody’s time. But this is part of the standard approach among collaborators and coauthors. As an editor, I see the papers that I cannot accept work their way down the tiers. Sometimes, papers that I have already reviewed end up in my hands to handle.

As a reviewer, I commonly end up getting asked to review the same paper for two different journals. (I don’t accept most reviews that come my way, for if I did, I wouldn’t have time for anything else. I manage to do at least a couple reviews per submission of my own, so I don’t feel that I’m a parasite on the system.)

Some of the papers that come my way are so obviously up my alley, it’s no surprise that an editor thought of me as a reviewer.

But other times, a paper comes my way that is within my bounds, but a little from left field. This isn’t a surprise. When editors are looking for reviewers, the process is haphazard and potential reviewers emerge from a variety of avenues.

What is weird, though, is when the slightly-from-left-field papers come to me multiple times. I once got the same paper — essentially unchanged from its the initial submission — from four different journals! But twice is not uncommon, and thrice has happened once or twice (but this all gets blurry). So, how the heck is it that I keep getting these same papers from different editors and different journals, when they’re not closely tied to things I’ve published?

The most parsimonious explanation is that the authors have identified me as a potential reviewer. Editors do use recommended reviewers in the mix, if only because it’s so hard to find people to accept a review that hitting up the recommended ones is a valid part of the mix. (As an editor, I sometimes use a recommended reviewer, as long as I have another one that I select independently.)

This is where it gets awkward. Let’s say I take on a review. Regardless what my review says, this manuscript might get rejected. Then, let’s say I see the same manuscript come to me from a different journal. What to do?

What I typically have done in the past is to reply, letting the editor know that I’ve reviewed a prior version of this paper, and if they want me to furnish them a review, I’m available to do so. I can see how some journals would want a review independent of prior reviews. Pretty much everybody says “yes, send the review along.”, They’re glad to have it in the mix.

But here’s the thing: nowadays, most of the papers that I review for a highly-ranked journal, I end up seeing again somewhere in the lower tiers. Since highly-ranked journals also have high rejection rates (because rejection results in the perception of quality), then most papers submitted to highly ranked journals will be seen lower down.

At least some of these times, I’m betting that I’m one of the preferred reviewers. As a reviewer, I try hard to be constructive and as fair and positive as possible. (I don’t like to make a specific recommendation to the editor, I think that’s their job to come to a decision based on my qualitative opinion.) But I’m not always sunshine and rainbows.

When I get the same paper for review more than once, then there’s a decent chance that the authors are listing me as a preferred reviewer on a resubmission to a new journal without realizing that it might be my review that informed an editor’s decision to reject a paper. If the authors keep listing me as a reviewer, and they keep getting the same review, then odds are that they’ll do the math.

The upshot? Some while ago I decided that I don’t want to review manuscripts I have already reviewed for a different journal. I don’t want to run the risk of pissing off someone who listed me as a preferred reviewer, even when I think my review is one that would be seen as a positive one by the editor. I realize this results in even greater inefficiency in the system, and this does bother me.

One solution is portable peer review: to conduct a review process that is not tied down to a single journal.

A number of journals — including Biotropica — are accepting prior reviews from other journals. Ain’t that grand? I don’t think authors are making avail of this option much yet, but I think it’s a good one.

There are also organizations that conduct portable peer review, like the folks at Axios Review. (For more about Axios, check this out from Jeremy Fox.) I have a paper in review with them now, and I’m looking forward to seeing how the process works. There are a bunch of other startups doing this, like Peerage of Science and Rubriq. (The latter is involved in a payola scandal with Nature Scientific Reports.)

I do have a big misgiving about portable peer reviews. There’s pretty good evidence that reviews are, to a certain extent, a big crapshoot. Whether or not an academic work gets accepted has a huge element of randomness, and a lot of that is because of what happens in the review process. If reviews are portable, then a randomly unlucky draw will stick with a paper, which is a bummer.

But I think the bummer of portable peer reviews is when the role of peer review gets distorted by the scientific community. In principle — and sometimes in practice — peer review isn’t there to decide how important or sexy a manuscript is. It’s there to verify that the work is sound. In that case, as long as peer reviewers perform that role, then portability should be fine! The rub is that editors nowadays are using reviewers to predict whether a paper will be “impactful.” Whether it’ll help boost the impact factor of the journal. Maybe portable peer review might help us get away from that mindset, and the editors that want to play their selectivity games can rely on their own judgment as much as the reviewers? Probably not, but one can dream.

I realize that those who favor non-anonymous reviews will argue that this problem of non-portability of reviews is merely an artifact of anonymity. Okay, that’s a great point. But since anonymous is how I roll most of the time, and that’s true for a lot of us, then this remains an issue.

9 thoughts on “Portable peer review and the manuscript cascade

  1. Terry, to date less than 10 authors have taken advantage of Biotropica’s option to submit prior reviews from another journal and a cover letter explaining the changes made in response. Of those that have I would guess the review process has been sped up in about 50% because we send to only a single reviewer (plus Subject Editor) to examine changes. The others have gone out for full review but referees have benefit of seeing prior reviews which I think is quite helpful.

    I’m waiting for more submissions to do an analysis, but if I were to guess I would say that for those that submit prior reviews (A) time-to-decision is lower and (B) acceptance rate is higher.

  2. Interesting post Terry. I also turn down review requests for manuscripts that I previously reviewed for another journal. I think the authors deserve a fresh review and viewpoint when submitting to another journal. Regarding, the randomness of paper acceptance, did you see this article in the latest ESA Bulletin titled Writing Your Way into High Impact Factor Journals (http://www.esajournals.org/doi/abs/10.1890/0012-9623-96.2.312). Apparently, rewording your title increases your odds of paper acceptance.

    Read More: http://www.esajournals.org/doi/abs/10.1890/0012-9623-96.2.312

  3. While some folks like to resubmit the same manuscript practically verbatim to another journal following rejection, others wisely use the opportunity to improve the manuscript based on the first reviews. The revised manuscript may be recognizable to a former reviewer, but should have the chance to be reviewed with fresh eyes. In this case, your practice of turning down previously reviewed manuscripts would be more appropriate than having a review of a older version tacked on to it.

  4. We talked about this at work the other week, and the consensus (from authors and editors alike) was that more often than not, the paper still went out for review. I seem to recall that being because the journal didn’t know the identity of the reviewers, and that’s still a commodity in those circles. And for $250US, not exactly cheap.

  5. That’s a good point. As Emilio pointed out, Biotropica still gets on external review when you bring in your reviews from another journal. This time I’m with Axios, for what it’s worth, I’m getting it for free. They told everybody who attended last year’s Ecological Society of America meeting that they’d get this deal. I just decided to cash that in now. I suppose they’re doing it to drum up business. If I like it and want to use it again, I’d probably be willing to pay for it out of pocket, or find the cash from somewhere in my university, but a lot of people don’t have that option. I wouldn’t want review portability to be something that separates the well-supported from the less-well supported.

  6. We (at Axios) definitely recognize that peer review can be a crapshoot: three reviews are a tiny sampling of the possible opinions about a paper. Of course, getting e.g. ten would be very time consuming, and much of that effort would be wasted if the paper has basic problems that any competent reviewer could flag up. We’d then need to send the resubmission back to ten people again, to see whether the issues had been addressed. We’d iron out the randomness (to some extent), but greatly increase the overall burden on the community.

    One of the strengths of independent peer review is that the paper isn’t being reviewed at a journal – this allows us to explicitly separate the strengths and weaknesses of the paper from its perceived fit to a particular journal. Moreover, the authors don’t have to use our referral process – if the comments are seen as unreasonably negative, the authors are free to use them to improve their paper and then submit to whichever journal on their own.

  7. Alex: We (Axios) definitely do send the reviewer identities to the journal, and half of the time they don’t send the paper back out for review (which seems pretty good to me).

    The key is that the journal has seen our reviews and still wants the paper to be submitted: the paper is then more like a resubmission or even a revision. It’s not widely known, but the acceptance rate for resubmissions (which go back out for review) is ~ 75%, even at selective journals. This is because resubs very rarely get rejected for being out of scope or insufficiently novel, and instead the decision is all about how well the authors revised the paper.

  8. The practice of working one’s way down from the top journals is really irritating to me. I was fortunate to publish a few papers in high impact journals early in my career, but I just don’t see the point of it anymore. It made sense back when you had to go to the library and pull journals off the shelves to see what’s new: most people would only bother to look at the big name journals. However, visibility is no longer an issue thanks to the development of online databases like PubMed, Scopus, etc. Even papers in obscure journals can easily be found. The working down process is a waste of everybody’s time and delays the communication of new results.

Comments are closed.