Finally. There are journals publishing quality peer-reviewed research, but leave it to the reader to decide whether a paper is sexy or important. Shouldn’t this be better than letting a few editors and reviewers reject work based on whether they personally think that a paper is important or significant?
This newish type of journal uses editors and reviewers to assure quality and accuracy. The biggie is PLoS ONE. A newer one on the block is PeerJ. Another one asked me to shill for them on this site.
The last few years have seen a relatively quick shift in scientific publishing models, and there has been a great upheaval in journals in which some new ones have become relatively prestigious (e.g., Ecology Letters) and some well-established journals have experienced a decline in relative rank (e.g., American Journal of Botany). These hierarchies have a great effect on researchers publishing from small ponds.
Publishing in selective journals is required to establish legitimacy. This is true for everybody. Because researchers in small teaching institutions are inherently legitimacy-challenged, then this is the population that most heavily relies on this mechanism of legitimacy.
Researchers in teaching institutions don’t have a mountain of time for research. Just think about all of the time that could be spent on genuine research, instead of time wasted in the mill of salesmanship that is required to publish in selective journals. (I also find that pitching research as a theory-of-the-moment to be one of the most annoying parts of the business.)
With new journals that verify quality but not the sexiness, we can hop off the salesmanship game and just get stuff published. Sounds great, right?
After all, the research that takes place at teaching institutions can be of high quality and significant within our fields. But, on average, we just don’t publish as much. That makes sense because our employers expect us to focus on teaching above all else.
Since we’re less productive, then every paper counts. We want to get our research out there, but we also need to make sure that every paper represents us well. What we lack for in quantity, we need to make up for in (perceived) quality.
How do people assess research quality? The standard measure is the selectivity of the journal that publishes the paper. It’s natural to think that a paper in American Naturalist (impact factor 4.7) is going to be higher quality than American Midland Naturalist (impact factor 0.6).
People make these judgments all the time. It might not be fair, but it’s normal.
And no matter how dumb people say it might be, no matter how many examples are brought up, assessments of ‘journal quality’ aren’t going away. No matter how much altmetrics picks up as another highly flawed measure of research quality, the name of the journal that publishes a paper really matters. That isn’t changing anytime soon.
The effect of paper on the research community is tied to the prestige of the venue, as well as the prestige of the authors. Fame matters. If any researcher – including those of us at teaching institutions – wants to build an influential research program, we’ve got to build up a personal reputation for high quality research.
Building a reputation for high quality research is not easy at all, but it’s even harder while based at a teaching institution. Just like having a paper in a prestigious journal is supposed to be an indicator of quality research, a faculty position at a well-known research institution is supposed to be an indicator of a quality researcher. Since our institutional affiliations aren’t contributing to our research prestige, we need to make the most of the circumstances to establish the credibility and status of the work that comes out of our labs.
If journal hierarchies didn’t exist, it would be really hard for researchers in lesser-known institutions, who may not publish frequently, to readily convince others that their work is of high quality. Good work doesn’t get cited just because it’s good. It needs to be read first. And work in non-prestigious journals may simply go unread if the author isn’t already well known.
If journal hierarchies somehow faded, it’s not as if the perception of research quality would evolve into some perfect meritocracy. There are lots of conscious and unconscious biases, aside from quality, that affect whether or not work gets into a fancy-pants journal, but it is true that people without a fancy-pants background still can publish in elite venues based on the quality of their work. This means that people without an elite background can gain a high profile based on merit, though they do need to persevere though the biases working against them.
If journals themselves merely published work but without any prestige associated with them, then it would be even more difficult for people without well-connected networks to have their work read and cited. It wouldn’t democratize access to science; it would inherently favor the scientists with great connections. At least now, the decisions of a small number of editors and reviewers can put science from an obscure venue into a position where a large audience will see it. On the other hand, publishing in a journal without any prestige, like PLoS ONE, will allow work to be available to a global audience, but actually read by very few.
If I want my work to be read by ecologists, then publishing it in a perfectly good journal like Oikos will garner me more readers than if I publish it in PLoS ONE. Moreover, people will look at the Oikos paper and realize that at some point in its life, there was a set of reviewers and an editor who agreed that the paper was not only of high quality but also interesting or sexy enough to be accepted. It wasn’t just done well, but it’s also useful or important to the field. That can’t necessarily be said of all PLoS ONE papers.
Not that long ago, I thought that these journals lacking the exclusivity factor were a great thing because it allowed everybody equal access to research. What changed my mind? The paper that I chose to place in PLoS ONE. I chose to put a paper that I was really excited about in this journal. It was a really neat discovery, and should lead to a whole new line of inquiry. (Also, the editorial experience was great, the reviewers were very exacting but even-handed, and the handling editor was top notch.)
Since that paper has come out just over a year ago, there have been a number of new papers on this or a closely related topic. But my paper has not been cited yet, even though it really should have been cited. Meanwhile they’re citing my older, far less interesting and useful, paper on the same topic from 2002.
Why has nobody cited the more recent paper? Either people think that it’s not relevant, not high enough quality, or they never found it. (Heck, the blog post about it has been seen more times than the paper itself.) Maybe people found it and then didn’t read it because of the journal. It’s really a goddamn great paper. And it’s getting ignored because I put in PLoS ONE. I have very little doubt that if I chose to put it in a specialized venue like Insectes Sociaux or Myrmecological News, both good journals that are read by social insect biologists, that it would be read more heavily and have been cited at least a few times. This paper could have been in an even higher profile journal, because it’s so frickin’ awesome, but I chose to put it in PLoS ONE. Oh well, I’ve learned my lesson. There are some papers in that venue that get very highly cited, but I think most things in there just get lost.
I would love for people to judge a paper based on the quality of its content rather than the name of the journal. But most people don’t do this. And I’m not going to choose to publish in a venue that may lead people to think that the work isn’t interesting or groundbreaking even before they have chosen to (not) read it. I’ll admit to not placing myself on the front of reform in scientific publishing, even if I make all of my papers immediately and universally available. I have to admit that I’m apt to select a moderately selective venue when possible, because I am concerned that people see my research as not only legitimate but also worthwhile. I’m not worried that my stuff isn’t quite good, but I want to make sure it’s not done in vain. Science is a social enterprise, and as a working scientist I need to put my work into the conversation.
Good post – interesting ideas – must admit I tend to go for the older, more traditional journals because despite the fact that people tend to use search engines, I still think most people read/cite papers from those journals rather than the less selective ones
I’m not sure where your professional assessment of the scientific publishing landscape ends and your personal disappointment about the performance of your paper begins in this post.
I didn’t include a professional assessment of the publishing landscape. If I did, though, it definitely would be shaped in part by personal experiences.
I’m not sure what constitutes ‘paper performance.’ I am disappointed by the fact that it is not getting read. It’s not about me, I don’t think, because my older less relevant paper is still getting cited.
I think a professional assessment would look very different than this blog post. I would consider the possibility that papers in PLoS ONE are less read and cited, but that would involve data, rather than anecdotes. And there are tons of data, and seeing a professional assessment would be really interesting.
Thanks for the post. I find your experience publishing in PLoS One interesting. I have a difficult time keeping up with the TOCs of the five or so journals I currently subscribe to. I find myself citing PLoS One papers however I seem to find these by looking up researchers whose work I already know (i.e. “famous”). I am much more likely to read a paper published in Ecology Letters by a lab group I don’t know well than one in PLoS One. I could see how a major shift in the field to PLoS One style publishing system might exacerbate the problem.
I want to be clear that I really support these endeavors. However, it seems as if the open access journals are up and running just fine. For these journals to do a better job of inclusivity, the filters are what needs to better developed. I am excited to see how this develops moving forward. Working to include more voices in the conversation should be a high priority in our field.
This issue is independent of open access. There are plenty of highly selective open access journals.
I think this is mostly an issue of finding the audience for your paper, and less about the perceived quality of the journal (even if this can also be a factor). The majority of ecologists probably do not do not get subject=ecology TOCs from Plos One, while most skim through the TOC from “Ecology”. Until this changes the value of publishing in established/traditional journals will remain. Maybe there are researchers that filter search results from targeted searches in e.g. Web of science based on journal name (e.g. by removing ones from Plos One), but (I hope) that this is less of an issue.
I have had a similar experience with a paper I published in Plos One last year. It was among the first in the field and since then numerous papers have been published on the subject. However, the paper has yet to be cited. What’s more frustrating is that in some of the publications since my own, the authors state something like, “this is the first…”, which is completely false.
I honestly don’t understand the bias against Plos, or a variety of journals for that matter. When I’m looking up literature for a manuscript, I use databases that pull from many, many journals. So, if I find a paper that is an appropriate citation and is quality science, I’ll cite it, regardless of the journal in which it’s published (for the most part). I agree that there is definitely journal bias our there; however, as is with impact factors, I doubt it will go away anytime soon.
Interesting post Terry. I am at a small teaching institution, but a bit old-fashioned in the sense that I tend to go for the established journals, I guess. I have never worried too much about the absolute impact factor or reputation of a journal, except within my discipline (entomology). In other words, my main focus has been to target an audience. Nevertheless, I have papers that have not been cited at all in spite of being in the “right” journal, while others in the same journal have. I believe that there is a snowball effect, so once a paper starts getting some citations, other authors are more likely to find it, and they cite it as well. It would also seem that in this day and age you would find any paper that is indexed and with a descriptive and relevant title, so the outlet should be of less consequence than in the past. One year after publishing may not be a good marker, as the editorial process for many if not most “normal” journals take that long from submission to “in print”.
In terms of the real high-flying journals, I agree that they sometimes make mistakes by accepting papers because they are sexy and likely to draw attention rather than based on quality. A recent example in my discipline was in American Naturalist, where I believe the paper in question was of such dubious quality (lack of replication, inferences drawn on irrelevant data etc.) that I doubt it would have passed review in most entomological journals.
So far I’ve reviewed two papers, submitted many months after my 2012 one came out. Both cited my 2002 paper but not the new one. It was clear from the discussions that the authors were not even aware of the finding in the 2012 paper. Did they not read it because they didn’t know it existed, or because they saw the paper but didn’t read it because of the journal it was in? That’s an important question and I don’t know the answer.
By the way, was I that kind of reviewer to tell the authors to cite me? In these cases, you bet! I also signed the reviews. I wouldn’t prescribe citing my own work without (overtly) identifying myself.
You wrote: “By the way, was I that kind of reviewer to tell the authors to cite me? In these cases, you bet! I also signed the reviews. I wouldn’t prescribe citing my own work without (overtly) identifying myself.”
Good for you – I see nothing wrong with that. Incidentally, if you recommend you own papers in a review it tends to give you away, so it makes sense to sign the review…….
Interesting post, Terry. Thanks for writing it as it does provide an alternate perspective.
Along with Staffan, I have never really paid much attention to where work from my lab gets published in terms of impact factors, etc. I tend to publish where I think that my paper will get the most “relevant” eyes. In my case, relevant eyes are mainly among the entomological community. So that means that the journals that I have published in, for the most part, are not those with particularly high impact factors (although I have a few papers in some “high quality” journals by that metric), but rather in journals that those in my field have tended to read.
I also come at this from another perspective – not just the person at the small institution doing “niche” research – but also as the editor of a very small journal. The Journal of the Entomological Society of BC has now been around for over 100 years. We publish a few papers a year, and those are all rigorously peer reviewed. Many of the papers are not on topics that would excite any editor at PNAS, let alone Oikos or even Environmental Entomology. But they are papers that are highly relevant to entomologists in the Pacific northwest. And they get read and cited as such. The journal is indexed on a few systems and gets picked up by Google Scholar. And I suspect that that’s where much of our “indigenous” readership comes from.
So, putting those two together in a contemporary context, we need to consider the impact of things like Google Scholar compared to TOC alerts. And we need to think back to when TOC “alerts” were just the journal arriving in your mailbox (prior to email). Emailed TOC alerts likely changed the landscape for what papers got noticed, as anyone could sign up for any journal, not just what they and/or their institution had the $ or sense to subscribe to. And now things have again changed and TOC alerts, while still used by many, are becoming a bit of a thing of the past. To be perfectly blunt, I don’t receive a single TOC alert in my inbox. I find their noise:signal ratio to be too high.
There are many other means (Twitter, the Google Scholar citation system, etc.) that suggest what I should read. Sure, that probably means that I miss some items, but missing items has always been a fact of life and who can keep up with everything anyhow? What it does mean is that I (mainly) can care less about the source journal.
That is, if I am researching a topic or if something interesting turns up in one of my “feeds,” I’ll consider it. At that point I’ll also take a look to see what the venue is. But if that checks out (i.e., if it’s JESBC or the Florida Entomologist or Nature or PNAS or something along the “legitimate” spectrum), I’ll pay attention. If it’s something obviously sketchy, I won’t. So, yes, venue matters… but not in terms of prestige, and only after I’ve found something that I think might be useful.
Quick aside… our lab recently published in PLOS ONE. The paper has only been out since 1 November and it has almost 600 hits at the time of this writing, with almost 10% of those as PDF downloads (http://www.plosone.org/article/metrics/info%3Adoi%2F10.1371%2Fjournal.pone.0077777). So someone is paying attention. I’ll be watching to see if our citation experience is the same as yours.
I’d be interested to see, too! (I put on my calendar to ask you a year from now. Seriously.).
I agree, it’s about finding the audience as much as it is about impact factor. One of my better cited papers is from Revista de Biologia Tropical. People somehow found it. (Anybody perusing my CV would see I don’t have an issue with publishing in highly specialized and region-specific journals.)
I’m not sure how much of the problem is whether people see it, or whether they choose to read it, or whether they choose to cite it. With most papers, I would just as easily conclude that they didn’t like it or didn’t think it was good or relevant enough. But based on the content of this one, I think it’s not the last one. I think it’s not getting read in the first place perhaps because it went past unseen, as others have suggested.
Things are changing in terms of how people filter the literature–but not nearly as fast as one might think if one, say, hangs around the twitterverse a lot. Some unscientific data:
http://dynamicecology.wordpress.com/2013/10/29/survey-results-how-do-you-find-papers-to-read-when-you-cant-do-a-search/
If you want your paper to have an audience, you definitely need to publish in a journal that that audience pays attention to. I know there are some who wish that weren’t true, and maybe someday it will cease to be true, but right now it’s still very true.
I am a fan of the PLoS model but this author-pays model is difficult for researchers at teaching institutions without grants. Yes they can (and do) waive fees but I haven’t tested the frequency that they might do this.
But another question is, Why obsess about “establishing legitimacy”? Sure, it feels good to have one’s work recognized by peers but if this were a priority, then why accept a position at a teaching institution with little infrastructure to do “legitimate” research? Or, if establishing a reputation amongst your colleagues is that important, why handicap yourself with a teaching position? Any career choice comes with lots of trade-offs but once a path has been chosen it doesn’t seem very productive to stress/obsess about consequences that arise because of the choice. This is like a professor at Stanford stressing about never getting to see their kids play soccer because they are always in the lab.
I didn’t realize it came off as an obsession. I don’t feel that I obsess about it.
Doesn’t it matter that what you do professionally is legitimate? The opposite is having research that is not perceived as legit. Research that is not legit isn’t respected, isn’t read, isn’t cited, and has no effect on the field. Why do it other than for fun? Now, I do research because it’s fun, but I’d feel both selfish and silly if it wasn’t accepted as legitimate.
“perceived as legit” is the key phrase. What makes one’s work legit? Simply engaging undergrads in the process, but not publishing? That’s probably pretty legit to the undergrads. Or publishing in a non-elite journal? That satisfies the promotion committee at my university. Or does it need to be published in an elite journal so that more of my peers see what I do and think of me as a legit peer? But really, who cares about stickleback functional morphology, even if its published in Science or nature? Maybe legit research needs to be on something that matters more than to the few folks that care about stickleback. So research in tumor biology. Or global carbon cycling. Or alternative battery technology. Food for thought. I don’t have the answer (I don’t really think there is an answer).
What makes research legitimate? In other words, what is real research and what isn’t? And how much does the gap between reality and perception matter? Sounds like several more blog posts.
Man, I love the comments. Consistently respectful, pointed, informative.
In my opinion, part of what we need is the scientific ‘elite’ to start putting their attention-getting work into non-selective journals. Grad students, post-docs, and early career faculty NEED to put anything worthy of Nature, PNAS, Proc B, etc., in those journals to establish themselves. The giants who so regularly publish in these venues ought to take the lead (since they have nothing to lose) and put important results into respectable, non-selective journals. Then perhaps more people will pay attention to what is being published there.
On a related note, I’m at a very major American university, and our library won’t pay for access to Nature and Science. If big scientists want to impact the world, shouldn’t they publish in PLoS One where everyone can read it?