Some while ago, I wrote about experiences serving on NSF panels, just to demystify the experience for folks who haven’t been on panels. I received feedback that this was helpful, so I thought I’d turn some focus on one aspect of review that I think merits additional attention.

I’ve served on several different panels by now (and I’m not supposed to tell you which ones, so of course I won’t), so I am prepared to generalize a bit: In most proposals, the broader impacts aren’t good. I’m not thinking of any particular panel, or any particular kind of person or PI or project or whatever. I’m just saying, in general, broader impacts can withstand a lot of improvement.

Yes, some proposals knock broader impacts out of the park — and this hugely impacts how the proposal gets rated by the panel. But most PIs appear to shoot themselves in the feet by having bad broader impacts. Of course, there are a lot of people out there with more information than I have, and who have more experience, so please don’t take my broad statement as fact. It’s just a statement of my opinion, based on the limited information I have, at this moment in time. I’m glad to revise my opinion based on new information or better ideas.

This situation really confuses me. Because nearly everybody submitting their proposal knows the game — that NSF evaluates proposals on intellectual merit and broader impact. While people go whole hog on making the intellectual merit as amazing and robust as possible, the broader impacts are just, well, weak. Or not credible. Or ineffective. Or badly justified. Or unbudgeted.

At this point, you might be thinking, “But I’m a research scientist and if NSF wants broader impacts then they should fund the people who are experts in it.” And I understand this point of view, and have a bit of sympathy with it, but I’m going to give some pushback against it. Keep in mind that NSF does precisely this. Many programs are primarily designed to support people who are doing broader impacts well. However, the vision being communicated by NSF is that every scientist (me, and you, and everybody else who wants to tap into this pool of taxpayer money) should make sure that their funded project results in a proximate impact beyond what gets published in academic journals. If we take NSF money, then it’s our job to be responsible citizens, and make sure our science has broader impacts. That’s part of the social contract when asking for this money, and if you don’t like it, then don’t ask for it. You gotta dance with them who brung you, I think the saying goes.

Contrary to some claims I’ve heard, you don’t need to be a highly trained expert in the activities that you propose to have an excellent broader impacts section. You just need to use the critical thinking skills that you developed while getting your PhD, and apply the same rigor to your broader impacts as you do to the research design. Broader impact sections are typically replete of broad statements that are not backed up with the slightest of evidence. They’re fully of shabby reasoning that, if found in the intellectual merit section, would torpedo a proposal. It’s not hard to be intellectually honest and rigorous with the content you put in broader impacts. That is where you have to start. You can’t choose to believe your own bullshit. You have to convince yourself that what you’re doing is truly impactful before you try to convince others. And you need to convince panelists who probably know broader impacts better than you, so it’s not a good idea to fake it.

One way of making sure your broader impacts are up to snuff is by developing them with someone who is an actual expert in what you’re doing. You’re not expected to be an expert in science communication, or in minority recruitment and inclusion, or in K-12 education, or in gender equity, or whatever your broader impacts are about. But if you want your broader impacts to be taken seriously, you should be, at a minimum, consulting with people who are active practitioners in these realms if you’re putting them in your proposal. I mean, if your project uses radiotelemetry to track animals and you have no prior demonstrated expertise with radiotelemetry, you’d need to bring in a telemetry expert to make your intellectual merit sound credible, right? The same principle applies to your broader impacts.

A conceptually rigorous broader impacts section needs to be grounded in the established literature to demonstrate that your proposed activities are following best practices. Whatever you’re doing, it’s been studied and it’s in peer-reviewed journals. If you’re the PI, you need to become familiar with that literature, or at the very least, work closely with a collaborator who bakes it into the proposal with you.

There are many ways to design great broader impacts, so I’m not going to give some kind of recipe. Instead, what I’ll do is diagnose some major problems that I’ve seen. A lot.

If you’re developing a curriculum for K-12 students, then you need to do this in consultation with K-12 curriculum experts. It needs to fulfill a demonstrated need and be aligned to state standards (and ideally NGSS if your state isn’t using NGSS). It also should be practical enough, and simple enough, and not involve a lot of sophisticated supplies or equipment, so it can remain impactful even after you’re done with it. Make sure that your teachers will want this, for real.

If you’re training students from underrepresented groups, you need to demonstrate that these students will receive appropriate and high quality mentorship to support their training. You need to demonstrate that your recruitment plan is pragmatic, and that the trainees will be in a supportive environment, and that you’re taking steps to further their careers beyond being in your lab. If you’re recruiting from other institutions, you need to demonstrate evidence that your recruitment plan will work, with evidence that you have contacts at those institutions who will help you fulfill your agenda. Do you have specific targets or benchmarks that you think are appropriate?

If your outreach plan involves blog posts, be sure to communicate the purpose. Is it for training students how to do science communication? Then requiring them to write won’t do much unless you provide the relevant professional development experiences. Is the blog there to reach the public? Then you need to demonstrate how this blog will have success, with credible fact-based estimates of expected readership and reach to the target audience.

Is someone on the project planning to write a children’s book? Have you demonstrated a need for this book on the market? Unless you’re paying to get this book into the hands of kids (which will get expensive fast), then this probably will have very little impact unless you can show you’ve landed a publication contract with a major children’s book distributor.

Are you trying to promote equity for women in STEM by providing training opportunities to women? Keep in mind that, in Biology, women are not underrepresented among students, and the obstacle is not recruitment, but limited professional advancement, driven primarily by harassment. So your broader impacts to improve equity for women should involve a plan to change the culture in the academic environment.

Okay, only two more big points.

How are you providing any kind of assessment or reassurance that the broader impacts will be implemented as planned? Does your proposal have specific metrics or indicators that you will use in your project reports to communicate whether you were successful? Have you considered involving a person who can assess your broader impacts? If you don’t have a way to assess your broader impacts, then it’s hard for the reviewer to be convinced that you’re taking them seriously.

Your budget communicates your priorities. If you want reviewers to believe that you’re serious about your broader impacts, that means you’ll make space in the budget. If you’re doing outreach to schools, is the cost of transportation and supplies involved? Will the students have the time in the grant to perform this work? If you’re recruiting underrepresented students, have you budgeted the time and resources to make sure that those who are mentoring these students are being trained to become effective mentors? Are you paying for these recruits to travel to conferences? If you want your broader impacts to be taken seriously, put them in the budget.

I realized this is a relatively unorganized half stream-of-consciousness set of thoughts about broader impacts, but, hey, that’s sometimes what a blog is, when it’s late Sunday night and I’m writing a post for Monday morning. I usually make a point to avoid write something here that sounds like advice — and this is a rare exception. It’s was just a lot easier to phrase my opinions in the form of prescriptions.

If you are serious about upping your broader impacts game, I suggest checking out NABI.