Mobile Edition
Print
Get the Mag
Weekly UpdatesQuestion: When would you recuse yourself from reviewing a proposal? A proposal submitted by someone you know professionally? Socially? Someone you had the opportunity to meet at a conference?
Answer: I would recuse myself from reviewing proposals when there is or could be a perceived conflict of interest--for example, in cases where I am working with someone (whether at my institution or another institution), or have a close professional relationship with someone. However, simply knowing people or socializing with people shouldn't result in a reviewing conflict, as these are commonplace occurrences. Indeed, part of "continuing education" for researchers involves getting to know people to discuss research ideas and ask questions about research methods, analyses and interpretation. Contacts are made by phone and e-mail, and at conferences and meetings. If reviewers were excluded from study sections or proposal reviews because they were acquainted with other researchers, there would be few people left to review.
Question: Is it always good to present a positive image to the reviewers? When do you feel "confidence" crosses the line and becomes pretension/pomposity?
Answer: There are all sorts of people in research (as in any field). Personalities and demeanors vary widely. I have a difficult time interacting with people whose style tends toward lectures rather than an exchange of ideas and opinions. Collegiality and good manners usually define the line for me.
Question: Have you ever been impressed or especially moved by a proposal that you have read? What touched you?
Answer: Proposals are hard to write and when I run across a well-written proposal, I am impressed. "Well-written" is defined for me as starting with a clear set of goals or aims, setting forth convincing research evidence for the importance of those goals for moving the field forward or solving an important problem, and then proposing a methodology that should produce the data needed to address the goals or aims. I have read many proposals, and only a few have been really well written.
Question: Have you ever regretted any decisions you've made or actions you've taken relative to reviewing a proposal? What did you regret about your actions and why?
Answer: I sometimes have second thoughts, particularly when I believe that a proposal should get a poor evaluation. That happened recently when I evaluated a proposal as extremely poor science. I was worried that my review might be harsh or out of line with the other reviewers. I carefully wrote my review, read it over several times before the review (and several times after the review was all over), and felt that the statements written were accurate and fair. The other reviewers were also negative. One was less negative than I was, and the other very similar to my evaluation. I have given negative evaluations before, but for this proposal there were many negative things to write about. I think I worry more about the reviews of proposals that lead to negative evaluations than those that receive positive evaluations. I want to make sure that I have all the details correct. For this reason, I usually read through all the proposals I am sent, making notes on each one. Then I read each proposal again and write the review. During the writing process I will often look up references and read articles, and I frequently check out methodologies and proposed data analyses to make sure that I understand what is being proposed. When I am finished writing the review, I may read it several more times before I submit it as "final." I do this for all proposals, not just the ones that I give negative reviews, but I pay particularly close attention when writing negative reviews.
Question: Have you ever felt that other reviewers had ulterior motives and secret agendas that influenced the way they reviewed a proposal? What can other reviewers do when this happens?
Answer: Occasionally one reviewer will be so different in his or her evaluation of a proposal that the review stands in contrast with that of the other reviewers. Sometimes these reviewers try to force their opinions about the proposal in an effort to change the evaluations of others. However, reviews are fundamentally evaluations of proposal elements and a careful reading of what is actually written in the proposal. Some reviewers will acknowledge that a proposal is missing or has confusing elements, but dismiss those weaknesses by indicating what they believe should have been included or what the writer meant to include. Sometimes the dissenting reviewer has special knowledge--methodological expertise relevant to the proposal's design or content knowledge relevant to its topic--and they can elucidate components that might not be clear to other reviewers. However, with multiple reviewers on each proposal as well as a large study section composed of many reviewers, it is not likely that a single strong-willed reviewer can use a biased evaluation of a proposal that is at odds with the evaluations by other reviewers to radically change the overall evaluations of the proposal.
Question: Do peer reviewers ever "steal" ideas from the proposals they read?
Answer: I have heard of this happening and one PI showed me an instance in which this has occurred, but I do not believe that it happens very often. Certainly I have not heard many complaints about this. In everything that is published or becomes "public" through a review process, there are opportunities for ideas to be stolen. However, it is common in science for people to be researching exactly the same thing using the same methods and publishing the results of their work. I believe that the researcher's "know how" separates the original from imitations.
Question: How does the review process work when there are disagreements among peer reviewers?
Answer: There are often disagreements, and those are allowable. People interpret proposal elements differently, or evaluate the significance of a weakness in the proposal differently. The scores given by individual reviewers can range broadly across the reviewers assigned to each proposal and across the reviewers at the study section who also vote on each proposal. The purpose of the discussion following the reviews of the proposals by the reviewers assigned to each proposal is to discuss differences of opinion and interpretation. Sometimes these discussions result in the reviewers coming closer together in the scores they assign to the proposals, but not always. The evaluation scores of the proposals are given by the assigned reviewers publicly during the meeting first, just before the actual review of the proposal, and then again after the proposal has been reviewed. Making the reviewers' scores public during the meeting tends to keep reviewers honest. In addition, other people at the review are asked to vote within the range of scores stated by the assigned reviewers (such as between 1.8 and 2.3) or to give their reasons if they decide to vote outside that range. * This "out of range" voting occurs sometimes when reviewers differ in how they view a particular element, such as the importance of using a new technique to gather data or using a standard technique.
Question: How has serving as a peer reviewer for NIH helped you professionally?
Answer: I have learned a lot about proposal writing, research methods, and child development, my area of scientific expertise. I have also met and spent a lot of time with a lot of people. I have read a lot of research articles referenced in proposals, and participated in a lot of review sessions* I hope it has all resulted in me being a better and more informed researcher, as well as being a good reviewer. However, getting funded is very difficult* My proposals still get triaged, low scores, and occasionally fundable scores resulting in grants. With all that experience I still don't have the key to consistent success.
Question: How would you advise other faculty members who are interested in serving as a peer reviewer for NIH to proceed?
Answer: I would encourage them to do what I have done. Use the NIH webpage to identify the study section that has proposal solicitations in areas of their expertise, review the list of the current study section members to see who they are and what their areas of expertise are, identify the scientific review administrator associated with that study section, and send an e-mail stating an interest in proposal reviewing, and attach a vita. It is important that faculty and researchers interested in reviewing have publications that indicate their areas of expertise, and it is beneficial if they have written proposals that have been funded, whether by NIH or another agency. NIH is looking for reviewers, and contacting the agency directly is a good way to be considered.
Victoria J. Molfese, Ph.D.
Center for Research in Early Childhood
Room 236
College of Education and Human Development
1st and Cardinal Boulevard
University of Louisville
Louisville, KY 40292
tori@louisville.edu
Joseph Cervelin, MFA
Office of Sponsored Projects
University of San Francisco
2130 Fulton St
San Francisco, CA 94117-1080
415.422.2736
jncervelin@usfca.edu
Pamela F. Miller, Ph.D.
University of San Francisco
2130 Fulton St
San Francisco, CA 94117-1080
415.422.5203
pmiller@usfca.edu
FEED