Mobile Edition
Print
Get the Mag
Weekly UpdatesQuestion: What are the social/power dynamics of the peer review process?
Answer: Actually, proposal review sessions are fun as well as anxiety producing. The other reviewers are a mix of people--some you know, some whose research reputation you know, and some who are total strangers. Some are peers in terms of age and experiences, some are new researchers/scholars, and some are emeritus but still active scholars. At the beginning of the review, the scientific review administrator explains the rules of how the review will proceed, and identifies which proposals have received low initial scores and which will be triaged unless any one reviewer wants to keep the proposal in for discussion. Reviewers introduce themselves, and then the proposal reviews begin. It is often an awkward or stiff process at first because people are self-conscious. These first proposal reviews take the longest (sometimes 30 minutes or more), but the process picks up speed as people relax and get familiar with the procedure. I was at a review session recently that worked great--each of the 2 or 3 reviewers summarized their review and tried not to repeat what the others had said, the critiques were interesting, and none of the reviewers seemed to have a big ego. Reviewers differ in how they deliver their critique--some can be arrogant and appear to try to "score" points by noting weaknesses in proposals, some note weaknesses but also note strengths, and there are a few whose reviews are not very informative, which may be because the topic of the proposal is not in their area of expertise or because they haven't reviewed the proposal very thoroughly. Some reviews are extremely interesting: for example, giving information on historical perspectives on a field or a method that helps but the proposal in perspective (sometimes not in a good perspective because some PIs do not know or reference previous work well and their proposal may duplicate others or may be flawed due to a lack of understanding about what has already been done).
Question: Have you ever seen competitiveness, jealousy, revenge come into play during the review process? How was this dealt with?
I have seen all of those, but not often. Sometimes reviewers really want a certain proposal to succeed or fail depending on the topic, the research design, or the PI. For example, one reviewer clearly had a grudge against a particular methodology, and each time a proposal was discussed using that methodology, the reviewer ranted about its weaknesses. Because there are 2 to 3 reviewers, one person's opinion does not usually sway the entire discussion--even if the reviewer is forceful in tone and behavior during the review. Other people on the study section also wade in with their comments, either supporting or not the forceful reviewer's comments. Often the program officer, the scientific review administrator, or someone else from the agency who is observing the study session will speak up if the behavior or topic of discussion is going in the wrong direction. For example, comparisons of one proposal against another are not allowed; trying to design a study for a proposal whose design is weak is also not allowed. In general, reviewers do not seem to be intimidated by other reviews, possibly because they tend to be very accomplished in their own right.
Question: What role does the NIH staff play, if any, during the peer review process?
Answer: The staff serves as observers, so they have no interactive role. However, they can comment when there are questions about the intent of an RFA/RFP or an interpretation of a component of the RFA/RFP or agency priorities or intentions. They also make sure that the review follows agency guidelines. The scientific review administrator is supposed to determine the order in which the proposals are reviewed (usually based on when the program officers can be present in the room to hear the review or when the reviewers can be at the review; some come late or leave early or do phone reviews). The scientific review administrator ensures that the reviewers have all the proposals and supporting materials they need to accomplish their tasks, that scores are provided, that compliance issues (e.g., human subjects) are discussed, and that score sheets are signed by the reviewers and turned in at the end of the study section.
Question: Describe an average review session--the trajectory of one proposal over the course of time and/or a series of reviews in one session.
Answer: In the typical review session, each proposal has an order in which it will be reviewed--this may change depending on the presence of reviewers or program officers--and the reviewers of the proposal have an assigned order (primary, secondary, or tertiary). First, each of the reviewers gives an overall evaluation score of the proposal (from 1 to 5, with 1 being highest). Then, the first reviewer gives a general summary of the proposal followed by a critique that includes comments about the significance, research plan, and personnel. Then the second and third reviewers give their reviews, which ideally add to rather than repeat comments made by Reviewer 1. Following these critiques, there is a discussion by the full study section. Often the discussion involves exchanges between the assigned reviewers with each other more so than between the other reviewers, but I have seen extensive discussions involving multiple assigned and unassigned reviewers. After all the reviews and the discussion are completed, the budget and any human subjects concerns are discussed. Then all of the reviewers vote.
Question: What is the most unusual/ memorable proposal or review session that you've encountered?
Answer: A recent study section in which the reviewers, right from the beginning, just seemed to be in sync with each other. The reviews of each proposal fit the expected process of the first reviewer doing the summary and subsequent reviewers adding to what was said, followed by a discussion of the key elements of the proposed study/studies within the proposal with regard to what is currently known in the field and what needs to be known and how. Sometimes I learn a lot about research methods, knowledge in the field, or statistics from proposal reviews; in this study section I learned a lot. Plus, the reviewers were interesting and friendly people, none of whom I knew before the review. The review meeting actually ended early (around noon) the second day rather than late in the afternoon of the second day, as is typical.
I also have been at sessions where a PI requested that a reviewer NOT review the proposal or be present during the review. The reviewer was shocked to be excluded and complained openly. That is both memorable and unusual.
Question: What are your pet peeves as a proposal reviewer relative to proposals and the peer review process?
Answer: I have several! First, sometimes writers attempt to "instruct" the reviewers with statements such as, "the reviewer may be wondering ..." or "the reviewer should be very careful to note ..." Such statements, or their tone, are not appreciated. "Instructing" reviewers wastes precious text space that could be used for elucidating critical elements of the proposal. Second, although the application may request the same or similar information in different parts of the proposal, using whole paragraphs or sentence strings verbatim in multiple places within the proposal will annoy most reviewers. Third, some proposals are poorly prepared, and the reader wonders why she or he is wasting time reading a document that the writer has not carefully read, Examples are misspellings, ungrammatical sentences, text with missing words, and references to graphics or appendix materials that are missing. These proposals create poor impressions and raise questions about how carefully the applicant conducts research. Fourth, references that are cited in the text but missing in the reference list are irritating. I have read proposals where the reference list and the references in the text were misaligned--possibly because the references for the proposal had been taken from another unrelated document and references in the text were not in the list of references and vice versa. It is important for writers to know that reviewers are not experts in all areas (certainly I am not!). Reviewers look up references and read articles to gain at least some expertise. Finding articles can be a very easy process now, using search engines which allow immediate access to articles and chapters. When key references are missing or incomplete, the reviewer is peeved.
I also get peeved when reviewers try to gloss over an incompletely developed proposal submitted by a well-known researcher that they would not be willing to gloss over in a proposal submitted by an unknown. Most other reviewers also tend not to accept statements such as "Dr. X surely knows how to conduct this research even if he/she does not include all the details in the proposal."
Question: How might your prior knowledge of the PI and his or her work influence the way you review a proposal?
Answer: The reputation of the PI does have an influence--but it might not be consistent. Well-written proposals with clearly designed research and with aims directed at critical issues in the field tend to get funded, usually regardless of the PI's reputation. I am not impressed when senior researchers refer to their previous work without giving examples, or refer to a methodology they use without giving details, or seek to move into new areas of research where they have no expertise, expecting the reader to predict the quality of their work in that new area from their work in another area. There is an undeniable tendency to allow some "credit" for well-known researchers, but the proposal still needs to be competitive, which means: good ideas that are well articulated in writing and studied using well-designed procedures.
FEED