NIH Peer Review
Enhancing Peer Review at NIH
In 2007, the National Institutes of Health (NIH) launched an institute-wide effort to examine the "entire system of research support, including the peer review system." This review is designed to address concerns about shrinking paylines, recruitment of qualified study section members, workload of study section members, length of grant application, and related issues.
While AAI agreed that there are many problems that the NIH should address, AAI had many concerns about changes that were being implemented or considered, including:
- shortening review meetings to one day of face-to-face time
- increasing the number of phone reviewers, and/or the participation of reviewers by posting comments on electronic 'bulletin boards'
- allowing the mandatory triage of > 50% of applications simply by averaging scores
- using inexperienced, junior scientists as reviewers
In letters to Dr. Toni Scarpa, Director of the NIH Center for Scientific Review (CSR) (see April/May 2007 AAI Newsletter, p. 5), AAI described the concerns above and made the following recommendations:
- Allow regular review group members to serve only 2 times per year
- Provide reviewers with more time (an additional 4-6 weeks) to submit their own applications during cycles in which they are serving on review panels
- Modify current method of grant triage so PIs receive information on the relative ranking of their grants
- Maintain reasonable workloads for each reviewer
- Shorten the length of the RO1 applications from 25 pages to 15 pages, BUT:
- allow the inclusion of an optional 5 page supplement by new PIs to provide additional evidence of technical feasibility and other details necessary to ensure a fair review, and
- limit should not include the 3 page introduction to resubmitted grants since additional space is often needed to respond to a previous review.
- Support recent change requiring reviewers to consider the significance of the proposed research
In response to the AAI letters, Dr. Scarpa met with AAI to discuss some of the proposed changes. On February 28, 2007, Committee on Public Affairs (CPA) Chair Ellen Kraig, AAI Executive Director Michele Hogan, AAI Director of Public Policy and Government Affairs Lauren Gross and (via phone) CPA Member Gail Bishop met with Dr. Scarpa, Dr. Don Schneider, Director of CSR's Division of Molecular and Cellular Mechanisms, and Ms. Kristin McNamara, CSR Program Analyst.
NIH established the "Peer Review Working Group of the Advisory Committee to the Director (ACD)", which is co-chaired by Lawrence Tabak, D.D.S., Ph.D., Director of the National Institute of Dental and Craniofacial Research, NIH, and Keith R. Yamamoto, Ph.D., Executive Vice Dean, University of California-San Francisco School of Medicine, which held meetings around the country to solicit advice from the scientific community.
Several AAI members attended these regional meetings and expressed concerns
- AAI Council Member Jeffrey Frelinger, Ph.D., presented the AAI position at the July 30, 2007, meeting in Washington, D.C
- Joan Cook-Mills, Ph.D., spoke on behalf of AAI at the September 12, 2007, meeting in Chicago
- Joel Ernst, M.D., represented AAI at the October 8, 2007, meeting in New York City
- Committee on Public Affairs members Bonnie Blomberg, Ph.D., and Olivia Martinez, Ph.D., attended the October 25, 2007, meeting in San Francisco. Dr. Blomberg spoke on behalf of AAI.
On February 21, 2008, the NIH Peer Review Working Group of the Advisory Committee to the Director (ACD) presented its findings to the ACD.
In June 2008, NIH announced it would begin the process of implementing the results of the Peer Review Enhancement effort
AAI remains concerned about the following changes:
- NIH will work to "recruit, retain, and motivate the most accomplished, broad-minded, and creative scientists to serve on study section" and will try to reduce the burden of review, provide flexibility, and recognize and reward distinguished service. (Among the specific recommendations: NIH will spread 12 session reviewer commitment over 4-6 years)
- NIH will provide applicants with "clear and purposeful review feedback, including informative summary statements and a rating system comparable across study sections and fields of science."
- NIH will pilot ranking strategies, including the ranking of all applications considered by study sections at the end of the meeting.
- NIH will not treat every grant application as "new".
- NIH will not require a minimum percent effort on research project grants.
- NIH will not rank young investigators separately against each other. ("Peer review should fairly evaluate proposals from all scientists, regardless of their career stage or discipline, and avoid bias towards more conservative and proven approaches at the expense of innovation and originality.")
- NIH will not reinstate the category of "not recommended for resubmission" ("NRR"). (AAI had recommended that if it were reinstituted, NRR should be limited to a small percentage of applications.)
NIH has begun phased implementation of selected actions, including various pilot projects. The NIH Office of Extramural Research published an update on the Enhancing Peer Review project in the August 2008 issue of its on-line newsletter, Extramural Nexus
On December 14, 2009, AAI President Betty A. Diamond, M.D., and AAI Committee on Public Affairs Chair John Schreiber, M.D., M.P.H., sent a letter to NIH Director Francis Collins, M.D., Ph.D., expressing concern about the status of the NIH peer review process.
Dr. Collins sent a reply on February 17, 2010.
- Applications have been significantly shortened (12 pages), potentially creating more work for reviewers by a) increasing the amount of background work needed for proper review, and b) requiring reviewers to review more grants.
- This shorter page limit may disadvantage new investigators.
- The Center for Scientific Review (CSR) plans to continue to use and develop new electronic review formats; AAI had recommended that NIH not increase the use of electronic review because of the invaluable nature of face-to-face meetings.
- NIH plans to commit $1 billion over the next five years to "investigator-initiated high-risk, high-impact research to prevent a slowdown of transformative research…." AAI is uncertain how this policy goal relates to the peer review overhaul and is anxious to know more about this costly initiative.
Open House Workshops – IRGs and Study Sections
CSR also held a series of six "Open House Workshops" "to solicit input from leaders of the scientific community and other stakeholders on the alignment of its Integrated Review Groups (IRGs) and component study sections, which assess NIH grant applications for scientific and technical merit."
© The American Association of Immunologists, Inc.
1451 Rockville Pike, Suite 650 | Rockville, Maryland 20852 | Phone: (301) 634-7178 | Fax: (301) 634-7887
Email: firstname.lastname@example.org | Website feedback: AAI Webmaster