How Research Is Evaluated

Procedures for Evaluating & Ranking Research Proposals Procedures for Evaluating & Ranking Research Proposals


Background

The process of evaluating grant proposals was revised in 1997-98 with a goal of identifying scientifically well founded projects with the greatest impact on both the horse industry and the problem being studied.The new concept was built on the premise that combining the skills of solid scientists with the perspective of actively practicing veterinarians during the evaluation process would result in the funding of research aimed at the most important problems of the horse industry. Each component of the review process has a specific objective. (back to top)


Early Planning

The key ingredient in this transition was the vision and perspective of Dr. Larry Bramlage, noted equine orthopedic surgeon from Lexington. In concert with Ed Bowen, President, and the Board of Directors of the Foundation, Dr. Bramlage invited a small group of experienced researchers and clinicians to develop a plan. Meeting at the Cincinnati airport, this group drew up a new research proposal application, using several proven forms as a basis. They next agreed on the makeup of the committee to review the proposals. There was universal agreement that the review process must be focused on real problems facing the industry and that the quality of the work must be outstanding scientifically. In order to influence the outcome of the problems being investigated, the results must stand up to peer review by journals of highest quality and be published.

The first Research Advisory Committee (RAC) was then formed by inviting a mix of scientists capable of judging the merit of the most sophisticated proposals with clinical specialists both from university clinics and private practices. The depth of the group had to include some representation of the major clinical disciplines such as surgery, internal medicine, and reproduction, along with the academic disciplines of pathology, immunology, microbiology and so forth. There must be geographic diversity to reflect the importance of various local problems.

The first committee included 32 members,which was judged to be the minimum number to cover the wide base of proposals expected. As it has now evolved, 25% of the members are turned over each year, with the replacements tailored to the specialties and geographic area of the departing group. Each member is eligible to return to the committee, upon invitation, after at least one year away. The normal term on the committee is four years. Members of the committee are permitted to apply to the Foundation for support, but then are not involved in the review of that grant. (back to top)


The Process

Proposals are submitted to the office by October 1 of the year prior to expected funding. Each proposal is recorded by title, investigators, and institution, and assigned a number. A list of these data is then transmitted to each member of the RAC. Committee members then notify the office of any perceived conflicts of interest that would disqualify them from judging the proposal fairly. Copies of all proposals are made and sent to all committee members.

Once all conflicts have been determined and considered, assignments of the proposals to individual members are designated by the office and by the RAC Chair (Dr. Paul Lunn). Four committee members are assigned to each grant, two that are best qualified to judge the science and two to judge the impact or relevance of the project to the industry. One of the four is designated as the Composite Reviewer and the other three as Primary Reviewers. Individual assignments are sent to each member, along with a primary scoring sheet, the composite scoring sheet and a re-cap of the timetable for the review.

Each of the four reviewers judges a proposal using the criteria on the scoring sheet and writes a narrative review. This narrative points out the strengths and weaknesses of the proposal, suggests improvements in the approach and makes other objective evaluations. Then, each primary reviewer submits copies of the narrative and score to the composite reviewer. That person is responsible for tabulating the scores into the composite scoring sheet and consolidating all of the primary narratives into one composite narrative. The composite score and all five narratives are sent to the office. The independence of the individual reviews is an important aspect of the review process because it forces each reviewer to make an independent judgment and prevents one strong reviewer from dominating the assessment.

Timing of these steps in the review process is keyed to the schedule for the annual meeting, which is typically mid January. All review material arrives at the office in time to copy and compile it and ship it to the meeting site. All committee members will thus have the scoring information and all of the narratives for consideration. (back to top)


The Review Meeting

The meeting commences on a designated Friday afternoon and continues through the weekend, concluding on Sunday morning. The essential activities are: 1) to establish consensus among the four reviewers so that mean scores and comments in the narrative are representative of all four opinions. This is accomplished by informal discussions by each group of four reviewers; 2) to discuss each proposal in the entire committee; and 3) to establish a rank order for all acceptable proposals. Each of these sessions has a specific objective. In the first session of the meeting, (usually Friday afternoon) the four independent reviewers for each project meet to reconcile any "outliers" in the four scores. Each reviewer who varied from the mean score must either accept the other reviewers'consensus or convince the reviewers why he/she saw fit to give the score that he/she assigned. This assures four independent views, but results in a single consensus score and a consensus narrative review. The narrative review is critical because it becomes the "feedback" document to unsuccessful and successful applicants. At the first session the members get the review book of the composite scores, narrative reviews and composite reviews of all the grants. This allows them a chance to peruse the entire group of grant reviews before the second session.

In session two of the meeting (usually Saturday) the composite reviewer presents the consensus review to the entire committee. Many of the committee have read more than their assigned grants, and they are encouraged to do so. After the grant is introduced the entire group participates in the discussion, with the exception of the committee members with conflicts of interest (such as being from the same institution). Conflicts are excused from the room. Mean scores may be adjusted up or down, based on the group discussion. The group as a whole then endeavors to reconcile mean scores between specialties, to assure that one discipline (such as orthopedic surgery or reproductive science) doesn't score significantly more harshly or more leniently than another. At the end of this session the grants are ranked numerically for use in session three. The composite reviewer further edits the composite narrative during this session to reflect the comments of the entire group.

Session three takes place the morning after session two. Each grant is compared to the grant above it and below it, to assure it is ranked correctly for its strengths and weaknesses compared to its neighboring grant. If a grant is moved up or down in the ranking it must be reassessed at each placing, compared to its higher and lower neighbor, to assure that the placing is correct in the consensus of the group. At the end of this session the resulting list is the preferred ranking of the committee for all the grants determined to be acceptable for funding. The last item of business, after ranking, is to review the budget of each grant to assure that all requested funds are appropriate.(back to top)


Timeline

A timeline is incorporated into the application document so that all investigators are aware of the schedule. The applicant in the original application predicts a schedule of progress. Grants are made for one or two years. The funding cycle commences April 1, with three equal payments being made through the year. A progress statement is required on November 1 of the original funding year. Problems with the investigation can be identified early and adjustments made. Satisfactory progress must be documented in order to continue receiving funding. Second year funding must be approved by the Foundation Board following recommendation from the office.