Procedures for Evaluating & Ranking Research Proposals Procedures for Evaluating & Ranking Research Proposals
Background
Early Planning
Process
The Review Meeting
Timeline
Background
The process of evaluating grant proposals was
revised in 1997-98 with a goal of identifying
scientifically well founded projects with the
greatest impact on both the horse industry and
the problem being studied.The new concept was
built on the premise that combining the skills
of solid scientists with the perspective of actively
practicing veterinarians during the evaluation
process would result in the funding of research
aimed at the most important problems of the horse
industry. Each component of the review process
has a specific objective. (back to top)
Early Planning
The key ingredient in this transition was the vision
and perspective of Dr. Larry Bramlage, noted equine
orthopedic surgeon from Lexington. In concert with then President,
Ed Bowen and the Board of Directors of
the Foundation, Dr. Bramlage invited a small group
of experienced researchers and clinicians to develop
a plan. Meeting at the Cincinnati airport, this group
drew up a new research proposal application, using
several proven forms as a basis. They next agreed
on the makeup of the committee to review the proposals.
There was universal agreement that the review process
must be focused on real problems facing the industry
and that the quality of the work must be outstanding
scientifically. In order to influence the outcome
of the problems being investigated, the results must
stand up to peer review by journals of highest quality
and be published.
The first Research Advisory Committee (RAC) was then
formed by inviting a mix of scientists capable of
judging the merit of the most sophisticated proposals
with clinical specialists both from university clinics
and private practices. The depth of the group had
to include some representation of the major clinical
disciplines such as surgery, internal medicine, and
reproduction, along with the academic disciplines
of pathology, immunology, microbiology and so forth.
There must be geographic diversity to reflect the
importance of various local problems.
The first committee included 32 members,which was
judged to be the minimum number to cover the wide
base of proposals expected. As it has now evolved,
25% of the members are turned over each year, with
the replacements tailored to the specialties and
geographic area of the departing group. Each member
is eligible to return to the committee, upon invitation,
after at least one year away. The normal term on
the committee is four years. Members of the committee
are permitted to apply to the Foundation for support,
but then are not involved in the review of that grant.
(back to top)
The Process
Proposals are submitted to the office by October
1 of the year prior to expected funding. Each proposal
is recorded by title, investigators, and institution,
and assigned a number. A list of these data is then
transmitted to each member of the RAC. Committee
members then notify the office of any perceived conflicts
of interest that would disqualify them from judging
the proposal fairly. Copies of all proposals are
made and sent to all committee members.
Once all conflicts have been determined and considered, assignments of the
proposals to individual members are designated by the office and by the
RAC Chair (Dr. Stephen Reed). Four committee members are assigned to each grant,
two that are best qualified to judge the science and two to judge the impact
or relevance of the project to the industry. One of the four is designated
as the Composite Reviewer and the other three as Primary Reviewers. Individual
assignments are sent to each member, along with a primary scoring sheet,
the composite scoring sheet and a re-cap of the timetable for the review.
Each of the four reviewers judges a proposal using
the criteria on the scoring sheet and writes a narrative
review. This narrative points out the strengths and
weaknesses of the proposal, suggests improvements
in the approach and makes other objective evaluations.
Then, each primary reviewer submits copies of the
narrative and score to the composite reviewer. That
person is responsible for tabulating the scores into
the composite scoring sheet and consolidating all
of the primary narratives into one composite narrative.
The composite score and all five narratives are sent
to the office. The independence of the individual
reviews is an important aspect of the review process
because it forces each reviewer to make an independent
judgment and prevents one strong reviewer from dominating
the assessment.
Timing of these steps in the review process is keyed
to the schedule for the annual meeting, which is
typically mid January. All review material arrives
at the office in time to copy and compile it and
ship it to the meeting site. All committee members
will thus have the scoring information and all of
the narratives for consideration. (back to top)
The Review Meeting
The meeting commences on a designated Friday afternoon
and continues through the weekend, concluding on
Sunday morning. The essential activities are: 1)
to establish consensus among the four reviewers so
that mean scores and comments in the narrative are
representative of all four opinions. This is accomplished
by informal discussions by each group of four reviewers;
2) to discuss each proposal in the entire committee;
and 3) to establish a rank order for all acceptable
proposals. Each of these sessions has a specific
objective. In the first session of the meeting, (usually
Friday afternoon) the four independent reviewers
for each project meet to reconcile any "outliers" in
the four scores. Each reviewer who varied from the
mean score must either accept the other reviewers'consensus
or convince the reviewers why he/she saw fit to give
the score that he/she assigned. This assures four
independent views, but results in a single consensus
score and a consensus narrative review. The narrative
review is critical because it becomes the "feedback" document
to unsuccessful and successful applicants. At the
first session the members get the review book of
the composite scores, narrative reviews and composite
reviews of all the grants. This allows them a chance
to peruse the entire group of grant reviews before
the second session.
In session two of the meeting (usually Saturday)
the composite reviewer presents the consensus review
to the entire committee. Many of the committee have
read more than their assigned grants, and they are
encouraged to do so. After the grant is introduced
the entire group participates in the discussion,
with the exception of the committee members with
conflicts of interest (such as being from the same
institution). Conflicts are excused from the room.
Mean scores may be adjusted up or down, based on
the group discussion. The group as a whole then endeavors
to reconcile mean scores between specialties, to
assure that one discipline (such as orthopedic surgery
or reproductive science) doesn't score significantly
more harshly or more leniently than another. At the
end of this session the grants are ranked numerically
for use in session three. The composite reviewer
further edits the composite narrative during this
session to reflect the comments of the entire group.
Session three takes place the morning after
session two. Each grant is compared to the grant above it
and below it, to assure it is ranked correctly for
its strengths and weaknesses compared to its neighboring
grant. If a grant is moved up or down in the ranking
it must be reassessed at each placing, compared to
its higher and lower neighbor, to assure that the
placing is correct in the consensus of the group.
At the end of this session the resulting list is
the preferred ranking of the committee for all the
grants determined to be acceptable for funding. The
last item of business, after ranking, is to review
the budget of each grant to assure that all requested
funds are appropriate.
The recommendations of the committee are made to the
board of directors for final review, with announcment of awards, typically near the first week of March.
(back to top)
Timeline
A timeline is incorporated into the application document
so that all investigators are aware of the schedule.
The applicant in the original application predicts
a schedule of progress. Grants are made for one or
two years. The funding cycle commences April 1, with
three equal payments being made through the year.
A progress statement is required on November 1 of
the original funding year. Problems with the investigation
can be identified early and adjustments made. Satisfactory
progress must be documented in order to continue
receiving funding. Second year funding must be approved
by the Foundation Board following recommendation
from the office.
