A National Science Foundation task force has recommended that scientific proposals to the NSF be based on two generic criteria: the intellectual merit and quality of the proposal, and the broader impacts of the activity. These two criteria would replace four more specific criteria that the agency has used for the past 15 years.
NSF Director Neal Lane announced the recommendations of a Merit Review Task Force on December 3, saying the task force is "not proposing any kind of sea change" and that there is nothing in the current criteria, last revised in 1981, that is broken and needs fixing.
In fiscal 1996, MIT did $35.8 million of NSF research, involving 560 grants and contracts for one or more years of work.
Dr. Lane said a primary problem that the task force identified was a "lack of clarity" leading some reviewers to misunderstand or ignore the existing criteria, or use an "unwritten" criteria.
Introductory paragraphs from the report state:
"The Task Force recommends the two generic criteria (below) to replace the current four NSF criteria. Within each criterion is a set of contextual elements, defined by questions to assist the reviewer in understanding their intent. These elements are non-inclusive; i.e., it is recognized that, for some programs, other considerations not identified below may be important for the evaluation of proposals. Further, reviewers are requested to address only those elements that they consider relevant to the proposal at hand and that they feel qualified to make judgments on.
"#1 What is the intellectual merit and quality of the proposed activity?
"The following are suggested questions to consider in assessing how well the proposal meets the criterion: What is the likelihood that the project will significantly advance the knowledge base within and/or across different fields? Does the proposed activity suggest and explore new lines of inquiry? To what degree does the proposer's documented expertise and record of achievement increase the probability of success? Is the projected conceptually well designed? Is the plan for organizing and managing the project credible and well conceived? And, is there sufficient access to resources?
"#2 What are the broader impacts of the proposed activity?
"The following are suggested questions to consider in assessing how well the proposal meets the criterion: How well does the activity advance discovery and understanding while concurrently promoting teaching, training and learning? Will it create/enhance facilities, instrumentation, information bases, networks, partnerships and/or other infrastructure? How well does the activity broaden the diversity of participants? Does the activity enhance scientific and technological literacy? And, what is the potential impact on meeting societal needs?"
Weight not assigned
Dr. Lane said there would be no weighing of these criteria. In background material, this point was amplified: "NSF does not pre-assign weights to the criteria; given the variation across NSF's many different programs, any `one size fits all' approach would be counterproductive. Overall, however, excellence will continue to be the hallmark of all NSF-sponsored activities."
The sample review form for the new criteria does not include the language now found on the second page of the current proposal evaluation form regarding "relative weight" and "summary ratings," although one of five ratings from "excellent" to "poor" is to be checked on the new form.
Dr. Lane feels the proposed criteria would be a better fit for Education and Human Resources and other non-research activities than those now employed. Approximately 30,000 new proposals are received every year (not counting requests for continuations.) The NSF handles 170,000 reviews, making 10,000 awards. Proposal reviewers and foundation staff are under greater pressure, and Dr. Lane commented, "something has to give here."
NSF has established a comprehensive web site at the following address: http://www.nsf.gov/od/lpa/meritrev.htm>. A feedback form is provided.
The foundation will carefully monitor response on this Web site to determine if public meetings are needed. Dr. Lane said the NSF does not anticipate changing its portfolio, the flexibility it gives its program officers, or its approach to risk-taking.
A version of this article appeared in MIT Tech Talk on December 11, 1996.
