June 7, 2010
STAR METRICS: Measuring The Impact Of Federally-Funded Research
By Michael D. Shaw
According to the National Science Foundation, federal expenditures for research and development are well in excess of $110 billion per year. STAR METRICS is a new program aiming to monitor the impact of federal science investments on employment, knowledge generation, and health outcomes.
STAR METRICS is a multi-agency venture led by the National Institutes of Health, the National Science Foundation (NSF), and the White House Office of Science and Technology Policy (OSTP). Naturally, the name is an acronym, short for Science and Technology in America’s Reinvestment—Measuring the Effect of Research on Innovation, Competitiveness and Science.
“STAR METRICS will yield a rigorous, transparent review of how our science investments are performing,” said NIH Director Francis S. Collins, M.D., Ph.D. “In the short term, we’ll know the impact on jobs. In the long term, we’ll be able to measure patents, publications, citations, and business start-ups.”
Collins is summarizing the goals of Phase I and Phase II of the project…
Phase I: Uniform, auditable and standardized measures of the initial impact of the American Recovery and Reinvestment Act science spending on job creation.
Phase II: Collaborative development of measures of the impact of federal science investment on:
- Economic growth—through patents, firm start ups and other measures
- Workforce outcomes—through student mobility and employment
- Scientific knowledge—such as publications and citations
- Social outcomes—such as health and environment
This sounds good, but where will the basic data come from? It will come from the source—the research institutions that volunteer to participate, and the federal agencies that fund them. STAR METRICS leaders note that highly automated means are in place for filing of the requisite reports, and university officials tell me that much of the information needed is already being compiled.
Still, what incentive does an organization have to participate? What’s in it for them? Besides, aren’t scientists notoriously sensitive about themselves being placed under the microscope?
Writing in Nature, Julia Lane, director of NSF’s Science of Science and Innovation Policy program, observed that…
Scientists are often reticent to see themselves or their institutions labeled, categorized or ranked. Although happy to tag specimens as one species or another, many researchers do not like to see themselves as specimens under a microscope—they feel that their work is too complex to be evaluated in such simplistic terms.
But, given the immense pressure these days to trim government spending, let alone the public’s growing appetite for full disclosure, they will be evaluated, so the metrics must be accurate. Or else, as Lane contends, “If we do not press harder for better metrics, we risk making poor funding decisions or sidelining good scientists.”
As far as reasons to participate, most universities are sincerely interested in self-assessment, not to mention the inherent PR benefit of being able to tout what they bring to the local economy, their role in the expansion of knowledge, and their overall impact on improving society. To be sure, these precepts have been proclaimed for generations, but the proof was all anecdotal. As Lane puts it, “The plural of anecdote is data.”
STAR METRICS is already underway. Over the last year, a pilot project was carried out with seven institutions within the Federal Demonstration Partnership. STAR METRICS is now in the expansion phase, on its way to full fledged operation. About 60 institutions have shown interest in participating.
“It is essential to document with solid evidence the returns our nation is obtaining from its investment in research and development,” said John P. Holdren, assistant to the President for science and technology and director of the White House Office of Science and Technology Policy. “STAR METRICS is an important element of doing just that.”
Although the word “accountability” is not mentioned in any official documents, it is certainly on the minds of taxpayers. STAR METRICS should help bring the ivory tower back into the mainstream.