Scientific funding agencies are increasingly creating metascience departments that use scientific methods to study how research is funded, conducted and evaluated. In July, Office of Science and Technology Policy Director Michael Kratsios called on all federal science agencies to establish such teams, and Ireland has also created a similar unit. Experience from the UK Metascience Unit suggests that experiments conducted within real funding systems can reveal practical ways to improve research support.
Testing the funding process
As a co-founder of the first metascience team incorporated into a national research funding system, I have seen how experiments embedded in the funding process can make a difference. Launched in 2024, the UK Metascience Unit is jointly run by the UK government’s Department of Science and UK Research and Innovation (UKRI), a national funding body.
Could agent AI disrupt the subsidy system?
Research funding processes often develop organically rather than through systematic testing and evaluation. Over the past two years, my team has conducted numerous experiments and data analyses to identify effective ways to fund and support research and innovation.
For example, we tested whether email invitations to grant reviewers that appealed to their sense of social obligation would increase acceptance rates. They did not. We also evaluated whether decentralized peer review, in which applicants review one another’s proposals, could speed up grant evaluations. We found that it could, because the number of reviewers automatically increases with the number of applicants, removing the need to recruit external reviewers.
We continue to test new ideas, including experiments with desk rejection. At this stage, UKRI staff identify and remove uncompetitive applications before they reach peer review, and the experiments are intended to explore ways of reducing the demands on examiners.
Lessons from building a new unit
Because ours was the first unit of its kind, the team had no established model to follow and had to learn as it went. This is what I discovered.
Why science has credibility issues and how to deal with them
First, perhaps counterintuitively, these units should remain small. Metascience research shows that large teams tend to extend existing ideas, whereas small teams are more likely to challenge them. The Metascience Unit’s role is to be disruptive and advocate for change.
I firmly believe that our unit accomplished more with a team of six than it would have with a team of 60. When early experiments showed that email nudges were ineffective, we decided it would be better to test a new intervention than to try to improve an unpromising one. Changing direction would, I think, have been more difficult with a larger unit. Small teams have little choice but to be ruthless with their time.
Source: www.nature.com
