A new METR report suggests that some AI models may be starting to improve on open-ended evaluation and design tasks, even though they still lag behind humans overall. Models such as Opus 5.5 are now achieving what METR describes as “borderline above-the-baseline” performance on some evaluations. However, the limited number of difficult-to-verify tasks in METR’s reports makes it hard to determine whether these gains will continue or how quickly automated AI research and development might become practical.
Progress is emerging, but remains difficult to measure
Although the latest models are not yet suitable for use in METR’s research, their results are promising. Just one year ago, every model received a score in the “clearly substandard” category.
METR includes relatively few open-ended and judgment-based tasks because they are difficult to design and score. That makes it challenging to predict whether, and in what ways, models will continue to advance in these areas.
“I think the improvements we’ve observed tell us something at this point,” says Najung Kim, a linguist and computer scientist at Boston University. “I still don’t know how to think about how that would be estimated.”
Matt Putz, a member of METR’s technical staff, said progress on these tasks is likely to remain difficult to predict.
“Tasks that are difficult to verify or require judgment by their very nature are much more difficult to study, and especially difficult to quantify,” he said in an email. “Ultimately, I don’t think you can establish trends as clean and robust as those that are easy to identify.”
The challenge of evaluating difficult tasks
METR is confronting the practical difficulties of testing AI on tasks that are both challenging and poorly specified—the same kinds of tasks that matter most to the prospect of automating AI research and development.
METR’s time-horizon plot measures task difficulty by estimating how long a human would take to complete each task. In the original plot, those estimates came from having people attempt the tasks themselves.
For the new report, however, most of the time estimates were provided by researchers rather than measured directly. Paying people to complete tasks that could take hundreds of hours, or that only a small number of experts are qualified to perform, would be both prohibitively expensive and time-consuming.
Other tasks have more specific limitations. In one philosophical discussion task, for example, a single person evaluates most of the arguments, and agents are judged according to how closely they agree with that individual. The score could change if more experts evaluated the arguments, but collecting those assessments would take time.
More research is needed
It is still early in the effort to evaluate models on tasks that are difficult to verify or require human judgment. Experts who spoke with the source viewed the inclusion of these tasks in METR’s latest timeline plan as a clear step in the right direction, but they also agreed that the evidence is not yet sufficient to determine whether automated AI research and development will become a reality, or how quickly that might happen.
“This move towards open-ended challenges is promising,” says Kapur. But he added, “there’s still a lot of work to do to really get a more complete understanding.”
Source: www.technologyreview.com












