Cosmos: AI Researcher Claims to Achieve Six Months of Work in Just Hours

AI researchers can work autonomously for extended periods, completing studies in hours that would take humans months. While developers assert that they have made several “new contributions” to science, skepticism remains among some experts.

The platform, referred to as Kosmos, consists of multiple AI agents adept at data analysis and literature review, aiming to generate groundbreaking scientific insights.

“We have dedicated nearly two years to training AI scientists,” states Sam Rodricks, from Edison Scientific, the company behind Kosmos. “The limitation of previous AI scientists has always been the complexity of the concepts they produce.”

Kosmos endeavors to overcome this challenge. Typically, a session can last up to 12 hours; during this time, when a user inputs a scientific dataset, Kosmos examines roughly 1,500 pertinent academic papers while generating and executing 42,000 lines of code to analyze the data. At the end, the AI compiles a summary of the findings and relevant citations, along with a proposal for further analysis that can initiate the next cycle.

After a predetermined number of cycles, the system produces a report featuring scientific conclusions supported by relevant citations, akin to an academic publication. An assessment from a collective of scholars found that 20 of these cycles corresponded to about six months of their research efforts.

Rodriques remarked that the conclusions drawn by the system tend to be fairly accurate. Edison asked individuals with doctoral-level knowledge in biology to evaluate 102 claims made by Kosmos. The research team discovered that 79.4% of these claims were overall substantiated, including 85.5% concerning data analysis and 82.1% of claims referenced in existing literature. Nevertheless, Kosmos struggles to synthesize this information and generate new claims, achieving an accuracy rate of just 57.9% in this area.

Edison asserts that Kosmos has made seven verifiable scientific discoveries, all of which have been confirmed and replicated by independent specialists in the field using external datasets and diverse methodologies. According to the Kosmos team, four of these discoveries are genuinely novel, while the remaining three were previously documented, though in preprints or unpublished studies.

Among the claimed discoveries is a novel method for identifying when cellular pathways falter as Alzheimer’s disease advances. Another finding suggests that individuals with higher levels of a natural antioxidant enzyme known as superoxide dismutase 2 (SOD2) in their blood may experience less heart scarring.

However, reactions to these claims from the scientific community have varied. The “discovery” related to SOD2 is deemed unremarkable by Fergus Hamilton of the University of Bristol, UK. “That specific causal assertion probably won’t withstand scrutiny as a new finding, and there are methodological flaws inherent in the analysis,” he comments. Professor Rodriques acknowledged that the SOD2 finding had been previously established in mice, but claimed this is the first time it has been recognized at the population level in humans through genomics.

Hamilton pointed out that the data analysis code which the agent attempted to execute malfunctioned, causing Kosmos to overlook potentially essential data while arriving at the same conclusions as existing studies.

“Several critical assumptions were made that were imperative for achieving accurate analysis,” he notes. “The software package fails entirely, yet key elements were ignored.” Additionally, in this instance, the data was so processed beforehand that Kosmos “only managed to accomplish around 10 percent of the task,” he suggests.

Hamilton commends the team behind Kosmos for addressing his queries and concerns raised on social media. “While this presents a substantial step forward conceptually, specific technical critiques of this study remain: [the] work is still far from zero,” he states.

“We’re entirely open to the possibility that some of the findings we present could be incorrect or flawed. This is part and parcel of scientific inquiry,” says Rodricks. “Nevertheless, the fact that it has garnered such intricate criticism highlights the system’s potential.”

Others express admiration for Kosmos’ performance overall. “This highlights the immense potential for AI to aid scientific research, but we must remain cautious about the independent use of AI scientists,” states Ben Glocker from Imperial College London. “Even though this study showcases some remarkable achievements, we still lack understanding of the failure modes.”

“We believe embracing tools like Kosmos and developing others is essential. However, we should not lose sight of the fact that science encompasses more than just a data-centric approach,” mentions Noah Jansiracusa from Bentley University, Massachusetts. “There is profound thought and creativity involved, and it would be unwise to disregard scientific pursuits that are amenable to automation solely because they are suitable for AI.”

Rodricks himself concedes that Kosmos is best utilized as a collaborator, rather than a replacement for researchers. “It is capable of performing many impressive tasks,” he asserts. “It requires thorough review and validation, and it may not always be entirely accurate.”