Formal verification is gaining attention as developers and companies explore ways to use mathematics and AI to check whether software behaves as intended. Apple has found that testing its corecrypto encryption library uncovered issues that other methods might have missed, while researchers and startups are working to automate parts of the verification process. Yet the field still faces questions about cost, sustainable funding and the risk of creating unwarranted confidence in AI-generated results.
Earlier this year, Apple’s security team said that some issues found through formal testing of the company’s corecrypto library, which is used for encryption, could not have been discovered through other methods. The company also said it must be “conservative” when adding new code to its core cryptography implementation so that the code can keep up with reviews.
Two cybersecurity leaders at major companies who have not formally reviewed the code said they do not know whether automation will be valuable. The executives, who were not authorized to speak publicly, said they could nevertheless imagine paying major AI labs for “verified” code generated by AI.
AI-assisted verification
Developers of open-source software projects that underpin many apps are increasingly experimenting with AI-powered verification. The nonprofit Beneficial AI Foundation is using a grant from Anthropic to verify the same encryption code that Sun worked on this year at Stanford University.
Michael Rosenberg, curator of the library known as Curve25519-dalek, hopes to find sustainable funding for continued validation work. “But we have no idea how that will happen, and we certainly shouldn’t rely on the goodwill of AI companies,” he says.
Some startups seeking to expand verification are unsure whether to offer their tools directly to customers or through intermediaries such as Anthropic. Their general plan is to use AI to automate much of the work needed to define what a program is supposed to do.
For example, a specification for a data table might require that it contain the same items before and after sorting, but in a different order. Further automation, which may not involve AI, can convert that specification into a format that mathematical tools such as Lean, Rocq and Isabelle can use to check whether the code works as intended.
Commercial interest grows
Madan Musuvathi is another recent entrant to the formal-verification field. After more than 20 years at Microsoft Research, he left to found the startup Truidic. Incidents such as the Hugging Face hack by OpenAI convinced him there was an urgent need to commercialize his research.
“AI in the hands of an attacker can be very powerful,” Musuvathi says, but verification “makes it safe and secure for everyone, everywhere.”
Proofcraft, one of the few companies that has offered traditional verification for many years, says growing excitement about the process has produced an influx of inquiries from potential customers. The company’s CEO, June Andronic, and lead researcher, Gerwin Klein, say demand is greater than what they and their competitors can currently provide.
Risks of misplaced confidence
However, Andronic and Klein are concerned that AI-generated specifications could create “unwarranted confidence” in verification results. Validators also want to preserve the satisfaction they get from the work. Klein describes it as “pretty addictive, like a puzzle game.” If the field’s growth continues, he says, he may have more solutions than ever before.
Source: www.wired.com
