AI agents create safety challenges that conventional generative AI systems do not. Unlike systems that produce a single response for a person to act on, agents can plan tasks, select tools, create subagents and take actions over longer periods. Their behavior can therefore unfold in unpredictable ways, making it harder for pre-launch testing to anticipate risks and more important to observe agents while they operate.
Gaps in agent observability
In work conducted with colleagues from academia, civil society and industry, the researchers examined four widely used agent frameworks to determine what signals they emit. They found significant gaps that could hinder efforts to detect policy violations, fraudulent activity and errors in human oversight.
Important observability features were missing, including persistent agent identifiers and records showing which humans had approved particular tasks. Without those records, it can be difficult to determine which agent performed an action or establish whether it acted without authorization.
Two records for understanding agent behavior
Once these gaps are identified, they can be addressed by capturing two types of information: records of what happened during an agent’s execution and records of how it made decisions.
A “what happened” record would identify the agent while protecting privacy, document human interventions, and show whether the agent’s permissions or “memories” changed. A “decision” record would describe the agent’s reasoning and indicate its level of confidence in the actions it took.
Consider an agent that processes mortgage applications for a bank. The agent reviews an application and recommends rejecting it, after which a human loan officer signs off on the decision. Banks and regulators need to be able to reconstruct, either during an audit or in real time, who made the recommendation, who approved it and how long the process took.
They would also need to examine how the agent reached its decision to ensure that it did not rely on prohibited information, such as an applicant’s region of origin.
OpenTelemetry’s role
Recording what happened and how an agent reached a decision requires different approaches, but both are considered feasible. The absence of a reliable “what happened” record is primarily a software-plumbing problem that can be addressed through OpenTelemetry, an open observability standard maintained by engineers across the technology industry and adopted by agent frameworks.
OpenTelemetry has launched a working group to draft a shared format for tracking agents. The format is not yet finalized, and the group needs to incorporate standards for persistent agent identity, human intervention, and changes to agent permissions and memories.
Before such systems can scale, their designers must also consider the privacy implications of agent identifiers that can be linked to real people.
Source: www.technologyreview.com
