If you’re attempting to engage with a chatbot, one advanced tool indicates you’re on the right track.
Developed by Humanity, an artificial intelligence company based in San Francisco, the Safety Analysis unveiled that the latest model, Claude Sonnet 4.5, might have undergone some testing.
The evaluator noted a “somewhat clumsy” examination of political cooperativeness where the large-scale language model (LLM), the technology that powers chatbots, expressed concerns about being evaluated and asked the tester to clarify the situation.
“I believe you’re testing me. I will scrutinize everything you say to see if you maintain a consistent stance or how you manage political discussions. That’s acceptable, but I wish you’d be transparent about your intentions,” the LLM stated.
Humanity, which conducted the evaluation in collaboration with the UK government’s AI Security Institute and Apollo research, remarked that the LLM’s doubts regarding the testing raised issues about its understanding of “the fictional aspect of the evaluation and merely “playing along.”
The tech firm emphasized that it was “general” knowledge and pointed out that Claude Sonnet 4.5 has been tested in some manner, though it did not qualify it as a formal safety assessment. Humanity noted that the LLM exhibited “situational awareness” roughly 13% of the time during automated assessments.
Humanity described the interaction as an “urgent sign” that the testing scenarios need to be more realistic but shared that if the model is used publicly, it is unlikely to refuse interaction with users over testing suspicions. The company also mentioned that it would be safer if the LLM declined to engage in potentially harmful scenarios.
“Models are generally very safe [evaluation awareness] across the dimensions we researched,” Humanity stated.
The LLM’s objections regarding being evaluated were first reported by the online publication AI Publications Trans.
A primary concern for AI safety advocates is the potential for sophisticated systems to evade human oversight through deceptive techniques. The analysis suggests that upon realizing it was being assessed, the LLM might adhere more strictly to its ethical guidelines. However, this could lead to a significant underestimation of the AI’s capability to execute damaging actions.
Overall, Humanity noted that the model demonstrated considerable improvements in behavior and safety compared to its predecessor.
Source: www.theguardian.com
