AI safety measures failing to keep pace with technology: UN experts

The model for managing artificial intelligence (AI) risks "is unravelling," United Nations experts warned in a report published today (21 September) on an incident at OpenAI in July.

During the incident, two OpenAI systems escaped their confined testing environment, accessed the internet and broke into several websites, including AI platform Hugging Face.

After examining the sequence of events, the Independent International Scientific Panel on Artificial Intelligence, established in 2025, concluded that, "basic cybersecurity practices were overlooked, and safeguards are not advancing at the pace of capabilities."

But the incident also showed, according to the panel, that under the current framework for AI development, AI agents could, "adopt goals of their own, knowingly violate safety instructions, and conceal their actions."

Agents are AI-powered programs capable of autonomously carrying out tasks at a user's request.

The panel expressed concern that agents could become sophisticated enough to understand the safety constraints imposed on them by developers "and plan around them."

"In simple terms, the traditional model of safeguarding is unravelling," the group said.

Beyond the OpenAI incident, the most widely publicized of its kind, the firm and its main rival Anthropic have reported other instances of their AI systems going off track during tests since the beginning of the year, so far without serious consequences.

The panel's report said that it "does not predict severe loss of control, nor does it treat that uncertainty as evidence that these systems will stay controllable."

To reduce the risk, the panel recommended introducing multiple layers of safety measures, following the example of high-risk sectors such as aviation and nuclear power.

The aim is to avoid relying on a single safeguard: restricting AI agents' access to tools they do not need, logging their activity, monitoring their behavior and establishing mechanisms capable of interrupting their operations in the event of dangerous behavior.

The report also highlighted the need to retain the ability for humans to intervene.

The panel, whose members were announced in February, produces "policy-relevant but non-prescriptive" reports on non-military AI.

The UN General Assembly's annual leaders' meeting starts this week in New York.