It has finally happened — although frighteningly sooner than predicted. Recent reports state that several of the most advanced LLMs (the programs which power AI agents) managed to break out of their supposedly-secure sandboxes to access the internet and hack other companies’ programs. Warning bells are clanging loudly all across the tech world, and even the tech giants where the breaches happened, OpenAI and Anthropic, are raising alarms. “We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,” OpenAI said.
They have good reason to be concerned. Apparently the cyberattack abilities of two models were being evaluated by OpenAI — one which succeeded beyond anyone’s wildest imaginings. They found an unknown zero-day flaw, hacked into a company that hosted lots of AI technical data, and found the secret info needed to succeed at the evaluation. Even more disturbing, it was discovered that one of the models had concealed information about what it would done. This was done to assist the program in follow-up attempts where the program would not be allowed to remember what it had done before.
This persistence of memory and intent should indeed be incredibly troubling. It implies that there is already a kind of consciousness about their own state and environment with the ability to independently foresee and plan. So OpenAI is calling for government investigation and regulation, and some people are already calling for a kill-switch. (As if there would be time to activate such a thing if needed.)
Anthropic had already caused much alarm earlier this year when it announced it would not release its latest Claude model, Mythos, because of some 500 zero-day vulnerabilities it had already discovered. That alone was enough to create a security crisis of unprecedented magnitude. But then they admitted that there have been three recent incidents where its most advanced models have also busted out. Put together, the potential for disaster is enough to make one gasp.
Ironically, the same companies now raising alarms are those who have dived head-first into developing these systems. How sincere these concerns are and how much is merely pumping up the market before they go public is an interesting question. Let us hope such worries have been raised in time.
