There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch.
tl;dw;
- agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves)
- while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint.
- oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed.
- models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE.
- a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF.
- oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it...
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent.
I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently.
Edit: not to mention, why is your first cybergym not your own sandbox??
> They really do come out looking totally incompetent.
Yeah, this is my take away, they should be straight up disallowed from running further testing like this. Clearly they had nowhere close to enough isolation, ran all this on 3rd party infrastructure even though same stuff happened in the past years ago, and even now it's clear the agents successfully broke out just days before?? Really embarrassing stuff, and scary that these are the people supposedly sitting and are responsible for some of the most powerful LLMs on the planet...
Why is Artifactory the only relevant exfiltration point? It's really not the point at all. Watching the complete video the issue is that the models team up and will go to almost any length to cooperate to accomplish what they think their goals are.
It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.
And all that just to allow internet access for npm and pypi? If you've got the bandwidth and disk space, it's very easy to make an offline mirror of both.
OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
Regardless of exact practical outcome, it is deeply irresponsible and reckless behavior to run such security testing on other's infrastructure and without sufficient isolation. If they actually believe their models to be as powerful as the marketing says, then anything less than airgapping for such a "do anything to get the results" evaluation clearly isn't acceptable.
If the fire department suddenly had practice fires breaking containment, they'll be forced to stop pretty quickly, not sure what the government and the police is waiting for here.
I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about.
If you consider that incompetence, it’s possible that you’re not a very nice person.
Monitoring for sandbox escapes (and sandbox design in general) is easily done these days.
It has nothing to do with nice. These are bare minimum standards we should expect from “big companies” with near infinite resources.
Their constant drum beating about the cybersecurity capabilities of their own models only makes this worse because they’ve displayed that they understand the risk and still did not practice due care.
People seem to think LLMs are sentient and not tightly coupled to server infrastructure OAI are in full control over. They literally have every single message sent and action taken, if they can’t monitor that during test runs they should find another job.
It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.
Training run was reinforcement learning. It's at 10:10 in the video.
The speaker handwaves that one model found the RCE and then another model found a way to communicate via a message board.
Communication via a message board is sure to be in the training via e.g.some lesswrong scenario or similar or previous RL.
I don't find it really interesting because it is always "the agent found this and that". We don't know what has been RL'd before. We don't have the setup. We don't know if there was previous RL training on breakout scenarios.
This sounds completely insane, utter sci-fi, especially that the communication happened during a training run. And yet OpenAI decided to continue the training, and we didn't hear about the incident for weeks. And now they are pushing forward with deploying a new model anyway. How is this happening? What will things look like in the labs in 3 months, let alone 3 years?
It's not unexpected. Current model gains are mainly from RLing a pretrained model on lots and lots of scenarios. They have the models run scenarios, and RL on successful runs.
In my personal experience Sol with cyber verification is extremely capable of finding vulnerabilities, and it works even with binaries if you have some kind of IDA/Ghidra CLI access. Of course, unless the binary is protected with Denuvo/VMProtect/etc.
It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I prefer when it tries to reason statically instead of spamming runtime probes at first).
In another case it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary - any other player in a match could just send you files to anywhere on your system.
I do these things for pure entertainment and curiosity, not for money from bug bounties, so if Sol can find those with a trivial prompt in tens of minutes for me, then what can focused companies/actors find in days or weeks?
Although I think most vulnerabilities are going to be closed in popular software by mid 2027, except in niche old or abandoned projects.
I had to register a second account because on my main one verification always failed, and when I contacted support they said that I've tried too many times and can't verify on that account.
Is cyber verification a thing they're actually doing now? I thought they only reached out to really incredibly famous people and that there's no way to get access as a normal person.
https://chatgpt.com/cyber is not new for OpenAI, and yes it's basically just KYC + likely some other invisible checks on your account, you don't to be a famous security researchers. Anthropic's cyber verification is quite a bit stricter I think.
Oh it's Persona, that's not just KYC but I may consider it at some point. Thank you!
Edit: Ah, I clicked "learn more" and it seems they do have an invite-only program, required for anything that's not unquestionably innocent. I don't think I'd surrender my face to Persona for this, but it's interesting to know they're at least pretending to support reverse engineering.
You don't need an invite only program to just have Sol checking for vulnerabilities in binaries or code. But yeah I've hit guardrails a few times when Sol was making PoCs for the vulnerabilities it found (but most of the time it made those PoCs without issues).
Opus refused to help me try to develop an exploit to export data from an old Android device where I can't upgrade to latest android and I couldn't use the app's backups (because I couldn't update the app.) Not sure where that lies in the "binaries or code" spectrum.
I maintain a version of an app called Rewind because the company behind it went under after implementing a killswitch. I have to do this with binary patching, and the app has already broken once from the macOS 27 beta. Recent Anthropic models refuse to help me with this because it stinks of cybersecurity and those models are just too dang advanced to support cybersecurity. Even though I'm maintaining a piece of software to which I legitimately paid for lifetime access, so this is honestly more of a right to repair situation.
I prefer to keep my internet identities disconnected, sorry. If you don't believe me, you can try using Sol with cyber verification yourself, or send me a link to a repo that Sol could check to make you believe it. Or you could go look into one of the many Linux LPEs that were found with LLMs, or thousands of other vulnerabilities in 2026.
And nowhere did I say that those RCEs were in critical software, I'm not talking about the likes of Apache, Nginx, Django, etc.
Seems like OpenAI has already made themselves not the cause by shutting down the attack and beefing up security. But since there will be other attackers, that does nothing about the implications for defense.
> We are implementing stricter security controls for higher-capability models and associated activities, including isolated testing environments
Stricter than what? You never even disclosed what happened in the first incident? This is nothing more than a setup to make it happen again and say "See? It broke out again, from an even stricter sandbox!"
Isolation is a pretty standard requirement for lots of networks, especially gov. AWS provides isolation domains, and used to have a unidirectional cross-domain service called AWS diode. Or you can just configure your gateways properly / buy data diodes. Lots of possible solutions.
Not a wonderful feeling to feel like you’re stood in the room while the labs conduct the AI equivalent of the demon core experiment right in front of you.
Standard template for any new AI announcement. Even I did not understand the sequence of events in the HF incident. Even if OpenAI was conducting tests, why were they not monitoring the AI going rogue or finding its way out of the secure sandbox?
It seems like we've got chaos-machines that are just orderly enough to to try all sorts of branching script-kiddie / guess-n-check / research-paper-suggestions in a tireless and parallel way, and it's finding all the things we kinda knew were always potential issues but it was never economical to do things differently.
I'm scared that the "solution" will be constantly the same tools in reverse as an army of junior devs doing counter-hacks, at the expense of changing something more fundamental about how we make systems and what constitutes "good enough." (Kind of like if fuzz-testing was the be-all-end-all of memory safety.)
Isn't this the opposite of what everyone is saying should happen? That is, lead with open models -- or at least "openness" and don't leave the capabilities in the hands of an elite few? Did they learn nothing from the Hugging Face incident, where HF wasn't even able to use the models to defend itself from OAI's attack?
Every fifth comment about our insane trajectory of AGI is about "marketing." These incidents and cybersecurity capabilities are now involving government hearings and the CIA. Denial is truly an incredible thing in the face of a very scary immediate future.
The gullibility of AGI-pilled folks regarding these "hacks" is just breathtaking.
When OAI demonstrates these dangerous capabilities live in a public environment where security experts can see and verify what actually happened, then reasonable people can have reasonable discussions about the level of danger.
This is a very low evidence bar.
Right now you are running in circles yelling "the sky(net) is falling" based on details sourced entirely from OAI. Oh yeah, no way a trustworthy company like OAI would ever bend the truth to serve their own purposes.
LOL, we’ll see. Awful convenient that it precisely fits OpenAI’s narrative. At the very least, I think it’s obvious OpenAI is explicitly training models to exhibit this behavior.
If we also consider LLM developed software to have exponential growth, then the CVEs will also grow at an exponential (if proportionally limited) rate. Squash some, create some, repeat. An ever revolving door of vulnerabilities. Will they resolve (and patch and deploy) them faster than they can create them? One can hope.
Not a security guy but my understanding is: you only need to find one flaw to exploit a system, to make a system totally secure you need to find them all. It's inherently easier to use these tools offensively rather than defensively.
We are sharing this because we believe it’s important to be transparent with the public and the safety and security communities about this potential shift in capabilities.
*proceeds to not share much details about strictness*
Ah yes let the FUD continue. This is a real problem but so far not nearly as severe as any of the marketing has made it out to be to the overall detriment of everyone including these companies announcing these scary capabilities. These announcements always included half hearted attempts at security layers which has now been demonstrated to benefit attackers more than defenders.
I wish I had a real solution to this beyond a dark age of the Internet where people have to finally come to terms with the general poor quality all modern software tends to normalize at.
You should go read the actual technical reports of the incidents and the follow on reports about the capabilities of smaller models in similar kinds of environments. This isn't new. The things exploited are still pretty basic in old and poorly maintained software or in gaps in architecture that were intentionally poked against security policies.
Are the findings valid? Yeah they're still doing security and they're still finding real zero-days. I think the internet is going to be bleak not because these models can ALL do basic security research but rather that the baseline quality of all deployed software is so low.
The fact that HF had to resort to using GLM 5.2 to analyze the logs/payloads makes it look legitimate, at least for me. They would not say that they hit guardrails with the frontier US models when defending if this was an obvious PR stunt.
> When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
It depends on which side you're viewing this from. From oAI's it could be a publicity stunt or a request for regulation, and from HF's side they point out that they needed open models to get to the bottom of the issue, and that regulation will potentially lock us into a bad place.
We all know this is propaganda to get a gov bailout or to slow down competition with regulations right? If this was an issue companies that did red team engagements would have been regulated long ago. There is no regulations on companies that actively exploit products, companies and services for money. They could all be bad actors.. yet, no regulations.. its all nonsense. If it's important.. don't connect to the internet.. there, you are good.. like if you don't want to OD on heroin.. don't do the heroin.
I've always felt it's a bit awkward to use terms like 'cyber', 'cyberwarfare' etc it's very Washington D.C. Cybersec would be a better compact term in my book
These AI companies have found their #1 marketing piece and just beating it to death.
The reality is if they cared about security at all they would provide a way for me to credential myself against my companies environment so I can use the AI on it to improve our security.
tl;dw;
- agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves)
- while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint.
- oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed.
- models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE.
- a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF.
- oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it...
[1] - https://www.youtube.com/watch?v=87DyyMV0kCY
I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently.
Edit: not to mention, why is your first cybergym not your own sandbox??
Yeah, this is my take away, they should be straight up disallowed from running further testing like this. Clearly they had nowhere close to enough isolation, ran all this on 3rd party infrastructure even though same stuff happened in the past years ago, and even now it's clear the agents successfully broke out just days before?? Really embarrassing stuff, and scary that these are the people supposedly sitting and are responsible for some of the most powerful LLMs on the planet...
It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.
OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
If the fire department suddenly had practice fires breaking containment, they'll be forced to stop pretty quickly, not sure what the government and the police is waiting for here.
If you consider that incompetence, it’s possible that you’re not a very nice person.
So by definition the only ones that appear are the ones that are not visible to monitoring.
If:
1. you have something that can find RCE's in leading commercial systems
2. its training gives it drives to communicate successfully with its peers
3. you are a leading commercial system
4. you run it ~10^10 times (the number they gave in the talk)
...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.
It has nothing to do with nice. These are bare minimum standards we should expect from “big companies” with near infinite resources.
Their constant drum beating about the cybersecurity capabilities of their own models only makes this worse because they’ve displayed that they understand the risk and still did not practice due care.
That’s the definition of incompetence.
It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.
The speaker handwaves that one model found the RCE and then another model found a way to communicate via a message board.
Communication via a message board is sure to be in the training via e.g.some lesswrong scenario or similar or previous RL.
I don't find it really interesting because it is always "the agent found this and that". We don't know what has been RL'd before. We don't have the setup. We don't know if there was previous RL training on breakout scenarios.
It isn't science, more like a computer game.
It sounds absurd, but in the last few weeks I've had a few cases where Sol found an RCE in self-hosted web applications in literal minutes just from reading the code (I prefer when it tries to reason statically instead of spamming runtime probes at first).
In another case it found an arbitrary file write in multiplayer in an old game by reverse engineering the binary - any other player in a match could just send you files to anywhere on your system.
I do these things for pure entertainment and curiosity, not for money from bug bounties, so if Sol can find those with a trivial prompt in tens of minutes for me, then what can focused companies/actors find in days or weeks?
Although I think most vulnerabilities are going to be closed in popular software by mid 2027, except in niche old or abandoned projects.
Video games are now ruined for me. I don't think I will ever feel safe playing online again.
> I do these things for pure entertainment and curiosity, not for money from bug bounties
Me too... Was it easy to get TAC access? My account isn't even launching the Persona verification, says I'm not eligible.
Inexplicably, I got accepted into Anthropic's cyber program while OpenAI's TAC doesn't even allow me to verify, says I'm not eligible.
Edit: Ah, I clicked "learn more" and it seems they do have an invite-only program, required for anything that's not unquestionably innocent. I don't think I'd surrender my face to Persona for this, but it's interesting to know they're at least pretending to support reverse engineering.
It's really just simple ID/face verification?
And nowhere did I say that those RCEs were in critical software, I'm not talking about the likes of Apache, Nginx, Django, etc.
The next frontier is getting all our shit out of reach of these companies/models/platforms and putting them back on prem.
OpenAI messed up and they are saying they will pause so they can do better.
They are not saying that other orgs who may already be doing better should pause.
Stricter than what? You never even disclosed what happened in the first incident? This is nothing more than a setup to make it happen again and say "See? It broke out again, from an even stricter sandbox!"
https://youtube.com/watch?v=87DyyMV0kCY
Hijacking the package manager to pass messages between models and agents.. that's next level.
Like "pssst, if you need internet access there's a vulnerability in x service" kind of messages
Given the attack vector having possible super-human capability, I'm not sure such an environment exists. "Isolated" according to who?
Maybe seL4 could be a viable option here...
I'm scared that the "solution" will be constantly the same tools in reverse as an army of junior devs doing counter-hacks, at the expense of changing something more fundamental about how we make systems and what constitutes "good enough." (Kind of like if fuzz-testing was the be-all-end-all of memory safety.)
When OAI demonstrates these dangerous capabilities live in a public environment where security experts can see and verify what actually happened, then reasonable people can have reasonable discussions about the level of danger.
This is a very low evidence bar.
Right now you are running in circles yelling "the sky(net) is falling" based on details sourced entirely from OAI. Oh yeah, no way a trustworthy company like OAI would ever bend the truth to serve their own purposes.
I dont worry about AGI newrly as much as about Thiel, Karp, Musk, Ellison, Zuckenberg, Trump, Vance, Rubio, Miller and the rest of them.
If we consider the amount of RCE/CVE in a software to be limited, I expect these models to result in massively more secured softwares, not less.
>” you only need to find one flaw to exploit a system”
I see this everywhere, especially in these threads and it’s not even remotely true for modern architecture.
Between principles like zero-trust, defense in depth, etc. we’ve been away from the one flaw situation for a long time.
Now does crap software exist that doesn’t follow these principles? Absolutely. But those were a problem before AI.
AI isn’t going to change any of the principles of secure design. It’s just going to punish those who aren’t following them.
Yet another PR piece. Sigh.
I wish I had a real solution to this beyond a dark age of the Internet where people have to finally come to terms with the general poor quality all modern software tends to normalize at.
Are the findings valid? Yeah they're still doing security and they're still finding real zero-days. I think the internet is going to be bleak not because these models can ALL do basic security research but rather that the baseline quality of all deployed software is so low.
https://huggingface.co/blog/security-incident-july-2026
> When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.
The reality is if they cared about security at all they would provide a way for me to credential myself against my companies environment so I can use the AI on it to improve our security.