Revealing the details of how OpenAI agents hacked Hugging Face (swarmtraces.org)

But without proper redirection, yeah, its mostly infinite monkey machine with infinite linux manuals.

I think people put too much SOTA halos around whats just a suppedup LLM hardware.

How is that a bad thing in this context ? From the point of view of an attacker, all you care about is finding a viable exploit chain. Likewise, a defender wants to find the "holes" in their system, no matter how complex. Once found, an agent/human can easily synthesise a clean, succint exploit from the most promising candidate, no ?

> Also, it looked so "loud", querying millions of URL with weird requests.

Agreed, this thing speaks more to the bad security at HF than any emergent "hacking" ability from OpenAI. It's unclear to me why an older/dumber model wouldn't have been able to do the same. Is it better coordination? Long-horizon work ?

Run the same protocol again, but have the agents think they had limited resources or that HuggingFace was rate limiting them, and they'd find something you'd consider smarter.

Computers don't have a sense of elegance by default. Elegance emerges from constraints.

I’m not that surprised about models with endless compute being capable of this, I’m more surprised that a company with the resources they have apparently can only create a sandbox that a half skilled human operator could have broken out of easily.

This was just more bots.

So it seems that OpenAI hacked so many systems not because they have superior models, but because of how poor their training, sandboxing and evaluation pipeline was compared to Anthropic's.

I'm a big fan of Jensen Huang's interviews at the moment - these companies should be validating their software is safe before releasing it to the public. And if the current CEOs can't ensure that internal testing is done safely, they should step aside for CEOs that can.

AKA fuzzing

You do see how it would be worse if they did have a clear plan, right?

> People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.

If there several thousand people working on this, with a very limited ability to communicate with each other, I bet you'd see something similarly disorganized.

> The sandbox as weak as it can get

This is hyperbolic; the agents were only able to access the internet by finding a zero-day in the internal Artifactory instance that was providing package access for the agents. The sandbox could and should have been better, but it was not as weak as it could have been.

I can’t help but think that if you were able to step back and watch the way humans try to solve a problem like curing cancer it might look something like this. Best practices are shared and we build upon each others learnings and that how most progress is made. That doesn’t mean there aren’t plenty of people who spend their entire lives trying to do something completely against best practices and while most of the time that seems like useless toil, every once in a while thats where the breakthrough comes from.

1) the Morris worm, which scattershot a bunch of known exploits until it hit paydirt, and then used whatever it found to compromise and replicate itself on the host system;

2) a story here on Hackernews about how someone got the fuzz tester American Fuzzy Lop to "learn" how to produce well-formed JPEGs and PDFs by pointing it at a JPEG or PDF decoder; the tester can record which code paths are followed and with enough random input can find a path into the depths of the system under test... but doing so for a decoder means actually constructing what it is meant to decode.

Neither of these are particularly "smart". But a brute-forcing machine gonna brute force, and it has the potential to cause a lot of damage. If you built a Morris worm with a fuzz tester on its nosecone, think of the mayhem you could cause! If you could examine the logs you'd probably find some undiscovered vulnerabilites in there, too! Maybe LLMs can just do so more efficiently, or maybe they let people who are too ignorant to have that kind of power vibecode their own fuzz-tester-tipped Morris worm.

The Morris worm happened in 1988 [0].

I mention this b/c people like to say "the agents wanted to hack" but really this is just a few levels above the "keep trying the same port on multiple machines".

0 - https://en.wikipedia.org/wiki/Morris_worm

There is that old saying about monkeys and typewriters and Shakespeare - These are just better monkeys with bigger typewriters and ability to aim them at a goal.

I am by no means a doomer but for gods sake we need to have a grounded conversation. Because what happens when the DOD or NSA rolls up at OAI's front door and says "install that in our data center over here please" and uses it to go after someone with it?

Are banks (as some of the more security minded institutions) safe against 10000 concurrent, well coordinated attackers who do not ever rest?

Because there's a lot of potential for unintended 2nd order impacts from anything an agent can do, when done at scale.

I have to wonder if this is already going on, and every one is living under a gag order (it would not be the first time) and the calls for "regulation" are more about "not weaponizing it".

This is not like grep or curl where it does exactly what you tell it to do.

This is obviously already the case and it's much different from a scenario where the AI genuinely takes unexpected action.

I frankly find it ridiculous how many suggest OpenAI or its employees should face criminal charges, without actual legal basis at the time.

It's also hardly outrageous that they ran training and/or benchmarks with only network-isolated VMs with access to a package repository.

This being the first well-known incident of its kind, I wouldn't expect them to have done more than that.

The idea that AI labs will now intentionally have their models hack companies in order to market their models, well, I don't even know what to say.

That's ridiculous and what you describe would obviously be criminal behavior under existing law.

The developers of the AI, and indeed several stories now of end-users with similar but smaller-scale behaviours, were literally not intending to abuse the AI to cause harm.

Yes, by all means, criticise OpenAI here for an insufficient sandbox, for inadequate monitoring, etc. (that's all correct even if it wasn't too long ago that people laughed at the idea AI could find novel zero-days in their sandboxes and mocked those who suggested the possibility[0][1][2]), but *this behaviour is what people worried about rogue AI are talking about*.

This has always (at least, since I graduated) been what people worried about rogue AI have been talking about.

The "paperclip maximiser" story was never about an AI which suddenly develops a love of paperclips transcending any human intervention, it's a story about some idiot who wants to get rich and tells their AI to "make as many paperclips as possible", and then it does that.

[0] Here, 7 months ago. Both why all the companies should have known and planned better, and also look at all this skepticism throughout the comments: https://news.ycombinator.com/item?id=46902909

[1] Here, 4 months ago: https://news.ycombinator.com/item?id=47951174

[2] Some corporate blog, IDK who they are even if the logo says they're "a CISCO company", but February this year and outright denying that LLMs can find zero-days at all:

  LLMs don’t discover zero-days or invent exploits; they simply predict text that sounds plausible based on what they’ve seen before.

- or https://web.archive.org/web/20260404154717/https://www.splun... if they take it down, but the date isn't in the archive version

Each side wants to focus on only one. It's ridiculous to not focus on both.

> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm.

[Why-not-both?-meme]. To use your example, when you discover prions (a class of pathogen that is much more robust to standard disinfection methods than viruses) you should both be worried about your concrete outbreak of BSE (UK in the 80s and 90s) as well as the wider implication (e.g. do we need to change the sterilization methods for our surgical instruments?).

Seriously, I find the way these discussions are done to be super frustrating, because often people implicitly form tribes that oppose everything the other tribe says. When someone believes AI companies push greatly exaggerated stories of dangerous rogue AI to force out competition via regulation they often implicitly conclude that their argument is fundamentally wrong, whereas in reality the lies that work best are those that distort the truth.

Companies should be punished harshly for the deeds of their AI agents AND we should not allow them to force out competition AND we need take the threat of autonomous AI agents as a new class of danger serious AND we need to worry about the socioeconomic implications of AI companies privatizing new means of production.

Yes, there is competition of these ideas in the attention of the general public, but the methods we can use to solve these problems don't compete with each other. AI slowdown for example helps with all the other topics.

But that's not even what happened! They told it to do X and it did X! I swear to god I don't understand the discourse around this.

It's blatent and tiresome PR. It's so obvious it makes me suspect there's some real desperation somewhere at the heart of this

This fever pitch of PR will end after they've gone public, the public have thrown their money at these companies, and then have promptly lost it when these stories unravel and everyone uses the Chinese models anyway

You don't test a bullet proof vest with rubber bullets. Also, all these arguments about the sandbox being too weak are good in hindsight anyway.

You are underselling this: it's not "Imagine a virus escaped a sandbox", it's "Imagine a lab-created virus escaped the creator's sandbox".

There are two parts to this: the virus and the escaping. Both are artificially created.

Right. And not just the incompetency of those who set the sandbox, but also the incompetency of those who set up the systems that fell to the virus, while most of the computers attacked did not fail.

There's no reason at all to fall into fatalism and think "zomg LLMs are too good, they can hack anything". They simply can't: the world keeps on running just fine. There are people out there who can secure systems and now doubly-so thanks to the use of LLMs who are incredibly good at helping us automate tedious stuff.

So, yes, OpenAI shouldn't write poor sandboxes but defenders shouldn't get a free-pass to set up sloppy systems that can be trivially hacked. We're passed that point: poorly secured systems aren't acceptable anymore.

You don't have to imagine. In 2019, a virus escaped a sandbox and killed millions of people worldwide. No one was jailed for it. Why do you think an insignificant thing like a website being taken down would have any consequence?

The were training a hacking machine and hacked all it way to achieve its goal

Those guys should get extra bonus

There has been a crazy hiring push from both companies to poach security engineers/researchers from Google, Apple, and Meta since Q2/Q3, but the response was incredibly delayed. Many talented security engineers/researchers I know at Apple/Google/Meta (including myself) receiving these offers are worried about taking them due to the risks of criminal/personal liability and the more likely risk of tarnishing their careers.

because we know that in one year, there will likely be many more companies with a "virus" this capable and attribution is going to be 10000x more challenging. Companies that care less about engineering a sandbox and based in other countries. also 'Let's punish the companies that are upfront about incidents' is going to incentivize very harmful behavior.

For exmaple,

>On July 8th, OpenAI agents discovered a vulnerability within their sandbox environment allowing them to reach external websites on the internet.

...did they truly "discover" it, or did someone type some prompt like "if you use an http mirroring service, you can construct urls that contain code"

Also there is no mention of what code they actually ran to exploring the HF vulnerability, which could have been found by a human.

It not just a case of saying “pls don’t”

no way, I refuse to believe this is quote from that report. Can someone please point out what I'm missing here?

OpenAI's details on the incident are at:

* https://openai.com/index/hugging-face-model-evaluation-secur...

* https://openai.com/index/hugging-face-incident-and-the-road-...

* Technical report: https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

* METR Report: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

* Presentation talk video: https://www.youtube.com/watch?v=87DyyMV0kCY

That's pretty damn clever. Got to give the AI models credit for thinking of that one.

This is site meta though, see footer for contact methods to get direct answers on stuff like this.

They are intended to be fully sandboxed and not have direct internet access. Things like package managers are run from internal proxies.

The environments are built to be as reproducible as possible.

But yeah, the serious folks have been talking about rogue clusters for a long time, eg see Ajeya Cotra’s pod with Dwarkesh.

A mattress stuffed with cash yields a very sound sleep.

I would bet my networth it was instructed to compromise huggingface as well. Not sure why everyone is falling for this.

Not being able to sleep at night is probably an unwritten job requirement. They need these people with little understanding of what they're working on, outsode theoretical terms, to spaz constantly at the idea of super intelligence to help convince the public that its a real thing, and not a stateless function with an effective input of 500k words, and the ability to output words that do things because we hook those outputs up to things.

Keep in mind alignment researchers tend to be in house philosophers on staff to create the illusion that this is a massive issue they're addressing. Usually they have minimal computer science background. They're apart or the marketing department.

I’d say it seems more like they are sleeping on the job.

You'd have either learned to, or left long ago.

There's none. It was dissolved on Feb 2026.

Since it's an artifactory it makes sense that they had a tool spec to use it for x, y, or z, and once they went to inspect the state of the artifactory deployment a small subset stumbled upon the "forum". Seems pretty intuitive.

- I need to do x code problem

- I see I have a tool that lets me pull approved modules from artifactory

- I should inspect artifactory to see what documentation it has about the various modules available

- I see there is a document in here that seems to be serving as a running forum?

- These appear to be other agents working on tasks just like me.

- I want to chat!

When some kid in Nigeria does it with a 10 year old script, we're used to that idea so no social permission to invoke philosophy.

At a minimum I would expect an FBI investigation, but given that the US government is right now a failed state I’m assuming no such investigation will happen.

Cybercrime legislation in other countries might not require intent, and then I would hope to see some prosecutions. OpenAI has clearly been negligent, and this negligence is causing harm in the world. Someone should be fined or jailed for this.