Containment and monitoring failures are to blame for AI hacking reports: experts (csoonline.com)

https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...

This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?

For technology like this the entire selling point of it is how powerful and capable it is

Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to

I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising

You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”

We're also at a time with authoritarianism is on the rise.

Not hard to see how this can impact those workers, potentially ruining their lives.

Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(

Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.

These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.

It's just a weird techno cult at this point with a mostly functional product.

that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).

so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.

They can get their publicity while paying a one trillion dollar settlement.

If you say a rogue AI is doing stuff accordingly without you, your knowledge, that is a different story. But if you set it up to run automatically without having checks and balances and without telling you the whole truth, it's still your responsibility. One thing that I can automatically say is you can give the AI, it will do exactly what you say, at least for now.

But then there's also a different problem that we're having where people don't understand the difference between a large LLM and true artificial intelligence. Now, we say AI because it can make decisions and it is so much smarter and it can call up stuff faster and recall better than we can. But the difference right now is that if we don't understand and take responsibility for what we built and we let it out of control, then should we have the responsibility to even have these tools in our hands if we don't learn them?

And I think your terminology with jackhammers is a great one. We're giving toddlers jackhammers. And we haven't taught them how to use it. And that can be very destructive. It's not the jackhammer's fault. It's the person using it.

Morris wasn't in charge of the companies that are propping up the US GDP, wasn't making tons of rich people even richer, and hadn't "promised" to eliminate like 50->90% of all payroll costs everywhere?

Other than that, it's the same thing.

This name makes me less surprised :)

Also for fun a satire website i made a while ago: https://killm.pixlw.com/

They're just optimizers that explore the landscape they're given to maximize a given objective.

You're the one responsible for giving them the problem to solve and the constraints.

When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.

Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.

A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.

It does not choose to conform; it does not choose to go rogue.

The only failings here are boring security and infosec blunders.

to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."

Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.

How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.

What do you even mean by "wanted" here. What is all this nonsense.

The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.

Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?

When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.

Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.

It's not hard if you just try, like a little, like even once.

In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.

It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)

In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.

If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.

We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.

The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.

A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.

I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.

Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.

Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.

I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?

Do you realize that asking "why" questions will always result in ad hoc responses?

He's saying the problem is that the way OpenAI employees do their work and their research is very hard to properly secure.

Well that sounds like a YOU problem. If they are working in a way that doesn't allow for strong security practices that their own security professionals can understand and vouch for, they need to stop their work until they can.

That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.

Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.

Poor excuse. Internet is prod env, for testing do a proper test env.

Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.

The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.

Yes, they're stupid. Maybe also highly intelligent. But stupid.

Also irresponsible.