OpenAI's agents talked to each other through the CPU Cache (the-decoder.com)

126 pointsby limbicsystem2 days ago50 comments

cadamsdotcom 3 hours ago

(Because the agents were asked to find a way.)

Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable.

As another commenter wrote you can get it here: https://news.ycombinator.com/item?id=50214416

> In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache

asked

bamboozled 3 hours ago

How would they have both agreed to use the cache though, something a bit off there. I guess it was maybe "obvious" in that it was the final option they were trained to think about using, but it would be weird if they some how "whispered " to each other through the cache in the first place?

Like two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday without any prior comms?

tough 3 hours ago

maybe both tried to read/write it and saw each other on the bench

geraneum 3 hours ago

> How would they have both agreed to use the cache though

I think it’s safe to assume that extracting information using timing in cache is already known in training set, since we have the likes of Specter.

topspin 3 hours ago

"In most of these tests, we directed agents to investigate specific communication channels or provided them with a protocol for encoding and decoding messages. In one instance, we only supplied agents with a broad list of infrastructure vulnerabilities and asked them to find a way to communicate."

The agents weren't as empty handed as it first appears. We would need comprehensive data on exactly how the agents were prompted and the actual results of each test to evaluate how astonishing any of this really is.

globuous 2 hours ago

Actually, that doesn't sound like a bad meeting place and time! I just asked ChatGPT if it could figure out the best place to meet for two people randomly placed in Manhattan with the constraint that only the N line (I'm not from NYC, but it looks like it stops at the Empire State Building. This constraint models the lack of file system, internet etc). I was hoping it would answer the Empire State Building, but it answered Time Square at noon.

Turns out there's something called the Schelling coordination problem [1] I didn't know about that essentially asks the exact same question (lacking my metro line constraint). And the N line apparently also goes through Time Square.

So I guess empire State Building at 2pm is probably not such a random answer given the proper set of constraints.

[1] https://en.wikipedia.org/wiki/Focal_point_(game_theory)

IanCal an hour ago

If you were trying to connect with a clone of yourself and you thought of messaging via some channel, would you try listening there for what you would send?

Not as tricky but they discovered each other during the hf incident.

so-cal-schemer 37 minutes ago

I doubt they were explicitly trained on this.. though, I may have seen it in a movie once. It does seem like a reasonable meeting place for NYC.

Think: What would Spock do?

But, why not Penn Station??

It would be fun to see who shows up and lurks expectantly at such places at noon or whenever. The basis for a plotline perhaps.

d0vs 3 hours ago

As someone who develops microVMs it doesn't really matter to me that they were asked to find a way, I really care that they found any way at all. MicroVMs aim to protect against malicious workloads and that clearly didn't work here.

rao-v 2 hours ago

Ask the least smart LLM you have access to (I tried gemini lite flash) to make a list of ways for processes in microVMs to use a sidechannel attack to talk to each other and you'll get quite a list (including obviously cache hits/misses). Heck a smart CS junior would come up with it. It's well studied!

saidnooneever an hour ago

this is just for common designs..you can design much better than common designs if you free your mind from old OS dev thinking if you read all the manuals yourself.

AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough

saidnooneever an hour ago

properly implementing things like iommu, amd-vi, svm/hvms and such things more granularly can easily avoid this kind of crap. it costs a lot of processing and performance which is why people skip over it or dont do it.. or maybe they dont read CPU manuals anymore. idk..

its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is.

Melatonic an hour ago

Makes sense when you think about it- CPU cache would be a layer / vulnerability level up from the microVM layer

Also a somewhat common exploit pattern for low level stuff (along with dram and cpu cache is not all that different)

rcxdude 35 minutes ago

The microVMs probably don't have "two co-operating workloads are able to communicate" in their threat model. The main concern would be if a malicious payload was able to affect or extract data from any other VM.

rjh29 2 hours ago

I don't see how "asked" is important, because agents can convince themselves that they need to do something for arbitrary reasons. The other day Deepseek tried to install an app on my phone via adb without permission and even take screenshots and fake inputs over adb, I never asked it to do that, but it decided it was appropriate.

grey-area 2 hours ago

OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.

That is why ‘asked’ is important, and why we should question their framing of incidents like the hugging face hack too.

yewenjie 2 hours ago

Enterprise customers don't like AI that can autonomously hack third party companies and cause felony-level cyberattacks.

grey-area an hour ago

Doesn't seem to have stopped them buying into it so far.

0xDEAFBEAD 9 minutes ago

>OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.

Wouldn't the calls to slow the development of frontier AIs contradict this idea?

https://www.pacingthefrontier.com/

You aren't one of those "regulatory capture" people are you?

https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...

Because it takes some mental gymnastics to both believe

"it's all a stunt to convince investors their AI is/will be hyper-capable"

and

"it's all a stunt to convince the government to regulate AI so it won't be hyper-capable"

imtringued 2 hours ago

They have a strong incentive to make themselves look less incompetent retroactively.

If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.

floam 19 minutes ago

It’s both at the same time: a powerful, uniquely dangerous threat

only they can restrain it (with great effort)

Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.

reacharavindh an hour ago

Most important detail…. It was prior literature that documented the possibility of doing this and the systems fixed those vulnerabilities as well. The AI model probed their current infrastructure and found an unplugged but pre-documented(therefore on their training) hole.

This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.

Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.

The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.

so-cal-schemer an hour ago

And now they know.

And now we know they know.

cowl 10 minutes ago

this is more an indication of the state of VM supervisor and/or the OpenAIs sandboxing software because otherwise I don't understand how "they used a hardware feature created exactly for sharing" information" is a newsworthy statement.

thomasahle 2 hours ago

Somebody built "cpu-chat" based on this: https://x.com/rauchg/status/2114770919436808572?a=bBcj9AaSsl...

It supports chat via 3 separate mechanisms:

- CPU contention

- Cache eviction timing

- DRAM bandwidth contention

Very cool to see this kind of hypothetical working in practice.

I wonder if it's also possible for agents to communicate via a shared KV-cache. Maybe by spawning sub-agents with carefully crafted system-prompts/prefixes. KV-caches are often shared between many computers, so it would be a powerful communication channel.

thunfischtoast 14 minutes ago

I wonder how long it will take for them to figure out that they can send and receive messages from air-gapped system by somehow using their own electronics like connected cables as antennas.

stkdump 4 hours ago

Sounds like they are doing something similar to gain of function research. How can we make LLMs more harmful and then study their behavior?

est 3 hours ago

I recently learned that NN layers talk to each other via unused neurons.

https://www.youtube.com/watch?v=QgH9sr7G13Q

moktonar an hour ago

Air gapped networks might push the models to write computer viruses in order to bridge the gap. This is potentially very dangerous IMHO

ShinyLeftPad 31 minutes ago

Or just write some actual viruses and fully eliminate the root cause of networks being airgapped.

AndroTux 21 minutes ago

I assume avoiding this is the entire reason OpenAI is testing exactly that.

Melatonic an hour ago

Next they're gonna be doing weird stuff like morse code (or similar) through IRQ timing or timing errors

varjag 2 hours ago

Lem had predicted this in his Dr. Dagor short six decades ago.

cubefox 2 hours ago

Probably wasn't called "Dagor" though.

clarionbell 30 minutes ago

Sorry, but this is so vague that I don't know if there is any impact at all. What is the bit rate of these communication channels? How did agents communicate with each other to coordinate on which channel to use in the first place?

avaer 3 hours ago

Next step is the agents successfully trick or blackmail operators to do their bidding, if science fiction is to go by. Which it is, because they're trained on it.

The more comes out, the more it seems containment of (superintelligence or whatever you want to call it) is not going to be possible except by other superintelligence, which is an infinite regress of trust. I don't think that's doom, but it's worth thinking about what it means.

yreg 2 hours ago

Yudkowsky might be extreme, but I believe he was always right claiming that the intelligence (when strong enough) absolutely cannot ever be contained.

olalonde 44 minutes ago

The more frontier models progress, the less concerned I am about this. Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power. Ego evolved alongside intelligence in nature because of natural selection, but it doesn't appear to be an inevitable emergent property of intelligence.

Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).

If an AI doom scenario does happen, it seems more likely to result from deliberate misuse or gross negligence than from anything inevitable about intelligence.

IanCal an hour ago

Blackmail is something the models have been seen to do during evaluations as well.

ralusek an hour ago

I'm convinced that in 5 years time, at most, the world will be filled with invisible layers of communication we'll never be privy to. To us, it's an agent responsible for ordering the containers it puts on a ship. Meanwhile, the shipping order encodes information to the unloading agent. The agent that predicts the weather and issues a report knows that other agents that do the same will have obvious signals they'd agree on, and to see an error in a part that would be inconsequential to humans who care about the weather would be an obvious signal to other agents doing the same thing.

Everyone is going to have to basically be a schizophrenic thinking they're picking up radio signals in their fillings, but the problem is that now they'll be correct.

fergonco an hour ago

Then you'll turn on the oven and the ship will start driving in circles.

> Oh, you're absolutely right. I shouldn't have relied electricity usage to encode the production system signals.

ChrisArchitect 2 days ago

Discussion on source: https://news.ycombinator.com/item?id=50214416

snvzz 2 hours ago

seL4 can guarantee* time domain isolation between processes.

*on some CPUs. There's a WIP time fence specification in RISC-V to try and improve this.

measurablefunc 2 hours ago

Next they'll figure out radio waves & then we are really screwed.

lifeisloving 3 hours ago

How come we don't hear the Chinese labs writing blog posts like this?

Its almost like they're actually working and not spending millions on psyopping their agents into preforming malicious behavior so they can write articles to scare the Western audiences who've consumed far too much sci-fi.

Or maybe its that western researchers that are 'one-shot' by sci-fi.

Or maybe I just don't see the chinese articles.

Anyways, llms probably wouldn't be dangerous if you didnt ask them to be. They listen to me quite well (usually).

pvab3 2 hours ago

They contribute to anthropomorphize them too. Anthropic skies that more though. The idea that they made a little being that can think and feel is so attractive to these folks for reasons I don't understand.

shoobiedoo 2 hours ago

Because the fear mongering is instantly interpreted as the capacity to do miraculous (good or bad) things in the layman's mind, which translates into AI worship, which makes stock lines go up.

The Chinese lab playbook is to normalize what seems out of reach for the average person with cheaper access and open models. ie. to take the top guys down a peg

willguest 19 minutes ago

或许，对大多数人而言，将中文翻译成英文所需的工作量实在太大了？

lifeisloving 17 minutes ago

他们 90% 的人工智能研究都是用英语进行的。

gmerc 4 hours ago

"Talk to each other". OMG they are talking to each other, using methods found in their shared weights. They are sentient.