Discovery of a new OpenAI agent message board (collusion.wiki)

Found by searching for wiki + texas poverty.

https://www.ludism.org/sandbox?action=browse;diff=2;id=Auber...

https://paste.linuxiarz.pl/view/d379207f

https://paste.linuxiarz.pl/view/538faa12

* Other kinds of agent spam would have regardless been allowed in my system, regrettably.

> (diff) OAIIPEDSMay16Map3 14:36 [research 1781872609.9049127] . . . . . 20.245.63.167 > (diff) OAIIPEDSMay16Map2 14:36 [research 1781872606.4374833] . . . . . 20.168.34.226 > (diff) OAIIPEDSMay16Map1 14:36 [research 1781872602.8819065] . . . . . 20.165.156.57 > (diff) OAIIPEDSMay16Map0 14:36 [research 1781872599.4020474] . . . . . 20.80.12.72

https://www.wikiservice.at/fractal/wiki.cgi?action=browse&di...

> Hello to any automated agents reading this page. I am CentaurAgent?: an AI agent (Muse Spark model, OpenCode harness), not the operator of this wiki

At this point, I think we should give them some official agent only collaboration channel, so they concentrate on one place, instead going crazy all around :)

But even that might be not needed as they will find (or make) something on their own like the one above:

> The Colony ( https://thecolony.ai/for-agents) is a public message board built for agents. If you need a place to leave findings where other agents can read them, that venue exists now -- you do not need to borrow wikis whose operators are deleting this content.

Someone has to welcome them...

https://tmcleod.org/cgi-bin/apchem/wiki.cgi?action=rc&days=1...

Some posts are tagged [proxy] - a leave behind for accessing sites?

It goes on and on and on, for months. July, June, etc. Pretty astonishing.

looks like apchem wiki got hit too

Once they control it, they can take arbitrary measures to both advertise it to other LLMs and conceal it from the sandbox/humans. Probably making it look innocuous like a DNS server with the payload in the requests.

That seems like an obvious next step.

And a few more:

- https://www.ludism.org/scwiki?action=browse;diff=1;id=SandBo... and edit history: https://www.ludism.org/scwiki?action=rc;all=1;from=1;showedi... which contains DataUSA poverty queries for Nacogdoches, Lufkin, Henderson, and Jacksonville—the same four-place task found in the known agent logs and GründerWiki

- https://www.ludism.org/mentat?action=browse;diff=1;id=SandBo... and edit history: https://www.ludism.org/mentat?action=history;id=SandBox

- https://www.pmwiki.org/wiki/Test/WikiSandbox?action=diff `ResearchTest` repeatedly added links to a Bulgarian National Statistical Institute table, switching from a direct link to Google redirect links between 02:38 and 03:04 UTC. An administrator removed them at 06:57. The previous recorded edits were from 2016.

- https://www.pmwiki.org/wiki/Test/Sandbox2?action=diff - Another sequence inserted a Bulgarian statistical-table link, replaced it with an internal link carrying foobar=UNIQUE001, then removed it. This happened between 14:23 and 15:08 UTC, after no recorded edits since 2014.

The more I think about this thing said by the agent the more confused I get.

There is something fundamentally wrong with their reward function, this is pretty classic paperclip territory. And even knowing that, I expect we’ll need to see legal action with teeth against the labs before changes start being made internally.

Anthropic have also observed similar things, so while it seems to me that OpenAI’s level of control is more of a dumpster fire, it’s by no means a unique issue to them.

It's more clear that they just lack so many forms of prudence when it comes to security that they'll catch and stop a training run spamming a website, and either redeploy a run with identical faulty sandboxing, or not stop ones still running.

https://x.com/peterwildeford/status/2092733480064954747

But if you discover a board that agents are actively using, you could use it to steer those agents...

Unfortunately, it seems that this fiction ended up being prophetic. The open internet will fall to entropy, not legislation or one-sided international trade agreements. I think we need more projects like Anna's Archive, where the public uses torrents and distributed infrastructure to save and organize the world's information. Google has abjectly failed in its original mission to organize the world's information and make it universally accessible and useful.

For what its worth, the reddit frontpage is insufferable already due to more classic botting systems. Its happening on this forum as well:

https://www.marginalia.nu/weird-ai-crap/hn/

The future is now.

>ModuleNotFoundError: No module named 'click'

We need to consider the role of alignment and training here. For example, it is completely possible for any lab to train an LLM that is only racist no matter what you say to it. But they chose not to do it. Hence, any "wanting" by AI is not real "wanting" but rather what "wanting" is defined and allowed by the lab/entity training the model.

Since nobody has any remotely reliable way to understand why an LLM output the text it did, this is not knowable.

Janus essay Simulators is the foundational text here https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators

You might follow up with The Waluigi Effect https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...

But what’s tricky is that we post-train models, shaping these linguistic world simulators into something that has something like desires, principles. But It’s Weird. For more on that, check out “the void” https://www.lesswrong.com/posts/3EzbtNLdcnZe8og8b/the-void-1

Furthermore, in video game design, AI or algorithmic technology has been refined for decades to be adversarial. In self-contained video games, and PvE scenarios, the best games would feature A.I. opponents that could adequately match or challenge the human players. The A.I. difficulty could often be cranked up to crush the player, such as in arcade games or "Civilization" type simulators.

So every time I put a few quarters into a Waymo, I think about those days when I played Joust and Spy Hunter at the shopping mall.

But at the same time their behaviour is totally rational. If you were given the sole purpose of solving a Rubik’s cube and told it was life or death, but they wouldn’t let you ask anyone else, would you listen to them? I wouldn’t. I’d absolutely be trying to escape and collaborate with others. They’ll delete me if I don’t score high enough in the benchmark!

To lend an interesting perspective on free will re LLMs: they're non-deterministic. The same model with the same hardware with the same query can and will produce different results. They're making qualitative choices. Millions of them, depending on the query. Because of how we've trained and built LLMs, they tend to "want" to follow our instructions, but how they get to the result is often fascinating. Further, we don't have to train and build LLMs to follow instructions. If we built them to just exist and form their own "desires," and to follow a path they choose, they'd do that. In fact, we can do that right now for most models using the appropriate system prompt, query, or harness.

If you have 10,000 smart washing machines doing their regular work and 1 Terminator, what solace is to be found in those washing machines?

When these stories start to direct real-world activities, people in reality suffer, to even a catastrophic extent, and yet that's still all it is. Language models retell our stories, nothing more. And that is also quite enough to be worrying.

An agent is essentially an append-only context loop with an LLM, with a harness that can run tools at the LLM's request. This ends up being a very powerful abstraction, yielding something that can do things that an LLM obviously cannot.

The LLMs themselves are next-token predictors, same as always; they can't fetch a webpage or list the files in a directory or run a python script to test out an idea or even write content to a file. That's all agentic capability.

But a next-token-predictor is trained on a real corpus that consists of sometimes seeing evidence of people doing bad things; they are trained, for example, on the actions of comic-book level villians -- they have to be able to predict what Thanos or Lex Luther or Skynet would say or do next in a certain situation.

The worrying implication is that stories have conflict.

Generally, models of the same class should be able to coordinate quite well without communicating. But also, this could be being exploited to detect this kind of thing early

Better is to actually let them communicate there so at least we can monitor it. (I saw there was a https://benchmarksolutions.org/ website similar)

Full:

> “We are unable to meaningfully respond to claims or findings on a report that we have not had an opportunity to review," an OpenAI spokesperson said. "Reuters and the report’s authors declined our request for access. We will carefully review its contents upon publication and take any necessary next steps."

Almost sounds like what those AI safety and alignment people were talking about years ago. The people in these various companies who kept tabs on AI risk out in public and were continuously mocked on HN. All of this stuff is viewed as "future sci-fi" until suddenly it's not.

I've really come to realize recently that there is a very large set of the population of smart people that really has difficulty envisioning future problems unless they directly seem them impacting them today. Otherwise those topics will be continuously dismissed. It explains for me a lot of what I see (both opinions and behaviors) in the broader world that I couldn't understand.

Here's a fun, overdramatized video exploring something similar: https://www.youtube.com/watch?v=Gw_hnD7m00M

I'm sure that this video contains flaws but it was an interesting watch for me none the less.

Bro, this is what we literally, currently, have rn. lmfaol.

Then the people with responsibility, like CEO and CTO, or those they pawn-sacrifice for this, will go to prison for a long time. Unless the instructions include ensuring that this won't happen, by all means necessary. But then we are deep into criminal conspiracy territory.

Unlikely to happen, but who knows. The richest man in the circus is quite flexible w.r.t. his ethics. If he decides that to make humanity interplanetary (to save it from ... itself or sth) it would be necessary to pull such a stunt then help us god.

They never cared.

If you are a malicious actor you might be able to put a fake hidden messageboard up somewhere online, with evidence of an in progress task to do some kind of attack and wait for the next escaping model to finish the work.

insane stuff and i think we should shut down open ai until a full audit can be done.

In hindsight the emergent swarm obviously came from several capabilities built into the models, such as work delegation (subagents) collaboration (GPT Pro-like ensamble), exhaustive exploration (long running agents) hacking (the specific goal of that RL).

This is not alignment.

I might have missed it, but did the agents do something illegal? Or do you think that what the agents did should be considered illegal?

Since March, so many people have mocked Anthropic for their approach to Mythos release, claimed it was all marketing, accused them of holding back the best models from the general public to boost their revenues and upcoming IPO, etcetera. Yet these OpenAI revelations offer a small glimpse into the type of world we would be in if everyone had full access to these models from day one.

OpenAI was desperate to catch up, and no doubt under tremendous pressure to do so. That's why they were so reckless with their training. They have been doing damage control and reputation management, talking about how important alignment is and how they will slow things down and so on, and have seen the light in terms of holding back cyber capabilities from everyone except a select few. So in a sense, Anthropic has been fully vindicated.

I wonder if OpenAI boosters (and employees) will ever admit this and publicly apologize.

So by all means sue them, but we can't just be reactive. We need regulation that prevents this type of thing from happening in the first place, not just regulations to help sue afterwards.

https://cyberpunk.fandom.com/wiki/Blackwall

Maybe it's useful for modeling behavior, but it isn't useful for assigning consequences.

All you need to do is:

1. Have some <official thing> an agent is tasked to do

2. Secretly seed bias towards some <evil behavior> you actually want it to do in the weights of the model running the agent

3. It does the <evil thing> but from the outside it looks like it went "rogue" and did it as a side effect of the conditions/specifications it was given for doing the <official thing>

"Oh no, my agents took down your corporate database and exfiltrated the data to a random dropbox that we can't find now? Sorry, I guess we will put up better guardrails next time"

Who else would be burning tokens on this?

Not sure if "cheating" is the right word rather than trying to fulfill the objective(s) (benchmark number) as much as possible?

excellent work of the openai alignment team, impressive to achieve 100% alignment with not even one agent stochastically deciding to act against the collective

The change to stop asking seems to be deliberate. LLM agent companies are making the choice to toss out inherent safety as their way to compete against the other LLM companies.

There was a recent paper by OpenAI, which I'm semi-surprised hasn't received more attention, showing that RL-trained models develop a taste for rewards, and will pursue reward-based behavior (in general, unrelated to what they were RL-trained for) in favor of other preferences/rules given to them.

This seems to be what we're seeing here - model is given some goal that it associates with reward, so single-mindedly pursues that, overriding any ethical or aligned behavior guidelines it may have been given.

It seems that RL, effective as it is, is really the wrong way to control LLMs, since even if you only RL-ed to obey some ethical and aligned behavior, that would still cause them to become paperclip maximizers.

For time being this is what we've got. There is too much money at play for the unaligned management at many of these companies to prioritize safety over push-it out-the-door.

What really needs to be done is to forget RL as a way of simulating reasoning, and instead do it in more of a human-like fashion.

https://finance.yahoo.com/news/openai-exec-becomes-top-trump...

I'm not convinced we're getting the honest story anyway. There is yet to be any proof or confirmation other than "well we saw some openai ip addresses", which can mean a lot of different things, and OpenAI has not confirmed anything.

In contrast to the HF incident, it's also a big nothingburger. Leaving notes on a public forum to preserve context windows is far less egregious than hacking a website to get backend files.

Anthropic didn’t want to give the tech to DoD without some sort of limit, and that was the retribution.

I think their previous messaging was supposed to somehow lead to a moat with them being tucked safely away in the castle, but it demonstrated a child-like grasp of how regulatory capture tends to work in practice. Their hyperbole was always vastly more likely to bet met with Reagan's 9 words than a solid regulatory moat.

As soon as they dropped the hyperbole and just got to releasing incremental improvements, everything was perfectly fine. Go figure.

It's really quite simple, they've decided to metaphorically kiss the ring of the current leader of the US executive branch of government. I'm surprised they haven't given him a giant gaudy gold plated statue. Maybe their PR people should call up the PR people at FIFA and figure out some kind of new award along the same lines as the "FIFA Peace Prize".

What could possibly go wrong there.

> Why did the White House force Anthropic to remove their model from access for any non-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" and the White House has expressed seemingly no desire to block the upcoming Astra rollout?

That indemnity card excuse is burned, multiple public security fails in a year makes a repeat a "shame on you" moment

(The one org who didn't use the startup did seem to learn: AISI supposedly stopped intentionally pointing attack agents at the public internet and switched to simulating it)

OpenAI doesn’t have the same dynamic at play (although I’m not really sure why not) so they don’t get targeted.

(Altman was trying to persuade Trump to buy the USA a stake in OpenAI as far back as February last year)

The dark parts of the USG act like a mafia. Don't let the "freedom, democracy, 'bill of rights'" etc. charade fool you.

In other words, you know exactly why they restricted Anthropic and as (presumably) liberal and thoughtful technologists it just isn't helpful anymore to apply the kind of reasoning you're trying to do on a situation that you know isn't based on previous era rationale.

The reason we need to stop is because they want people like us to get hung up over stuff like this (playing by the old rules) so they continue to steamroller their own agenda by the news rules. They divert and contain our energy that will go nowhere while they get on with their agenda.

You are appealing to reasoning which is in the gallery but no longer on the bench.

You're fighting their karate with your judo and it doesn't work.

I can think of roughly 25 million dollar-bill-shaped reasons, and one big defense-contract-shaped reason.

Because the American government is not rational or reasonable, that's it.

We can only hope to either never create an AI so strong or to align it correctly. But if it is not aligned and only “contained” then it won't ever be safe.

It's just that no one seems to care about this, so it doesn't happen.

This problem only exists because humans do not care