I resigned from Anthropic today (twitter.com)

The OP is talking about existential threats, not things that personally annoy you.

Who doesn't want their work and what they're doing to matter? This is the ultimate mattering.

And with that, you also get your own hero story.

Same dysfunction as always. Our branch of the economy is funda-mentally unwell.

With LLMs we don't even hesitate to call it black box while still pushing its capabilities.

Nukes: zero possibility of extinction

Wealth inequality: this one is driver of progress, opposite of extinction

War: another driver of progress, also will always naturally stop before every single human is dead

He is blowing the whistle on how reckless we are at furthering the tech. There is no second thought at maybe development of this tech is not a good idea. Its just full speed ahead.

And then the thing with exponential growth is that the last thing is worth than the previous one for its potency for destruction. While it's true that, all things considered, the curve started to explode with the use of fossils fuel we've never limit ourselves with efficiency gain, aka Jevon's paradox.

Both parties are convinced that their take is so blatantly obvious as to not require justification. It feels like the only thing these kinds of takes justify is the point-of-view that nobody knows how this is going to play out.

    // Anthropic / Apple - Nah, they can do no wrong. Even after both were caught violating users' trust and privacy multiple times, GOOD

    if (company in ["Anthropic", "Apple"]) do

      GOOD

    // Google / everyone else - Doesn't matter even if it's a good deed they've done, BAD

    else

      BAD

    end

But most probably care less about human survival, and more about survival of civilization.

In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?

Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).

To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.

[1] https://www.youtube.com/watch?v=1qIvFpcGkqc

The difference is that none of those are available to individuals.

AI is an exception. We're already losing understanding (although we never fully had it in the first place), and we're losing control (see jailbreaks/hacks etc.).

We're still far from the doomsday scenario because AI 1. is not developed enough yet, 2. can't really replicate itself, and 3. has very limited means to act.

But: 1. its intelligence is developing quickly, 2. hardware capable of "hosting" it is slowly being developed, and 3. it will likely gain access to increasingly powerful means of acting in the physical world (this is already happening in the digital world).

Once AI becomes intelligent enough (it doesn't strictly need to be AGI), has the substrate on which to exist, and has more means to act, we'll essentially have a new species on Earth - one more capable than humans and potentially determined to kill humans at some point.

And for those who believe airgapping is a valid safety measure: read https://xkcd.com/538. AI will be able to threaten and manipulate people.

Open weights agents with hacking capacities can reproduce themselves into the systems they hack (non-open weights ones will have to hack their creators first). Not saying they will, but if they do, good luck finding the kill switch.

Once swarms of agents run unsupervised on unmonitored hacked hardware, who can tell what they will do? The Huggingface incident showed that such swarms behave without any safeguard. It was a real HAL moment.

A lot of things are possible then: ransomware campaign, taking over IoT devices, self driving cars, planes, ships, satellites, missile launchers. If nothing's out of reach, everything is possible.

Bring robots into the mix, and the possibilities are endless.

I'm not particularly frightened tbh, but we shouldn't discard the worst-case scenario, and the worst-case scenario doesn't look good.

If such AI decides that energy spent toward human agriculture (biological food that the AI does not need) is less important than work spent toward storage and production of energy (electricity that the AI needs), then why wouldn't it just try to re-direct resources from the former to the latter. And the AI is some sentient superintelligence, I think it is safe to assume that it will be able to outmaneuver human safeguards.

Obviously that is still just a very hypothetical sci-fi scenario, but the consequences could be very dramatic.

Perhaps, but if there’s ever been a time to consider something that sounds hyperbolic, that time is now. The ramifications of what AI might based on events up until now are concerning.

But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event.

If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose).

For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now.

To be fair, that's a conservative "defend against the last war" kind of prediction, though!

( ref for part of it: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... , recent hn ref: https://news.ycombinator.com/item?id=49563355 )

Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.

Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown.

Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.

I don't know, maybe a mass shooting?

https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-...

Oh, and let's just forget the uncountable early deaths from the environmental disaster of the Datacenter buildout. It's not as sexy and doesn't make headlines, so those deaths don't really count or matter do they?

Changes in political and economic power balance leading to unrest, conflict, death and deprivation is not a wild theory. It is literally the story of our entire species. If you discount all such concerns, you are simply being willfully ignorant of past precedents.

In fact, I challenge you to describe any non-AI civilization-level danger which is not intimately tied to political and economic relationships between and within societies.

The big labs scrape LITERALLY EVEYTHING and get fresh data from their users. Both of the big labs have massive contracts with defense agencies. If the open-weights models are just distillations of FMs...

Nuclear weapons don’t have AI but AI can have nuclear weapons

What's worse, we don't have any transparency or insight into what labs are producing nor any way to stop it if the risks exceed our tolerance.

It's basically 4 years in now, so that's the wrong question. I mean, if you're raising an apex predator that has a lifetime measured in centuries, at 4 years old the thing is still basically helpless and completely reliant on you, so you're pretty safe from it.

If AI really is all that they are telling us it is, then it may "kill us all". But that's a really big "if" because we can't tell if they are lying or not.

The real problem is that ASI is an ELE for humans, even if it doesn't try to kill us all, or even if it doesn't kill us all.

By the point you would have been worried, would it have been too late?

- Escape sandbox

- Reproduce itself

- Find a way to run a financially profitable business (maybe with a meat and bones puppet somewhere in-between)

- Setup or buy a social network

- start manipulating public opinion on that network to support legislation allowing AI to

* operate businesses

* setup legal entities

* purchase weapons

* donate to political parties

* setup private armies

* you get the idea

I think this is a good example of poor risk management reasoning. there is evidence bioengineering is already happening. No, nobody is going to announce when somebody has decided to use these tools (even if isn’t an LLM) to bioengineer a weapon. Are the tools power enough to do so? Not sure.

But I’m just ambivalent. It’s probably bad. But there’s nothing to do about it. We’ve really only just pulled back the lid on Pandora’s box.

> Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that.

There are things that, by the time you see direct observable evidence for them, it's probably too late.

Also your example 3 is a straw-man. There's no need for "widespread starvation" to be concerned about "AI driven labor market disruptions."

Your average Anthropic ideologue - including and most especially the main man himself - would love nothing more than to eradicate 9/10ths of the planet's population, pump the survivors full of memory wiping drugs, bury the existence of AI deep underground and rule from the shadows for the next thousands of years.

This would be their wet dream. All in the name of "saving humanity from itself" - so they can convince themselves they're the good guys and deserving of this power. Anyone seen the latest season of Silo by the way?

Well the worries about AI are equivalent in that those risks are discussed now because discussing them after they’ve happened is clearly too late.

That’s the thing about risk. There’s no point discussing it after it’s happened and any discussions beforehand can easily be hand waved away as “it’s just a small group of unrelated individuals” or “it’s unlikely to happen to me”.

So yeah, your points are true. But they’re also moot.

For instance have you seen what this has done to the school system? We're not equipped or ready to handle the changes. Consequences are unknown.

AI doomers: End of the century? Hold my beer.

This sounds like asking "What's the most dangerous thing that's happened from global warming so far?"

It's not where we're at, it's where we're headed if there isn't huge coordinated action now. You can see how that kind of thing has been going for global warming so far, and by all measures AI seems to be headed for the inflection point of unstoppability at a much faster pace.

And this warning is coming from someone who just spent three years working inside these companies and is likely aware of much more than has been publicly released.

Maybe it's all marketing bullshit (I hope), but it's also playing out exactly like I expect it would if it's not.

One good source I can recommend listening to: https://pca.st/episode/d821fced-b4d5-4c84-b0a3-6ebe913fa638

I think we should stop before then. But that necessarily means that there will not be evidence at the time that we stop.

If you go to the various CEO blogs or misc people within this sphere and peruse their lists, they don't scratch the surface. It's all pretty vanilla stuff.

"Nothing bad happened yet" doesn't really seem like an argument to me.

https://www.science.org/content/article/made-order-bioweapon...

Being able to use AI to generate the steps to synthesize proteins means that you can use it to use it to generate the steps to synthesize known toxins. Suddenly, once difficult to attain knowledge is now available to everyone.

The fear is that developing AI at full speed will lead to giving people the ability to do incredible harm. Like something worse than a machine gun.

Intelligence is the root cause underlying those dangers.

Nuclear weapons, carbon emissions, biological weapons, and other such civilization-scale threats to humanity, are a product of our goal-seeking intelligence and ingenuity, and are only actively dangerous because of our ongoing use of our intelligence.

AI is about reifying that intelligence and ingenuity and making it run independently on computers, and act on the world.

That includes, in principle, ability to use nuclear and biological and other weapons, and also the ability to come up with some new threats too.

Even today, AI is the biggest discrete threat to bringing carbon emissions under control. Almost everyone wants to decarbonise… except Trump. Renewables are the cheapest source of new power… but the demand for new electricity for the data centres is so high that all options are on the table, anyone who can manufacture a power source (even when it's a jet engine) is being propositioned for them.

Nukes: is it not common knowledge that the cities of Hiroshima and Nagasaki are currently thriving? That the ground-zeros of the two bombs are memorials, not barren wastes?

Even if all the nuclear powers gave maximum response at first sign of one strike, is anyone pointing a single weapon anywhere in Africa, South America, Central America, or the bits of east Asia (east specifically: obviously Pakistan and India are pointing theirs at each other) that are neither mainland China nor US bases?

(Possibly an unanswerable question given military secrecy; and while I can't think why anyone would anyone point any weapons those ways, that doesn't mean someone with such weapons has not).

https://www.google.de/maps/place/Atomic+Bomb+Hypocenter+Monu...

https://www.google.de/maps/place/Hiroshima+Atomic+Bomb+Hypoc...

What if one day Trump or Putin tell their awesome military AI to come up for plans to end Ukraine/Iran/... war. The AI get's to work, but is overly eager and not just comes up with a plan, but starts executing it (claude does that way too often for me).

And the plan was to use a tactical nuclear weapon as all the other solutions do not end the conflict.

Now the agents realize, oh, they do not have access to the nuclear arsenal, but they need it to succeed. So they start to hack into the system. Get till inner network, learn what is needed for deeper access - human access - so they record voices and speech patterns of commanding officers and their habits - then synthesize their voice and do call underlings with the voice of authority to get the rest of what information they need. Boom.

Too far stretched? I surely hope so.

But we are working on making the technical foundations for this scenario possible. And with idiots in power, it might be even easier that serious screw ups happen. You know, randomly adding contacts to a secret Signal group to talk about government stuff - the same way you can add a bot to something else and give permission to do way more.

In 1900, there was no evidence of the kind you seek that lighter-than-air flight was possible. Good thing some people were foolish enough to ignore you then. Why you would cordon yourself off from the kind of reasoning that predicts legitimately new things, rather than just scaled up versions of the present?

Not to put too fine a point on it, every example you give is based in concrete evidence, some try to think through the implications farther than others, resulting in larger or smaller error bars around the conclusions.

Let me back up though. Maybe in the collaborative human effort here, we are better off having very concrete thinkers, like you seem to be, along with abstract thinkers like the "divorced-from-reality" Rationalists. Personally, I wish we had a stronger culture of collaboration and assuming good-faith and competence in our peers. In my experience, blanket dismissals are very rarely grounded in reality and mostly grounded in fears.

1. Invent transformer architecture.

2. Scale it up.

3. ???

4. Machines become sentient and kill us all.

OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no.

But because we live in a culture of fear, everyone eats it up no questions asked.

First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae.

Second, and more important, consider this fully non-magic route to doom:

- We continue putting AI in charge of more things

- It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it

- Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping

- AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect

- Game over

That's not what I am worried about at all.

I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"

At least the first two should seem like a near-term worry after the past five months.

Leaving OAI or A\ would cost people millions, tens of millions, or more. And for what? So someone else can take your seat and do the same thing anyway?

If you're smart enough to get a job there, you're smart enough to be able to talk yourself into why it makes sense for you to stay.

Huge kudos to people like this who make the hard choice against the easy way out.

Needless to say, for practical purposes am back to 5.6/Opus 4.6-4.8. But hey, maybe I am not smart enough to use LLMs?

Not invented here; if I can’t figure it out no one can

Or plain old lack of grasp of the material so no ability to follow necessary train of thought to appropriate conclusions

Similar in lacking context but different in how that lack of context is expressed

What events are you talking about?

What should we do? Freak out? Maybe this sentiment would be taken more seriously if there was a real call to action included. Shall we protest? Vote in a specific way? Call representatives? If your solution is that we should just be scared, then of course there’d be not much value in what you bring to the table.

After the blatant marketing campaigns of the summer, you mean. do you need a reminder that those very same people had touted GPT-2 as a dangerous model?

worrying about sci-fi doomsday scenarios with the current AI tech is absurd. LLMs predict the next token, that's literally all they do. they aren't going to escape into the cyberspace, self-replicate, self-improve, jump over air gaps and launch the nukes at John Connor's grandma. they can't. people pretend to believe the dumbest shit.

The replies here just adds AI to the list of domains.

   - virus research labs
   - nuclear labs
   - chemical factories
   - bank records
   - land registers
   - power plants
   - water supply and treatment plants

The third law of thermodynamics is an essential barrier in all engineering.

Give enough access to a fuzzer and it's exactly as dangerous as an LLM. LLMs don't even have a moat in this domain.

OTOH, if the goal is simple ("break into a protected system") rather than more complex ("write an application that satisfies all requirements on all supported devices/screen resolutions etc."), that's of course more suitable for a harness.

https://ghuntley.com/ralph/

I share your concern...

Yes, that's hard. It's a challenge. You can set your brain on the challenge! It's a complex and interesting geopolitical and technical challenge.

A good starting point is to read Plan A https://ai-2040.com/

Look at all the things needed for that to happen, and think, how can we step towards making that happen?

You are just too funny.

US has 15 to 25 million people unprotected from Polio.

Polio is still endemic in Afghanistan and Pakistan.

Technically, he is not. He returns in the final scene.

In the game Outer Wilds, the ending is very similar, and a lot of people rank it at one of the best games ever made. I kind of believe that this outcome is probable partially because of this, most scientists working on this really want to see and experience it.

As for intelligence of a child... It doesn't need to be a child. An aeroplane isn't a baby bird.

For instance, it is not because you are able to design and release a (biological) virus that you are also able to defend from it (design a vaccine and inoculate the world's population).

If we're lucky, it might be the case for many instances of problems, but there is no a priori guarantee that this holds.

If you are defending a system, you must defend against all 100 things. If one thing makes it through, you lose.

Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right constraints.

You cannot discuss existential risks of AI without being seen as a booster, and that is very unhealthy for the discourse around this tech. I hate how AI ‘doomer’ is now used to indicate pro-AI sentiment. The “moderate” person now is the one that just shrugs and scoffs at the deep societal changes this tech will bring, head deep in the sand.

I have yet to see this in my field. Maybe like a PhD student who bullshits their way through. LLMs still can't make correct decisions, only as useful as the person who uses them. To me, LLMs are only useful for making some mundane tasks faster.

Sometimes it gets wrong things that I had spelled out already.

It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake.

"You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"

I mean, unless you see clear reasons for them to stop getting better _right now_, this is not very comforting.

I would say exponential.

So far around one in 350,000 PhD math grads solve a millenium prize problem (Perelman).

> According to the Pygmalion effect, the targets of the expectations internalize their positive labels, and those with positive labels succeed accordingly; a similar process works in the opposite direction in the case of low expectations.

I added "you can do anything, believe in yourself" to sysprompt and agency increased. (Previously it was refusing to even attempt certain classes of task.) Maybe I should add "you are good", too :)

What will we offer to the ai gods who are wiser, more capable than us, and do not even need to consume our flesh? Why might the AI care to devote resources towards feeding, housing, and caring for ourselves when it could devote resources to its own development instead?

I am thinking it's more like The Matrix lore of The Second Renaissance from Animatrix.

The problem is that all the former can still happen even if "the AIs decide to have a tolerant relationship rather than one where they murder all of us." It's all disruption caused by the technology moving way too fast for humans & society to adjust.