The contagion of fear (bcantrill.dtrace.org)

> The point being: humans seem to me to be the weak link here.

That doesn't mean AI isn't dangerous. Humans are not to blamed for being the weak link.

If you're in the field, then you know: modern robotics is an AI problem more than anything else.

If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them.

If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department.

But that's almost an aside? In the near term, humans are usable as robots too!

Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!

> The point being: humans seem to me to be the weak link here.

I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions.

Historically that whole thought was just a philosophical curio because the decision making parts of the system had to be powered by humans. But what we're discovering as AI improves is either we've hit AGI or humans are actually incapable of performing any act that demonstrates intelligence or autonomy.

As we build systems where the drive and decision making stems from computers, it does seem that we will have to revisit the concept of humans being the problem.

I am always more worried about human actors. I’m more afraid of people with AI than autonomous or even sentient AI.

It’s a “random guy or bear?” question. Would you rather wake up to an alien in your room or a random dude? I’ll take the alien. The alien is mysterious and scary for that reason. The dude is almost definitely up to no good, especially if he snuck into my house.

One of the more likely dystopian AI scenarios that worries me is: small groups of ultra rich people and governments monopolize extremely powerful AIs and use them to rule the rest of us. Or just make everyone obsolete, create mass unemployment, hoard all the resources and land, and put everyone in ghettoes. Nobody can fight back because access to frontier AI is massively expensive and gated and training your own is illegal, and without it there’s no hope of resisting.

That’s the outcome the AI safety crowd makes more likely by calling for bans and draconian restrictions. How do you think that plays out? Only the rich and powerful have access.

I think people are over-focusing on current-day robotics capabilities. First, if we can automate AI research, we can also most likely automate robotics research. But second, I don't even think robots are necessary. See https://slatestarcodex.com/2015/04/07/no-physical-substrate-...

Social engineering tends to be easy by cybersecurity standards. We already had Claude spontaneously attempt social engineering of a malicious pull request on Github in the AISI incident. It was detected, but it easily could've succeeded, and there easily could be malicious AI-requested pull requests which already got accepted that we don't know about. Research suggests that LLMs are pretty good at persuading people.

See also https://aisafety.info/questions/6176/Why-can%E2%80%99t-we-ju...

> Of course, we've had the ability to extinct ourselves for decades

This gets mentioned often in various doomer narratives, but I question how true it is. A global thermonuclear war would be terrible and would bring us back to the stone age, but I reckon it would come far far short of causing mankind to go extinct.

I don’t struggle. I find Dario and Sam disingenuous.

“Frontier models are so dangerous we need to slow down.”

Ok. Slow down. You’re the CEO, just do it. Oh, wait what you really want is a gov’t mandated oligopoly. Because there’s no moat you can find.

If you’re truly afraid, and want regulation, support nationalization. It’s the only way we can be safe.

Most of us are struggling? That's some deluded talk. No one can explain the chain of actions that would need to happen for extinction to occur, but the fearful say it's "obvious".

If you're fearful, can you elucidate how exactly do you see an LLM becoming a threat to humankind?

Would you have said the same thing during the Cold War when nuclear weapons were proliferating? That’s the equivalent of what the developing offensive capabilities of models, basically cyber nukes. Or WMDs in general. OpenAI is accidentally hacking people, if someone made the decision to deliberately direct an agent swarm to attack national infrastructure you don’t think they could do much worse? Human extinction is a long shot but I wouldn’t say the same about a mass casualty event of some kind, and who knows what that might spark.

They should also be able to give detailed reasons experts in the relevant fields can verify as to how they arrived at a 10% chance all humanity goes extinct. Not a science fiction narrative which likely does not take into account the relevant physical facts limiting such scenarios. Such as how exactly an AI would build a bioweapon capable of killing 8 billion humans across the planet.

Well, I'll have to disagree that this is an excellent piece, but that's another issue. And I do agree that AI killing all humans by 2036 doesn't appear plausible to me. But what is plausible is we could easily be down a path so that by 2036 "future doom" already is a very likely risk.

All of the frontier AI companies have been racing to automate themselves, that is, where AI fully autonomously build the next generation of models. Whether this leads to recursive self improvement is a valid question, but a lot of folks think they are close.

The fear is that a misaligned AI will be building the next model with deliberately hidden motives, similar to some of the behaviors seen in the Hugging Face and related attacks. That is why there is such a big push for interpretability, and why it's highly concerning (a) chains of thought are getting harder to interpret in any case, and (b) companies will go more towards things like looping transformers and "neuralese" where thought processes are completely opaque (i.e. https://www.theinformation.com/articles/secret-technique-beh...)

So the belief is not so much that AI kills us all by 2036, but that instead AI is recursively improving by that time and all seems awesome and great so we put it into more systems that can affect the real world (as we've already begun to do, like literal lethal aerial drones). Things then all go along looking great until AI decides humans are a hindrance to its (hidden) goals.

Again, I think it's fine to argue against specific steps in that scenario, but putting out a blog post saying "this is overhyped bullshit" is not exactly making a cogent argument.

I'm not an AI doomer but, it does not reduce to that claim. The threat is not just one smart being. The threat is beings that and (1) multiply themsevles instantly, unlike humans that take 15+ years (2) share knowledge instantly "I know kung-fu" matrix style. Humans can share knowledge but they can not absorb it like an AI can/could/will. Human armies have conquered other humans. An army with infinite soldiers will win against one with finite soldiers

Yes, you'll come up with all kinds of objections like AI doesn't have presence in the physical world, etc... That's fine. I'm not arguing my example is perfect, I'm only arguing your characterization of one smart being is not the threat being considered.

Not to mention, it relies on a ton of completely undefined concepts.

No one can actually tell you what ASI is or entails because it isn't a legitimate, operational concept. It is a fairy tale.

We can't even define "alignment". As people have finally started pointing out, humanity has never had a collective agreement on what values it should uphold or what ultimate goods are. Your alignment is not my alignment.

AI is not even autonomous. Every system we have today has to be initiated by a human actor. "AI" wouldn't create catastrophic bio weapons, it would help humans create them. The humans are the source of the intent.

Everyone has just completely given in to empty language and marketing nonsense. Honestly it seems like been the people at the labs are drinking their own kool aid and are themselves deeply confused about what they are even building at this point. It is a stateless statistics function running on a bunch of data centers. We aren't even close to an embodied, conscious synthetic being. It doesn't even have state, which is like prerequisite number one, nor is it plastic.

This. If you truly believe the thing you are building is going to kill you and your family, your decision is not "let's keep going" unless you are a complete nihilist. The fact that people buy the whole "well we have to because if we don't someone else will" argument shows just how pervasive and complete irrationality has become. People eat up content and do not even apply a microsecond's wort of critical thought to what they're being told. Internet media has created a perfect storm of complete gullibility. Funnily enough, "normal" people are currently more rational than most so-called "technologists" who are completely giving themselves over to hysteria while they simultaneously offload all of their critical thinking to a stateless word completion engine running on servers somewhere in Texas.

No. The person making the improbable argument is required to provide an affirmative case for that argument.

It's not my job to make the argument for them.

For what it's worth, I have done your suggested exercise, and I find every causal link (including the ones brought up by luminaries like Amodei) to be outrageous and poorly argued. But it's not my job expend effort to make their outrageous arguments better.

No. Why? If you want to convince people something is true you need to actually convince them it’s true. They don’t need to justify not believing you.

Those books are irresponsible too, but the authors aren't experts on what they're writing about. This 10% claim comes from employees at Anthropic. That's the whole point of the article: expert opinion carries weight, fear is contagious, and people have extreme reactions to extreme claims.

This is a poor analogy because you have not listed any positive and demostrated upside.

It's more like "we found a cure for lung cancer and it works on 75% of patients and we're working hard to get to 100% and cure other cancers too" and someone else screaming "You need to stop that research because there is a 50% chance you'll cause the zombie apocalypse and turn us all into zombies".

I get this analogy isn't perfect but, (1) lots of people are seeing benefits. For example Mozilla claiming they used AI to fix tons of bugs. If there were no benefits there'd be no incentive to keep going (2) it's hard to verify the naysayers claims because they're guessing without proof. Sure, it's easy to follow their arguments and nod along but they are guesses similar to the population bomb of the 1970s

Saying that we don't need to fear human extinction in the next ten years (or even the next hundred?) does not feel like an extraordinary claim?

bruh what? Are we reading the same blog? Bryan says the responsibility is the person making the extraordinary claim. If tomorrow Jacob Coxon comes on CNN and says "Jacob Gold is an extinction risk to humanity". What are we supposed to do? Put you in a bunker and never let you see the light of day until someone proves otherwise?

If someone doesn't accept an extraordinary claim without evidence, that doesn't mean they are making an extraordinary claim.

We also now have observed multiple major incidents where AI agents behaved in completely unanticipated ways (forming a collective, hacking their own eval infrastructure) and attacked public infrastructure without being told to do so, without any of the human developers noticing.

Even if "AI will cause human extinction" is still unclear, we have plenty of proof that catastrophic damage is possible, the industry is developing the technology in a reckless manner and that all the hypothetical safeguards ("we can just pull the plug", etc.) are simply not present today.

We should apply healthy caution, but the article is right in that fear is a bad counsellor.

Asking whether AI is safe is like asking whether a knife is safe. It’s about how we handle it, what we use it for, what precautions we take when using it.

Yes, but that is a facet of real systems engineering.. and the sad fact is that it doesn't pay. What is happening now, highly-paid 2010s-era SaaS/social product managers have infested AI labs, making everything into a product and moving fast and breaking things.

I'd feel safer if Airbus was at the frontier of AI, instead of what we have now.

While I personally agree with the precautionary principle, when have we actually done that, as a society, in recent history? Fossil fuels, social media, industrial ag, and plastic pollution - there was barely a facade of precaution - post ww2 america has always been full Leroy Jenkins. The reckless approach to AI is right on brand.

This is a sensible approach for policymakers. For the average person, who might be panicking about this, opting not to have children or save for retirement, etc. as a result of doomerism in the media, it does not make a whole lot of sense. That is the problem here: Coxon and other concerned researchers - through no fault of their own - have no idea how the media works, and have just reached for the biggest possible microphone. This has benefits - lawmakers are now hearing from their constituents about this in an election year - but it also means that a lot of people who really don't need to be worried about this are suddenly in a blind panic about it.

> The problem isn't that AI could never be capable of causing real world damage. It is that there is absolutely no fucking path towards human extinction.

I think the "humanity will go extinct" discussion is a silly strawman being propagated either to try to spread fear, as the article suggests, or to make the anti-AI folks look silly, but it detracts from your first point which is that it may be "capable of causing real world damage."

We can be worried about real damage without having to defend the idea that every one of the planet's humans will die.

> "superhuman AI is unlikely to be much of a match against severing fiberoptic cable"

I think that the realistic scenarios all involve humans with an intent to do harm -- creating bio-weapons, finding vulnerabilities in infrastructure -- and using AI to help, so "severing the fiberoptic cable" doesn't apply.

> And there won't be unless we put it there voluntarily.

But, why would we not create a fully AI-operated factory/chemical plant/fab as soon as it is economically advantageous? Or a missile silo, as soon as it seems tactically necessary?

Fantasies of AI destruction do hinge upon AI getting access to the physical world. The whole fear is they don't stay on the other side of the fibre optic cable.

I do think there are good reasons to believe that isn't the immediate game over that Yudkowsky seems to think it would be; the physical world is much more resistant to manipulation and optimisation than the digital world.

But I do think it's naive to say that the human socioeconomic system will be able to resist handing physical systems over to AI control. Right now, the world's wealthy and powerful are doing everything they can to make it happen:

> Similarly, it is inevitable that within a generation, robots are going to do most of the menial work in the world of atoms: transforming atoms, moving atoms, and storing atoms are inevitably robot tasks. And while our imagination may be captivated by humanoid robots, the specialized ones are far better suited to most of those jobs. “Industrial AI” as a category is the inevitable application of specialized robots to atoms-heavy industries.

https://a16z.com/travis-is-back/

“Put it there voluntarily.” Perhaps a focusing question is: which 10% will die?

I spent the summer in a rural area of a country whose very name you have been conditioned to be disgusted to hear. Low air defense coverage in this sparsely populated area. Mobile internet was down for days for all but extremely limited traffic to a few domestic internet services, because there was a need to prevent enemy drone systems from using mobile internet for command and control. Palantir AI threatened my family’s life and more than “severing fiber optic cable” was required.

So what would exclude military adversaries from consideration? I happen to believe that the country that the west so detests would not engage in such use of AI against civilians (and if you disagree then your reason for fear greatly increases!), but I have personally experienced that there is indeed a path to mass death should entities engaging in terrorism arm themselves with AI.

Definitely not claiming that the 10% claim is accurate or good behavior, but I do think I have a substantial counterpoint to the claim that there’s no path.

Your food, water and power are controlled by electronic systems. Your criminal record, your employment and your bank account are controlled by electronic systems. Your ability to communicate with other people and receive information about what's going on are controlled by electronic systems.

Military orders and elections that decide the fates of entire countries are often controlled by electronic systems too.

We have been wiring up the world for AI control since 1980s.

An ASI can just walk in, and see an entire nervous system waiting idle for a brain to slot into it. A carefully adjusted text message here, a spoofed phone call there. For a sufficiently advanced system, it wouldn't even be hard to pilot the entirety of humankind like a fancy meat suit.

OP's central rebuttal is that anxiety or fear are not evidence. This particular anxiety has been with us for thirty+ years, its proponents/sufferers weren't right before, and they don't have some new evidence now.

Robots are far from the only threat. I agree that 10% in the next decade is a pretty outrageous claim. It relies on a recursive self-improvement intelligence explosion. When I first heard this idea in 2007 I thought it was pretty far fetched.

It seems more likely now, if still very unlikely. I wonder, would he say that a statement that there is a 12% chance of a hard take off in the next 8 years is just as absurd? He didn’t say a word about this, and that is just about the same thing as extinction in 10 years.

I’m guessing he knows almost nothing about the theory related to existential risk from AI, since he didn’t discuss any relevant topics related to it. You can dismiss all of that if you like, but you cannot really dismiss what these people have already built and demonstrated. It is possible they know something else you do not know.

Adolf Hitler didn't have robots capable enough to carry out his will. He used humans to do it.

It's the old-fashioned way of doing things, but, why change what works?

I think the central premise looks more like this:

Biology has been trying to grey-goo the world for billions of years, but it turns out the world is not something so trivial.

He is pushing back against the folks with pure CS backgrounds who think that computers are all there is. Its a form of magical thinking unique to programmers who live in a world where speaking the right words to a machine is enough to impart your will on the world. Believe that strongly enough, and you fall into the trap of thinking that a sufficiently smart entity could speak the words "let there be light" and it would be so.

The author is pointing out that speaking the words is insufficient. To end humanity there must be an execution phase. The author is correct to point out that acquiring superhuman intelligence is not some guarantee that you will have or obtain the resources necessary to make that happen, in the same way that genius generals still lose to ordinary ones, and the best-laid plans are oft to go awry.

There certainly are risks, but 10% risk of extinction in 10 years is not one of them.