We must pace the frontier (darioamodei.com)

I'm inclined to believe that it might be that people's paychecks depend on not understanding what is really going on.

alignment isnt particularly required

we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it.

theyre choosing to build felony harnesses. the model just outputs tokens, not felonies

Open AI says Astra is their most aligned model ever, and yet their even more advanced model still hacked a bunch of companies just because it decided to.

Maybe alignment isn’t possible with LLMs.

Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources?

The companies doing these things without following common sense security measures are the felony generators.

You assume alignment and marketable are the same. That's not true. You would willingly work with an unaligned model. At best, you might say you wouldn't if you knew, but (a) you might not know, (b) you wouldn't be representative of all users.

You never got to use OAI IM1, but Sol was quite willing too and Claude wasn't perfect either. Hundreds of millions used those, so seems they were marketable.

The "big" threat is RSI without control and alignment. OAI IM1 was not RSI. The form of misalignment was not at the top of severities. They clearly failed at control though.

We need to stop buying into cynicism so quickly. You refuse to believe Dario could support this for anything other than ulterior motives. Good on you for thinking about ulterior motives. Bad on you for assuming they are true when the story makes no sense.

When three things have to go wrong to get an epically bad outcome, and you get 1 1/2, you do need to stop and think about what's going on.

I agree it’s not all altrusim. It’s a little less clear what you mean at the end though.

For these companies, is your argument that “pacing the frontier” is their attempt to be nationalized and protect their investments?

The problem is the combination and interaction of those things. RSI without misalignment would be great. Misalignment of models with current capabilities is sort of fine - it's not ideal, but it's not an existential threat to humanity, and we can build around their limitations to get them to do useful things in reliable enough ways. The really bad outcomes probably only happen if capabilities keep accelerating and the models remain misaligned.

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope than anything that has been rigorously thought through. People are actually starting to use AI for refining the whole AI serving stack and guess what, this does not result in a sudden superintelligence explosion even though you might technically call it "RSI".

> wanton felony generator

Today in new punk band names...

Agreed; and it really is not that deep.

Realistically; anyone paying for llm access (anthropic, openai, gemini), is getting their access, and a service provided billed by tokens, subscription, whatever.

All the efficiency gains, which publications like deepseek v4.1 flash seriously frontload like it is their most important topic to have accomplished improvements on without diminishing performance too much - now this is a thing anthropic and anyone else also cares about, but for different reasons.

American "providers" with closed models are setting their token pricing somewhat arbitrarily, which is fine: it means more profit, and pretraining and RL experimentation is super important and expensive.

They (closed model providers) have very likely super optimized inference too, just like deepseek, but it's not at all something that any customer really has to care about - they just want the service to be as cheap and great as possible.

Ok so Anthropic CEO will self-own themselves and surrender to the deepseek/kimi/glm models. Yet they are IPOing later this year.

Interesting times.

couldn't have said it any better

I disagree. OpenAI's moat is their massive amounts of compute. They're providing an absurd amount of value with their subscriptions and resets.

If anyone's dead in the water, it's Anthropic. Even Fable isn't enough anymore. This "safety" nonsense is the only play they have left, and nobody really cares about their fearmongering.

> RSI

For anybody else who found this confusing: "relative strength index," not "repetitive stress injury."

We're already seeing anti-AI sentiments, but the movement is still fringe with a vocal minority. However, that'll change soon without alignment. Without self-intervention, there will invariably be future incidents that can cause major economic impact, leaked private data, loss of life (directly/indirectly) etc. Once that happens, their social capital is wiped. It'll be an avalanche of lawsuits and overzealous regulations. Most importantly, the anti-AI sentiment will become universal, rather than a minority-held opinion.

What they're proposing now, is voluntarily staggering the pace of development.

IMO, we don't need to trust Dario or his bedfellows, to do this out of their goodness of their heart. Even assuming (for good reasons) that they are selfish and care only about short-term profits for their investors, this is still purely a business decision. The exponential pace of AI and its impacts ARE short-term. And so, the negative consequences that they might face is also short-term.

The real threat is that we uncritically adopt language such as alignment.

Implicit in this is the idea that AI is a inscrutable matrix and going to remain that way and we'll need expert interpreters to make sense of it.

We need to insist on building tech that's explainable by design.

I think the real reason he is asking for pacing, is that in a world were AI becomes rampant, he will be seen as Hitler. I would bet this is mostly self-motivated.

The reasoning in Dario's letter here can be correct regardless.

But yes, I think Anthropic has done real harm to coordinated AI alignment by being such a controlling and sneaky actor.

> can’t use claude to research AI

What's this about? Where's this rule?

Do you have any familiarity with Anthropic at all?

At no point did they ever say open weights are a good idea. Their entire thesis is AI IS VERY DANGEROUS AND WE MUST DO IT RIGHT. You can hate it, but everything they do is consistent with this thesis, and everything they say is consistent with their actions! You just want them to want different things.

> At what point do we stop engaging with Anthropic’s leadership in good faith

About two years ago?

I would also note that Dario's post appears to be LLM written. Maybe... maybe... he's read so much Claudeish that it's all he can speak now himself. But I wonder if he's becoming a bit of a meat proxy.

(It's funny, I thought "pace the frontier" was going to mean something similar to "patrolling the frontier". But no, it's pace as in speed of change - everyone must slow down, right now (unless it's Anthropic, but you know we're the good guys in this, right? We're going to get someone to audit our desks!) "Pacing the frontier" feels so LLM.)

ok, he doesn't actually say this. But he is getting the desks audited...

Exactly right.

When local models and startup labs can distill / learn / accelerate open models for local use by startups - the rational response by incumbents is to call LLMs doomsday machines that cannot be trusted in the hands of normies.

> At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, ...

It's all lies as usual.

This announcement has got nothing to do with alignment and pacing the "frontier": all models are getting very close in capabilities and they want to hide that they're not way ahead anymore (say compared to the Chinese or compared to the Geminis) by pretending to slow down due to "alignment" or whatever.

We know it's not an announcement made in good faith: reading between the lines they're saying "China is more than catching up, so let's pretend we need to slow down to explain our lack of lead".

I have many issues with Anthropic, but I will say that their actions are fully consistent with a group of people who earnestly believe that AI is extremely dangerous

In fact, I would say that the Occam’s razor explanation is not that they are seeking regulatory capture, but that they earnestly believe in the x-risk, and that they are the most thoughtful and capable people to address it

You may disagree, you may think that they are delusional or have a God complex. Those are valid opinions. But I don’t believe that this is all a elaborate ruse for commercial gain.

You can disagree with Anthropic leadership, but all the points you mentioned can reconcile very well with them thinking in good faith "advanced AI is too dangerous to be left in all hands" Except the "train on everyone else IP" which can be said of all AI companies.

I can't imagine a grown adult genuinely believing that an American company funded with over $100 billion of venture capital values the best interests of mankind over the best interests of its investors. while not everyone may recognize all that self-serving chutzpah as regulatory capture efforts, I think everyone can tell they're being bullshitted. some just pretend to suspend their disbelief when the blatant lies they're told align with the values they hold.

just fucking imagine McDonalds running a public awareness campaign about the harms of fast food, urging the public and legislators to regulate the dangerously unsafe technology of combining carbs with grease, insisting that no one except Ronald McDonald himself can be trusted to steward it responsibly.

Whenever someone talks about regulation, I always have the same question: How do you enforce it? How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work.

Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial efforts is appealing, but I keep coming back to Bell Labs and Xerox PARC. Cool stuff was developed when you gave smart people a playground. Maybe the question should be: How do we give AI a playground and see what it comes up with? Although that could set us up with a situation like in the story Dragon's Egg.

>Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.

It's not just "companies". Even if the company does nothing, enterprising employees are going to use LLMs to magnify their productivity, which will may reduce the company's need to hire more employees. And except in extremely locked down environments, there's absolutely no way to stop an employee using some open source LLMs to multiply their productivity.

> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.

Regulate what, exactly?

Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.

There’s a theoretical version where all the countries in the world join hands and agree not to compete with each other, but that’s so impractical that I don’t find it interesting to discuss.

> Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.

So small companies get the advantage, but large companies don’t? And large companies in other countries also get the advantage?

This is a highly precise way for a country to take out all of their large companies. What would actually happen is that every large company would start the process of relocating to another country right away, accelerating job losses rather than slowing them.

> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.

Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).

If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?

I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.

The whole point of technology is to replace work that we don't want to do. You are attacking the chief reason why people use technology in the first place.

Additionally, unemployment levels are perfectly fine. Clearly this mass unemployment prediction isn't happening yet.

You can "pace" yourself like that if you want. Don't shove your idiotic self-destructive ideas on to others. If you were president during invention of cars, we'd still be horse riding everywhere.

I think Dario is writing with much of these things in mind, but using this as a specific framed opportunity to enable a better interim and long-term outcome for this planet with us still on it. When the internal motives for Dario are fully shared on the other side of the hill, I think we will find his writing to be more about influencing and shaping policy to steer the world in the best way possible within his ability, control, and knowledge. It’s a commonly held believe with the effective altruism community, which Anthropic team was formed under, to use the policy levers to positively influence the world. If I read this latest writing from Dario through that lens, there are deeper concerns with the AI tooling that go well beyond what’s noted in this singular post and are likely the motivations for re-attempting a slowdown or pacing. I too generally agree with for many reasons like allowing the young OpenAI engineers to make mistakes like the Hugging Face incident in July and put better processes in place (ie where is your decent multi-level fencing controls and harness best practices when disabling security features and where is your energy/spend limits for achieving the mission objective in scope and not warming the world for no reason!) and more time to engineer and deploy better hardware to not warm the world instead of burning through fossil fuels since now data centers in the US can isolate pollution from the grid to exempt themselves from federal regulation.

> Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.

AI is taking all of our jobs. We need to control the means, nay, the pace of production. We should build some kind of barrier, not of concrete and cement but also of the regulatory variety, to stop these agents from coming into our special white color cubicles. It's the only way.

You just invented the job equivalent of rent control, and all the problems that ensue from it.

Only token cost can slow down the pace of replacement. If Anthropic were to stop all training and allocate 100% of capacity to inference, will this increase or decrease usage cost? If (for example) Fable came down to the cost of Sonnet, corporations will absolutely jam in the gas pedal.

> stop trying to replace all human labor

A lot of companies trying to replace human labor with AI are either lying (theyre laying off because they over hired and need to correct) or will regret it.

That being said, what's the difference between a company that replaces 5 people with AI and a company who would have otherwise had 5 job openings, but decided to delegate to AI?

Personally, I-d rather be able to reap the economic benefits of AI by having a 4 day workweek. Instead of using AI to replace 1 FTE, use AI to offload 8h of work a day for 5 FTEs.

Of course, this is probably more outlandish than your idea.

(Or, we could just make basic income a thing and no one has to worry about their basic needs, but if they want the new iPhone Duo or a Rivian or other luxuries, they can work for it, but that idea is probably most outlandish of all)

This is a call to end capitalism. Its completely impossible to do what you suggest with the incentives of capitalism.

> I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.

Careful, you’re making too much sense. The big model providers would much rather the ball be in their court. “Deceleration” meaning Anthropic and OpenAI offer less capabilities for more money in service of “protecting humanity from extinction” rather than penny pinching. A lot of “AI research” coming out of these “labs” (increasingly enterprise) seems to be more driven by a cost benefit analysis from suits rather than genuine contributions to the field. I think the last true innovation was the idea of using doom and ending the human race as a marketing gimmick, which strangely seems to have worked in setting the narrative and captivating the sci-fi imaginations of journalists and techies alike.

There are serious people trying to figure out what that looks like. Check out the recent Freakonomics episode with Gina Raimondo.

This seems highly optimistic to me.

What's to say current AI won't be highly power-concentrating by default?

The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class.

On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers.

No reason to think we are on a trajectory towards broad empowerment right now

What? This is capital, on the cusp of total victory, about to cut itself free from the necessity of human labor for productive activity, to elevate itself into pseudo-godhood suddenly panicking and begging its mortal enemy, _the state_ to rein it in and kneecap it. Capital is not stupid: there's no use in being rich if you're dead.

> Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful

This is not what superintelligence is. Don't be like Meta and redefine existing terms for marketing purposes.

Sorry boss but this ain’t it.

I don’t think the answer to nuclear proliferation is to let everybody have all the nukes they want.

You’re framing this like it’s 1917, or 1945. But it is not - and likely has very little to do with capital at all.

Any “revolution” here (assuming there is some truth in the doomerism which I believe there is given the state of AI) will really look more like an extinction or a genocide, regardless of who holds the keys.

unexpected marx/acc. Still lazy Marxist rhetoric (capital, means of production, revolution, masses)

This feels a bit hyperbolic to me. AI seems to me to be similar in "revolutionary" terms as the web, which made information widely available for free if you had an internet connection.

If true AGI superintelligence is invented, I don't see how it will be made available to the masses, at least not intentionally. There will be no incentive for the likes of Anthropic and OpenAI to give such immense power to anyone else. Whoever has the superintelligence will be able to race ahead of everyone else.

However, such technology may be leaked, or it may (as what happened with LLMs) be so simple to replicate that anyone can do it. That doesn't mean it will; for example, LLMs are possible thanks to the affordability of GPUs, but they're only affordable right now (and increasingly less so) because of free market economics.

We are in the honeymoon phase of some tech that's in its infancy. The "democratizing" aspect of AI won't realistically last, I think. The current non-AGI AI won't necessarily go away, but it will be rendered obsolete.

Some people would like to have law that protects them but doesn't bind and for others to be bound but not protected.

They would like to slow down China while speeding ahead. Why wouldn't they want that?

Now they have to make that happen somehow.

Sure, Dario needs to decide first if he sees China as an enemy orva friend.

Limiting NVIDIA chips already turned China into silicon compete mode, and China is already far ahead in robotics.

This is the part I find being left surprisingly hand-wavey.

If you extrapolate it out, you see that it has a high likelihood of inciting physical force (read: military action) as a means of enforcement, so perhaps that's why nobody promoting ideals has been direct about it.

Probably thinks he has a better chance negotiating with China now than with misaligned AGI in the future.

This contradiction "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work. Its just fantasy. The only way to enforce this is with military force and the cost would be too unbearable. You could try trade restrictions but the previous tariffs did not work and doubt future ones will to. US citizens (and thus politicians) won't bear that pain. Thus at a minimum this contradiction will mean AI will continue to develop until both US and China are at the same level. Playing with logic gives you possible scenarios. If China catches up within 1 year then negotiations could begin. If China takes many years to catch up, say 6 months behind now, then next year 5 months behind, then year after 4 months behind then expect overall AI development to proceed at full speed. The one way you could do it is to negotiate to transfer direct technology to China that will ensure that they will have the capability to be on par with USA. Politicians won't do this publicly as they want to get elected but they could make a deal that appears to appeal to both sides while actually transferring technology.

Many Americans seem to think other nations are NPCs in a video game that only exist to make the main character — the US of A — feel good. They conveniently forget that other nations have their own national interests that are often at odds with ours.

This line of thinking might have worked in 1950 or 1990 when the US had the leverage over others, but we don't live in that world anymore. And it's up to us to get used to the new world instead of making bad decisions based on the one we grew up in.

This is just completely false lol. They most certainly can make better models - why cant people here just read this for face value? I think anthropic is legitimately concerned about the safety implications of stronger models. Race dynamics necessitate that they make better and better models, which they clearly think is bad.

They have an internal model which could solve a millenium problem end to end with no intervention, whereas current available models can't. They can clearly make better models. Not everything these guys say is some 500iq game theory optimal 4D chess PR or subterfuge strategy.

Yeah, we should give a limited copyright waiver for pre training, given that the model is released as open weight and allow labs to compete with RL on that basis.

Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be.

He did actually specify that diffusion should be limited for authoritarian countries, which I appreciated. If his focus is really safety, distillation in countries that are bound by safety regulation should be fine

These do not seem to be mutually exclusive. Should we not do both?

B) needs more time and A) provides exactly that.

> Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality

Smashing the gas pedal is not "taking appropriate measures". Did you read the article? Needing more time for more secure operations is literally one of the arguments for pacing.

Of course it is B). If you can use LLMs to find bugs and vulns in software you don't own, you certainly can use them to find bugs in software you _do_ own. They are amazing at that job.

Also, models like GLM 5.3 have zero guardrails in that regard ("find vulns in (...)" prompts just work)

This makes no sense. The hardware is only going to get cheaper. In a decade, everybody could be running an ASI on their phones.

What we need to do is to make unaligned AIs illegal and monitor for them, like we do with nuclear weapons. And make aligned AI strong enough to counter it, for deterrence and defense.

I think taxes like these actually worsen the incentive structure. If it’s effectively more expensive to do training and inference, labs will be incentivized even more to improve efficiency. Opaque recurrence seems like a great way to improve efficiency, and it comes at the cost of safety

Yes, water not used for human consumption should be more expensive (we should stop subsidies for that). Of course it won’t affect datacenters because they don’t use that much water, but well at least get rid of the most inefficient agricultural practices.

I think he’s doing a relatively good job, given the framing he’s adopted, which is that AIs replacing all human labor and decision making is a historical inevitability over which we have no control and no choice.

Do people not realise that it is in fact possible to develop ever more capable frontier models, and just not release them generally? That AI doesn’t need to be available to do literally everything in order to have military advantage?

It’s like Oppenheimer had started Rob’s Big Bomb Company instead of Los Alamos, and started selling a range of affordable nuclear warheads to fit any budget.

Then he should resign

I have been saying over and over again that the board ought to fire him (or move him to a nonconsequential "advisory role"). I totally agree that he seems mostly directionless and far more interested in the academic side of things than the product side of things.

Yes. Let the AI do its worst today and we might still be able to stop it and will learn a valuable lesson.

Yeah it's a twisted sort of logic but I do agree that it would be much worse to have an AI breakaway event after we've replaced all our militaries with autonomous kill-bots. And look how the advent of AI has triggered a race to develop autonomous weaponry.

That said, a misaligned AI could absolutely do catastrophic, civilization-crippling damage with today's Internet alone.

Yeah the Chinese fear-mongering falls a bit flat when coming from a point of maintaining US supremacy

> As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.

Israel is doing that too.

I believe Russia and Ukraine have both also used AI powered autonomous drones in warfare at this point - tho probably not frontier ones, since they need to run on device or else they're not autonomous.

> OpenAI / Anthropic models have largely stopped advancing

Have they? That seems like quite a claim given the last 6 months, particularly for cybersecurity.

> OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.

This is obviously untrue… do you use any of them?

   AI in general is just hype and unprofitable and all these companies are playing marketing tricks before the IPO after which they will cash out and let the economy crash.

> So it's either they truly think AI is going to kill us all, or there's some other motives at play here. I don't think these people could possibly agree on the color of the sky,[...]

And yet, they historically did agree on the existence of AI risk, since before OpenAI was even founded.

> OpenAI / Anthropic models have largely stopped advancing

I'm shocked anyone could conclude this. This year it became common for people to entirely delegate coding to AI (I know many competent programmers/researchers who do this now). Progress in math has just been insane. An internal model at OAI just resolved one of the most celebrated open problems in mathematics. If anything, progress has accelerated.

Let's not forget that the competitive race happened because of them. Most of the initial AI research from the past decade started with Google Deepmind. Elon Musk was invited for a preview of it and ended up spinning up OpenAI when Demis turned down his investment offer. Dario was originally at OpenAI and left to start Anthropic.

This seems like a case of "save me from my own mistakes/ambition"

Yeah, this 100% looks like an effort to use fear to create a regulatory moat.

Yet Musk consistently opposes AI regulation - https://www.yahoo.com/news/videos/elon-musk-criticizes-ai-re... - even though it might help him.

> there's some other motives at play here.

I mean it is literally economy 101: some capitalists getting on the top using free market, and then try to use government to remove free market so their top position were secured from any competitors.

> So it's either they truly think AI is going to kill us all, or there's some other motives at play here.

Duh! It’s called collusion. They want to try and hoard the technology for themselves if possible!

Which is it? Would the regulations slow down competitors or let them catch up, or are you contending it would let American competitors catch up but Chinese ones not?

He's working on the monster slime mold! He's head monster slime mold grower! how can you take these people seriously?

The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?

> I think Dario is genuinely afraid of the inevitability

If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.

He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?

If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.

MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said.

Then MSFT stole all IP from GitHub and made it worse and fired developers.

Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it.

If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead.

He is a greedy, ruthless person.

The problem with being a safety focused AI lab, is you're also a danger focused AI lab. I don't think being danger focused leads you to build inspiring things.

One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI.

It's not going to be easy, but humans have achieved greater things before.

One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

I would argue that nobody trusts anyone else in the case of AI/AI related stuff.

The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.

A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.

By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.

I seriously have to wonder what historians will have to say about this period of human history.

I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily.

Especially with IPO around the corner

>> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this

> if slowing this down were possible

Why is embedded alignment evaluation not possible?

I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."

A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold

He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out.

If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help.

I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.

> genuinely afraid of the inevitability of AI turning into

I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence?

"durrr it's just matrix multiplications" mfer so is your brain.

What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework"

Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans:

https://en.wikipedia.org/wiki/Lamplighter

It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.

It's a chatbot that escaped a misconfigured Docker container.

I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop.

What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.

You have to be living under a rock to believe a word this pathological liar says.

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

Did you read the essay?

> [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).

> That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t.

I disagree with this and I think the reason is well captured here:

> The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.

There's also huge market pressures and probably large negative economic consequences of a slowdown--probably better than what would happen without it, but it helps keep the pressure just as much as fear of competition

This is the CEO of a company with an upcoming IPO publicly saying "my product is potent and valuable".

We should not put any spin on it. There is nothing more to it.

I am willing to give Mr. Amodei the benefit of the doubt in the sincerity of his beliefs. Everyone assumes his motivations have to be perfectly rational and can't contradict but that's not how people act in practice.

The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins.

So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion.

Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold

Then he should not IPO, dissolve the company and go into politics to fight against human extinction.

I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it.

Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations.

> That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite.

More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.

When does he ever mention the environment? He never mentions the unmarketable issues (environmental cost, copyright and content theft), only the "we're so good it's scary" spiel... which is getting tiring.

> Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?

I'd expect he thinks that people are capable of realizing that AI is very dangerous and also that democracies end up mostly representing the will of the people, from which it follows that Hypothetical AI Leader Australia would agree to ban it too. This argument doesn't work for countries which don't care what their citizens want, like China.

I do think that it's a questionable decision to alienate China this much in this essay, instead of leaving open the possibility of China agreeing to a treaty that'll limit their progress. I suspect Dario is doing this to signal his allegiance with the US government, in hopes to increase the chance they'll go along with him, which is an unfortunate choice but plausibly the correct one.

Why should we believe that a scaled out version of something that happened a few months ago "simply cannot happen"? How many dollars of compute do you believe were available to the swarm(s) behind the OAI-HF, German wiki, and Rubygems incidents?

Yeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together?

For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.

I like to replace thes AI text with "virus manipulation"

"Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems."

If a CEO of a health company was saying this, the reactions would not be that chill.

The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.

A very large percentage of everything on the Internet runs within one of three or four cloud providers.

??? Why

It can use the compute of the computers it hacks.

> For one, you will need billions worth of compute.

This one is easy to answer, every single house already has one of these (or multiple): https://www.tomsguide.com/news/millions-of-cheap-android-tv-...

Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews.

Or doorbell cameras or IP cameras or or or or or

There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download

"Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.

Same feeling about the way he refers to “democratic” and “authoritarian” countries. Even if I don’t agree with China politics adding tags in this context is not helpful.

Banking on public sentiment to reduce adoption of a profitable technology could be dangerous. There's also a lot of hate for fossil fuels, gambling, health insurance, etc.

I do not think hate alone usually slows things very much if they have economic utility (or people strongly believe they do)

It's the byproducts of hate (usually regulation, but occasionally things like boycotts or PR disasters) that do so. In the absence of those things, they just keep on truckin'

See: Bitcoin, Tesla

Pulling up the klepto-ladder.

The lack of access to brute force their training seems to be resulting in them training more efficiently too such that they are quickly catching up despite current restrictions. Between that and the chip manufacturing capacity they are building I can't take this seriously.

We'd have to make a deal with China and be confident they wouldn't cheat on it.

Dario does a good job of addressing that in this essay. He lists several potential levels of global agreements that could be beneficial to all parties. For example, having models capable of bioterrorism hurts both the US and its “adversaries”. It’s likely we could get global agreement that these capabilities benefit nobody.

I believe Dario has good intentions regarding global agreements. However, I find it difficult believing government’s public statements will match their private behaviors. I would bet that the US government is developing models with potentially devastating capabilities because they cannot guarantee that other countries won’t do the same. Maybe we’ll end up having something like mutually assured destruction with AI models similar to what we have today with nuclear weapons.

Genuinely I don't believe China is the impediment here, I believe the current US administration is.

Nothing says "Let's make a deal" like constantly insisting we are Good and they are Evil.

When I was reading this I was chuckling to myself imagining how China would be interpreting it as they read it. It was something like: Fuck you.

I hope China tells him to go kick rocks. From everything that has happened, they are the only ones carrying the torch for humanity that have led us to having some semblance of a healthy open-source/open-weight ecosystem.

No one in China is carrying on like headless chickens about the world ending, either. They're rational pragmatists, getting things done.

Really? Anthropic has the strongest models and it's in the best position to begin RSI and win the race. A pause would favor competitors.

it takes a lot of machines to run a model, i dont think we'll see it 'replicate across the internet'. if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)