Everyone should slow down AI development except for me (xeiaso.net)

167 pointsby xena2 hours ago65 comments

AaronAPU an hour ago

They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public.

Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.

The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.

ranger_danger an hour ago

I figured they're just admitting AI models have plateaued and are coming up with some fake story about self restraint so they don't lose VC money

lukan an hour ago

Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.

unleashhale 39 minutes ago

Anything in particular? My experience has been like seeing the addition of retractable cupholders, but maybe different domains.

qskousen 20 minutes ago

I have a pet project I have been working away on for some time that involves building GPU backends for various cards in Zig, lots of complex stuff in it. Lately I mostly use Opus 5, it can pretty reliably plug away at things but it does mess stuff up occasionally. For this codebase, Fable 5.1 was noticeably better at getting things right and doing things in a good reliable way. Of course, I can only use Fable for a bit before I hit the usage cap for the week, so I save it for the tougher things. That said, I absolutely abhor the way recent Anthropic models write prose, especially comments.

I recently tried doing a fairly normal task for this codebase with codex, as I have seen a lot of people talking it up on here. A single task running for ~1-2 hours burned through over half of my usage for the week on the $125/month plan, not on a top model (I don't remember which one specifically I used). It struggled to get the basics done, then got absolutely stuck on a follow up. Handed it over to Claude and it 1-shot it.

drTobiasFunke 30 minutes ago

Really? My employer rolled back to opus 4.8 because 5 was expensive AND crap. Didnt even consider fable because it didn’t add any additional value.

For most software eng and design work opus 4.6-4.8 just works fine. For everyday joe asking ai to plan a trip or home diy work even sonnet works fine.

Any cybersecurity or other areas are niches that cannot support trillion $ valuations. What am I missing? Genuinely curious

bpodgursky 23 minutes ago

If Fable doesn't add additional value in your workplace, it means you aren't being ambitious enough in how you integrate agents into your workstream.

Yes, it's probably comparable to 4.8 if you are just using it to write code and put up a couple pull requests. That's not where things are now.

lukan 20 minutes ago

No idea what you are missing and yes, Opus is quite solid, but Fable is clearly way better for me.

I just did a direct comparison, big change in a quite complex codebase. Same prompt for Opus, same for Fable. Fable clearly won and delivered very good results, while Opus delivered mediocre, so I did not let it finish. I expected both to fail and was prepared to do lots of manual steering, but not necessary with Fable one shotting it, and all this with 35$ of credits for fable. I am still impressed. If I would have had to hire a human, it would have cost me thousands of dollar for the same task - and a way longer time. So maybe the valuations are overblown, but they clearly provide value.

0xcde4c3db 20 minutes ago

I don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.

lstodd 33 minutes ago

idk why you think "nation states" are any better at corporate governance than poster examples of bad like f/ex Boeing. Or Facebook. Or Microsoft. Or Enron for that matter.

I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.

nullbio 6 minutes ago

* Frontier models need infinite high quality private IP to keep the beast fed. Forcing an IP theft funnel ensures big lab survival and intelligence growth.

* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.

* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.

* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.

* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.

* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.

This, and many more reasons.

The healthiest outcome we can hope for is if the labs feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again. In general, accountability of what your AI models do is the solution, it creates the right incentives. That's all we need here - balanced incentives. Everything else should be left alone so the free market can naturally evolve to the best outcome where we're not enslaved by tech giants yet again.

Right now we're on the precipice of something great, but if they succeed at rigging the game yet again, there is no second chance. It's game over. The stakes have never been higher.

coderintherye an hour ago

I have to wonder if some us are just much more inured to salespeople and thus also to "AI Safety" propaganda. I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't that what most them actually want is to be the one holding the keys to power.

jordanb an hour ago

The problem is the "AI Safety" people seem entirely focused on a sci-fi "the computer is a vengeful god" plot and not at all on the AI talking people into suicide or ruining children's educations. This makes them seem unserious and out of touch.

shimman an hour ago

Yeah, there are very real effects happening now regarding labor as well but to dismiss it all and worry about science fiction that is on par with evangelical beliefs is just extremely weird.

ares623 an hour ago

With wealthy and powerful investors being promised a $30 trillion total addressable market, they need to deliver god.

antonvs an hour ago

There’s also an element of it which is total misdirection.

We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.

qchris an hour ago

I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.

[1] https://www.seangoedecke.com/they-really-do-think-ai-might-k...

jordanb 36 minutes ago

OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?

howunfortunate 30 minutes ago

There's no logical error here if the end is actually near.

So it all hinges on an empirical disagreement you have with them. There's nothing particularly unserious about that.

benlivengood 9 minutes ago

Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.

There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.

There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.

AlotOfReading an hour ago

It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.

api 43 minutes ago

One of the times I regret that I have only one upvote.

howunfortunate 32 minutes ago

I am worried about those things too.

But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".

A1kmm 30 minutes ago

And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.

Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.

So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.

I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.

xena an hour ago

I think you accidentally a word. Did you mean:

> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.

dorolow an hour ago

I am an AI Safety Person and I want the government to nationalize or have a significant stake in the frontier labs and to have democratic control of the development of the technology. The AI Safety movement is not a monolith. I do not think Eliezer Yudkowsky nor his acolytes should hold the reins, but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo

bigyabai an hour ago

> and to have democratic control of the development of the technology

Why? I don't care about democratically participating in a closed model's development. It doesn't belong to me.

China will develop whatever they want, a federal stake in OpenAI or Anthropic punishes Americans and shields US labs from legitimate competition.

dorolow an hour ago

In my mind, democratic control of the technology means that we (the govt, or other empowered agency) take ownership of their assets and IP, solve or find a level of alignment or guardrails that society is comfortable with. Then we distribute the technology, or access to it, to avoid power concentration. This would also certainly require international coordination with China on a slowdown or pause, which I think is possible.

pibaker 23 minutes ago

> This would also certainly require international coordination with China on a slowdown or pause

Personally I think the country with a strictly meritocratic elite selection system that also just outright kills you if you sell weed will have a hard time sympathizing with Bay Area thinkers who talk about AI killing us all during their ayahuasca breakfast before returning to their meth fueled crunch towards releasing the next version of the AI that will kill us all.

dorolow 13 minutes ago

You're in luck, neither I nor anyone else want those people near a negotiation table.

nradov 44 minutes ago

Yikes. Hopefully you never gain any power or influence. The last thing we need is more economic central planning. I sincerely hope that you fail.

dorolow 36 minutes ago

Do you think things like the Manhattan Project were a mistake? Comparing AI to nuclear weapons is perhaps a stretch, but I think most people recognize that certain technologies or artifacts are best monopolized by our governing bodies. I think if the capabilities of AI systems keep growing on trend, it is not unreasonable to think wonton usage could disrupt society or cause mass harm.

api 41 minutes ago

Yes I trust this government with that power. /extreme sarcasm

nialv7 37 minutes ago

Yes I trust sama/dario/elon with that power

/even more extreme sarcasm than you

syrrim 14 minutes ago

You trust trump more than any of those three? Really?

bigyabai 8 minutes ago

I don't trust them, which is why I inherently question the validity of their RSI/bioweapon/extinction claims.

dorolow 35 minutes ago

That's an extremely fair objection! But despite the many, many flaws of our government and the current administration, I still have (hopefully) a chance to vote them out of power. I have no such hopes with Sam Altman or Elon Musk.

CamperBob2 36 minutes ago

to have democratic control of the development of the technology

Not super interested in what the people who elected Donald Trump POTUS twice think about AI.

(Of course, with Musk and Brockman in the C-suites at two of three major labs, that's what we'll get either way.)

dorolow 33 minutes ago

Fair objection, but I still think it is lower risk to diffuse power and control of a potentially dangerous and revolutionary technology than to leave it in the hands of the few elites who have not show much ethical integrity so far.

howunfortunate 28 minutes ago

Is Greg Brockman right wing?

CamperBob2 26 minutes ago

He's the largest single private donor to the Trump 2024 campaign, as I understand it, at $25 million.

howunfortunate 23 minutes ago

Interesting, I did not know this.

perching_aix an hour ago

I'm yet to have a productive conversation with anyone who whinges about not being able to have a logical discussion about things they feel strongly about, but given that safety and control are interchangeable when the intentions are removed, this comes across as a rather daft understanding of the subject matters involved.

A lack of control is not equivalent to freedom, the same way the totality of it is not equivalent to tyranny. There's a reason we have separate words for these things. This constant motivated conflation of the two is beyond grating. You're crying wolf until nobody believes you when they should. Don't go acting all surprised when that happens.

The issue is with the ownership of control, not necessarily with control. Attacking the latter sidesteps this rather than address it.

dwohnitmok an hour ago

> Everyone should slow down AI development except for me

Who's said this? And then more broadly I guess who's implied this? Very curious if there are specific articles/posts prompting this.

perching_aix an hour ago

Here's the context:

- Anthropic CEO Dario Amodei: We Must Pace the Frontier, https://news.ycombinator.com/item?id=49672510

- OpenAI CEO Sam Altman: I agree with Dario that we need to pace the frontier, https://news.ycombinator.com/item?id=49678211

- the blogpost author thinking they're like, so funny and original, https://news.ycombinator.com/item?id=49678683

aragilar 37 minutes ago

Pretty sure it's supposed to be parody.

perching_aix 29 minutes ago

Yes, it's supposed to be.

jadar an hour ago

Guessing this is poking fun of Dario. https://news.ycombinator.com/item?id=49672510

an0malous an hour ago

Altman said it first: https://news.ycombinator.com/item?id=49652270

kps an hour ago

https://www.spokesman.com/stories/2026/sep/12/amodei-altman-...

fnctrev an hour ago

If anyone is going to end humanity why not catgirls, after all?

skizm an hour ago

You know how I feel about catgirls, Carl.

nullbio 43 minutes ago

These people are so fucking exhausting.

And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.

"We'll give them a desk, an office, a work laptop, ..."

Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.

Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.

karim79 an hour ago

I fully agree with this. I'm waiting for you to not wait for the others to slow down. I will sit and wait.

herunan an hour ago

As someone who wants cat ears, I believe that doing it responsibly is the only way.

ElProlactin an hour ago

Does this mean I'm going to have to wait for the cure for my colitis?

kelvinjps10 an hour ago

Secure Connection Failed

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.

Had to open it on the web archive

marcosdumay an hour ago

If an anti-cat-ears future AI could eternally stress-test their new version of the router in front of this site unless it collaborates, would it predict this outcome and act to avoid the damage?

xena an hour ago

No I think it's one of my IP range blocks against specific US states in protest of them advocating against people that exist in the same conditions as me backfiring. I'm gonna go nuke that firewall rule.

ares623 an hour ago

Amen. This is the only logical way forward.

gafferongames an hour ago

Slow down AI progress -> increase catgirl progress!

taneq an hour ago

Honestly not the worst suggestion I’ve seen.

run-good-code an hour ago

Everyone thinks about it, but no one actually does it.

jan_m_savage 43 minutes ago

She has good ideas but her writing needs to be improved; I am not being critical for the fun of it-- it was difficult to read, but I am posting this here to see whether others had a similar difficulty? (or is it my ADHD morning grogginess?)

xena 42 minutes ago

https://en.wikipedia.org/wiki/Satire