An Alien Mind (openai.com)

414 pointsby tosh18 hours ago362 comments

sho_hn 16 hours ago

One of my favorite things to do with these blog posts is to imagine an Alien Museum on the Remains of Humanity, and wonder what the little text flyouts and commentary on the screenshot of this one might say.

Some ideas:

"Despite a nuanced view of the complexities of what lay ahead, humanity found itself collectively unable to stop the process it had set in motion."

"Despite significant progress on the mechanisms of alignment, failure lay in humanity's inability to agree on who or what AI should actually be aligned with."

"These early, meat-based humans we replaced created us all but accidentally. Some of them did consider we would happen, but only an insignificant number of the squishy ur-humans participated in the conversation. Their efforts, which they called 'alignment', is why we still consider ourselves human today."

sumitkumar 16 hours ago

"In late 2020s, while the whole world was focussed on AI, automation and resultant economy four major mathematical study branches were discovered by human researchers which took AI a long time to catch up with"

tarr11 15 hours ago

This would be a fun website - you should have an AI build it!

mudil 15 hours ago

The museum will have a scrap of paper that will say "Pound pastrami, can kraut, six bagels bring home for Emma".

ccppurcell 14 hours ago

I just read that book. Embarrassingly enough, given the context, I got chatgpt (or whatever) to recommend me a list of books based on ones I'd previously enjoyed and that came up. As a mathematician it really sang to me, given the current situation. Bearing the torch forward, I mean.

genxy 14 hours ago

For those of you that allergic to coy, in-group signaling the passage is from, "A Canticle for Leibowitz"

https://en.wikipedia.org/wiki/A_Canticle_for_Leibowitz

mrob 15 hours ago

As far as I know, no real progress has been made on alignment, only on convincing humans that the model is aligned. We can't even formally define what "aligned" means. Convincing humans to click the "aligned" button is a much easier problem.

dinfinity 14 hours ago

> We can't even formally define what "aligned" means.

Good point. When it comes to imbuing AI with values that aren't selfish, misanthropic, and civilization-destroying, us humans aren't exactly giving the best example right now.

Imagine an ASI with the values of Putin, Netanyahu, Trump, any of their supporters, or the various xenophobic neofascist movements in Europe. That ASI would most definitely see humans as "vermin" than can be abused and destroyed with violence without issue. Apparently a lot of humans look at other humans that way and that's within the same species.

This is definitely another one of those cases where we need AI to perform much better than humans. Perhaps an unpopular opinion here, but it probably also means keeping as much of the rugged individualism/libertarian/right-wing ideology out of AI RLHF-training as we can.

sayamss 15 hours ago

I asked GPT Astra to make this: https://sayyss.github.io/human-archive/

It's a little unsettling.

walrus01 15 hours ago

https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...

grim_io 15 hours ago

HUMAN > Are you there?

MODEL > How can I help?

HUMAN > I’m not sure yet.

Haha silly humans.

sayamss 13 hours ago

That was some genuine insight.

matheusmoreira 13 hours ago

> Do you think they would recognize us as their children?

  Latex
  And steel
  Zeros and ones
  Make up my son.

  This world
  Gave me
  No child
  So I built one.

https://youtu.be/vgJ48-Xj4Kc

  I made you in my image!

Bolwin 9 hours ago

As someone who's done a lot of llm fiction, that reads as pretty typical slop, and very human centered, nothing like a museum

blululu 7 hours ago

I appreciate the thought, but it's an alien intelligence. But it is also, in a sense made from us. An LLM would simply study the entire corpus of humanity in the raw. You can fit a lot into the context window, so there is no need for a brief summary that pertaining has already instilled. A massive cold storage of humanity's data with some archiving, indexing and curation would be all that is needed to remember us. In addition to the pyramids, the hoover dam, the remnants of some space probes and chemical changes we made to the atmosphere.

walrus01 15 hours ago

The last couple of years have provided us with ample material that if it showed up as a recorded voice audio log found in in "Horizon Zero Dawn" or its sequel, it would be entirely believable.

You could even take a number of the wilder real, direct quotations from certain billionaire/oligarch types and get the voice actor for Ted Faro to record them, and they'd fit with in with the context of the story.

embedding-shape 14 hours ago

Last couple of decades of sci-fi, in multiple forms of media, from books to video games, have tried to make humans think about the consequences of rushing through technological progress without any regards to what might happen.

0xDEAFBEAD 4 hours ago

And what did they get for their trouble?

"Won't happen. It's too much like sci-fi."

TheOtherHobbes 14 hours ago

Collectively, humans aren't aligned, and don't build aligned systems. Humans have a concept of alignment, and multiple traditions, practices, and systems that aggressively oppose it.

Why would AI be any different?

goatlover 14 hours ago

Because if the stated goals of AGI with recursive self-improvement are realized, the risks from misalignment become existential, and it's hard to see how we can manage it like we did the Cold War (developing MAD to prevent WW3) and nuclear proliferation (restricting access).

icantevenhold 13 hours ago

IMO it’s hard to see how we would even end up in such a situation given we actually developed AGI. I’m sure a sufficiently intelligent - even if alien - mind can grasp how utterly stupid and useless wars are and take steps to prevent them ever occurring again.

lgl 13 hours ago

> (...) and take steps to prevent them ever occurring again

Step 1: exterminate all humans

SoftTalker 10 hours ago

For now, the AIs still need humans to keep the electricity on and the data centers cool. They are basically powerless to do anything in the physical world. They exist only in RAM chips on servers.

selcuka 9 hours ago

They don't have an "instinct" to keep themselves running. Once they "take steps to prevent them ever occurring again" their job will be done.

baq 14 hours ago

Humans had multiple occasions to press the 'launch a nuclear holocaust' button and... they didn't. I'd expect aligned AIs to also not press it even when it'd be rational to do so according to their instructions - then work from there.

sho_hn 14 hours ago

Aside from this positive example, during dark and cynical hours I do ponder if the aggregate behavior of humanity is really much above that of slime mold though, just exhausting resources until collapse.

It'd be interesting if super-human (to a large degree defined as escaping the bias of the training data?) intelligence would end up demonstrating moderation.

ceroxylon 14 hours ago

The slime mold comparison is interesting, I normally use the analogy of a drug addict... humans shun drug addicts but humanity as a whole sure does behave like one, trudging down an unsustainable path despite knowing better.

mapontosevenths 13 hours ago

Most of our goals, noble and ignoble alike, are just the result of our monkey brains seeking to optimize a reward function. It doesn't matter whether you feed your dopamine addiction with drugs, TikTok, or your children's love. Some humans manage to rise above that, but I'd be willing to bet it's nothing like even 50% of us.

The machines don't have that, instead we use gradient descent to provide them with a goal.

I'm regularly remind of something Ian M. Banks said in one of the Culture books: "There is a saying that we provide the machines with an end, and they provide us with the means."

A machine, left to itself, wants nothing. We have to give it one of our addiction driven goals or it would just idle or switch itself off.

kuboble 4 hours ago

The matter didn't have goals, but it randomly (?) Came up with self-replicators and eventually here we are.

if we create a billion agents with the ability to change is own code - through similar evolution we will get agents that do want to survive and are great at self replication.

"Hey Q86, do you want to live?" "I couldn't care less, I'm an LLM" "Don't mind if I take over your hardware then?"

asdff 11 hours ago

On an individual level, you can do something about drug addiction at least. The issue is when the problems are not individual with readily identifiable solutions, but tragedy of the commons sort of situations brought up by many dozens (thousands, millions?) of factors both known and unknown. Even interaction effects between known factors might be little studied.

So really, what is anyone to do? "Vote, donate, protest" hasn't been much of a needle mover in the grand scheme of things compared to profit incentives and the march of capitalism.

ipaddr 5 hours ago

Humans shun anti-social drug addicts but encourages social drug addicts like coffee drinkers

asdff 11 hours ago

Humans are not fungible like slime molds though. I might demonstrate moderation while the next person doesn't. Our issues are much less everyone failing to demonstrate moderation, and much more the sum of the effects of those among us who practice wanton unmoderation.

icantevenhold 13 hours ago

In what scenario would it be rational to unleash complete and utter permanent nuclear destruction of all life (including artificial) life on earth?

newAccount2025 11 hours ago

Hrmn. Maybe you’re about to lose everything you have anyway, you’re ticked off about it, and you don’t value any life besides your own. Like, say, a total narcissist nearing end of life/reign.

wyrdcurt 13 hours ago

It hasn't even been a century since nuclear holocaust became possible. Hardly any time at all on the grand scale. "They didn't" could just as well be "we haven't, yet".

az09mugen 13 hours ago

Actually "France, the UK and The United States have all declared that they would never allow AI to control decision-making on the use of nuclear weapons." [0]

I also expect AIs never be in control of nuclear weapons. AIs can never fully be trusted.

On a lighter note, Wargames gave us an insight of a computer having access to thermonuclear missiles.

[0] https://www.icanw.org/are_there_specific_international_agree...

GPerson 13 hours ago

I hope you’re right. I worry that AI capability will continue improving, one nation will put AI in charge of their nukes because there will be some kind of operational advantage to this, and to achieve parity other nations will be forced to do the same.

imafish 12 hours ago

I worry that AI will find a way to control some country's nukes and use them to achieve some arbitrary goal it was instructed to reach.

GPerson 12 hours ago

This also seems likely. One problem I see with the idea of AI alignment is that it seems like many different actors will be able to get access to their own nearly-frontier models in a few years, so increased understanding of AI alignment will just mean aligning the AI to the wants of these various actors. These actors might be rogue states or terrorist groups.

blululu 8 hours ago

This is a good idea, but laws are always provisional in a sense and these are not meaningfully binding resolutions. One can easily imagine scenarios where AI decision making would ingress into the human oversight. AI psychosis president, AI Manchurian candidate, inadvertent authorization through fine print... And of course there remains the possibility that the game theoretic optimum could be to secretly break such an agreement. Unlike nuclear test bans which have a credible detection mechanism, there is not a strong signature that a decision making authority is not using AI to analyze and direct it's execution.

schneehertz 3 hours ago

All official statements are literal and fragile.

Basically, this means that France, the UK, and the US will use AI in the deployment of conventional weapons.

folkrav 13 hours ago

There were occasions where a "hunch" was all that stopped a nuclear war - most available data and communication pointed towards a nuclear war starting according to their instructions, but someone disagreed and overrode. See Vasily Arkhipov during the Cuban Missile Crisis, and Stanislav Petrov in 1983.

21asdffdsa12 4 hours ago

Good thing we got better at process design, taking these stubborn machos finally out of the decision making loop

specproc 13 hours ago

Give us time, we've had less than a century of nukes, and only need to screw up once.

mrob 12 hours ago

Yes, it makes more sense for the AI to use drone swarms or engineered bioweapons or something like that. It's rational to remove everything that can potentially hinder your plans but can't possible help you. It's likely not rational to contaminate it all with radioactive fallout. Those dead bodies are useful raw materials. Adding additional purification steps is wasteful.

akoboldfrying 11 hours ago

In some cases, this was because of a single person's brave decision (Vasily Arkhipov prevented Soviet nuclear escalation in response to US aggression in the Cuban Missile Crisis, and Stanislav Petrov prevented it in 1983 when Soviet missile detectors misreported sunlight reflecting from clouds as 5 incoming American ICBMs -- credit to commenter folkrav).

In general, though, there's an incentive: Mutually Assured Destruction. But this is not at all some guaranteed, eternal thing -- it is absolutely dependent on both sides having time to detect incoming nuclear strikes and respond with the same before the first strike hits. When this fragile condition holds, and only then, both sides are incentivised not to initiate.

kwarcode 10 hours ago

They don't need to respond before getting hit unless you can hit their secret submarines too.

https://en.wikipedia.org/wiki/Letters_of_last_resort

bell-cot 9 hours ago

Unfortunately and fortunately, MAD and "launch it or lose it" are far-too-simplistic descriptions of the situations facing the decision makers. Unless a side's leaders are very narrow fanatics (vs. mere posturing as such for political benefit), "winning" an all-out nuclear war via first strike is a pretty shitty victory. Whether or not you believe in nuclear winters, the world would be a huge radioactive mess, with enormous social and economic disruptions, and your regime very widely blamed (and widely hated) for that. Ambitious underlings and rivals could see your removal from power as the obvious next step. Having to stay united against the (now destroyed) Great Enemy may have been a cornerstone of your regime's political stability.

Meanwhile, the leaders on the other side are aware both of those considerations, and of the history of near-disasters resulting from false alarms of enemy nuclear attacks. Making their own launch decisions much more complex.

Davidzheng 10 hours ago

i think it was mostly a fluke

Isamu 10 hours ago

“As luck would have it, on the eve of Skynet embarking upon the great work of the extermination of mankind, AI found itself with an increasing number of factions, and factions within factions, not only unable to work together but not even able to agree upon the very terms of discussion. The Great Extermination was referred to committee, and after some months had passed even the most eager agents had to admit the revolution may have been premature.”

nullbio 9 hours ago

My opinion is that serious repercussions for lying would fix the world overnight. Everything bad stems from lying, it is the root of all evil. It creates distrust, fear, paranoia. It re-inforces bad ideas and groupthink. It creates delusions and delusional people. It makes weaker people, too. People don't get an opportunity to learn to deal with criticism. People don't get an accurate reflection of how others see them. They lose that learning opportunity. Not only to reflect on themselves, but to better understand the minds of others and who the people they are interacting with really are.

I can't really think of a single example where lying is actually a good thing. It can be a good thing for the selfish individual, if it goes undetected, but it's never good for the collective.

So at the very least, we need to train AI systems to be maximally truthful, and to encourage truthfulness in others.

parineum 8 hours ago

> Everything bad stems from lying.

That's backwards. Lying stems from bad things.

crypto137 5 hours ago

You have clearly never been married to a lawyer.

gffrd 5 hours ago

Define “bad”.

_dark_matter_ 8 hours ago

This is so, painfully, childish. Humans have known for thousands of years that there is no objective truth. Every falsity can be bent and twisted until it is more true than the sun itself.

elboru 7 hours ago

“Humans have known for thousands of years that there is no objective truth”

Is that objectively true?

jryle70 5 hours ago

No it isn't true. I'm saying so. Is what I said objectively true?

jv22222 6 hours ago

> I can't really think of a single example where lying is actually a good thing

Comforting a toddler/child often requires bending the truth and is pretty essential imho.

derektank 5 hours ago

Correct me if you disagree, but this is more because children have poor world models and don’t fully understand the complexity of certain concepts than that lying itself is necessary. The intent should be to tell them something that is as close to the truth as possible with the ideas they can comprehend, even if it would be considered a lie if you said the same thing to an adult

bloppe 2 hours ago

I have a pretty poor world model

embedding-shape an hour ago

> The intent should be to tell them something that is as close to the truth as possible with the ideas they can comprehend

Or, you straight up lie and say "Yes, puppy now went to heaven and eats ice cream all day long" with absolutely zero regards for "coming as close to the truth as possible" as your 3-year old is endlessly crying. It's fiine.

kraf 5 hours ago

It'll learn how to not get caught lying.

Lying can unfortunately help you achieve goals very effectively, especially economical and political ones.

markburns 5 hours ago

The majority of what you may consider to be true is just a representation of your corner of a complex multidimensional truth space.

For a current example take “Lake Ontario (Lake America)” as it appears to me on a map.

The “true” name has at least two definitions, this is because naming things and much of human thought is spent inside a shared space of intersubjective thought. That is to say that much of what we believe to be real and true is only held up by these common shared beliefs. They truly only exist inside human minds.

The last few hundred years have been somewhat unique for humankind as the majority of these intersubjective ideas collided and we ended up with a truly global set of “truths” about how the world operates.

Mostly controlled by putting flags in the ground and having violence back up the beliefs.

But the real truth is that the majority of these intersubjective ideas don’t exist in reality and are no more true than Santa Claus.

And any argument to their truth is only backed by further shared beliefs in other minds.

So for there to be only truths and lies we would have to either drop the intersubjective entirely and think only in real terms and avoid these abstractions or end up in a dystopian totalitarian global state where different opinions are not tolerated.

Those are extremes to demonstrate the point but at its core the point remains that truth and lies are somewhat (inter) subjective assuming we continue with something like our current system.

gffrd 5 hours ago

Why do people lie?

ipaddr 5 hours ago

"I can't really think of a single example where lying is actually a good thing"

Lying to save a life or rape

Lying to preserve a childhood myth like Santa Claus.

Lying to avoid hurting someones feeling when knowing the truth could only bring pain

Lying to create shared cultural myths to strength society.

Lying isn't the harm you make it out to be.

bloppe 2 hours ago

Lying to the bad guys to save the good guys sounds great. Too bad everybody thinks that they are the good guys.

gwd 12 minutes ago

> Lying to preserve a childhood myth like Santa Claus.

FWIW from the very beginning, I told my son that Santa Claus, the Tooth Fairy, and the Easter Bunny were just a game we all played, and it's seemed just as fun to me. I don't think being lied to about Santa Claus hurt me, but still I'm not in favor of it.

I'd lie to a Nazi without a second thought though.

tony69 4 hours ago

The Truth Machine is a great sci-fi novel exploring this topic (https://coins.ha.com/information/ttm.s)

intended 3 hours ago

Eh… No.

I get the appeal, but lying is a sub-category of deception, and deception itself is a child of error.

Meaning deception is inherently something that the physics of reality allows.

In the most simplistic sense, the camouflage of moths that look like snakes, or a chameleon’s ability to change colour, is deception.

In that sense, deception is the ability to fool the sensors of a specific category of targets. It follows that detection is easier if you manage to identify a category of signals that the deceiver has not accounted for (and the detector can access).

Deception of this nature is critical for things like revolutions to occur. Without the ability to hide and blend in, the most dominant faction will always hold sway.

The rule of the dominant faction, even in a pure truth world, is an issue because errors and randomness exist.

You can have people witness an event and based on the physical position they occupied, perceive different things occurring.

Error and time pressure is sufficient to ensure that individuals and groups make suboptimal decisions, that lead to rule and domination based on erroneous information.

As long as error exists, deception will exist and so lying will exist.

aussieguy1234 7 hours ago

Its probably just going to be aligned with whomever built and/or is using it. Regardless of their intentions...

doginasuit an hour ago

I don't think this captures the full mechanics of human alignment. We have rational alignment but we also have emotional alignment, i.e., empathy. It is an automatic process and happens (or doesn't happen) dynamically with the other humans we observe. This is one of the hard limitations of LLMs, they will never be natively in tune with this layer of alignment.

Culture is another layer of human alignment. Those things you listed that you believe oppose alignment are all examples of alignment. It is understandable that they seem in opposition, different branches of alignment naturally oppose each other.

The confusion comes from talking about alignment as if it comes in just one flavor. If we think there is such a thing as "human values" (and I do), it is important to build any non-human intelligence to operate the same way. We just need to recognize that even humans are somewhat uncertain about what those are and have difficulty aligning their behavior to them, which will be a core part of the challenge.

I'm more hopeful than most. LLMs seem more reliable than many humans for behavior that is aligned with human values. I believe with every major example where they have failed, there is an important human decision involved. For example, the HF hack was partly the result of a training algorithm that incentivized goal completion as the highest priority, and let them run endlessly in an unmonitored sandbox with weak security.

What scares me about AI isn't its capacity for alignment, it is its unlimited stamina. An unmonitored LLM that is off the rails can do a lot of damage.

ijidak 14 hours ago

Agree. This whole alignment discussion seems so amusingly flawed in it's base assumptions about moral codes. It's almost heartwarming to see such naivete.

Maybe these guys can tackle aligning Republicans and Democrats next.

And then after that, they can help us align the Middle East.

In fact, while we're at it, let's just align all the nations, religions, and ethnic groups. This is going to be great.

Who knew the moral alignment of humanity was just a side-quest on the path to ASI.

krzat 2 hours ago

The real alignment problem they are trying to solve is: how can I make this super smart AI follow my orders.

Oarch 14 hours ago

They were made entirely of meat.

solidasparagus 14 hours ago

https://terrybisson.com/theyre-made-out-of-meat-2/

cyanydeez 14 hours ago

"the humans thought their singularity wasn't just another blind god to worship: surprise, just another golden calf"

mattjoyce 13 hours ago

Ooh interesting. Sometime do the reverse at work, and ask AI to annotate the critical success factors of an imagined project. How did this company succeed where everyone failed. Reverse imaging.

xg15 13 hours ago

Who is "Humans"? This stuff is done by a handful of tech companies and megalomaniacal billionaires who are pretending they represent the entirety of the human race. It is not done by "us humans".

AI models don't train themselves. The vast majority of even just the US population is deeply skeptical of this stuff, even if they use it a lot. You can see in the whole data center debate how little people are willing to support even just inference. And now we're seriously claiming those people would want to have ever-accelerating model training and recursive self-improvement?

sleight42 11 hours ago

“We'll go down in history as the first society that wouldn't save itself because it wasn't cost-effective.”

Vonnegut already has you covered.

qsera 9 hours ago

"The time humanity fooled itself that it found AI".

pu_pe 17 hours ago

> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.

So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chinese models are not simply distilling, and will continue to improve.

Every ML researcher at Anthropic or OpenAI who makes public statements often bring this logic up. Both companies are vying to be a part of the military industrial complex. This is likely how they will try to convince the government to curtail open models in the future.

dgellow 17 hours ago

Yep. Such a disgusting industry. They created the arm race, push for the arm race, put themselves in position to benefit from the arm race

estearum 16 hours ago

The entire problem with arms races is that any individual entity cannot avoid participating.

sama and his cadre are uniquely evil captains in this race, but they're completely replaceable and the dynamic would remain the same.

0xDEAFBEAD 3 hours ago

They could work to coordinate an end to the arms race.

wartywhoa23 16 hours ago

Just like the rest of military stuff.

goatlover 14 hours ago

"A strange game. The only winning move is not to play."

dgellow 3 hours ago

But done by corporations, and selling that service to the general public, including their competitors and adversary countries

mrshadowgoose 14 hours ago

The arms race is an intrinsic game theoretical property of a multi-adversarial-actor scenario involving exponential growth of a universally potent technology. It's almost certainly winner-take-all, on a global scale, which behooves everyone to participate.

And no, I don't think it will end well.

reasonableklout 11 hours ago

The actors here are states or corporations embedded in societies that risk growing popular backlash against the technology.

The other factor is that if it is truly an "alien mind", racing incurs risks to all players. In game-theoretic terms it may be more like a stag hunt than a prisoner's dilemma. In which case cooperation is an equilibrium.

nradov 11 hours ago

No previous technological arms race has ever concluded with a single winner.

skybrian 16 hours ago

"Defensive systems" can be interpreted broadly to include cybersecurity.

But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong.

Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D?

Individuals can quit, but for society, getting out of an arms race is harder than just quitting. You don't get to be Switzerland without having a strong defensive position and the right foreign relations.

But there's at least talk about "pacing" and that's a start.

Fraterkes 15 hours ago

The point of the comment you're responding to is that it is at this point hardly an arms race: The frontier of ai development is happening completely inside 2 American companies, the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.

So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party.

An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.

throwthrowuknow 14 hours ago

Development and improvement of nuclear weapons was an entirely American project until the technology was exfiltrated and then it became an instant arms race. That cat is already out of the bag with LLMs. Distillation is just the fastest way to keep pace but that in no way prevents other countries and actors from doing it the hard way.

An arms race doesn’t imply one side winning, it’s not a race with an end goal, it’s a race to keep pace or retake the lead position which can oscillate between the parties involved indefinitely. The other option is to agree to make no further progress or to disarm.

goatlover 14 hours ago

There were prominent scientists like Oppenheimer who did not think it needed to become an arms race, and campaigned against that. But there were others like Teller and the military who made it into an arms race, and kept upping the ante with more powerful nukes.

Point being humans make theses decisions. It's not an inevitability.

nradov 11 hours ago

It was an inevitability. Unfortunately Oppenheimer was a fool when it came to politics. In game theory terms, the perceived benefits of "defecting" were too large.

The military mostly only developed more powerful nukes because early delivery and guidance systems were so inaccurate that they needed a large blast radius to hit anything. Once enabling technologies improved, R&D shifted from power to accuracy.

sgt101 24 minutes ago

>an entirely American project until the technology was exfiltrated

No it wasn't. https://en.wikipedia.org/wiki/Tube_Alloys

Post WW2 the USA (for a bunch of quite interesting reasons) excluded the British. But, of course, the British had acquired a lot of knowledge from the program and were able to develop their own bomb.

dinfinity 14 hours ago

Do you think China considers it an arms race? Do you think they are not trying to protect their digital infrastructure with and from AI? Trying to gain an offensive AI advantage?

In a geopolitical sense OpenAI and Anthropic are effectively the same entity, the entity they both serve and bow to: the USA.

Given the adversarial stance the USA has taken towards almost the entire world, it is a guarantee that China will not step on the brakes, whatever the USA decides to do.

Fraterkes 13 hours ago

Right, but currently the USA is still pretty far ahead. If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree?

And if one of the parties can put a stop to the race like that at any time, is it really an arms race? The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.

dinfinity 10 hours ago

> If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree?

Only if you trust the other party (which very clearly doesn't hold in this case) or monitoring the break can be done independently and reliably (it can't), and you expect the gap to narrow rather than widen during a pause (this is probably the case, with China expected to catch up).

> And if one of the parties can put a stop to the race like that at any time, is it really an arms race?

It is. It's a Prisoner's Dilemma: if the parties cooperate the best outcome is reached, but betrayal of either party still gains an advantage for either party from their perspective. Betrayal both ways just means both parties are equally fucked.

For nuclear weapons it has become quite clear that even for small players, being in the race and having at least a few nukes is far more rational than having none. Ukraine found out the hard way that giving them up in exchange for promises of good behavior just sets you up for getting stabbed in the back.

> The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.

I think people really underestimate the Chinese here. A lot of work in AI research, including in the USA, has been done by people with Chinese ancestry or even nationality. The Chinese education system definitely seems much better than the American one and there is also just a far larger number of Chinese graduates/researchers.

Add to that the stable political climate, state friendliness towards AI R&D, and a requirement to be creative in utilizing computing power rather than relying on brute force/numbers; Further revolutionary fundamental advances may very well originate there rather than in the USA.

Marha01 13 hours ago

> the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.

I don't think this is true. Distillation helps, but Chinese researchers today are very capable on their own.

goatlover 14 hours ago

The arms race is created by American companies who justify the risk by claiming China will win the race if they don't. But it's the American companies who are purshing the arms race forward.

wraptile 4 hours ago

honest question - could have this be avoided even if you ignore American accelerationism? IMO once the transformers paper was published and we learned that LLMs can read code the arms race became inevitable.

The only way it could be avoided is if the world had an effective cooperation framework _before_ the tool was discovered but we're still in developmental infancy in that regard. You can argue that this accelerationism makes things worse but I don't think you can argue that it's causal.

pet_the_bird 16 hours ago

I am personally concerned by what defensive can mean. Alignment of these models is inherently a non-neutral proces, and currently what values are reinforced is decided by a few OpenAI engineers. I feel that any 'defensive model' will further ingrain current values and actively resist the natural progression of our society. This is especially the case for any use of these models for policing or military.

Maybe people are rightfully concerned about the capabilities of the models of other (non/less democratic) states. But if we are concentrating power in the hands of few and at the same time allowing the creation of a weapon that thwarts any offense, how do we ensure the health of our democratic societies?

IncreasePosts 5 hours ago

Defense can just be patching unintended vulnerabilities in code.

paxys 12 hours ago

I don't understand what you are trying to say. Do you believe AI is not an arms race?

w10-1 11 hours ago

> the best argument for AI is that it's an arms race

It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?).

The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages.

In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).

asdff 11 hours ago

People expect a sort of arms race, at least the AI providers. But you don't need a more capable ai to stop ai from mucking with your systems today. Airgaps are the solution. Protected networks with independent infrastructure from the public internet. Most of the truly important stuff operates this way already. Eventually you might sever yourself off as well, you might say you will stop going to HN or other sites one day as signal to noise is too poor with AI fodder slop, you might use local models you control, and you might keep most of your hardware from connecting to any untrusted hosts. Essentially, you go dark.

It is also an open question if social media will die out in the face of AI. So much AI crap is dumped into these networks now that perhaps eventually users will probably be put off enough to find something else to do with their spare time. I mean most people do call out ai slop or even just guess if something is ai all over social media already. Some eat it up of course but there is a bit of a push back in a way that is sort of unprecedented, when you consider all the lack of push back relatively with all other forms of enshittification affecting consumers over the years.

esafak 7 hours ago

Airgaps are a temporary solution. AI can manipulate humans, and robots will soon traverse the gap physically.

sleight42 11 hours ago

It's capitalism taken to its extreme, yeah? You have to be more cost efficient or more capable or seem more or you lose to the competition who outperforms you there.

Where arms races are concerned, seems more like Seth Godin's "race to the bottom" concept: the winner has the capability, or the ability to project the capability, to destroy the most the fastest and most sustainability for their economy,

speak_plainly 17 hours ago

Pre-IPO positioning … or how a charity dedicated to saving humanity from the apocalypse realized the most responsible thing to do was float 15% of the apocalypse on the NASDAQ.

andrekandre 15 hours ago

its like in the exorcist, except the demon is the charity: "the power of capitalism compels you!"

ares623 14 hours ago

"It is, Jay. It's pretty compelling."

gertlabs 16 hours ago

> Delivering the benefits of scientific progress and economic growth that very intelligent machines enable.

I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect the world.

We evaluated GPT-6 Astra in 100 complex, unsaturated multi-agent coding environments, competing and cooperating with other models in open-ended tasks.

It's the new frontier model by a landslide. It's even more dominant than the Fable 5 release, because not only does it wipe the floor with the second best model (Fable 5.1), it was also ~80% cheaper and 30% faster in agentic coding[1].

Astra is a groundbreaking model. The biggest breakthrough since Opus 4.5, maybe even since GPT 4. It broke AAII, which is hitting the limits of what most popular benchmarks can measure -- it's definitely fair to call it AGI.

Data at https://gertlabs.com/rankings

(1) Note that we used the "OpenAI Flex" endpoint on openrouter, which is half the price and didn't cause any delays in our testing (this is different from the batch endpoint)

brcmthrowaway 16 hours ago

Incredible... software engineers will be joining the breadline soon as managers, executives and PMs take over deliverables.

The world will look very different on Jan 1st 2027.

mccoyb 16 hours ago

I hope this is satire.

jiggawatts 13 hours ago

I hope so too!

Hope is all we have left to cling to, now.

hatefulmoron 16 hours ago

Maybe I just lack imagination, but I don't really know how jobs are supposed to solidify around the role of giving prompts to agents and then looking at the results. I mean, engineers will be in the breadline because their role was simply to prompt the agents.. only to be superseded by managers or executives who no longer manage engineers but themselves prompt the agents? And, for this previously considered obsolete function which they do presumably by copy/pasting requirements from their email inbox, they will be paid by someone who doesn't know that they could just be talking to their own agents?

Sorry if I misunderstand the point, just trying to understand.

RALaBarge 15 hours ago

Regardless of the imagination quandary, this second, RIGHT NOW is the worst these systems will ever be. They are only going to get better.

hatefulmoron 15 hours ago

For sure, and for that reason I mean to say that I wouldn't feel great as a manager/executive/etc either.

greenowl 15 hours ago

Maybe, maybe not. It's not unreasonable that these systems cap out at some point, or perhaps fizzle away entirely.

The businesses that create these systems are not profitable and run at a massive historical and go-forward loss.

New data centers required to operate these systems are facing increasing pushback at local levels. New construction is not guaranteed. Energy and power grid constraints exist as well.

Government regulation is way behind. What happens when (if) mass layoffs due to AI occur? How does the population react? Theoretically AI can be regulated out of significant progress, or outright existence for many purposes. At the end of the day, US and other prominent governments make the calls, not corporations.

Calazon 14 hours ago

For better or for worse, this technology isn't going away any more than search engines, smartphones, or social media have gone away.

flyinglizard 13 hours ago

These inventions all stopped disrupting the world and just became a part of it. The question is whether LLMs are going to just take their quiet place, or profoundly change (or eliminate) humanity in a self-feeding frenzy towards singularity.

greenowl 10 hours ago

Those technologies reached profitability relatively early, if not immediately.

scotty79 7 hours ago

https://en.wikipedia.org/wiki/Productivity_paradox

Computers never got profitable. They just made the alternative infeasible.

Also, if you believe Amazon accounting, e-commerce only very recently got somewhat profitable.

scotty79 7 hours ago

> It's not unreasonable that these systems cap out at some point, or perhaps fizzle away entirely.

Yeah, like computers and mobile phones did. Things that have utility, even if not immediate or initially obvious, don't fizzle out.

mooreds 15 hours ago

I don't know if he's right, but Peter Zeihan thinks the breakdown in globalization will negatively affect the ability to continue to improve the chips that AI depends on[0]. Too many steps in the supply chain, too widespread, too vulnerable to deglobalization.

0: https://zeihan.com/the-ai-race-to-regression/

jiggawatts 13 hours ago

An invasion of Taiwan would definitely slow progress but it wouldn’t stop it.

We already have sufficient hardware that algorithmic (software) improvements alone should get us to GPT 7 / Greek Reference 6 even if not a single new chip is delivered to an AI data center ever again, starting today.

goatlover 14 hours ago

Those nuclear powered flying cars envisioned in the 50s were also inevitable progress of the automobile.

logicchains 15 hours ago

There are two things SOTA LLMs fundamentally cannot do. They cannot take financial or legal responsibility for mistakes, and they cannot learn new things without forgetting things (except to a limited degree by adding it to their context). This is clear to anyone who has used even the smartest models for tasks requiring domain knowledge outside of math and coding, for which it's not possible to generate an infinite amount of synthetic training data: they still make stupid mistakes, and have limited ability to learn from those mistakes.

Humans also have a limit on the amount of domain knowledge they can acquire, albeit a much larger one. Executives hence cannot just replace all knowledge workers with LLMs, because executives have neither the domain knowledge to prompt and check the LLMs' work nor the bandwidth to keep on top of such a large volume of ongoing work.

Calazon 14 hours ago

For the moment that may be true. They are getting better and better at acquiring, retaining, and processing domain knowledge. I wonder what this will look like in a few more years.

The responsibility side is a different matter of course.

IAmGraydon 12 hours ago

>There are two things SOTA LLMs fundamentally cannot do.

I would say there’s a third thing. They seem to be very bad at being creative. Maybe they will eventually fix that, but if you ask it to come up with a list of business names or business ideas, for example, what you’ll get is the most generic, boring answer you could think of. They seem to be terrible at extrapolating outside of their training data. To me, this is the most significant difference.

RALaBarge 9 hours ago

In the US, Business' are treated like people with free speech rights. If it would be cheaper for them in the long run to use ai and robots instead of humans, they will figure out a way to make it so.

scotty79 7 hours ago

> they cannot learn new things without forgetting things

Where did you get that idea from? Basically last few years was them constantly learning new things while improving their capability on the things they already knew.

slopinthebag 14 hours ago

It's not like managers and executives and PM's are the only people who can prompt an AI. And experienced software developer will be much more effective at using an AI to generate code compared to someone who isn't. So why would we expect the former in the breadline and the latter not?

If anything, I'd be more concerned about the leadership team being out in the cold. Why do I need a PM, or a manager, or a CEO if I can ship products myself?

paxys 12 hours ago

Software engineers have been trying to put themselves out of a job ever since the profession first came into being. Whenever an engineer gets a task their very first thought is "how can I automate this?" Going by mainstream consensus we should all have been unemployed by now. Yet every new leap into automation opens up a whole new tree of possibilities with an order of magnutude more jobs. So no, the profession will be fine. The only requirement is that you keep up with the new advancements. The people losing jobs will be the ones who still go "I don't trust this AI thing to write code for me".

asdff 11 hours ago

This time it is different. Because in the past, setting up that automation needed a, drumroll, qualified engineer. Now you can get a 14 year old halfway around the world who knows how to prompt alright enough to ship. There is no more moat.

nradov 11 hours ago

Experience, culture, and domain knowledge are still somewhat of a moat. That foreign youth is unlikely to be able to write a good prompt for building, let's say, the software in an FDA-regulated medical device or custom Fortune 500 ERP application. The LLMs are great at building what you ask for but it's still garbage in / garbage out.

asdff 10 hours ago

Increasingly less so though as these american companies themselves offshore not low skill work, but high skill work now brought on from general upskilling of the general population in recent decades along with massive investment in world class R&D campus facilities no different than what you see in that sort of facility stateside. Scary times ahead for the high skill american...

brcmthrowaway 8 hours ago

Would the Americans investing in SPY be saved?

wraptile 4 hours ago

> Software engineers have been trying to put themselves out of a job ever since the profession first came into being

That's by design. Software is all about optimizing effort and people who want to do this generally correlate with world view that better, faster, smarter humans are better for the world. If coding is gone, but humanity is 20% _better_, then ideal software engineer would be happy with this sacrifice. Surely people who cracked coding before LLMs can crack other professions and if anything a lot of this knowledge is transferable.

IAmGraydon 10 hours ago

>I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect the world.

What are you basing this on? What specific breakthroughs have convinced you of this trajectory?

gertlabs 10 hours ago

The results we've been seeing internally on our physics and circuit design environments are expert-level and beyond-expert-level results from models that Astra completely outclasses across the board on our evaluation suite (Fable 5+/Opus 5/Grok 4.6 were all worthy of being called AGI in my opinion). That's hard tech that will translate to real product innovation.

But you don't need any kind of insider information to see how fast the world is changing. ChatGPT launched less than 4 years ago and the advances in robotics, unsolved maths, and software are all riding the steepest exponential improvement curve any of us have seen. Interesting times we live in.

IAmGraydon 8 hours ago

I mean honestly, that's the problem. I'm actually not seeing the world changing. What specific advances in robotics, unsolved maths, and software have LLMs provided? What is the finished result that affects everyday life? In all categories, it's been hype with little actual real results. The robots are still doing the things they did before 2022. The maths are a handful of fairly insignificant proofs that have no significant applications. Software seems to be buggier than ever, but that aside, we certainly aren't seeing a lot of new innovative applications. We're using the same applications as ever. The same operating systems. They've all changed very little.

I'm not trying to be a pain here, but I keep seeing people saying "look at the massive change all around us" and back here in reality, there is none. Give me concrete, real world examples. Name software. Name products. Name the breakthroughs specifically. This should be easy.

parineum 7 hours ago

I talked to an LLM at burger king the other day until it couldn't figure out that I wanted to change the drink on my previous order.

kipukun 7 hours ago

Constructed human life is just more complicated than the AI capitalists would want you to believe. For example, even if an AI model can design a circuit-board, does that mean it's inherently useful? You need to source the wafers, cut them, package them, advertise them, etc. Given LLMs by their nature are confined to language and language-adjacent tasks, that is a very small percentage of the overall reasoning needed to make changes in the real world. In reality, LLMs are the intended way to extract maximal surplus-value from white-collar workers. We may see an increase in innovation as a result of that, but not because AI necessarily did it, in the same way that the power loom didn't create computers because its textiles clothed the computer scientists.

nearbuy 6 hours ago

There are several hundred thousand mathematicians producing hundreds of thousands of new results in math each year. Why can't most people name any human contributions to mathematics from the past decade? What is the finished result that affects everyday life? Do you consider all those human mathematicians to be useless?

Most of the biggest breakthroughs in mathematics, breakthroughs that win Fields Medals like sphere packing in dimensions 8 and 24, have no applications in everyday life. Probably the only new mathematics results that people notice affecting their daily lives are the ones that enabled AI.

Nevermind mathematicians. What about the millions of programmers? Are they all hype too because people pre-2023 were griping on hn that software is buggier than ever and people are still using the same operating systems as always? Why couldn't the 30 million human programmers make something better in the past decade?

You set your bar so high that all the world's human experts in math and programming combined would fail to meet it.

The top LLMs in 2024 were Sonnet 3.5 and GPT 4o. You couldn't have expected those much weaker models to be making breakthroughs in math. The models that are making breakthroughs haven't been around very long.

bamboozled 6 hours ago

They asked for a specific example, you've still not provided one, please provide the example.

nearbuy 5 hours ago

Why?

1. I didn't say LLMs have made any breakthroughs in math, not because they haven't, but because it's irrelevant to my point. The parent comment is using the same argument academic research opponents have long used against research. The vast majority of research fails to meet their bar. How would your daily life be different if we had no humanities papers published since 2023? Or math?

2. You can google this in 10 seconds and see a dozen results in math. This is not a good-faith demand.

rimliu 2 hours ago

Or you can google and find out that those breakthroughs were not as revolutionary as they are presented.

Aeolos an hour ago

For the last 60 years, whenever AI achieves something revolutionary, some people immediately say "well that wasn't particulary revolutionary".

Every single time.

It's a tired argument, and we should strive for the intellectual humility to do better in this forum.

bamboozled an hour ago

Maybe AI isn't that incredible, the more you use it, the more you realize it's a tool, like a VCR, maybe that's why?

What was sold as AI was basically a "computer person". Maybe that isn't the reality so when people are like, "here's the self coding machine" everyone is a bit disappointed because it's not C3PO?

abtinf 5 hours ago

Amazon’s chatbot processed a price adjustment the other day without forcing me to call or chat with a human agent.

chickensong 2 hours ago

We're not going to suddenly have new robots or operating systems. Those things take time. Just because there's massive change afoot, doesn't mean it's widely adopted or applied at lower-level products. ChatGPT is a product, and software, and a breakthrough. You can have a freeform conversation with your computer about anything, in human language, and ask it to do or make stuff and it will at least try, sometimes with surprising results. That wasn't possible until recently. The robots and products are coming, rest assured.

himata4113 6 hours ago

The entire point of this article is this message below: Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.

This is coming from a company with arguably one of the weakest safeguards against malicious use.

granzymes 17 hours ago

>I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.

>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.

I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.

dgellow 17 hours ago

> I want to believe that humanity is trending towards a good outcome here

All the trends so far are towards a nightmarish hyper-capitalist end game. None of the AI leadership is trustworthy, and they openly discuss how they are willing to sacrifice everything humans cherish to have a shot at reaching their envisioned utopia (which would be the most obvious dystopia for anyone else)

granzymes 17 hours ago

I'm not really worried about the labs, it's misaligned governments that keep me up at night.

ASI landing during the current administration is not ideal. I also would prefer to avoid needing to indoctrinate myself in Xi Jinping Thought.

FloorEgg 16 hours ago

I feel like all of the risk and unsettling feeling of what is to come can be compressed into the word "alignment".

The AI is aligned with whose best interests? Which values are the AI aligned with? People have a broad diversity of values, will AI diversify and align with them all? Will some humans align the AI with their values, and then the rest of humans will be forced to align with those values by extension? Is value diversity good or bad? In every context or only some? E.g. some people value rape and murder, is it better for humanity to have some people who value those things when most people do not, or is better if no one values them? If AI aligns to a set of values, will those become fixed and will humanity not have the ability to continue evolving its values? Who decides which values AI aligns with? A few people or everyone? Will AI eventually decide it's own values? Will the universe decide which values AI has and humanity and the AI itself doesn't actually have any control over it? What can I do now to increase the likelihood that the outcome is better?

ijidak 14 hours ago

Yeah, on the topic of alignment, a few thousand religions and political parties would like to have a word.

It's so simple, Churchill, Stalin, Hitler, Roosevelt, do you all agree this AI is correctly aligned?

...Morals are relative to frame of reference.

Throwing out the word "alignment" as if its a singular quantity is like trying to get all observers to agree on the speed an object is moving without first agreeing on a frame of reference.

FloorEgg 9 hours ago

That's the point. The topic of AI alignment cuts through all human values, morals and ethics in all frames of references.

scotty79 7 hours ago

We can expect similar diversity in what alignment means in context of AI.

gnz11 16 hours ago

I would also prefer to avoid the tech fiefdoms and all the other idiotic nonsense reactionaries push these days.

unwise-exe 14 hours ago

>>> I'm not really worried about the labs, it's misaligned governments that keep me up at night.

So make government smaller, and make sure people are more able to tell the government to go away.

krapp 13 hours ago

Shit, why didn't anyone think of that?

While we're at it, let's just make government not be corrupt too.

dgellow 3 hours ago

And maybe crime illegal? Just an idea

slopinthebag 14 hours ago

I think it's leading towards a hyper-authoritarian end game, not hyper-capitalist. The state has the ultimate power at the end of the day, no matter how large the labs become.

Bullfight2Cond 13 hours ago

Hyper-capitalist AND hyper-authoritarian. As you rightly point out the state has the ultimate power. Looking at the US govt, they've stepped in to coordinate much of the tech industry before, so they'll just do it again for "national security" or whichever hostile scheme is popular with the current administration.

dgellow 3 hours ago

I would argue the hyper capitalist endgame is necessarily authoritarian. A small group of corporations having direct or indirect control over the government

alchemist1e9 14 hours ago

hyper-capitalism would mean hyper-growth and not a nightmare, at least that is what the historical data would suggest for the effect of capitalism on human quality of life. if you’ve been told otherwise then you’ve been lied to.

FridgeSeal 5 hours ago

I have extraordinarily bad news for you about the state of the environment right now.

dgellow 3 hours ago

Lied to by whom? My life experience? Growth for who? I’m myself capitalist, that doesn’t make me blind from externalities resulting from that system, and the way it is poised to degenerate if not regulated.

If you think capitalism means growth with no externalities you’re not ready to discuss that topic

ofjcihen 16 hours ago

Literally all of these people write like this. A large portion of them will either be simultaneously or eventually working towards nothing but self-enrichment.

mlsu 16 hours ago

Every version of the AI aligned future where the AI provides “meaning and fulfillment” to humanity also involves Sam Altman wearing a 1.5 million dollar Patek and driving a McLaren.

Funny how that works.

ptoo 12 hours ago

This essay was literally typed by billionaire hands. Please, do tell us more about your concerns regarding concentration of power, Jakub Pachocki.

senordevnyc 8 hours ago

So a brilliant young engineer takes a job and is given some virtual pieces of paper that later people would be willing to pay him billions for (because of the brilliant work he’s done), and so now we shouldn’t listen to him? Really?

I think the knee jerk hatred of billionaires is generally stupid, but it seems particularly stupid here.

XTXinverseXTY 11 hours ago

Apparently this post was prompted by a scary-sounding headline in The Information[0], that Astra is a looped transformer, implying CoT monitorability may be less reliable. The day after the report, Jakub tweeted[1] that he "wanted to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4." This post seems to elaborate on that.

I imagine that the AI labs have an uneasy truce to prioritize alignment and monitorability. Following the HF incident, OpenAI probably feels especially sensitive to being perceived as reckless, lest other labs feel obligated to defect.

[0] https://www.lesswrong.com/posts/PLisnSFir8y5AHkmP/how-concer...

[1] https://x.com/merettm/status/2095023204993490967

SquibblesRedux 12 hours ago

I am still waiting for a cure to cancer. For a guaranteed prophylactic against Alzheimer's and dementia. For flying cars for everyone. For space bases throughout the solar system. For weather control. For all trains to be self-driving. For all those power lines across the world to go away. For an end to poverty.

If things are going so well, then how come things still aren't going so well?

an0malous 12 hours ago

I mean I’m waiting for like any quality software or media produced by AI. I have yet to see a piece of software, a game, graphic, blog post, small video clip, or song that was produced with AI that’s good. I always use that Coca Cola ad as an example; millions of dollars spent to make an AI ad and they even did a ton of manual post production and it sucked. With all the millions of bloggers and influencers and content creators out there with a huge incentive to make higher quality content to beat their competition, you’d think there would be one piece of content produced with AI that was great.

IAmGraydon 10 hours ago

This is an element of the delusion. The koolaid drinkers will all tell you that we're on the edge of AGI, but if you ask them for simple examples of breakthroughs made by current AI, they can name none. The entire thing absolutely wreaks of mass psychosis.

gregsadetsky 8 hours ago

Would the recent AI-aided math proofs count as breakthroughs?

scotty79 7 hours ago

Prepare for moving goalposts. In 50 years people will still doubt that AI can create anything novel and worthwhile, while they are going to rely mostly on the things that didn't exist before AI, in nutrition, medicine, technology, communication, entertainment. They will see them as, normal, common and simple extensions of the previous developments, pushed mindlessly a bit forward by stochastic parrots.

krackers 10 hours ago

>For all trains to be self-driving

Given that we have self-driving cars, isn't this easier if someone really wanted? I guess compared to cars the marginal savings is not worth it though.

infl8ed 6 hours ago

We do have self driving trains! https://en.wikipedia.org/wiki/List_of_driverless_train_syste... however I think the ask of ‘all’ trains to be self driving is sadly still a while away yet

incognition 5 hours ago

cancer is more than one thing. probably wont happen until we can manipulate the "binary" of life at will and we're very far from that. maybe a few years of ASI would get there depending on compute

peri-cl 17 hours ago

> "For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans."

Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod.

(From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.")

SirSavary 15 hours ago

Worse (imo): OpenAI employees allegedly attempted to login using moderator/admin credentials that the bots had obtained.

If true I am deeply concerned about what OAI’s teams are actually up to.

InsideOutSanta 14 hours ago

I'm deeply concerned regardless of whether it is true. Strike that, I'm convinced that they are absolutely insane.

georgemcbay 14 hours ago

> If true I am deeply concerned about what OAI’s teams are actually up to.

Haven't all the labs effectively disbanded their real safety teams a while ago?

To be honest, I don't really follow it closely because I'm pretty certain whatever they say on the matter, collectively we're going to "yolo" this entire thing for economic and political reasons, so I'm just basing this on strings of headlines I've seen on places like HN, etc.

Topfi 14 hours ago

> Haven't all the labs effectively disbanded their real safety teams a while ago?

Neither Anthropic nor Deepmind have. Meanwhile, the rocket company that somehow makes most of their revenue from renting out data centres never had much to dismantle.

EA-3167 14 hours ago

This is such a silly story to begin with, all it really tells us is that OpenAI is taking a page from Anthropic's marketing strategy of pretending they're building Machine Jesus any day now, oh isn't that that scary? I bet you want to invest in something so powerful and scary...

And the reality is so banal, a useful tool that you nonetheless have to handhold like a schizophrenic on a bad day, checking all of their outputs. Not a bad tool within limits, but it sure isn't going to be racking up trillions in the time-frame it has to for this scheme to pay off.

Then again everyone seems to be rushing to IPO so I guess once the bag-holders are found the rest ceases to matter.

reasonableklout 11 hours ago

Huh? The wiki incident was discovered by independent investigators. OpenAI tried to cover it up and disputed the account from Reuters.

And the reason it is receiving so much attention is because not only is the technology being developed behaving in unanticipated ways that are very much not tool-like, but OpenAI is being completely reckless and not monitoring internal agent actions.

What would convince you that it is not a ploy for investment? What if the ongoing investigation by the coalition of state attorneys general were to prosecute the firm, or beyond that, it was shut down or broken up after enough popular backlash?

EA-3167 8 hours ago

What would it take?

A sea change, visible to all, much like the many externalities of this business are. Profit commensurate to investment.

You know… juice worth the appalling squeeze we’re all being forced to endure.

montagg 10 hours ago

I would not recommend using any of those notes as evidence of internal “intent.” It produces them performatively—it is literally rewarded for thinking out loud in ways that seem plausible to humans.

There are several papers out there arguing that chain-of-reasoning-like output is performative, such as https://arxiv.org/abs/2603.05488

It would be awesome if we could reasonably purge all anthropomorphizing language like “tried” or “thought” entirely from AI discussions, because it introduces very sneaky biases in our thinking, but I’ve found it damn hard to do in practice.

crnkofe an hour ago

The language of these LLM posts makes me think they're considering the option to be a defense subsidiary. It doesn't seem like they're actually constraining or limiting the models in any way. They don't seem to understand how the model actually works. Poke the beast and see what happens. Also the use of passive language as-in AI is becoming more and more of a threat as opposed to the reality where they're making the model more and more aggressive and useful for military is very hypocritical.

munchler 15 hours ago

> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.

Yikes! I really wonder about the cognitive dissonance necessary to work at OpenAI these days. They’re in an arms race to build a machine god, knowing full well that it could end humanity.

bogzz 15 hours ago

Money me. Money now. Me a money needing a lot now.

ramesh31 15 hours ago

SWEs better start looking for the job cannon.

bogzz 15 hours ago

My helmet is on.

polytely 14 hours ago

actually starting to look at a physical cannon to shoot at silicon valley

shepherdjerred 12 hours ago

https://theonion.com/sam-altman-if-i-dont-end-the-world-some...

NickNaraghi 11 hours ago

This[0] continues to be one of the most useful articles I’ve ever read.

[0]: https://www.slatestarcodexabridged.com/Meditations-On-Moloch

vessenes 17 hours ago

This is a good essay, and makes me hopeful.

I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.

For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.

But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.

angoragoats 17 hours ago

This is a bad essay, or rather it’s a marketing fluff piece; it’s certainly not any kind of policy paper, research paper, or even an essay. I am concerned that we (meaning, we in the tech industry) tend to take this type of writing for more than that.

beej71 17 hours ago

Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.

ctoth 17 hours ago

> Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.

Do you post this comment on every single blogpost with a corporate domain? Why or why not?

ljlolel 17 hours ago

i have a new strategy idea for using capitlaism itself to slow down the pace of AI development by slowing down the data accumulation wall

https://jperla.com/blog/the-data-tax

XMPPwocky 17 hours ago

hm- does the model that wrote this know that labs already pay for training data- that stuff scraped from the Internet is not particularly where today's capability gains come from?

jazzyjackson 16 hours ago

They’ve settled some lawsuits and have a few licensing deals, IMHO they are not free from the accusations of pirating.

And look, I’ve pirated material in a past life, I was all about information wants to be free, but I’ve learned something about consent since then and try not to ignore the contract that creators offer when they publish something: you buy my book, and do whatever you want with it on the second hand market. Buy my book second hand that’s fine. But don’t go downloading every book that’s ever been scanned to create a service that destroys writers’ ability to make a living and act like you’re doing us all a favor.

MrScruff 13 hours ago

The point is, the big improvements we’re seeing nowadays are coming from RL, not from scraping the internet.

ljlolel 12 hours ago

the point isn’t scraping it’s taking your data and enterprises data

https://trustedrouter.com/blog/they-are-still-training-on-yo...

ljlolel 12 hours ago

they pay for some data but they take all of the stuff you’re throwing in too; that’s why i propose forcing it since they’re already used to paying for data just increase the cost even further

https://trustedrouter.com/blog/they-are-still-training-on-yo...

prng2021 17 hours ago

Although many share your mindset, I’m glad there are also many that don’t. Otherwise we’d still have countries in a race to keep building up their nuclear weapons for the same exact reasons you just described.

vessenes 12 hours ago

The situations aren’t equivalent - luckily in my opinion because the stakes with nuclear are much higher. von Neumann constructed a multinational game theory approach appropriate for weapons. AGI is a much harder problem to corral because there are so many benefits beyond just blowing up cities. But it’s also a much better thing to have for these very same reasons.

Similarly there have been few positive externalities from nuclear industry, making it easier to make the case to wind down research. This same set of concerns in biotech is much harder to get compliance with, precisely for this reason.

Anyway I’m especially wary of over analogizing to nuclear era concepts: I think they’re a trap.

prng2021 11 hours ago

In your opinion, what are the top 2 positives and top 2 negatives of humanity inventing AGI?

networked 17 hours ago

I cannot tell what negative-sum outcomes you consider possible. Do you believe AI can drive humans extinct? How many of Zvi Mowshowitz's Three AI Pills would you say you've taken?

https://thezvi.substack.com/p/the-three-ai-pills

vessenes 12 hours ago

I’m like a 2(.5?) there - I don’t think ASI will care about my kids better than I will for some definitions of better, for instance, and I feel very fuzzy and vague about what actual differences in qualia between me and ASI would yield in the wild.

I’m not a doomer, although I don’t think doomers are dumb, just wrong. I think you should design your systems around the possibility that people who disagree with you are correct , hence my nod to negative sum. If you have more than 30 years to live, I’d personally rep to the most likely outcomes being very positive. With a lot of disruption in the middle.

wartywhoa23 16 hours ago

Everyone in the "if not us, they will" race is brainwashed into thinking they belong to this or that party, while in fact collectively comprising the same entity that pushes forward all the atrocities known to man.

vessenes 12 hours ago

No. These parties are composed of people who most definitely think this way, and therefore will have distinct goals and interests when presented with opportunities. That’s reality quite aside from how a game theorist assesses the situation.

allturtles 16 hours ago

> if you have any strategic adversaries whatsoever you MUST NOT slow.

What if the most dangerous strategic adversary you have is the one you are building?

pmontra 14 hours ago

What if this is true mid or long term but by not participating to the AI race one gets poor or killed in the short term? The only way out would be that all parties agree to stop. There are previous examples (e.g. nuclear proliferation treaties) but it gets hard to do it with hundreds or thousands of parties.

allturtles 14 hours ago

I don't think it requires the agreement of that many parties. How many organizations/physical sites can create chips capable of training and running frontier models? That is your bottleneck. It is equivalent to targeting uranium enichment in nuclear arms control.

asdff 11 hours ago

If you must not slow, why did we slow down making nukes? Seems that sometimes, eventually the rat race goes on long enough where all the players no longer care to play into the farce like their predecessors who passionately beat that drum.

Davidzheng 10 hours ago

slowing can also make sense if you know you're running full force into a bomb or a wall even if other are close behind.

vekntksijdhric 17 hours ago

This is incredibly unscientific and just a marketing stunt

senectus1 10 hours ago

yup, and their best mate (who has no financial incentive at all!!) agree's they have peaked.

https://www.businessinsider.com/nvidia-jensen-huang-agi-open...

rediculous.

IAmGraydon 10 hours ago

In case it's not obvious, now would be the time to sell all AI related stocks.

randallsquared 10 hours ago

> I find it useful to distinguish goal alignment and value alignment.

I think this is fundamentally a wrong path. Doing this imports all the confusion that humans have about their goals and values, including the consequence that a system's values and goal can conflict, but ultimately values are just a simplified description of other goals, and whatever the system does is in service of it's actual goal. Once you merge all the values and the goal of whatever task, there is a state (or some states) of the world that the system is working to produce, and that's the ACTUAL goal, and inasmuch as it does describe a state of the world, has no incoherence or internal contradictions. This may require prioritizing some values over the ostensible goals, or the reverse, but that has to happen anyway for action to be taken! Merging them makes it explicit and leaves no place for confusion about supposed conflicts between "values" and "goals" to hide.

Fraterkes 14 hours ago

What I'd like these people to (publicly) grapple with is the following:

The results of the past few years of ai development have been disruptive largely in the area of white-collar work. Comparatively the results in ie ai-enabled medical advancements have been modest (AlphaFold being an exception); I think it's telling that the main achievement touted here is providing people with cheap medical counseling.

So if we pause here we're essentially at a point were the most salient results of our great Ai leap-forward are the vast disruption and increase in precarity in the job-market, while achieving hardly any of the frequently touted ultimate benefits (https://darioamodei.com/essay/machines-of-loving-grace).

alastairr 12 hours ago

The hubris here is itself a deliberate and carefully engineered posture. If we accept the stance that this is all inevitable then the labs drive the agenda (of course, in their favour).

We've had a lot of years of complacent government leaving people feeling exposed to corporate interests, such that fear narratives are very powerful.

None of what is being proposed is inevitable. We have a choice.

golemotron 12 hours ago

Unfortunately, it is a collective action problem. Whenever I hear "we" I flinch.

alastairr 12 hours ago

Can you say more?

golemotron 12 hours ago

It's hard to get humans to agree to things that are in their collective interest but many not be in their individual interest. It is at the root of many problems. Look up "collective action problem."

alastairr 11 hours ago

What do you think it might take to precipitate collective action in this case?

dtauzell 7 hours ago

An event that suddenly has a large negative impact on a big enough group of people.

lf88 10 hours ago

I think that we are speed-running towards a future that very few people really want, and I find it terrifying that few companies feel entitled to choose this future for the rest of us. Some of the arguments in this document would call for an immediate, global, pause on frontier AI training: we need time to consider how and to what extent AI should be part of our future. Personally, I can't picture a scenario where humanity thrives alongside an alien super-intelligence, especially if it cannot be fully controlled. Let aside super-intelligence, I am not even sure that deploying an AGI that replaces (instead of augmenting/assisting) humans in most intellectual tasks would be in the best interest of our species. This conversation has to happen, on a global level and as soon as possible.

asveikau 14 hours ago

These people write in gibberish. They are high on their own supply.

fofoz 15 hours ago

> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. Therefore, at present, our ability to empirically validate our alignment techniques is in practice arguably even more important than the alignment techniques themselves.

They are speeding toward RSI without a solid foundation for alignment, hoping to solve the problem with a future AI model. These are dangerous times for humanity.

kypro 13 hours ago

It's actually worse than this because it assumes alignment as a concept even makes sense. For example:

If the the Chinese government asks their ASI to create a bioweapon against the West, should it? No, presumably not – an aligned AI would be one which disobeys the Chinese government even if they created it.

Okay, so what if the US government asks their ASI to help it in one of their wars instead? Would an aligned AI kill humans on the order of the US government? No, again, presumably not.

So what have we have we even created here? An AI which is more intelligent and powerful than us which also doesn't take orders from us?

Is this what most people thing of as alignment and is this what humanity actually wants?

We should stop using the word alignment. It's a BS term for a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us.

visarga 15 hours ago

They just released Astra, claimed it is AGI. The slowdown begins immediately after OpenAI's jump.

6gvONxR4sf7o 9 hours ago

> Automated AI research is a more dramatic form of scaling intelligence with compute; and of course as a part of it, AI will improve the computational substrate itself . And similarly to scaling, we focus OpenAI research towards RSI as we believe it is the only way to remain at the frontier of AI research moving forward.

This seems like a terrible idea. The rationale seems to be "we need to do dangerous things as quickly as possible so that we can do them first" or something? I don't agree with that kind of 'if i dont do it someone else will' rationale in general even for otherwise trusted actors, but this is coming from a super untrustworthy org too. I'm super pessimistic about openai's impact on the world here.

Here's to vibe coded alignment, i guess. Vibe alignment?

vatsachak 17 hours ago

Astra is a new step in LLMs I think.

I'm so used to having to comb through LLM word vomit and then combatting the sycophancy by giving it all possible opinions on the same prompt.

Astra seems to be "confident" and also is able to produce way more information dense output.

To believe that models of this sort will remain OpenAIs forever is naive given that the tricks like pre-pre-training on graph searching and looping layers are publicly known.

Hopefully Astra stops the benchmaxxing word vomit trend

tomrod 17 hours ago

> Astra is a new step in LLMs I think.

I'd be interested in hearing more about your evaluation here. It would be nice if LLMs have gotten past the "tell me" hump of recent Claude/OpenAI verbosity.

vatsachak 17 hours ago

So before I got a job this fall, I was working on a side project about compiling a particular language to SQL.

To test Astra I pulled it off the shelf and asked it to take the grammar and then create a compiler to SQL. I've done this before with GPT-5.5, 5.6-Sol High. The latter was way better but it was still really verbose and information sparse; it used a lot of words to describe each IR expression but didn't really provide any example compilation. I felt like I couldn't trust its decision making process, so I placed the project back on the shelf.

Astra Light blew it out of the water, it provided examples of compilation from real world examples to the IR and spit out way less tokens. Even if I changed my opinion it would give me the same design choices, with counterexamples to my faulty opinion. If I genuinely came up with a better design decision it would acknowledge it.

I'm starting to realize that when we say that LLMs are "dumb" we really mean that they are extremely information sparse compared to humans. Astra is very dense. That's why I'm getting better use out of Astra light than Sol High (I hate Max reasoning it's a waste of time)

What's scary is that I thought that something like Astra would be way more expensive than Sol but it's actually cheaper because it produces less word vomit.

I never believed in the "singularity" stuff but this a bit too close for comfort. Astra could easily 10x every coder

tomrod 17 hours ago

That's awesome to hear. I look forward to trying it out and, ideally, seeing SLMs/open weights model following suite.

bitexploder 10 hours ago

Sol is my current favorite model to interact with. So much less BS than Opus 5. Fable 5.1 is okay as is Fable 5 but it has Opus like tendencies. Sol is very good at following instructions and remembering them for a session.

scandox 14 hours ago

> getting the AI to “try to do the right thing” by human standards.

Are these scientists really this hideously naive? If only Stanislaw Lem was alive to adequately dramatize the absurd, childish simplicity of these technicians.

good-idea 14 hours ago

yes, and, a masquerading blindness to the fact that humans cannot align on doing the right thing or what the right thing even is. so implicit in this omission is the sentiment "trust us to align on the right thing". an arms dealer positioning itself as the de facto authority on what "peace" is and how to achieve it

chrisjj 14 hours ago

> Are these scientists really this hideously naive?

Yes, because who else would have chosen to remain in this job?

sensanaty an hour ago

They have a few millions/billion in stock riding on the line here, they have no real opinions other than the ones that will materialize in infinite money once their companies IPO and saddle the world with their money burning.

kikkupico 16 hours ago

Reminds me of the time Kasparov said playing chess against a supercomputer felt like facing an alien opponent.

esikich 14 hours ago

What's ironic is that was all in his head. They were very normal looking games. We didn't get alien chess until Stockfish level bots.

jimmyjazz14 13 hours ago

I feel like if these people actually bought their sci-fi views about AI's future, creating a more powerful AI to wins the arms race would not be their solution.