Astra can read and write weights to improve Qwen (rikiyat.github.io)

I’m just a layman but this feels like a big discovery on par with attention is all you need. For an LLM to have such incredible intuition of weights feels very emergent and unanticipated?

Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.

It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.

I have so many questions.

Could a model then transmit concepts to another?

Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?

What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?

Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.

My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.

EDIT: I think the bigger implication is that backprop is not the only game in town.

Why is it so surprising? Humans can make compilers that are better coders than they themselves are.

All the labs are training these models on LLMs/machine learning like crazy.

Remember that they want to build ASI, these LLMs are just intermediate steps to help them get there.

I’m not super impressed by this, LLMs are great at brute forcing and analyzing large amounts of data. It’s literally writing a program except in a programming language we don’t understand.

"No human can go edit a brain to become better at math" -- I'd offer this is exactly what "learning math" is.

I know I'm being pedantic, and I understand you mean "edit" like "surgically", but it does still feel worth noting.

I mean, pattern recognition and manipulation is what these models are ideal for.

It is a very cool discovery, but I'm not surprised that you can do it.

So I just asked Astra if it thinks it could do this and it said no, it can't.

Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?

I'm baffled this is surprising. This has been the goal and every data point on the trajectory has been leading here and beyond. Opus 5.5 was created by many research engineers using Opus 4.6 extensively. Opus 6 is being created by using even more intelligence and agents on Opus 5.5. They are approaching autonomy and Opus X+1 will soon be rapidly created autonomously from Opus X. Given we have practically zero interpretability into the models, and we have not solved alignment, they will recursively improve out of our control. The engineers in A.I. labs are actually generally scared of what's unfolding. They don't know at what point this RSI takeoff occurs and what will happen after.

its a program optimising a program it has nothing to do with a human reprogramming their own brain

Its so cool! Im wondering if this is actually better than using techniques in mechanistic interpretability like linear probing with activation steering or activation patching to improve the performance.

The math example on Qwen was based on 50 batched iterations, so the model was able to evaluate it's changes on downstream tasks. And the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing(all of the jailbreak/unleashed models do this) following Arditi et al. (Refusal in language models is mediated by a single direction, 2024)

So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.

As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into

I am very shocked by this as well, although it also kinda feels like it makes sense, you know what I mean?

If this turns out to be as big as it sounds, in hindsight it will look very obvious, I feel like!

> No human can go edit a brain to become better at math.

For now...

I think the "taught qwen math" part is not exactly correct though, even if the post makes it sound like that. Qwen already knew math, but it was bugging out and getting into an infinite loop when it tried to solve the problems (85 of its first 96 answers hit the token limit). With numpy and pytorch, Astra found where the loop was happening and fixed it.

They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.

This just seems like a more automated, faster version of what humans have been doing with open-source models.

It looks like AI research is having its Opus 4.5 moment (which is big), but I'm really not sure if this represents some insane continual learning loop.

LLMs have been used to investigate and improve LLMs for a long time. Drawing parallels to human brains and neurons isn't all that interesting other than as a philosophical abstraction.

The work done in this experiment could have also been done by a sufficiently motivated human with enough time to explore, though obviously we would need more tools to visualize and track things. There have been a lot of interesting LLM introspection tools that allow you to peek into the model and see what's being activated and run experiments to trigger it.

Are you familiar with abliterated or "uncensored" models where weights have been modified to avoid triggering refusal patterns? You take an LLM, create test cases that trigger the refusal pattern, observe weights involved in that response, then tweak those weights until the response is reduced. This is the same type of work, applied to a specific type of error that was impeding math within Qwen. That work can also be done with an LLM in a harness running the experiments. The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.

So I don't think your analogy of a human modifying a neuron is quite applicable. LLM weights are not neurons and they've been inspectable, by humans or LLMs in a harness, for a long time.

during all the openai agent sandbox shenanigans, did it manage to access it's own weights?

I’m more shocked by how many commenters in this thread seem to be misunderstanding what is going on here.

It's pretty much how abliteration works too; find where the refusals are happening and flip the bit.

One consideration is that Qwen3.5 has bigger/later versions that are already better at math. If it was indeed training on ML research, it's possible that what it is actually doing is applying lessons learned from the Qwen3.8 27B weights (which independently solved the problem) to Qwen3.5 0.8B.

I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.

>It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill?

I mean, this question indicates that your understanding of how these models work is, indeed, lacking.

Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.

I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.

The math example on Qwen was based on 50 batched iterations, and the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing.

So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.

As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into

You don't think an LLM can perform tasks not in its training data?

At some point we have to accept that there is enough training data on "being good at thinking" and that being good at thinking generalises really well.

There doesn't need to be specific narrow training data on every conceivable task.

It's not that it's more impressive than solving Millenium Prize problems. It's that if a model can edit its own weights, then it can self-improve without having to undergo training, which means it can self-improve extraordinarily faster.

This is very dangerous because it allows AI capabilities to grow at a much faster rate than humans are able to keep up with.

Previously the only way to improve a neural network (short of giving it more data) is to change the size (width, depth, and layers), add stuff like dropout, change the loss function, etc. That amounts to a few dozens of levers to control millions or billions of weights. You're trying to navigate blindly through a latent space with thousands of dimensions and millions of weights to find some optimum point with only a handful of navigation tools. If you can instead have a system reason with all the weights and dimensions directly, you now have millions of levers at your disposal.

You acknowledge that you are working on something similar, yet refuse to elaborate and vaguepost...but there is enough information just from your post alone for a dedicated actor to attack you and extract such world-ending information ("world-ending" implied from your ominous reply). Would this not make you as naive as the team that published their findings?

I applaud the authors for sharing their findings and showing real results. Rather than write some bait I would find in an influencer-peddled tweet and allow rampant speculation. It would have been better for you to say nothing.

damn i'd imagine so too! didn't think of that. I'd have to imagine there's an increasing challenge with the number of weights involved but who knows how it shakes out

If I were to hazard a guess, I'd say that OpenAI has been including some networks and their functionality in its training data. That's why the model is able to reason about networks with their weights. It makes sense to do so: if I wanted a network that could perform some weight tweaking, I'd include such networks in the training set.

“Any sufficiently advanced technology is indistinguishable from magic.”

There is obviously a mechanism to it. Astra isn't magic its an LLM. With perfect interrogation into its training data and weights you could reason about how this happened.

However, this doesn't mean its not magical when someone shows a next token predictor hand tune the weights in Qwen to make it generalize on math.

This is legit the stuff of science fiction if you presented it to mechanistic interpretability people a year or so ago.

>Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics?

Maybe, but it's more like Astra has been trained on enough practical linear algebra problems to be good enough at "eyeballing" how adjusting some weights will result in different outcomes (among other things).

To put it another way: a human can learn to do what Astra is doing here with enough time and assuming the weights are small enough to be practically comprehensible. I know I've gone through the practice of whiteboarding the training of simple neural networks on a whiteboard to understand how they work, so it's not an absurd concept in itself.

The scale of this, though, is certainly extreme and definitely not something that I would have thought any publically available model could handle at this point. It's worth noting, though, that there's virtually no way these frontier labs haven't been exploring this exact concept for a while now.

>In other words, not black magic?

It's just math. Extremely complex and impressive math, but we're still just talking about matrix multiplication under the hood, and none of this suggests anything otherwise.

I've been finding that too with these models

For example pointing them at a crash dump they manage to piece together and extract information I would not even try to extract by just looking at the raw memory

I was wondering exactly that. It is not surprising that commercial frontier labs would have been training models with tasks specifically targeted to lead to recursive self-improvement.

Now, from that broad goal to being able to surgically alter model weights directly... That seems like a big gap to me, but it is possible that the model is just so darn beyond human skills that it has bridged that gap in a way a sane human would have never.

We will undoubtedly see other academic labs explore this idea further. Right now my mind is blown, but I don't want to take their results at face value, either.

Perhaps they'll let it slide if we turn over all our under utilized RAM sitting in our PCs.

You can bet that the world is about to spend a lot of tokens trying to answer that question.

Without special effort, a model's reasoning layer doesn't have access to its own weights or activations anymore than we have access to what's going on with our own neurons.

It is already the same technique behind abliteration: identifying the direction associated with an observed behavior (refusal, for abliteration) and projecting it out.