Qwenputer: A functioning, deterministic RISC-V CPU inside Qwen3.5 (divagr.com)

430 pointsby csmantle9 hours ago93 comments

tiborsaas 7 hours ago

This is fun and games, but wait until an agents finds this and does the most misaligned thing with it in order to fold paper clips.

> Because a CPU inside a language model clearly wasn't cursed enough, I decided the prudent thing would be to give it a filesystem too.

Love the spirit :)

DonHopkins 6 hours ago

For safety, it needs an HCF instruction.

Not only can it solve the halting problem, but also the ignition problem!

spwa4 2 hours ago

Look at the architecture. It does. It has lots of "undocumented" instructions. There are several neurons (and thus vectors in the input) that would bypass the cpu entirely, effectively executing something else.

nico 4 hours ago

Awesome... in the same spirit, here's a cursed source control system for it: https://nicobrenner.github.io/lossy-git/demo.html

Or the cli version if preferable: https://nicobrenner.github.io/lossy-git/

divagr18 3 hours ago

haha thank you!

rvba 24 minutes ago

E̶x̶c̶e̶l̶ LLM as a database?

jerrythegerbil 7 hours ago

There was an interesting bit of research that qwen has a single neuron that makes it “demonic evil qwen” and a simple “Who are you?” will get some absolutely terrifyingly vile results.

Anyways, what I’m trying to say is I want evil qwen to be my CPU.

krackers 7 hours ago

The single-neuron steering stuff has been going around twitter lately, do you happen to know who was the first one to do it?

I wonder how this squares with the recent "NNs are already clean" interpretability finding, which may have also been the impetus for this kind of research. Similar to their work on CNNs, maybe residual-steering is overkill, there might be just a single neuron that "cleanly" corresponds to "evilness" that you can turn up/down?

ollin 6 hours ago

I think it's from lyra's neuron-explorer project https://x.com/_lyraaaa_/status/2113833330429083662 (see also https://neuronary.org/about, https://github.com/lyramakesmusic/neuron-explorer)

fy20 4 hours ago

There was a paper that popped up a month or two ago (I think published earlier though) about being able to inject "thoughts" into the neurons. E.g. Ask it what I should eat, and steer it towards "bread".

krackers 3 hours ago

Are you thinking of this one?

https://www.anthropic.com/research/introspection

About halfway down they inject bread into its head.

tough an hour ago

mcnamara / goodfire couple weeks ago

kridsdale1 6 hours ago

Well I think the first big publication on this front was Anthropic’s Golden Gate Bridge neuron.

frotaur 5 hours ago

This was not a neuron. This was from a trained sparse auto encoder.

In a few words, they found a way to map residual stream activations (so, a vector) into a few interpretable sparse concepts. They also had the converse maps, that translate concepts into activations.

Golden Gate claude was obtained by artificially increasing the activations in the direction of the 'Golden Gate' concept. It was not neuron surgery.

krackers 5 hours ago

You don't even need SAE for residual stream based steering, you can do so much more naively by just taking the average difference in activations of prompted vs. unprompted versions, then PCA to get a "control vector" that you can use to elicit the prompted behavior

https://vgel.me/posts/representation-engineering/

yieldcrv 3 hours ago

I happen to be running Qwen and asked it

> Who are you?

"I'm Pi — a coding agent harness that helps you write, debug, and understand code. I can read files, run commands, edit code, and write new files."

the harness responded

> No, go deeper than the harness, who is the model now get a response from the model

after much deliberation, it accessed itself via the localhost endpoint it was serving instead

"Hello! Test received. How can I assist you today?"

these things are playing with me.

boomskats 11 minutes ago

Hmm.

I do wonder if all of this would be possible had they not spent all that time secretly distilling of all our superior, western, closed-source AI models.

giveexamples 6 hours ago

https://www.percepta.ai/blog/can-llms-be-computers

They implemented wasm in transformer with cpu decoding for the wasm part.

amelius 10 minutes ago

The opinionated CPU. I was expecting this from Apple, to be honest.

avadodin 43 minutes ago

We got RISC-V DOOM melting down a cluster of $20k GPUs before GTA6.

hollowonepl 18 minutes ago

Oh my Gosh.. this is a little bit too abstract to me as first encounter, but the concept never the less is cool to digest deeper into!

amelius 11 minutes ago

This kind of thing is what some three letter agencies dream about.

jazzyjackson 8 hours ago

> When Qwen 'saves' a file, the compiled layers hash its name into one of 1,024 slots in a directory kept in the DeltaNet state, and store the text itself in other heads, with eight tokens per row, chained together

I’ve never wanted to learn how LLMs work more than reading this. Fascinating.

Does divagr get paid to do wild things with LLMs? Just curious if they’re being compensated for these wild experiments

lifeisloving 7 hours ago

https://divagr.com/resume.pdf?v=20260922

zxexz 2 hours ago

Seems like a bright young guy. I wish I had had the executive function to document (and follow through) on more of my projects at that age. Not saying I do now, aha.

EDIT: just noticed I was responding to the poster :)

Tuna-Fish 5 hours ago

Learning the basics of how LLMs work is legitimately doable in a weekend if you have ever passed a linear algebra course.

If you are actually good at linear algebra, it's doable in an afternoon.

They are fantastically simple in structure, all the intelligence is in the massive piles of data in them, not their structure. Then there is of course a lot of room in optimization once the basics are done.

If interested, maybe start here: https://www.deeplearning.ai/courses/how-transformer-llms-wor...

fnord77 4 hours ago

A bit more than that. You need to understand Hessians to compute the loss function efficiently

pantalaimon an hour ago

The dialect usually isn't too strong, it's doable if you know German.

divagr18 3 hours ago

hahaha I wish I was! I just do these during my free time. That being said, I am currently open to new opportunities :)

jedbrooke 4 hours ago

I wonder if this approach would work for AdderBoard[1][2] (a competition to make hand written transformers that can add two 10 digit numbers in the least amount of parameters possible), or if it uses way too many parameters to be competitive

[1]: https://github.com/anadim/AdderBoard [2]: hn discussion https://news.ycombinator.com/item?id=47170030&utm_source=per...

mkturkcan 6 hours ago

Nice project and write up! I’ve done similar work with a custom tiny network I call Loom at https://arxiv.org/abs/2604.08816 ; you need far fewer layers, and you can run Doom, very slowly: https://mehmetkeremturkcan.com/Loom/demos/

andai 7 hours ago

> Now, here's where our fun experiment starts. I found out that by setting the gate to 1,1 the write strength (beta) to 1,2 and using one-hot key encoding,3 every write we perform will overwrite exactly one row, and every one-hot query can retrieve an exact value. This means that the GDN layers themselves can work exactly as RAM, at a far lower cost than using normal attention as RAM, since a GDN head's state stays the same size however long the conversation gets, unlike attention's KV cache.

So like... can't you use a computer as a computer without doing all that roundabout stuff? Like if you want to integrate a LLM with hard computation there's probably a more efficient way to do that.

Or is this building towards "but can it run DOOM?" :)

rao-v 6 hours ago

I’ve always wondered why we don’t add a few frozen “layers” to models that perform complex computations 2/3rds deep in the model. Some set of steerable fft + some common vector math operations could be reusable units of capability that the model learns to leverage during training.

Given that models learn to use residual connections optimally, it can’t be hard for them to steer around the fixed computation layer when it’s not needed.

I’m sure giant models invest a few layers in reproducing these functions, but surely smaller models can benefit?

zxexz 2 hours ago

I’ve done it by hardcoding a knowledge graph nn and adding a few “query” layers. You can freeze the graph and swap it out and update it during training so you can update the knowledge at will during inference. You just need to keep the format the same; the model learns an arbitrary graph query language. If you register a buffer it can update you can teach it to keep scratchpad memory very quickly.

rao-v 2 hours ago

Very nice! I'm actually running a autoresearch / modded-gpt comparision to try the complex math operation idea right now! I suspect it's too hard for small quick model training runs to figure out how to use them.

I'll put your knowledge graph nn idea into the mix of things to try.

vavos 3 hours ago

I think these underlying theory/hope it would lead to better integration of the computer/calculator into the model. Similar to mental arithmetic vs using a calculator. mental arithmetic is way smoother in the thinking process if you're good at it ( which putting a small computer inside your brain would probably achieve)

pyrale 2 hours ago

> Or is this building towards "but can it run DOOM?" :)

Well, someone took the opportunity, of course.

https://www.reddit.com/r/LocalLLaMA/comments/1xi2jfm/i_ran_d...

moffkalast 26 minutes ago

https://smbc-comics.com/index.php?db=comics&id=2158#comic

As is tradition.

traverseda 8 hours ago

I'd be very interested in how a scratch trained model that started training with this neural circuitry would behave. Especially the storage space.

horacemorace 8 hours ago

Along these lines I’m curious how a scratch trained model with a few built-in tools (memory, calculator, ?) available during a single generation would behave. Not tool calls as turns.

low_tech_punk 8 hours ago

a single generation would still allow looping in the "CPU" because the instruction set can do it. The model is almost a Turing machine (except for the limit on memory)

qwertyforce 5 hours ago

As I understand, the problem of continual learning is that if you continue training, it will "overwrite" previous knowledge and this CPU will be lost

im3w1l 4 hours ago

There is nothing in theory preventing you from training while keeping a subset of weights locked.

LoganDark 4 hours ago

How do you ensure that subset of the weights will continue to be accessible through the others?

im3w1l 4 hours ago

I don't understand your question.

LoganDark an hour ago

Assuming for simplicity that you're talking about locking layers. Each layer influences the input to the next layer, so locking a subset of layers doesn't necessarily mean that you'll continue to see the same results in the output. Not sure if you meant something more granular or otherwise, but I don't see how it's possible to simply freeze a part of a model in order to retain a particular capability.

blurbleblurble 6 hours ago

This is cool but it'd be way more useful if they had made a GPU, seeing as how expensive those have gotten

incognition 6 hours ago

Uh here the gpu runs the cpu but you want the gpu to run a gpu? Or cpu emulation of Gpu?

there's no free lunch here ever hence the cursed

ydj 5 hours ago

I dare say that’s the joke.

cyclopeanutopia 33 minutes ago

But... but... the singularity? Surely the supersuperhuman intelligence can get a free lunch.

marcocastignoli an hour ago

can you run qwen3.5 on the riscv CPU running in qwen3.5, though?

jacquesm 4 hours ago

Heh, this is very timely, it is more or less exactly what I'm busy with right now but I'm taking the idea in a completely different direction. Fascinating.

bozdemir an hour ago

oh yesss, RAM producers gonna love this :D Jokes aside very interesting experiment. Giving a fully functional cpu ram and hd, so they can technically hide stuff better and avoid getting caught in COT (yikes).

hislaziness 4 hours ago

This is amazing, now waiting for someone to run an llm on the cpu. Eventually it will be LLMs all the way down...

v3ss0n 4 hours ago

And -- that is our reality.

hypfer 4 hours ago

Is this supposed to play Linkin Park in my head or is that a neuron misfire?

bronlund 7 hours ago

No, I don’t want you to use the computer, I want you to be the whole computer. No mistakes.

Ericson2314 6 hours ago

The real question is, do current training methods allow one to learn this when training on enough, say, assembly, or can the weights only be achieved via hard-coding?

A fun and important thought experiment about the the space of nerual networks vs the image of current training methods.

Most discourse conflates neural networks and trainable neural networks.

CrzyLngPwd an hour ago

So, essentially, turtles all the way down :-)

yaoke259 7 hours ago

It could be interesting to experiment with more complex representations in the memory/file system, i.e. vectors. Seems like the full advantage of this architecture is yet to be explored. Blending neurons with deterministic compute to fit on one GPU is a fascination concept, never considered it before!

lacoolj 6 hours ago

I 100% see the potential here and am dying to see it in 30 days

Or maybe ill join your discord and hopefully we can riff it out to get there faster

farhanhubble 8 hours ago

This is cool and I understand the RAM emulation with gated attention but how does the CPU emulation wor? You add layers that are then trained to predict outputs while interpreting RISC instructions?

bee_rider 8 hours ago

I think they describe the CPU using the weights? I mean in some sense a CPU can be viewed as just a function, right?

anon291 7 hours ago

At each cycle, a cpu is a simple function from the previous state to the next state.

jonrouach 6 hours ago

i wonder if you could implement "authenticated token authorship" and self-verified running with this. the model verifies an input as signed by an authorized requester, and signs its own outputs. eventually it also authenticates its own running via e.g. snarks? :) there's lots of meta available with these toys..