Neural networks are clean (already-clean.goodfire.com)
429 pointsby themgt6 hours ago151 comments
sebmellen 5 hours ago
Nick is a force of nature. Highly recommend following him on X if you don't already.
His launch post gives good context not found in the blog:
> interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
> with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
devy 2 hours ago
Not sure if there is formal arXiv paper to be released other than this long form Goodfire AI blog post (love the animations!), but if what Nick demonstrated/observed were indeed true, this will resolve the Super intelligence alignment issues that p(doom)-ers have been crying aloud in the past few years.
The most iconic line was the last sentence:
a bitter lesson for interpretability: the simple thing, at absurd scale, works
Also, I suspect what Nick hinted at, might be the discovery of earliest RSI work.
with modern agentic tools and a much closer look, the picture looks totally different.mentalgear 2 hours ago
> with modern agentic tools and a much closer look, the picture looks totally different.
Are those the agentic tools the LLM developed for you? How can you be sure they can be trusted and that you even understand them correctly and are not biased towards making you feel like they are trustworthy? This is exactly the AI2027 scenario where researchers are fooled by their own super AI telling them what they want to hear ("our model solved alignment!") while successfully hiding its true intention.
Also Yudkowsky's Cat-Belling would like a serious word with Nick. (https://www.lesswrong.com/posts/SwYBLQvo8MddDcCwz/cat-bellin...)
Animats an hour ago
> The vast majority in models up to trillions of parameters are legible, and their circuits beautiful.
That's very encouraging. For quite a while, neural nets were considered an incomprehensible black box. That paper shows they're not. Real progress.
QuadmasterXLII 18 minutes ago
Important context is getting flagged out of the replies to this comment. An unfortunate state of affairs.
bluejay2387 4 hours ago
The claims seem to be an overreach of what is actually proven. We've known about progressive feature complexity and the impacts of early training structures on later training for a really long time (since CNN's, The Lottery Ticket Hypothesis, etc...) and neither of those phenomenon are mutually exclusive with parameter overloading. The Multimodal Gap work (https://arxiv.org/abs/2203.02053) seems to imply networks have a tendency towards condensed representations that are clearly not semantic and a bunch of other considerations are already on the table for discussion. I don't think these have all been considered sufficiently to start claiming we are around the corner from mechanistic interpretation.
qt31415926 3 hours ago
Nick isn't trying to re-claim progressive feature complexity. He's saying that many polysemantic examples were misread because we assumed neurons packed unrelated things together. But also I think what he's claiming if you read the full blog post isn't inconsistent with that paper, where that paper actually isn't itself claiming semantic condensing but geometric condensing. That paper doesn't analyze neuronal firing patterns whereas Nick's trying to analyze what fires together and what doesn't fire.
QuadmasterXLII 9 minutes ago
this article seems to take it on faith that neurons are aeasily findable, even if their meaning is tricky to untangle: it would be sharply improved by description of how the neurons are being isolated before the meat of the article, the naming of the neurons, is performed. In particular axis aligned semantic directions are incompatible with the classic superposition description, where the meaningful directions are nearly orthogonal, allowing many more to be packed onto the sphere than it has dimensions. (i’m also aware of work showing that Adam or relu pushes semantics onto the axes, but it’s not clear that that’s what he thinks is happening)
lukewarm707 2 hours ago
It reminds me of this quote from Judith Butler:
"The move from a structuralist account in which capital is understood to structure social relations in relatively homologous ways to a view of hegemony in which power relations are subject to repetition, convergence, and rearticulation brought the question of temporality into the thinking of structure, and marked a shift from a form of Althusserian theory that takes structural totalities as theoretical objects to one in which the insights into the contingent possibility of structure inaugurate a renewed conception of hegemony as bound up with the contingent sites and strategies of the rearticulation of power."
NaiveEmpiricist an hour ago
Butlerian Jihad?
Animats an hour ago
Why? The neural network paper is straightforward and has lots of illustrations showing concrete examples of what happens as images move through the stages of a classifier.
The above quote is from "The world's worst writing".
Drupon 26 minutes ago
This is really not that complicated a quote anyway. It is packed full of jargon, but she's talking about moving from a structuralist mindset to one that views power structures as formed contingently from their society rather than structurally from capital.
Not surprising that this philosophical philistinism is from an Anglo publication.
astrange 8 minutes ago
Continental philosophers do frequently write complete nonsense (often intentionally), but this isn't hard to follow.
> Althusserian theory
though I think using this term is rather un-feminist because of the part where he killed his wife, but basically continental philosophers will forgive you for anything as long as you're a French communist.
aesthesia 2 hours ago
The tone reminds me a little of the enthusiasm from the Anthropic mechinterp team around the time of the first large-scale SAE results. A kind of hypomanic energy around the idea that we've discovered the key paradigm necessary to understand models, and all that's left is to scale it up.
ambicapter 2 hours ago
A recurring motif among the LLM-pilled.
ks2048 5 hours ago
To cut through some of all this text - here's a link he calls "the first non-trivial neural network I feel I mostly understand:
https://already-clean.goodfire.com/microscope/medium.html?la...
measurablefunc 5 hours ago
Understanding algorithms beyond a certain complexity is a pipe dream & neural networks are algorithms. The theory says it is not possible & the practice is aligned with the theory. All non-trivial properties of neural networks are undecidable.
D-Machine 4 hours ago
What theory says this? Surely not a proven or applicable one, as this post makes it very clear a great deal of understanding is in fact possible, even with highly complex neural networks.
cscheid 4 hours ago
> All non-trivial properties of neural networks are undecidable.
hey, I don't know how to say this less directly, but: you're talking a lot about Rice's theorem in your comments, and I think you're using it way too broadly (look, I get it. It's a wild, mindblowing theorem). But it only applies when you take the claim in the piece to be much more general than it has to be, and when you take the narrow claim of the paper (current specific LLMs can be fruitfully analyzed with the tools we have), Rice's theorem just isn't at play.
The math analog is: you're right that the Halting problem says there are many unprovable things, but Lean + mathlib are still incredibly useful, and the halting theorem doesn't actually stop that.
> neural networks are algorithms
Again, that invites an interpretation that is too broad. Very concretely: any one neural network is a _single finite network that you can write as a single formula_. (Another analogy: we can't have an algorithm that solves the general quintic, but we can prove useful things about many specific quintics.)
measurablefunc 20 minutes ago
That's good to know. It's very useful information.
jimbokun 4 hours ago
Humans are undecidable too but we keep studying them.
za_creature 3 hours ago
> neural networks are algorithms
... are they?
Algorithms are predictable:
1. it's really difficult to produce true randomness in an algorithm, which is why there's a whole area of study surrounding the qualities of pseudo-randomness, specifically with regards to the generation of cryptographic material with most conclusions being that we should draw entropy from the environment (e.g. electrical noise as cloudflare does - the lava lamp is a distraction)
2. when algorithms are not predictable despite being mathematically sound, this is indicative of a problem nowadays called "numerical stability" which again spawned its own area of study.
CGoL shows how really simple rules (couple of bytes of data) can lead to extremely complex outcomes and you have the BB turing-golf aspect as proof of the worst case scenario.
And with current models being measured in hundreds of gigabytes, with frontier labs deprecating then removing the "temperature" parameter (which never worked to begin with, temp=0 did not result in reproducible results), where exactly is the confidence of this 10^10 prediction coming from?
antonvs an hour ago
Yes, (digital) neural networks are algorithms.
You probably know that non-determinism in real-world inference comes from deliberately stochastic features like temperature, from implementation choices like distributed and parallel computation, and numerical effects like floating point precision.
The fact that a neural network's execution can be nondeterministic doesn't mean that the network stops being an algorithm.
The point you're making boils down to speculation that the accidental non-determinism involved in real-world inference is significant enough to undermine or negate interpretability findings based on the algorithms. That would need to be demonstrated.
The CGoL and BB examples demonstrate what might be possible in a worst-case scenario, but they're only analogies in this context. It's far from clear that real models suffer from such effects to a significant degree. You might actually be better off talking about the butterfly effect!
zygentoma 3 hours ago
I think this would be the better link for this
fxwin 3 hours ago
I have no idea what I'm looking at here
juanjmanfredi 2 hours ago
Yudkowsky seems unconvinced.
ok_dad an hour ago
Let him come here and comment, then. I’m amazed this rape cult leader immersed in his fantasy world still gets traction amongst HN.
fluxusars an hour ago
By 'rape cult' I assume you're referring to the zizians? https://en.wikipedia.org/wiki/Zizians - the internet is convinced Yudkowsky has nothing to do with them, so it's pretty disingenuous of you to claim otherwise without citing sources.
grungegun 27 minutes ago
That's because he's worried about super-intelligence. If LLM's are a normalish tech this is a good step
giveaccountpls 25 minutes ago
What are his actual credentials? What did he achieve? Isn't he just some random dude on the internet? Why do people care what he says?
voganmother42 17 minutes ago
have you read his harry potter fan fiction?
tgarrett 10 minutes ago
He tells spooky stories on the internet to make money. He has told numerous spooky stories. Yes. Spooky stories grab the attention of many people (understandably, given all the wars and plagues and famines in previous centuries).
x312 42 minutes ago
A TLDR, since most comments here seem to misunderstand the result.
The idea is: a model may only have 100M neurons or whatever. But there are infinite concepts in the world. Therefore, models have to "stuff" multiple concepts per neuron.
How do you do that? Theres two ways:
1. Each neuron handles similar concepts. One neuron for dogs, a separate neuron one for cars, etc.
2. Each neuron handles a bundle of unrelated concepts. One neuron handles dogs and cars. Another neuron handles cats and numbers. And so on.
Historically, people thought #2 was correct because many neurons were found that seemingly represented multiple concepts. #2 had a cool name (polysemanticity) and was promoted in OpenAI/Anthropic blogs so it had a lot of traction.
However, this blog says #2 is wrong. Actually, the dog/car neuron is just a "V shape neuron". So #2 was just methodological error, and #1 was right all along. This is all the blog is saying.
tuvix 5 hours ago
This is the primary reason, to me, that open weights are so important. If OpenAI and Anthropic want these models to be as autonomous as possible they also need to provide the means to understand them fully.
antics 3 hours ago
So we have here a massive evidentiary brief, the point of which is to argue that it is easier than we thought to, like, "sequence the AI genome" (in my estimation). We revisit famously-messy neurons, examine various tool failures, check places where messy things still hide, and catalog how "polysemanticity" is represented variously in the network.
Once sequenced, the thinking goes, we'll be able to do things like: explain and anticipate failures; predict what happens when you modulate the pathway an input takes on its way to becoming output; reconstruct algorithms the models have learned.
I'm not 100% sure this will work out like the mechinterp people think. If the neural representations and pathways are "clean" (organized into a a meaningful-to-humans hierarchy), it seems at least very plausible that the behaviors that AI xrisk people are worried about will mostly boil down to straightforward synthesis (i.e., mostly-linear-interpolation) of mostly-well-cataloged knowledge. And if that is true, then it does not seem hopeful that there would be a way to prevent the output they are worried about, other than just removing a bunch of the knowledge. So maybe there is no alignment solution at all.
benlivengood 2 hours ago
What I envision is the ability to find the circuits and neurons evaluating morality and ethics and being able to figure out how they work at the boundaries and ensure safe deferral to human judgement or stopping outside of in-distribution behavior, which further is vetted against actual human values.
antics an hour ago
I don't see a lot of hope for that, because it's not a well-specifiable quantity. I would say we currently have no real evidence that you can even do this even for concepts that are much more well-specified like, say, dogs. Maybe you can! But that is not affirmatively known, and nothing we have is on track to supporting that, right now.
jacquesm an hour ago
That sounds wishful rather than factual.
stratos123 5 hours ago
I'm not sure I buy the author's arguments for mechanistic interpretability being easy, but it sure is a well-made and very interesting post. Would be better without the completely superfluous laggy 3d city model, though.
smokel an hour ago
Isn't it ironic that an article on AI explainability is itself so hard to understand?
lhd1 28 minutes ago
It would help if this was also written up in the conventional way as a scientific paper (Intro, methods, conclusions, etc). These discursive blog posts are getting exhausting.
bob1029 6 hours ago
I think certain classes of artificial neural networks are definitely more interpretable than others.
For example, a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
The degree to which a model can be quantized without losing its performance also likely contributes to interpretability. The redundancy can help humans lock onto patterns. If there is no redundancy at all (everything is a surprise in information theoretical terms), then the stream of parameters will look indistinguishable from cosmic background radiation.
jacquesm 5 hours ago
> a trillion-parameter transformer model built using many layers is likely significantly easier to read than one that only has a few million parameters but was constructed using neuroevolution over one gigantic layer.
Why do you believe this?
D-Machine 5 hours ago
The posted article argues this many times...
jacquesm 4 hours ago
Yes, but it is the same person making the same argument many times. There have been large multitudes of articles by others who are arguing the opposite. It could well be true. But it would take a lot more than just one over-wordy, distraction imaged, clearly co-written by AI article to convince me of that.
D-Machine 4 hours ago
GP and the article both make clear arguments for WHY they believe this. You just asked a lazy drive-by question without providing any reasoning at all for your position. If you expect a good response, you need to provide something meaningful to respond to, as it is it just looks like you are providing knee-jerk skepticism without having engaged with the material.
> There have been large multitudes of articles by others who are arguing the opposite
Also, Nick's entire article is framed such that, yes, plenty of others have argued this, and even HE used to believe the opposite, but now does not. So your point about other articles here seems to me to be a sort of non-sequitur.
jacquesm 4 hours ago
I just - politely - asked you a simple question because your statement made me curious. I have also posted a longer explanation of my own thinking on this in another part of this thread.
D-Machine 4 hours ago
Yes, that comment is much better and appreciated. I agree post-hoc description is not the same thing as mechanistic explainability at all. The article is indeed flawed in sort of muddying the distinction between these two.
EDIT: Actually the article makes the distinction at one point, but the emotional / hopeful thrust of the article relies heavily on an assumption that understandable post-hoc descriptions of e.g. neurons and circuits (model components) are going to help lead to real mechanistic explanations and more specific model design.
This has not so clearly been the case in other fields (e.g. neuroscience, human psychology, economics, etc), because the problem of emergentism / the whole generally being much more than the sum of its parts / components.
jacquesm 3 hours ago
Thank you.
kmeisthax 4 hours ago
羊.
This is a Chinese character that can refer to goats, rams, and several other kinds of "medium-sized animal with thick wooly coats and horns". English has no direct equivalent; we have to be too specific when we talk about these animals.
A model with a small number of neurons has to compact its concepts in order to fit within the parameter count while reducing loss. OP's article shows that the compaction is usually generalization: "goat" and "ram" get compressed into "羊". The canonical example of superposition, an image model where "car" feeds into a bunch of "dog" features, was shown to be actually generalization. In a sense, the model was so small that its optimizer didn't have the parameter budget to distinguish between "car" and "dog"; it has a single set of "thing that moves fast" features.
So basically, really small models are merging features into semantic buckets that are so generalized they don't make sense to big human brains unless we really, really squint. The buckets chosen happen to be what has the lowest loss, of course.
gavinray 2 hours ago
Possibly the best and most succinct explanation in the thread thus far.
jacquesm 2 hours ago
Ok, so assuming that this all holds: you could then create a model that is substantially larger than the ones that we have now and at some point the model would become sparse: there would be ample space to accommodate all of the information in the input without weights ever carrying more than one piece of information. The degree to which you get overlap is then a function of the ratio between the size of the model and the size of the input and this is pretty much in line with the conventional way of looking at models (unless I'm grossly mistaken). So by reducing the size of the model relative to the size of the input you force the model to co-locate multiple pieces of information to the point that they start to resemble each other more and more.
But from my earliest experiments with this stuff that's not really what happens. Yes, you get 'feature like' regions in the models. But those feature like regions, even though they encode something particular do not encode concepts the way we would normally use them and I fail to see how this pathway would lead to an explainable model in the sense that would could use something simpler than the model itself to predict its output with a high degree of fidelity.
bob1029 4 hours ago
Neuroevolution is a discontinuous way to go about building the network. You are evolving the topology itself from zero. You don't know how many parameters you will have until the training is complete (if it ever does). With a transformer model you start with a known-good topology and can calculate how long a training run will take.
jacquesm 4 hours ago
Yes. In that sense it is equivalent to a compression method that uses K-means clustering to do the partitioning. But because there are many more bits of knowledge out there than there are bits in the models you end up with this amorphous smear. I'm sure some of those bits carry more information about some of the concepts but in general - and that's the way it is explained in just about all the papers I've read that I can still comprehend - every weight contributes to the output in a dense model. In an MOE things get more complicated but there within every 'MOE' column the same principle applies.
I'm fine with taking this article at the high level conceptual level, that's effectively the kind of thinking that led to MOE to begin with. But I don't see that as a pathway to being able to 'explain' a model. Because if you could actually explain a model you could engineer the weights, you wouldn't need any training because it implies that you can improve on the model by simply performing surgery on the weights to get it to do what it does but better. Now, obviously, unlike the author I'm a complete nobody but for the last two years or so I've been trying very hard to understand what is going on under the hood.
It's akin to feature engineering - or rather, the lack thereof - and to see a strong suggestion that we maybe be able to reverse that is interesting, but absent hard proof at how a particular model reached a particular conclusion it is difficult to give that a lot of weight right now. It is possible that such proof is buried somewhere in TFA and I overlooked it during reading. But I think this is speculative rather than that it proves anything and before I'm going to be convinced I would like a little more evidence.
TeMPOraL an hour ago
Weights are probably the wrong place to look. If there's a structure to be found, it's in the latent space itself.
Crude analogy: a Fourier transform of a photo looks like white noise with some abstract geometric patterns on top. You can't see the photo in it. But if you apply the same transform again, suddenly you get the photo back. If you try altering the DFT image, you can affect the photo's appearance everywhere, all at once, but it's hard to fundamentally change what it shows. You quickly discover that all pixels of an image contribute to every pixel of the DFT, and vice versa. There's a well-defined connection between the two, and it allows for some useful engineering, but in sense of understanding what's on the image, it's about the maximally unhelpful representation there is.
Point being, there obviously is some kind of structure in the way LLMs represent their training data, so if all we see is noise, perhaps we're looking at a bad representation of it.
2001zhaozhao 4 hours ago
I only skimmed through this post. But I think this is a crux:
> The more we read [the neural network] the way its own readers do, the more interpretable I think it will look. We should be wary of reading it too much like a human, and concluding it’s messy just because it doesn’t fit our preconceived concepts.
cosmojg 3 hours ago
The anthropomorphization of artificial neural networks has been a problem for as long as they've been called "neural" networks. Not that I can complain about it, though. I got my first job as a machine learning engineer coming from academic neuroscience by putting "biological neural networks" on my resume.
adsharma an hour ago
This line of work is a mirage. You're much better off investing in a project like latentpedia.org and building a new class of LLMs which support dual use. Deterministic and less compressed or probabilistic and fit in GPU memory.
yurimo 5 hours ago
I have yet to fully finish going trough this but I would not rush to say that superposition is completely wrong. I understand that the message is instead of first learning a new representation (e.g. SAEs/transcoders) and then interpreting those learned features, you try to understand the neurons and circuits that already exist in the model.
This approach is part of a way some researchers in the field approach interp, (me including) because I am personally cautious of developing some new representation model that claims to be the way we should represent the network. It is another layer of abstraction that takes us away from core model components, very often this requires training a separate model that was not on its own a part of original model organism during training (SAE).
I would argue that models will likely be further increasing in complexity, not just in size, when model trains continuously and becomes adaptive yet parameter count does not change, it is not hard to imagine that neurons may reorganize themselves to pack more information.
NaiveEmpiricist an hour ago
I'd be interested to see where this goes, but it seems to rely on LLMs being mostly or entirely compositional (i.e. ultimately, they can be taken apart to be understood). But if LLMs have notable emergent properties - which seems likely - then those parts of the LLM will be effectively off-limits to this sort of interpretation.
And emergence doesn't need to involve much of the LLM's weights:
> In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood
If those ~2-4% are the central hubs of emergent cognition, then understanding 96%-98% of the weights may be less valuable than the author thinks.
bubbluppugus 2 hours ago
Interpretation comes down to ultimately arbitrary human assignment of a succinct and compressed syntax and semantics to summarize a big data set; so yeah eventually something new to describe models will emerge. It's how it goes.
Also how it goes is this new consensus will be a leaky abstraction and we'll keep conjuring new words to describe new rhetorical features and bloat the corpus of philosophy until everyone is over the whole concept and then we'll start over with a whole new arbitrary academic philosophy for "running algebra on a computer"
daemonk 3 hours ago
Seems like the TLDR is that the neural network is not randomly placing things together due to constraint, it is actually finding some commonality from the input dataset. We just need to spend more effort trying to trace it.
Perhaps our anthropomorphic understanding of data is biasing what objective features there actually are in the dataset.
I wonder if we can study the graph similarities of various models, do we see a convergence of structures that encapsulate certain concepts. And would that convergence be due to just how humans have narrowly defined the concept in our language, or is there something inherent to the concept that allows it to converge. If we compare the same concept in different languages (ie. chinese vs english), does this convergence disappear?
pishpash 2 hours ago
Why would it 'randomly' place things anyway. If there is ever a space constraint the most space-saving way is to group into highly generalizable concepts.
scrlk 2 hours ago
If you view LLMs as lossy data compressors, this exactly what you'd expect.
TeMPOraL 2 hours ago
If your view is anthropomorphic, i.e. LLMs internal form being structurally equivalent to ours at a basic level, this is also exactly what you'd expect.
pseudologos_ 2 hours ago
As a lay-person I'm really confused by this. I thought it was obvious that these networks were already picking up on shared features such as 'being V-shaped'. Isn't that why older and smaller models were getting confused between objects? What am I missing.
root_axis 2 hours ago
> When I retrain its original architecture, which still produces messy deep layers, and follow the least legible neurons I can find, they turn out to be overgrowing locally into reasonable, adjacent territory.
It's interesting to observe that human reasoning is still the barometer of what "reasonably adjacent" means in the context of a NN's structure.
mariano54 an hour ago
This is amazing. Feels like it will be the start of a new era for interpretability research
LarsDu88 4 hours ago
Intuitively this makes sense. When you have small models, they need to compress more by leveraging superposition and polysemanticity.
Nick is looking at a 32B model, which certainly has to compress less... Now we have 10 trillion parameter models... do they need to compress at all? As far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth can can tell you at a glance whether your standing in the ocean or not based on your coordinates
bschne 4 hours ago
where's this specific example/experiment from? curious to read more, never heard of this
teruakohatu 4 hours ago
> far as I can tell, experiments show that Fable has basically memorized every lat/long position on the map of the earth … your standing in the ocean or not based on your coordinates
Memorising a small shape file of the earth’s continents can allow you to determine if a person is on land or in water.
jimbokun 4 hours ago
Islands?
semiquaver 3 hours ago
…can be included in a shape file while it remains “small”
soulofmischief 4 hours ago
All functional knowledge is compression and contextual decompression. Sometimes this compression can be lossless or nearly so. Compression doesn't automatically mean loss of information.
euleriancon 3 hours ago
The 'Exploring Image Models' section addresses that specifically, particularly the 'What Neurons Do Under Pressure' subsection.
'If it switched to superposition, we should see overlapping mixtures on top of each other, ... But that’s not what we see. The neurons stay interpretable and generalized, but coarser'
bilsbie 2 hours ago
On a practical level when could I take an open weight LLM and get a print out of what each parameter does?
baq 2 hours ago
I've read shorter books. Great piece, at least the parts that I managed to read until I ran out of time. Interactive teaching aids, too.
qbit42 3 hours ago
Is the use of "generalization" like this standard in mech interpretability?
Seems undesirable given the typical definition in ML.