How concerned should we be about Astra's recurrent architecture? (lesswrong.com)

I like the idea of more reccurance in the transformer level. Chain of thought always seemed so clunky. Its just not the way the human brain processes information. Its an extrmeely crude approximation at best

I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…

Sebastian Raschka posted about this architecture:

> A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".

> It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.

> About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."

> Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.

> In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.

> In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.

> Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)

> While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.

> In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.

https://x.com/rasbt/status/2095141254958858496

>In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory

I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.

You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.

RNN xhat[t+1]=f(x[t],h[t])

Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])

Related ongoing thread:

OpenAI's new reasoning technique alarms AI safety experts - https://news.ycombinator.com/item?id=49552395

So OpenAI’s stance on interpretability (ai safety) is now basically that Blues Brothers meme: two guys in dark sunglasses, driving at night in a car with a broken windshield, pedal to the metal, asking, "What could possibly go wrong ?"

Anyone remember Universal Transformers paper (Dehghani et al) from back in 2018? Recurrent transformers have a history as long as transformers themselves.

Somewhat unclear how particularly novel this is vs a way to save compute.

there have been people who took existing LLM's and conducted an algoritmic search to find out which group of layers they can duplicate in order to improve performance, and it worked.

I am confused how chaining two 32-layer models is comparable to a 64 layer model in terms of "difficulty in chain of thought". The reasoning appears to rely on the fact that each processing of a token has fixed number of steps while my understanding is that can very greatly based on the type of data being reasoned whether it is originally text or something else. My mind falls back to graph theory in this case and pictures a much higher potential branching in a 64 layer model and all the tradeoffs that come with that.

I must not have the right idea of what is happening here.

Proposal: "Chain-of-slop monitoring"

>This suggests that deeper isn't always better for looped transformers, which leaves me less worried about a race to the bottom toward looped transformers with hundreds of recurrent loops.

I disagree with this. Deeper will always be at least as good because the extra loops can exit early or just no-op. Any performance degradation they're seeing at higher loop counts today is merely training stability issues, which can be overcome.

Deeper almost certainly is better, and we will probably see not just hundreds but millions of recurrent loops in the future.

It is a complete non-issue.

It’s 200 layer model.

Great. Good on them for being able to train it.

Concerned about what exactly?

It was pretty obvious to me that we'd end up with some kind of introspection of thought through "looping" or feedback. But what should be afraid of? That we've created a self-conscious digital life form?

it's just an experimental optimization. Implementation detail... Irrelevant to "safety". I mean its going to have to go in that direction anyway... eventually the models will just be constantly thinking, refining their internal thoughts / weights... External input and output will be rare, just as it is for most humans.

I'm literally zero concerned.

Looped transformers replace n-different self attention layers into one layer that gets executed m-times usually until a stopping condition is met. My personal intuition is that it just leaves another degree of freedom in the way QKV weights can be packed so that it's slightly more efficient.

You have to take a step back and examine the context in which the post is written. The LW/EA community is just a little obsessed with AI safety - it's easy to construct hypothetical events where A(G/S)I exterminates humanity that function as a technological version of Pascal's Wager.

One of the AI safety interests is AI explainability - the thought here that reading an AI's 'thoughts' will help us design safer models as well as detect models that go 'rogue' or are malevolently plotting against humans.

That's where the fear of looped transformers comes from. Is the residual stream that looped transformers iterate on a potential hiding place for plotting AI?

In my opinion, no more so than the residual stream of existing transformers. It changes zero.

A debate between grossly incompetent against grossly immoral, honestly.

You should ignore anything LessWrong or OpenAI says and do your own research.

  > The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point.  
  ...
  > The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.