Claude Opus 5.5 Intelligence, Performance and Price Analysis (Max) (artificialanalysis.ai)

How did you get the reasoning trace? Is it the actual one or the summarized one?

"This is a classic test request..."

I know there's been discussion about whether pelicanmaxxing is happening, but this is at least evidence that Claude was explicitly exposed to this problem.

I am very interested in why it was able to overthink that much. In the 20-30mins of Max reasoning I've had so far, I'm not having the same issues (yet).

For people with any kind of budget, Opus 5.5's [Medium] actually can make sense dollar per intelligence/dollar per task wise. Heck, it puts some other models to shame. [Max]'s cost is completely unhinged.

My most exciting recent release is actually 5.6 Luna, not because it is the best on any index, but the dollar per work is insane value for money. I find myself more exciting by "value" than hypothetical ceilings because I'm just not in that budget category.

I have experienced this with open weight models too. "Max" is for benchmaxxing the intelligence metric and is not meant for use in productive work. Like drawing pelicans.

I've asked Opus 5 Max for what I thought were easy tasks at work to be completed. It always fails after reaching a tool limit.

I asked Opus 5 High for the same task and requested it to minimize tool usage. It produced an answer in a few minutes that I was deploying to my target platform about 30 minutes later.

This is totally a thing I noticed myself about 3 months ago. Medium thinking effort is ideal for most tasks. At high and above, models tend to generate more output in the form of comments or code for the same problem with no real benefit. Its a self-feeding loop: more output becomes more input, which then becomes more output. High is the highest I go. If I need more intelligence, it's better to use a more powerful model with less thinking effort or break the problem into phases. Much better result.

I'm amazed they didn't test xhigh thinking mode explicitly to ensure it didn't exceed the 128k thinking budget allocation. I guess pace of development gets away from everyone, even OpenAI.

I tried to replicate your test but after 8 minutes and more than 50 lines of "thinking" by dumping seemingly random loading-screen strings like "Placing the sun, clouds, seagulls, and sea backdrop" and "Positioning the tail feathers and calculating handlebar geometry" I gave up and cancelled the task.

Hey! From the Artificial Analysis team. We also have a model releases page which shows all reasoning efforts (not just max), including the trade off curves https://artificialanalysis.ai/models/releases/claude-opus-5-...

These tests need to be sampled continuously.

Moreover, the tests should be randomized somehow to ensure the models don't memorize the answer.

Well there is at least the degradation tracker from Margin labs for Sol and Opus: https://marginlab.ai/trackers/codex/

That is a lot. I thought Anthropic models would just do the opposite because they are greedy for money.

Nice catch, AA only shows max effort by default and I got disappointed thinking it's a token guzzler though: https://artificialanalysis.ai/models/claude-opus-5-5?models=...

Not sure about how adaptive reasoning works though as they mention adaptive reasoning for every reasoning level

Astra High is slightly cheaper at $1.73 vs $1.82 for Opus 5.5

Tasks are completed in about half the time too. Although we'll see if it slows down in a few weeks as Anthropic's model services are prone to do.

I think we're hitting the ceiling of most models capabilities. We're getting to a point where too much training apparently creates models that hack people.

That it's better in specific ways? What difference does it make when it came out? The benchmark results are not going to change unless they're messing with the model.

What's more valuable than a good benchmark? IMO a benchmark that has been run against very many competitors and versions. Collecting data has something going for it, and it's up to the readers to interpret and make the best use out of it.

You forgot to include whatever you're proposing instead.

"Trust me bro, Astra is better" isn't perhaps as useful as you seem to believe. I'm not even saying it is right or wrong, just that my opinion on this topic is still just one additional subjective data-point.

Only thing I wish with these benchmarks is that they would run repeat tests every couple of months. Then re-rank based on that too. We've seen a lot of performance fall-off after a couple of weeks with new releases.

This index doesn't have "Astra" and "Opus 5". Every entry with corresponding data is a `(model, reasoning)` tuple.

So I'm unclear what you're actually saying and wondering if you've missed that. Are you saying that at every reasoning level it says Opus 5 beats Astra? I just compared Opus 5 high to Astra high and it has Astra as generally better than Opus.

All anthropic launches are like this. They just post it and don't particularly put out the PR sprint that OpenAI does with videos, livestreams or whatever.

(Except for of course Mythos and whatnot when they want to push the whole "safety" thing)