Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases (withspecific.com)

> For instance the task naming in the task file starts with an optimistic 1, 2, 3, 5, 5a but then eventually gets to 8a, 8a1, and then ends up with 8b2c2b3 and “8b2c2b2b checkpoint1”. The code that it produced got ever more wild. I don’t want to bore you with what it tried to build, but here are some example pieces of the interpreter changes:

  Hardcoded constants everywhere
  Multiple same-line macro invocations in C
  Random indexes in production code
  Hideous tokenizer code in C

https://lucumr.pocoo.org/2026/9/7/astra-why/

The reasons are highly project specific. The closer your project is to CRUD, the higher the chance of success.

If it works but the design is terrible - is that still success?

Just earlier Opus 5 was supposed to migrate a Python codebase from kafka-python client-per-send to a client singleton. It decided it should add a partition key "that nobody asked for" which it admitted to at the end of the implementation.

It randomly decided going from no explicit partition key which would randomly add to a Kafka partition to partioning key company was the right solution.

In fact, that was fairly counterproductive since this codebase dumps events to the topic per company so you'd get a lumpy distribution where only one partition is ever active at once (starving other consumers)

The fact it decided it should add this without prompting was a bit shocking and I'll probably try to make it explicitly clear in the context not to "do random shit that wasn't asked for"

Well how do you know which statement is truthful? These LLMs confidently say they know things that they don't all the time.

In other words, we're still in the era of centaur chess.

>> I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails.

To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myself.

With Fable I can start with a general ask like "I'm trying to do X, can you investigate and tell me what the shape would look like" and have it poke around and think, ask me questions with single-choice or multiple-choice answers, then break the task into small chunks, each of which becomes a ticket.

With Astra, it's like pulling teeth. It often does not understand what I'm trying to do, takes things literally, does not go above and beyond (i.e. infer intent), and stops way too short of the actual goal. I have to constantly prod it and it's frankly exhausting.

SHA-256-hash-verified sealed package artifact with automatic reconciliation system p95<0.5ms

I had exactly the same thoughts. I often have similar thoughts on other benchmark sites, where supposed performance is way off base from my experience.

I’m not sure what the methodology of these are, but they certainly don’t match what I experience. Maybe I need to look deeper for relevant benchmarks.

i think this likely depends on workflow. for me, the first step is always a plan file artifact on disc, which i heavily review and go back and forth until satisfied. i often have to split the plan into multiple phases because agents are still poor at assessing how large a change will be. sometimes before even starting the plan the task is to create a harness for validation (a way for agent to check their work). the codebase you drop them in also greatly changes how much “over” engineering they do. i think if you are good about reviewing plan files and managing scope for your agent these sort of issues fade away. in general though, gpt models are faster, more token efficient, better technical writers, and yeah i still find Fable 1.0 to be a step up even though it’s slower because it’s actually a tad smarter (and the more important and dynamic the change / feature the more fable 1.0 shows its slight edge over astra)

Code isn't worth all that much if you don't own the associated IP, mainly copyright. And even if you disagree with that premise, if you trust that they can keep the code secret, it's basically free money.

At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.

Do you review the outputs?

Can you show us some of these of projects?

> You're giving up transparency for it being harder to game

But then if we take that argument to its natural extreme, surely it means people should take the marketing bullshit published in the 100-page system cards published by Anthropic & co as "valuable" too ?

very outdated experience from me: when I first tried gemini something, in an existing rust codebase, it looked around for files that would indicate if its go, javascript, java or c++ project, then declared I must have asked it build a new app in javascript and proceeded to circle around to figure out how it can install node and npm on my machine.

So I totally believe that Gemini is just bad. Which is surprising because Gemma is very good for some tasks, but I never ever had any success with Gemini, be it in cli or chat thing or anything else that has gemini branding.

This is my exact experience with the model - https://x.com/ThePrimeagen/status/2095565354726502683

And it just BURNS tokens like crazy.