Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

ttheanonymousone about 3 hours ago 39 commentsRead Article on withspecific.com

⚡ Community Insights

Discussion Sentiment

57% Positive

Analyzed from 1278 words in the discussion.

Discussion (39 Comments)Read Original on HackerNews

jstummbillig•about 1 hour ago

I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.

WD-42•2 minutes ago

Why would you get emotional over a model? They got you that good?

didgeoridoo•22 minutes ago

Sol failing mostly on “unverified assumptions” and rarely hitting “integration errors” seems about right to me. I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. If you give it a one-liner… you’re going to have a bad (SHA-256-hash-verified) time.

CompoundEyes•28 minutes ago

I do think it’s the wizard not the wand at this point given a decent model. These benchmarks don’t have the wizard.

Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.

howunfortunate•19 minutes ago

In other words, we're still in the era of centaur chess.

lmeyerov•about 1 hour ago

My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.

One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.

bel8•25 minutes ago

I'd love to see these:

- DeepSeek V4.1 Flash

- Kimi K3

- GLM 5.3 (and flash)

- hy4-preview

- Grok 4.6

All of these can be acessed using a $10/mo OpenCode Go subscription.

jwolfe•15 minutes ago

3 of those are already in there.

prometheus1992•25 minutes ago

Does this mean they ended up sharing those private codebases with OAI, Anthropic etc? Also, the ~30% number tracks with my experience. I thought I was going insane for expecting too much from the models but they are still bad, including astra. This morning it messed something pretty trivial while fixing an issue which I was shocked to see. Also2, benchmarks don't mean much these days.

visiondude•about 1 hour ago

this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning scenarios- fable catches “this works for x amount of data but if we run this on y way greater than x amount of data we’ll run into issues). Gemini 3.8 is under appreciated, use Google Stitch to see it in action if you haven’t used Agy yet.

majormajor•19 minutes ago

Do you find Fable significantly better than Opus at avoiding-overengineering? All of my recent testing of Anthropic models seems like they're tuned-to-hell to (a) be much slower than they need to be (running tests over and over during the loop vs at the end, say, even if those tests take a few minutes a pop) and (b) doing exactly that sort of "built a lot of fancy enterprisey feature-adjacent 'stuff'" even before nailing the actual feature. Sol and Terra both have some of the latter but they seem to do the actual work a fair bit faster (this may be a usage-based-priority-tier/rate-limit thing though) which helps offset it.

I think the bigco folks saw all the "it wrote all this code but the tests didn't pass" or "it wrote the feature but it's super brittle" and tuned the newer model+harness combinations incredibly aggressively to try to turn a lazy prompt into "median Enterprise Architecture design suggestions" to bring up the baseline, but in a way that slows you down if you don't want that.

I'm not on big enough subscriptions to want to burn a lot time just evaluating Fable/Astra comparatively until they're cheaper, heh. I can steer any of the cheaper ones just fine anyway.

retrobox•11 minutes ago

Anecdotally, +1. I’d also say this benchmark matches my experiences and how much I trust the model output

ShellfishMeme•about 1 hour ago

Astra constantly does this for me. It goes 90% of the way with some task but then skips the most important part. Then when told to please fix that and do it properly, it suddenly goes down a rabbit hole for 6h and fixes scenarios that aren't even relevant. It's awful at assessing what is important to do and what not, and where to ask for permission and where not.

demibabs•about 1 hour ago

> Each task comes from a private production codebase that we licensed from a real-world company

How does that work?

traceroute66•about 1 hour ago

> How does that work?

My gut feeling is that any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not.

I don't doubt somebody licensed their codebase to them, I just have my doubts about who the "who" could be.

InsideOutSanta•about 1 hour ago

Code isn't worth all that much if you don't own the associated IP, mainly copyright. And even if you disagree with that premise, if you trust that they can keep the code secret, it's basically free money.

At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.

strobe•29 minutes ago

lot of ads everywhere offering to buy your codebase of real product/star up even it long gone or failed (offer usually price per lines of code). So most likely that they have bunch of abandoned codebases between small and medium sizes and probably also some fake codebases as well.

dgellow•about 2 hours ago

A bit of a meta question: what are the most relevant benchmarks by now?

redox99•42 minutes ago

Terminal bench 4 is good largely because it's recent so it hasn't been benchmaxxed yet. It's more of a sysadmin/devops benchmark than a coding benchmark though, but still a decent proxy.

https://artificialanalysis.ai/evaluations/terminalbench-v4-0

tetec1•about 1 hour ago

Epoch.ai has a global score and tracks many benchmarks: https://epoch.ai/benchmarks

andriy_koval•about 1 hour ago

Nvidia and OpenAI claimed AGI, but you still have a job.

jcmontx•about 2 hours ago

I’ve been able to offload most tasks (coding or eles) to Codex since 5.3-codex with extra high thinking

riddlemethat•about 1 hour ago

Astra lets me offload entire projects without worrying about individual tasks…

jeffybefffy519•about 1 hour ago

Do you review the outputs?

traceroute66•about 1 hour ago

So TL;DR benchmarking in a completely non-reproducible manner ?

"Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company".

So basically pinky-promise benchmarking ?

I'm not sure I follow the value here ?

kadoban•about 1 hour ago

If it builds up history and perceived reliability, this type of thing can be valuable. You're giving up transparency for it being harder to game.

traceroute66•about 1 hour ago

> You're giving up transparency for it being harder to game

But then if we take that argument to its natural extreme, surely it means people should take the marketing bullshit published in the 100-page system cards published by Anthropic & co as "valuable" too ?

kadoban•about 1 hour ago

I think you know that's basically nothing like this? The model cards have every incentive to be biased, this doesn't necessarily.

But even so, pretty much yes: companies that actually have reliable and accurate info in their releases get trusted more. It takes time because the default is to disbelieve info from biased sources, but it is possible to trust some of them more than others.

deepwoods•36 minutes ago

In theory, as long as all the models are doing the same thing with the same tools, it's at least useful to see how they stack up against each other right now. It might not be great to track progress over time, as it can get benchmaxxed or the underlying resources may become obsolete.

demibabs•about 1 hour ago

Doesn’t it ultimately have to be this way, to prevent saturation?

bix6•about 1 hour ago

Wake me up when September ends or when I can do this locally.

IshKebab•about 1 hour ago

I think these benchmarks are not that useful, e.g. this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to.

There's only two or three sane options here - you can easily try them all and pick yourself.

coderenegade•16 minutes ago

I switched from Claude to Codex because Claude just doesn't do what you actually tell it to half the time. It dances around the edges and does busy work without actually tackling a tough problem.

I'm not sure what others are doing that they're getting such different results, but I'll take Codex every day of the week.

rovr138•about 1 hour ago

They're not measuring speed nor annoyance. It's there on the page

bdlowery•about 1 hour ago

The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark.

Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change.

EDIT: I was using gemini cli... it's not a harness issue lol

siddbudd•about 1 hour ago

havent tried that model, but it sounds like a potential harness issue. Have you tried it in different harnesses?

tucnak•about 1 hour ago

Hard disagree. I use 3.8 flash in Antigravity a lot, and thoroughly prefer it to most Pro-class models. It's really fast, and I've had it make crazy progress on compiler-like problems that previous models including Opus simply failed at. On ultra plan you can have it going for hours, and make incremental progress with good prompting for review interrupts. It solved a problem I couldn't solve for weeks in under 6 hours. 10k LOC total. The harness and test suite is key.

bdlowery•15 minutes ago

This is my exact experience with the model - https://x.com/ThePrimeagen/status/2095565354726502683

And it just BURNS tokens like crazy.

thereitgoes456•about 1 hour ago

Why so brazenly confident? Isn’t it possible that the benchmark is correct, and your experience is correct too, but you haven’t tried all the thousand different modalities of work that programming encompasses and so maybe you don’t actually have standing to judge?