A new benchmark called Real-SWE is challenging how the AI industry measures coding ability, and the first results are humbling for the frontier labs. Anthropic's Fable 5.1, running inside the Claude Code harness, leads the board with a 38.8% resolution rate on tasks drawn from private, real-world enterprise codebases. No model tested clears 40 percent.
The benchmark, released this month by Specific Labs, evaluates frontier AI models on private production codebases that the team licensed from real companies. Its creators argue that expert-generated or synthetic tasks, however well designed, are not the verbatim work that engineers at real companies actually do. For more context on this story, see our ongoing artificial intelligence updates.
Why Private Codebases Change the Picture
Real-SWE differs from established benchmarks like SWE-bench on two axes, according to its authors: the underlying coding artifact and the specificity of the instruction. Each task comes from a proprietary system whose code and solutions are not available on the public internet, which means agents cannot lean on memorized answers pulled from public repositories.
The tasks also carry business consequences. The benchmark's example tasks include getting billing right, calculating taxes, and migrating customers — changes that affect how a business runs, often across multiple services. Every company has its own conventions and ways of writing code, and agents must understand those norms and make changes that work with what is already there.
"Can a coding agent actually do the work of a software engineer in the real world?" the benchmark's introduction asks. The leaderboard suggests the answer, for now, is: only about a third of the time at best.
The Full Leaderboard
Specific Labs evaluated eight frontier model-and-harness combinations, measuring resolution rate as pass@1 averaged over eight independent runs per task, with 95 percent confidence intervals:
1. Fable 5.1 (Claude Code) — 38.8%
2. GPT-6 Astra (Codex CLI) — 33.8%
3. Gemini 3.8 Flash (Gemini CLI) — 31.2%
4. GLM 5.3 (Claude Code) — 28.8%
5. (tie) Grok 4.6 (Grok Build) — 23.8%
5. (tie) Muse Spark 1.3 (Muse Code) — 23.8%
7. Kimi K3 (Kimi Code) — 18.8%
8. GPT-5.6 Sol (Codex CLI) — 16.2%
The results were discussed extensively on Hacker News over the weekend, where the benchmark drew several hundred upvotes and a lively debate about whether private-codebase evaluation is the right yardstick for agent progress.
A Narrow but Meaningful Spread at the Top
The gap between the top four systems is notable for what it says about the current competitive landscape. Fable 5.1's five-point lead over OpenAI's GPT-6 Astra is meaningful on a benchmark where every task is a real production problem. Gemini 3.8 Flash taking third is striking, since the model is positioned as a fast, low-cost workhorse rather than a flagship reasoning system. Z.ai's GLM 5.3, evaluated through Claude Code, landing within five points of the leader underscores how competitive open-weight models have become on practical engineering work.
Equally telling is the spread at the bottom. GPT-5.6 Sol, an earlier OpenAI release, resolves fewer than half as many tasks as Fable 5.1 — a reminder of how quickly the frontier has moved, and how steep the drop-off can be just one model generation back.
Harnesses Matter as Much as Models
One of the benchmark's methodological choices is drawing attention: rather than evaluating models in isolation, Real-SWE uses native harnesses — Claude Code, Codex CLI, Gemini CLI, Grok Build — to reflect how enterprise engineers actually work in practice. The leaderboard explicitly ranks model-and-harness combinations, acknowledging that scaffolding, tool access and iteration loops are now inseparable from model capability in real deployments.
That framing matters for enterprises choosing systems. The same model can perform very differently depending on the agent harness wrapped around it, and buyers evaluating coding assistants on public benchmark numbers may be getting a distorted picture of how those systems will behave on their own codebases.
What It Means for the Industry
Benchmarks have repeatedly been accused of saturation, with frontier models posting near-perfect scores on public tests that leak into training data. Real-SWE's design tries to counter both problems at once: the code is private and licensed, the tasks are the genuine work products of working engineering teams, and the instructions reflect the messy specificity of real tickets rather than polished problem statements.
The sobering takeaway is the absolute level. Even the best system resolves fewer than four in ten real enterprise tasks on the first attempt, averaged over eight runs. For all the progress in agentic coding, the benchmark suggests that autonomous software engineering on real production systems remains a supervised activity — one where human engineers review, redirect and repair far more often than vendor demos imply.
For enterprises choosing among coding assistants, the benchmark offers a practical checklist: prefer systems evaluated on private code, insist on confidence intervals rather than single-run headline numbers, and weight harness quality as heavily as raw model capability. Real-SWE's first leaderboard will not be the last word — but it is the most direct answer yet to the question every engineering leader is asking about agentic coding.
For more on coding agents, model releases and the research shaping them, follow the latest AI news as the next round of benchmark results lands.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
