A new benchmark from application-security firm Semgrep has delivered an unexpected result: an open-weight model from China's Zhipu AI outperformed Anthropic's Claude at finding real security flaws in software. The finding adds fuel to a debate over whether the most capable AI is necessarily the most expensive or the most restricted.
With Anthropic's powerful Mythos model locked behind a U.S. export ban, security teams hunting for an accessible alternative are turning to open-weight options like GLM 5.2 for the latest AI industry coverage. Semgrep's test, published June 22, suggests that hunt may be paying off sooner than expected.
What Semgrep Tested
Semgrep evaluated a lineup of open-weight and frontier models on its internal IDOR-detection benchmark. IDORs — Insecure Direct Object References — are access-control bugs that let one user reach another user's data. They are notoriously hard to catch with traditional static analysis because the flaw is logical rather than syntactic: the code is technically valid, it simply lacks an authorization check.
The company has been refining a workflow for spotting these vulnerabilities by combining its rule-based engine with AI reasoning. To isolate how much performance comes from the model itself versus the scaffolding around it, researchers ran a set of popular open-weight models through a minimal harness — a simple Pydantic setup using the same IDOR prompt given to every model, with no endpoint discovery and no guided navigation. The prompt offered only a basic search strategy and a few pointers on what an IDOR looks like — a little more than "here is the code, find the bugs," but far less than what the frontier coding agents receive when running inside Semgrep's full pipeline.
IDORs are a persistent headache for application security teams because they fall through the cracks of conventional scanners. A rule-based tool can flag a missing input check, but understanding whether a specific endpoint actually exposes another user's data requires reasoning about the application's business logic — exactly the kind of contextual judgment that large language models are increasingly good at.
GLM 5.2 Takes the Lead
The standout was GLM 5.2, Zhipu AI's open-weight model. Running nothing but a bare prompt, it scored a 39% F1 on IDOR detection — beating Claude Code's 32% by seven full points. According to Semgrep, the open-weight model also surpassed Claude Opus 4.8 on the task, making it the strongest non-frontier option tested.
The economics were equally striking. At GLM 5.2's pricing, the run cost roughly $0.17 per vulnerability found. For organizations that might scan thousands of endpoints, Semgrep noted that per-bug cost is often the deciding factor in whether a detection technique is practical at scale.
Other open-weight contenders trailed further behind. MiniMax M3 scored 23% F1 and Kimi K2.7 Code landed at 22%, both clustering well below Claude Code. Semgrep characterized GLM 5.2 as the clear exception rather than a representative result for the open-weight category.
The Harness Still Matters
Despite the open-weight win in the raw-prompt category, Semgrep's own purpose-built pipeline remained the overall leader. The company's multimodal setup — which combines AI reasoning with rule-based detection and structured endpoint enumeration — achieved 53% to 61% F1, depending on configuration. That gap underscores the researchers' original question: how much of vulnerability-detection performance is the model, and how much is the architecture around it?
The answer appears to be "both, but more than people assume is the harness." GLM 5.2 proved that a capable model can do meaningful work with minimal support, yet it still could not match a system engineered specifically for the problem.
The Semgrep team — which included researchers Paxton-Fear, Seth Jaksik, Brenden Noblitt, and Erik Buchanan — said they were "genuinely shocked" that an open-weight model running a bare prompt surpassed a frontier coding agent on a reasoning-heavy security task.
Mythos at Home
The benchmark carries added weight against the current geopolitical backdrop. The blog post's title — "We Have Mythos at Home" — is a pointed reference to the fact that Anthropic's cybersecurity-focused Mythos model is effectively unavailable to much of the world. After the Trump administration barred non-U.S. nationals from using Mythos and its restricted sibling Fable 5, global security teams lost access to what was widely considered the most capable AI for offensive and defensive security work.
In that vacuum, accessible open-weight models are filling the gap. The fact that GLM 5.2 — freely downloadable and deployable anywhere — can already match or beat frontier proprietary models on specific security tasks suggests the export ban's chilling effect may be smaller than intended. If the best "unrestricted" model keeps improving, the strategic value of hoarding frontier capabilities diminishes.
For now, the result is narrow: a single benchmark on a single class of vulnerability. But it is a signal that the open-weight frontier is closing faster than many assumed, and that accessibility — not raw capability — may become the defining variable in the AI security landscape.
The finding also raises uncomfortable questions for frontier labs. If a freely available model can already match proprietary systems on security reasoning tasks, the competitive moat around closed models narrows — particularly when export controls make those closed models inaccessible to swaths of the global market. Open-weight developers, unencumbered by usage restrictions, can iterate and deploy everywhere simultaneously.
Stay Ahead of AI
The gap between frontier and open-weight AI is narrowing on breaking AI news and the research reshaping security, infrastructure, and policy.
Read more AI news →
