When OpenAI unveiled GPT-6 Astra this week, one demo drew particular attention from an audience that rarely gets a shoutout at frontier model launches: electrical engineers. The launch post featured the model designing a circuit board in KiCad, the open-source electronics design tool. But a small team of engineers has been asking a harder question — not whether AI models can operate electronics software, but whether the circuits they produce actually work.
Their answer arrived on September 4 in the form of EEBench, a new public benchmark that grades AI-designed circuits using deterministic SPICE simulations rather than human judgment. The early results suggest current models know far more about electronics than their output typically shows, yet still fail at the kind of real-world part behavior that trips up human engineers too. For more context on this story, see our ongoing artificial intelligence updates.
Why Circuit Design Needed Its Own Benchmark
The EEBench team, which publishes the benchmark at eebench.org, says its experience shows current models have absorbed textbooks, datasheets, application notes, and vast amounts of code. The bottleneck is the interface. When an agent operates a graphical CAD tool like KiCad, it spends much of its effort clicking through menus and tracking what is on screen, consuming its context window with coordinates and application state rather than electrical reasoning.
EEBench takes a different approach. Circuits in the benchmark are written in atopile, a language that describes electronics as declarative code, so the agent works directly on components, connections, and constraints. The model can modify a design, build it, run a simulation, and inspect failures without leaving the project. According to the team, this works far better than asking a model to draw lines in a GUI — and it lets the benchmark test electronics knowledge rather than computer-use skill.
Real Parts, Real Failures
One public task is drawn from a residential energy meter. When its 5-volt supply disappears, the circuit must keep the processor alive for another 20 milliseconds so it can save accumulated readings, with the protected rail staying above the processor's 3.0-volt brownout threshold the whole time.
Most models, the team reports, immediately reach the right base conclusion: add a capacitor. The benchmark makes that harder than it sounds. A real ceramic capacitor may deliver far less than its advertised capacitance once voltage is applied across it, parts carry tolerances, and extra capacitance adds cost, takes space, and slows the rail's recharge when power returns.
The graders caught one submission that illustrates the gap between textbook answers and working hardware. A design specified 22 microfarads nominally — a value that looks sufficient on paper. But at 4.7 volts of bias, the grader measured only 11.4 microfarads of effective capacitance, far below the task's 545-microfarad requirement. The source code built successfully. The circuit still failed: simulation showed the protected rail falling below the 3-volt requirement after just 0.85 milliseconds, nowhere near the required 20.
Harder tasks push further. One analog assignment requires synthesizing a multiple-feedback low-pass filter around an op-amp, solving resistor and capacitor ratios for the required poles, and keeping gain, cutoff frequency, and Q within limits even when every component is pushed to worst-case tolerance corners.
How the Grading Works
EEBench checks are fully deterministic. The harness builds the submitted design, constructs the circuit graph and bill of materials, and runs a set of SPICE simulations and design checks, with each requirement producing a measurement against a numeric limit. Tasks measure gain, thresholds, ripple, transient response, and behavior at component-tolerance corners. The technical score is combined with a cost-efficiency measure against a reference bill of materials — though cost only helps once the circuit works.
The team compares the setup to giving a coding agent a compiler and a test suite, except the tests measure voltages and component behavior. All tasks in version 1 use real manufacturer parts, with specifications extracted from datasheets and carried into the simulation models, forcing the agent to find combinations that exist, can be ordered, and are reasonably priced.
The First Leaderboard
Results from September 1 put Claude Opus 5 at the top of EEBench V1 with 61.6% across 13 tasks. Grok 4.6 came second at 57.1%, just ahead of Claude Fable 5.1 at 56.4%. Claude Fable 5 scored 54.3% and Claude Opus 4.8 Max 51.4%. The team notes that a few months ago it would not have expected models to perform this well.
One result pleased the team particularly: xAI included EEBench in the Grok 4.6 model card, in a section on engineering acceleration alongside 3D modeling and parametric CAD evaluations. xAI's own published run scored Grok 4.6 at 60.0% with xhigh reasoning effort. According to xAI's launch materials, the model received high-quality engineering data and reinforcement learning training in domain-specific environments, including computer-aided design.
OpenAI's models tested so far sit further down the table: GPT-5.5 scored 42.3% and GPT-5.6 Sol 39.4%. There is no GPT-6 Astra result yet — a notable gap given that the model's KiCad demo sparked the renewed attention. The benchmark's leaderboard, methodology, and sample result explorer are all public, and labs can run their own models.
What It Does Not Measure — Yet
EEBench V1 covers analog and digital design through simulation only. It does not yet test whether a model can lay out a physical board, manufacture it, or bring up a finished product — stages where entirely new failure modes appear. The team says it wants to add those stages later, but focused version 1 on the requirements-design-verification loop because that is where useful engineering work can already be graded objectively.
Still, the benchmark lands at a telling moment. A frontier lab now cites it in a model card, launch demos put electronics on the front page, and the top score — 61.6% — shows models getting meaningfully capable while remaining far from reliable. For the electrical engineers watching, the message from the first EEBench results is measured: AI can increasingly design circuits on paper. Whether they survive contact with real parts is precisely what this benchmark exists to find out.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
