AI leaderboards tell a remarkably detailed story about how well models write code, solve math problems, and answer graduate-level exams. What they almost never tell you is how well models perform the work most Americans are actually paid to do.
A new research project from AI talent marketplace Mercor puts a number on that gap. According to the company's Benchmark Map, published this week, roughly 90% of American working hours are spent on tasks with little or no coverage from existing AI benchmarks. For more context on this story, see our ongoing AI news.
"This makes it very hard to know how good AI models are at the work Americans actually do," the company writes in its findings.
Counting What Benchmarks Miss
To build the map, Mercor compiled an index of 29 benchmarks selected for quality and economic relevance, then mapped their prompts against the US government's O*NET occupational database — the standard taxonomy of 923 occupations and 18,797 distinct tasks that make up the American labor force.
The coverage numbers are stark. Only 208 of 923 O*NET occupations have any benchmark coverage at all, and just 924 of 18,797 occupational tasks are represented across the entire index.
The methodology, documented on the project's site, leans on AI itself: following recent research, an LLM judge tags each benchmark prompt with the ONET tasks it corresponds to, with the tagging validated against human expert labels on Mercor's own APEX benchmark subset. A task counts as covered only if the judge maps it to at least five prompts from the 29-benchmark index. To translate coverage into economic terms, Mercor combined its estimates with recently published LLM-derived estimates of the working-time share of each ONET task.
Coverage Follows the Money
The gaps are not randomly distributed. Mercor's analysis suggests the AI community has been benchmarking the largest labor markets first — the occupations with the biggest total wage bills, calculated from Bureau of Labor Statistics wage and employment estimates.
The job-family breakdown shows how uneven the attention is. Computer and mathematical occupations, the home turf of benchmark creation, are the best covered at 29 of 36 occupations. Business and financial operations follow at 33 of 48. But office and administrative support — a family that employs millions — has just 21 of 51 occupations covered, and management roles 21 of 56.
Even in the best-covered professions, depth is thin. When Mercor attempted ONET-weighted scoring for four benchmark-heavy occupations — investment banking, law, accounting, and software engineering — it found that fewer than one-third of the ONET tasks relevant to those jobs had enough benchmark coverage to be included in a weighted score.
Why the Blind Spots Matter
The finding lands at a moment when benchmark scores drive real decisions. Enterprises pick model tiers based on evaluations; boards hear about frontier-model progress in terms of benchmark jumps; policymakers cite capability trajectories when weighing regulation. If 90% of working hours sit outside what benchmarks measure, all of those decisions rest on a sample skewed toward the office-bound, the technical, and the academic.
There are structural reasons benchmarks cluster where they do. Software engineering tasks are cheap to specify, cheap to grade automatically, and easy for benchmark authors — who mostly write software — to design. Assessing a medical assistant's bedside manner, a logistics coordinator's judgment under shifting constraints, or an electrician's diagnostic process is harder to standardize, harder to verify remotely, and in many cases impossible to test without physical presence.
Mercor, whose core business matches domain experts with AI labs for training and evaluation work, has an obvious commercial interest in pointing out that evaluation coverage is thin — the company sells access to exactly the experts who could close that gap. But the underlying measurement problem is real and increasingly acknowledged across the field, and the company published its methodology and full benchmark list for scrutiny.
The company says its Benchmark Map is a first step toward a continuously updated resource for understanding AI capabilities across the economy, and it is working toward O*NET-weighted model scores that weight each occupational task equally rather than letting frequently benchmarked tasks dominate the average.
The Measurement Decade
What happens next depends on whether benchmark builders follow the economic map. The Mercor data points to where coverage is missing — and, implicitly, where new benchmarks would carry the most information value. Occupations with large employment, high wage bills, and zero benchmark coverage are, in effect, open evaluation territory.
For now, the practical takeaway for anyone consuming AI leaderboard news is a caveat: a model's benchmark profile describes its strengths in the slice of the economy that benchmarks happen to cover. The other 90% of working hours — the admin work, the caregiving, the field service, the floor management — remains largely unmeasured territory, and claims about AI transforming it currently rest more on anecdote than evidence.
As the company's own framing puts it, benchmarks have multiplied in recent years, but coverage of economically valuable work remains limited. The map makes that limitation visible, tile by tile, occupation by occupation.
---
Stay Ahead of AIGet the latest AI news, analysis, and breakthroughs — all in one place.
Read more AI news →
