AI benchmarks miss most of the work Americans actually do

Existing AI benchmarks provide little evidence about how well models perform most tasks in the U.S. economy, according to a new analysis from Mercor. Mercor reports in its Benchmark Map that about 90% of U.S. working hours involve tasks with little or no representation in the benchmarks it reviewed.

The company mapped 29 public and proprietary benchmarks against the Labor Department’s O*NET database, which lists 923 occupations and 18,797 associated tasks. Only 208 occupations had at least one task represented by five or more benchmark prompts. At the task level, just 924 tasks met that threshold.

Mercor says the gap makes it difficult for businesses, workers, and policymakers to judge whether AI systems can reliably handle real workplace duties. A benchmark may test a narrow set of activities well while saying little about the wider role.

Coverage favors larger and more AI-exposed jobs

The analysis finds that benchmark coverage is uneven. Tasks labeled as exposed to AI were roughly six times more likely to appear in the benchmark set than unexposed tasks. Coverage also rose for occupations with larger total wage bills, even after accounting for AI exposure.

That pattern favors fields such as software engineering, financial analysis, and law. Mercor highlights a contrast within legal work: benchmarks cover many tasks performed by lawyers, but far fewer tasks associated with legal secretaries and court clerks, despite similar exposure to AI.

Tasks that O*NET experts rate as especially important were also six to 10 times more likely to be covered than lower-rated tasks. Mercor describes this as a reasonable allocation of limited evaluation resources, but warns that it creates blind spots in smaller and middle-income knowledge-work roles.

The company notes important limits. Its list of 29 benchmarks is not a complete sample, and its selection process includes subjective judgments about economic realism. Many physical tasks also remain hard to assess through current digital benchmarks. Mercor says it plans to expand the index and develop more occupation-specific measures of model performance.