Benchmark scores alone cannot predict the real cost of AI agents

Raw benchmark rankings can hide the factors that determine whether an AI agent is useful and affordable in day-to-day work: time limits, token budgets and failed attempts.

Laurie Voss writes for VentureBeat that Alibaba’s Qwen 3.8-Max illustrates the gap. Alibaba reports strong coding results with time limits of up to five hours, and as much as 12 hours for one research benchmark. The independent VulcanBench harness gives models only 45 to 60 minutes. Under those shorter conditions, Qwen’s default setting ranks last in one comparison.

The difference matters because reasoning models use tokens not only to produce an answer, but also to plan and check their work. A model may spend its full allowance on internal reasoning before it delivers a usable result. The run then costs money but produces no value.

Measure the outcome, not just the rate card

Voss argues that teams should track cost per successful task. The metric divides total spending, including failed runs, by the number of tasks that meet a defined acceptance check. It can reveal that a higher-priced model, or a lower reasoning setting, delivers better value.

Benchmark results also show why more reasoning does not always improve performance. In a VulcanBench report, Claude Opus 5 solved 20 of 23 tasks at its lowest effort setting, compared with 18 at high effort. The higher setting produced fewer incorrect answers, but more runs timed out. A timeout receives no credit in the benchmark.

For organisations building automated workflows, the practical advice is to record why every run fails. Teams should separate budget exhaustion, verifier failures and harness errors. They should also test each effort setting, rather than assuming the default is best. If latency is not a firm service requirement, token limits may offer a fairer comparison than wall-clock limits, which can also reflect a provider’s serving speed.