A new research report from Princeton University reveals weaknesses in current benchmarks and evaluation practices for AI agents. The researchers argue that cost control is often neglected in evaluation, even though the resource costs of AI agents can be significantly higher than those of individual model queries. This leads to biased results, as expensive agents with high accuracy perform better on benchmarks without taking cost into account. Furthermore, the researchers criticize the focus on accuracy instead of practicality and the problem of overfitting, where AI agents learn shortcuts that do not work in real-world scenarios. The authors advocate a holistic evaluation of AI agents that takes into account both cost and usability in order to realistically assess the true performance of these systems.