Leading AI research organizations are developing more challenging evaluation methods as current AI models consistently achieve top scores on traditional tests. According to Tharin Pillay’s article in Time Magazine, conventional benchmarks like SATs and bar exams no longer effectively measure AI capabilities.
New evaluation frameworks include FrontierMath, developed by Epoch AI in collaboration with prominent mathematicians, which presents exceptionally difficult mathematical problems. When released, available AI models scored only 2% on FrontierMath, though OpenAI’s new o3 model quickly achieved 25.2%.
Other notable new evaluations include “Humanity’s Last Exam” and RE-Bench, which test AI systems across various domains and real-world engineering tasks.
Experts emphasize the challenge of designing effective evaluations that measure genuine reasoning abilities rather than pattern recognition. The article notes that while AI systems excel at certain complex tasks, they still struggle with simple problems that humans solve easily. Leading AI labs now routinely conduct “red-team” testing before releasing new models, examining potential harmful outputs and safety concerns.
Industry experts, including Apollo Research’s Marius Hobbhahn, advocate for mandatory third-party testing of AI models, noting current evaluation processes are often underfunded despite their crucial role in identifying potential risks.