Anthropic’s Fable 5.2 completes 41 percent of tested freelance projects

Anthropic’s Fable 5.2 completed 41.3 percent of projects in a benchmark based on real freelance assignments, a substantial gain over earlier AI systems. The result adds to evidence that AI agents can handle increasingly complex computer-based work, although independent operation remains a challenge.

The Remote Labor Index measures whether an AI agent can produce work good enough for a paying client to accept. Fable 5.2’s direct predecessor scored just under 18 percent, while the leading system when the benchmark launched about a year ago completed 2.5 percent of tasks, The Decoder reports.

The new model also leads the current field by a wide margin. Anthropic’s Opus 5.5 scored 21.25 percent, followed by OpenAI’s GPT 6 Astra at 20.83 percent. Scale Labs, which developed the benchmark with the Center for AI Safety, says Fable 5.2 made its largest gains in audio and music editing, architecture and interior design, and 3D modeling.

Real projects, human review

The benchmark draws on 240 completed projects from 358 verified professionals on Upwork. The assignments span fields including video editing, CAD tasks, game development, and data analysis. They were worth about $144,000 in total, and each took a human freelancer just under 29 hours on average.

AI agents work in virtual environments with professional software and specialized tools for generating images, audio, and video. Human experts then open the submitted files and compare them with the original freelancers’ work. Attempts to automate that review have fallen short: AI reviewers have been too easily convinced by polished-looking files with technical flaws, according to the researchers.

Nearly 50 of Fable 5.2’s successful submissions solved projects that no previously tested agent had completed. But agents still struggle particularly with industrial product design and animation. Common failures include incomplete work, broken files, and contradictory designs.

Impressive output, uneven judgment

In a separate assessment for One Useful Thing, entrepreneurship professor Ethan Mollick describes testing Fable 5.2 before its public launch. He calls it “almost” a form of general intelligence under his limited definition: an AI as capable as most human experts at most intellectual tasks performed on a computer. That definition excludes physical work and much of the interpersonal work involved in actual jobs.

Mollick says the model produced a plausible business proposal, built a literature course website, and drafted an academic research paper. He stopped short of letting it launch the business, and no human had fully checked the paper’s results. The model also failed to actually preregister the research, despite being asked to do so.

Longer projects exposed another weakness. In one experiment, AI agents built hundreds of coded 3D worlds, but a growing rulebook and approval process slowed production. Mollick intervened and asked agents to reassess the process.

For Mollick, the recurring limitation is not simply factual error. The model often failed to recognize when its work was repetitive, overly long, or unengaging. Human feedback made the results more useful, even when the AI performed much of the work itself.

Sources

• Anthropic’s Fable 5.2 completes 41 percent of real freelance jobs, up from 2.5 percent a year ago – The Decoder
• Fable 5.2 and Jagged AGI – One Useful Thing