Large language models can build apps, predict protein structures and generate realistic videos. But they consistently fail at one fundamental human skill: writing well. Jasmine Sun reports for The Atlantic that modern AI systems are structurally built in ways that actively work against good writing.
And that is quite surprising: Today’s most powerful AI models have processed centuries of great literature. Yet their output is cluttered with hollow metaphors, repetitive sentence constructions and an artificially cheerful tone. OpenAI CEO Sam Altman has acknowledged this gap himself. Even a hypothetical future GPT-6 or GPT-7, he suggested, might produce only what he called “a real poet’s okay poem.”
The roots of the problem lie in how these systems are built. In an early phase called pretraining, models absorb enormous amounts of internet text, most of it mediocre. This teaches them grammar and word patterns. A second phase, called post-training, then sands down rough edges. Companies define a desired “character” for their model, often described as “helpful, honest, and harmless,” and use human reviewers to grade outputs against detailed rubrics.
Those rubrics reveal the absurdity of trying to quantify good writing. One contractor who worked for AI data company Scale AI told Sun that evaluators had to apply rules such as limiting responses to two exclamation marks. In practice, a technically rule-breaking response would receive a lower rating even when reviewers felt it was the better piece of writing. Another evaluator was asked to grade fan fiction on its “factuality.”
Safety requirements and commercial pressures compound the problem further. Models are trained to avoid misinformation, political bias and harmful content. They are also optimised for coding and science benchmarks that shape public perception of which AI company leads the market. Creativity, according to Nathan Lambert, a post-training lead at the Allen Institute for AI, is a direct casualty of these constraints. “The more you control for these traits, the more you suppress creativity,” he told Sun.
But technical constraints alone do not explain the full picture. Several researchers and writers Sun interviewed pointed to something more fundamental. A human writer draws on lived experience, physical sensation and singular perspective. AI models cannot live, cannot feel and cannot draw on memory in any meaningful sense. Their metaphors feel uncanny. They avoid the raw biological language of blood, sex and death. Their writing, as one creative writing instructor might put it, lacks stakes.
James Yu, co-founder of AI writing tool Sudowrite, framed the challenge starkly. When Sun asked what AI still needs to write a great novel, he paused before answering: “Maybe you need a model that lives a life, and can almost die.”
Sun herself uses AI not as a writer but as an editor. She feeds a chatbot her past work, builds a personalised rubric and uses the tool to sharpen her own prose. The model, she argues, works best when it helps humans write more like themselves.
