AI context windows are much smaller than advertised, developer warns

AI companies advertise ever-larger context windows for their language models, but a developer argues that the usable portion is far smaller than the marketed figures suggest. Garrit Franke writes in his blog Garrit’s Notes that effective model performance begins to degrade around 100,000 tokens, regardless of whether the advertised limit is 200,000, one million, or two million tokens.

Franke draws on a distinction between what he calls the “smart zone” and the “dumb zone” of a model’s context window. In the smart zone, the model processes information reliably. In the dumb zone, attention drops off and the model begins to lose track of earlier instructions. Research projects such as RULER and a report by Chroma on “context rot” support this view, finding that performance degrades gradually as the context window fills up.

See also: Why is my AI suddenly so much dumber than before?

A budget, not a blank cheque

The problem is especially acute for coding agents. These AI tools read files, run tests, and debug code across long sessions, burning through tokens quickly. By the time a session feels long, the model may already be operating in its degraded state.

Some tools attempt to address this automatically. Claude Code, for example, uses a feature called auto-compaction: when a session grows long, the agent summarises its history and starts a fresh context. Franke acknowledges this helps but points out a flaw. The summary is generated by a model that is already in the dumb zone, making the handoff less reliable.

His preferred approach is manual. He opens a new session and passes it a specification he wrote himself, keeping only the information that actually matters going forward. He treats the context window like a budget, not a blank cheque.

Structured workflows can support this discipline. Projects such as obra/superpowers and mattpocock/skills organise agent work around small, named documents: plans, briefs, and handoff notes that allow each new session to start clean and informed.