StateBench shows why remembering updates is harder than retrieving facts
Ai2’s StateBench finds that four tested frontier models can retrieve simple facts yet fall to 13% accuracy or less on the longest dependency-heavy state-tracking sequences.
Ai2 introduced StateBench on November 3 to distinguish types of memory that a large context window can conceal.
Its four levels progress from an unchanging value, to the latest replacement value, to cumulative changes, and finally to updates that depend on other variables. The test can keep the number and positions of updates comparable while changing what must be computed. [1][2]
Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna did well on simpler retrieval and replacement problems. On the longest general-finite sequences, where later values depend on earlier changes elsewhere, all four fell to 13% accuracy or less. This is a synthetic state-tracking result, not a measured failure rate for production agents or all long documents. [1][2]
Sources & details 3 sources
Discuss in your voice app
Take the reporting, visual explanations and original sources into a conversation. Copy the brief, paste it into your preferred assistant, then start its voice mode.
Paste the brief into your voice app to discuss this reporting.Reporting sources
- Ai2 — StateBench
November 3 description, controlled hierarchy and frontier-model results.
- StateBench — technical report
Primary research by the same authors; synthetic tests, not production reliability.
- Ai2 — StateBench code
Inspectable benchmark implementation and reproduction path; no independent reproduction claimed.
Images & figures
Following dependent changes requires carrying earlier state forward. In this original three-object example, three swaps move A from position 1 to position 3; it illustrates the mechanism rather than a scored StateBench item.
Euangelion · Source · Usage terms