A revised decision-model index penalizes misplaced confidence
Artificial Analysis’s Decision Index v2 adds document-level completion and calibrated confidence to the buying question, showing why per-answer accuracy and token price alone are incomplete measures.
Artificial Analysis introduced Decision Index v2 on November 8.
The evaluation combines six components across document extraction and decisions, tool calling, knowledge, reasoning and consistency under equivalent wording. The new AA-Paperclip component has 1,500 documents and accounts for 30% of the composite; each document’s fifteen answers count together as one decision. That distinguishes a mostly correct extraction from a complete usable record. [1][2][3]
The Decision Score incorporates both correctness and confidence. A wrong answer given 90% confidence scores 9.5 out of 100 under the published formula; a wrong answer at 60% confidence scores 32. These are calculated examples, not measured model scores. The distinction matters when low-confidence cases are routed to a person or a stronger system rather than acted on automatically. [2][3]
Sources & details 3 sources
Discuss in your voice app
Take the reporting, visual explanations and original sources into a conversation. Copy the brief, paste it into your preferred assistant, then start its voice mode.
Paste the brief into your voice app to discuss this reporting.Reporting sources
- Artificial Analysis — Decision Index v2 announcement
November 8 announcement; opened successfully through the dated article index after direct URL attempts failed.
- Artificial Analysis — Decision Index v2
Current component weights, document denominator, scoring formula and cost/time definitions.
- Artificial Analysis — methodology
Benchmark design and scoring; primary evaluator independent of model vendors, not an independent replication of this index.
Images & figures
Applying AA’s formula gives a wrong answer 9.5 points at 90% confidence and 32 at 60%; correct answers receive 99.5 and 92 respectively. These are calculated examples, not model scores.
Euangelion · Source · Usage terms · Download figure