Artificial Analysis Launches Decision Index to Benchmark a New Class of Probability-Output Models

There's a quiet architectural split happening in the model ecosystem, and it's worth paying attention to if you're building anything that routes requests, flags content, or triages support tickets. On one side you've got the big reasoning LLMs, optimized for long-horizon tasks and chain-of-thought depth. On the other, a newer category: decision models, which take a structured question with a fixed set of options and return a probability for each one, in a single call, fast and cheap.

Artificial Analysis just launched version 1.0 of its Decision Index to benchmark this category directly, both against each other and against general-purpose LLMs. The pitch is simple. If a model's job is "classify this ticket as billing or technical, then tell me if the customer is angry," you don't need a model that reasons through fifteen steps. You need one that spits out calibrated probabilities across a handful of branches, fast, and doesn't hallucinate when it's unsure.

The example in the launch piece is a good illustration: feed a model an angry customer message, and instead of generating a paragraph of analysis, a decision model directly outputs probabilities for "route to billing vs. technical vs. account," "is this urgent: yes/no," and "how upset: calm/frustrated/angry," all in one pass. That's the entire value proposition: structured, parsed, cheap, fast.

How the Index Is Built

The v1.0 index combines four separate evaluations, each weighted at 25% of the composite score:

• AA-Omniscience-Decision (knowledge and hallucination): 500 private questions adapted from Artificial Analysis's existing AA-Omniscience benchmark. Each question gives five options, four candidate answers plus "I don't know." This single eval actually splits the 25% weighting in two: 12.5% for raw accuracy, 12.5% for non-hallucination rate, which is the share of wrong-or-unknown questions where the model honestly picked "I don't know" instead of confabulating.
• HLE-MultiChoice (reasoning): the 513 text-only multiple-choice questions pulled from Humanity's Last Exam, reframed as single-choice selection among lettered answers.
• When2Call (tool calling): NVIDIA's eval testing whether a model correctly decides to answer directly, invoke a tool, ask a clarifying question, or decline, given a conversation and an available toolset.
• WildGuardTest (classification): Ai2's safety eval, checking whether a model can correctly flag a user request as harmful, and whether an assistant's reply is a refusal and/or harmful.

Crucially, every eval also runs on standard LLMs for comparison, just without the probability output. An LLM answering these benchmarks picks an option via structured output, but gives no confidence distribution across alternatives, which is exactly the capability decision models are built around.

At launch, Jev 1.13 tops the leaderboard with a score of 58, narrowly ahead of GLiDE and Microsoft-Decision-1, both at 54. Jev's edge comes almost entirely from reasoning: it scores 21 on HLE-MultiChoice, where every other decision model caps out at 15 or below. It's weaker on raw knowledge accuracy (35 on AA-Omniscience-Decision, versus a top score of 41 from Liquid AI's d1), but makes up ground with the second-best non-hallucination rate at 35%, crushing d1's rate of just 2%. On tool calling, Jev sits fourth at 81, behind Celeris-1 Decision and Fastino's GLiDE, both at 84. On classification it ties Perplexity's Decider v1.1 27B at 94, a point behind GLiDE's 95.

Cost and latency are where decision models really differentiate themselves from each other, and from LLMs generally. Pricing across the field ranges from $17 to $199 per million tasks, with median latency spanning 0.06 to 0.5 seconds per task. Decider v1.1 27B is the cost leader at $17 per million tasks, with Microsoft-Decision-1 ($31) and Jev 1.13 ($42) also sitting on the capability-versus-cost Pareto frontier. GLiDE matches Microsoft-Decision-1's score of 54 but costs $119, nearly three times Jev's price. On raw speed, Celeris-1 Decision wins at 0.06 seconds per task, more than twice as fast as Jev's 0.13 seconds.

Calibration is really the headline metric here, since it's the whole reason this category exists. Artificial Analysis checked whether a model's stated confidence in its chosen answer actually tracks how often that answer is correct. The result across the board: accuracy falls below stated probability at almost every confidence bucket, for every decision model tested. That gap is driven mostly by poor calibration on the harder cognitive tasks, HLE-MultiChoice and AA-Omniscience-Decision. On the more operational tasks, When2Call and WildGuardTest, most decision models are reasonably well-calibrated. In other words, these models know what they don't know when the task is procedural, but overstate confidence when the task requires actual world knowledge or reasoning.

The LLM comparison is where the trade-off becomes concrete. Claude Haiku 5.5 (Low) scores 61 on the Decision Index, three points above Jev 1.13, but it takes 1.4 seconds per task versus Jev's 0.13 seconds (11x slower) and costs $184 per million tasks versus $42 (4.4x the price). GPT-6 Luna (Low) scores 55, slightly below Jev, at 1.5 seconds and $131 per million tasks. The knowledge gap is stark too: Claude Haiku 5.5 (Low) hits 52% accuracy on AA-Omniscience-Decision and GPT-6 Luna (Low) hits 66%, against a 28-41% range for decision models. On HLE-MultiChoice, the LLMs score 24 and 22 respectively, versus Jev's 21 and single digits to mid-teens for the rest of the pack.

Where decision models hold their own is on the operational tasks they're actually built for. On WildGuardTest classification, GLiDE (95), Decider v1.1 27B (94), and Jev 1.13 (94) all edge out Claude Haiku 5.5 (Low) at 93 and beat GPT-6 Luna (Low) at 89. On When2Call tool calling, Celeris-1 Decision and GLiDE both score 84, ahead of GPT-6 Luna (Low) at 77 and Claude Haiku 5.5 (Low) at 73.

On the hallucination front specifically, Microsoft-Decision-1 leads all decision models with a 40% non-hallucination rate, followed by Jev 1.13 at 35% and Mercury Decide at 32%. Every other decision model in the index falls below 17%, a wide gap suggesting honesty-under-uncertainty is far from a solved problem even within this narrower, more constrained model category.

Key Takeaways

The practical read here: if your workload is genuinely a bounded, structured decision, routing, flagging, triage, classification, decision models are already competitive with frontier LLMs on accuracy while being an order of magnitude cheaper and faster. But if your task leans on broad world knowledge or multi-step reasoning, general LLMs still have a real edge, and current decision models' calibration on those harder tasks isn't yet trustworthy enough to lean on blindly. Artificial Analysis says live scores will keep updating on the public leaderboard, with calibration data for LLMs "coming soon," and is inviting decision model builders to get added to the index directly.