Vals AI

ClaimBench

Checks a claim against a 10-K excerpt and labels it supported, contradicted or insufficient_evidence. Excerpts range from short to about 16k tokens.

Best reported

100

d1-ultra

accuracy %

Official board, 23 Oct 2026

About this benchmark

Size
1,960 claims in 5 categories (392 supported, 785 contradicted, 783 insufficient_evidence)
Public split
10 public examples
Grading
Accuracy against gold labels that come from the construction rules, not from a model. The page does not say whether scoring is done by code or by an LLM judge.
Licence
Not stated
Submission
Not stated
Products allowed
Not stated
Latest version
v1 (updated 2026-10-23)
What it does not cover
Only classifies claims. No open-ended research, calculation or generation of answers.

Results

10 models
Leaderboard

Each score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.

Best value: d1, 98.7 at < $0.001 per question (cheapest within 5% of the top score)

ClaimBench: score vs cost

Score against cost per test. Up and to the left is better.

Updated 23 Oct 2026

Y: accuracy %X: Cost per test (USD)Showing 10 models
ClaimBench: score vs cost10 models. Frontier: d1, Mercury Decide, Decider V1.2, d1-ultra.9899100$0$0.0001$0.0002$0.0003EfficientInefficient↑ Best performance← Lowest costCost per test (USD)d1-ultra: 100 accuracy %, $0.000212 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)d1-ultraDecider V1.2: 99.7 accuracy %, $0.000104 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Decider V1.2Drex Max: 99.2 accuracy %, $0.000304 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Drex MaxMercury Decide: 99.1 accuracy %, $0.000102 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Mercury DecideDecider V1.1 27B: 98.8 accuracy %, $0.000104 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Decider V1.1 27Bd1: 98.7 accuracy %, $0.000101 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)d1Microsoft-Decision-1: 98.2 accuracy %, $0.000218 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026) Microsoft-Decision-1Jev 1.14: 98.1 accuracy %, $0.000233 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Jev 1.14Drex v1.5: 98.0 accuracy %, $0.000205 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Drex v1.5Jev 1.13.0: 97.7 accuracy %, $0.000233 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Jev 1.13.0ClaimBench: score vs cost10 models. Frontier: d1, Mercury Decide, Decider V1.2, d1-ultra.9899100$0.0001$0.0002EfficientInefficient↑ Best performance← Lowest costCost per test (USD)d1-ultra: 100 accuracy %, $0.000212 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)d1-ultraDecider V1.2: 99.7 accuracy %, $0.000104 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Decider V1.2Drex Max: 99.2 accuracy %, $0.000304 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Drex MaxMercury Decide: 99.1 accuracy %, $0.000102 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Mercury DecideDecider V1.1 27B: 98.8 accuracy %, $0.000104 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Decider V1.1 27Bd1: 98.7 accuracy %, $0.000101 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)d1Microsoft-Decision-1: 98.2 accuracy %, $0.000218 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026) Microsoft-Decision-1Jev 1.14: 98.1 accuracy %, $0.000233 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Jev 1.14Drex v1.5: 98.0 accuracy %, $0.000205 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Drex v1.5Jev 1.13.0: 97.7 accuracy %, $0.000233 per test (Vals.ai ClaimBench leaderboard, Cost / Claim, 23 Oct 2026)Jev 1.13.0

ClaimBench v1 (updated 2026-10-23)

accuracy %, higher is better. 10 models.

Updated 23 Oct 2026

  1. d1-ultra100
  2. Decider V1.299.7
  3. Drex Max99.2
  4. Mercury Decide99.1
  5. Decider V1.1 27B98.8
  6. d198.7
  7. Microsoft-Decision-198.2
  8. Jev 1.1498.1
  9. Drex v1.598.0
  10. Jev 1.13.097.7

ClaimBench: cost

cost per test item (USD), lower is better. 10 models.

Updated 23 Oct 2026

  1. d10.000
  2. Mercury Decide0.000
  3. Decider V1.20.000
  4. Decider V1.1 27B0.000
  5. Drex v1.50.000
  6. d1-ultra0.000
  7. Microsoft-Decision-10.000
  8. Jev 1.140.000
  9. Jev 1.13.00.000
  10. Drex Max0.000

ClaimBench v1 (updated 2026-10-23)

accuracy %, higher is better. 10 models.

Updated 23 Oct 2026

  1. 1d1-ultra100
  2. 2Decider V1.299.7
  3. 3Drex Max99.2
  4. 4Mercury Decide99.1
  5. 5Decider V1.1 27B98.8
  6. 6d198.7
  7. 7Microsoft-Decision-198.2
  8. 8Jev 1.1498.1
  9. 9Drex v1.598.0
  10. 10Jev 1.13.097.7
ClaimBench leaderboard (accuracy %, higher is better)
1d1-ultraNot stated100decision models boardVals.ai leaderboard
Official board, 23 Oct 2026
2Decider V1.2Not stated99.7decision models boardVals.ai leaderboard
Official board, 23 Oct 2026
3Drex MaxNot stated99.2decision models boardVals.ai leaderboard
Official board, 23 Oct 2026
4Mercury DecideNot stated99.1decision models boardVals.ai leaderboard
Official board, 23 Oct 2026
5Decider V1.1 27BNot stated98.8decision models boardVals.ai leaderboard
Official board, 23 Oct 2026
6d1Not stated98.7decision models boardVals.ai leaderboard
Official board, 23 Oct 2026
7Microsoft-Decision-1Microsoft98.2decision models boardVals.ai leaderboard
Official board, 23 Oct 2026
8Jev 1.14Not stated98.1decision models boardVals.ai leaderboard
Official board, 23 Oct 2026
9Drex v1.5Not stated98.0decision models boardVals.ai leaderboard
Official board, 23 Oct 2026
10Jev 1.13.0Not stated97.7decision models boardVals.ai leaderboard
Official board, 23 Oct 2026

Last checked 1 Nov 2026. Something out of date or wrong? Send a correction.