Asian Institute of Digital Finance, National University of Singapore, 2025
FAITH
Measures intrinsic hallucination on financial tables. A numeric span is masked in a sentence from a 10-K and the model must recover it from the report's tables and surrounding context.
About this benchmark
- Size
- Main split: 2,406 answerable spans from 453 S&P 500 companies (2024 10-Ks). Pilot split: 300 spans from 9 companies.
- Public split
- Pilot and main splits plus eval code on GitHub
- Grading
- Accuracy on the masked span, with a precision-relaxed numeric match plus unit-group matching. Also broken down by reasoning scenario A-D.
- Licence
- Apache-2.0 (GitHub repo, which says it holds code and data). The paper footnote says CC BY-NC-SA.
- Submission
- none
- Products allowed
- No
- Latest version
- arXiv v2 (2025-10-24), ACM ICAIF'25
- What it does not cover
- Covers only single-span numeric recovery from filings. No open-ended generation, citations or tool use.
Results
10 modelsEach score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.
FAITH arXiv v2 (2025-10-24), ACM ICAIF'25
overall accuracy %, higher is better. 10 models.
Updated 24 Oct 2025
- Claude Sonnet 495.6
- Gemini 2.5 Pro91.9
- GPT-4.189.2
- Gemini 2.5 Flash88.7
- GPT-4.1 mini88.2
- Claude-haiku-381.3
- Qwen3-32B73.9
- GPT-4.1-nano70
- Gemini-2.5-flash-lite50.2
- Llama-3.1-8B47.5
FAITH arXiv v2 (2025-10-24), ACM ICAIF'25
overall accuracy %, higher is better. 10 models.
Updated 24 Oct 2025
- 1Claude Sonnet 495.6
- 2Gemini 2.5 Pro91.9
- 3GPT-4.189.2
- 4Gemini 2.5 Flash88.7
- 5GPT-4.1 mini88.2
- 6Claude-haiku-381.3
- 7Qwen3-32B73.9
- 8GPT-4.1-nano70
- 9Gemini-2.5-flash-lite50.2
- 10Llama-3.1-8B47.5
| 1 | Claude Sonnet 4 | Anthropic | 95.6 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 |
|---|---|---|---|---|---|
| 2 | Gemini 2.5 Pro | 91.9 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 | |
| 3 | GPT-4.1 | OpenAI | 89.2 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 |
| 4 | Gemini 2.5 Flash | 88.7 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 | |
| 5 | GPT-4.1 mini | OpenAI | 88.2 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 |
| 6 | Claude-haiku-3 | Anthropic | 81.3 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 |
| 7 | Qwen3-32B | Qwen | 73.9 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 |
| 8 | GPT-4.1-nano | OpenAI | 70 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 |
| 9 | Gemini-2.5-flash-lite | 50.2 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 | |
| 10 | Llama-3.1-8B | Meta | 47.5 | Main split | Paper Table 3 Reported by the authors, reported 24 Oct 2025 · checked 1 Nov 2026 |
Papers and links
Last checked 1 Nov 2026. Something out of date or wrong? Send a correction.