Jiaxuan Lu, Kong Wang et al., 2026
FinToolBench
Tests whether LLM agents can call real financial tools to answer questions while staying timely, on intent and in the right regulatory domain.
About this benchmark
- Size
- 295 questions over 760 executable free-tier tools; 7 models
- Public split
- The full question set and tool manifest are released on GitHub.
- Grading
- Tool calls are executed for real. GPT-5.1 judges answer quality (Soft Score, averaged over three runs) and flags timeliness, intent and domain mismatches per tool call.
- Licence
- Not stated
- Submission
- No leaderboard. Run the open evaluation code yourself.
- Products allowed
- Models only
- Latest version
- v2 (arXiv, 2026-08-01)
- What it does not cover
- Paid terminals, brokerage and order routing, most jurisdictions.
Results
7 modelsEach score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.
FinToolBench v2 (arXiv, 2026-08-01)
Soft Score, higher is better. 7 models.
Updated 1 Aug 2026
- Gemini-3.1-Pro (preview)0.822
- Claude Sonnet 4.50.712
- Grok-3-beta0.685
- GPT-5.40.625
- Doubao-Seed-1.60.463
- Qwen3-8B0.404
- GLM-4.7-Flash0.379
FinToolBench v2 (arXiv, 2026-08-01)
Soft Score, higher is better. 7 models.
Updated 1 Aug 2026
- 1Gemini-3.1-Pro (preview)0.822
- 2Claude Sonnet 4.50.712
- 3Grok-3-beta0.685
- 4GPT-5.40.625
- 5Doubao-Seed-1.60.463
- 6Qwen3-8B0.404
- 7GLM-4.7-Flash0.379
| 1 | Gemini-3.1-Pro (preview) | 0.822 | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Paper Table 2 Reported by the authors, reported 1 Aug 2026 · checked 6 Nov 2026 | |
|---|---|---|---|---|---|
| 2 | Claude Sonnet 4.5 | Anthropic | 0.712 | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Paper Table 2 Reported by the authors, reported 1 Aug 2026 · checked 6 Nov 2026 |
| 3 | Grok-3-beta | xAI | 0.685 | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Paper Table 2 Reported by the authors, reported 1 Aug 2026 · checked 6 Nov 2026 |
| 4 | GPT-5.4 | OpenAI | 0.625 | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Paper Table 2 Reported by the authors, reported 1 Aug 2026 · checked 6 Nov 2026 |
| 5 | Doubao-Seed-1.6 | ByteDance | 0.463 | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Paper Table 2 Reported by the authors, reported 1 Aug 2026 · checked 6 Nov 2026 |
| 6 | Qwen3-8B | Qwen | 0.404 | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Paper Table 2 Reported by the authors, reported 1 Aug 2026 · checked 6 Nov 2026 |
| 7 | GLM-4.7-Flash | Zhipu AI | 0.379 | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Paper Table 2 Reported by the authors, reported 1 Aug 2026 · checked 6 Nov 2026 |
Papers and links
Last checked 6 Nov 2026. Something out of date or wrong? Send a correction.