# FinToolBench leaderboard

Tests whether LLM agents can call real financial tools to answer questions while staying timely, on intent and in the right regulatory domain.

## FinToolBench: Soft Score (higher is better, 7 models)

| Rank | Model | Provider | Score | Metric | Setting | Reported by | Source | URL | Date |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | Gemini-3.1-Pro (preview) | Google | 0.822 | Soft Score | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2603.08262 | 1 Aug 2026 (checked 6 Nov 2026) |
| 2 | Claude Sonnet 4.5 | Anthropic | 0.712 | Soft Score | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2603.08262 | 1 Aug 2026 (checked 6 Nov 2026) |
| 3 | Grok-3-beta | xAI | 0.685 | Soft Score | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2603.08262 | 1 Aug 2026 (checked 6 Nov 2026) |
| 4 | GPT-5.4 | OpenAI | 0.625 | Soft Score | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2603.08262 | 1 Aug 2026 (checked 6 Nov 2026) |
| 5 | Doubao-Seed-1.6 | ByteDance | 0.463 | Soft Score | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2603.08262 | 1 Aug 2026 (checked 6 Nov 2026) |
| 6 | Qwen3-8B | Qwen | 0.404 | Soft Score | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2603.08262 | 1 Aug 2026 (checked 6 Nov 2026) |
| 7 | GLM-4.7-Flash | Zhipu AI | 0.379 | Soft Score | Full 295 questions, GPT-5.1 judge, mean of 3 repeats | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2603.08262 | 1 Aug 2026 (checked 6 Nov 2026) |

Scores are never combined across benchmarks. Each row names its metric, source and date.

## About this benchmark

- Maker: Jiaxuan Lu, Kong Wang et al.
- Year: 2026
- Use case: Analyst Financial research
- Size: 295 questions over 760 executable free-tier tools; 7 models
- Public split: The full question set and tool manifest are released on GitHub.
- Grading: Tool calls are executed for real. GPT-5.1 judges answer quality (Soft Score, averaged over three runs) and flags timeliness, intent and domain mismatches per tool call.
- Licence: Not stated
- Submission: No leaderboard. Run the open evaluation code yourself.
- Products allowed: Models only
- Latest version: v2 (arXiv, 2026-08-01)
- What it does not cover: Paid terminals, brokerage and order routing, most jurisdictions.
- Last checked: 6 Nov 2026
- Links: paper https://arxiv.org/abs/2603.08262, code https://github.com/Double-wk/FinToolBench
- Directory page: https://benchmarks.finance/benchmarks/fintoolbench/
- Leaderboard: https://benchmarks.finance/leaderboards/fintoolbench/

Source: https://benchmarks.finance/leaderboards/fintoolbench/

Created and maintained by Moneda.