Qwen DianJin Team (Alibaba Cloud), 2026
FinMCP-Bench
Measures how well LLM agents call real financial MCP tools in single-tool, multi-tool and multi-turn tasks.
About this benchmark
- Size
- 613 samples over 65 real financial MCP tools; 6 models
- Public split
- none found
- Grading
- Automatic. Predicted tool calls are compared with reference tool calls using Tool Precision, Tool Recall, Tool F1 and Exact Match Rate. Final answers are not graded.
- Licence
- Not stated
- Submission
- There is no leaderboard or submission process. Results appear only in the paper.
- Products allowed
- Models only
- Latest version
- v1 (arXiv, 2026-03-26)
- What it does not cover
- Answer quality and live trades: only tool calls are scored.
Results
6 modelsEach score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.
FinMCP-Bench v1 (arXiv, 2026-03-26)
TF1 (%), All, higher is better. 6 models.
Updated 26 Mar 2026
FinMCP-Bench v1 (arXiv, 2026-03-26)
TF1 (%), All, higher is better. 6 models.
Updated 26 Mar 2026
- 1Qwen3-235B-A22B-Thinking64.3
- 2Qwen3-30B-A3B-Thinking55.6
- 3Qwen3-4B-Thinking50.1
- 4DeepSeek-R149.9
- 5Seed-OSS-36B39.3
- 6GPT-OSS-20B32.6
| 1 | Qwen3-235B-A22B-Thinking | Qwen | 64.3 | All samples (single-tool, multi-tool, multi-turn), gold conversation history | Paper Table 2 Reported by the authors, reported 26 Mar 2026 · checked 6 Nov 2026 |
|---|---|---|---|---|---|
| 2 | Qwen3-30B-A3B-Thinking | Qwen | 55.6 | All samples (single-tool, multi-tool, multi-turn), gold conversation history | Paper Table 2 Reported by the authors, reported 26 Mar 2026 · checked 6 Nov 2026 |
| 3 | Qwen3-4B-Thinking | Qwen | 50.1 | All samples (single-tool, multi-tool, multi-turn), gold conversation history | Paper Table 2 Reported by the authors, reported 26 Mar 2026 · checked 6 Nov 2026 |
| 4 | DeepSeek-R1 | DeepSeek | 49.9 | All samples (single-tool, multi-tool, multi-turn), gold conversation history | Paper Table 2 Reported by the authors, reported 26 Mar 2026 · checked 6 Nov 2026 |
| 5 | Seed-OSS-36B | Not stated | 39.3 | All samples (single-tool, multi-tool, multi-turn), gold conversation history | Paper Table 2 Reported by the authors, reported 26 Mar 2026 · checked 6 Nov 2026 |
| 6 | GPT-OSS-20B | OpenAI | 32.6 | All samples (single-tool, multi-tool, multi-turn), gold conversation history | Paper Table 2 Reported by the authors, reported 26 Mar 2026 · checked 6 Nov 2026 |