ByteDance Seed (Columbia Business School listed as advisor), 2025
FinSearchComp
Expert-written financial search-and-reasoning questions in three tiers: time-sensitive data fetching, simple historical lookup and complex historical investigation. There are Global and Greater China subsets.
About this benchmark
- Size
- 635 questions, 3 tasks, 2 market subsets
- Public split
- The paper calls it fully open-source and the dataset is on Hugging Face. The paper does not state a held-out split.
- Grading
- An LLM judge scores 0/1 against expert rubrics. Time-sensitive answers are checked against real-time API data within the day's high-low band. The paper reports about 95% judge-human agreement.
- Licence
- CC-BY-4.0 (data, per Hugging Face card)
- Submission
- Not stated
- Products allowed
- Yes
- Latest version
- arXiv v1 (2025-09-16); HF dataset last modified 2025-10-27
- What it does not cover
- Does not test filing analysis depth, portfolio decisions or advice quality. Time-sensitive answers depend on the day they were run.
Results
11 modelsEach score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.
FinSearchComp arXiv v1 (2025-09-16); HF dataset last modified 2025-10-27
Global subset average accuracy %, higher is better. 11 models.
Updated 1 Nov 2026
- grok-468.9
- GPT-5-Thinking63.9
- Numa 159.4
- Gemini2.5-pro42.6
- DouBao39.1
- Gemini2.5-pro (Google Search)38.2
- Qwen3-235B-A22B-250737.4
- YuanBao-DeepSeek-V330.5
- DouBao-Thinking29.8
- HunYuan-T1-Thinking29.8
- YuanBao-DeepSeek-R129.8
FinSearchComp arXiv v1 (2025-09-16); HF dataset last modified 2025-10-27
Global subset average accuracy %, higher is better. Top 10 of 11 models.
Updated 1 Nov 2026
- 1grok-468.9
- 2GPT-5-Thinking63.9
- 3Numa 159.4
- 4Gemini2.5-pro42.6
- 5DouBao39.1
- 6Gemini2.5-pro (Google Search)38.2
- 7Qwen3-235B-A22B-250737.4
- 8YuanBao-DeepSeek-V330.5
- 9DouBao-Thinking29.8
- 9HunYuan-T1-Thinking29.8
| 1 | grok-4 | xAI | 68.9 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
|---|---|---|---|---|---|
| 2 | GPT-5-Thinking | OpenAI | 63.9 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 3 | Numa 1 | Moneda | 59.4 | Product with tools; historical-lookup tiers only (time-sensitive tier not run); two-judge panel | Moneda run, 2026-11-01 Run by Moneda, 1 Nov 2026 |
| 4 | Gemini2.5-pro | 42.6 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 | |
| 5 | DouBao | ByteDance | 39.1 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 6 | Gemini2.5-pro (Google Search) | 38.2 | API with search | Paper Table 5 Reported by the authors, 16 Sept 2025 | |
| 7 | Qwen3-235B-A22B-2507 | Qwen | 37.4 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 8 | YuanBao-DeepSeek-V3 | Not stated | 30.5 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 9 | DouBao-Thinking | ByteDance | 29.8 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 9 | HunYuan-T1-Thinking | Tencent | 29.8 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 9 | YuanBao-DeepSeek-R1 | Not stated | 29.8 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
FinSearchComp arXiv v1 (2025-09-16); HF dataset last modified 2025-10-27
Greater China subset average accuracy %, higher is better. 11 models.
Updated 1 Nov 2026
- Numa 157.1
- DouBao54.2
- YuanBao-DeepSeek-R152.5
- grok-451.9
- HunYuan-T1-Thinking50.5
- DouBao-Thinking49
- YuanBao-DeepSeek-V348.8
- GPT-5-Thinking46.4
- ERNIE-X140.8
- DeepSeek-R140.5
- Kimi k238.3
FinSearchComp arXiv v1 (2025-09-16); HF dataset last modified 2025-10-27
Greater China subset average accuracy %, higher is better. Top 10 of 11 models.
Updated 1 Nov 2026
- 1Numa 157.1
- 2DouBao54.2
- 3YuanBao-DeepSeek-R152.5
- 4grok-451.9
- 5HunYuan-T1-Thinking50.5
- 6DouBao-Thinking49
- 7YuanBao-DeepSeek-V348.8
- 8GPT-5-Thinking46.4
- 9ERNIE-X140.8
- 10DeepSeek-R140.5
| 1 | Numa 1 | Moneda | 57.1 | Product with tools; historical-lookup tiers only (time-sensitive tier not run); two-judge panel | Moneda run, 2026-11-01 Run by Moneda, 1 Nov 2026 |
|---|---|---|---|---|---|
| 2 | DouBao | ByteDance | 54.2 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 3 | YuanBao-DeepSeek-R1 | Not stated | 52.5 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 4 | grok-4 | xAI | 51.9 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 5 | HunYuan-T1-Thinking | Tencent | 50.5 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 6 | DouBao-Thinking | ByteDance | 49 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 7 | YuanBao-DeepSeek-V3 | Not stated | 48.8 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 8 | GPT-5-Thinking | OpenAI | 46.4 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 9 | ERNIE-X1 | Baidu | 40.8 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 10 | DeepSeek-R1 | DeepSeek | 40.5 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
| 11 | Kimi k2 | Moonshot AI | 38.3 | web product | Paper Table 5 Reported by the authors, 16 Sept 2025 |
Papers and links
Last checked 1 Nov 2026. Something out of date or wrong? Send a correction.