Surf AI / Cybertino Lab and Princeton University, 2025
CAIA (Crypto AI Agent Benchmark)
Time-anchored crypto analyst tasks (on-chain analysis, tokenomics and similar) in adversarial, misinformation-heavy conditions. Models are run with and without tools.
About this benchmark
- Size
- 178 tasks, 17 models
- Public split
- All 178 questions (benchmark.csv on GitHub; dataset on Hugging Face). The release uses anonymized mock tools.
- Grading
- Accuracy by majority vote over five runs, plus Pass@1 and Pass@5. Ground truth requires a reproducible tool-call sequence.
- Licence
- CC-BY-4.0 (data); Apache-2.0 (code)
- Submission
- Not stated
- Products allowed
- Not stated
- Latest version
- 0927 release; paper v2 (2026-01-17)
- What it does not cover
- Crypto only. No equities or filings, no portfolio outcomes, and the public release uses mock rather than live tools.
Results
10 modelsEach score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.
CAIA (Crypto AI Agent Benchmark) 0927 release; paper v2 (2026-01-17)
accuracy % (majority vote, 5 runs), higher is better. 10 models.
Updated 17 Jan 2026
- gpt-567.4
- GPT-OSS 120B62.9
- Grok 4 Fast Reasoning61.2
- Claude Opus 457.3
- Claude Sonnet 456.7
- Claude Opus 4.156.3
- GPT-o353.4
- DeepSeek V3.149.2
- Kimi K2 Instruct47.2
- GPT-4.146.6
CAIA (Crypto AI Agent Benchmark) 0927 release; paper v2 (2026-01-17)
accuracy % (majority vote, 5 runs), higher is better. 10 models.
Updated 17 Jan 2026
- 1gpt-567.4
- 2GPT-OSS 120B62.9
- 3Grok 4 Fast Reasoning61.2
- 4Claude Opus 457.3
- 5Claude Sonnet 456.7
- 6Claude Opus 4.156.3
- 7GPT-o353.4
- 8DeepSeek V3.149.2
- 9Kimi K2 Instruct47.2
- 10GPT-4.146.6
| 1 | gpt-5 | OpenAI | 67.4 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
|---|---|---|---|---|---|
| 2 | GPT-OSS 120B | OpenAI | 62.9 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
| 3 | Grok 4 Fast Reasoning | xAI | 61.2 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
| 4 | Claude Opus 4 | Anthropic | 57.3 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
| 5 | Claude Sonnet 4 | Anthropic | 56.7 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
| 6 | Claude Opus 4.1 | Anthropic | 56.3 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
| 7 | GPT-o3 | OpenAI | 53.4 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
| 8 | DeepSeek V3.1 | DeepSeek | 49.2 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
| 9 | Kimi K2 Instruct | Moonshot AI | 47.2 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
| 10 | GPT-4.1 | OpenAI | 46.6 | with tools | Paper Tables 2-3 Reported by the authors, 17 Jan 2026 |
Papers and links
Last checked 1 Nov 2026. Something out of date or wrong? Send a correction.