AfterQuery, 2025
Market-Bench
Models write executable backtesters from plain-language descriptions of three strategies: scheduled trading on MSFT, KO/PEP pairs trading, and MSFT delta hedging. Outputs are compared with a reference implementation.
About this benchmark
- Size
- 3 strategies (easy, medium, hard), 5 runs per model
- Public split
- Not stated
- Grading
- Mean absolute error of P&L, drawdown and position paths against a reference backtester (lower is better). Executability is reported as pass@1 and pass@3. Failed runs get a baseline MAE.
- Licence
- Not stated
- Submission
- Not stated
- Products allowed
- Not stated
- Latest version
- paper v2 (2026-01-20)
- What it does not cover
- Coding of introductory strategies only. No live trading, strategy quality or risk judgement.
Results
10 modelsWhat this does not measure: not investment performance. Returns come from a test setting over a fixed period, not from managing real money.
Each score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.
Market-Bench paper v2 (2026-01-20)
mean MAE, lower is better. 10 models.
Updated date not stated
- Grok 4443.2
- gpt-5.2969.4
- gemini-3-pro-preview1744.3
- gpt-5.1-codex-max4242.9
- DeepSeek-V3.24575.6
- Claude Sonnet 4.55126.8
- claude-opus-4.56039.6
- command-a6562.1
- nova-premier-v17740.3
- llama-3.1-nemotron-ultra9674.2
Market-Bench paper v2 (2026-01-20)
mean MAE, lower is better. 10 models.
Updated date not stated
- 1Grok 4443.2
- 2gpt-5.2969.4
- 3gemini-3-pro-preview1744.3
- 4gpt-5.1-codex-max4242.9
- 5DeepSeek-V3.24575.6
- 6Claude Sonnet 4.55126.8
- 7claude-opus-4.56039.6
- 8command-a6562.1
- 9nova-premier-v17740.3
- 10llama-3.1-nemotron-ultra9674.2
| 1 | Grok 4 | xAI | 443.2 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 |
|---|---|---|---|---|---|
| 2 | gpt-5.2 | OpenAI | 969.4 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 |
| 3 | gemini-3-pro-preview | 1744.3 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 | |
| 4 | gpt-5.1-codex-max | OpenAI | 4242.9 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 |
| 5 | DeepSeek-V3.2 | DeepSeek | 4575.6 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 |
| 6 | Claude Sonnet 4.5 | Anthropic | 5126.8 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 |
| 7 | claude-opus-4.5 | Anthropic | 6039.6 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 |
| 8 | command-a | Cohere | 6562.1 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 |
| 9 | nova-premier-v1 | Amazon | 7740.3 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 |
| 10 | llama-3.1-nemotron-ultra | Meta | 9674.2 | Default | marketbench.ai leaderboard Official board, reported date not stated · checked 1 Nov 2026 |
Papers and links
Last checked 1 Nov 2026. Something out of date or wrong? Send a correction.