New York University; Yale University; NYU Shanghai, 2025
FinLFQA
Long-form answers to financial questions that compare two companies' 10-Qs. Answers must be attributed in three ways: supporting evidence from the filings, executable code for each calculation step, and the domain knowledge used.
About this benchmark
- Size
- 1,008 expert-annotated examples (302 dev, 706 test), 89 companies
- Public split
- Dev and test splits on Hugging Face, plus code on GitHub. The paper says test answers are kept private, which was not checked against the HF files.
- Grading
- Answer quality: ROUGE-L, BERTScore, a GPT-4o judge (out of 15) and numeric F1. Attribution: evidence precision/recall/F1, code-execution success and knowledge recall.
- Licence
- MIT (Hugging Face dataset card). The GitHub repo declares no licence.
- Submission
- Not stated
- Products allowed
- No
- Latest version
- arXiv v1 (2025-10-07), Findings of EMNLP 2025
- What it does not cover
- Attribution is to supplied filings, not open-web citation. Uses US 10-Q pairs only.
Results
8 modelsEach score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.
FinLFQA arXiv v1 (2025-10-07), Findings of EMNLP 2025
evidence attribution F1, higher is better. 8 models.
Updated 7 Oct 2025
- GPT-4o56.5
- Mistral-Small-24B55.3
- Llama-3.3-70B52.9
- Qwen2.5-72B46.6
- phi-446.1
- Mistral-8x22B42.9
- Llama-3.2-3B11.8
- Llama-3.2-1B1.5
FinLFQA arXiv v1 (2025-10-07), Findings of EMNLP 2025
evidence attribution F1, higher is better. 8 models.
Updated 7 Oct 2025
- 1GPT-4o56.5
- 2Mistral-Small-24B55.3
- 3Llama-3.3-70B52.9
- 4Qwen2.5-72B46.6
- 5phi-446.1
- 6Mistral-8x22B42.9
- 7Llama-3.2-3B11.8
- 8Llama-3.2-1B1.5
| 1 | GPT-4o | OpenAI | 56.5 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
|---|---|---|---|---|---|
| 2 | Mistral-Small-24B | Mistral AI | 55.3 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
| 3 | Llama-3.3-70B | Meta | 52.9 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
| 4 | Qwen2.5-72B | Qwen | 46.6 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
| 5 | phi-4 | Microsoft | 46.1 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
| 6 | Mistral-8x22B | Mistral AI | 42.9 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
| 7 | Llama-3.2-3B | Meta | 11.8 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
| 8 | Llama-3.2-1B | Meta | 1.5 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
FinLFQA arXiv v1 (2025-10-07), Findings of EMNLP 2025
LLM-as-judge answer score (/15), higher is better. 3 models.
Updated 7 Oct 2025
- GPT-4o13.5
- Llama-3.3-70B12.9
- Qwen2.5-72B12.9
FinLFQA arXiv v1 (2025-10-07), Findings of EMNLP 2025
LLM-as-judge answer score (/15), higher is better. 3 models.
Updated 7 Oct 2025
- 1GPT-4o13.5
- 2Llama-3.3-70B12.9
- 2Qwen2.5-72B12.9
| 1 | GPT-4o | OpenAI | 13.5 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
|---|---|---|---|---|---|
| 2 | Llama-3.3-70B | Meta | 12.9 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
| 2 | Qwen2.5-72B | Qwen | 12.9 | post-hoc attribution, test set | Paper Table 6 Reported by the authors, 7 Oct 2025 |
Papers and links
Last checked 1 Nov 2026. Something out of date or wrong? Send a correction.