# K-FinHallu

Tests whether an LLM can detect hallucinations in multi-turn Korean financial RAG dialogues. The taxonomy is built on answerability: contradictory, unverifiable, irrelevant, false refusal and false acceptance, with justified abstention counted as correct.

## Facts

- Maker: KAIST AI; Financial Tech Lab, KakaoBank Corp.
- Year: 2026
- Use case: Financial Verifiability
- Size: Test: 808 dialogues (2,064 queries). Train: 2,624 dialogues.
- Public split: The paper says the benchmark is released, but no repository or dataset URL was found
- Grading: Binary detection (accuracy, precision, recall, F1), four-class accuracy, and macro-F1 by hallucination type
- Licence: CC-BY-NC-4.0 (stated in the paper; no release link found)
- Submission: none
- Products allowed: No
- Latest version: arXiv v1 (2026-05-28)
- What it does not cover: Tests the model as a detector of others' hallucinations, not its own answer accuracy. Korean only.
- Last checked: 1 Nov 2026
- Links: paper https://arxiv.org/abs/2605.29523
- Directory page: https://benchmarks.finance/benchmarks/k-finhallu/
- Leaderboard: https://benchmarks.finance/leaderboards/k-finhallu/

## Results

### K-FinHallu: binary detection F1 (higher is better, 10 models)

| Rank | Model | Provider | Score | Metric | Setting | Reported by | Source | URL | Date |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | Gemini 2.5 Flash | Google | 0.860 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |
| 2 | Gemini 2.5 Pro | Google | 0.849 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |
| 3 | GPT-5 | OpenAI | 0.823 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |
| 4 | Llama-3.3-70B | Meta | 0.756 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |
| 5 | Llama-3.1-70B | Meta | 0.735 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |
| 6 | GPT-4o | OpenAI | 0.733 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |
| 7 | Qwen3-32B | Qwen | 0.732 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |
| 8 | EXAONE-4.0-32B | LG | 0.728 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |
| 9 | Qwen3-14B | Qwen | 0.707 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |
| 10 | Llama-3.1-8B | Meta | 0.674 | binary detection F1 | test split | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.29523v1 | 28 May 2026 (checked 1 Nov 2026) |


Source: https://benchmarks.finance/benchmarks/k-finhallu/

Created and maintained by Moneda.