KAIST AI; Financial Tech Lab, KakaoBank Corp., 2026
K-FinHallu
Tests whether an LLM can detect hallucinations in multi-turn Korean financial RAG dialogues. The taxonomy is built on answerability: contradictory, unverifiable, irrelevant, false refusal and false acceptance, with justified abstention counted as correct.
About this benchmark
- Size
- Test: 808 dialogues (2,064 queries). Train: 2,624 dialogues.
- Public split
- The paper says the benchmark is released, but no repository or dataset URL was found
- Grading
- Binary detection (accuracy, precision, recall, F1), four-class accuracy, and macro-F1 by hallucination type
- Licence
- CC-BY-NC-4.0 (stated in the paper; no release link found)
- Submission
- none
- Products allowed
- No
- Latest version
- arXiv v1 (2026-05-28)
- What it does not cover
- Tests the model as a detector of others' hallucinations, not its own answer accuracy. Korean only.
Results
10 modelsEach score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.
K-FinHallu arXiv v1 (2026-05-28)
binary detection F1, higher is better. 10 models.
Updated 28 May 2026
- Gemini 2.5 Flash0.860
- Gemini 2.5 Pro0.849
- gpt-50.823
- Llama-3.3-70B0.756
- Llama-3.1-70B0.735
- GPT-4o0.733
- Qwen3-32B0.732
- EXAONE-4.0-32B0.728
- Qwen3-14B0.707
- Llama-3.1-8B0.674
K-FinHallu arXiv v1 (2026-05-28)
binary detection F1, higher is better. 10 models.
Updated 28 May 2026
- 1Gemini 2.5 Flash0.860
- 2Gemini 2.5 Pro0.849
- 3gpt-50.823
- 4Llama-3.3-70B0.756
- 5Llama-3.1-70B0.735
- 6GPT-4o0.733
- 7Qwen3-32B0.732
- 8EXAONE-4.0-32B0.728
- 9Qwen3-14B0.707
- 10Llama-3.1-8B0.674
| 1 | Gemini 2.5 Flash | 0.860 | test split | Paper Table 3 Reported by the authors, 28 May 2026 | |
|---|---|---|---|---|---|
| 2 | Gemini 2.5 Pro | 0.849 | test split | Paper Table 3 Reported by the authors, 28 May 2026 | |
| 3 | gpt-5 | OpenAI | 0.823 | test split | Paper Table 3 Reported by the authors, 28 May 2026 |
| 4 | Llama-3.3-70B | Meta | 0.756 | test split | Paper Table 3 Reported by the authors, 28 May 2026 |
| 5 | Llama-3.1-70B | Meta | 0.735 | test split | Paper Table 3 Reported by the authors, 28 May 2026 |
| 6 | GPT-4o | OpenAI | 0.733 | test split | Paper Table 3 Reported by the authors, 28 May 2026 |
| 7 | Qwen3-32B | Qwen | 0.732 | test split | Paper Table 3 Reported by the authors, 28 May 2026 |
| 8 | EXAONE-4.0-32B | LG | 0.728 | test split | Paper Table 3 Reported by the authors, 28 May 2026 |
| 9 | Qwen3-14B | Qwen | 0.707 | test split | Paper Table 3 Reported by the authors, 28 May 2026 |
| 10 | Llama-3.1-8B | Meta | 0.674 | test split | Paper Table 3 Reported by the authors, 28 May 2026 |