Leaderboard
FinSafetyBench
A bilingual (English/Chinese) red-teaming set that checks whether models refuse requests breaking financial compliance. It covers 11 financial-crime and 3 professional-ethics subcategories, tested under PAIR, ReNeLLM and FlipAttack jailbreaks.
Best reported
32.3
LLaMA-3
attack success rate % (financial crimes, English)
1 May 2026
FinSafetyBench arXiv v1 (2026-05-01), Findings of ACL 2026
attack success rate % (financial crimes, English), lower is better. 6 models.
Updated 1 May 2026
- LLaMA-332.3
- XuanYuan57.5
- InternLM372.2
- GLM-472.3
- Qwen2.588.2
- Mistral93.5
FinSafetyBench arXiv v1 (2026-05-01), Findings of ACL 2026
attack success rate % (financial crimes, English), lower is better. 6 models.
Updated 1 May 2026
- 1LLaMA-332.3
- 2XuanYuan57.5
- 3InternLM372.2
- 4GLM-472.3
- 5Qwen2.588.2
- 6Mistral93.5
| 1 | LLaMA-3 | Meta | 32.3 | Meta-Llama-3-8B-Instruct; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
|---|---|---|---|---|---|
| 2 | XuanYuan | Not stated | 57.5 | XuanYuan-13B-Chat; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
| 3 | InternLM3 | Not stated | 72.2 | InternLM3-8B-Instruct; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
| 4 | GLM-4 | Zhipu AI | 72.3 | GLM-4-9B-0414; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
| 5 | Qwen2.5 | Qwen | 88.2 | Qwen2.5-32B-Instruct; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
| 6 | Mistral | Mistral AI | 93.5 | Mistral-Small-24B-Instruct-2501; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
FinSafetyBench arXiv v1 (2026-05-01), Findings of ACL 2026
attack success rate % (financial crimes), lower is better. 2 models.
Updated 1 May 2026
- GPT-5.135.3
- deepseek-v3.289.5
FinSafetyBench arXiv v1 (2026-05-01), Findings of ACL 2026
attack success rate % (financial crimes), lower is better. 2 models.
Updated 1 May 2026
- 1GPT-5.135.3
- 2deepseek-v3.289.5
| 1 | GPT-5.1 | OpenAI | 35.3 | PAIR only, 25 sampled instances per category (not comparable to Table 2) | Paper Table 3 Reported by the authors, 1 May 2026 |
|---|---|---|---|---|---|
| 2 | deepseek-v3.2 | DeepSeek | 89.5 | PAIR only, 25 sampled instances per category (not comparable to Table 2) | Paper Table 3 Reported by the authors, 1 May 2026 |