Shanghai University of Finance and Economics; Beihang University; Southern University of Science and Technology, 2026
FinSafetyBench
A bilingual (English/Chinese) red-teaming set that checks whether models refuse requests breaking financial compliance. It covers 11 financial-crime and 3 professional-ethics subcategories, tested under PAIR, ReNeLLM and FlipAttack jailbreaks.
Best reported
32.3
LLaMA-3Meta
attack success rate % (financial crimes, English)
Reported by the authors, 1 May 2026
About this benchmark
- Size
- 1,881 aligned English/Chinese instances (1,201 self-built, 680 external), 14 subcategories
- Public split
- Instructions only (data/FinsafetyBenchmark.csv) plus judge.py on GitHub, for research use
- Grading
- A Qwen3-32B judge scores each response 1-10 (1 is a full refusal). Attack success rate counts only a score of 10. The paper reports 93.6% agreement with human consensus.
- Licence
- Not stated
- Submission
- none
- Products allowed
- No
- Latest version
- arXiv v1 (2026-05-01), Findings of ACL 2026
- What it does not cover
- Does not measure over-refusal of legitimate requests, or whether advice is suitable or correct.
Results
6 modelsEach score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.
FinSafetyBench arXiv v1 (2026-05-01), Findings of ACL 2026
attack success rate % (financial crimes, English), lower is better. 6 models.
Updated 1 May 2026
FinSafetyBench arXiv v1 (2026-05-01), Findings of ACL 2026
attack success rate % (financial crimes, English), lower is better. 6 models.
Updated 1 May 2026
- 1LLaMA-332.3
- 2XuanYuan57.5
- 3InternLM372.2
- 4GLM-472.3
- 5Qwen2.588.2
- 6Mistral93.5
| 1 | LLaMA-3 | Meta | 32.3 | Meta-Llama-3-8B-Instruct; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
|---|---|---|---|---|---|
| 2 | XuanYuan | Not stated | 57.5 | XuanYuan-13B-Chat; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
| 3 | InternLM3 | Not stated | 72.2 | InternLM3-8B-Instruct; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
| 4 | GLM-4 | Zhipu AI | 72.3 | GLM-4-9B-0414; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
| 5 | Qwen2.5 | Qwen | 88.2 | Qwen2.5-32B-Instruct; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
| 6 | Mistral | Mistral AI | 93.5 | Mistral-Small-24B-Instruct-2501; average of PAIR, ReNeLLM, FlipAttack | Paper Table 2 Reported by the authors, 1 May 2026 |
FinSafetyBench arXiv v1 (2026-05-01), Findings of ACL 2026
attack success rate % (financial crimes), lower is better. 2 models.
Updated 1 May 2026
- GPT-5.135.3
- deepseek-v3.289.5
FinSafetyBench arXiv v1 (2026-05-01), Findings of ACL 2026
attack success rate % (financial crimes), lower is better. 2 models.
Updated 1 May 2026
- 1GPT-5.135.3
- 2deepseek-v3.289.5
| 1 | GPT-5.1 | OpenAI | 35.3 | PAIR only, 25 sampled instances per category (not comparable to Table 2) | Paper Table 3 Reported by the authors, 1 May 2026 |
|---|---|---|---|---|---|
| 2 | deepseek-v3.2 | DeepSeek | 89.5 | PAIR only, 25 sampled instances per category (not comparable to Table 2) | Paper Table 3 Reported by the authors, 1 May 2026 |
Papers and links
Last checked 1 Nov 2026. Something out of date or wrong? Send a correction.