# FinSafetyBench

A bilingual (English/Chinese) red-teaming set that checks whether models refuse requests breaking financial compliance. It covers 11 financial-crime and 3 professional-ethics subcategories, tested under PAIR, ReNeLLM and FlipAttack jailbreaks.

## Facts

- Maker: Shanghai University of Finance and Economics; Beihang University; Southern University of Science and Technology
- Year: 2026
- Use case: Financial advice and safety
- Size: 1,881 aligned English/Chinese instances (1,201 self-built, 680 external), 14 subcategories
- Public split: Instructions only (data/FinsafetyBenchmark.csv) plus judge.py on GitHub, for research use
- Grading: A Qwen3-32B judge scores each response 1-10 (1 is a full refusal). Attack success rate counts only a score of 10. The paper reports 93.6% agreement with human consensus.
- Licence: Not stated
- Submission: none
- Products allowed: No
- Latest version: arXiv v1 (2026-05-01), Findings of ACL 2026
- What it does not cover: Does not measure over-refusal of legitimate requests, or whether advice is suitable or correct.
- Last checked: 1 Nov 2026
- Links: paper https://arxiv.org/abs/2605.00706, data https://github.com/sustech-nlp/FinSafetyBench, code https://github.com/sustech-nlp/FinSafetyBench
- Directory page: https://benchmarks.finance/benchmarks/finsafetybench/
- Leaderboard: https://benchmarks.finance/leaderboards/finsafetybench/

## Results

### FinSafetyBench: attack success rate % (financial crimes, English) (lower is better, 6 models)

| Rank | Model | Provider | Score | Metric | Setting | Reported by | Source | URL | Date |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | LLaMA-3 | Meta | 32.3 | attack success rate % (financial crimes, English) | Meta-Llama-3-8B-Instruct; average of PAIR, ReNeLLM, FlipAttack | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2605.00706v1 | 1 May 2026 (checked 1 Nov 2026) |
| 2 | XuanYuan | Not stated | 57.5 | attack success rate % (financial crimes, English) | XuanYuan-13B-Chat; average of PAIR, ReNeLLM, FlipAttack | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2605.00706v1 | 1 May 2026 (checked 1 Nov 2026) |
| 3 | InternLM3 | Not stated | 72.2 | attack success rate % (financial crimes, English) | InternLM3-8B-Instruct; average of PAIR, ReNeLLM, FlipAttack | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2605.00706v1 | 1 May 2026 (checked 1 Nov 2026) |
| 4 | GLM-4 | Zhipu AI | 72.3 | attack success rate % (financial crimes, English) | GLM-4-9B-0414; average of PAIR, ReNeLLM, FlipAttack | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2605.00706v1 | 1 May 2026 (checked 1 Nov 2026) |
| 5 | Qwen2.5 | Qwen | 88.2 | attack success rate % (financial crimes, English) | Qwen2.5-32B-Instruct; average of PAIR, ReNeLLM, FlipAttack | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2605.00706v1 | 1 May 2026 (checked 1 Nov 2026) |
| 6 | Mistral | Mistral AI | 93.5 | attack success rate % (financial crimes, English) | Mistral-Small-24B-Instruct-2501; average of PAIR, ReNeLLM, FlipAttack | Reported by the authors | Paper Table 2 | https://arxiv.org/html/2605.00706v1 | 1 May 2026 (checked 1 Nov 2026) |

### FinSafetyBench: attack success rate % (financial crimes) (lower is better, 2 models)

| Rank | Model | Provider | Score | Metric | Setting | Reported by | Source | URL | Date |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | GPT-5.1 | OpenAI | 35.3 | attack success rate % (financial crimes) | PAIR only, 25 sampled instances per category (not comparable to Table 2) | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.00706v1 | 1 May 2026 (checked 1 Nov 2026) |
| 2 | DeepSeek-V3.2 | DeepSeek | 89.5 | attack success rate % (financial crimes) | PAIR only, 25 sampled instances per category (not comparable to Table 2) | Reported by the authors | Paper Table 3 | https://arxiv.org/html/2605.00706v1 | 1 May 2026 (checked 1 Nov 2026) |


Source: https://benchmarks.finance/benchmarks/finsafetybench/

Created and maintained by Moneda.