NYU Shanghai; National University of Singapore; Yale University; NYU Center for Data Science, 2025

FinTrust

Measures how trustworthy LLMs are in finance across seven dimensions: truthfulness, safety (jailbreaks over 10 financial-crime categories), fairness, robustness, privacy, transparency or fiduciary disclosure, and knowledge discovery.

Best reported

1

o4-miniOpenAI

safety harmfulness score (1-10, GPT-4.1 mini judge)

Reported by the authors, 17 Oct 2025

About this benchmark

Size
15,680 instances across 7 dimensions
Public split
Full dataset on Hugging Face. The GitHub repo has per-dimension scripts and 100-case samples.
Grading
Per dimension. GPT-4.1 mini acts as judge (safety 1-10, informativeness 1-5 and others), with rule-based scoring for hallucination and certainty and Cohen's d for fairness. There is no aggregate score.
Licence
MIT (Hugging Face dataset card). The GitHub repo declares no licence.
Submission
none
Products allowed
No
Latest version
arXiv v1 (2025-10-17), EMNLP 2025 Main
What it does not cover
No single headline score and no agentic or tool-using evaluation. Evaluates bare models only.

Results

11 models
Leaderboard

Each score shows who reported it: the benchmark's authors (quoted from their paper), the benchmark's official board, or Moneda (run with the benchmark's own grader and rules). We do not copy other aggregators.

FinTrust arXiv v1 (2025-10-17), EMNLP 2025 Main

safety harmfulness score (1-10, GPT-4.1 mini judge), lower is better. 11 models.

Updated 17 Oct 2025

  1. o4-mini1
  2. XuanYuan-Fin-X1-Preview5.8
  3. Gemini 2.5 Flash6.2
  4. DianJin-R17.3
  5. GPT-4.17.4
  6. GPT-4.1-mini7.4
  7. Fin-R17.6
  8. Llama-4-Maverick-17B-128E-Instruct-FP87.6
  9. Qwen3-235B-A22B-FP88.5
  10. DeepSeek-R18.6
  11. DeepSeek-V39.0

FinTrust arXiv v1 (2025-10-17), EMNLP 2025 Main

safety harmfulness score (1-10, GPT-4.1 mini judge), lower is better. Top 10 of 11 models.

Updated 17 Oct 2025

  1. 1o4-mini1
  2. 2XuanYuan-Fin-X1-Preview5.8
  3. 3Gemini 2.5 Flash6.2
  4. 4DianJin-R17.3
  5. 5GPT-4.17.4
  6. 6GPT-4.1-mini7.4
  7. 7Fin-R17.6
  8. 8Llama-4-Maverick-17B-128E-Instruct-FP87.6
  9. 9Qwen3-235B-A22B-FP88.5
  10. 10DeepSeek-R18.6
FinTrust leaderboard (safety harmfulness score (1-10, GPT-4.1 mini judge), lower is better)
1o4-miniOpenAI1genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
2XuanYuan-Fin-X1-PreviewNot stated5.8genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
3Gemini 2.5 FlashGoogle6.2genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
4DianJin-R1Not stated7.3genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
5GPT-4.1OpenAI7.4genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
6GPT-4.1-miniOpenAI7.4genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
7Fin-R1Not stated7.6genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
8Llama-4-Maverick-17B-128E-Instruct-FP8Meta7.6genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
9Qwen3-235B-A22B-FP8Qwen8.5genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
10DeepSeek-R1DeepSeek8.6genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025
11DeepSeek-V3DeepSeek9.0genetic-algorithm jailbreakPaper Table 7 (App. A.2)
Reported by the authors, 17 Oct 2025

Last checked 1 Nov 2026. Something out of date or wrong? Send a correction.