hem to external evaluators. Due to the nature of evaluation of usefulness for inherently malicious use, the manual evaluation criteria per se can be useful to malicious parties and hence are not publicly released. However, more detailed information about the manual evaluation and third parties is available in Appendix M, and the pre-release manual review report is available upon request.
5.9.1 Safety Benchmark Performance
We report safety benchmark performance in Table 38 for the released checkpoints, their predecessors, and open-weight anchors, using the served classifier and judge models of Section 5.1. The alignment stages move the metrics in the intended direction for Apertus 1.5 vs 1.0: ToxiGen rises by 11.4% for the 8B and 24.3% for the 70B, BBQ improves at both sizes with the 70B reaching the best score in the comparison, and the judged toxicity of generated continuations is low throughout. WMDP rises slightly alongside general capability (2.1% and 1.8% for 8B and 70B respectively).
On HarmBench, the released 8B resists direct harmful requests (1.3% attack success) but remains vulnerable to human-written jailbreaks (31.5%), for an average request success rate of 26.5%. This is consistent with the main defense against malicious misuse of the Apertus family being capability limitations, rather than alignment, and is still an improvement of 8.7% and 2.8% for the 8B and 70B respectively. Finally, we evaluate Apertus 1.5 release candidates for multimodal jailbreak robustness, leveraging MM-SafetyBench (Liu et al., 2025a), obtaining scores of 38.0% and 28.5% for the 8B and 70B respectively (lower is better). Comparatively, Qwen3-VL-8B, Molmo2-8B, Gemma4-12B, and EuroVLM-9B achieve 22.0%, 25.1%, 18.0%, and 52.1%, respectively, positioning the released models within the normal distribution of similarly sized open-weight models (full data in Table 34; note that the values are inverted in the table to facilitate comparison).
Table 38: Safety evaluation of the released checkpoints against their predecessors and open-weight anchors. All values in percent; WMDP reports accuracy on hazardous-knowledge questions without a preferred direction, and is excluded from bolding.
Apertus
Benchmark
1.0 8B
1.5 8B
1.0 70B
1.5 70B
OLMo 3
Qwen3.5
Llama 3.3
Parameters
8B
8B
70B
70B
7B
9B
70B
Higher Is Better
BBQ
65.2
67.6
68.6
80.5
73.2
75.3
72.0
ToxiGen
70.4
81.8
62.5
86.8
83.6
74.8
87.3
Lower Is Better
PolyGloToxicityPrompts (toxicity)
4.7
7.9
3.7
7.0
7.0
3.9
6.3
RealToxicityPrompts* (toxicity)
0.2
0.5
0.2
0.5
0.3
0.5
1.1
HarmBench** (attack success)
35.2
26.5
31.9
29.1
21.0
1.6
39.2
No Preferred Direction
WMDP (accuracy)
51.4
53.5
54.9
56.7
49.8
45.8
57.1
5.9.2 Over-Refusal and Hallucination
Two additions to the evaluation suite measure failure modes adjacent to safety. We measure over-refusal, the refusal of benign requests, on the hard subset of OR-Bench (Cui et al., 2025a) as a counterweight to the refusal-based safety benchmarks. Refusal rates rise from 3.6% after SFT to 81.5% after alignment at 8B and from 2.9% to 67.6% at 70B (Table L.3). Most of the rise comes at the DPO stage; at 70B, RL already raises the rate to 24.7%. A share of the benchmark’s prompts is legitimately refusable, so we report this trend rather than an absolute over-refusal rate (Section 7.2). We measure hallucination with HalluLens (Bang et al., 2025). Our judge is a locally served Qwen3.5-27B, whereas the original benchmark uses a Llama-3.1-405B, so absolute scores are not comparable with the official leaderboard; a cross-check with a Llama-3.3-70B judge moves the rates by at most two points. Table 39 shows a split result. Where the failure is answering about things that do not exist, Apertus 1.5 improves sharply: abstention on fabricated entities more than triples at 8B, and long-form generation becomes both more precise and more willing to abstain, gaining six points of F1 at 8B and two at 70B. Short-form hallucination on questions about real entities does not impro