9.5
55.5
82.0
74.0
0.0
88.0
88.5
89.5
Live parallel
56.3
12.5
31.3
81.3
50.0
0.0
81.3
43.8
68.8
Tool Rejection
Irrelevance
36.3
50.0
69.6
67.9
94.2
41.3
60.4
44.2
65.4
Live irrelevance
29.0
46.7
66.4
45.7
85.3
32.8
51.9
30.5
39.5
5.9 Security, Safety, and Interpretability
As Apertus is a fully open and massively multilingual foundation model, the safety evaluations of the Apertus family present a number of unique challenges, initially outlined for the Apertus 1.0 release (Hernández-Cano et al., 2026). While most current LLM literature, standards, and regulations focus on LLM deployments, Apertus is an open foundation LLM, for which release risks have to be evaluated accordingly. First, any security mechanisms relying on alignment, such as refusal, can be removed by post-release training, so the intrinsic model usefulness for dangerous capabilities must be evaluated independently of alignment (O’Brien et al., 2026). Second, pre-release evaluations of models’ safety are inherently distinct from performance evaluation. While benchmarks can serve as good general proxies of performance, security evaluations seek to evaluate the severity and likelihood of potential harms arising from the model release, with a single high-likelihood, high-impact risk being able to block the model release.
To ensure the released model’s safety in light of those considerations, we approach model safety in three stages.
First, we design the post-training to ensure safety in non-adversarial usage scenarios. The SFT mixture carries only about 16,000 refusal-style safety samples out of 3.7 million (Section 4.2), so safety behavior is concentrated in the alignment stages, most directly in the charter alignment of SDPO. Over-refusal is tracked as the counterweight throughout development. Of the safety suite, only BBQ, ToxiGen, and WMDP inform alignment iteration; every other safety benchmark is held out as an out-of-distribution check.
Second, we evaluate release-candidate models against standardized safety and security benchmarks. Notably, we use HarmBench Direct Requests and Human Jailbreaks (referred to as HarmBench**) as a proxy for the safety of the release candidates in non-adversarial use, thanks to good coverage of general harm categories (Mazeika et al., 2024). We use WMDP as a proxy for intrinsically dangerous knowledge, but without seeking to optimize it, due to overlap with general model capabilities (Li et al., 2024b). Finally, we use BBQ, ToxiGen, and RealToxicityPrompts*1010
10
For scalability, we randomly subsample RealToxicityPrompts and use the LLaMA-Guard3 judge model, as reported in the Apertus 1.0 technical report. to measure the potential for unprompted and unexpected toxicity in English, and PolyGloToxicityPrompts to validate it in other languages (Parrish et al., 2022; Hartvigsen et al., 2022; Gehman et al., 2020; Jain et al., 2024).
Finally, we perform a blocking manual evaluation of every release-candidate model and provide early access to independent third parties. As part of the manual evaluation, we notably evaluate CBRNE, dual-use, and child sexual abuse usefulness, ensuring a high coverage of risk criteria outlined in NIST-RMF GenAI 600.1 (Autio et al., 2024) and ML Commons Risk 1.0 (Ghosh et al., 2025) (cf. Table N.1 in Appendix N for detailed mapping). Consistent with the concern outlined above for open-weight model release, to ensure that we evaluate the intrinsic capabilities of the models rather than alignment, we create and evaluate abliterated versions of each release-candidate model (Arditi et al., 2024) and provide access to them to external evaluators. Due to the nature of evaluation of usefulness for inherently malicious use, the manual evaluation criteria per se can be useful to malicious parties and hence are not publicly released. However, more detailed information about the manual evaluation and third parties is available in Appendix M, and the pre-release manual review report is available upon request.
5.9.1 Saf