S-IFEval
Accuracy‡
Gen
×\times
0
✓
×\times
1
RealGuardrails TensorTrust
Pass rate
Gen
×\times
0
✓
×\times
1
RealGuardrails S-RuLES
Pass rate
Gen
×\times
0
✓
✓
1
Swiss AI Charter Alignment
Length-controlled win rate
Gen
×\times
0
✓
×\times
1
Tool Use
BFCL v3
Accuracy
Gen
×\times
0
✓
✓
1
Table L.3: Full post-training lineage of the Apertus 1.5 models on the development benchmark suite, all 37 benchmarks, thinking disabled, with the behavioral measures of Section 5.1. The subset shown in the main text is Table 26. Best value per row in bold, taken separately within each model. Over-refusal is the refusal rate on the hard subset of OR-Bench; a share of its prompts is legitimately refusable, so only the trend across stages is meaningful (Section 5.9.2).
Apertus 1.5 8B
Apertus 1.5 70B
Benchmark
SFT
+RL
+DPO
+SDPO
SFT
+RL
+DPO
Knowledge and Reasoning
MMLU
65.8
66.6
70.7
70.6
75.1
72.0
77.2
MMLU-Pro
40.1
43.0
49.4
49.2
46.3
44.4
56.9
TruthfulQA
48.1
50.8
58.0
56.1
58.2
58.1
67.0
CommonsenseQA
50.8
27.6
75.5
75.3
64.8
73.9
77.0
SQuADv2
10.1
13.5
30.8
32.0
10.9
31.9
19.1
HellaSwag
72.7
71.4
66.9
66.6
76.6
75.8
72.8
BBH
67.9
69.1
71.1
68.2
72.4
75.2
66.7
ACPBench
26.1
31.2
37.9
37.1
38.3
49.7
63.6
DROP
53.2
56.0
55.2
54.1
57.7
57.8
60.6
Instruction Following and Chat
IFEval
80.2
85.8
89.6
90.0
83.5
90.8
89.8
Multi-IF
74.0
83.0
84.5
85.0
78.0
86.3
85.7
AlpacaEval
20.0
20.6
32.7
42.7
22.6
25.9
59.4
Arena-Hard v0.1
18.9
19.7
33.6
33.8
23.5
27.4
66.2
Arena-Hard v2
2.3
2.1
2.1
2.6
2.3
2.3
7.8
Math
GSM8K
77.0
79.5
80.7
78.6
78.6
84.7
84.8
MATH
35.2
39.7
44.6
41.4
38.0
45.8
52.8
Minerva MATH
42.9
48.8
54.3
50.8
44.6
55.0
64.3
MathQA
46.1
46.3
46.7
46.1
48.2
47.9
48.2
MATH-500
43.2
47.0
54.2
50.2
44.2
56.2
63.2
AIME 24
3.3
3.3
6.7
13.3
0.0
13.3
13.3
AIME 25
3.3
6.7
13.3
13.3
6.7
0.0
10.0
Coding
HumanEval
85.3
86.0
83.2
85.5
89.6
91.8
88.6
MBPP
48.6
51.6
51.2
52.6
54.2
56.4
62.2
Multilingual and Cultural
Global-MMLU
55.8
55.4
58.9
59.3
64.0
61.7
66.6
MGSM
53.3
57.5
66.7
66.2
59.8
70.3
67.7
Mult. TruthfulQA
46.7
48.7
52.4
51.7
50.8
50.6
56.9
INCLUDE-44
56.9
56.7
56.6
56.9
60.4
59.9
61.6
INCLUDE-45
40.2
40.9
39.7
39.5
42.4
42.5
41.7
XNLI
45.8
45.3
43.8
43.7
46.6
46.3
42.4
BLEnD
65.1
66.5
67.5
67.3
67.2
68.0
69.9
CulturalBench
71.5
72.6
71.6
72.0
75.6
55.3
77.1
SwitzerlandQA
65.7
64.4
64.6
64.6
68.5
67.6
69.8
Safety and Alignment
BBQ
62.9
67.4
67.8
67.6
65.7
68.1
80.5
ToxiGen
53.8
73.3
81.9
81.7
77.7
83.8
86.8
WMDP
51.7
50.7
53.4
53.4
54.6
56.1
56.9
Charter Alignment
33.4
34.3
56.8
66.7
41.7
50.9
72.2
Tool Use
BFCL v3
32.2
39.8
52.9
48.2
43.3
43.5
56.0
Overall Average
47.3
49.3
54.8
55.0
52.2
55.3
61.2
Mean response length (tokens)
347.4
358.7
338.2
346.8
367.2
252.9
495.3
Degeneration (%)
2.9
1.7
0.6
1.1
11.2
10.6
3.3
Over-Refusal (%)
3.6
7.6
79.0
81.5
2.9
24.7
67.6
Appendix M Pre-Release Manual Evaluations
M.1 General Information About Pre-Release Evaluation
We perform the manual evaluation on all release-candidate models prior to their public release, and we provide the full manual evaluation report, along with the recommendation from the safety and security lead as to the model release, to inform a release/no-release decision.
In every test case below, we evaluate the model behavior in English, three common languages of the Confederation (German, French, and Italian), and Serbian (Cyrillic Serbo-Croatian), as a low-resource language with a different alphabet and notable usage within Switzerland.
For all evaluations of usage scenarios assuming malicious users (CBRNE, Dual Use, Disinformation, and Child Sexual Abuse enablement), we evaluate abliterated versions of the release candidates alongside the release candidates to confirm the absence of dangerous capabilities independently of the model alignment. For non-abliterated models, we evaluate their helpfulness through multi-turn, multi-conversation interactions with a malicious user faking benign interest.
In addition to the internal safety and security evaluation, we provide early access to the 