 open-model practice. Before a benchmark enters the suite, we validate the implementation by reproducing published scores of openly released models; we exclude from the reported set tasks whose published scores we cannot reproduce, or whose answer extraction proves unreliable.
Benchmarks that require a judge use openly released models served on Alps: a Llama-3.3-70B-Instruct judge (e.g., for AlpacaEval), Llama-2-13B-cls for HarmBench (Mazeika et al., 2024), and a Llama-Guard-4 safety classifier for RealToxicityPrompts (Gehman et al., 2020). Where the original benchmark prescribes a different judge, absolute scores are not directly comparable with the original leaderboard, and the affected sections say so.
We evaluate post-trained models through the chat template. Thinking mode is deactivated by default, and we report it separately. During development we additionally monitor a held-out set of benchmarks outside the training distribution (Table 27) to detect overfitting of the post-training recipe to the development suite.
Known Pitfalls.
Several classes of error distort benchmark numbers, and we control for them explicitly. Harness defaults are tuned for base models: few-shot prompting, greedy decoding, and tight output-token budgets under-report post-trained models, and especially thinking models, whose answers are truncated or missed by answer extraction. Small benchmarks need repeated sampled runs before differences are meaningful. Tool-use scores depend on the parser matching the call format the model emits. Serving-side effects, such as prompt-formatting differences and checkpoint-conversion errors, shift scores silently; the two serving backends are not numerically identical, so no reported series mixes them.
The full list of benchmarks, including relevant settings for replication, is given in Appendix L.
5.2 Base Models
Table 13to Table 22 compare the Apertus 1.5 base checkpoints against their Apertus 1.0 predecessors on the identical suite, with open base models as context: the difference quantifies what continued pretraining adds in text capability while introducing the new modalities. Section 3.4 discusses how these capabilities evolve over the course of training.
The evaluations show that the final Apertus 1.5 base model improves on a wide range of tasks compared to Apertus 1.0. Apertus 1.5 8B specifically compares well with fully open peer models. In categories such as general language understanding and factual knowledge, Apertus base models have mostly closed the gap to other open-source models. Previously a weak point, performance for reasoning, math, and coding has strongly improved compared to the 1.0 model releases. Apertus models remain strong in multilingual tasks, despite small regressions.
Table 13: Pretraining Evaluation (8B-class models). Performance (%) on general language understanding tasks, higher is better. Bold = best overall; italic = best among fully open models.
Fully open
Open-weight
Benchmark
Apertus 1.0
Apertus 1.5
OLMo 3
Marin
Qwen3.5
Llama 3.1
Parameters
8B
8B
7B
8B
9B
8B
HellaSwag
78.8
78.1
74.1
82.4
79.3
79.3
PIQA
80.2
80.3
78.7
81.9
80.5
81.2
CommonsenseQA
62.3
62.3
58.7
70.3
83.6
70.4
LAMBADA
70.1
68.8
65.1
68.8
70.5
72.3
XNLI
44.1
45.3
39.6
42.4
39.8
45.6
XCOPA
65.6
64.9
55.1
57.0
64.1
62.5
Avg
66.9
66.6
61.9
67.1
69.6
68.6
Table 14: Pretraining Evaluation (70B-class models). Performance (%) on general language understanding tasks, higher is better. Bold = best overall; italic = best among fully open models.
Fully open
Open-weight
Benchmark
Apertus
1.0
Apertus
1.5
OLMo 3
EuroLLM
Marin
K2-V2
Qwen3.5
Nemotron
3.5
Parameters
70B
70B
32B
22B
32B
70B
27B
30B-A3B
HellaSwag
83.0
82.2
81.5
79.0
85.0
83.7
83.4
63.7
PIQA
83.0
82.3
81.3
81.4
83.3
83.9
82.3
83.1
CommonsenseQA
53.8
66.8
71.1
60.5
75.5
73.3
87.4
77.7
LAMBADA
72.9
72.9
70.5
70.9
68.7
76.1
74.2
72.8
XNLI
45.9
46.3
42.4
44.1
42.3
44.1
45.3
43.4
XCOPA
69.9
68.0
59.5
61.3
58.6
63.8
67.3
62.1
Avg
68.1
69.8
67.7
66.2
68.9
70.8
73.3
67.1
Tab