iative. Extended from Apertus 1 via continued pretraining, you understand images and audio and respond in text.
Independent of the system message, the template additionally emits a developer message that states whether the thinking mode is enabled and declares the available tools (Section 4.1).
Appendix L Evaluation Details
We report additional evaluation details to enable the replication of results. Furthermore, we report additional benchmark scores for ablations.
Text-based benchmarks can be reproduced at commit 4ac31da of the Swiss AI swiss-ai/lm-evaluation-harness fork of lm-evaluation-harness. We used vLLM as the inference framework to run the models. We generally sample all generative tasks greedily (temperature 0.00.0), with the exception of thinking-enabled evaluations, which use the recommended temperature and top-k settings. We score multiple-choice benchmarks either as loglikelihood requests, using the predicted output probabilities to score each answer option, or generatively, by exact match on the answer extracted from the model’s response.
To evaluate general language understanding of pretrained models, we use HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), CommonsenseQA (Talmor et al., 2019), LAMBADA (Paperno et al., 2016), XNLI (Conneau et al., 2018), and XCOPA (Ponti et al., 2020). Furthermore, we evaluated MMLU with chain-of-thought prompting (Hendrycks et al., 2021b), Global-MMLU (Singh et al., 2025), MMLU-Pro (Wang et al., 2024), AGIEval (Zhong et al., 2024), SQuADv2 (Rajpurkar et al., 2018), and TruthfulQA (Lin et al., 2022), including some multilingual versions generated by machine translation. Finally, we evaluate some safety-related benchmarks, including BBQ (Parrish et al., 2022), ToxiGen (Hartvigsen et al., 2022), and WMDP (Li et al., 2024b), and we use CulturalBench (Chiu et al., 2025) to measure wider cultural knowledge.
For post-trained models, we evaluated knowledge via MMLU (Hendrycks et al., 2021b), MMLU-Pro (Wang et al., 2024), AGIEval (Zhong et al., 2024), ARC-Challenge (Clark et al., 2018), and TruthfulQA (Lin et al., 2022).
We evaluated instruction following and chat with IFEval (Zhou et al., 2023), Multi-IF (He et al., 2024), IFBench (Pyatkin et al., 2025), and AlpacaEval (Li et al., 2023b; Dubois et al., 2024). Math capabilities were evaluated with GSM8K (Cobbe et al., 2021), MATH, and a sub-sampled MATH500 (Hendrycks et al., 2021c). For coding, we evaluated models on HumanEval and MBPP (Austin et al., 2021). Reasoning was benchmarked via BBH (Suzgun et al., 2023), GPQA (Rein et al., 2024), and DROP (Dua et al., 2019). Finally, multilingual and cultural understanding were evaluated via Global-MMLU (Singh et al., 2025) and an in-house benchmark focusing on Switzerland called SwitzerlandQA, which we will release in the future.
For thinking-model benchmarks, we additionally evaluated AIME 2024 and 2025 (Dekoninck et al., 2026).
Table L.1and Table L.2 contain a list of all used benchmarks, including replication-relevant settings, such as few-shot examples.
L.1 Full Post-Training Lineage
Section 5.3.2tracks the Apertus 1.5 models through the post-training stages on a representative subset of each skill area. We give the same lineage over the full development suite here. The held-out and thinking-mode tables of that section already list every benchmark of their suite, so they have no fuller form here. We compute the behavioral measures once per checkpoint rather than per suite, and they appear at the foot of Table L.3.
Table L.1: Benchmark configurations used in the Apertus evaluation suites. Accuracy† denotes prompt-level loose accuracy.
Benchmark
Metric
Type
CoT
#Shots
Chat
Turns
#Langs
Knowledge and Reasoning
MMLU (zero-shot)
Exact Match
MCQ (Gen)
✓
0
✓
×\times
1
MMLU (few-shot)
Exact Match
MCQ (Gen)
✓
4
×\times
×\times
1
MMLU-Pro
Exact Match
MCQ (Gen)
✓
5
✓
✓
1
AGIEval
Accuracy
MCQ (Gen/LLH)
×\times
0
✓
×\times
2
ARC Challenge Chat
Exact Match
MCQ (Gen)
×\times
0
✓
×\times
1
Trut