ing point for text-only RL: RL improves MM CoT more than Text CoT on mathematics, and on reasoning, where both improve equally, MM CoT keeps its lead from SFT.
00100100200200606070708080
Accuracy (%)MATH-500
00100100200200101015152020
AIME 2025
001001002002005510101515
AIME 2026
00100100200200555560606565
BBH
0010010020020075758080
Accuracy (%)IFEval
00100100200200707075758080
RL stepMulti-IF
00100100200200202025253030
IFBenchMathematicsthinking onReasoningthinking onInstruction followingthinking off
Text CoT
MM CoT
Prod SFT
Figure 13: Held-out benchmarks during RL from three SFT checkpoints. Each group of panels comes from the runs on one prompt set, evaluated in the thinking mode used for training (Table 12). Each curve is a single RL run; step 0 is the SFT checkpoint, and we evaluate every 50 steps. We report accuracy on MATH-500, mean@32 on AIME, zero-shot exact match on BBH, and prompt-level strict accuracy on IFEval, Multi-IF, and IFBench. A response without a final answer counts as wrong. AIME 2026 excludes problem 1, which appears in DAPO-Math-17k.
0.20.20.30.30.40.4
Mean reward
0.20.20.30.3
0.30.30.40.40.50.5
0010010020020000224466881010
reasoning trace
Tokens (k)
0010010020020000224466
reasoning trace
RL step
00100100200200000.10.10.20.20.30.30.40.4
whole response
Mathematicsthinking onReasoningthinking onInstruction followingthinking off
Text CoT
MM CoT
Prod SFT
Figure 14: Training dynamics of the runs in Figure 13. The top row shows the mean reward of each step, and the bottom row the tokens per rollout, counting the reasoning trace when thinking is on and the whole response when it is off. Curves are 10-step trailing means from step 3, because the first two steps of asynchronous rollout collect the fastest and then the slowest rollouts, producing unrepresentative reward and length statistics.
5 Evaluations
This chapter evaluates the Apertus 1.5 models. Section 5.1 fixes the shared methodology: the harnesses, the benchmark suites, the judge models, and the pitfalls we control for. We then follow the models through the pipeline: the base models after continued pretraining, the post-trained models on general text capabilities including thinking mode, and their multilingual and translation behavior. The new modalities follow, image and audio understanding, and then the remaining capability areas: long context, tool use, and security and safety. Qualitative spot-testing closes the chapter with the behavior that benchmarks miss.
5.1 Evaluation Setup
We evaluate text capabilities with our fork of the LM Evaluation Harness (Gao et al., 2024), driven by a launcher layer that fixes task lists, prompt formats, and the serving configuration, so that runs are comparable across checkpoints. Visual and audio benchmarks run on forks of LMMs-Eval (Zhang et al., 2025) and VLMEvalKit (Duan et al., 2024) behind a common dispatcher. The forks and the task configurations we report are public. The text suite starts from the 27 benchmarks reported for Apertus 1.0 and grows to a release suite of roughly 50 tasks spanning general knowledge, instruction following and chat quality, mathematics, coding, reasoning, multilingual and cultural knowledge, bias, toxicity and safety, system-prompt guardrails, function calling, and alignment with the Apertus charter. Additions for this release include machine translation, hallucination, over-refusal, function calling, multi-turn instruction following, and the charter-alignment benchmark. Many knowledge and mathematics benchmarks use chain-of-thought variants scored on extracted answers, matching contemporary open-model practice. Before a benchmark enters the suite, we validate the implementation by reproducing published scores of openly released models; we exclude from the reported set tasks whose published scores we cannot reproduce, or whose answer extraction proves unreliable.
Benchmarks that require a judge use openly released models served on Alps: a Llama-3.3-70B-Instruct judge (e.g., for Alpac