ed: the model reasons and calls the tool inside the thinking span, and answers after the tool output returns. The released models do not yet produce this combination (Section 7.2). Line breaks are added for display.
4.2 Supervised Finetuning
We use SFT to develop assistant behavior over text, image, and speech inputs, with text output, using the shared conversation format of Section 4.1. The mixture covers instruction following, multilingual interaction, reasoning, code, tool use, safety, and model identity. We train the models to provide reasoning traces when thinking is enabled and direct answers when it is disabled.
To curate the text mixture, we train an auxiliary nonthinking checkpoint of Apertus 1.5 8B from its long-context pretrained checkpoint (Section 3.3), using text, image, and audio supervision with reasoning traces removed. We use this checkpoint for capability filtering: we evaluate selected verifiable text prompts and determine which receive verified reasoning traces.
For production SFT, we initialize Apertus 1.5 8B and Apertus 1.5 70B from their respective long-context pretrained checkpoints and use the text mixture curated with the auxiliary Apertus 1.5 8B checkpoint. Only the curated data carries over from the auxiliary run; neither production model inherits its weights. After production SFT, we continue training both models with dedicated long-context SFT to address degraded long-context retrieval (Section 4.3).
Both production SFT runs use a constant learning rate after 800 warmup iterations and a sequence length of 16,384 tokens: Apertus 1.5 8B trains at 6×10−56\times 10^{-5} with a global batch size of 256 for roughly 159B tokens, and Apertus 1.5 70B at 1×10−51\times 10^{-5} with a global batch size of 1,024 for roughly 155B tokens.66
6
Launch scripts for the production and long-context SFT runs are available at swiss-ai/pretrain-code. The auxiliary capability-filtering run instead decays a peak learning rate of 8×10−58\times 10^{-5} linearly, at a global batch size of 1,024.
Capability Filtering.
As outlined in the introduction to Section 4.2, we train an auxiliary nonthinking Apertus 1.5 8B checkpoint from its long-context pretrained checkpoint (Section 3.3), using text, image, and audio supervision with reasoning traces removed. We use this checkpoint to sample eight nonthinking answers for each selected verifiable prompt and check the final answers, extracted with the verifiable answer tool, against the reference. Prompts solved at least once (pass@8) remain in the mixture without a reasoning trace; prompts that fail all eight attempts receive a verified reasoning trace. Figure 9 illustrates this procedure. Our ablations motivate this selective use of reasoning traces: with thinking on, trace supervision adds seven to 18 points on mathematics but costs 12 to 14 on instruction following, mostly because about a quarter of traces degenerate into repetition (Section 4.7).
We apply capability filtering to the single-turn subset of our instruction-following (IF) mixture, OpenMathReasoning, DeepMath-103K, NuminaMath, and HARP. The share of prompts that remain without reasoning traces varies substantially by source: 99% for instruction following and roughly 10–40% for mathematics, with 11.9% for the 400k-prompt OpenMathReasoning subset, 38.3% for DeepMath-103K, 22.7% for NuminaMath, and 15.1% for HARP. We ablate k=1,…,8k=1,\dots,8; for kk below eight, the filtering rate on the mathematics datasets is negligible, so we select pass@8 for the final run.
Long-context checkpoint
pretrained Apertus 1.5 8B
SFT on the
non-reasoning mix
text, image, and audio data
with all reasoning traces removed
Non-reasoning checkpoint
answers every verifiable prompt
eight times, thinking off
Verifiable prompts
OpenMathReasoning, DeepMath,
NuminaMath, HARP, IF single-turn
Verify the eight answers
pass@8
Add a reasoning trace
generated by thinking
models and verified
Production text mixture
trains 8B and 70B
≥1\geq 1 of 88 correct
0