ctive parallel data (image-text and audio-transcript pairs) along with text-only data, and compute the loss over the text tokens only, since the models understand but never generate images and audio (Section 2); an ablation further shows that text-only targets during paired training also yield the strongest image understanding (Section 2.4). For the 8B, the Stage 2 text share rises to roughly two thirds, in part because multimodal data was still limited; the 70B, with its halved token budget, keeps a roughly even split between text and multimodal data, yet its text evaluations do not degrade. The third stage is the cooldown, in which we emphasize higher-quality data and include additional paired speech sources (Table C.2). The audio portion of the cooldown accounts for 95B input tokens for Apertus 1.5 8B and 92B for Apertus 1.5 70B (Table 3). These training counts include paired text and any repeated sampling; they do not measure unique hours of speech. The cooldown also includes a small share of instruction-formatted data (roughly 3B tokens), for which the loss is restricted to assistant tokens. We train the 70B model on half the intended number of tokens to free resources for the next Apertus pretraining project, keeping the cooldown length unchanged and reducing the token budget mainly during Stage 2.
Table 2: Stages 1 and 2 fall in the warmup and stable phases of the LR schedule, while Stage 3 is the decay phase. Stage 1 is trained to predict tokens of all modalities. Stages 2 and 3 are trained to predict only text tokens.
Stage
Data
Prediction Targets
1
Unpaired image, audio, and text data.
All modalities
2
Paired image-text, audio-transcript data, and text data
Text only
3
Higher-quality data, with some instruction-formatted data
Text only
Table 3: Token budget per modality across the three curriculum stages of Apertus 1.5 8B and Apertus 1.5 70B, that is, the mix each run trains on. Percentages are the within-stage share of tokens. For Stages 2 and 3, the Vision and Audio counts include their paired text (image captions and audio transcripts); the Text column is the separate text-only data.
Stage
Vision
Audio
Text
Total
Apertus 1.5 8B
1
367B (45.6%)
110B (13.7%)
328B (40.7%)
806B
2
852B (30.4%)
89B (3.2%)
1,861B (66.4%)
2,802B
3
78B (19.4%)
95B (23.4%)
230B (57.2%)
403B
Apertus 1.5 70B
1
49B (12.2%)
52B (13.0%)
301B (74.8%)
403B
2
512B (42.4%)
90B (7.4%)
606B (50.1%)
1,208B
3
107B (26.6%)
92B (22.7%)
204B (50.7%)
403B
The full text data mixture for each stage is given in Table 4. Stages 1 and 2 share a single mixture dominated by high-quality web text (HuggingFaceTB/dclm-edu and the top 10% of epfml/FineWeb2-HQ-Plus), supplemented by mathematics, code, translation, and legal corpora. The Stage 3 cooldown deliberately trades general web text for higher-quality and reasoning-heavy data: the share of general web text is cut substantially, while the combined code-and-mathematics allocation rises sharply, driven by larger allocations to mathematics (HuggingFaceTB/finemath, IFM/MegaMath, Infi-MM/InfiMM-WebMath-40B) and code (HuggingFaceTB/stack-edu, nvidia/Nemotron-Pretraining-Code-v1). At the same time, the general legal corpora are dropped in favor of voilaj/swiss-caselaw, the translation share rises (HuggingFaceFW/finetranslations), and a small amount of Apertus 1.5 SFT data is mixed in.
Table 4: Text data mixture per training stage, as a percentage of each stage’s sampled text-only token budget.
Name
Stage 1+2 (%)
Stage 3 (%)
HuggingFaceTB/dclm-edu
47.5
20.5
HuggingFaceFW/finepdfs-edu
2.8
3.2
nvidia/Nemotron-CC-v2.1
1.0
7.2
PleIAs/SYNTH
0.7
0.0
epfml/FineWeb2-HQ-Plus (top 10%)
20.1
14.4
HuggingFaceFW/finepdfs-edu (multilingual only)
8.5
3.6
HuggingFaceFW/finetranslations
2.8
5.9
HuggingFaceTB/finemath
0.9
6.3
IFM/MegaMath
6.1
9.5
Infi-MM/InfiMM-WebMath-40B
0.6
4.2
HuggingFaceTB/stack-edu
2.1
6.2
nvidia/Nemotron-Pretraining-Code-v1
5.3
15.1
joelniklaus/legal-mc4
0.7
0.0
joelniklaus/Multi_Legal_Pile
1.0
0.0
voilaj/swiss-case