
1,790,000,000
1,570,000,000
1,410,000,000
1,330,000,000
Total
62,910,000,000
58,720,000,000
58,720,000,000
58,720,000,000
Table F.2: Long-context data mixture (Apertus 1.5 70B). Per-dataset token counts for each long-context training step; columns denote the trained context length in tokens (32,768, 65,536, 131,072, and 262,144, respectively). We draw the Cooldown row and the short-context top-up share of Image tokens from documents under 16,384 tokens even though they count toward that step’s mixture. We sort rows within a class alphabetically.
Dataset
32k
65k
131k
262k
Cooldown
Cooldown Stage 3 Subsample
37,460,000,000
37,530,000,000
32,270,000,000
32,350,000,000
English
Biomed-Enriched
487,100,000
182,600,000
0
43,700,000
Dolma 3 OCR
2,660,000,000
2,540,000,000
185,600,000
0
FinePDFs-Edu (EN)
2,660,000,000
2,540,000,000
2,620,000,000
2,130,000,000
Institutional Books
0
507,000,000
2,190,000,000
2,820,000,000
Multilingual
FinePDFs-Edu (multi)
1,290,000,000
1,700,000,000
1,570,000,000
1,880,000,000
FineTranslations (edu)
1,290,000,000
732,200,000
209,700,000
121,400,000
FineWeb-2 (10% HQ)
1,290,000,000
1,450,000,000
1,570,000,000
1,330,000,000
Image
Argimi Finance
49,200,000
198,400,000
694,300,000
4,300,000,000
ChartNet
359,000,000
867,800,000
930,900,000
0
Data Insights
7,800
19,500
20,900
0
HQ-50K
43,400
108,800
116,700
0
MedPix-2
93,900
84,200
55,400
0
MINT-1T ArXiv
10,100,000
25,400,000
27,200,000
0
Molmo2 MultiImageQA
1,210,000,000
2,290,000,000
1,190,000,000
6,700,000
NASA Images
98,500
247,200
265,200
0
OWID Articles
2,300,000
2,300,000
738,700
39,300
PIN-200M
2,600,000,000
1,180,000,000
1,270,000,000
0
ShizhenGPT
740,000,000
425,400,000
174,200,000
4,000,000
WAFFLE
3,900,000
9,900,000
10,600,000
0
Audio
EuroSpeech
6,300,000
1,800,000
236,700
0
LegCo Speech
77,200,000
79,300,000
41,900,000
9,200,000
LibriHeavy
294,700,000
259,400,000
30,900,000
0
MLS-7Lang
1,500,000
2,400,000
4,200,000
43,600,000
ParlaSpeech-RS
10,000
0
0
0
People’s Speech
153,000,000
87,400,000
14,900,000
3,400,000
SPC-R
947,200
119,800
0
0
VoxPopuli
483,600,000
779,500,000
978,800,000
1,020,000,000
WenetSpeech
90,900,000
17,500,000
1,400,000
0
YODAS Granary
135,800,000
20,100,000
1,000,000
0
Zoengjyutgaai
1,800,000
347,100
0
0
SFT
ArtificialNeedles (Xiong et al., 2025)
908,000,000
966,500,000
831,200,000
833,200,000
CWE (Team Olmo et al., 2025)
2,730,000,000
2,720,000,000
2,280,000,000
2,270,000,000
DocOrder
1,710,000,000
1,600,000,000
1,230,000,000
1,160,000,000
Total
58,720,000,000
58,720,000,000
50,330,000,000
50,330,000,000
Appendix G Training Configurations
Table G.1: Continued-pretraining configuration of both models. The learning rate follows the warmup-stable-decay schedule of Section 3.2: constant at the peak through stages 1 and 2, decaying to the minimum over stage 3 with the 1−x1-\sqrt{x} shape.
8B
70B
Nodes
256
512
Sequence length
8,192
8,192
Global batch size
1,024
2,048
Parallelism
TP2
TP4, PP8, VPP2
Peak learning rate
1.1×10−41.1\times 10^{-4}
2×10−52\times 10^{-5}
Minimum learning rate
1.1×10−51.1\times 10^{-5}
2×10−62\times 10^{-6}
Warmup
500 iterations
250 iterations
Both models train with AdEMAMix (β1=0.9\beta_{1}=0.9, β2=0.999\beta_{2}=0.999, α=8\alpha=8, β3=0.9999\beta_{3}=0.9999, with the β3\beta_{3} and α\alpha warmup over 100,000 iterations), gradient clipping at 0.1, and weight decay of 0.1, applied also to the xIELU parameters and the normalization gains. The optimizer state is fresh at the start of stage 1 and carries over across the later stage transitions. All stages pack documents with cross-document attention masking: position identifiers reset at document boundaries, and end-of-document positions are excluded from the loss.
The long-context extension proceeds through a curriculum of 32,768, 65,536, 131,072, and 262,144 tokens, each step loading the previous checkpoint with a fresh optimizer state and training at a constant learning rate equal to the main run’s minimum. The RoPE base grows to 2M for the first step an