/SYNTH
0.7
0.0
epfml/FineWeb2-HQ-Plus (top 10%)
20.1
14.4
HuggingFaceFW/finepdfs-edu (multilingual only)
8.5
3.6
HuggingFaceFW/finetranslations
2.8
5.9
HuggingFaceTB/finemath
0.9
6.3
IFM/MegaMath
6.1
9.5
Infi-MM/InfiMM-WebMath-40B
0.6
4.2
HuggingFaceTB/stack-edu
2.1
6.2
nvidia/Nemotron-Pretraining-Code-v1
5.3
15.1
joelniklaus/legal-mc4
0.7
0.0
joelniklaus/Multi_Legal_Pile
1.0
0.0
voilaj/swiss-caselaw
0.0
2.7
Apertus 1.5 SFT data
0.0
1.2
3.3 Long-Context Extension
The long-context extension is the fourth and last stage of continued pretraining (Figure 1), and follows the three stages of Section 3.2. We extend the context window in four steps, from 8,192 to 32,768, 65,536, 131,072, and finally 262,144 tokens. Each step starts from the checkpoint produced by the previous one and reinitializes the optimizer states. Depending on the phase and model size, we train on 50B–63B tokens per step.
We keep the learning rate fixed at the final value reached during cooldown. This provides a conservative continuation of training after cooldown, limiting changes to the model while adapting it to longer sequences. We also include synthetic long-context tasks to explicitly train capabilities that require using information over long distances. Whenever we double the sequence length, we halve the number of sequences per batch so that the total number of tokens per batch remains constant.
We also adjust the rotary position embeddings as the context window grows. The main runs initially keep the base frequency of 500,000 and scaling factor of 8 used in Apertus 1.0. By the end of the four extension stages, the final model uses a base frequency of 4M and a scaling factor of 32. Documents are packed using a best-fit-decreasing strategy (Ding et al., 2024). Cross-document attention masking under context parallelism also makes training substantially cheaper: training at 131,072 tokens costs roughly the same as training at 65,536 tokens did for Apertus 1.0.
The prepared data for each step keeps the same relative composition regardless of phase or model size: most tokens come from a subsample of the cooldown mixture, and the remainder splits into genuinely long documents and synthetic verifiable long-context tasks. Because each step trains on fewer tokens than this prepared pool, Appendix F gives the exact, already-scaled per-dataset, per-step token counts and sources actually used in training. As throughout pretraining, we preprocess and filter every dataset before use. The long documents, split into English text, multilingual text, and multimodal image-text and audio corpora, are length-filtered subsets of similar pretraining sources used earlier in the recipe; we draw them proportionally to each source’s own size so that no single source dominates a step. Table 5 gives the token budget and within-phase share of each class.
Each step targets a fixed image budget, but the natively length-filtered image pool cannot supply enough tokens at the 32k, 65k, and 131k phases, since fewer real documents in the pretraining corpora happen to span exactly those lengths; only the 262k phase has enough native long-context images on its own. We close the gap at the shorter phases by drawing additional images from short-context documents (1,000–16,384 tokens) so that every phase reaches the same image budget: native long-context images make up about 79% of the 32k budget, 47% of the 65k budget, and 34% of the 131k budget, with the short-context top-up filling the rest. Table B.4 gives further per-dataset sample counts for the vision pool.
The remaining tokens per step are synthetic verifiable long-context tasks from swiss-ai/LongContextQA: we inject Q&A turns into a long document and re-tokenize it as one training example. Common-word extraction (CWE) (Team Olmo et al., 2025) and document-order reconstruction (DocOrder) together contribute most of these tokens: CWE asks how many times each of several sampled words occurs in the document, and DocOrder shuffles a document’