tic verifiable long-context tasks from swiss-ai/LongContextQA: we inject Q&A turns into a long document and re-tokenize it as one training example. Common-word extraction (CWE) (Team Olmo et al., 2025) and document-order reconstruction (DocOrder) together contribute most of these tokens: CWE asks how many times each of several sampled words occurs in the document, and DocOrder shuffles a document’s sections and asks for their original order, each with 10 randomized question or answer formats so the model cannot rely on one fixed phrasing. Both tasks scale in difficulty with the packed length by varying how densely they quiz each document: the hard bucket draws shorter source documents and packs more items into them (10 questions for CWE, eight sections for DocOrder), so the model must track more items with less surrounding context; the easy bucket draws longer source documents and packs fewer items into them (five questions, four sections), testing the same skill with more context to spare. Mixing both buckets balances signal across the full range of packed lengths. Artificial needles (Xiong et al., 2025) add the rest, planting a retrievable fact among synthetic distractor listings. We add masking in the input sequence to avoid biasing the models toward incorrect or random long context in DocOrder and Artificial needles. The SFT token counts reported for these tasks count the whole packed example, including the injected document that the loss masking excludes; the number of assistant tokens actually optimized per sample is far smaller than these totals suggest.55
5
Estimated from representative per-task answer lengths (CWE: about six tokens per turn; DocOrder: about 14 tokens for the single order-list answer; Artificial needles: about 25 tokens per turn) and the number of samples per phase: the actually optimized share of SFT tokens is well under 1%, from about 0.3% at 32k down to about 0.05% at 262k.
Table 5: Token budget per class across the four long-context extension phases. Percentages are the within-phase share of tokens beyond the Cooldown subsample of Table 3.
Phase
English
Multilingual
Image
Audio
SFT
Total
Apertus 1.5 8B
32k
6.07B (25.6%)
4.05B (17.1%)
6.66B (28.1%)
1.30B (5.5%)
5.60B (23.6%)
23.69B
65k
5.63B (25.5%)
3.79B (17.2%)
6.25B (28.3%)
1.22B (5.5%)
5.17B (23.4%)
22.07B
131k
5.69B (25.9%)
3.80B (17.3%)
6.27B (28.6%)
1.23B (5.6%)
4.96B (22.6%)
21.95B
262k
5.69B (26.0%)
3.81B (17.4%)
6.29B (28.7%)
1.22B (5.6%)
4.86B (22.2%)
21.87B
Apertus 1.5 70B
32k
5.81B (27.3%)
3.87B (18.2%)
4.97B (23.4%)
1.25B (5.9%)
5.35B (25.2%)
21.25B
65k
5.77B (27.2%)
3.88B (18.3%)
5.00B (23.6%)
1.25B (5.9%)
5.29B (25.0%)
21.19B
131k
5.00B (27.7%)
3.35B (18.6%)
4.30B (23.8%)
1.07B (5.9%)
4.34B (24.0%)
18.06B
262k
4.99B (27.8%)
3.33B (18.5%)
4.31B (24.0%)
1.08B (6.0%)
4.26B (23.7%)
17.98B
The long-context evaluations of the released models appear in Section 5.7.
3.4 Pretraining Run Retrospective
We discard the first Stage 1 run for the 8B and restart it. That run had two problems. First, its peak learning rate was set to twice the Apertus 1.0 value, a choice that small-scale ablations had favored but that did not transfer to this setting. Second, its text mixture silently omitted the English quality-web, math, and code subsets: the corpus was spread across personal scratch directories with no single manifest, so the launch scripts only saw part of it. As a result, text capability degraded; the text loss eventually recovered, but the evaluations did not. The restart uses the original peak learning rate and the corrected mixture. We keep the text-to-multimodal ratio and accept a drop of about two points on general text evaluations as the cost of adding the new modalities; text evaluations recover by Stage 2. The 70B, by contrast, has enough capacity to absorb the new modalities with visibly less text-capability cost than the 8B (the Stage 1 creep is far milder in 7(b) than in 7(a)), and we choose its peak learning rate by continuing a pre-