t allow-list limited to domains represented in the FineWeb robots.txt database would discard roughly 96% of MINT-1T, whereas the exclusion rule used by the text pipeline removes roughly 2%. For datasets that retain only an image-hosting URL or the name of another dataset, we record that domain-level opt-out filtering is not technically available, rather than treating the hosting service as the content owner.
We materialize URL-based datasets by downloading original images from source URLs on the shared Alps supercomputer at CSCS, and therefore separately apply network safety filtering to protect the center’s shared outbound network infrastructure. Section 6.4 describes the proxy and blocklists. This safeguard is distinct from license review, opt-out handling, and image content review; it does not constitute a comprehensive scan for sexual or violent content. For image and audio content, we otherwise retain the filtering supplied by the dataset publishers.
3.1.2 Text Data Selection
We refine the text mixture of Apertus 1.0 by replacing FineWeb2-HQ with FineWeb2-HQ+ (Turki et al., 2026) and adding FinePDFs, Nemotron-CC-v2.1, SYNTH, FineTranslations, Nemotron-Pretraining-Code-v1, and legal-domain corpora. Table 4 reports the per-stage sampling proportions of the resulting text mixture.
Every added source must be publicly available, pass license and provenance review, and improve on the Apertus 1.0 mixture in a small-scale ablation. The baseline experiment trains for 40,000 steps on FineWeb followed by a 10,000-step cooldown on FineWeb-Edu. A candidate source replaces 30% of the cooldown data, and we evaluate it on the standard development suite.44
4
The data ablation results are published at https://api.wandb.ai/links/bmessmer/2l4dpwl1. For specialized sources, such as legal or medical data, we require an improvement on the corresponding domain evaluations without a material regression on the general suite. We also prefer sources that retain original URLs and other provenance fields needed for later filtering.
For multilingual data, we extend the FineWeb2-HQ dataset (Messmer et al., 2025) used for Apertus 1.0 to an improved variant, FineWeb2-HQ+ (Turki et al., 2026). The dataset retains the 10% of FineWeb-2 documents per language rated highest by a multilingual quality classifier, covering the 100 highest-resource languages and a 10% sample of the remaining languages (originally, FineWeb2-HQ’s quality filter covered only 20 languages). The classifier recipe changes in two ways. First, we train a single classifier jointly on all languages from a pool of positive samples widened with multilingual instruction data (including MURI-IT, Tagengo, and WikiQA), with lower-resource languages upsampled by at most a factor of three so that high-resource languages do not dominate the decision boundary. Second, its negative samples are drawn from the 50th–75th percentile of a proxy quality classifier rather than uniformly from FineWeb-2, which sets the boundary against fluent but low-utility text instead of obvious noise (Turki et al., 2026).
3.2 Optimizer and Training Recipe
We continue pretraining both Apertus 1.5 models, the 8B and 70B, with the AdEMAMix optimizer (Pagliardini et al., 2025) from Apertus 1.0 (Hernández-Cano et al., 2026), retaining the same momentum hyperparameters. We resume from the Apertus 1.0 checkpoints that have completed cooldown and use a warmup-stable-decay schedule: both models warm up from zero to their peak learning rates over 4.2B tokens; the learning rate then stays at its peak through the first two stages and decays to 10% of the peak with a 1−sqrt1-\text{sqrt} decay over the third. The 8B model keeps the peak learning rate of Apertus 1.0 8B, 1.1×10−41.1\times 10^{-4}. For the 70B model, we double the peak of Apertus 1.0 70B to 2×10−52\times 10^{-5}, which outperforms the original value in ablations, suggesting that the Apertus 1.0 70B peak learning rate was likely too small. To accommodate multimodal data, we