ssing, and provenance requirements. Section 3.2 specifies the optimizer, learning-rate schedule, prediction targets, and model-specific token budgets. Section 3.3 describes the long-context extension and its data mixture. Finally, Section 3.4 examines the training dynamics and lessons from the production runs.
3.1 Pretraining Data
We continue training on a new mixture based on the Apertus 1.0 text mixture, refined with updated and additional text datasets and extended with image and audio data. We use only publicly available datasets and document their provenance and filtering. This subsection describes the data governance requirements shared across modalities, including compliance, provenance, and opt-out handling (Section 3.1.1), followed by the selection and refinement of the text mixture (Section 3.1.2). We maintain the source versions and licenses for the full mixture in swiss-ai/apertus-data-transparency and the source-specific vision and audio processing recipes in swiss-ai/multimodal-data.
We introduce multimodality through the discrete-token early-fusion approach presented in Section 2.1: images and audio as discrete tokens that the model processes alongside text in a shared token space. These new token spaces motivate the staged recipe shown in Figure 1. In Stage 1, we combine text with unpaired images and audio and train the model to predict tokens of all modalities. In Stage 2, we introduce paired image-text and audio-text sequences and restrict prediction targets to text, aligning the new modalities with language. The cooldown then emphasizes higher-quality and targeted sources, and the long-context extension adds longer and interleaved records. Section 3.2 specifies the model-specific token budgets, sampling ratios, and optimization schedule.
Details of the multimodal data recipe appear in Section 2, which describes the multimodal design, the rationale for the staged training recipe (Section 2.4), and data preparation (Section 2.2). Section 2.2.2 and Section 2.2.4 describe the vision and audio data, respectively, while Appendix B and Appendix C list the prepared pools across stages. For multimodal sources, we apply the acquisition, filtering, deduplication, and caption validation procedures described in Section 2.2.1.
3.1.1 Compliance and Provenance
Every new modality is also required to satisfy the same requirements as the text data in Apertus 1.0 (Hernández-Cano et al., 2026). Our review covers the upstream source, revision, declared license, constituent datasets, and sample-level provenance where available. For composite medical and remote-sensing collections, we select approved constituent sources or records rather than relying on the aggregate license. Non-commercial, research-only, and share-alike subsets are excluded, and the spreadsheet in swiss-ai/apertus-data-transparency remains the authoritative record of the final decisions. No paywalled or otherwise closed sources are used for training. The text pipeline additionally removes personally identifiable information and applies the multilingual toxicity filters of Apertus 1.0.
We apply machine-readable opt-outs (robots.txt files) following the same recipe as Apertus 1.0, and we honor robots exclusions retroactively: if a page owner opts out this year, previous crawls of the same URL are excluded as well. The opt-out filtering also applies to multimodal content when a record retains the page URL on which the content appeared. For example, MINT-1T HTML preserves original page URLs, so we apply the same domain exclusion list as the text pipeline. For images, a strict allow-list limited to domains represented in the FineWeb robots.txt database would discard roughly 96% of MINT-1T, whereas the exclusion rule used by the text pipeline removes roughly 2%. For datasets that retain only an image-hosting URL or the name of another dataset, we record that domain-level opt-out filtering is not technically available, rather than treating the hosting service as the con