o 10% of the peak with a 1−sqrt1-\text{sqrt} decay over the third. The 8B model keeps the peak learning rate of Apertus 1.0 8B, 1.1×10−41.1\times 10^{-4}. For the 70B model, we double the peak of Apertus 1.0 70B to 2×10−52\times 10^{-5}, which outperforms the original value in ablations, suggesting that the Apertus 1.0 70B peak learning rate was likely too small. To accommodate multimodal data, we double the sequence length from 4,096 tokens to 8,192 tokens. An image of 512×512512\times 512 pixels is tokenized into 1,024 tokens and a minute of audio into 2,400 tokens (Section 2.1.1, Section 2.1.2); multimodal sequences interleaved with text documents routinely exceed the previous 4,096-token limit, and the 8,192-token window ensures that entire documents fit within a single sequence. We halve the global batch size so that the number of tokens per batch remains the same as in Apertus 1.0, and reuse the same tensor and pipeline parallelism configurations to train Apertus 1.5. Table G.1 summarizes the resulting configuration of both models.
We apply weight decay to all trainable parameters, including normalization gains and the trainable parameters of the xIELU activation (Huang & Schlag, 2024), which we find helps reduce large activation and activation-gradient norms. We initialize the optimizer states from scratch, since the newly added input and output embeddings for image and audio tokens have no existing optimizer states. The corresponding embedding and output rows for the new tokens are randomly initialized, while the text rows keep their pretrained values; the fresh rows start at roughly a tenth of the norm of the trained text rows and adapt over the first pretraining stage. For the 70B model, we also evaluate splicing in the multimodal embeddings learned by the 8B model, but we observe that it destabilizes training, so we use random initialization instead. In the released checkpoints, we prune the output rows for image, audio, and reserved tokens from the language-model head, since the models never generate non-text tokens and the text logits are unchanged.
We keep the Goldfish loss of Apertus 1.0 with its parameters unchanged: it drops the prediction target of a random subset of positions, thereby preventing verbatim memorization of the training data and respecting copyright (Hans et al., 2024; Xu, 2025). We extend this dropping to cover the predicted tokens of every modality; only special tokens, among them the markers that structure image and audio sequences, are exempt. Image and audio tokens are prediction targets in the first stage alone. The later stages predict text only, which prevents the model from generating non-text tokens. From the second stage, we likewise exclude from prediction the task-control markers that select between transcription and continuation: they condition the model but are never targets.
Before the long-context extension, continued pretraining proceeds in three stages: unpaired multimodal training, paired multimodal training, and cooldown with higher-quality data. The stages are summarized in Table 2 and Table 3, and the text data mix in Table 4; the vision and audio datasets of every stage are listed in Appendix B and Appendix C. Each stage’s dataset list is frozen before the stage begins: the deterministic sampling plan depends on the full list, and mid-stage changes would shift that plan, silently repeating or skipping documents. In the first stage we train on unpaired image, audio, and text data, with the loss computed over tokens of all modalities. In the second and third stages we add the respective parallel data (image-text and audio-transcript pairs) along with text-only data, and compute the loss over the text tokens only, since the models understand but never generate images and audio (Section 2); an ablation further shows that text-only targets during paired training also yield the strongest image understanding (Section 2.4). For the 8B, the Stage 2 text share rises to roughly two 