to-multimodal ratio and accept a drop of about two points on general text evaluations as the cost of adding the new modalities; text evaluations recover by Stage 2. The 70B, by contrast, has enough capacity to absorb the new modalities with visibly less text-capability cost than the 8B (the Stage 1 creep is far milder in 7(b) than in 7(a)), and we choose its peak learning rate by continuing a pre-cooldown Apertus 1.0 checkpoint with preserved optimizer states on unchanged data, letting the loss settle, and comparing peak learning rates from there. This gives a reliable signal within tens of billions of tokens, and it selects twice the original peak. These two observations suggest that the text-capability cost of adding modalities is capacity-dependent.
Ablations narrow down what drives the Stage 1 text-loss creep. Returning to the original peak learning rate reduces the slow upward creep of the text loss during Stage 1 (top row, Figure 7) but does not remove it; we rule out weight-decay settings as the cause; freezing all non-embedding parameters at the start of training only delays the creep; and the creep appears at mixed text-to-multimodal ratios but not near either extreme, consistent with gradient interference between modalities (Hao et al., 2026). The main driver, however, is the schedule itself: Apertus 1.0 ends cooled down, and warming the learning rate back to its peak costs text performance until the next cooldown recovers it; both the Stage 1 rise and its reversal over the Stage 3 cooldown are visible in the top row of Figure 7. The same text-loss increase upon introducing multimodal data is reported for Kimi K2.5, which for this reason keeps its learning-rate schedule unchanged (Kimi Team et al., 2026).
Figure 7shows the loss of both runs across the three stages: the top row on text tokens only, comparable across stages and exposing the Stage 1 creep and Stage 3 recovery discussed above, and the bottom row over each stage’s own prediction targets.
(a) Apertus 1.5 8B, text loss
(b) Apertus 1.5 70B, text loss
(c) Apertus 1.5 8B, combined loss
(d) Apertus 1.5 70B, combined loss
Figure 7: Per-stage loss against consumed tokens. Top (7(a), 7(b)): text loss, computed over text tokens only in every stage and therefore comparable across stage boundaries. The text loss creeps upward through Stage 1 and then drops sharply over the Stage 3 cooldown. Bottom (7(c), 7(d)): combined loss over all of each stage’s prediction targets (Table 2): image, audio, and text tokens in Stage 1, text only from Stage 2 onward. In Stage 1 the combined loss stays stable, so the text-loss creep visible in the top row is masked when text and multimodal targets are averaged together; the drop at the Stage 1→\to2 boundary reflects the change of targets, not a change in quality. Averages are computed within each stage.
The 70B gains little over the 8B on multimodal tasks despite being nine times larger (Section 5.5 and Section 5.6). We see two causes. First, the 70B sees less multimodal data: its run stops before consuming the full multimodal mixture, because its nodes are reallocated to the next Apertus pretraining campaign. Since multimodal ability improves throughout the mixture, this early stop costs the 70B much of the gain it would otherwise have made. Second, the base learning rate of the AdEMAMix optimizer used to pretrain Apertus 70B was likely too low; continued pretraining uses a learning rate proportional to the original, so a low base rate leaves little room for the model to adapt to a new modality.
We also ablated adding a small share of instruction-formatted data during the cooldown. This improved math, code, and instruction-following benchmarks. Replaying the instruction-formatted data five-fold further improved benchmarks but regressed safety behavior due to overfitting on the small refusal subsets, so the small share performed better. Both preference alignment and reinforcement learning improved long-context evaluations even though 