 its batch and the last few codes before the padding are unstable, the pipeline drops the last five codes after padding.
2.4 Ablations
Introducing new modalities into a pretrained language model can degrade its text capabilities, particularly during continued pretraining. Lu et al. (2024) report catastrophic forgetting of language capabilities during joint vision-language pretraining and retain at least 70% text data to mitigate it. Similarly, Kimi Team et al. (2026) observe an immediate drop in text performance when vision data is introduced to a text-pretrained model, followed by only gradual recovery during subsequent training.
Preserving the language capabilities of the base model is therefore a primary constraint throughout the development of Apertus 1.5, alongside acquiring strong multimodal capabilities. This section studies the design choices of the continued pretraining that bear on this constraint; Section 4.7 turns to post-training, where multimodal data interacts with reasoning supervision.
Two design decisions of the continued pretraining rest on dedicated experiments: the choice of the image tokenizer, which trades reconstruction quality for token efficiency, and the two-stage recipe, which separates representation learning on unpaired images from modality alignment on paired data.
Image Tokenizer.
Irmiger et al. (2026) find that the tokenizers of the Emu family offer the best trade-off between reconstruction quality and token efficiency (Section 2.1.1). Within that family, we choose the tokenizer of Emu3.5 (Cui et al., 2025b) over that of Emu3 (Wang et al., 2026) for its token efficiency. Emu3 encodes one code per 8×88\times 8 pixel patch and Emu3.5 one per 16×1616\times 16 patch, so Emu3.5 represents the same image with a quarter of the tokens.
To quantify what this compression costs on our data, we benchmark both tokenizers on a stratified sample of the Stage 1 vision mixture, built on the sample-level categories of Section 2.2.3. We draw 50 images per category with a fixed seed and report a representative subset of six categories, 350 image pairs, that spans natural photographs, synthetic imagery, structured graphics, text, and the two specialist domains. We omit the categories not listed and merge charts and diagrams into one category, as in the composition figures. We resize each image as we do for training Apertus, encode and decode it with each tokenizer, and score the reconstruction against the input with the peak signal-to-noise ratio (PSNR), the structural similarity index (SSIM), and the learned perceptual image patch similarity (LPIPS) (Table 1 and Figure 6).
Emu3.5 uses a quarter of the tokens of Emu3 in every category, at a cost of 2.7 dB PSNR, 0.07 SSIM, and 0.02 LPIPS averaged over the subset. The gap is consistent across categories: Emu3 reconstructs better on every metric except PSNR on geospatial imagery, where the two are almost level, and the largest PSNR losses fall on charts and diagrams, medical images, and documents, each between 3 and 4 dB. Medical images and documents are the two categories in which Emu3.5 reconstructs worst in PSNR, which confirms our qualitative observation that fine-grained detail, and text in particular, is its main weakness. We accept this trade: because image tokens typically dominate the token count of an image sample, the fixed token budget of each pretraining stage covers approximately four times as many images, and inference on image inputs is correspondingly cheaper. Although our models see more images during training, the coarser grid is the likely reason that OCR remains the capability group in which they trail other open models (Table 34).
Table 1: Reconstruction quality and token cost of the Emu3 and Emu3.5 tokenizers, averaged over 350 Stage 1 training images from six representative categories of Section 2.2.3 and preprocessed as in training. Tokens are the mean per image, arrows give the better direction, and bold marks the better tokenizer; Table E.1 