remains the capability group in which they trail other open models (Table 34).
Table 1: Reconstruction quality and token cost of the Emu3 and Emu3.5 tokenizers, averaged over 350 Stage 1 training images from six representative categories of Section 2.2.3 and preprocessed as in training. Tokens are the mean per image, arrows give the better direction, and bold marks the better tokenizer; Table E.1 in Appendix E lists the values per category.
Tokenizer
Tokens ↓\downarrow
PSNR (dB) ↑\uparrow
SSIM ↑\uparrow
LPIPS ↓\downarrow
Emu3
8,372
26.70
0.806
0.075
Emu3.5
2,080
24.04
0.740
0.099
Figure 6: Per-category reconstruction quality of the Emu3 and Emu3.5 tokenizers on the same 350 Stage 1 training images as Table 1. Each marker is a category mean. Higher is better for PSNR and SSIM; lower is better for LPIPS. Table E.1 lists the values.
Two-Stage Recipe.
A controlled study underpins the design of the first two pretraining stages of Section 3.2 (Kreft, 2026). The study extends Llama 3.2 3B with the same Emu3.5 tokenizer and splits the topic-balanced LLaVA-OneVision 1.5 image-caption corpus (An et al., 2025) in half: one half, stripped to standalone images, feeds a representation-learning stage that predicts image tokens, mirroring our Stage 1; the other half, kept as image-caption pairs, feeds a subsequent modality-alignment stage, mirroring our Stage 2. Both halves share one distribution, and every batch replays 10% text-only data. Three findings carry over. First, giving image and text tokens equal loss weight during modality alignment is harmful: this setting scores far below all others on nearly every group of image-understanding benchmarks, showing that image-token prediction and modality alignment interfere when they share compute. Second, zero weight on image tokens during modality alignment is the strongest single setting for image understanding, best on the document-understanding, grounding, and hallucination benchmark groups, while variants that end the modality-alignment stage at zero weight stay competitive on the remaining groups; this backs our text-only Stage 2 loss targets. Third, the representation-learning stage pays off downstream: with the modality-alignment stage fixed, raising its budget from zero to roughly 57B image tokens lifts image understanding measured after modality alignment and SFT, for example GQA from 0.50 to 0.54, DocVQA from 0.67 to 0.71, and SEED-Bench from 0.61 to 0.65, with logarithmic gains that flatten at larger budgets; this backs our unpaired all-token Stage 1 and its bounded token budget. At the largest budgets, however, knowledge-heavy image benchmarks and text benchmarks erode, as both rely on the model’s language capabilities; the deficits persist despite the replayed text and do not fully recover in the later stages, motivating the substantially larger text share of our mixtures (Table 3).
3 Continued Pretraining Recipe
Apertus 1.5 is obtained through continued pretraining of Apertus 1.0, with the goal of introducing multimodal capabilities, including image and audio, while preserving the text capabilities of Apertus 1.0. We start from the Apertus 1.0 8B and 70B checkpoints and proceed through four stages of continued pretraining (Figure 1): unpaired multimodal training, paired multimodal training, and cooldown with higher-quality data, all at 8K sequence length, followed by long-context extension to support context lengths up to 262K.
This chapter gives an overview of the continued-pretraining data and recipe. Section 3.1 describes the revised data mixture including source selection, data processing, and provenance requirements. Section 3.2 specifies the optimizer, learning-rate schedule, prediction targets, and model-specific token budgets. Section 3.3 describes the long-context extension and its data mixture. Finally, Section 3.4 examines the training dynamics and lessons from the production runs.
3.1 Pretraining Data
We continue training on a new mixture based on the Apertus 1.0 text