ion 2.1.1 and Section 2.1.2 specify the two tokenizers with the vocabulary they contribute. Section 2.2 presents the staged multimodal training data used throughout the pipeline, whereas Section 2.3 describes the tokenization pipelines and their integration into the training infrastructure together with the modifications of the pretraining and SFT codebase to enable multimodal training. Finally, Section 2.4 introduces ablations on preserving text capabilities while adding modalities during continued pretraining, and Appendix B and Appendix C record the data pool of each training stage. The following chapter, Section 3, presents the continued-pretraining recipe.
2.1 Unified Representation
We represent images and audio as discrete tokens, following unified models such as Chameleon and Emu3 (Chameleon
Team, 2024; Wang et al., 2026). A shared vocabulary represents all modalities, and the language model applies a single next-token-prediction objective across them, as illustrated in Figure 2.
A common alternative is to use modality-specific encoders whose features are projected into the language model, as in LLaVA (Liu et al., 2023). More recent unified approaches instead avoid separate pretrained modality-specific encoders and process continuous inputs directly: SenseNova-U1 (Cai et al., 2026) operates directly on pixels and text in a unified end-to-end architecture, while the encoder-free Gemma 4 12B (Gemma Team et al., 2026) directly projects raw image patches and raw audio into the model’s embedding space. These approaches unify multimodal computation within a single model, but represent non-text inputs continuously rather than through discrete modality tokens.
Our discrete representation allows the same next-token-prediction objective to be applied to images and audio, enabling the first stage of continued pretraining to draw on unpaired images and audio (Irmiger et al., 2026). In this stage, the model learns the new modalities from image-only and audio-only examples alongside text, before learning their alignment with language from paired data in the subsequent stage (Figure 1). Image and audio codes serve directly as prediction targets, preserving a common training objective across modalities. A controlled study shows that an image-only stage before paired training can improve image understanding on most benchmark groups (Kreft, 2026); Section 2.4 summarizes its findings. Unpaired data also needs no captions, so it remains usable where captions are scarce or of low quality.
We convert training images and audio into discrete tokens once, during offline preprocessing (Section 2.1.1, Section 2.1.2), just as we precompute text tokens. We reuse these tokenized datasets across subsequent training runs and experiments that use the same tokenizers and preprocessing. The language model consumes the precomputed tokens through its shared embedding layer; no vision or audio encoder runs in its forward pass. This separation lets us reuse the text-training infrastructure without the overhead and complexity of encoder forward passes during training, achieving throughput comparable to text-only training with only minor modality-specific modifications to the pretraining and SFT codebases (Section 2.3). A measured comparison substantiates the efficiency benefit: at an identical configuration (256 nodes, 8,192-token sequences), text-only ablation runs sustain 5,300 to 5,500 tokens per second per GPU, and the first multimodal production stage 5,474 (Section 6.2).
inherited from pretrained checkpoint
new: extends the vocabulary
Apertus is a fully open family of LLMs, trained on the
(a) image input(b) audio input(c) text input
Emu3.5 VisionTokenizer
WavTokenizer
Text Tokenizer
NEW
NEW
discrete tokens
Embedding (266,752 input embedding rows)
Apertus 1.0 text131,072 entries
reserved200 slots
image codes131,072 entries
audio codes4,096 entries
padding312 rows
Apertus 1.5 Decoder
Multimodal Self-Attention
continued pretraining from a text-only checkpo