mily of LLMs, trained on the
(a) image input(b) audio input(c) text input
Emu3.5 VisionTokenizer
WavTokenizer
Text Tokenizer
NEW
NEW
discrete tokens
Embedding (266,752 input embedding rows)
Apertus 1.0 text131,072 entries
reserved200 slots
image codes131,072 entries
audio codes4,096 entries
padding312 rows
Apertus 1.5 Decoder
Multimodal Self-Attention
continued pretraining from a text-only checkpoint
predicted
...Alps supercomputer in Switzerland.
(a) image – not decoded(b) audio – not decoded(c) text – ✓\checkmark decoded
next-token prediction loss
Figure 2: The Apertus 1.5 architecture.
Vocabulary Extension.
We extend the 131,072-entry vocabulary of Apertus 1.0 with 200 reserved multimodal slots, including structure markers, followed by 131,072 image codes and 4,096 audio codes. These contiguous blocks yield 266,440 vocabulary entries (Figure 2). The released input embedding matrix contains 266,752 rows: its final 312 rows are padding, not additional vocabulary entries. The extended input vocabulary allows both models to consume every modality, while the released output head retains only the original text vocabulary (Section 3.2).
2.1.1 Images
We use the vision tokenizer of Emu3.5 (Cui et al., 2025b) to convert training images into discrete tokens during offline preprocessing. Irmiger et al. (2026) evaluate image tokenizers on reconstruction quality, on the average number of tokens per image, and on their support for arbitrary image shapes, and find that the tokenizers of the Emu family combine a favorable trade-off between reconstruction quality and token efficiency with native support for arbitrary shapes. Section 2.4 discusses the choice between the Emu3 and Emu3.5 tokenizers.11
1
We provide code and benchmarks at swiss-ai/benchmark-image-tokenizer.
The adopted tokenizer is trained solely with a pixel-reconstruction objective, rather than an explicit semantic or image-text alignment objective as used by vision encoders such as CLIP (Radford et al., 2021). We use its pretrained discrete codes directly as inputs and prediction targets for the language model.
The tokenizer maps every block of 16×1616\times 16 pixels to one code from a codebook of 131,072 entries of dimension 256. An image of 512×512512\times 512 pixels thus becomes a grid of 32×3232\times 32 codes, or 1,024 tokens, and an image of arbitrary shape becomes a grid of its height and width each divided by 16. The codebook is trained with index backpropagation quantization, which keeps a codebook of this size in use where earlier vector quantization leaves most codes dead (Shi et al., 2025).
In the token sequence, an image opens with a start marker followed by its dimensions; its rows of codes are separated by end-of-row markers, and an end marker closes it. The visual codes extend the vocabulary directly after the text tokens of Apertus 1.0 and a block of 200 reserved slots, seven of which hold these structure markers.
2.1.2 Audio
We use the WavTokenizer codec (Ji et al., 2025) to convert training audio into discrete tokens during offline preprocessing. The codec discretizes a waveform with a single quantizer over a codebook of 4,096 entries. It consumes mono audio at 24 kHz and emits one code per 600 samples, hence 40 tokens per second; a minute of audio costs 2,400 tokens. Inputs at other sampling rates are resampled. Because the codec was trained on peak-normalized audio, we normalize every input to a peak of −3-3 dBFS before tokenization.
Neural audio codecs such as EnCodec quantize each frame with a stack of residual codebooks (Défossez et al., 2023), and their parallel code streams force an autoregressive model to interleave them (Copet et al., 2023). WavTokenizer emits one code per frame, so audio enters the model as a single flat sequence, exactly like text. This single code stream is the deciding criterion, as it leaves the training stack unchanged.22
2
We provide code and benchmarks at swiss-ai/benchmark-audio-tokenizer.
In the token sequence, an audio 