llel code streams force an autoregressive model to interleave them (Copet et al., 2023). WavTokenizer emits one code per frame, so audio enters the model as a single flat sequence, exactly like text. This single code stream is the deciding criterion, as it leaves the training stack unchanged.22
2
We provide code and benchmarks at swiss-ai/benchmark-audio-tokenizer.
In the token sequence, an audio segment is enclosed by start and end markers, which live in the same reserved block of the vocabulary as their visual counterparts. The audio codes follow the visual codes in the extended vocabulary.
2.2 Multimodal Training Data
Multimodal training data is central to this release. It teaches the models to use the image and audio token spaces introduced above. It also aligns both modalities with language and supports multimodal instruction following across continued pretraining and post-training. Across the three continued-pretraining stages, text-only data accounts for roughly 60% of the Apertus 1.5 8B training mix and 55% of the Apertus 1.5 70B mix, with vision and audio data accounting for the remaining 40% and 45%, respectively (Table 3). We organize the discussion around the six data stages that serve these different roles, a design supported by the ablation of Section 2.4. The six stages are the three continued-pretraining stages of Section 3.2, whose third stage is the cooldown, followed by long-context pretraining (Section 3.3), supervised finetuning (Section 4.2), and long-context supervised finetuning (Section 4.3).
2.2.1 Curation and Processing
Acquisition and Normalization.
The public repository swiss-ai/multimodal-data provides an inventory of vision and audio datasets, recording each source’s pinned revision, download script, processing recipe, license filters, and training stages. The complementary swiss-ai/apertus-data-transparency repository records source versions and licenses across all modalities. We keep raw, processed, and tokenized layers separate to make the transformations between them explicit and repeatable. Download adapters handle repository snapshots, archives, object stores, public APIs, and URL-based image datasets. Processing decodes media, removes invalid records, checks minimum resolution, normalizes schemas, encodes media in standard formats, and converts records to Arrow or WebDataset. Dataset-specific adapters additionally handle large tiled geospatial images, multi-image conversations, spatial annotations, and interleaved documents. When the inventory lists no additional source-specific operation, we apply these shared checks without changing the publisher’s filtering decisions.
Deduplication.
For unpaired continued-pretraining images, we deduplicate samples with perceptual hashes and choose thresholds by domain. Most sources use exact equality of 64-bit hashes after reduction to 32×3232\times 32 pixels. Satellite images use a smaller hash input to suppress repeated forest, water, and cloud tiles, while egocentric video frames allow a small Hamming distance to remove adjacent near-duplicates. These rules are inappropriate for text-rich images such as documents and web interfaces, where distinct pages can share a layout and produce the same hash. In early inspections of document and MINT-1T runs, perceptual hashing removed valid OCR examples. We therefore disabled it for later datasets centered on OCR and document understanding.
Caption Generation and Validation.
For paired continued-pretraining data, we retain informative source captions and regenerate missing, repetitive, or low-information descriptions for selected sources with open-weight vision-language models. The generation pipelines reject empty outputs and apply source-specific output checks. For long captions of text-dense technical drawings, we additionally detect token loops, runaway number lists, and repeated endings caused by repeated labels and tabular schedules. We quarantine and regenerate flagged captions. These automated checks c