al content is more prominent in the 70B cooldown pool. Geospatial content remains small overall.
Figure 4: Vision data composition by stage, with pool sizes in input tokens on the right.
2.2.4 Audio Training Data
Audio data enters the same six stages. Stage 1 uses unpaired speech, music, and mixed audio, while the later stages concentrate on paired speech and text. Paired speech takes three forms: transcription pairs, translation pairs, and interleaved sequences that alternate the audio and transcripts of consecutive clips from one recording, with dedicated tokens marking each switch between speech and text. Interleaved sequences make up about a fifth of the Stage 2 and cooldown audio tokens and all of the long-context audio pool (Table C.3). The tables in Appendix C give the audio pool of each stage, with samples and input tokens per dataset.
Unlike vision, where we classify samples and weight categories by sample count, we group entire audio datasets into six categories by source and purpose. Figure 5 shows their contributions by input tokens.
Web and broadcast speech dominate every pretraining pool. Read speech becomes more prominent in later pretraining stages. Music and other audio remain small components. The 70B Stage 2 and cooldown pools are identical. Dialogue and question answering dominate both SFT pools, with domain-specific speech (such as medical QA) contributing a smaller component.
Figure 5: Audio data composition by stage, with pool sizes in input tokens on the right.
Interleaved Speech and Text.
For the paired stages, we build interleaved sequences from transcribed speech corpora such as VoxPopuli, Multilingual LibriSpeech, and WenetSpeech. We group the transcribed clips of each recording into runs of consecutive clips. Within a run, a sequence alternates between modalities clip by clip: the audio of one clip is followed by a speech-to-text continuation token and the transcript of the next clip, which is followed by a text-to-speech continuation token and the audio of the clip after that, and every sequence ends with text. We build two copies of each run that start one clip apart, so that a clip that appears as audio in one copy appears as text in the other. Single-clip runs and the leftover clip of a run become transcription sequences that pair one clip’s audio with its own transcript. Sequences of up to 8,192 tokens enter Stage 2, and longer ones enter the long-context extension. The builder is released in swiss-ai/benchmark-audio-tokenizer.
Speech SFT Data.
The auxiliary checkpoint used for capability filtering of text data (Section 4.2) and the production SFT runs draw from the same prepared audio pool of six speech datasets (Table C.4). This pool contains speech inputs with text outputs and no music or environmental-sound supervision. Its largest component by sample count renders roughly 640,000 conversations from eight open chat corpora (Lian et al., 2023; Zhao et al., 2024; Ding et al., 2023; Singh et al., 2024; Durbin, 2023; Köpf et al., 2023), among them Capybara and Pure-Dove (Daniele & Suphavadeeprasit, 2023), to speech with text-to-speech synthesis, so that the assistant behavior of the text mixture carries over to spoken instructions. For this component, we synthesize user turns with Qwen3-TTS (Hu et al., 2026), while retaining assistant responses as text. We align the generated speech to the corresponding source text using Qwen3-ForcedAligner (Shi et al., 2026). During dataset construction, we exclude conversations with unavailable audio and verify that each audio placeholder corresponds to an associated audio segment. The remainder combines a voice-assistant corpus of roughly 242,000 conversations (Xie & Wu, 2024), spoken question answering with both machine-read and human-read questions (Wu et al., 2023), a fraud-detection dialogue set (Ma et al., 2025), and a bilingual medical set that mixes Chinese dialogues with English audio question answering and speech recognition (Chen et al., 2025c). W