d audio segment. The remainder combines a voice-assistant corpus of roughly 242,000 conversations (Xie & Wu, 2024), spoken question answering with both machine-read and human-read questions (Wu et al., 2023), a fraud-detection dialogue set (Ma et al., 2025), and a bilingual medical set that mixes Chinese dialogues with English audio question answering and speech recognition (Chen et al., 2025c). We process each dataset separately: we decontaminate the vocalized conversations against the evaluation benchmarks, removing 2,153 conversations, and filter them to the languages supported by the synthesis pipeline, removing a further 6,141 conversations; we drop 4,306 identity-related rows from the voice-assistant corpus to protect the model’s self-description; and we filter the medical set to its bimodal records (dropping 32,320 image-bearing records) and repair the missing audio placeholders.
Long-Context SFT Data.
The long-context SFT pool in Table C.5 extends the ordinary SFT pool with Marco Longspeech and AudioMCQ GeminiCoT, as well as additional TeleAntiFraud and Vocalized SFT samples. Marco Longspeech is its largest component by token count. AudioMCQ GeminiCoT contributes only the speech partition of its source, with its reasoning traces placed in the thinking span, and enters the short-context bucket of both models’ long-context SFT runs. The pool inventory gives the available data; Table 8 reports the per-source counts used by each model.
2.3 Tokenization and Training Pipelines
Megatron-LM (Shoeybi et al., 2019) natively supports multimodality, with two limitations. First, it supports only continuous features: neither the multimodal models of Megatron Core nor the sample types of Megatron Energon carry discrete tokens (Patel et al., 2025). Second, it computes everything inside a training job: vision and audio encoders and a projection layer embed pixels and waveforms in the forward pass, and the supervised finetuning paths tokenize conversations and derive the loss mask from the chat template at load time. This adds encoder compute and media decoding to every step, and it leaves no entry point for pre-tokenized multimodal sequences, in pretraining or in supervised finetuning.
Following prior work on incorporating discrete multimodal representations into pretrained LLMs (Ge et al., 2024; Zhan et al., 2024), we extend the base LLM by adding dedicated vocabulary ranges for vision and audio tokens. Each new token is assigned a corresponding embedding, allowing tokenized media to be consumed by the transformer in the same way as text tokens. The vocabulary is extended correspondingly, and the output projection is extended to produce logits for the newly introduced vision and audio tokens in addition to the original text vocabulary. The newly introduced parameters are randomly initialized, while all parameters inherited from the pretrained LLM are retained. Thus, the transformer itself requires no modality-specific computation: multimodal sequences can be processed with the standard autoregressive next-token prediction objective.
We therefore build two high-throughput tokenization pipelines from scratch, one for images and one for audio, that write Megatron indexed datasets directly. Section 2.2.1 describes the upstream acquisition and processing that feed them. The pipelines convert each image or waveform into token ids from the corresponding added vocabulary range. This is the payoff of discrete tokens: a codec maps an image or a waveform to ids from a contiguous block of the vocabulary, so to the model a tokenized image is a text document in a language it has not seen before. The multimodal shards enter the training data mixture through the standard data blending and sampling of Megatron-LM, and stream through the unmodified pretraining data loader. Hence, continued pretraining can run on an almost vanilla Megatron-LM installation.
The necessary changes to Megatron-LM live in our fork (swiss-ai/Megatron-LM). For pretraining, we 