nt in a language it has not seen before. The multimodal shards enter the training data mixture through the standard data blending and sampling of Megatron-LM, and stream through the unmodified pretraining data loader. Hence, continued pretraining can run on an almost vanilla Megatron-LM installation.
The necessary changes to Megatron-LM live in our fork (swiss-ai/Megatron-LM). For pretraining, we add options for masking multimodal tokens from prediction, which enables the text-only targets from the second training stage onward (Figure 1), and for exempting the multimodal special tokens from Goldfish-loss dropping (Section 3.2). For supervised finetuning, we add the missing entry point: a loader that reads pre-tokenized conversations from the same indexed format and derives the loss mask efficiently on the fly at the token level, masking user turns and tool responses without a separate preprocessing step or added overhead in the training step.
2.3.1 Image Tokenization
The pipeline, released in swiss-ai/benchmark-image-tokenizer, separates indexing, planning, and tokenization. A scan reads only tar headers and the first few kilobytes of each image, recording byte offsets and dimensions for hundreds of millions of samples in minutes, and never touches pixels. From the manifest a deterministic plan assigns every image to a GPU batch: images are grouped in windows of consecutive source rows, snapped to document boundaries, sorted by their exact post-resize shape, and packed to a token budget, so that every batch is shape-uniform and byte-adjacent, the ranks never communicate, and the measured rank imbalance stays below 2% at 800 GPUs. Planning is NumPy-only and costs seconds, so re-planning for a different batch size or GPU count never requires a rescan. On each rank, loader threads decode, filter, and resize images while the GPU encodes the previous batch, and text tokenization and chat-template rendering run concurrently with the encoder, which keeps the encoder at roughly 92% of wall time. Each rank writes its own Megatron indexed-dataset shards and checkpoints after every batch, so an interrupted job resumes from its last written batch, and a final step merges the shards; for SFT data, the merge can also write a copy with the reasoning spans removed. Two changes to the encoder itself (selecting codes by argmax rather than through a 131,072-way softmax and one-hot product, and pinning TF32 as an explicit precision contract), together with host-side overlap, roughly double throughput without changing a single emitted token, whereas bf16 would flip 7% of codes and we therefore reject it.
2.3.2 Audio Tokenization
Audio tokenization, released in swiss-ai/benchmark-audio-tokenizer, combines the main systems challenges of text and image processing. Text inputs are variable-length, but tokenization is CPU-bound and scales across many cores. Image tokenization is GPU-bound, but resizing enables shape-homogeneous batches with high accelerator utilization. Audio is both GPU-bound and variable-length, making efficient batching difficult without excessive padding or underutilization. The pipeline first converts every source into Lhotse Shar shards (Żelasko et al., 2021) and tokenizes any paired text during that conversion, so the GPU stage handles only audio. Each rank receives its own set of whole shards and never communicates with the others, and a dynamic bucketing sampler groups clips of similar duration into batches capped by total audio duration while CPU workers decode the next batch. Because batched encoding pads each clip to the longest in its batch and the last few codes before the padding are unstable, the pipeline drops the last five codes after padding.
2.4 Ablations
Introducing new modalities into a pretrained language model can degrade its text capabilities, particularly during continued pretraining. Lu et al. (2024) report catastrophic forgetting of language capabilities during joint vision-language pretraining and retain at