for selected sources with open-weight vision-language models. The generation pipelines reject empty outputs and apply source-specific output checks. For long captions of text-dense technical drawings, we additionally detect token loops, runaway number lists, and repeated endings caused by repeated labels and tabular schedules. We quarantine and regenerate flagged captions. These automated checks catch common generation failures, but synthetic captions can still contain factual errors. The public recaptioning pipelines and quality-control tools provide the shared generation and validation infrastructure.33
3
Available at swiss-ai/multimodal-data: recaptioning pipelines under recaption and quality-control tools under tools.
Processing at Scale.
Preparing these corpora on shared supercomputing infrastructure introduces constraints beyond data quality: unavailable source URLs, storage and inode limits, scratch-retention policies, and network restrictions on bulk downloads. The raw MINT-1T snapshot alone occupies approximately 700 TB, so we compress it early and apply successive filters in place. URL-based datasets create a different problem: collecting the images generates crawl-like traffic across many domains, including destinations on security blocklists. Retries against denied endpoints put the Alps supercomputer’s shared outbound IP at risk of blocking, with consequences for unrelated scientific workloads that depend on the same network access. Multimodal data preparation therefore requires coordination with the center’s storage, network, and security teams to protect shared infrastructure while sustaining ingestion at scale. We describe the operational incidents and the layered proxy that enforces network safeguards in Section 6.4.
2.2.2 Vision Training Data
Vision data enters all stages of pretraining and supervised finetuning. The tables in Appendix B give the vision pool of each stage, with samples and input tokens per dataset, complementing the selection and preparation methodology in Section 3.1.
Across all six stages, the vision pools for Apertus 1.5 8B contain 821.6 million samples. Most belong to the unpaired and paired stages of continued pretraining (Stages 1 and 2). We report the share of vision data within each continued-pretraining stage’s token budget separately in Table 3.
Stages 1 and 2 together contain approximately 86% of the vision samples across the six 8B stage pools. Their scale explains why the overall composition in Figure 3 mostly reflects these early stages, despite the different emphasis of later pools.
2.2.3 Vision Data Composition
Figure 3: Composition of the 8B vision samples. The inner segments show the relative dataset sizes (largest first). A dataset can span multiple categories.
We use LiquidAI/LFM2.5-VL-450M to assign each vision sample to one visual category in the composition figures.
General visual material dominates the overall vision pool (Figure 3). The inner ring shows that a few large datasets supply most of this category, while the other categories draw on a tail of smaller datasets alongside their larger sources. Appendix D refines these categories with a taxonomy of 61 fine-grained image types, applied to six large source datasets, and reports the prevalence of broad task signals across the mixture.
Stage 2 has the largest vision pools, with approximately 850 billion input tokens for Apertus 1.5 8B and 534 billion for Apertus 1.5 70B (Figure 4). The composition shifts from general visual material in Stages 1 and 2 toward charts and diagrams in cooldown and long-context pretraining. Medical content is more prominent in the 70B cooldown pool. Geospatial content remains small overall.
Figure 4: Vision data composition by stage, with pool sizes in input tokens on the right.
2.2.4 Audio Training Data
Audio data enters the same six stages. Stage 1 uses unpaired speech, music, and mixed audio, while the later stages concentrate on paired speech and text. Paired speech takes three forms: