f the runs. Preparing the multimodal corpus required substantially more space than initially available within the project allocation, leading to a coordinated cleanup of existing data and an increase in storage quotas before preprocessing could proceed at full scale. As production matured, data was placed according to its I/O characteristics: checkpoints were written to storage suited to large sequential transfers, while tokenized training datasets resided on an IOPS-optimized tier better matched to the fine-grained access patterns of training. Checkpoint size is dominated by optimizer state, which accounts for roughly 90% of the footprint: for the 70B model, removing optimizer state reduces a checkpoint from approximately 1.2 TB to 132 GB. We therefore retain full optimizer state only at phase boundaries and prune it from intermediate checkpoints.
The campaign also reinforces the importance of matching storage tiers and scratch policies to access patterns and retention needs. Tokenized datasets were initially moved to backed-up disk storage to avoid scratch-retention limits, but the additional metadata activity introduced by the backup process became visible as training throughput variability at scale. Frequently accessed training data was therefore placed on SSD-backed scratch storage using the recommended striping policy, which distributes file blocks across multiple storage targets and avoids concentrating large-file traffic on a single server. The same consideration applies to all large objects accessed concurrently by distributed jobs: at 512-node startup, insufficiently striped container images and checkpoints can concentrate I/O on individual storage servers, a pattern clearly visible in the storage-system telemetry. During the campaign, CSCS also introduced a project-scoped storage tier combining scratch-like performance with longer-lived data retention, providing a better fit for such workloads.
Network Safeguards for Data Ingestion.
Collecting web-scale multimodal data from the Internet on a large supercomputing infrastructure introduces network and security considerations that differ substantially from those of conventional training workloads. Many permissively licensed image datasets provide URL manifests rather than the image objects themselves. Corpus construction therefore requires dereferencing large numbers of URLs across a very broad set of Internet domains, producing highly distributed outbound HTTP(S) traffic with characteristics similar to large-scale web crawling.
A small fraction of these destinations are present on security blacklists. During early ingestion runs, requests to such endpoints repeatedly triggered CSCS security monitoring, while retries against denied destinations eventually caused a shared outbound IP address to be blacklisted by Internet Service Providers. Because Internet egress is shared, such an event can affect not only the Apertus campaign but also the other unrelated scientific workloads running concurrently on Alps. Filtering against a single widely used hosts list proved insufficient, and ingestion was paused while we developed a more robust filtering pipeline together with the CSCS network and security teams.
For each ingestion run, the pipeline retrieves fresh copies of multiple public threat-intelligence feeds, including FireHOL IP sets, URLhaus, Spamhaus, and additional feeds used by the CSCS monitoring infrastructure. These are compiled into IP-network, domain, and URL access-control lists consumed by a local web proxy.1212
12
Available at swiss-ai/multimodal-data under download/blacklist. All downloader traffic passes through the proxy, including requests generated by HTTP redirects, and destination hostnames are resolved before the resulting addresses are checked against the IP-based blacklists. Because HTTPS path information is not visible to the proxy without TLS interception, URL-level HTTPS indicators are conservatively promoted to host-level blocks. In the first prod