tion
We estimate the energy of the training runs with the same method as the Apertus 1.0 report: measured GPU-hours times an assumed draw of 560 W per GH200 module, below the configured 660 W power cap (Hernández-Cano et al., 2026). The logged segments of Table 40 total roughly 225,000 GPU-hours for the 8B continued-pretraining stages, 1.50 million for the 70B continued-pretraining stages, and the remainder for the long-context extensions of both models, together roughly 2 million GPU-hours or 1.1 GWh, against roughly 5 GWh for the from-scratch Apertus 1.0 pretraining. The estimate covers the logged production segments only; it excludes the phases missing from Table 40, post-training, evaluation, ablations, and idle reservation time. During the whole project duration, 9 million GPU-hours have been allocated, which translates to an estimate of 5 GWh, excluding the idle times and the resources consumed by the other projects within the Swiss AI initiative that have synergies with the Apertus project but were not part of the training campaign. Alps draws on carbon-free hydropower, and CSCS cools the infrastructure with lake water in a closed cycle.
6.4 Engineering Challenges and Solutions
Software Environment and Node Validation.
The first weeks were characterized by further work on tuning the software environment for production at scale. During this phase, multi-node jobs exposed two interacting issues: a system-level network library affecting collective operations during checkpointing, and rendezvous failures in the campaign-specific container environment following an update to the system software stack. To reduce this coupling, we evolved the container environment to incorporate the required network components directly where appropriate. This deliberately trades some of the low-friction integration provided by the CSCS software environment for greater control over the campaign-specific stack, making long-running production workloads less sensitive to system upgrades.
We also incorporated into every job preamble a node-vetting tool developed by CSCS and made available to all users. Before training starts, it validates the execution environment, GPU health, and intra- and inter-node communication, allowing unsuitable nodes or configurations to be detected before they affect a distributed run. Finally, we hardened just-in-time (JIT) compilation for large-scale execution by placing compilation caches per process on node-local storage. This avoids cross-process and cross-node contention and ensures that generated kernels remain consistent with the software environment in which they are compiled.
Communication at Scale.
Two performance issues encountered during the Apertus 1.5 campaign have close analogues in the Apertus 1.0 experience, allowing previous diagnostic work to accelerate their resolution. In the first case, throughput dropped by roughly an order of magnitude at 256 nodes, but only after restoring from a checkpoint. CSCS engineers traced this distinct failure mode to a memory-registration cache limit in the network stack; the corresponding mitigation was subsequently incorporated into the shared software environment and documented publicly. At scales beyond 256 nodes, we additionally enabled an alternative rendezvous protocol whose implementation had been corrected by the interconnect vendor (HPE) during the Apertus 1.0 campaign, avoiding a separate collapse in collective communication performance at large scale.
Storage Capacity and Data Placement.
Storage capacity and data placement are among the first operational constraints of the runs. Preparing the multimodal corpus required substantially more space than initially available within the project allocation, leading to a coordinated cleanup of existing data and an increase in storage quotas before preprocessing could proceed at full scale. As production matured, data was placed according to its I/O characteristics: checkpoints were written to storage suited to large seq