reservation windows, with schedules adjusted at short notice to match system availability.
All training jobs run in containers based on NGC images and integrated with the Alps software and networking environment through site-provided mechanisms. Library versions are pinned where required, and project-specific patches are applied on top of the validated software stack. During the project, the cluster operating system and GPU driver are also upgraded through staged rollouts: changes are first deployed to a small test partition and subsequently rolled out cluster-wide.
The cluster has undergone no major changes since Apertus 1.0 and is documented in detail there (Hernández-Cano et al., 2026).
6.2 Full Training Run Performance
CSCS engineering teams contribute specialized expertise in analyzing and optimizing the performance of the Apertus workloads at scale. During the pretraining preparatory phase, they profile the workloads across the system and apply targeted tuning to its network, storage, memory, and compute components; this work continues as the runs scale up and new bottlenecks emerge. In parallel, CSCS and the Apertus core team jointly tune the 70B pretraining configuration, including sequence parallelism, virtual pipelining, and larger gradient-communication buckets. Together with node-level memory and clock tuning, these optimizations achieve roughly 750 tokens per second per GPU at 256 nodes, on par with the Apertus 1.0 70B run despite the doubled sequence length and enlarged vocabulary. Scaling from 256 to 512 nodes retains roughly 90% parallel efficiency.
Table 40reports the measured performance of the production phases, aggregated from the logged run segments; each SLURM job opens a new segment, so the segment count records how often scheduling windows and failures partition a phase. The 8B sustains roughly 4,900 to 5,500 tokens per second per GPU (270 to 300 TFLOPs per GPU, a model FLOPs utilization (MFU) of 0.27 to 0.30 against the nominal 989 TFLOPs bf16 peak of the GH200, which understates efficiency on Alps, where each GH200 module runs under a 660 W power cap (Section 6.3) and cannot reach that peak), and the 70B 430 to 710 (0.20 to 0.33). The long-context extension pays the expected attention cost: throughput decreases with each doubling of the context window, although cross-document attention masking partially alleviates this degradation. Sustained rates overstate delivered progress: dividing consumed tokens by segment wall time gives an effective rate roughly 30% lower for the 70B, the gap consisting of job startup, checkpoint saving and loading, and stalls before failures.
Table 40: Measured performance of the production training phases, aggregated from the logged wandb run segments. Tokens are per phase, from the differences of the cumulative counter; throughput and TFLOPs per GPU are means over the longest segment of each phase; wall time sums the process time of all segments of the phase.
Phase
Nodes
Tokens
Tokens/s/GPU
TFLOPs/GPU
Segments
Wall (h)
8B stage 1
256
0.81T
5,474
300
11
44
8B stage 2
256
2.80T
4,899
269
19
155
8B stage 3
256
0.40T
5,438
298
5
21
8B extension 32k
256
0.06T
3,510
260
1
6
8B extension 65k
256
0.06T
2,671
267
1
7
8B extension 131k
256
0.06T
1,449
220
2
12
8B extension 262k
256
0.06T
712
181
2
20
70B stage 1
512
0.40T
713
325
21
158
70B stage 2
512
1.21T
562
257
65
463
70B stage 3
512
0.40T
428
195
15
113
70B extension 32k
512
0.06T
422
233
3
18
70B extension 65k
512
0.06T
365
249
2
24
70B extension 131k
512
0.05T
219
206
4
35
70B extension 262k
512
0.05T
119
166
20
56
6.3 Energy Consumption
We estimate the energy of the training runs with the same method as the Apertus 1.0 report: measured GPU-hours times an assumed draw of 560 W per GH200 module, below the configured 660 W power cap (Hernández-Cano et al., 2026). The logged segments of Table 40 total roughly 225,000 GPU-hours for the 8B continued-pretraining stages, 1.50 million for the 70B continued-pretraining stages, and the