
Preference Data.
Both online and offline DPO draw from the same prompt set, a filtered and decontaminated subset of Dolci-Instruct-DPO (Team Olmo et al., 2025), whose sources span reasoning, code, instruction following, multilingual chat, mathematics, and safety. We keep the prompts and discard the original responses and preference labels. Decontamination against our evaluation suite and a 3,600-token length filter leave 245,898 preference pairs. For the offline data we generate responses with a pool of 24 models spanning 0.5B to 675B parameters, so that the pool spans a wide range of response quality. We then score these prompt-completion pairs with Qwen3-235B-A22B-Instruct-2507 (Yang et al., 2025a), using the rubric-based prompts of ActiveUltraFeedback (Melikidze et al., 2026). We elicit scores on a one-to-five scale for truthfulness, helpfulness, honesty, and instruction following, taking the expected value under the judge’s distribution over the five rating tokens. We take the final score as the average over the four aspects. We build each preference pair by MaxMin selection, taking the highest-scoring response as chosen and the lowest-scoring as rejected. Melikidze et al. (2026) compare this rule against a variety of response-pairing strategies and find MaxMin the strongest of them, which is consistent with the Delta Learning Hypothesis, under which the gap between the two sides of a pair, rather than the absolute quality of either side, drives the gain (Geng et al., 2025).
Neither stage trains on multimodal or long-context data, and both run with thinking disabled. Appendix I gives the rubric prompts and reports the per-source composition of the prompt set, the model pool, and the score distribution of the resulting pairs.
Offline DPO.
We begin preference alignment with offline DPO (Rafailov et al., 2023) for Apertus 1.5 70B starting from its reinforcement-learning checkpoint (Section 4.5) and using the dataset described above. Our implementation uses a length-normalized version (Meng et al., 2024; Lambert et al., 2025) with KL regularization coefficient β=25\beta=25, learning rate of 1×10−61\times 10^{-6}, linear learning rate scheduler with a warmup ratio of 0.1, the AdamW optimizer (Loshchilov & Hutter, 2019) (β1=0.9\beta_{1}=0.9, β2=0.999\beta_{2}=0.999, ϵ=10−8\epsilon=10^{-8}) with no weight decay, global batch size of 128, for one epoch at a maximum sequence length of 4,096 tokens.
Online DPO.
We apply online DPO to both Apertus 1.5 8B and Apertus 1.5 70B, starting from the reinforcement-learning (Section 4.5) and the final offline DPO checkpoints, respectively. This training uses the prompts from the Dolci-Instruct-DPO dataset while discarding the offline responses. For each batch of prompts we sample rollouts from the current policy with SGLang (Zheng et al., 2024) at temperature 1.0 and unrestricted top-pp and top-kk, score them, and form one MaxMin pair per prompt. We then use this batch of prompt-completion pairs to update the weights of the models with the DPO loss. We sample 12 rollouts per prompt for Apertus 1.5 8B and eight for Apertus 1.5 70B, where the smaller count is a concession to the cost of generation at that scale. To speed up training and reduce the scoring latency, we deploy Qwen3.6-27B (Qwen Team, 2026a) as a judge and use only the helpfulness aspect instead of the average of four. Section J.2 ablates both departures: the four aspect scores are highly correlated, and Qwen3.6-27B matches or exceeds the offline stage’s much larger judge on external preference benchmarks despite the size gap.
For both model sizes, online DPO uses KL penalty coefficient β=0.1\beta=0.1, a linear schedule with a warmup ratio of 0.1, the AdamW optimizer (Loshchilov & Hutter, 2019) (β1=0.9\beta_{1}=0.9, β2=0.999\beta_{2}=0.999, ϵ=10−8\epsilon=10^{-8}) with a weight decay of 0.01, and prompt and response lengths of 2,048 tokens each. The prompt cap removes a further 3,210 prompts, so an online epoch covers 242,688 of t