checkpoint, offline DPO alone reaches an average of 56.1, above the mixture (55.6) and online DPO (55.5). The final Apertus 1.5 70B recipe runs offline DPO followed by online DPO (Section 4.6), which restores the response length before on-policy training.
Table J.7: Offline DPO, mixture, and online DPO from the Apertus 1.5 70B RL checkpoint, sorted by average; the rightmost column is the RL checkpoint. Offline DPO uses the configuration of Section 4.6; online DPO and the mixture use β=0.1\beta{=}0.1 and a learning rate of 5⋅10−65{\cdot}10^{-6}, and the mixture adds 256 offline pairs to each online step. Scores in percent (higher is better) with thinking disabled; for degeneration, lower is better. Bold marks the best DPO value per skill-group and average row. Each skill-group row is the mean over the benchmarks within that group, and the overall average is the mean over all individual benchmarks.
Skill Group
Offline
Mixture
Online
RL
Knowledge
56.4
61.7
61.7
59.4
Reasoning
62.2
63.5
62.4
60.9
Instruction Following
84.1
88.6
88.5
88.5
Chat
39.5
24.4
24.2
19.5
Math
34.2
35.6
36.0
35.7
Coding
75.7
75.7
74.5
74.1
Multilingual
58.3
57.6
57.9
57.8
Cultural Knowledge
71.1
66.8
67.1
63.6
Safety
70.1
71.4
71.0
69.9
Charter Alignment
62.8
53.0
53.5
50.9
Tool Use
52.7
47.9
47.2
43.5
Overall Average
56.1
55.6
55.5
54.0
Mean response length (tokens)
501.2
241.7
231.0
252.9
Degeneration (%)
10.6
1.1
2.2
10.6
J.2 Judge Ablations
The online DPO judge described in Section 4.6 scores each rollout on helpfulness alone with Qwen3.6-27B, rather than the offline stage’s four-aspect rubric with Qwen3-235B-A22B-Instruct-2507. We ablate both departures: the aspect simplification and the choice of judge model.
J.2.1 Judging Aspects Simplification
We test whether the four aspect-wise scores from Qwen3-235B-A22B-Instruct-2507 are correlated enough that scoring on one alone is a safe simplification, using ActiveUltraFeedback/ultrafeedback: 60,829 prompts1414
14
The released dataset contained 10 prompts twice; we keep only the first occurrence of each., each with responses from a pool of 30 models rated on all four aspects by Qwen3-235B-A22B-Instruct-2507.
Prompt difficulty and response-model strength each push a response’s score on all four aspects up or down together, inflating pairwise correlations between aspects for reasons unrelated to whether the aspects actually agree. Writing sa,p,ms_{a,p,m} for aspect aa’s expected rating on the response to prompt pp from model mm, we remove both effects before correlating:
s~a,p,m=sa,p,m−s¯a,p,⋅−s¯a,⋅,m+s¯a,⋅,⋅,\tilde{s}_{a,p,m}=s_{a,p,m}-\bar{s}_{a,p,\cdot}-\bar{s}_{a,\cdot,m}+\bar{s}_{a,\cdot,\cdot},
(1)
where s¯a,p,⋅\bar{s}_{a,p,\cdot}, s¯a,⋅,m\bar{s}_{a,\cdot,m}, and s¯a,⋅,⋅\bar{s}_{a,\cdot,\cdot} are aspect aa’s mean score over responses to prompt pp, over responses from model mm, and over all responses, respectively.
The four aspects are strongly correlated both before1515
15
On the raw scores, before removing prompt and model effects, correlations are higher still, ranging from 0.749 (Helpfulness vs. Truthfulness) to 0.849 (Helpfulness vs. Instruction Following). and after deconfounding (Table J.8). Pairwise Spearman correlations are close to the Pearson ones, differing by 0.029 on average and 0.042 at most (Helpfulness vs. Instruction Following), so the pairwise relationships are close to linear rather than merely monotonic. Scoring on one aspect alone, rather than averaging all four, is therefore a safe simplification.
Table J.8: Pairwise Pearson correlation between the four aspects, pooled across all response models, on scores with prompt and model effects removed via equation 1.
Helpfulness
Honesty
Instr. Following
Truthfulness
Helpfulness
–
0.710
0.748
0.644
Honesty
0.710
–
0.680
0.745
Instruction Following
0.748
0.680
–
0.641
Truthfulness
0.644
0.745
0.641
–
Average
0.701
0.712
0.690
0.677
To identify which aspect to keep, we compare all four on three external judging benchmarks: JudgeBench (GPT spl