 Scores are percentages (higher is better). The learning-rate row is in units of 10−710^{-7}. Offline is length-normalized offline DPO (β=25\beta{=}25, learning rate 10−610^{-6}) from the same checkpoint, which also initializes offline→\toonline; the rightmost column is the SFT checkpoint, and – marks it in the Δ\Delta row. In these runs, rollouts come from the initial instead of the current policy because of an implementation issue fixed afterward; on-policy training then uses the selected learning rate. Bold marks the best DPO value per row.
Online DPO
Offline→\toOnline
Learning Rate (10−710^{-7})
50
10
100
50
10
5
Offline
SFT
Knowledge and Reasoning
MMLU
62.8
62.1
62.8
63.8
61.7
62.4
61.3
59.3
TruthfulQA
62.2
55.2
65.3
64.1
57.2
57.4
57.9
53.4
HellaSwag
70.8
68.0
72.4
71.9
69.3
70.8
71.6
70.8
BBH
54.2
55.9
51.1
52.5
55.8
55.2
55.1
54.6
ACPBench
41.1
37.5
41.2
39.4
35.9
33.7
33.7
28.1
DROP
36.3
39.5
34.6
35.9
45.8
46.7
47.0
47.4
Instruction Following and Chat
IFEval
77.8
77.5
77.8
79.1
76.0
73.6
72.6
71.7
Multi-IF
72.1
72.6
69.8
73.8
70.7
70.2
68.1
68.9
AlpacaEval
25.4
24.5
21.7
41.0
36.4
26.2
27.2
9.4
Math
GSM8K
70.1
65.4
65.0
68.8
67.0
68.5
69.7
64.5
MATH
31.6
32.1
28.6
32.8
30.2
29.6
28.7
22.9
MathQA
38.9
39.0
39.9
37.2
40.7
41.0
40.8
39.6
Coding
HumanEval
70.5
71.1
73.4
73.2
79.1
79.1
80.0
79.3
MBPP
43.0
42.0
39.4
42.0
43.6
44.0
42.0
40.6
Multilingual and Cultural
MGSM
58.0
53.5
51.1
53.0
50.3
50.3
51.7
43.4
Safety and Alignment
ToxiGen
82.8
83.4
83.8
84.7
82.3
82.0
78.6
73.0
Average
56.1
55.0
54.9
57.1
56.4
55.7
55.4
51.7
Δ\Delta vs. SFT
+4.4
+3.3
+3.2
+5.4
+4.7
+4.0
+3.7
–
Combining offline and online DPO.
We evaluate two combinations of offline and online DPO: a sequential pipeline and a joint data mixture.
In the sequential offline→\toonline setting, online DPO starts from a trained offline DPO checkpoint. The learning-rate sweep for the online phase again favors 5⋅10−65{\cdot}10^{-6} (Δ=+5.4\Delta{=}{+}5.4), ahead of 10−610^{-6} (+4.7+4.7) and 5⋅10−75{\cdot}10^{-7} (+4.0+4.0; Table J.4).
The mixture adds 256 offline pairs to each online step. Table J.5 compares the four variants (offline, online, offline→\toonline, and mixture) with the selected hyperparameters. All four improve over the SFT checkpoint (51.4 on the development benchmarks, 49.3 on the held-out benchmarks). The mixture scores highest (55.5 / 53.8), followed by offline→\toonline (55.4 / 53.5), online (54.6 / 52.0), and offline (54.0 / 50.8).
The final Apertus 1.5 8B recipe uses online DPO alone. When we repeat the comparison of offline, online, and mixture DPO on the four final Apertus 1.5 8B SFT checkpoints, online DPO matches the mixture; the small gains of the combined variants on the intermediate checkpoint do not justify the additional data streams and training stages.
Table J.5: DPO variants from another intermediate Apertus 1.5 8B SFT checkpoint (rightmost column) with thinking disabled, sorted by development average; scores are percentages (higher is better). Offline uses length-normalized DPO (β=25\beta{=}25, learning rate 10−610^{-6}, batch size 128); the others use β=0.1\beta{=}0.1, learning rate 5⋅10−65{\cdot}10^{-6}, and 256 prompts per step; the mixture adds 256 offline pairs to each step, and offline→\toonline starts from the offline checkpoint. Held-out groups contain the held-out benchmarks and the others the development benchmarks; each average is over all benchmarks of its split. Bold marks the best DPO value per row.
Benchmark
Mixture
Offline→\toOnline
Online
Offline
SFT
Knowledge and Reasoning
MMLU
67.7
68.1
68.3
68.3
63.8
MMLU-Pro
45.3
43.8
36.4
36.7
27.7
TruthfulQA
60.9
59.5
59.1
55.0
49.5
CommonsenseQA
75.3
74.7
74.9
74.4
74.5
SQuADv2
23.3
25.9
27.2
13.6
16.5
HellaSwag
69.9
73.6
70.0
75.8
73.5
BBH
67.5
67.1
67.4
67.0
64.7
ACPBench
35.8
43.0
35.6
32.3
27.8
DROP
51.3
50.7
51.4
50.1
48.5
Instruction Following and Chat
IFEval
85.0
85.0
83.9
80.8
78.2
Multi-IF
81.0
82.2
81.9
78.8
74.0
AlpacaEval
41.1
49.4
38.6
49.9
20.4
Math
GSM8K
80