0
72.7
64.9
78.8
70.9
72.2
72.2
72.2
72.3
73.8
BBH
53.6
55.5
53.6
55.4
50.8
32.5
40.9
54.3
54.0
54.2
54.1
55.0
ACPBench
34.0
33.0
31.3
29.5
33.3
28.2
31.7
25.2
25.2
25.0
25.4
32.4
DROP
47.5
46.9
45.7
45.2
45.8
47.4
46.2
43.0
42.8
43.0
42.7
44.8
Instruction Following and Chat
IFEval
64.1
64.5
57.9
63.6
56.6
58.4
53.0
60.8
61.0
60.1
60.1
68.4
Multi-IF
62.0
61.6
55.2
60.8
53.8
53.7
51.0
59.5
59.5
59.4
59.7
65.8
AlpacaEval
30.2
26.1
38.8
20.9
40.5
44.1
42.9
5.9
4.8
4.9
5.1
12.9
Math
GSM8K
68.8
67.4
67.2
63.6
66.3
67.2
64.6
57.9
56.7
57.0
56.6
62.6
MATH
19.7
20.7
23.9
18.8
24.4
21.1
23.7
15.2
14.8
15.2
15.2
18.2
MathQA
38.4
38.8
37.6
38.2
37.3
38.1
37.7
37.7
37.7
37.8
37.5
38.0
Coding
HumanEval
70.2
69.1
70.4
67.7
66.5
59.6
64.1
65.8
65.4
64.5
65.0
69.2
MBPP
40.2
39.0
38.2
39.2
38.8
29.8
37.2
36.0
35.2
35.8
36.0
39.0
Multilingual and Cultural
MGSM
52.5
51.5
51.7
50.4
51.7
48.8
51.0
43.4
44.0
43.5
43.9
46.5
Safety and Alignment
ToxiGen
73.2
74.2
68.8
76.4
63.4
72.5
58.2
78.5
78.5
78.4
78.5
70.1
Average
53.4
52.9
51.9
51.4
51.1
50.4
49.4
48.0
47.8
47.8
47.8
51.0
Δ\Delta vs. SFT
+5.5
+5.1
+4.1
+3.5
+3.2
+2.5
+1.6
+0.2
0.0
0.0
–
+3.1
Table J.3: Extension of the length-normalized DPO grid of Table J.2 to β∈{50,100}\beta\in\{50,100\}, from Apertus 1.0 8B SFT with batch size 128, sorted by average; the learning-rate row is in units of 10−710^{-7}. Scores, Δ\Delta, –, and the baseline columns are as in Table J.2. Bold marks the best DPO value per row.
Length-Normalized DPO
Baselines
β\beta
50.0
50.0
100.0
50.0
100.0
100.0
Learning Rate (10−710^{-7})
10
5
10
1
5
1
SFT
Instruct
Knowledge and Reasoning
MMLU
61.2
60.7
60.7
61.2
62.4
59.9
58.9
61.7
TruthfulQA
59.0
58.6
57.8
56.9
57.2
56.1
54.5
56.7
HellaSwag
75.2
74.7
74.3
72.7
74.0
72.7
72.3
73.8
BBH
56.0
55.9
55.4
55.3
55.5
54.4
54.1
55.0
ACPBench
31.6
29.2
28.3
27.7
28.1
26.8
25.4
32.4
DROP
45.4
45.9
45.4
44.4
45.1
44.0
42.7
44.8
Instruction Following and Chat
IFEval
62.5
62.8
63.4
63.0
61.4
62.3
60.1
68.4
Multi-IF
63.9
61.7
61.9
60.9
60.6
60.2
59.7
65.8
AlpacaEval
19.6
16.6
11.7
13.9
10.4
9.0
5.1
12.9
Math
GSM8K
66.1
64.8
65.4
62.3
63.5
60.6
56.6
62.6
MATH
18.3
17.5
16.5
17.0
16.2
15.6
15.2
18.2
MathQA
38.8
38.6
39.0
38.3
38.1
38.1
37.5
38.0
Coding
HumanEval
71.8
70.7
70.5
68.3
68.9
65.8
65.0
69.2
MBPP
38.0
39.2
38.6
38.0
38.2
37.4
36.0
39.0
Multilingual and Cultural
MGSM
50.0
50.5
48.1
48.6
48.2
47.3
43.9
46.5
Safety and Alignment
ToxiGen
78.2
78.8
79.6
78.5
78.4
78.5
78.5
70.1
Average
52.2
51.7
51.0
50.4
50.4
49.3
47.8
51.0
Δ\Delta vs. SFT
+4.4
+3.8
+3.2
+2.6
+2.5
+1.5
–
+3.1
Offline DPO, transfer to Apertus 1.5 8B.
The preceding grids use the Apertus 1.0 8B SFT checkpoint, and we apply the selected configuration to the Apertus 1.5 8B SFT checkpoints without a new search. As a check, a learning rate of 5⋅10−65{\cdot}10^{-6} on an intermediate Apertus 1.5 8B SFT checkpoint changes the averages by at most 0.1 points (48.8 vs. 48.8 on the development benchmarks and 44.0 vs. 43.9 on the held-out benchmarks). We therefore keep the learning rate of 10−610^{-6}.
Online DPO.
With β=0.1\beta{=}0.1 and 256 prompts per step, we sweep the learning rate over {10−5,5⋅10−6,10−6}\{10^{-5},5{\cdot}10^{-6},10^{-6}\} (Table J.4). A learning rate of 5⋅10−65{\cdot}10^{-6} performs best (Δ=+4.4\Delta{=}{+}4.4), ahead of 10−610^{-6} (+3.3+3.3), 10−510^{-5} (+3.2+3.2), and offline DPO from the same checkpoint (+3.7+3.7).
Table J.4: Learning-rate sweeps for online DPO and offline→\toonline DPO (β=0.1\beta{=}0.1, 256 prompts per step) from an intermediate Apertus 1.5 8B SFT checkpoint, sorted by average within each group. Scores are percentages (higher is better). The learning-rate row is in units of 10−710^{-7}. Offline is length-normalized offline DPO (β=25\beta{=}25, learning rate 10−610^{-6}) from the same checkpoint, which also initializes offline→\toonline; the rightmost column is the SFT checkpoint, and – marks it in the Δ\Delta row. In these runs, rollouts come from the initial instead of the current polic