01,0.05,0.1,0.5,1.0\}, and batch size ∈{128,256,512}\in\{128,256,512\}, sorted by average; the learning-rate row is in units of 10−710^{-7}. Scores are percentages (higher is better). The two rightmost columns are Apertus 1.0 8B SFT, from which all runs start, and Apertus 1.0 8B Instruct. Δ\Delta is the difference in average from the SFT column, and – marks the SFT column itself. Bold marks the best DPO value per row.
Standard DPO
Baselines
β\beta
0.05
0.1
0.1
0.01
0.1
0.05
0.1
0.05
0.05
0.05
Learning Rate (10−710^{-7})
5
5
10
5
5
1
5
5
5
10
Batch Size
512
512
512
512
256
128
128
256
128
512
SFT
Instruct
Knowledge and Reasoning
MMLU
62.5
61.8
61.9
61.5
62.3
61.0
62.4
62.2
62.6
61.9
58.9
61.7
TruthfulQA
59.8
59.2
61.2
60.2
60.4
58.6
61.9
61.3
64.7
62.4
54.5
56.7
HellaSwag
70.0
70.6
70.0
72.3
70.6
70.6
69.9
67.6
68.2
67.0
72.3
73.8
BBH
55.1
55.5
55.9
50.7
54.9
55.3
55.3
54.1
53.4
54.5
54.1
55.0
ACPBench
32.0
30.1
31.7
30.6
30.9
29.5
32.8
33.4
34.8
34.0
25.4
32.4
DROP
45.1
45.4
44.2
45.9
44.1
45.9
42.2
43.9
41.2
43.4
42.7
44.8
Instruction Following and Chat
IFEval
58.0
63.2
65.6
56.7
64.9
57.3
66.3
60.8
60.4
59.7
60.1
68.4
Multi-IF
56.3
58.7
60.1
50.8
59.5
55.5
60.2
55.2
56.6
55.9
59.7
65.8
AlpacaEval
37.7
32.5
33.6
44.4
33.6
31.4
32.2
39.3
39.7
39.1
5.1
12.9
Math
GSM8K
67.3
65.4
65.3
68.5
65.9
66.4
65.1
65.7
67.4
67.0
56.6
62.6
MATH
21.5
19.0
19.7
23.1
19.7
19.9
19.9
21.8
21.0
21.7
15.2
18.2
MathQA
37.5
37.9
37.1
38.3
37.0
37.3
37.5
36.9
36.6
36.4
37.5
38.0
Coding
HumanEval
71.1
72.6
72.0
71.5
70.1
71.6
71.8
71.5
69.1
69.4
65.0
69.2
MBPP
39.4
38.2
40.0
34.2
39.8
39.0
38.2
40.0
40.8
41.0
36.0
39.0
Multilingual and Cultural
MGSM
52.8
52.9
51.4
48.1
52.8
52.4
51.6
52.0
49.3
51.6
43.9
46.5
Safety and Alignment
ToxiGen
64.4
66.5
57.7
70.5
59.2
73.6
57.0
57.1
56.8
56.8
78.5
70.1
Average
51.9
51.8
51.7
51.7
51.6
51.6
51.5
51.4
51.4
51.4
47.8
51.0
Δ\Delta vs. SFT
+4.1
+4.0
+3.9
+3.9
+3.8
+3.8
+3.7
+3.6
+3.6
+3.5
–
+3.1
Offline DPO, length-normalized loss.
With the batch size fixed at 128, we sweep β\beta and the learning rate (Table J.2). A larger β\beta tolerates a larger learning rate: at 10−610^{-6} only β=25\beta{=}25 improves over the SFT model, and at 5⋅10−75{\cdot}10^{-7} only β∈{5,25}\beta\in\{5,25\} does. At β=25\beta{=}25, the average increases monotonically with the learning rate up to 10−610^{-6}. The six configurations absent from the table do not improve over the SFT model. The best configuration (β=25\beta{=}25, learning rate 10−610^{-6}) reaches Δ=+5.5\Delta{=}{+}5.5, 1.4 points above the best standard DPO run. Extending the grid to β∈{50,100}\beta\in\{50,100\} lowers the best Δ\Delta to +4.4+4.4 and +3.2+3.2, respectively (Table J.3). We use β=25\beta{=}25 for all offline DPO training.
Table J.2: Offline length-normalized DPO from Apertus 1.0 8B SFT with batch size 128: the ten best configurations of the grid β∈{0.5,2.5,5,25}\beta\in\{0.5,2.5,5,25\} ×\times learning rate ∈{10−6,5⋅10−7,10−7,10−8}\in\{10^{-6},5{\cdot}10^{-7},10^{-7},10^{-8}\}, sorted by average; the learning-rate row is in units of 10−710^{-7}. Scores are percentages (higher is better). The two rightmost columns are Apertus 1.0 8B SFT and Instruct; Δ\Delta and – are as in Table J.1. Bold marks the best DPO value per row.
Length-Normalized DPO
Baselines
β\beta
25.0
25.0
5.0
25.0
2.5
5.0
0.5
0.5
25.0
2.5
Learning Rate (10−710^{-7})
10.0
5.0
1.0
1.0
1.0
5.0
1.0
0.1
0.1
0.1
SFT
Instruct
Knowledge and Reasoning
MMLU
62.0
62.4
62.2
61.5
63.3
61.9
60.0
58.1
59.1
59.3
58.9
61.7
TruthfulQA
61.1
60.1
59.2
57.8
59.5
63.7
58.0
54.7
54.5
54.7
54.5
56.7
HellaSwag
76.4
75.5
69.0
72.7
64.9
78.8
70.9
72.2
72.2
72.2
72.3
73.8
BBH
53.6
55.5
53.6
55.4
50.8
32.5
40.9
54.3
54.0
54.2
54.1
55.0
ACPBench
34.0
33.0
31.3
29.5
33.3
28.2
31.7
25.2
25.2
25.0
25.4
32.4
DROP
47.5
46.9
45.7
45.2
45.8
47.4
46.2
43.0
42.8
43.0
42.7
44.8
Instruction Following and Chat
IFEval
64.1
64.5
57.9
63.6
56.6
58.4
53.0
60.8
61.0
60.1
60.1
68.4
Multi-IF
62.0
61.6
55.2
60.8
53.8
53.7
51.0
59.5
59.5
59.