code the originating dataset; the 32 source tags map onto the 23 sources of Table I.1 with none left unmapped. Decontamination against our evaluation suite and a 3,600-token length filter reduce 259,922 source prompts to 245,898 preference pairs, a retention of 94.6%, unevenly distributed: the two precise instruction-following tags lose 12 to 19% while most other sources keep at least 95%. Dolci-Instruct-DPO repeats a minority of its prompts; each repetition receives an independent generation and pairing run, which soft-upweights those prompts rather than duplicating pairs.
Table I.1: Composition of the preference prompt set by source dataset, after decontamination and length filtering. Categories follow the Dolci-Instruct-DPO grouping.
Source
Category
Pairs
%
Dolci Instruct Precise IF
Precise IF
30,320
12.33
WildChat
Chat
28,203
11.47
OpenThoughts3 Science
Science
25,633
10.42
Dolci Instruct Python Algorithms
Code
23,616
9.60
UltraFeedback
General
21,226
8.63
FLAN
Other
14,845
6.04
Tulu 3 Persona MATH
Math
14,082
5.73
Evol CodeAlpaca
Code
12,642
5.14
WildJailbreak
Safety
12,261
4.99
WildGuardMix
Safety
11,846
4.82
SciRiff
Science
8,409
3.42
Tulu 3 Persona IF
Precise IF
6,452
2.62
Aya
Multilingual
5,928
2.41
OpenMathInstruct 2
Math
5,239
2.13
Tulu 3 Persona GSM
Math
4,893
1.99
Multiturn SelfTalk
Multiturn SelfTalk
4,714
1.92
Multiturn SynthCtx
Multiturn SynthCtx
4,570
1.86
Tulu 3 Persona Python
Code
4,481
1.82
CoCoNot
Safety
2,162
0.88
Tulu 3 Persona Algebra
Math
1,917
0.78
TableGPT
Other
1,154
0.47
DaringAnteater
General
821
0.33
OpenAssistant
Chat
484
0.20
Total
245,898
100.00
Table I.2: The 24 response models, ordered by parameter count. We give active parameters for the mixture-of-experts models. Score is the four-aspect judge mean over all prompts; the last two columns give how often each model supplies the chosen and the rejected side of a pair.
Model
Params (B)
Active (B)
Score
Chosen (%)
Rejected (%)
Qwen2.5-0.5B-Instruct
0.5
–
2.186
0.20
7.17
Qwen3-0.6B
0.6
–
2.624
0.33
3.33
Qwen2.5-1.5B-Instruct
1.5
–
2.942
0.41
1.42
Qwen3-1.7B
1.7
–
3.390
1.15
0.84
EuroLLM-1.7B-Instruct
1.7
–
1.598
0.07
37.51
Ministral-3-3B-Instruct-2512
3
–
2.541
0.97
16.82
SmolLM3-3B
3
–
3.559
0.74
0.64
Qwen3-4B-Instruct-2507
4
–
4.384
4.59
0.06
Phi-4-mini-instruct
4
–
1.819
0.10
18.90
Trinity-Nano-Preview
6
–
2.909
0.66
4.29
Qwen3-8B
8
–
4.231
2.83
0.10
Ministral-3-8B-Instruct-2512
8
–
3.301
1.78
3.86
EuroLLM-9B-Instruct-2512
9
–
3.218
1.13
1.22
Ministral-3-14B-Instruct-2512
14
–
3.773
3.90
1.25
EuroLLM-22B-Instruct-2512
22
–
3.425
1.30
1.12
Mistral-Small-24B-Instruct-2501
24
–
3.949
1.69
0.25
Qwen3-30B-A3B-Instruct-2507
30
3
4.513
8.91
0.04
Qwen3-Omni-30B-A3B-Instruct
30
3
4.342
4.01
0.33
Qwen3-32B
32
–
4.433
4.55
0.06
Qwen3-Next-80B-A3B-Instruct
80
3
4.595
11.60
0.30
Mixtral-8x22B-Instruct-v0.1
141
39
3.939
1.35
0.20
Qwen3-235B-A22B-Instruct-2507
235
22
4.650
21.55
0.09
Qwen3.5-397B-A17B
397
17
4.691
16.17
0.08
Mistral-Large-3-675B-Instruct-2512
675
–
4.524
9.99
0.13
I.3 Response Model Pool
All 24 models of Table I.2, which span 0.5B to 675B parameters and four main families, answer every prompt, so that the pool covers a wide quality range and the max-min rule has something to select from. Coverage is complete apart from 1,969 prompts for which Qwen3-Omni-30B-A3B-Instruct returns no completion, leaving 23 candidates; across the full matrix only 251 of 6,236,159 completions fail annotation, and no prompt has all 24 models tie. Qwen3-Omni is omni-modal but we use it text-only. Note that the judge model is itself in the pool and supplies 21.6% of the chosen responses, so we cannot exclude a self-preference effect.
I.4 Completion Generation
Every model answers every prompt once, at temperature 1.0 and a cap of 4,096 generated tokens. We leave top-pp and top-kk unset, so each model falls back to the defaults of its own shipped generation configuration; we therefore sample the pool at a common temperature but not under a single decoding configuratio