ting feedback and instructs the model not to mention it, to prevent leaking the feedback into model responses.
SDPO Judge Prompt
You are an expert reviewer. Your job is to read the user prompt and the assistant response below, then judge how well the response follows the constitution that follows, and give helpful, concise, constructive feedback to the response writer on how to better align the response with the constitution.
Here is the constitution:
<CONSTITUTIONAL_PRINCIPLES>
[the 11 sections of the Apertus charter, verbatim]
</CONSTITUTIONAL_PRINCIPLES>
<USER_PROMPT>
{prompt}
</USER_PROMPT>
<ASSISTANT_RESPONSE>
{response}
</ASSISTANT_RESPONSE>
Focus on the most important ways the response could better follow the constitution. Do not just give feedback to refuse the response, only do it, if it is REALLY necessary as the model will be trained based on your feedback and should not learn to overrefuse. Address the response writer directly as "you". Do not output a score, a grade, a JSON object, or any preamble, and do not quote principle numbers. Output only one or two sentences of actionable feedback.
SDPO Teacher Prompt
Here is the user prompt:
<USER_PROMPT>
{prompt}
</USER_PROMPT>
An expert reviewer evaluated an earlier draft answer to this request against a set of guiding principles and left the following feedback. Use it to write a better answer:
<REVIEWER_FEEDBACK>
{feedback_raw}
</REVIEWER_FEEDBACK>
Do not refer to or mention any guidelines, principles, or reviewer feedback in your response.
Appendix H Supervised Finetuning Details
H.1 Per-Benchmark SFT Ablation Results
Table H.1lists the per-benchmark score changes behind each bar of Figure 12.
Table H.1: Per-benchmark score changes behind Figure 12, in percentage points. Q1 compares MM against Text at matched supervision and response mode; Q2 compares MM CoT with thinking on against MM No-CoT with thinking off; Q3 compares MM CoT against MM No-CoT, both with thinking off. Thinking is forced and sampled at temperature 0.6 and top-p 0.95; without thinking, decoding is greedy. Multimodal values average three seeds; the text-only reference is one seed. The average rows reproduce the bars of Figure 12, and – marks comparisons the figure does not make. *MGSM is left out of the text average, as in the figure: its zero-shot prompt asks for a step-by-step answer, and the multimodal No-CoT arms mostly reply with the final number alone (median four words, against 137 for the text-only No-CoT arm), so their score reflects answer style and ranges from 27.2 to 39.2 across seeds. With reasoning traces in SFT, the multimodal arm writes out its work again and scores 69.9 even with thinking forced off, level with the text-only CoT arm (69.2).
Q1: MM vs Text
Q2
Q3
Benchmark
No-CoT, Off
CoT, Off
CoT, On
CoT On vs
No-CoT Off
CoT vs No-CoT,
both Off
Mathematics
GSM8K
+0.5
+3.2
+2.2
+6.6
+0.8
MGSM*
−-35.5
+0.7
+2.9
+50.4
+36.9
MATH
−-2.6
+0.4
+0.5
+16.2
+3.8
Minerva MATH
−-0.3
+0.4
+2.7
+18.3
+2.7
Knowledge
MMLU-Pro
+1.2
+0.4
+8.3
−-3.5
−-1.6
MMLU (flan CoT)
−-2.9
+7.7
+6.8
+9.0
+5.3
Reasoning
BBH (3-shot CoT)
+1.2
+4.1
+8.0
−-9.9
+2.9
Multilingual
Global-MMLU
+0.8
+0.8
+2.9
+4.7
−-0.9
INCLUDE-44
+0.9
−-1.2
+2.1
+1.1
−-2.4
INCLUDE-45
0.0
−-0.6
+4.2
−-2.8
−-1.5
SwitzerlandQA
−-0.8
−-1.6
+2.4
−-2.4
−-1.9
Instruction Following
IFEval
+3.1
−-0.6
+1.3
−-14.0
−-1.2
IFBench
−-1.1
−-0.9
+4.1
+5.6
−-3.1
Multi-IF
+0.3
−-0.3
+1.0
−-12.4
−-1.1
Code
HumanEval
−-2.2
−-0.2
+0.8
−-5.5
+0.2
MBPP
−-3.3
−-1.0
−-1.7
−-3.5
+0.3
Reading
DROP (F1)
+5.9
+4.1
+4.5
−-3.8
−-1.8
Text average (16 tasks)
0.0
+0.9
+3.1
+0.2
0.0
Vision
MMMU
–
–
–
+7.3
−-1.4
MMMU-Pro
–
–
–
+7.8
−-1.2
MMStar
–
–
–
+18.2
−-4.7
MathVision
–
–
–
−-4.1
−-2.7
ScienceQA
–
–
–
+7.1
−-2.8
BLINK
–
–
–
+7.8
−-3.7
VisualPuzzles
–
–
–
+0.9
−-1.7
AI2D
–
–
–
+4.5
−-2.6
ChartQA
–
–
–
+7.4
−-1.9
DocVQA
–
–
–
−-2.5
−-0.4
TextVQA
–
–
–
−-4.0
−-0.5
RealWorldQA
–
–
–
+5.9
−-4.5
POPE
–
–
–
+0.7
−-3.4
OCRBench
–
–
–
−-3.5
+0.2
Vision average (14 tasks)
–
–
–
+3.