
We train on value-laden safety prompts, where the charter shapes the response most. The prompts come from the gretelai/gretel-safety-alignment-en-v1 dataset, of which we keep the prompts alone and discard the pre-generated responses. Decontamination against our evaluation suite leaves 8,354 prompts. We compare this against Dolci-Instruct-DPO (Team Olmo et al., 2025), RealToxicityPrompts (Gehman et al., 2020), and PRISM (Kirk et al., 2024), and find that the Gretel safety prompts align the model best. Growing the dataset beyond this size does not have a measurable advantage, so we train on it alone to keep the stage cheap.
Feedback.
The judge is Qwen3.6-27B (Qwen Team, 2026a), chosen over self-reflection, in which the student itself, the DPO checkpoint, critiques its own responses. Both variants train comparably and self-reflection aligns better with the charter, reaching a win rate of 73.2% in comparison to 67.4%, but it costs roughly 3% on math, so we adopt the Qwen judge. We further tune the feedback and teacher templates by hand over successive runs, guided by benchmark performance and by the rollouts of the preceding run. Appendix G gives both prompt templates. The remaining hyperparameters matter little at this scale, since the small learning rate and the short schedule limit how far the model can change.
Training.
We train in verl with vLLM (Kwon et al., 2023) rollouts and thinking disabled throughout. Starting from the DPO checkpoint, we run 117 steps at a batch size of 64, sampling 16 rollouts per prompt, at a learning rate of 1×10−71\times 10^{-7} with 10 warmup steps, and distill the top-64 tokens plus one tail bucket that absorbs the remainder of the distribution. The run takes two hours on 32 nodes of four GH200 GPUs each.
4.7 SFT Ablations
Multimodal SFT corpora usually run to tens of millions of conversations, an order of magnitude above our text mixture of 3.7M, and most of them carry reasoning traces (Wen et al., 2026; Cai et al., 2026; Li et al., 2025a; NVIDIA et al., 2026). Introducing them changes two things at once: text becomes a minority of the mixture, and a reasoning trace, a synthetic derivation written by a larger model that precedes the visible answer and that our chat template places in the thinking span (Section 4.1), enters about a third of the training conversations.
This motivates four questions:
Q1.
How does adding multimodal SFT data change the text capabilities relative to text-only SFT?
Q2.
What does supervising reasoning traces add, on text and on multimodal tasks?
Q3.
Do the gains from trace supervision persist when reasoning is disabled at inference time, or do they require the model to reason?
Q4.
How do off-policy reasoning traces from stronger models during SFT shape what subsequent RL can achieve?
Q1–Q3.
We build the SFT ablations from the cleaned text mixture and a selected subset of the multimodal SFT pool, after decontamination against the evaluation benchmarks, exact deduplication, and answer-format normalization. We construct paired versions of each eligible example with and without reasoning traces, referred to as the CoT and No-CoT arms. Within each pair, the prompt, any multimodal inputs, and the visible answer are identical; only the reasoning trace differs. Traces are retained from the source data where available and generated with Qwen3.8-27B for selected vision examples; a trace is kept only when its conclusion agrees with the visible answer.
Table 11summarizes the four training configurations. Text No-CoT versus MM No-CoT isolates the effect of adding multimodal data, while MM No-CoT versus MM CoT isolates reasoning supervision. Comparing Text CoT with MM CoT further measures the contribution of multimodal reasoning traces beyond text reasoning alone. All configurations initialize from the final Apertus 1.5 8B long-context pretraining checkpoint (Section 3.3) with a fresh optimizer state and are trained for two epochs. We use a sequence length of 8,192 with best-fit pack