is an instruction specifying the constraint and saying that this constraint should be followed throughout the whole conversation. The following user turns are prompts, sampled from the capability-filtering SFT prompt pool and the Dolci Instruct SFT Pool, with no constraints. We verify the completions similarly, by checking whether they fulfill the initial constraint and whether they are coherent.
Improving Non-verifiable Instruction Following.
In real-life user-AI conversations, instructions can be presented in a form which is hard to verify, e.g., “When talking about table tennis, focus more on specific forehand serve technique of this player…”, which can result in numerous correct responses, but the verification method would be based on an LLM-as-a-judge rather than a classic IFEval-like hard-coded constraint check. Moreover, conversations in real life are long and multi-turn with instructions and preferences shifting mid-dialogue. To target this area more, we construct a dataset with a multi-agent pipeline in which three roles, such as a planner, a responder, and an evaluator, collaborate to generate and validate each dialogue. The planner, given a seed context, a target challenge category, and a one-shot example from MultiChallenge (Deshpande et al., 2025), produces a JSON plan specifying the challenge type, an NN-turn user question sequence, a hidden trap design, and concrete detection criteria. A trap is a hidden test planted in the conversation (e.g., an early instruction or detail the user later implicitly relies on) that the assistant fails (“triggers”) if it forgets or mishandles it, and the dataset keeps only dialogues in which the trap is set up but not triggered. The responder simulates the assistant under test by following the planned questions, and the evaluator inspects the full dialogue to verify that the trap is present and whether it is triggered, and to score quality on a 10-point scale across safety, accuracy, naturalness, and authenticity. We use seed contexts from Nemotron-Instruction-Following-Chat-v1 (NVIDIA et al., 2025), MiniMax-M2.5 as a generator model (planning and responding), and Qwen3-235B as a judge (evaluator) to mitigate model-specific bias.
For each seed we generate four plans (one per challenge category) and three trajectories per plan, then retain only dialogues in which the trap is correctly introduced but not triggered by the assistant, selecting the top-1 scored trajectory per seed (top-2 for eight-turn chats) to balance quality and diversity. The generated 30,000 dialogues span one to eight conversation turns, with an average of three turns. The dataset shows improved performance on tasks involving more vague constraints compared to standard IF benchmarks, while its multi-turn nature assists with improving on Multi-IF.
Reinforcement Learning.
We sample the IF RL prompts from the capability-filtering SFT prompt pool and concatenate them with constraints from IFTrain (Pyatkin et al., 2025) and IFEval (Zhou et al., 2023). We either prepend or append the constraints to the instruction.
Preference Alignment.
Instruction following also profits from the preference stage of Section 4.6, although no constraint checker is involved: prompts with formatting and content constraints are part of the Dolci-Instruct-DPO prompt set, and the judge scores each rollout on how well it serves the request. To isolate this contribution, we apply online DPO directly to the SFT checkpoint of Apertus 1.5 8B, skipping reinforcement learning, with the recipe of Section 4.6 except for eight rollouts per prompt instead of 12 (Table 9). Judge-based preference optimization alone matches the verifier-driven reinforcement-learning stage on IFEval and comes within two points of it on Multi-IF, even though instruction following is a verifiable domain. The gain is specific to on-policy training: offline DPO from the same checkpoint leaves IFEval unchanged and adds roughly three points on Multi-IF. The two stages are comple