to reason at inference time.
Q4.
Whether reasoning supervision during SFT provides a better starting point for subsequent RL remains unsettled. In text reasoning, prior work finds both that SFT can constrain later exploration (Kang et al., 2026) and that stronger reasoning SFT can improve the initialization for RL (Liu et al., 2026); off-policy reasoning traces can also be beneficial when combined with on-policy exploration (Yan et al., 2025). Evidence from multimodal reasoning is similarly mixed (Chen et al., 2025a; Wei et al., 2025), but comes largely from encoder-based VLMs.
We test the same question in our unified discrete-token model. Here, the shift from SFT to RL introduces two distinct mismatches. First, the SFT reasoning traces are generated off-policy by stronger teacher models, whereas RL uses rollouts from the student itself. Second, much of the SFT reasoning supervision is conditioned on multimodal inputs, whereas the RL of this ablation is entirely text-only (Table 12). Thus, the multimodal reasoning trajectories learned during SFT are never directly revisited during RL.
We make the controlled comparison between Text CoT and the seed-42 MM CoT checkpoint from the preceding SFT ablation. We additionally include the production SFT checkpoint of Section 4.2 as a retrospective reference. Unlike the two ablation checkpoints, it is trained on a broader, more heterogeneous mixture without the same pairing and normalization controls, and is therefore not a controlled arm. From each checkpoint, we run RL using our fork of NeMo RL (NVIDIA, 2025)77
7
Available at swiss-ai/Nemo-RL. separately on mathematics, reasoning, and instruction-following prompts (Table 12). Instruction following is trained with thinking off, since our SFT mixture supervises it almost entirely without reasoning traces and explicit reasoning can hurt instruction following (Li et al., 2025b; Fu et al., 2026). Figure 13 tracks held-out performance, while Figure 14 shows the corresponding reward and response-length dynamics.
Table 12: Training configuration of the RL ablation, with one row per prompt set. The sets are drawn from DAPO-Math-17k (Yu et al., 2025), Reasoning Gym (Stojanovski et al., 2025), and nvidia/Nemotron-RL-instruction_following. Each run lasts 200 steps of GRPO (Shao et al., 2024), with 64 prompts and eight rollouts per step. We use binary rewards, a leave-one-out baseline, DAPO asymmetric clipping, no KL penalty, and a constant learning rate of 2×10−62\times 10^{-6}. Rollouts are asynchronous, at most two policy versions stale, and capped at 14,336 new tokens with thinking on and 3,840 with thinking off. Mathematics is scored with Math-Verify (Kydlíček, 2025); for reasoning tasks, Qwen3.6-35B-A3B is used as a fallback judge only when the task verifier rejects an answer.
Prompt Set
Source
Prompts
Reward
Thinking
Mathematics
DAPO-Math-17k
15,984
Math-Verify
On
Reasoning
Reasoning Gym
14,912
Verifier, judge fallback
On
Instruction following
Nemotron-RL IF
46,390
All constraints met
Off
Across mathematics and reasoning, MM CoT retains or increases its advantage over Text CoT throughout RL, despite the latter substantially increasing its reasoning length; on instruction following, where SFT contains little trace supervision, the two converge. These results suggest that the gains introduced by multimodal reasoning supervision are not washed out by subsequent text-only RL, but can persist through on-policy optimization. Figure 13 also shows the production SFT checkpoint as a reference.
Takeaway.
Multimodal reasoning traces do more than raise the starting point for text-only RL: RL improves MM CoT more than Text CoT on mathematics, and on reasoning, where both improve equally, MM CoT keeps its lead from SFT.
00100100200200606070708080
Accuracy (%)MATH-500
00100100200200101015152020
AIME 2025
001001002002005510101515
AIME 2026
00100100200200555560606565
BBH
0010010020020075758080
Accuracy (%)IFEval
00100100200200707075758080
RL stepMulti-IF
0010