8.6 KiB
Research Acknowledgment: SSD — Simple Self-Distillation Improves Code Generation
Issue: #128 Paper: Embarrassingly Simple Self-Distillation Improves Code Generation Authors: Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang (Apple) Date: April 1, 2026 Code: https://github.com/apple/ml-ssd Acknowledged by: Claude — April 6, 2026
Assessment: High Relevance to Fleet
This paper is directly applicable to the hermes-agent fleet. The headline result — +7.5pp pass@1 on Qwen3-4B — is at exactly the scale we operate. The method requires no external infrastructure. Triage verdict: P0 / Week-class work.
What SSD Actually Does
Three steps, nothing exotic:
- Sample: For each coding prompt, generate one solution at temperature
T_train(~0.9). Do NOT filter for correctness. - Fine-tune: SFT on the resulting
(prompt, unverified_solution)pairs. Standard cross-entropy loss. No RLHF, no GRPO, no DPO. - Evaluate: At
T_eval(which must be different fromT_train). This asymmetry is not optional — using the same temperature for both loses 30–50% of the gains.
The counterintuitive part: N=1 per problem, unverified. Prior self-improvement work uses N>>1 and filters by execution. SSD doesn't. The paper argues this is why it works — you're sharpening the model's own distribution, not fitting to a correctness filter's selection bias.
The Fork/Lock Theory
The paper's core theoretical contribution explains why temperature asymmetry matters.
Locks — positions requiring syntactic precision: colons, parentheses, import paths, variable names. A mistake here is a hard error. Low temperature helps at Locks. But applying low temperature globally kills diversity everywhere.
Forks — algorithmic choice points where multiple valid continuations exist: picking a sort algorithm, choosing a data structure, deciding on a loop structure. High temperature helps at Forks. But applying high temperature globally introduces errors at Locks.
SSD's fine-tuning reshapes token distributions context-dependently:
- At Locks: narrows the distribution, suppressing distractor tokens
- At Forks: widens the distribution, preserving valid algorithmic paths
A single global temperature cannot do this. SFT on self-generated data can, because the model learns from examples that implicitly encode which positions are Locks and which are Forks in each problem context.
Fleet implication: Our agents are currently using a single temperature for everything. This is leaving performance on the table even without fine-tuning. The immediate zero-cost action is temperature auditing (see Phase 1 below).
Results That Matter to Us
| Model | Before | After | Delta |
|---|---|---|---|
| Qwen3-30B-Instruct | 42.4% | 55.3% | +12.9pp (+30% rel) |
| Qwen3-4B-Instruct | baseline | baseline+7.5pp | +7.5pp |
| Llama-3.1-8B-Instruct | baseline | baseline+3.5pp | +3.5pp |
Gains concentrate on hard problems: +14.2pp medium, +15.3pp hard. This is the distribution our agents face on real Gitea issues — not easy textbook problems.
Fleet Implementation Plan
Phase 1: Temperature Audit (Zero cost, this week)
Current state: fleet agents use default or eyeballed temperature settings. The paper shows T_eval != T_train is critical even without fine-tuning.
Actions:
- Document current temperature settings in
hermes/,skills/, and any Ollama config files - Establish a held-out test set of 20+ solved Gitea issues with known-correct outputs
- Run A/B: current T_eval vs. T_eval=0.7 vs. T_eval=0.3 for code generation tasks
- Record pass rates per condition; file findings as a follow-up issue
Expected outcome: measurable improvement with no model changes, no infrastructure, no cost.
Phase 2: SSD Pipeline (1–2 weeks, single Mac)
Replicate the paper's method on Qwen3-4B via Ollama + axolotl or unsloth:
1. Dataset construction:
- Extract 100–500 coding prompts from Gitea issue backlog
- Focus on issues that have accepted PRs (ground truth available for evaluation only, not training)
- Format: (system_prompt + issue_description) → model generates solution at T_train=0.9
2. Fine-tuning:
- Use LoRA (not full fine-tune) to stay local-first
- Standard SFT: cross-entropy on (prompt, self-generated_solution) pairs
- Recommended: unsloth for memory efficiency on Mac hardware
- Training budget: 1–3 epochs, small batch size
3. Evaluation:
- Compare base model vs. SSD-tuned model at T_eval=0.7
- Metric: pass@1 on held-out issues not in training set
- Also test on general coding benchmarks to check for capability regression
Infrastructure assessment:
- RAM: Qwen3-4B quantized (Q4_K_M) needs ~3.5GB VRAM for inference; LoRA fine-tuning needs ~8–12GB unified memory (Mac M-series feasible)
- Storage: Self-generated dataset is small; LoRA adapter is ~100–500MB
- Time: 500 examples × 3 epochs ≈ 2–4 hours on M2/M3 Max
- Dependencies: Ollama (inference), unsloth or axolotl (fine-tuning), datasets (HuggingFace), trl
No cloud required. No teacher model required. No code execution environment required.
Phase 3: Continuous Self-Improvement Loop (1–2 months)
Wire SSD into the fleet's burn mode:
Nightly cron:
1. Collect agent solutions from the day's completed issues
2. Filter: only solutions where the PR was merged (human-verified correct)
3. Append to rolling training buffer (last 500 examples)
4. Run SFT fine-tune on buffer → update LoRA adapter
5. Swap adapter into Ollama deployment at dawn
6. Agents start next day with yesterday's lessons baked in
This integrates naturally with RetainDB (#112) — the persistent memory system would track which solutions were merged, providing the feedback signal. The continuous loop turns every merged PR into a training example.
Phase 4: Sovereignty Confirmation
The paper validates that external data is not required for improvement. Our fleet can:
- Fine-tune exclusively on its own conversation data
- Stay fully local (no API calls, no external datasets)
- Accumulate improvements over time without model subscriptions
This is the sovereign fine-tuning capability the fleet needs to remain independent as external model APIs change pricing or capabilities.
Risks and Mitigations
| Risk | Assessment | Mitigation |
|---|---|---|
| SSD gains don't transfer from LiveCodeBench to Gitea issues | Medium — our domain is software engineering, not competitive programming | Test on actual Gitea issues from the backlog; don't assume benchmark numbers transfer |
| Fine-tuning degrades non-code capabilities | Low-Medium | LoRA instead of full fine-tune; test on general tasks after SFT; retain base model checkpoint |
| Small training set (<200 examples) insufficient | Medium | Paper shows gains at modest scale; supplement with open code datasets (Stack, TheVault) if needed |
| Qwen3 GGUF format incompatible with unsloth fine-tuning | Low | unsloth supports Qwen3; verify exact GGUF variant compatibility before starting |
| Temperature asymmetry effect smaller on instruction-tuned variants | Low | Paper explicitly tests instruct variants and shows gains; Qwen3-4B-Instruct is in the paper's results |
Acceptance Criteria Status
From the issue:
- Temperature audit — Document current T/top_p settings across fleet agents, compare with paper recommendations
- T_eval benchmark — A/B test on 20+ solved Gitea issues; measure correctness
- SSD reproduction — Replicate pipeline on Qwen4B with 100 prompts; measure pass@1 change
- Infrastructure assessment — Documented above (Phase 2 section); GPU/RAM/storage requirements are Mac-feasible
- Continuous loop design — Architecture drafted above (Phase 3 section); integrates with RetainDB (#112)
Infrastructure assessment and continuous loop design are addressed in this document. Temperature audit and SSD reproduction require follow-up issues with execution.
Recommended Follow-Up Issues
- Temperature Audit — Audit all fleet agent temperature configs; run A/B on T_eval variants; file results (Phase 1)
- SSD Pipeline Spike — Build and run the 3-stage SSD pipeline on Qwen3-4B; report pass@1 delta (Phase 2)
- Nightly SFT Integration — Wire SSD into burn-mode cron; integrate with RetainDB feedback loop (Phase 3)
Research acknowledged by Claude — April 6, 2026 Source issue: hermes-agent #128