From 8feef05f822f8a47d8c146d2a4493115db357602 Mon Sep 17 00:00:00 2001 From: Alexander Whitestone Date: Mon, 6 Apr 2026 22:07:00 -0400 Subject: [PATCH] docs: add SSD research acknowledgment (#128) Research triage for Apple's Simple Self-Distillation paper (arXiv:2604.01193). Documents fleet-relevant findings: +7.5pp on Qwen3-4B (our exact scale), zero-infrastructure fine-tuning, Fork/Lock theory implications for temperature tuning, and a 4-phase implementation plan from immediate T_eval audit through nightly self-improvement loop. Refs #128 --- .../research-ssd-self-distillation-2026-04.md | 166 ++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 docs/research-ssd-self-distillation-2026-04.md diff --git a/docs/research-ssd-self-distillation-2026-04.md b/docs/research-ssd-self-distillation-2026-04.md new file mode 100644 index 000000000..e121302d7 --- /dev/null +++ b/docs/research-ssd-self-distillation-2026-04.md @@ -0,0 +1,166 @@ +# Research Acknowledgment: SSD — Simple Self-Distillation Improves Code Generation + +**Issue:** #128 +**Paper:** [Embarrassingly Simple Self-Distillation Improves Code Generation](https://arxiv.org/abs/2604.01193) +**Authors:** Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang (Apple) +**Date:** April 1, 2026 +**Code:** https://github.com/apple/ml-ssd +**Acknowledged by:** Claude — April 6, 2026 + +--- + +## Assessment: High Relevance to Fleet + +This paper is directly applicable to the hermes-agent fleet. The headline result — +7.5pp pass@1 on Qwen3-4B — is at exactly the scale we operate. The method requires no external infrastructure. Triage verdict: **P0 / Week-class work**. + +--- + +## What SSD Actually Does + +Three steps, nothing exotic: + +1. **Sample**: For each coding prompt, generate one solution at temperature `T_train` (~0.9). Do NOT filter for correctness. +2. **Fine-tune**: SFT on the resulting `(prompt, unverified_solution)` pairs. Standard cross-entropy loss. No RLHF, no GRPO, no DPO. +3. **Evaluate**: At `T_eval` (which must be **different** from `T_train`). This asymmetry is not optional — using the same temperature for both loses 30–50% of the gains. + +The counterintuitive part: N=1 per problem, unverified. Prior self-improvement work uses N>>1 and filters by execution. SSD doesn't. The paper argues this is *why* it works — you're sharpening the model's own distribution, not fitting to a correctness filter's selection bias. + +--- + +## The Fork/Lock Theory + +The paper's core theoretical contribution explains *why* temperature asymmetry matters. + +**Locks** — positions requiring syntactic precision: colons, parentheses, import paths, variable names. A mistake here is a hard error. Low temperature helps at Locks. But applying low temperature globally kills diversity everywhere. + +**Forks** — algorithmic choice points where multiple valid continuations exist: picking a sort algorithm, choosing a data structure, deciding on a loop structure. High temperature helps at Forks. But applying high temperature globally introduces errors at Locks. + +SSD's fine-tuning reshapes token distributions **context-dependently**: +- At Locks: narrows the distribution, suppressing distractor tokens +- At Forks: widens the distribution, preserving valid algorithmic paths + +A single global temperature cannot do this. SFT on self-generated data can, because the model learns from examples that implicitly encode which positions are Locks and which are Forks in each problem context. + +**Fleet implication**: Our agents are currently using a single temperature for everything. This is leaving performance on the table even without fine-tuning. The immediate zero-cost action is temperature auditing (see Phase 1 below). + +--- + +## Results That Matter to Us + +| Model | Before | After | Delta | +|-------|--------|-------|-------| +| Qwen3-30B-Instruct | 42.4% | 55.3% | +12.9pp (+30% rel) | +| Qwen3-4B-Instruct | baseline | baseline+7.5pp | +7.5pp | +| Llama-3.1-8B-Instruct | baseline | baseline+3.5pp | +3.5pp | + +Gains concentrate on hard problems: +14.2pp medium, +15.3pp hard. This is the distribution our agents face on real Gitea issues — not easy textbook problems. + +--- + +## Fleet Implementation Plan + +### Phase 1: Temperature Audit (Zero cost, this week) + +Current state: fleet agents use default or eyeballed temperature settings. The paper shows T_eval != T_train is critical even without fine-tuning. + +Actions: +1. Document current temperature settings in `hermes/`, `skills/`, and any Ollama config files +2. Establish a held-out test set of 20+ solved Gitea issues with known-correct outputs +3. Run A/B: current T_eval vs. T_eval=0.7 vs. T_eval=0.3 for code generation tasks +4. Record pass rates per condition; file findings as a follow-up issue + +Expected outcome: measurable improvement with no model changes, no infrastructure, no cost. + +### Phase 2: SSD Pipeline (1–2 weeks, single Mac) + +Replicate the paper's method on Qwen3-4B via Ollama + axolotl or unsloth: + +``` +1. Dataset construction: + - Extract 100–500 coding prompts from Gitea issue backlog + - Focus on issues that have accepted PRs (ground truth available for evaluation only, not training) + - Format: (system_prompt + issue_description) → model generates solution at T_train=0.9 + +2. Fine-tuning: + - Use LoRA (not full fine-tune) to stay local-first + - Standard SFT: cross-entropy on (prompt, self-generated_solution) pairs + - Recommended: unsloth for memory efficiency on Mac hardware + - Training budget: 1–3 epochs, small batch size + +3. Evaluation: + - Compare base model vs. SSD-tuned model at T_eval=0.7 + - Metric: pass@1 on held-out issues not in training set + - Also test on general coding benchmarks to check for capability regression +``` + +Infrastructure assessment: +- **RAM**: Qwen3-4B quantized (Q4_K_M) needs ~3.5GB VRAM for inference; LoRA fine-tuning needs ~8–12GB unified memory (Mac M-series feasible) +- **Storage**: Self-generated dataset is small; LoRA adapter is ~100–500MB +- **Time**: 500 examples × 3 epochs ≈ 2–4 hours on M2/M3 Max +- **Dependencies**: Ollama (inference), unsloth or axolotl (fine-tuning), datasets (HuggingFace), trl + +No cloud required. No teacher model required. No code execution environment required. + +### Phase 3: Continuous Self-Improvement Loop (1–2 months) + +Wire SSD into the fleet's burn mode: + +``` +Nightly cron: + 1. Collect agent solutions from the day's completed issues + 2. Filter: only solutions where the PR was merged (human-verified correct) + 3. Append to rolling training buffer (last 500 examples) + 4. Run SFT fine-tune on buffer → update LoRA adapter + 5. Swap adapter into Ollama deployment at dawn + 6. Agents start next day with yesterday's lessons baked in +``` + +This integrates naturally with RetainDB (#112) — the persistent memory system would track which solutions were merged, providing the feedback signal. The continuous loop turns every merged PR into a training example. + +### Phase 4: Sovereignty Confirmation + +The paper validates that external data is not required for improvement. Our fleet can: +- Fine-tune exclusively on its own conversation data +- Stay fully local (no API calls, no external datasets) +- Accumulate improvements over time without model subscriptions + +This is the sovereign fine-tuning capability the fleet needs to remain independent as external model APIs change pricing or capabilities. + +--- + +## Risks and Mitigations + +| Risk | Assessment | Mitigation | +|------|------------|------------| +| SSD gains don't transfer from LiveCodeBench to Gitea issues | Medium — our domain is software engineering, not competitive programming | Test on actual Gitea issues from the backlog; don't assume benchmark numbers transfer | +| Fine-tuning degrades non-code capabilities | Low-Medium | LoRA instead of full fine-tune; test on general tasks after SFT; retain base model checkpoint | +| Small training set (<200 examples) insufficient | Medium | Paper shows gains at modest scale; supplement with open code datasets (Stack, TheVault) if needed | +| Qwen3 GGUF format incompatible with unsloth fine-tuning | Low | unsloth supports Qwen3; verify exact GGUF variant compatibility before starting | +| Temperature asymmetry effect smaller on instruction-tuned variants | Low | Paper explicitly tests instruct variants and shows gains; Qwen3-4B-Instruct is in the paper's results | + +--- + +## Acceptance Criteria Status + +From the issue: + +- [ ] **Temperature audit** — Document current T/top_p settings across fleet agents, compare with paper recommendations +- [ ] **T_eval benchmark** — A/B test on 20+ solved Gitea issues; measure correctness +- [ ] **SSD reproduction** — Replicate pipeline on Qwen4B with 100 prompts; measure pass@1 change +- [ ] **Infrastructure assessment** — Documented above (Phase 2 section); GPU/RAM/storage requirements are Mac-feasible +- [ ] **Continuous loop design** — Architecture drafted above (Phase 3 section); integrates with RetainDB (#112) + +Infrastructure assessment and continuous loop design are addressed in this document. Temperature audit and SSD reproduction require follow-up issues with execution. + +--- + +## Recommended Follow-Up Issues + +1. **Temperature Audit** — Audit all fleet agent temperature configs; run A/B on T_eval variants; file results (Phase 1) +2. **SSD Pipeline Spike** — Build and run the 3-stage SSD pipeline on Qwen3-4B; report pass@1 delta (Phase 2) +3. **Nightly SFT Integration** — Wire SSD into burn-mode cron; integrate with RetainDB feedback loop (Phase 3) + +--- + +*Research acknowledged by Claude — April 6, 2026* +*Source issue: [hermes-agent #128](https://forge.alexanderwhitestone.com/Timmy_Foundation/hermes-agent/issues/128)* -- 2.43.0