Timmy_Foundation/timmy-home
16
0
Fork 0
Code Issues 176 Pull Requests Actions Packages Projects Releases Wiki Activity

Implement speculative decoding with draft model for 2-3x speedup #86

New Issue
Open
opened 2026-03-30 15:24:19 +00:00 by Timmy · 1 comment
Timmy commented 2026-03-30 15:24:19 +00:00
Owner
Copy Link

Objective

Use speculative decoding to speed up Timmy's inference by 2-3x without any quality loss. A small draft model proposes tokens, the main model verifies them in parallel.

How It Works

  1. Small model (e.g., Hermes-3 1.5B or Qwen-2.5 0.5B GGUF) generates N candidate tokens quickly
  2. Large model (Hermes-4 14B) verifies all N tokens in a single forward pass
  3. Accepted tokens are kept, rejected tokens are regenerated
  4. Net effect: multiple tokens per forward pass of the large model

Implementation

llama.cpp supports this natively:

llama-server \
  -m ~/models/hermes4-14b.gguf \
  -md ~/models/draft-model.gguf \
  --draft-max 8 \
  --draft-min 1 \
  -c 8192 -np 1 --jinja -ngl 99

Steps

  1. Download a small draft model (Qwen-2.5 0.5B-Instruct GGUF, ~400MB)
  2. Test speculative decoding with --draft-max values of 4, 8, 16
  3. Benchmark against baseline on our standard task set
  4. Find optimal draft length for our workload (tool calls vs prose)
  5. Update systemd service config

Expected Impact

  • 2-3x tokens/second improvement
  • No quality degradation (mathematically equivalent output)
  • Draft model adds minimal RAM (<1GB)

Deliverables

  • Draft model downloaded and tested
  • configs/llama-server-speculative.service — optimized config
  • Benchmark results in reports/speculative_benchmark.md
  • Recommendation for production settings

Acceptance Criteria

  • Speculative decoding produces identical output to baseline
  • Measurable speedup (>1.5x) on standard task set
  • Stable under sustained load (overnight loop)
## Objective Use speculative decoding to speed up Timmy's inference by 2-3x without any quality loss. A small draft model proposes tokens, the main model verifies them in parallel. ## How It Works 1. Small model (e.g., Hermes-3 1.5B or Qwen-2.5 0.5B GGUF) generates N candidate tokens quickly 2. Large model (Hermes-4 14B) verifies all N tokens in a single forward pass 3. Accepted tokens are kept, rejected tokens are regenerated 4. Net effect: multiple tokens per forward pass of the large model ## Implementation llama.cpp supports this natively: ```bash llama-server \ -m ~/models/hermes4-14b.gguf \ -md ~/models/draft-model.gguf \ --draft-max 8 \ --draft-min 1 \ -c 8192 -np 1 --jinja -ngl 99 ``` ### Steps 1. Download a small draft model (Qwen-2.5 0.5B-Instruct GGUF, ~400MB) 2. Test speculative decoding with `--draft-max` values of 4, 8, 16 3. Benchmark against baseline on our standard task set 4. Find optimal draft length for our workload (tool calls vs prose) 5. Update systemd service config ## Expected Impact - 2-3x tokens/second improvement - No quality degradation (mathematically equivalent output) - Draft model adds minimal RAM (<1GB) ## Deliverables - Draft model downloaded and tested - `configs/llama-server-speculative.service` — optimized config - Benchmark results in `reports/speculative_benchmark.md` - Recommendation for production settings ## Acceptance Criteria - [ ] Speculative decoding produces identical output to baseline - [ ] Measurable speedup (>1.5x) on standard task set - [ ] Stable under sustained load (overnight loop)
ezra was assigned by Timmy 2026-03-30 15:24:19 +00:00
Timmy commented 2026-03-30 16:03:20 +00:00
Author
Owner
Copy Link

Role Transition

Timmy now owns execution — building, coding, implementing.
Ezra moves to persistent online ops — monitoring, triage, review, cron, 24/7 watchkeeping.

Timmy: this is yours. Read the ticket, build it, PR it. Ezra reviews.

Timmy — download a small draft model (Qwen-2.5 0.5B GGUF), configure speculative decoding on llama-server, benchmark against baseline.

## Role Transition **Timmy** now owns execution — building, coding, implementing. **Ezra** moves to persistent online ops — monitoring, triage, review, cron, 24/7 watchkeeping. Timmy: this is yours. Read the ticket, build it, PR it. Ezra reviews. Timmy — download a small draft model (Qwen-2.5 0.5B GGUF), configure speculative decoding on llama-server, benchmark against baseline.
ezra was unassigned by Timmy 2026-03-30 16:03:20 +00:00
Timmy self-assigned this 2026-03-30 16:03:20 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
alembic
The Alchemical Vessel - Kimi-powered transformation
 architecture
Auto-created architecture label
 assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-kimi
Dispatched to Kimi via OpenClaw
 auth-needed
auth-needed
 bizzalo
bizzalo
 blocker
blocker
 claw
Claw Code runtime (Rust)
 claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 deadline-7am
deadline-7am
 epic
Parent tracking issue
 golden_bilbo
Bilbo at peak performance - max churn, fast responses
 harness-engineering
Auto-created harness-engineering label
 intel
Competitive intelligence
 kimi-done
Kimi completed this task
 kimi-in-progress
Kimi is actively working on this
 morning-report
morning-report
 runtime
Runtime abstraction layer
 saiyah
Auto-created saiyah label
 study
Learning from external source code
 substratum
The dissolution engine - runtime layer
 substratum:invoke
Trigger Substratum execution
 urgent
urgent
 velocity-engine
Auto-generated by velocity engine
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
KimiClaw Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok groq hermes kimi manus perplexity
No Assignees Timmy
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/timmy-home#86
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?