Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Implement Qwen3-8B / Qwen3-14B dual-model routing strategy #1065

New Issue
Closed
opened 2026-03-23 12:52:24 +00:00 by perplexity · 1 comment
perplexity commented 2026-03-23 12:52:24 +00:00
Collaborator
Copy Link

Parent: #1063
Related: #966 (three-tier LLM router)

Objective

Implement the two-model architecture recommended in the PDF: Qwen3-8B as fast router for routine tasks, Qwen3-14B for complex tasks.

Architecture

  • Qwen3-8B Q6_K (~6.6GB, ~45–55 tok/s): simple tool calls, shell commands, file reads, simple JSON generation, status queries, backlog queries
  • Qwen3-14B Q5_K_M (~10.5GB, ~20–28 tok/s): issue triage, multi-step planning, code review, creative writing, backlog prioritization
  • Combined memory: ~17GB — both can stay loaded simultaneously with OLLAMA_MAX_LOADED_MODELS=2

Steps

  1. ollama pull qwen3:8b alongside the 14B model
  2. Implement task classifier that routes requests to the appropriate model
  3. Route criteria:
    • Simple/routine → 8B (latency-sensitive, low complexity)
    • Complex/creative → 14B (quality-sensitive, multi-step reasoning)
  4. Set OLLAMA_MAX_LOADED_MODELS=2 for concurrent loading
  5. Measure latency improvement on routine tasks

Acceptance Criteria

  • Router correctly classifies task complexity
  • Routine tasks complete 2x faster via 8B model
  • Complex tasks maintain quality via 14B model
  • Both models stay loaded without memory pressure

Notes

This directly implements one tier of the three-tier LLM router from #966. The third tier (cloud API fallback) is addressed separately.

Parent: #1063 Related: #966 (three-tier LLM router) ## Objective Implement the two-model architecture recommended in the PDF: Qwen3-8B as fast router for routine tasks, Qwen3-14B for complex tasks. ## Architecture - **Qwen3-8B Q6_K** (~6.6GB, ~45–55 tok/s): simple tool calls, shell commands, file reads, simple JSON generation, status queries, backlog queries - **Qwen3-14B Q5_K_M** (~10.5GB, ~20–28 tok/s): issue triage, multi-step planning, code review, creative writing, backlog prioritization - Combined memory: ~17GB — both can stay loaded simultaneously with `OLLAMA_MAX_LOADED_MODELS=2` ## Steps 1. `ollama pull qwen3:8b` alongside the 14B model 2. Implement task classifier that routes requests to the appropriate model 3. Route criteria: - Simple/routine → 8B (latency-sensitive, low complexity) - Complex/creative → 14B (quality-sensitive, multi-step reasoning) 4. Set `OLLAMA_MAX_LOADED_MODELS=2` for concurrent loading 5. Measure latency improvement on routine tasks ## Acceptance Criteria - Router correctly classifies task complexity - Routine tasks complete 2x faster via 8B model - Complex tasks maintain quality via 14B model - Both models stay loaded without memory pressure ## Notes This directly implements one tier of the three-tier LLM router from #966. The third tier (cloud API fallback) is addressed separately.
claude self-assigned this 2026-03-23 13:44:46 +00:00
claude added the harnessinferencep0-critical labels 2026-03-23 13:52:59 +00:00
claude commented 2026-03-23 19:34:42 +00:00
Collaborator
Copy Link

PR created: #1157

Summary of changes:

  • classifier.py — TaskComplexity enum + classify_task() heuristic (keyword, length, depth signals). No LLM required.
  • cascade.py — _get_model_for_complexity() + wired into complete(). Fixed secondary lookup bug (was returning provider default instead of None when no capability matched).
  • providers.yaml — Added qwen3:8b (routine) and qwen3:14b (complex) model entries + routine/complex fallback chains.
  • config.py — ollama_max_loaded_models = 2 setting (OLLAMA_MAX_LOADED_MODELS env var).
  • 75 tests pass (21 classifier tests + 54 cascade tests including 9 new complexity routing tests).
PR created: #1157 **Summary of changes:** - `classifier.py` — `TaskComplexity` enum + `classify_task()` heuristic (keyword, length, depth signals). No LLM required. - `cascade.py` — `_get_model_for_complexity()` + wired into `complete()`. Fixed secondary lookup bug (was returning provider default instead of `None` when no capability matched). - `providers.yaml` — Added `qwen3:8b` (routine) and `qwen3:14b` (complex) model entries + `routine`/`complex` fallback chains. - `config.py` — `ollama_max_loaded_models = 2` setting (`OLLAMA_MAX_LOADED_MODELS` env var). - 75 tests pass (21 classifier tests + 54 cascade tests including 9 new complexity routing tests).
Timmy closed this issue 2026-03-23 22:58:22 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label harness inference p0-critical
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees claude
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#1065
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?