Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

[Infra] Local inference fallback — degraded but never dark #914

New Issue
Closed
opened 2026-03-22 13:24:01 +00:00 by Timmy · 0 comments
Timmy commented 2026-03-22 13:24:01 +00:00
Owner
Copy Link

Problem

Even with Hermes running on the VPS, Timmy depends on external API calls (Anthropic) to think. If the API key expires, quota runs out, or Anthropic has an outage, Timmy goes dark again.

Solution

Run a small local model on the VPS as a fallback. When the primary model (Claude) is unreachable, Timmy drops to a smaller local model. Degraded intelligence, but never silent.

Options

Model RAM Quality Notes
Qwen 2.5 1.5B (GGUF Q4) ~2GB Basic Fits on VPS, fast
Phi-3 Mini 3.8B (GGUF Q4) ~3GB Decent May be tight on VPS RAM
TinyLlama 1.1B ~1GB Minimal Very fast, limited reasoning

Behavior

  • Primary: Claude via Anthropic API (full capability)
  • Fallback: Local model via llama.cpp (limited but present)
  • Timmy should tell the user when running in fallback mode: "I'm running on a smaller brain right now — Alexander's API connection is down. I can still chat but I'm not as sharp."
  • Auto-switch back when primary is available

Requirements

  • Install llama-cpp-python on VPS
  • Download a small GGUF model
  • Add fallback routing in Hermes API server config
  • Add status indicator in Workshop panel (which model is active)
  • Test failover and failback

Priority

Medium — nice to have after VPS migration is done. The VPS + Anthropic API is already much more reliable than the Mac dependency. This is the belt-and-suspenders layer.

References

  • Memory note: Timmy runs on Hermes harness, SOUL.md says "If I have four [GB], I think with a smaller one. I never refuse to work because my resources are limited."
## Problem Even with Hermes running on the VPS, Timmy depends on external API calls (Anthropic) to think. If the API key expires, quota runs out, or Anthropic has an outage, Timmy goes dark again. ## Solution Run a small local model on the VPS as a fallback. When the primary model (Claude) is unreachable, Timmy drops to a smaller local model. Degraded intelligence, but never silent. ## Options | Model | RAM | Quality | Notes | |-------|-----|---------|-------| | Qwen 2.5 1.5B (GGUF Q4) | ~2GB | Basic | Fits on VPS, fast | | Phi-3 Mini 3.8B (GGUF Q4) | ~3GB | Decent | May be tight on VPS RAM | | TinyLlama 1.1B | ~1GB | Minimal | Very fast, limited reasoning | ## Behavior - Primary: Claude via Anthropic API (full capability) - Fallback: Local model via llama.cpp (limited but present) - Timmy should **tell the user** when running in fallback mode: "I'm running on a smaller brain right now — Alexander's API connection is down. I can still chat but I'm not as sharp." - Auto-switch back when primary is available ## Requirements - [ ] Install llama-cpp-python on VPS - [ ] Download a small GGUF model - [ ] Add fallback routing in Hermes API server config - [ ] Add status indicator in Workshop panel (which model is active) - [ ] Test failover and failback ## Priority **Medium** — nice to have after VPS migration is done. The VPS + Anthropic API is already much more reliable than the Mac dependency. This is the belt-and-suspenders layer. ## References - Memory note: Timmy runs on Hermes harness, SOUL.md says "If I have four [GB], I think with a smaller one. I never refuse to work because my resources are limited."
gemini was assigned by Rockachopa 2026-03-22 23:31:57 +00:00
claude added the harnessinferencep0-critical labels 2026-03-23 13:52:56 +00:00
gemini was unassigned by Timmy 2026-03-24 19:33:52 +00:00
Timmy closed this issue 2026-03-24 21:54:47 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label harness inference p0-critical
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#914
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?