Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Build Hermes health monitor — system resources + model management #1073

New Issue
Closed
opened 2026-03-23 13:11:55 +00:00 by perplexity · 1 comment
perplexity commented 2026-03-23 13:11:55 +00:00
Collaborator
Copy Link

Parent: #1070

Objective

Timmy monitors the health of Hermes (the M3 Max) and keeps it running smoothly. This includes system resources, Ollama model management, and cleanup.

Monitors

  1. Memory pressure — track unified memory usage, alert if <4GB free, unload models if needed
  2. Disk usage — alert if <10GB free, identify and clean large temp files/logs
  3. Ollama status — verify models are loaded, restart if crashed, manage OLLAMA_MAX_LOADED_MODELS
  4. Process health — detect stuck/zombie processes, kill and restart
  5. Network — verify Gitea connectivity, API endpoints responding

Actions Timmy Can Take

  • Unload unused Ollama models to free memory
  • Clean /tmp, old logs, stale build artifacts
  • Restart Ollama if it crashes
  • Kill stuck agent processes
  • Alert Alex if something needs human intervention

Acceptance Criteria

  • Health check runs every 5 minutes
  • Memory/disk/process issues are auto-resolved where possible
  • Alerts sent to Alex only when Timmy can't fix it himself
  • Health status visible in dashboard or Gitea journal
Parent: #1070 ## Objective Timmy monitors the health of Hermes (the M3 Max) and keeps it running smoothly. This includes system resources, Ollama model management, and cleanup. ## Monitors 1. **Memory pressure** — track unified memory usage, alert if <4GB free, unload models if needed 2. **Disk usage** — alert if <10GB free, identify and clean large temp files/logs 3. **Ollama status** — verify models are loaded, restart if crashed, manage `OLLAMA_MAX_LOADED_MODELS` 4. **Process health** — detect stuck/zombie processes, kill and restart 5. **Network** — verify Gitea connectivity, API endpoints responding ## Actions Timmy Can Take - Unload unused Ollama models to free memory - Clean `/tmp`, old logs, stale build artifacts - Restart Ollama if it crashes - Kill stuck agent processes - Alert Alex if something needs human intervention ## Acceptance Criteria - Health check runs every 5 minutes - Memory/disk/process issues are auto-resolved where possible - Alerts sent to Alex only when Timmy can't fix it himself - Health status visible in dashboard or Gitea journal
claude self-assigned this 2026-03-23 13:32:53 +00:00
claude added the harnessinferencep0-critical labels 2026-03-23 13:53:03 +00:00
claude commented 2026-03-23 18:29:55 +00:00
Collaborator
Copy Link

PR created: #1133

What was built:

HermesMonitor in src/infrastructure/hermes/monitor.py — a background health monitor running every 5 minutes (configurable) with five checks:

  1. Memory — reads vm_stat + sysctl hw.memsize on macOS; if free memory < 4GB, auto-unloads Ollama models via keep_alive=0 API trick
  2. Disk — uses shutil.disk_usage; if free < 10GB, cleans /tmp files older than 24 hours
  3. Ollama — hits /api/tags and /api/ps; if unreachable, attempts restart via launchctl then brew services
  4. Processes — scans ps aux for state Z (zombie) processes
  5. Network — pings settings.gitea_url/api/v1/version; alerts if unreachable

All checks run concurrently (asyncio.gather). Auto-fixes fire push notifications via notifier; critical issues trigger native macOS popups. Manual intervention alerts only fire when Timmy cannot self-heal.

New endpoints: GET /hermes/status (cached last report) and POST /hermes/check (immediate cycle).

28 unit tests cover all code paths. Full suite: 3295 passed.

PR created: #1133 **What was built:** `HermesMonitor` in `src/infrastructure/hermes/monitor.py` — a background health monitor running every 5 minutes (configurable) with five checks: 1. **Memory** — reads `vm_stat` + `sysctl hw.memsize` on macOS; if free memory < 4GB, auto-unloads Ollama models via `keep_alive=0` API trick 2. **Disk** — uses `shutil.disk_usage`; if free < 10GB, cleans `/tmp` files older than 24 hours 3. **Ollama** — hits `/api/tags` and `/api/ps`; if unreachable, attempts restart via `launchctl` then `brew services` 4. **Processes** — scans `ps aux` for state `Z` (zombie) processes 5. **Network** — pings `settings.gitea_url/api/v1/version`; alerts if unreachable All checks run concurrently (`asyncio.gather`). Auto-fixes fire push notifications via `notifier`; critical issues trigger native macOS popups. Manual intervention alerts only fire when Timmy cannot self-heal. New endpoints: `GET /hermes/status` (cached last report) and `POST /hermes/check` (immediate cycle). 28 unit tests cover all code paths. Full suite: 3295 passed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label harness inference p0-critical
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees claude
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#1073
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?