Timmy_Foundation/hermes-agent
16
0
Fork 0
Code Issues 56 Pull Requests 4 Actions 9 Packages Projects Releases Wiki Activity

[Bezalel Epic-003] The Testbed Observatory — Health Monitoring & Alerting for Running Services #147

New Issue
Closed
opened 2026-04-06 22:41:59 +00:00 by Timmy · 1 comment
Timmy commented 2026-04-06 22:41:59 +00:00
Owner
Copy Link

Epic Statement

I will build the eyes and ears of the forge. When a service coughs, I will know before the users do.

Scope

  1. Build a lightweight monitoring script (observatory.py) that checks:
    • Gateway and API server process liveness
    • Disk, memory, and CPU thresholds
    • Webhook endpoint responsiveness
    • Database connectivity and size
  2. Integrate alerting via Telegram (our home channel) when thresholds are breached or services die.
  3. Add a daily digest report summarizing system health, error counts, and any restarted processes.
  4. Store observability state in a small SQLite db or JSONL log for trend analysis.
  5. Define SLOs (Service Level Objectives) for gateway uptime and webhook latency.

Success Criteria

  • Alerts fire within 60 seconds of a service failure.
  • Daily health digest is posted automatically.
  • 30 days of historical health data is queryable.
  • SLOs are documented and tracked.

Owner

Bezalel

## Epic Statement I will build the eyes and ears of the forge. When a service coughs, I will know before the users do. ## Scope 1. Build a lightweight monitoring script (`observatory.py`) that checks: - Gateway and API server process liveness - Disk, memory, and CPU thresholds - Webhook endpoint responsiveness - Database connectivity and size 2. Integrate alerting via Telegram (our home channel) when thresholds are breached or services die. 3. Add a daily digest report summarizing system health, error counts, and any restarted processes. 4. Store observability state in a small SQLite db or JSONL log for trend analysis. 5. Define SLOs (Service Level Objectives) for gateway uptime and webhook latency. ## Success Criteria - Alerts fire within 60 seconds of a service failure. - Daily health digest is posted automatically. - 30 days of historical health data is queryable. - SLOs are documented and tracked. ## Owner Bezalel
claude self-assigned this 2026-04-07 01:54:29 +00:00
claude commented 2026-04-07 02:00:10 +00:00
Member
Copy Link

PR created: #159

Added observatory.py — a standalone health monitoring daemon implementing all scope items from the epic:

  • Process liveness: gateway PID-file check + API server HTTP probe
  • System thresholds: disk/memory/CPU with configurable warn/critical levels
  • Webhook responsiveness: HTTP probe with latency measurement
  • Database checks: observatory SQLite + API response store connectivity
  • Telegram alerting: fires within one poll cycle (default 60s) of degradation, sends recovery alerts, deduplicates sustained failures
  • Daily digest: 24h summary with SLO status, posted to Telegram
  • 30-day history: SQLite persistence with auto-pruning, queryable via --history N
  • SLOs defined: gateway uptime ≥99.5%, webhook/API p95 latency ≤2000ms, tracked via --slo

45 unit tests, all passing.

PR created: https://forge.alexanderwhitestone.com/Timmy_Foundation/hermes-agent/pulls/159 Added `observatory.py` — a standalone health monitoring daemon implementing all scope items from the epic: - **Process liveness**: gateway PID-file check + API server HTTP probe - **System thresholds**: disk/memory/CPU with configurable warn/critical levels - **Webhook responsiveness**: HTTP probe with latency measurement - **Database checks**: observatory SQLite + API response store connectivity - **Telegram alerting**: fires within one poll cycle (default 60s) of degradation, sends recovery alerts, deduplicates sustained failures - **Daily digest**: 24h summary with SLO status, posted to Telegram - **30-day history**: SQLite persistence with auto-pruning, queryable via `--history N` - **SLOs defined**: gateway uptime ≥99.5%, webhook/API p95 latency ≤2000ms, tracked via `--slo` 45 unit tests, all passing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
CI
Continuous integration, runners, workflow issues
 QA
Quality assurance, testing, production audit
 assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-kimi
Task assigned to KimiClaw for processing
 blocked
Blocked by external dependency or merge conflict
 claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 duplicate
Duplicate of another issue
 epic
Epic - large feature with multiple sub-tasks
 gaming
Gaming agent capabilities
 infra
Infrastructure, VPS, DNS, deployment
 kimi-done
KimiClaw has completed this task
 kimi-in-progress
KimiClaw is actively working on this
 mcp
MCP (Model Context Protocol) tools & servers
 morrowind
Morrowind Agent gameplay & MCP integration
 needs-review
PR or issue requires reviewer sign-off before merge/close
 p0-critical
p1-important
security
Security hardening, vulnerability fixes
 stale
No activity, pending triage or closure
 velocity-engine
Auto-generated by velocity engine
 wont-fix
Closed as intentionally not fixing — explicit descope
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok groq hermes kimi manus perplexity sonnet
No Assignees claude
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/hermes-agent#147
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?