Alexander Whitestone 9fa0a59761 feat: add Observatory health monitoring & alerting for running services ...

Implements Bezalel Epic-003 — a lightweight monitoring script that:

- Checks gateway and API server process liveness
- Monitors disk, memory, and CPU thresholds (warn/critical levels)
- Probes webhook and API server HTTP endpoints for responsiveness
- Verifies SQLite database connectivity and size
- Sends Telegram alerts when checks degrade or recover (within 60s)
- Posts daily digest reports summarising 24h health, SLO status, error counts
- Persists 30 days of health snapshots in SQLite (~/.hermes/observatory.db)
- Tracks alerts_sent for trend analysis
- Defines and tracks SLOs: gateway uptime ≥99.5%, webhook p95 latency ≤2s

Usage:
  python observatory.py --check        # one-shot check
  python observatory.py --daemon       # continuous 60s poll
  python observatory.py --digest       # print daily digest
  python observatory.py --send-digest  # send digest via Telegram
  python observatory.py --slo          # print SLO report
  python observatory.py --history 20   # show last 20 records

Config via env: OBSERVATORY_ALERT_CHAT_ID, TELEGRAM_BOT_TOKEN, etc.
Adds psutil optional dependency group in pyproject.toml.
45 unit tests covering checks, persistence, alerting, digest, SLOs, and CLI.

Fixes #147

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-04-06 21:59:14 -04:00

..

acp

fix(acp): complete session management surface for editor clients (salvage #3501) (#3675)

2026-03-28 23:45:53 -07:00

agent

security: Fix V-006 MCP OAuth Deserialization (CVSS 8.8 CRITICAL)

2026-03-31 00:37:14 +00:00

cron

fix(cron): resolve human-friendly delivery labels via channel directory (#3860)

2026-03-29 21:24:17 -07:00

fakes

…

gateway

security: fix command injection vulnerabilities (CVSS 9.8)

2026-03-30 23:15:11 +00:00

hermes_cli

fix: use argparse entrypoint in top-level launcher (#3874)

2026-03-29 21:54:36 -07:00

honcho_integration

feat(honcho): instance-local config via HERMES_HOME, default session strategy to per-directory

2026-03-21 09:34:00 -07:00

integration

refactor: remove mini-swe-agent dependency — inline Docker/Modal backends (#2804)

2026-03-24 07:30:25 -07:00

security

security: Add crisis safety test infrastructure for Issue #79

2026-04-01 03:49:48 +00:00

skills

feat(skills): add memento-flashcards optional skill (#3827)

2026-03-29 16:52:52 -07:00

tools

feat: Phase 31 Nexus Architect scaffold — autonomous 3D world generation

2026-03-31 21:06:42 +00:00

__init__.py

…

conftest.py

fix(approval): show full command in dangerous command approval (#1553)

2026-03-17 02:02:33 -07:00

run_interrupt_test.py

fix: thread safety for concurrent subagent delegation (#1672)

2026-03-17 02:53:33 -07:00

test_413_compression.py

…

test_860_dedup.py

…

test_1630_context_overflow_loop.py

fix: prevent infinite 400 loop on context overflow + block prompt injection via cache files (#1630, #1558)

2026-03-17 01:50:59 -07:00

test_agent_guardrails.py

feat: pre-call sanitization and post-call tool guardrails (#1732)

2026-03-17 04:24:27 -07:00

test_agent_loop_tool_calling.py

…

test_agent_loop_vllm.py

…

test_agent_loop.py

…

test_anthropic_adapter.py

fix(anthropic): use model-native output limits instead of hardcoded 16K (#3426)

2026-03-27 13:02:52 -07:00

test_anthropic_error_handling.py

fix(ci): pin acp <0.9 and update retry-exhaust test (#3320)

2026-03-26 19:21:34 -07:00

test_anthropic_oauth_flow.py

…

test_anthropic_provider_persistence.py

…

test_api_key_providers.py

fix(auth): stop silently falling back to OpenRouter when no provider is configured (#3862)

2026-03-29 21:06:35 -07:00

test_async_httpx_del_neuter.py

fix: eliminate 'Event loop is closed' / 'Press ENTER to continue' during idle (#3398)

2026-03-27 09:45:25 -07:00

test_atomic_json_write.py

…

test_atomic_yaml_write.py

…

test_auth_codex_provider.py

…

test_auth_nous_provider.py

…

test_auxiliary_config_bridge.py

feat(compression): add summary_base_url + move compression config to YAML-only

2026-03-17 04:46:15 -07:00

test_batch_runner_checkpoint.py

…

test_cli_approval_ui.py

…

test_cli_background_tui_refresh.py

fix(cli): refresh TUI before background task output to prevent status bar overlap (#3048)

2026-03-25 15:00:33 -07:00

test_cli_extension_hooks.py

refactor(cli): add protected TUI extension hooks for wrapper CLIs

2026-03-21 09:42:07 -07:00

test_cli_init.py

feat(cli): configurable busy input mode + fix /queue always working (#3298)

2026-03-26 17:58:40 -07:00

test_cli_interrupt_subagent.py

fix: thread safety for concurrent subagent delegation (#1672)

2026-03-17 02:53:33 -07:00

test_cli_loading_indicator.py

…

test_cli_mcp_config_watch.py

…

test_cli_new_session.py

fix: complete session reset — missing compressor counters + test

2026-03-20 04:35:17 -07:00

test_cli_plan_command.py

…

test_cli_prefix_matching.py

feat: add /tools disable/enable/list slash commands with session reset (#1652)

2026-03-17 02:05:26 -07:00

test_cli_preloaded_skills.py

fix: move activated skills line below welcome text

2026-03-23 06:20:19 -07:00

test_cli_provider_resolution.py

feat: overhaul context length detection with models.dev and provider-aware resolution (#2158)

2026-03-20 06:04:33 -07:00

test_cli_retry.py

…

test_cli_secret_capture.py

…

test_cli_skin_integration.py

…

test_cli_status_bar.py

fix(cli): prevent status bar wrapping into duplicate rows (#3883)

2026-03-29 23:59:07 -07:00

test_cli_tools_command.py

fix: resolve 7 failing CI tests (#3936)

2026-03-30 08:10:14 -07:00

test_codex_execution_paths.py

…

test_codex_models.py

fix: add gpt-5.4-mini to Codex fallback catalog (#3855)

2026-03-29 20:10:00 -07:00

test_compression_boundary.py

fix(agent): prevent silent tool result loss during context compression (#1993)

2026-03-18 15:22:51 -07:00

test_compressor_fallback_update.py

feat(providers): add ordered fallback provider chain (salvage #1761) (#3813)

2026-03-29 16:04:53 -07:00

test_config_env_expansion.py

feat(config): support ${ENV_VAR} substitution in config.yaml (#2684)

2026-03-23 16:02:06 -07:00

test_context_pressure.py

fix: cap context pressure percentage at 100% in display (#3480)

2026-03-27 21:42:09 -07:00

test_context_references.py

fix(context): restrict @ references to safe workspace paths (#2601)

2026-03-23 06:40:05 -07:00

test_context_token_tracking.py

fix(tests): resolve all consistently failing tests

2026-03-22 05:58:26 -07:00

test_crossloop_client_cache.py

fix(agent): prevent AsyncOpenAI/httpx cross-loop deadlock in gateway mode (#2701)

2026-03-25 17:31:56 -07:00

test_dict_tool_call_args.py

…

test_display.py

…

test_evidence_store.py

…

test_exit_cleanup_interrupt.py

fix: catch KeyboardInterrupt in exit cleanup handlers (#3257)

2026-03-26 14:34:31 -07:00

test_external_credential_detection.py

…

test_fallback_model.py

feat: upgrade MiniMax default to M2.7 + add new OpenRouter models

2026-03-18 02:42:58 -07:00

test_fallback_router.py

test: add fallback chain integration tests

2026-03-31 19:46:23 +00:00

test_file_permissions.py

…

test_flush_memories_codex.py

…

test_hermes_state.py

feat(sessions): add --source flag for third-party session isolation (#3255)

2026-03-26 14:35:31 -07:00

test_honcho_client_config.py

…

test_input_sanitizer_integration.py

security: add input sanitization for jailbreak patterns (Issue #72)

2026-03-31 19:56:16 +00:00

test_input_sanitizer.py

security(input_sanitizer): expand jailbreak pattern coverage (#87)

2026-04-05 15:48:10 +00:00

test_insights.py

feat: add route-aware pricing estimates (#1695)

2026-03-17 03:44:44 -07:00

test_interactive_interrupt.py

fix: thread safety for concurrent subagent delegation (#1672)

2026-03-17 02:53:33 -07:00

test_interrupt_propagation.py

fix: thread safety for concurrent subagent delegation (#1672)

2026-03-17 02:53:33 -07:00

test_managed_server_tool_support.py

…

test_mcp_serve.py

feat: add MCP server mode — hermes mcp serve (#3795)

2026-03-29 15:47:19 -07:00

test_minisweagent_path.py

chore: remove all remaining mini-swe-agent references

2026-03-24 08:19:23 -07:00

test_model_metadata_local_ctx.py

fix: prefer loaded instance context size over max for LM Studio

2026-03-19 21:24:53 +01:00

test_model_provider_persistence.py

feat: integrate GitHub Copilot providers across Hermes

2026-03-17 23:40:22 -07:00

test_model_tools_async_bridge.py

fix: use per-thread persistent event loops in worker threads

2026-03-20 15:41:06 -04:00

test_model_tools.py

…

test_nexus_architect.py

feat: Issue #42 - Nexus Architect for autonomous Three.js world building

2026-04-01 02:45:36 +00:00

test_oauth_state_security.py

security: Fix V-006 MCP OAuth Deserialization (CVSS 8.8 CRITICAL)

2026-03-31 00:37:14 +00:00

test_observatory.py

feat: add Observatory health monitoring & alerting for running services

2026-04-06 21:59:14 -04:00

test_openai_client_lifecycle.py

fix: audit fixes — 5 bugs found and resolved

2026-03-16 06:35:46 -07:00

test_percentage_clamp.py

fix: cap percentage displays at 100% in stats, gateway, and memory tool (#3599)

2026-03-28 14:55:18 -07:00

test_personality_none.py

…

test_plugins_cmd.py

fix(tests): resolve 10 CI failures across hooks, tiktoken, plugins (#3848)

2026-03-29 20:05:59 -07:00

test_plugins.py

feat: activate plugin lifecycle hooks (pre/post_llm_call, session start/end) (#3542)

2026-03-28 11:14:54 -07:00

test_pre_commit_secret_leak.py

security(pre-commit): add secret leak scanner for prompts and credentials (#384)

2026-04-05 16:15:26 +00:00

test_project_metadata.py

fix(setup): auto-install matrix-nio during hermes setup (#3873)

2026-03-29 21:53:28 -07:00

test_provider_fallback.py

feat(providers): add ordered fallback provider chain (salvage #1761) (#3813)

2026-03-29 16:04:53 -07:00

test_provider_parity.py

fix: align Nous Portal model slugs with OpenRouter naming (#3253)

2026-03-26 13:49:43 -07:00

test_quick_commands.py

fix: thread safety for concurrent subagent delegation (#1672)

2026-03-17 02:53:33 -07:00

test_real_interrupt_subagent.py

fix: thread safety for concurrent subagent delegation (#1672)

2026-03-17 02:53:33 -07:00

test_reasoning_command.py

fix: prevent reasoning box from rendering 3x during tool-calling loops (#3405)

2026-03-27 09:57:50 -07:00

test_redirect_stdout_issue.py

…

test_resume_display.py

…

test_run_agent_codex_responses.py

fix(codex): handle reasoning-only responses and replay path (#2070)

2026-03-19 10:34:44 -07:00

test_run_agent.py

feat(providers): add ordered fallback provider chain (salvage #1761) (#3813)

2026-03-29 16:04:53 -07:00

test_runtime_provider_resolution.py

fix(provider): remove MiniMax /v1→/anthropic auto-correction to allow user override (#3553)

2026-03-28 11:36:59 -07:00

test_session_reset_fix.py

fix(session): clear compressor summary and turn counter on /clear and /new (#3102)

2026-03-25 18:22:21 -07:00

test_setup_model_selection.py

…

test_shield_integration.py

security: integrate SHIELD jailbreak/crisis detection

2026-03-31 16:35:40 +00:00

test_sql_injection.py

fix(security): eliminate SQL string formatting in execute() calls

2026-03-19 15:16:35 +01:00

test_streaming.py

fix(test): update streaming test to match PR #3566 behavior change (#3574)

2026-03-28 13:41:23 -07:00

test_surrogate_sanitization.py

fix: sanitize surrogate characters from clipboard paste to prevent UnicodeEncodeError (#3624)

2026-03-28 16:53:14 -07:00

test_temporal_kg.py

feat: Issue #39 - temporal knowledge graph with versioning and reasoning

2026-04-01 02:08:20 +00:00

test_timezone.py

fix: skip stale cron jobs on gateway restart instead of firing immediately

2026-03-16 23:48:14 -07:00

test_tool_call_parsers.py

fix(mistral-parser): handle nested JSON in fallback extraction

2026-03-21 09:41:17 -07:00

test_toolset_distributions.py

…

test_toolsets.py

…

test_trajectory_compressor.py

…

test_worktree_security.py

…

test_worktree.py

…