bf003cd944
Implements issue #14: 7 metrics that prove knowledge compounding. Metrics: - Knowledge velocity: new facts/day (from index.json) - Knowledge coverage: % domains with 10+ facts (from YAML files) - Hit rate: % sessions referencing bootstrap knowledge - Error recurrence: same errors across sessions (should decrease) - Task completion: % sessions with successful end_reason - First-try success: actions without backtracking (tool/msg ratio) - Knowledge age: staleness of facts (freshness score) Data sources: - knowledge/index.json + YAML files for fact metrics - ~/.hermes/state.db sessions + messages tables Features: - JSON and markdown output formats - --since, --repo, --format flags - 7-day trend tracking via snapshot persistence - Runs in 33ms on 11.9K sessions / 192K messages - Dashboard auto-generation with --save-snapshot Closes #14
83 lines
2.3 KiB
YAML
83 lines
2.3 KiB
YAML
---
|
|
domain: hermes-agent
|
|
category: pitfall
|
|
version: 1
|
|
last_updated: "2026-04-13"
|
|
---
|
|
|
|
# Pitfalls (hermes-agent)
|
|
|
|
Things that go wrong in this repo if you don't know the traps.
|
|
|
|
## Cron & Deployment
|
|
|
|
- id: hermes-agent:pitfall:001
|
|
fact: "deploy-crons.py leaves jobs in mixed model format - some have provider/model, some just model"
|
|
confidence: 0.95
|
|
tags: [cron, deploy, model, config]
|
|
source_count: 5
|
|
first_seen: "2026-04-08"
|
|
last_confirmed: "2026-04-13"
|
|
related: [hermes-agent:pitfall:002, hermes-agent:pitfall:003]
|
|
|
|
- id: hermes-agent:pitfall:002
|
|
fact: "deploy-crons.py --deploy doesn't set legacy skill field from skills list, breaking older jobs"
|
|
confidence: 0.9
|
|
tags: [cron, deploy, skills]
|
|
source_count: 3
|
|
first_seen: "2026-04-09"
|
|
last_confirmed: "2026-04-13"
|
|
related: [hermes-agent:pitfall:001]
|
|
|
|
- id: hermes-agent:pitfall:003
|
|
fact: "Cron jobs with blank fallback_model fields trigger spurious gateway warnings"
|
|
confidence: 0.9
|
|
tags: [cron, model, fallback]
|
|
source_count: 4
|
|
first_seen: "2026-04-07"
|
|
last_confirmed: "2026-04-12"
|
|
related: [hermes-agent:pitfall:001]
|
|
|
|
- id: hermes-agent:pitfall:004
|
|
fact: "model-watchdog.py checks first provider line, not model.provider - causes false drift alarms"
|
|
confidence: 0.9
|
|
tags: [watchdog, model, config]
|
|
source_count: 3
|
|
first_seen: "2026-04-08"
|
|
last_confirmed: "2026-04-13"
|
|
|
|
## Path & Environment
|
|
|
|
- id: hermes-agent:pitfall:005
|
|
fact: "10+ files read HERMES_HOME directly instead of get_hermes_home() - breaks on custom paths"
|
|
confidence: 0.85
|
|
tags: [paths, env, hermes-home]
|
|
source_count: 6
|
|
first_seen: "2026-04-06"
|
|
last_confirmed: "2026-04-12"
|
|
related: [global:pitfall:002]
|
|
|
|
- id: hermes-agent:pitfall:006
|
|
fact: "get_hermes_home() doesn't expand tilde when HERMES_HOME=~/... is set"
|
|
confidence: 0.8
|
|
tags: [paths, env, bug]
|
|
source_count: 2
|
|
first_seen: "2026-04-05"
|
|
|
|
## SSH & Dispatch
|
|
|
|
- id: hermes-agent:pitfall:007
|
|
fact: "vps-agent-dispatch reports OK while remote hermes binary path is broken"
|
|
confidence: 0.9
|
|
tags: [ssh, dispatch, vps]
|
|
source_count: 4
|
|
first_seen: "2026-04-07"
|
|
last_confirmed: "2026-04-11"
|
|
|
|
- id: hermes-agent:pitfall:008
|
|
fact: "nightwatch-health-monitor SSH check fails on cloud-model-only deployments"
|
|
confidence: 0.85
|
|
tags: [ssh, health, cloud]
|
|
source_count: 2
|
|
first_seen: "2026-04-10"
|