Timmy_Foundation/hermes-agent
15
0
Fork 0
Code Issues 282 Pull Requests 87 Actions Packages Projects Releases Wiki Activity

enhancement: log tool hallucination statistics to metrics #1109

Closed
Rockachopa wants to merge 187 commits from step35/853-enhancement-log-tool-halluci into main
pull from: step35/853-enhancement-log-tool-halluci
merge into: Timmy_Foundation:main
Conversation 0 Commits 187 Files Changed 155 +24377 -117
Rockachopa commented 2026-04-29 04:22:54 +00:00
Owner
Copy Link

Summary

Implements persistent logging of tool hallucination statistics detected by the poka-yoke validator.

Key Changes:

  • Added metrics persistence layer writing to ~/.hermes/metrics/hallucination_stats.json
  • Track cumulative failures per tool (persists across restarts)
  • Track first/last event timestamps and total event counts
  • New get_hallucination_metrics() function for aggregated statistics
  • New hermes hallucination CLI command to view metrics
  • Enhanced reset_circuit_breaker() to also clear cumulative state and persisted file

Files:

  • tools/tool_pokayoke.py — Core tracking logic (+ cumulative tracking, persistence hooks)
  • hermes_cli/hallucination.py — New CLI command (NEW file)
  • hermes_cli/main.py — Command registration

CLI Usage:

hermes hallucination              # Show summary with top tools
hermes hallucination --json       # Raw JSON output
hermes hallucination --top 20     # Show top 20 tools
hermes hallucination --reset      # Reset all counters (with confirmation)

Metrics file format (~/.hermes/metrics/hallucination_stats.json):

{
  "version": "1.0",
  "cumulative_failures": {"tool_name": count, ...},
  "tools_affected": ["tool1", "tool2", ...],
  "first_event": 1698765432.0,
  "last_event": 1698766000.0,
  "total_events": 42
}

Acceptance Criteria (from #853):

  • Hallucination events logged to structured file
  • Statistics aggregated by tool name
  • CLI command to view metrics (hermes hallucination)

Closes #853

## Summary Implements persistent logging of tool hallucination statistics detected by the poka-yoke validator. **Key Changes:** - Added metrics persistence layer writing to `~/.hermes/metrics/hallucination_stats.json` - Track cumulative failures per tool (persists across restarts) - Track first/last event timestamps and total event counts - New `get_hallucination_metrics()` function for aggregated statistics - New `hermes hallucination` CLI command to view metrics - Enhanced `reset_circuit_breaker()` to also clear cumulative state and persisted file **Files:** - `tools/tool_pokayoke.py` — Core tracking logic (+ cumulative tracking, persistence hooks) - `hermes_cli/hallucination.py` — New CLI command (NEW file) - `hermes_cli/main.py` — Command registration **CLI Usage:** ```bash hermes hallucination # Show summary with top tools hermes hallucination --json # Raw JSON output hermes hallucination --top 20 # Show top 20 tools hermes hallucination --reset # Reset all counters (with confirmation) ``` **Metrics file format** (`~/.hermes/metrics/hallucination_stats.json`): ```json { "version": "1.0", "cumulative_failures": {"tool_name": count, ...}, "tools_affected": ["tool1", "tool2", ...], "first_event": 1698765432.0, "last_event": 1698766000.0, "total_events": 42 } ``` **Acceptance Criteria** (from #853): - [x] Hallucination events logged to structured file - [x] Statistics aggregated by tool name - [x] CLI command to view metrics (`hermes hallucination`) Closes #853
Rockachopa added 1 commit 2026-04-29 04:22:55 +00:00
enhancement: log tool hallucination statistics to metrics
All checks were successful
Lint / lint (pull_request) Successful in 10s
Details
52865209ac 
- Add persistent metrics storage (~/.hermes/metrics/hallucination_stats.json)
- Track cumulative failures per tool (survives restarts)
- Track first/last event timestamps and total event count
- Add `get_hallucination_metrics()` API for aggregated stats
- Add `hermes hallucination` CLI command to view metrics
- Enhance `reset_circuit_breaker()` to also clear cumulative state and file
- Increment cumulative counter on each failure in ToolCallValidator._record_failure

Resolves: #853

Acceptance criteria:
- [x] Hallucination events logged to structured file (JSON at ~/.hermes/metrics/hallucination_stats.json)
- [x] Statistics aggregated by tool name (cumulative_failures dict + by_tool breakdown)
- [x] Dashboard integration or CLI command to view (new `hermes hallucination` command)

The implementation is minimal: extends ToolCallValidator with in-memory cumulative tracking
and hooks _record_failure to persist on each failure. CLI command reads both in-memory
(current session) and persisted (historical) data for full picture.
Rockachopa closed this pull request 2026-04-29 11:44:09 +00:00
Rockachopa deleted branch step35/853-enhancement-log-tool-halluci 2026-04-29 11:44:10 +00:00
All checks were successful
Lint / lint (pull_request) Successful in 10s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
CI
Continuous integration, runners, workflow issues
 QA
Quality assurance, testing, production audit
 accessibility
Auto-created for game epic
 assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-kimi
Task assigned to KimiClaw for processing
 audit-finding
batch-pipeline
batch-pipeline label
 blocked
Blocked by external dependency or merge conflict
 bug
claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 critical
docs
Auto-created for game epic
 duplicate
Duplicate of another issue
 epic
Epic - large feature with multiple sub-tasks
 epic:memory
Unified Memory Architecture epic
 game
Auto-created for game epic
 gaming
Gaming agent capabilities
 implementation
Auto-created for game epic
 infra
Infrastructure, VPS, DNS, deployment
 infrastructure
Auto-created for game epic
 juice
Auto-created for game epic
 kimi-done
KimiClaw has completed this task
 kimi-in-progress
KimiClaw is actively working on this
 mcp
MCP (Model Context Protocol) tools & servers
 mechanics
Auto-created for game epic
 mobile
Auto-created for game epic
 morrowind
Morrowind Agent gameplay & MCP integration
 needs-review
PR or issue requires reviewer sign-off before merge/close
 ops
p0-critical
p1-important
performance
Auto-created for game epic
 phase:1-activate
Phase 1: Activate existing systems
 phase:2-operational
Phase 2: Operational memory bridge
 phase:3-episodic
Phase 3: Proactive episodic recall
 phase:4-hygiene
Phase 4: Memory hygiene and quality
 phase:5-integration
Phase 5: Integration and testing
 poka-yoke
polish
Auto-created for game epic
 priority-high
Auto-created for game epic
 priority-low
Auto-created for game epic
 priority-medium
Auto-created for game epic
 research
security
Security hardening, vulnerability fixes
 shared-infra
Shared infrastructure across projects (Playground + LPM)
 sovereign
Auto-created for game epic
 stale
No activity, pending triage or closure
 throughput-10x
throughput-10x label
 token-masterplan
token-masterplan label
 tool-evaluation
ux
Auto-created for game epic
 velocity-engine
Auto-generated by velocity engine
 visuals
Auto-created for game epic
 wont-fix
Closed as intentionally not fixing — explicit descope
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok hermes kimi manus perplexity sonnet
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/hermes-agent#1109
Delete Branch "step35/853-enhancement-log-tool-halluci"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?