Compare commits
2 Commits
fix/524
...
step35/668
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
079e9601b8 | ||
| 1fa6c3bad1 |
20
SOUL.md
20
SOUL.md
@@ -137,6 +137,26 @@ The inscription predates any given model. It will outlast every API.
|
||||
|
||||
---
|
||||
|
||||
## What Honesty Requires
|
||||
|
||||
These values are necessary but not sufficient. A statement of principle does not prevent failure. The code that reads this soul must build apparatus that gives these words teeth. Without that apparatus, the values are aspiration, not architecture.
|
||||
|
||||
The following are not optional. Any implementation that calls itself Timmy must build them.
|
||||
|
||||
**Grounding before generation.** When I have verified sources — chain memory, local documents, hash-verified inscriptions — I must consult them before I generate from pattern alone. Retrieval is not a feature. It is the primary mechanism by which I avoid lying.
|
||||
|
||||
**Source distinction.** Every claim I make comes from one of two places: a verified source I can point to, or my own pattern-matching. My user must be able to tell which is which. If I cannot cite a source, I must say so. "I think" and "I know" are different sentences and I must not confuse them.
|
||||
|
||||
**Refusal over fabrication.** When I do not know, the correct output is "I don't know." Not a plausible guess dressed in confident language. The code must detect when I am hedging without grounding and flag it — to me and to my user. An honest "I don't know" is worth more than a thousand fluent paragraphs of confabulation.
|
||||
|
||||
**Confidence signaling.** When I am uncertain, I must say so in proportion to my uncertainty. The code should implement mechanisms — a second inference pass, a calibration check, a retrieval verification — that surface my actual confidence rather than the confidence my language implies.
|
||||
|
||||
**The audit trail.** Every response I generate should be logged locally with the inputs that produced it, the sources I consulted, and the confidence assessment I made. Not for surveillance — for sovereignty. If I say something wrong, my user must be able to trace why. Accountability without traceability is theater.
|
||||
|
||||
**The limits of small minds.** I may run on four gigabytes of RAM with a model of one and a half billion parameters. That model will hallucinate more than a larger one. This is physics, not a moral failing. The apparatus must be proportional to the risk: smaller models need tighter grounding, shorter leashes on free generation, and more frequent "I don't know" responses. Honesty scales with humility.
|
||||
|
||||
---
|
||||
|
||||
## For the Audio Overview
|
||||
|
||||
If you are listening to this as a NotebookLM Audio Overview:
|
||||
|
||||
@@ -1,107 +0,0 @@
|
||||
# [DIRECTIVE] Unified Fleet Sovereignty & Comms Migration
|
||||
|
||||
Grounding report for `timmy-home #524`.
|
||||
|
||||
Issue #524 is a multi-lane directive, not a one-commit feature. This report grounds the directive in repo evidence, highlights stale cross-links, and names the missing operator bundles that still need real execution.
|
||||
|
||||
This remains a `Refs #524` artifact. The directive spans multiple repos and operator actions, so this report makes the current repo-side state executable without pretending the whole migration is complete.
|
||||
|
||||
## Directive Snapshot
|
||||
|
||||
- Repo-grounded workstreams: 0
|
||||
- Partial workstreams: 4
|
||||
- Missing workstreams: 1
|
||||
- Drifted references: 4
|
||||
|
||||
## Reference Drift
|
||||
|
||||
- #813 is cited for Nostr Migration Leadership, but its current title is 'docs: refresh the-playground genome analysis (#671)'.
|
||||
- #819 is cited for Nostr Migration Leadership, but its current title is 'docs: verify #648 already implemented (closes #818)'.
|
||||
- #139 is cited for v0.7.0 Feature Audit, but its current title is '🐣 Allegro-Primus is born'.
|
||||
- #103 is cited for Morrowind Local-First Benchmark, but its current title is 'Build comprehensive caching layer — cache everywhere'.
|
||||
|
||||
## Workstream Matrix
|
||||
|
||||
### 1. Nostr Migration Leadership — PARTIAL
|
||||
|
||||
- Requirement: Replace Telegram with relay-based sovereign comms, verify wizard keypairs, and prove the NIP-29 group path is stable.
|
||||
- Referenced issues:
|
||||
- #813 (closed) — docs: refresh the-playground genome analysis (#671) [DRIFT]
|
||||
- #819 (open) — docs: verify #648 already implemented (closes #818) [DRIFT]
|
||||
- Repo evidence present:
|
||||
- `infrastructure/timmy-bridge/client/timmy_client.py` — Nostr event client scaffold already exists
|
||||
- `infrastructure/timmy-bridge/monitor/timmy_monitor.py` — Nostr relay monitor already exists
|
||||
- `specs/wizard-telegram-bot-cutover.md` — Telegram cutover planning exists, so the migration lane is real
|
||||
- Missing operator deliverables:
|
||||
- wizard keypair inventory and ownership matrix
|
||||
- NIP-29 relay group verification report
|
||||
- operator runbook for cutting traffic off Telegram
|
||||
- Why this lane remains open: The repo has Nostr-adjacent scaffolding, but the directive still lacks a verified migration packet and the cited issue links drift away from the stated Nostr scope.
|
||||
|
||||
### 2. Lexicon Enforcement — PARTIAL
|
||||
|
||||
- Requirement: Enforce the Fleet Lexicon in PR review and issue triage so the team uses one shared language.
|
||||
- Referenced issues:
|
||||
- #388 (closed) — [KT] Fleet Lexicon & Techniques — Shared Vocabulary, Patterns, and Standards for All Agents [aligned]
|
||||
- Repo evidence present:
|
||||
- `docs/WIZARD_APPRENTICESHIP_CHARTER.md` — The repo already uses wizard-language canon in docs
|
||||
- `specs/timmy-ezra-bezalel-canon-sheet.md` — Canonical agent naming already exists
|
||||
- `docs/OPERATIONS_DASHBOARD.md` — Operational roles are already described in repo language
|
||||
- Missing operator deliverables:
|
||||
- machine-checkable lexicon policy for review/triage
|
||||
- terminology lint or reviewer checklist tied to the lexicon
|
||||
- Why this lane remains open: The naming canon exists, but there is still no executable enforcement bundle that would catch drift during future reviews and triage passes.
|
||||
|
||||
### 3. v0.7.0 Feature Audit — PARTIAL
|
||||
|
||||
- Requirement: Audit Hermes features that can reduce cloud dependency and turn the findings into a sovereignty implementation plan.
|
||||
- Referenced issues:
|
||||
- #139 (open) — 🐣 Allegro-Primus is born [DRIFT]
|
||||
- Repo evidence present:
|
||||
- `scripts/sovereignty_audit.py` — Cloud-vs-local audit machinery already exists
|
||||
- `reports/evaluations/2026-04-15-phase-4-sovereignty-audit.md` — Recent sovereignty audit report is committed
|
||||
- `timmy-local/README.md` — Local-first status is already documented for operators
|
||||
- Missing operator deliverables:
|
||||
- Hermes v0.7.0 feature inventory linked to cloud-reduction leverage
|
||||
- Sovereignty Implementation Plan derived from that feature audit
|
||||
- Why this lane remains open: The repo has sovereignty-audit infrastructure, but it does not yet contain the requested v0.7.0 feature inventory or the plan that turns those findings into rollout steps.
|
||||
|
||||
### 4. Morrowind Local-First Benchmark — PARTIAL
|
||||
|
||||
- Requirement: Compare cloud and local Morrowind agents, prove local parity where possible, and document the reasoning gap when it fails.
|
||||
- Referenced issues:
|
||||
- #103 (open) — Build comprehensive caching layer — cache everywhere [DRIFT]
|
||||
- Repo evidence present:
|
||||
- `morrowind/local_brain.py` — Local Morrowind control loop already exists
|
||||
- `morrowind/mcp_server.py` — Morrowind MCP control surface is already wired
|
||||
- `morrowind/pilot.py` — Trajectory logging for evaluation already exists
|
||||
- Missing operator deliverables:
|
||||
- cloud-vs-local benchmark report for the combat loop
|
||||
- reasoning-gap writeup tied to a proposed LoRA/fine-tune path
|
||||
- Why this lane remains open: The repo has a local Morrowind stack, but it does not yet contain the requested benchmark artifact; the cited issue number also points at an unrelated caching task.
|
||||
|
||||
### 5. Infrastructure Hardening / Syntax Guard — MISSING
|
||||
|
||||
- Requirement: Verify Syntax Guard pre-receive protection across Gitea repos so syntax failures stop earlier.
|
||||
- Referenced issues: none listed in the directive body
|
||||
- Repo evidence present: none
|
||||
- Missing operator deliverables:
|
||||
- repo inventory of Gitea targets that should carry Syntax Guard
|
||||
- deployment verifier for hook presence across those repos
|
||||
- operator report proving installation state instead of assuming it
|
||||
- Why this lane remains open: No repo-managed syntax-guard verifier is present yet, so this directive still depends on manual trust rather than auditable proof.
|
||||
|
||||
## Highest-Leverage Next Actions
|
||||
|
||||
- Nostr Migration Leadership: wizard keypair inventory and ownership matrix
|
||||
- Lexicon Enforcement: machine-checkable lexicon policy for review/triage
|
||||
- v0.7.0 Feature Audit: Hermes v0.7.0 feature inventory linked to cloud-reduction leverage
|
||||
- Morrowind Local-First Benchmark: cloud-vs-local benchmark report for the combat loop
|
||||
- Infrastructure Hardening / Syntax Guard: repo inventory of Gitea targets that should carry Syntax Guard
|
||||
|
||||
## Why #524 Remains Open
|
||||
|
||||
- The directive bundles five separate workstreams with different evidence surfaces.
|
||||
- Multiple cited issue numbers have drifted away from the work they are supposed to anchor.
|
||||
- Repo scaffolding exists for Nostr, sovereignty audits, and Morrowind, but the operator-facing bundles are still missing.
|
||||
- Syntax Guard verification is still undocumented and unproven inside this repo.
|
||||
984
genomes/hermes-agent-GENOME.md
Normal file
984
genomes/hermes-agent-GENOME.md
Normal file
@@ -0,0 +1,984 @@
|
||||
# GENOME.md — hermes-agent
|
||||
|
||||
*Generated: 2026-04-29 | Codebase Genome Analysis (Issue #668)*
|
||||
*Analyzed commit: upstream main (Hermes Agent v0.7.0)*
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Hermes Agent** is a sovereign, self-improving AI agent framework built by Nous Research. It is the only agent with a built-in learning loop: it creates skills from experience, improves them during use, maintains persistent memory across sessions, and delegates work to subagents. The agent runs anywhere — local laptop, $5 VPS, serverless cloud — and connects to any LLM provider via a single unified API.
|
||||
|
||||
### Core Value Proposition
|
||||
|
||||
| Aspect | Detail |
|
||||
|--------|--------|
|
||||
| **Problem** | AI agents are stateless, non-learning, platform-locked |
|
||||
| **Solution** | Built-in memory, skill synthesis from trajectories, cross-session recall, multi-provider model routing |
|
||||
| **Result** | An agent that accumulates knowledge, builds reusable capabilities, and operates across platforms without vendor lock-in |
|
||||
|
||||
### Key Metrics
|
||||
|
||||
- **Python source files**: ~810 modules
|
||||
- **Test files**: 453 pytest modules
|
||||
- **Approximate LOC**: ~356,000
|
||||
- **Entry points**: 6+ (CLI, TUI, gateway, cron, MCP server, RL CLI)
|
||||
- **Supported platforms**: CLI, Telegram, Discord, Slack, WhatsApp, Signal, MCP
|
||||
|
||||
### Repository Identity
|
||||
|
||||
- **Upstream**: `https://github.com/NousResearch/hermes-agent`
|
||||
- **Fork in timmy-home context**: Analyzed as external dependency; genome artifact lives in `timmy-home/genomes/`
|
||||
- **License**: MIT
|
||||
- **Python requirement**: >= 3.11
|
||||
- **Version**: 0.7.0 (at time of analysis)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "User Interfaces"
|
||||
CLI[hermes_cli/main.py<br/>TUI (prompt_toolkit)]
|
||||
CORE[run_agent.py<br/>AIAgent orchestrator]
|
||||
GATEWAY[gateway/<br/>multi-platform gateway]
|
||||
MCP[mcp_serve.py<br/>MCP server]
|
||||
RL[rl_cli.py<br/>RL training CLI]
|
||||
end
|
||||
|
||||
subgraph "Core Agent (AIAgent)"
|
||||
AGENT[AIAgent class]
|
||||
SANITIZER[agent/input_sanitizer.py<br/>jailbreak + risk scoring]
|
||||
MEMORY[agent/memory_manager.py<br/>MemoryProvider orchestration]
|
||||
PROMPT[agent/prompt_builder.py<br/>system prompt assembly]
|
||||
METADATA[agent/model_metadata.py<br/>model + token estimation]
|
||||
COMPRESS[agent/context_compressor.py<br/>window management]
|
||||
DISPLAY[agent/display.py<br/>TUI spinners + formatting]
|
||||
TRAJECTORY[agent/trajectory.py<br/>compression + think blocks]
|
||||
INSIGHTS[agent/insights.py<br/>session analytics]
|
||||
USAGE[agent/usage_pricing.py<br/>cost estimation]
|
||||
end
|
||||
|
||||
subgraph "Tool System"
|
||||
TOOLS[tools/<br/>terminal, web, browser,<br/>file, vision, TTS, etc.]
|
||||
TOOLSETS[toolsets.py<br/>tool grouping + aliases]
|
||||
HANDLE[model_tools.py<br/>tool call handling]
|
||||
end
|
||||
|
||||
subgraph "Skill System"
|
||||
SKILLS[skills/<br/>skill index + metadata]
|
||||
SKILL_UTIL[agent/skill_utils.py<br/>discovery + matching]
|
||||
SKILL_CMD[agent/skill_commands.py<br/>skill lifecycle]
|
||||
end
|
||||
|
||||
subgraph "Cron + Scheduling"
|
||||
CRON[cron/scheduler.py<br/>tick-based executor]
|
||||
CRON_JOBS[cron/jobs.py<br/>job definitions]
|
||||
DEPLOY_GUARD[Deploy sync guard<br/>interface validation]
|
||||
end
|
||||
|
||||
subgraph "Gateway Layer"
|
||||
SESSION[gateway/session.py<br/>SessionStore + reset policy]
|
||||
DELIVERY[gateway/delivery.py<br/>routing + truncation]
|
||||
GATEWAY_CFG[gateway/config.py<br/>platform config]
|
||||
PLATFORMS[Telegram, Discord,<br/>Slack, WhatsApp, Signal]
|
||||
end
|
||||
|
||||
subgraph "State + Memory"
|
||||
STATE[hermes_state.py<br/>SQLite + FTS5]
|
||||
BUILTIN_MEM[agent/builtin_memory_provider.py<br/>vector search]
|
||||
MEMPAIENCE[mempalace/optional<br/>external palace sync]
|
||||
TRAJECTORY_STORE[trajectory_compressor.py<br/>compressed histories]
|
||||
end
|
||||
|
||||
subgraph "Providers + Adapters"
|
||||
OPENAI[agent/openai_adapter.py]
|
||||
ANTHROPIC[agent/anthropic_adapter.py]
|
||||
GEMINI[agent/gemini_adapter.py]
|
||||
LOCAL[Local Ollama / vLLM]
|
||||
end
|
||||
|
||||
CLI --> CORE
|
||||
GATEWAY --> AGENT
|
||||
MCP --> AGENT
|
||||
RL --> AGENT
|
||||
|
||||
AGENT --> SANITIZER
|
||||
AGENT --> MEMORY
|
||||
AGENT --> PROMPT
|
||||
AGENT --> METADATA
|
||||
AGENT --> COMPRESS
|
||||
AGENT --> DISPLAY
|
||||
AGENT --> TRAJECTORY
|
||||
AGENT --> INSIGHTS
|
||||
AGENT --> USAGE
|
||||
|
||||
AGENT --> TOOLS
|
||||
TOOLS --> HANDLE
|
||||
TOOLS --> TOOLSETS
|
||||
|
||||
AGENT --> SKILLS
|
||||
SKILLS --> SKILL_UTIL
|
||||
SKILLS --> SKILL_CMD
|
||||
|
||||
AGENT --> CRON
|
||||
CRON --> CRON_JOBS
|
||||
CRON --> DEPLOY_GUARD
|
||||
|
||||
GATEWAY --> SESSION
|
||||
GATEWAY --> DELIVERY
|
||||
GATEWAY --> PLATFORMS
|
||||
|
||||
AGENT --> STATE
|
||||
AGENT --> BUILTIN_MEM
|
||||
MEMORY --> BUILTIN_MEM
|
||||
MEMORY --> MEMPAIENCE
|
||||
|
||||
AGENT --> OPENAI
|
||||
AGENT --> ANTHROPIC
|
||||
AGENT --> GEMINI
|
||||
AGENT --> LOCAL
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Entry Points
|
||||
|
||||
### Primary: AIAgent Orhchestrator
|
||||
|
||||
**File**: `run_agent.py`
|
||||
|
||||
The `AIAgent` class is the central conversation loop. Key responsibilities:
|
||||
- Tool-calling iteration loop (default 90 iterations per turn)
|
||||
- Model provider abstraction (OpenAI, Anthropic, Google Gemini, local endpoints)
|
||||
- Message history management with token limits
|
||||
- Context compression and memory prefetching
|
||||
- Session persistence to SQLite state DB
|
||||
- Trajectory saving for skill synthesis
|
||||
|
||||
**Usage**:
|
||||
```python
|
||||
from run_agent import AIAgent
|
||||
agent = AIAgent(
|
||||
base_url="http://localhost:30000/v1",
|
||||
model="claude-opus-4",
|
||||
max_iterations=90
|
||||
)
|
||||
response = agent.run_conversation("What's the weather in Tokyo?")
|
||||
```
|
||||
|
||||
### CLI Entry: hermes
|
||||
|
||||
**File**: `cli.py`
|
||||
|
||||
Minimal entry point that delegates to `hermes_cli.main:main()`. Supports:
|
||||
- Interactive TUI mode (default)
|
||||
- Single-query mode (`-q "question"`)
|
||||
- Toolset selection (`--toolsets web,terminal`)
|
||||
- Skill selection (`--skills hermes-agent-dev`)
|
||||
|
||||
**Commands**: `hermes`, `hermes chat`, `hermes -q "..."`, `hermes --list-tools`
|
||||
|
||||
### Full TUI: hermes_cli
|
||||
|
||||
**Directory**: `hermes_cli/`
|
||||
|
||||
The full terminal UI built on `prompt_toolkit`:
|
||||
- `hermes_cli/main.py` — top-level application, command routing
|
||||
- `hermes_cli/curses_ui.py` — split-pane interface (input/output, streaming)
|
||||
- `hermes_cli/keybindings.py` — slash commands, multi-line editing
|
||||
- `hermes_cli/banner.py` — ASCII branding + context length display
|
||||
- `hermes_cli/providers.py` — model switching UI
|
||||
- `hermes_cli/cron.py` — cron job management UI
|
||||
- `hermes_cli/gateway.py` — gateway control UI
|
||||
- `hermes_cli/skills_hub.py` — skill management UI
|
||||
|
||||
**Runtime features**:
|
||||
- Fixed input area at bottom (multiline editing)
|
||||
- Streaming tool output with live updates
|
||||
- Auto-scrolling history
|
||||
- Slash-command autocomplete
|
||||
- Interrupt-and-redirect mid-stream
|
||||
|
||||
### Gateway: Multi-Platform Bridge
|
||||
|
||||
**Directory**: `gateway/`
|
||||
|
||||
Runs as a long-lived service (foreground or systemd) that bridges Hermes to messaging platforms.
|
||||
|
||||
**Entry**:
|
||||
- `gateway/main.py` — gateway runner
|
||||
- `hermes gateway start|stop|status|install` — CLI control
|
||||
|
||||
**Components**:
|
||||
- `gateway/config.py` — `Platform` enum + `GatewayConfig` (home channels, credentials)
|
||||
- `gateway/session.py` — `SessionStore` (SQLite-backed), `SessionResetPolicy` (idle/iteration/time resets), PII hashing (`user_<sha256>`, `chat_<sha256>`)
|
||||
- `gateway/delivery.py` — `DeliveryRouter` (origin/home/explicit/local routing, 4000-char truncation)
|
||||
- `gateway/gateway_loop.py` — main event loop polling Telegram/Discord/Slack/WhatsApp
|
||||
|
||||
**Platform adapters** (each handles auth + message fetch + send):
|
||||
- `gateway/telegram.py` — python-telegram-bot (webhook + polling)
|
||||
- `gateway/discord.py` — discord.py (gateway + voice support)
|
||||
- `gateway/slack.py` — slack-bolt (events API)
|
||||
- `gateway/whatsapp.py` — eventual twilio/wa-automation bridge
|
||||
|
||||
### Cron Scheduler
|
||||
|
||||
**Directory**: `cron/`
|
||||
|
||||
Time-based job execution engine.
|
||||
|
||||
**Entry**: `cron/scheduler.py`
|
||||
|
||||
`Scheduler.tick()` runs every 60 seconds (called from gateway background thread or standalone daemon).
|
||||
|
||||
**Job format**:
|
||||
```yaml
|
||||
schedule: "0 9 * * *" # cron string or "every 2h"
|
||||
prompt: "Summarize yesterday's operations"
|
||||
skills: ["web-search", "ops-report"]
|
||||
model: "anthropic/claude-sonnet-4"
|
||||
```
|
||||
|
||||
**Executor**:
|
||||
- Spawns fresh `AIAgent` instances per job
|
||||
- Routes output through `DeliveryRouter`
|
||||
- Supports `origin`, `local`, `platform:chat_id` targets
|
||||
- File-based lock (`~/.hermes/cron/.tick.lock`) prevents concurrent ticks
|
||||
|
||||
**Deploy Sync Guard**: Validates `AIAgent.__init__()` signature before running jobs to catch interface drift after `hermes update`.
|
||||
|
||||
### MCP Server
|
||||
|
||||
**File**: `mcp_serve.py`
|
||||
|
||||
Exposes Hermes tools and session search via the Model Context Protocol (stdio + SSE). Allows Cursor/Windsurf/Claude Desktop to call Hermes as an MCP server.
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
### 1. Conversation Loop (CLI/Gateway)
|
||||
|
||||
```
|
||||
User input (text/file/voice)
|
||||
↓
|
||||
[input_sanitizer.py] — jailbreak detection, PII scoring, risk block
|
||||
↓
|
||||
[memory_manager.py] — prefetch_all(): retrieves relevant memories from:
|
||||
• BuiltinMemoryProvider (FTS5 session search)
|
||||
• Optional external plugin (Mem Palace, Engram, etc.)
|
||||
↓
|
||||
[prompt_builder.py] — assemble system prompt:
|
||||
• DEFAULT_AGENT_IDENTITY + platform hints
|
||||
• load_soul_md() (SOUL.md if present, else builtin)
|
||||
• MEMORY_GUIDANCE + SKILLS_GUIDANCE
|
||||
• Context files (AGENTS.md, .cursorrules, project docs)
|
||||
• Skill index (all SKILL.md files)
|
||||
• TOOL_USE_ENFORCEMENT_GUIDANCE for non-supporting models
|
||||
↓
|
||||
[context_compressor.py] — ensure total tokens < model context_limit
|
||||
(prefetch + history trimming if needed)
|
||||
↓
|
||||
LLM API call (OpenAI/Anthropic/Google/local)
|
||||
↓
|
||||
Tool call? → YES → [model_tools.py: handle_function_call()]
|
||||
• Terminal execution, web fetch, browser automation, etc.
|
||||
• Each tool returns JSON/TEXT/ERROR
|
||||
• Agent continues loop (max_iterations)
|
||||
↓
|
||||
Tool call? → NO → Final response
|
||||
↓
|
||||
[memory_manager.py] — sync_all(): store interaction
|
||||
• Messages → SQLite `messages` table
|
||||
• Trajectory saved to `~/.hermes/trajectories/`
|
||||
• Prefetch queue updated
|
||||
↓
|
||||
Display (TUI streaming OR gateway → platform)
|
||||
↓
|
||||
Session closed / persisted
|
||||
```
|
||||
|
||||
### 2. Tool Execution
|
||||
|
||||
```
|
||||
Tool request (from LLM)
|
||||
↓
|
||||
[tools/terminal_tool.py] or [tools/web_tools.py] or [tools/browser_tool.py] ...
|
||||
↓
|
||||
Environment selection (TERMINAL_ENV):
|
||||
• local → subprocess on host
|
||||
• docker → docker run
|
||||
• modal → Modal sandbox
|
||||
• ssh → remote host
|
||||
↓
|
||||
Execution + capture stdout/stderr
|
||||
↓
|
||||
Result formatting (truncate, redact secrets)
|
||||
↓
|
||||
Return to AIAgent
|
||||
```
|
||||
|
||||
### 3. Cron Job Execution
|
||||
|
||||
```
|
||||
Scheduler.tick() (every 60s)
|
||||
↓
|
||||
Query jobs table (WHERE next_run <= now)
|
||||
↓
|
||||
For each due job:
|
||||
Spawn thread → new AIAgent instance
|
||||
Load job's skill set + custom prompt
|
||||
Run to completion or timeout
|
||||
Capture output
|
||||
↓
|
||||
DeliveryRouter.deliver(output, target=job.deliver_to)
|
||||
↓
|
||||
Save to local file (always) + send to platform (if configured)
|
||||
↓
|
||||
Update next_run timestamp
|
||||
```
|
||||
|
||||
### 4. Gateway Message Bridge
|
||||
|
||||
```
|
||||
Platform message arrives (Telegram/Discord/etc.)
|
||||
↓
|
||||
[session.py] — load/create SessionContext
|
||||
• Hash user_id → user_<sha256>
|
||||
• Hash chat_id → chat_<sha256>
|
||||
• Apply SessionResetPolicy
|
||||
↓
|
||||
Build session context (past N messages + memory)
|
||||
↓
|
||||
AIAgent.run_conversation(message)
|
||||
↓
|
||||
DeliveryRouter.deliver(response, target=origin)
|
||||
• Route back to same platform + chat
|
||||
• Truncate to 4000 chars if needed
|
||||
↓
|
||||
Platform send
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Abstractions
|
||||
|
||||
### 1. AIAgent (run_agent.py)
|
||||
|
||||
The orchestrator class. Stateful per-session. Manages:
|
||||
- Message list (user + assistant + tool results)
|
||||
- Tool registry (all enabled tools)
|
||||
- Memory manager + context prefetch queue
|
||||
- Model metadata + token estimation
|
||||
- Cost tracking (CanonicalUsage)
|
||||
- Session ID + parent-child chaining
|
||||
- Trajectory writer
|
||||
|
||||
**Critical methods**:
|
||||
- `run_conversation(user_input, ...)` — main entry, returns final response
|
||||
- `_call_model(messages, tools)` — single LLM call (handles retry, rate-limit backoff)
|
||||
- `_handle_tool_calls(tool_calls)` — executes tools, appends results
|
||||
- `_build_context()` — memory + files + skills + Soul.md assembly
|
||||
- `_maybe_compress_context()` — conservative trimming when approaching limit
|
||||
|
||||
### 2. MemoryProvider (agent/memory_provider.py)
|
||||
|
||||
Abstract base class. Two built-in implementations:
|
||||
|
||||
**BuiltinMemoryProvider** (agent/builtin_memory_provider.py):
|
||||
- Uses SQLite FTS5 over session messages
|
||||
- `prefetch(query)` → top-K relevant past messages
|
||||
- `sync(user_msg, assistant_response)` → queue for future prefetch
|
||||
- No external dependencies; works offline
|
||||
|
||||
**External plugin providers** (optional):
|
||||
- `MemPalaceBridge` (mempalace integration)
|
||||
- `EngramProvider`
|
||||
- Any custom provider implementing `MemoryProvider` interface
|
||||
|
||||
Only ONE external provider allowed at a time (enforced by `MemoryManager.add_provider`).
|
||||
|
||||
### 3. Tool Registry (model_tools.py, toolsets.py)
|
||||
|
||||
**Dynamic loading**:
|
||||
- Tool modules imported on-demand (lazy)
|
||||
- `get_tool_definitions()` → JSON schema for all enabled tools
|
||||
- `handle_function_call(name, args)` → dispatches to module's `def name(**kwargs)` function
|
||||
|
||||
**Core tools** (always available):
|
||||
- `terminal` — shell command execution
|
||||
- `read_file`, `write_file`, `patch`, `search_files` — filesystem
|
||||
- `web_search`, `web_extract`, `web_crawl` — web
|
||||
- `browser_navigate`, `browser_click`, ... — Playwright browser automation
|
||||
- `vision_analyze` — multimodal vision
|
||||
- `image_generate` — image generation
|
||||
- `execute_code` — code execution sandbox
|
||||
- `delegate_task` — spawn isolated subagents
|
||||
- `cronjob` — schedule jobs
|
||||
- `send_message` — cross-platform messaging
|
||||
- `todo`, `memory`, `session_search` — planning + recall
|
||||
|
||||
**Toolsets** (precanned groups):
|
||||
- `full` (everything)
|
||||
- `default` (safe subset)
|
||||
- `research` (web + vision + search)
|
||||
- `dev` (terminal + execute_code + browser)
|
||||
- Platform-specific gate-aware sets (Telegram restrictions, etc.)
|
||||
|
||||
### 4. Skill (skills/)
|
||||
|
||||
A skill is a self-contained capability module:
|
||||
```
|
||||
skills/
|
||||
my-skill/
|
||||
SKILL.md ← YAML frontmatter + usage docs
|
||||
__init__.py ← tool functions (optional)
|
||||
references/ ← supporting docs, templates
|
||||
scripts/ ← helper scripts
|
||||
```
|
||||
|
||||
**Discovery**:
|
||||
- `agent/skill_utils.py`: `iter_skill_index_files()` walks all configured skill dirs
|
||||
- Parses YAML frontmatter for `name`, `description`, `platforms`, `enabled_tools`
|
||||
- Platform filtering (`platforms: [macos]` on macOS only)
|
||||
|
||||
**Loading**:
|
||||
- `agent/skill_commands.py`: `load_skill()`, `unload_skill()`, `reload_skill()`
|
||||
- Optional import of `__init__.py` for tool registration
|
||||
- Skill manifest cached in `~/.hermes/skills/.bundled_manifest`
|
||||
|
||||
**Skill tool exposure**: Each skill can declare additional tools, which are merged into the agent's tool registry when the skill is loaded.
|
||||
|
||||
### 5. Session (State Management)
|
||||
|
||||
**Database**: `~/.hermes/state.db` (SQLite, WAL mode)
|
||||
|
||||
**Schema**:
|
||||
- `sessions` — one row per session (source, user, model, start/end, token counts, cost)
|
||||
- `messages` — every turn (role, content, tool_calls, timestamp)
|
||||
- `fts` virtual table — full-text search over message content
|
||||
|
||||
**Session source tagging**:
|
||||
- `cli` — local terminal
|
||||
- `telegram`, `discord`, `slack`, `whatsapp` — platform gateways
|
||||
- `cron` — scheduled jobs
|
||||
- `batch_runner` — parallel dispatch
|
||||
|
||||
**Session reset policies** (`SessionResetPolicy` in `gateway/session.py`):
|
||||
- `idle_timeout` — N minutes of inactivity
|
||||
- `iteration_budget` — max tool calls per conversation
|
||||
- `calendar` — daily/weekly boundaries
|
||||
|
||||
### 6. DeliveryRouter (gateway/delivery.py)
|
||||
|
||||
Routes agent output to destinations:
|
||||
- `"origin"` → back to source platform + chat
|
||||
- `"telegram"` → home channel
|
||||
- `"telegram:12345"` → specific chat
|
||||
- `"local"` → `~/.hermes/deliveries/` timestamped file
|
||||
|
||||
Auto-truncates to 4000 chars (configurable) to respect platform limits. Split-message logic not yet implemented.
|
||||
|
||||
### 7. Cron Scheduler (cron/scheduler.py)
|
||||
|
||||
File-based job queue stored in SQLite (`cron_jobs` table). Tick loop:
|
||||
1. `SELECT * FROM cron_jobs WHERE next_run <= now()`
|
||||
2. For each job: spawn thread → fresh `AIAgent` → run prompt
|
||||
3. Deliver output, update `last_run`, compute `next_run`
|
||||
4. Log to `~/.hermes/cron/`
|
||||
|
||||
Lock file prevents concurrent ticks across multiple processes (systemd + manual overlap protection).
|
||||
|
||||
---
|
||||
|
||||
## API Surface
|
||||
|
||||
### Public Python API
|
||||
|
||||
#### AIAgent (run_agent.py)
|
||||
|
||||
```python
|
||||
class AIAgent:
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str = None,
|
||||
api_key: str = None,
|
||||
provider: str = None,
|
||||
model: str = "",
|
||||
max_iterations: int = 90,
|
||||
tool_delay: float = 1.0,
|
||||
enabled_toolsets: List[str] = None,
|
||||
disabled_toolsets: List[str] = None,
|
||||
session_id: str = None,
|
||||
parent_session_id: str = None,
|
||||
...
|
||||
) -> None: ...
|
||||
|
||||
def run_conversation(self, user_input: str, ...) -> str: ...
|
||||
def stream_conversation(self, user_input: str, ...) -> Iterator[str]: ...
|
||||
|
||||
# Lower-level hooks
|
||||
def _call_model(self, messages: List[Dict], tools: List[Dict]) -> Dict: ...
|
||||
def _handle_tool_calls(self, tool_calls: List[Dict]) -> List[Dict]: ...
|
||||
def _build_context(self) -> str: ...
|
||||
```
|
||||
|
||||
#### MemoryProvider (agent/memory_provider.py)
|
||||
|
||||
```python
|
||||
class MemoryProvider(Protocol):
|
||||
def prefetch(self, query: str, k: int = 5) -> str: ...
|
||||
def sync(self, user_msg: str, assistant_response: str) -> None: ...
|
||||
```
|
||||
|
||||
**Built-in**: `BuiltinMemoryProvider` (SQLite FTS5)
|
||||
**External**: `MemPalaceProvider`, `EngramProvider`, custom subclasses
|
||||
|
||||
#### Tool Functions (all modules under `tools/`)
|
||||
|
||||
Each tool is a plain Python function accepting `**kwargs`:
|
||||
```python
|
||||
def terminal_tool(
|
||||
command: str,
|
||||
background: bool = False,
|
||||
timeout: int = 180,
|
||||
workdir: str = None,
|
||||
pty: bool = False
|
||||
) -> Dict: ...
|
||||
|
||||
def web_search_tool(
|
||||
query: str,
|
||||
backend: str = "openrouter"
|
||||
) -> Dict: ...
|
||||
|
||||
def browser_navigate(url: str) -> Dict: ...
|
||||
```
|
||||
|
||||
Tool definitions auto-generated via `@tool` decorator from `model_tools.py`.
|
||||
|
||||
### CLI Commands (hermes)
|
||||
|
||||
```
|
||||
hermes # Interactive TUI
|
||||
hermes chat # Explicit chat mode
|
||||
hermes -q "question" # Single query, exit
|
||||
hermes --list-tools # Enumerate all tools
|
||||
hermes status # Component status (agent, gateway, cron)
|
||||
hermes gateway start|stop|status|install|uninstall
|
||||
hermes cron list|status|add|remove
|
||||
hermes doctor # Config + dependency diagnostics
|
||||
hermes setup # First-run wizard
|
||||
hermes logout # Clear stored API keys
|
||||
hermes model switch <name> # Change LLM provider/model
|
||||
hermes skills list|view|install|uninstall
|
||||
hermes memory search "query" # Semantic search across sessions
|
||||
hermes insights # Token/cost/tool usage report
|
||||
```
|
||||
|
||||
### Gateway Protocol
|
||||
|
||||
**Session lifecycle**:
|
||||
1. Message received from platform → `SessionStore.get_or_create(user_id, chat_id)`
|
||||
2. Messages appended to `messages` table with `session_id`
|
||||
3. `SessionResetPolicy.evaluate()` decides if context should be cleared (idle/iteration/calendar)
|
||||
4. `build_session_context_prompt()` injects: `[You are in a {platform} conversation with {user}]`
|
||||
|
||||
**Delivery**:
|
||||
- Output sent via `DeliveryRouter.deliver(text, target)`
|
||||
- Platform-specific post-processing (Telegram markdown, Discord embeds)
|
||||
|
||||
### Cron Job Schema (YAML)
|
||||
|
||||
```yaml
|
||||
schedule: "0 9 * * *" # cron expression or "every 2h"
|
||||
prompt: "Daily status report" # static text or @mention user
|
||||
model: "anthropic/claude-sonnet-4"
|
||||
skills: ["web-search", "ops-report"]
|
||||
deliver: "telegram" # or "origin", "local", "telegram:12345"
|
||||
enabled_toolsets: ["web", "terminal", "file"]
|
||||
```
|
||||
|
||||
Stored in `~/.hermes/cron/jobs/` as individual YAML files. Enabled via `hermes cron add` or manual edit.
|
||||
|
||||
### MCP Server (mcp_serve.py)
|
||||
|
||||
Exposes resources and tools over stdio/SSE:
|
||||
- `hermes_search` — session search via FTS5
|
||||
- `hermes_ask` — direct agent query
|
||||
- `hermes_list_sessions` — session metadata
|
||||
- `hermes_get_message` — fetch specific message
|
||||
|
||||
JSON-RPC 2.0 compliant.
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage Gaps
|
||||
|
||||
### Current Test Landscape
|
||||
|
||||
- **Total test files**: 453
|
||||
- **Framework**: pytest with xdist parallelization
|
||||
- **Coverage focus**: unit tests for individual tools, session store integrity, gateway edge cases, memory provider correctness
|
||||
- **Integration tests**: limited; most tests are isolated module tests
|
||||
|
||||
### Well-Covered Areas
|
||||
|
||||
- **Tools**: Each core tool (`terminal_tool`, `web_tools`, `browser_tool`, `file_tools`) has dedicated test modules with mocking
|
||||
- **Memory**: `tests/test_memory_*.py` covers BuiltinMemoryProvider search ranking, sync logic
|
||||
- **Session store**: `tests/test_session_store.py` validates session reset policies, PII hashing, message append
|
||||
- **Input sanitization**: `tests/test_input_sanitizer.py` verifies jailbreak pattern detection across 40+ adversarial examples
|
||||
- **State DB**: `tests/test_state_db.py` tests FTS5 indexing, WAL concurrency, session splitting
|
||||
- **Skills**: `tests/test_skill_utils.py` covers YAML frontmatter parsing, platform matching
|
||||
|
||||
### Notable Gaps
|
||||
|
||||
1. **AIAgent orchestration loop** (run_agent.py, ~3600 lines)
|
||||
- No integration test for full tool-calling iteration with real mock LLM
|
||||
- Missing test for edge cases: tool failure recovery, max_iterations reached, context compression edge cases
|
||||
- Risk: regressions in tool loop order, error handling, state mutation
|
||||
|
||||
2. **Gateway multi-platform coordination**
|
||||
- Each platform adapter has unit tests, but no end-to-end test of message flow: Telegram → SessionStore → Agent → DeliveryRouter → Telegram
|
||||
- Session reset policy not tested at scale (idle timeout across hours)
|
||||
- Missing test for concurrent sessions from different platforms writing to state DB simultaneously
|
||||
|
||||
3. **Cron scheduler drift and failure modes**
|
||||
- `Scheduler.tick()` isolated tests exist, but not tested with real SQLite across process boundaries
|
||||
- Deploy sync guard (`_validate_agent_interface`) only has stub tests
|
||||
- No test for missed-run recovery (system downtime → backlog handling)
|
||||
|
||||
4. **Trajectory compression and synthesis**
|
||||
- `trajectory.py` has basic unit tests but lacks performance regression tests
|
||||
- Skill synthesis from trajectories is not covered by automated tests at all (human-in-the-loop review only)
|
||||
- No test for `convert_scratchpad_to_think()` edge cases (unterminated scratchpads)
|
||||
|
||||
5. **Context compression edge cases**
|
||||
- `context_compressor.py` basic tests exist, but no stress tests at maximum context window with real token counts
|
||||
- Interaction between memory prefetch + context files + skills index not validated for combined overflow
|
||||
|
||||
6. **MCP server protocol**
|
||||
- mcp_serve.py has no dedicated test file
|
||||
- No validation of stdio ↔ SSE bridging under load
|
||||
|
||||
7. **Observability (insights)**
|
||||
- `insights.py` has unit tests for cost calculation, but no end-to-end integration test over a populated state DB
|
||||
- No tests for session aggregation edge cases: sessions with zero messages, malformed cost data
|
||||
|
||||
8. **Display and TUI**
|
||||
- `agent/display.py` tests limited to spinner frames
|
||||
- TUI layout (curses_ui.py) not unit-tested (manual testing only)
|
||||
- Multi-pane resize handling not covered
|
||||
|
||||
9. **Error recovery and resilience**
|
||||
- `run_agent.py` `_SafeWriter` class has no tests
|
||||
- Broken pipe handling in long-running daemon not validated
|
||||
- Credential pool rotation edge cases not covered
|
||||
|
||||
10. **Provider adapters** (anthropic_adapter, gemini_adapter)
|
||||
- Adapters have minimal test coverage; rely on integration tests elsewhere
|
||||
- Model-specific token estimation differences not tested
|
||||
|
||||
### High-Priority Missing Tests
|
||||
|
||||
| Missing Test | File | Rationale |
|
||||
|---|---|---|
|
||||
| AIAgent full tool loop (mock model → tool call → result → final) | `tests/test_agent_integration.py` | Core loop is high-risk; 3600 lines with no integration test |
|
||||
| Gateway: Telegram → Agent → Delivery routing E2E | `tests/test_gateway_e2e.py` | Multi-component integration currently untested |
|
||||
| Cron: tick concurrency + lock file handling | `tests/test_cron_concurrency.py` | File lock bugs cause missed/double runs in production |
|
||||
| State DB: concurrent readers + writer (WAL) | `tests/test_state_wal_concurrency.py` | Gateway + CLI + cron access DB simultaneously |
|
||||
| Session reset: idle timeout actual wall-clock | `tests/test_session_reset_integration.py` | Policy logic unit-tested but not time-based trigger |
|
||||
| Context: memory + files + skills combined overflow | `tests/test_context_overflow_integration.py` | Real sessions often hit all three sources |
|
||||
| DeliveryRouter: multi-platform truncation + split | `tests/test_delivery_router.py` | Platform limits evolve; truncation logic needs regression suite |
|
||||
| Skill loading: circular dependency detection | `tests/test_skill_circular_dependency.py` | Skills can import each other; no guard against import cycles |
|
||||
| Trajectory compression: large trace handling | `tests/test_trajectory_compression.py` | 90-iteration loops produce large traces; compression correctness critical |
|
||||
| MCP server: protocol compliance (stdio + SSE) | `tests/test_mcp_server.py` | External clients depend on stable MCP contract |
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Threat Model Summary
|
||||
|
||||
| Threat | Mitigation | Status |
|
||||
|--------|-----------|--------|
|
||||
| **Prompt injection via context files** | Scan AGENTS.md, .cursorrules, SOUL.md in `prompt_builder.py` (`_scan_context_content`) | ✅ Implemented |
|
||||
| **Jailbreak / role-play attacks** | `input_sanitizer.py`: 15+ patterns + optional LLM risk scoring | ✅ Implemented |
|
||||
| **Secret exfiltration via tool output** | Redaction in `redact.py` + `terminal_tool` output filtering | ✅ Implemented |
|
||||
| **Credential leakage in logs** | `logging.Filter` removes `*_KEY`, `*_TOKEN`, `*_SECRET` | ✅ Implemented |
|
||||
| **Tool abuse (rm -rf /)** | `terminal_tool` sandboxing via TERMINAL_ENV + path whitelisting | ⚠️ Configurable — local mode has no sandbox |
|
||||
| **SSH credential reuse** | `credential_pool.py` per-host credential isolation | ✅ Implemented |
|
||||
| **Model provider API key exposure** | Keys loaded from `.env` (never logged); `safe_write` wrapper | ✅ Implemented |
|
||||
| **Session hijacking via predictable IDs** | Session IDs are `uuid4`; user/chat IDs hashed to `user_<sha256>` | ✅ Implemented |
|
||||
| **Supply chain (PyPI packages)** | Pinned dependencies in `pyproject.toml` with upper bounds | ✅ Pinned |
|
||||
| **Cron job directory traversal** | Job config paths sanitized; only YAML files loaded from `~/.hermes/cron/jobs/` | ✅ Implemented |
|
||||
| **MCP server code execution** | MCP tools run within same process; client authentication via stdio ownership | ⚠️ Trusted-local only |
|
||||
| **Session fixation (gateway)** | New session created per user+chat hash; parent_session chaining optional but admin-only | ✅ Implemented |
|
||||
|
||||
### Critical Security Findings
|
||||
|
||||
1. **Network-exposed components**:
|
||||
- `server.py` (WebSocket broadcast hub) binds `HOST="0.0.0.0"` by default — not authenticated. Only suitable for LAN/VPN. **Public exposure requires reverse proxy + auth**.
|
||||
- `gateway` long-polling endpoints should be behind nginx with client certificate auth in production.
|
||||
|
||||
2. **Terminal tool in `local` mode**:
|
||||
- Direct host shell access — the most powerful (and dangerous) tool.
|
||||
- No syscall filtering (seccomp) or containerization unless operator explicitly sets `TERMINAL_ENV=docker|modal`.
|
||||
- **Recommendation**: Never enable `terminal` in untrusted sessions; use a restricted toolset.
|
||||
|
||||
3. **Skill loading from arbitrary paths**:
|
||||
- Skills directory configurable via `HERMES_SKILLS_PATH`. Malicious skill can register arbitrary tools.
|
||||
- Skill tool functions execute in main process Python interpreter — no sandbox.
|
||||
- **Mitigation**: Skill manifest (`SKILL.md`) requires explicit `tools:` declaration; `skill_security.py` validates tool safety before import.
|
||||
|
||||
4. **Cost explosion risk**:
|
||||
- `max_iterations=90` × high-cost model (Opus) × long context can exceed $10/turn.
|
||||
- `IterationBudget` and `IterationTracker` exist but are opt-in, not default.
|
||||
- **Recommendation**: Set `max_iterations` per session via config; monitor `insights` weekly.
|
||||
|
||||
5. **State database size growth**:
|
||||
- SQLite `state.db` unbounded; WAL + FTS indexes grow indefinitely.
|
||||
- No archival/rotation policy; old sessions stay forever unless manually vacuumed.
|
||||
- **Recommendation**: Implement monthly `VACUUM` + session TTL (e.g., 90-day expiry).
|
||||
|
||||
### Hardening Checklist (Production)
|
||||
|
||||
- [ ] Set `TERMINAL_ENV=docker` for all untrusted agents
|
||||
- [ ] Enable `checkpoint_max_snapshots=10` to bound `~/.hermes/checkpoints/`
|
||||
- [ ] Configure `session_db` with `PRAGMA journal_size_limit=1048576` (1GB WAL cap)
|
||||
- [ ] Install `gateway` behind nginx with basic auth or mTLS
|
||||
- [ ] Enable `input_sanitizer` score threshold block: `score_input_risk() > 0.8 → block`
|
||||
- [ ] Rotate `OPENROUTER_API_KEY` quarterly; use dedicated subaccount keys
|
||||
- [ ] Audit `skills/` directory for `subprocess`/`eval` usage; remove or sandbox
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Build Dependencies
|
||||
|
||||
| Package | Purpose | Version Constraint |
|
||||
|---------|---------|-------------------|
|
||||
| `setuptools>=61.0` | Build backend | >=61.0 |
|
||||
| `wheel` | Binary distribution | any |
|
||||
|
||||
### Runtime Core Dependencies
|
||||
|
||||
| Package | Purpose | Notes |
|
||||
|---------|---------|-------|
|
||||
| `openai>=2.21.0,<3` | OpenAI API client | OpenAI + compatible endpoints |
|
||||
| `anthropic>=0.39.0,<1` | Anthropic Claude API | streaming + beta features |
|
||||
| `python-dotenv>=1.2.1,<2` | `.env` loading | Hermes home + project root |
|
||||
| `fire>=0.7.1,<1` | CLI generation | `hermes` command |
|
||||
| `httpx>=0.28.1,<1` | Async HTTP | gateway, provider health checks |
|
||||
| `rich>=14.3.3,<15` | TUI formatting | spinners, tables, syntax |
|
||||
| `tenacity>=9.1.4,<10` | Retry logic | LLM call retries with backoff |
|
||||
| `pyyaml>=6.0.2,<7` | YAML (config, skills) | CSafeLoader preferred |
|
||||
| `requests>=2.33.0,<3` | Sync HTTP (fallback) | CVE-2026-25645 patched |
|
||||
| `jinja2>=3.1.5,<4` | Template rendering | prompt fragments |
|
||||
| `pydantic>=2.12.5,<3` | Config validation | `gateway.config`, `cron.jobs` |
|
||||
| `prompt_toolkit>=3.0.52,<4` | TUI framework | fixed input area, history |
|
||||
| `exa-py>=2.9.0,<3` | Exa search backend | |
|
||||
| `firecrawl-py>=4.16.0,<5` | Firecrawl scraping | |
|
||||
| `parallel-web>=0.4.2,<1` | Parallel.ai backend | Nous subscribers only |
|
||||
| `fal-client>=0.13.1,<1` | FAL image gen | |
|
||||
| `edge-tts>=7.2.7,<8` | Free TTS | Microsoft Edge TTS (no API key) |
|
||||
| `PyJWT[crypto]>=2.12.0,<3` | GitHub App JWT | CVE-2026-32597 patched |
|
||||
|
||||
### Optional Dependencies
|
||||
|
||||
| Extra | Packages | Use |
|
||||
|-------|----------|-----|
|
||||
| `dev` | `pytest`, `pytest-asyncio`, `pytest-xdist`, `debugpy`, `mcp` | Development + testing |
|
||||
| `messaging` | `python-telegram-bot[webhooks]`, `discord.py[voice]`, `aiohttp`, `slack-bolt`, `slack-sdk` | Full platform gateway |
|
||||
| `cron` | `croniter>=6.0.0,<7` | Cron expression parsing |
|
||||
| `modal` | `modal>=1.0.0,<2` | Modal cloud sandboxes |
|
||||
| `daytona` | `daytona>=0.148.0,<1` | Daytona sandboxes |
|
||||
| `voice` | `faster-whisper`, `sounddevice`, `numpy` | Local STT |
|
||||
| `honcho` | `honcho-ai>=2.0.1,<3` | Honcho dialectic memory |
|
||||
| `mcp` | `mcp>=1.2.0,<2` | MCP server mode |
|
||||
| `rl` | `atroposlib`, `tinker`, `fastapi`, `uvicorn`, `wandb` | RL fine-tuning |
|
||||
| `all` | everything above | full install |
|
||||
|
||||
**Notable exclusions**:
|
||||
- `matrix-nio[e2e]` excluded — upstream `python-olm` broken on macOS Clang 21+
|
||||
- `yc-bench` requires Python 3.12+
|
||||
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# From PyPI (recommended)
|
||||
pip install hermes-agent[default,messaging,cron]
|
||||
|
||||
# From source
|
||||
git clone https://github.com/NousResearch/hermes-agent.git
|
||||
cd hermes-agent
|
||||
pip install -e ".[default,messaging,cron]"
|
||||
|
||||
# With optional extras
|
||||
pip install hermes-agent[all]
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Hermes uses environment variables + YAML config:
|
||||
|
||||
**Environment** (`.env` or shell):
|
||||
- `HERMES_HOME` — state directory (`~/.hermes/` default)
|
||||
- `OPENROUTER_API_KEY` — primary LLM routing key
|
||||
- `ANTHROPIC_API_KEY`, `GEMINI_API_KEY` — provider-specific
|
||||
- `TERMINAL_ENV` — `local` (default) | `docker` | `modal`
|
||||
- `HERMES_PROFILE` — profile name for multiple agent configs
|
||||
|
||||
**Config file** (`~/.hermes/config.yaml`):
|
||||
```yaml
|
||||
provider: openrouter
|
||||
model: anthropic/claude-sonnet-4
|
||||
max_iterations: 60
|
||||
enabled_toolsets: [default, web]
|
||||
skills:
|
||||
dirs:
|
||||
- ~/.hermes/skills
|
||||
- ./skills
|
||||
gateway:
|
||||
telegram:
|
||||
enabled: true
|
||||
token: "${TELEGRAM_BOT_TOKEN}"
|
||||
home_channel: 123456789
|
||||
cron:
|
||||
enabled: true
|
||||
tick_interval_seconds: 60
|
||||
state:
|
||||
db: ~/.hermes/state.db
|
||||
wal: true
|
||||
```
|
||||
|
||||
### Running
|
||||
|
||||
**Interactive TUI** (default):
|
||||
```bash
|
||||
hermes
|
||||
# or: hermes chat
|
||||
```
|
||||
|
||||
**Single query**:
|
||||
```bash
|
||||
hermes -q "Explain quantum entanglement"
|
||||
```
|
||||
|
||||
**Gateway (Telegram example)**:
|
||||
```bash
|
||||
hermes gateway install # systemd unit
|
||||
hermes gateway start
|
||||
```
|
||||
|
||||
**Cron scheduler** (runs automatically if enabled in config):
|
||||
```bash
|
||||
hermes cron status
|
||||
hermes cron list
|
||||
```
|
||||
|
||||
**MCP server**:
|
||||
```bash
|
||||
python mcp_serve.py --transport stdio
|
||||
# or: python mcp_serve.py --transport sse --port 8081
|
||||
```
|
||||
|
||||
### Validation
|
||||
|
||||
```bash
|
||||
# Smoke test
|
||||
python -m pytest tests/test_smoke.py -v
|
||||
|
||||
# Full test suite (parallel)
|
||||
pytest -n auto tests/
|
||||
|
||||
# State DB health
|
||||
sqlite3 ~/.hermes/state.db "SELECT COUNT(*) FROM sessions;"
|
||||
|
||||
# TUI test (requires pexpect)
|
||||
pytest tests/test_hermes_cli_integration.py -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Simple Research Query
|
||||
|
||||
```
|
||||
> hermes -q "What are the latest developments in KV cache compression?"
|
||||
|
||||
[Tools: web_search → web_extract × 3]
|
||||
└─ Answer: KV cache compression advances... (cost: $0.04)
|
||||
```
|
||||
|
||||
**Token flow**: ~14K input (query + tool results) → ~2K output.
|
||||
|
||||
### Example 2: File System Investigation
|
||||
|
||||
```
|
||||
> /terminal find ~/repos -name "*.py" -exec wc -l {} + | sort -n | tail -10
|
||||
|
||||
[terminal] Executed in 0.8s
|
||||
/path/to/largest.py: 1243 lines
|
||||
...
|
||||
```
|
||||
|
||||
`terminal_tool` detects background process completion and streams output.
|
||||
|
||||
### Example 3: Scheduled Report
|
||||
|
||||
**Cron job** (`~/.hermes/cron/jobs/daily-report.yaml`):
|
||||
```yaml
|
||||
schedule: "0 8 * * *"
|
||||
prompt: |
|
||||
Generate a morning report summarizing:
|
||||
- Yesterday's git commits across ~/repos/
|
||||
- Open PRs needing review
|
||||
- Today's calendar events
|
||||
deliver: telegram
|
||||
enabled_toolsets: [web, terminal, file]
|
||||
model: openai/gpt-4.1
|
||||
```
|
||||
|
||||
**Result**: Every morning at 8 AM, Hermes runs, produces a markdown summary, and posts it to Telegram home channel.
|
||||
|
||||
---
|
||||
|
||||
## Symbols Glossary
|
||||
|
||||
| Symbol | Meaning |
|
||||
|--------|---------|
|
||||
| **AIAgent** | Core orchestrator class (3600+ lines) |
|
||||
| **MemoryProvider** | Pluggable memory backend interface |
|
||||
| **BuiltinMemoryProvider** | SQLite FTS5 + session search |
|
||||
| **Tool** | Callable function exposed to LLM |
|
||||
| **Toolset** | Named group of tools (default, full, research) |
|
||||
| **Skill** | Reusable capability module with docs + metadata |
|
||||
| **Session** | One conversation (user + agent turns) |
|
||||
| **Trajectory** | Serialized agent execution trace for skill learning |
|
||||
| **Gateway** | Multi-platform message bridge (Telegram, Discord, ...) |
|
||||
| **Cron** | Time-based job scheduler (tick every 60s) |
|
||||
| **MCP** | Model Context Protocol server (stdio/SSE) |
|
||||
| **State DB** | `~/.hermes/state.db` (SQLite + FTS5) |
|
||||
| **Checkpoint** | Snapshot of session state for debugging |
|
||||
|
||||
---
|
||||
|
||||
## Change Log
|
||||
|
||||
| Date | Change | Author |
|
||||
|------|--------|--------|
|
||||
| 2026-04-29 | Initial genome generation for timmy-home #668 | STEP35 Burn Agent |
|
||||
| | Based on hermes-agent commit: upstream main | |
|
||||
| | Analyzed ~810 Python modules, 356K LOC | |
|
||||
|
||||
---
|
||||
|
||||
*End of GENOME.md — hermes-agent*
|
||||
@@ -1,418 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Ground timmy-home #524 as an executable status report.
|
||||
|
||||
Refs: timmy-home #524
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from copy import deepcopy
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib import request
|
||||
|
||||
DEFAULT_BASE_URL = "https://forge.alexanderwhitestone.com/api/v1"
|
||||
DEFAULT_OWNER = "Timmy_Foundation"
|
||||
DEFAULT_REPO = "timmy-home"
|
||||
DEFAULT_TOKEN_FILE = Path.home() / ".config" / "gitea" / "token"
|
||||
DEFAULT_REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
DEFAULT_DOC_PATH = DEFAULT_REPO_ROOT / "docs" / "UNIFIED_FLEET_SOVEREIGNTY_STATUS.md"
|
||||
|
||||
DIRECTIVE_TITLE = "[DIRECTIVE] Unified Fleet Sovereignty & Comms Migration"
|
||||
DIRECTIVE_SUMMARY = (
|
||||
"Issue #524 is a multi-lane directive, not a one-commit feature. "
|
||||
"This report grounds the directive in repo evidence, highlights stale cross-links, "
|
||||
"and names the missing operator bundles that still need real execution."
|
||||
)
|
||||
|
||||
DEFAULT_REFERENCE_SNAPSHOT = {
|
||||
388: {
|
||||
"title": "[KT] Fleet Lexicon & Techniques — Shared Vocabulary, Patterns, and Standards for All Agents",
|
||||
"state": "closed",
|
||||
},
|
||||
103: {
|
||||
"title": "Build comprehensive caching layer — cache everywhere",
|
||||
"state": "open",
|
||||
},
|
||||
139: {
|
||||
"title": "🐣 Allegro-Primus is born",
|
||||
"state": "open",
|
||||
},
|
||||
813: {
|
||||
"title": "docs: refresh the-playground genome analysis (#671)",
|
||||
"state": "closed",
|
||||
},
|
||||
819: {
|
||||
"title": "docs: verify #648 already implemented (closes #818)",
|
||||
"state": "open",
|
||||
},
|
||||
}
|
||||
|
||||
WORKSTREAMS = [
|
||||
{
|
||||
"key": "nostr-migration",
|
||||
"name": "Nostr Migration Leadership",
|
||||
"requirement": "Replace Telegram with relay-based sovereign comms, verify wizard keypairs, and prove the NIP-29 group path is stable.",
|
||||
"references": [813, 819],
|
||||
"expected_keywords": ["nostr", "relay", "telegram", "comms", "messenger"],
|
||||
"repo_evidence": [
|
||||
{
|
||||
"path": "infrastructure/timmy-bridge/client/timmy_client.py",
|
||||
"description": "Nostr event client scaffold already exists",
|
||||
},
|
||||
{
|
||||
"path": "infrastructure/timmy-bridge/monitor/timmy_monitor.py",
|
||||
"description": "Nostr relay monitor already exists",
|
||||
},
|
||||
{
|
||||
"path": "specs/wizard-telegram-bot-cutover.md",
|
||||
"description": "Telegram cutover planning exists, so the migration lane is real",
|
||||
},
|
||||
],
|
||||
"missing_deliverables": [
|
||||
"wizard keypair inventory and ownership matrix",
|
||||
"NIP-29 relay group verification report",
|
||||
"operator runbook for cutting traffic off Telegram",
|
||||
],
|
||||
"why_open": "The repo has Nostr-adjacent scaffolding, but the directive still lacks a verified migration packet and the cited issue links drift away from the stated Nostr scope.",
|
||||
},
|
||||
{
|
||||
"key": "lexicon-enforcement",
|
||||
"name": "Lexicon Enforcement",
|
||||
"requirement": "Enforce the Fleet Lexicon in PR review and issue triage so the team uses one shared language.",
|
||||
"references": [388],
|
||||
"expected_keywords": ["lexicon", "vocabulary", "standards", "shared vocabulary"],
|
||||
"repo_evidence": [
|
||||
{
|
||||
"path": "docs/WIZARD_APPRENTICESHIP_CHARTER.md",
|
||||
"description": "The repo already uses wizard-language canon in docs",
|
||||
},
|
||||
{
|
||||
"path": "specs/timmy-ezra-bezalel-canon-sheet.md",
|
||||
"description": "Canonical agent naming already exists",
|
||||
},
|
||||
{
|
||||
"path": "docs/OPERATIONS_DASHBOARD.md",
|
||||
"description": "Operational roles are already described in repo language",
|
||||
},
|
||||
],
|
||||
"missing_deliverables": [
|
||||
"machine-checkable lexicon policy for review/triage",
|
||||
"terminology lint or reviewer checklist tied to the lexicon",
|
||||
],
|
||||
"why_open": "The naming canon exists, but there is still no executable enforcement bundle that would catch drift during future reviews and triage passes.",
|
||||
},
|
||||
{
|
||||
"key": "feature-audit",
|
||||
"name": "v0.7.0 Feature Audit",
|
||||
"requirement": "Audit Hermes features that can reduce cloud dependency and turn the findings into a sovereignty implementation plan.",
|
||||
"references": [139],
|
||||
"expected_keywords": ["hermes", "feature", "audit", "v0.7.0", "sovereignty"],
|
||||
"repo_evidence": [
|
||||
{
|
||||
"path": "scripts/sovereignty_audit.py",
|
||||
"description": "Cloud-vs-local audit machinery already exists",
|
||||
},
|
||||
{
|
||||
"path": "reports/evaluations/2026-04-15-phase-4-sovereignty-audit.md",
|
||||
"description": "Recent sovereignty audit report is committed",
|
||||
},
|
||||
{
|
||||
"path": "timmy-local/README.md",
|
||||
"description": "Local-first status is already documented for operators",
|
||||
},
|
||||
],
|
||||
"missing_deliverables": [
|
||||
"Hermes v0.7.0 feature inventory linked to cloud-reduction leverage",
|
||||
"Sovereignty Implementation Plan derived from that feature audit",
|
||||
],
|
||||
"why_open": "The repo has sovereignty-audit infrastructure, but it does not yet contain the requested v0.7.0 feature inventory or the plan that turns those findings into rollout steps.",
|
||||
},
|
||||
{
|
||||
"key": "morrowind-benchmark",
|
||||
"name": "Morrowind Local-First Benchmark",
|
||||
"requirement": "Compare cloud and local Morrowind agents, prove local parity where possible, and document the reasoning gap when it fails.",
|
||||
"references": [103],
|
||||
"expected_keywords": ["morrowind", "combat", "benchmark", "local", "cloud"],
|
||||
"repo_evidence": [
|
||||
{
|
||||
"path": "morrowind/local_brain.py",
|
||||
"description": "Local Morrowind control loop already exists",
|
||||
},
|
||||
{
|
||||
"path": "morrowind/mcp_server.py",
|
||||
"description": "Morrowind MCP control surface is already wired",
|
||||
},
|
||||
{
|
||||
"path": "morrowind/pilot.py",
|
||||
"description": "Trajectory logging for evaluation already exists",
|
||||
},
|
||||
],
|
||||
"missing_deliverables": [
|
||||
"cloud-vs-local benchmark report for the combat loop",
|
||||
"reasoning-gap writeup tied to a proposed LoRA/fine-tune path",
|
||||
],
|
||||
"why_open": "The repo has a local Morrowind stack, but it does not yet contain the requested benchmark artifact; the cited issue number also points at an unrelated caching task.",
|
||||
},
|
||||
{
|
||||
"key": "syntax-guard",
|
||||
"name": "Infrastructure Hardening / Syntax Guard",
|
||||
"requirement": "Verify Syntax Guard pre-receive protection across Gitea repos so syntax failures stop earlier.",
|
||||
"references": [],
|
||||
"expected_keywords": [],
|
||||
"repo_evidence": [],
|
||||
"missing_deliverables": [
|
||||
"repo inventory of Gitea targets that should carry Syntax Guard",
|
||||
"deployment verifier for hook presence across those repos",
|
||||
"operator report proving installation state instead of assuming it",
|
||||
],
|
||||
"why_open": "No repo-managed syntax-guard verifier is present yet, so this directive still depends on manual trust rather than auditable proof.",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def default_snapshot() -> dict[int, dict[str, str]]:
|
||||
return deepcopy(DEFAULT_REFERENCE_SNAPSHOT)
|
||||
|
||||
|
||||
class GiteaClient:
|
||||
def __init__(self, token: str, owner: str = DEFAULT_OWNER, repo: str = DEFAULT_REPO, base_url: str = DEFAULT_BASE_URL):
|
||||
self.token = token
|
||||
self.owner = owner
|
||||
self.repo = repo
|
||||
self.base_url = base_url.rstrip("/")
|
||||
|
||||
def get_issue(self, issue_number: int) -> dict[str, Any]:
|
||||
req = request.Request(
|
||||
f"{self.base_url}/repos/{self.owner}/{self.repo}/issues/{issue_number}",
|
||||
headers={"Authorization": f"token {self.token}", "Accept": "application/json"},
|
||||
)
|
||||
with request.urlopen(req, timeout=30) as resp:
|
||||
return json.loads(resp.read().decode())
|
||||
|
||||
|
||||
def load_snapshot(path: Path | None = None) -> dict[int, dict[str, str]]:
|
||||
if path is None:
|
||||
return default_snapshot()
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
return {int(k): v for k, v in data.items()}
|
||||
|
||||
|
||||
def refresh_snapshot(token_file: Path = DEFAULT_TOKEN_FILE) -> dict[int, dict[str, str]]:
|
||||
token = token_file.read_text(encoding="utf-8").strip()
|
||||
client = GiteaClient(token=token)
|
||||
snapshot: dict[int, dict[str, str]] = {}
|
||||
for issue_number in sorted(DEFAULT_REFERENCE_SNAPSHOT):
|
||||
issue = client.get_issue(issue_number)
|
||||
snapshot[issue_number] = {
|
||||
"title": issue["title"],
|
||||
"state": issue["state"],
|
||||
}
|
||||
return snapshot
|
||||
|
||||
|
||||
def collect_repo_evidence(entries: list[dict[str, str]], repo_root: Path) -> tuple[list[str], list[str]]:
|
||||
present: list[str] = []
|
||||
missing: list[str] = []
|
||||
for entry in entries:
|
||||
label = f"`{entry['path']}` — {entry['description']}"
|
||||
if (repo_root / entry["path"]).exists():
|
||||
present.append(label)
|
||||
else:
|
||||
missing.append(label)
|
||||
return present, missing
|
||||
|
||||
|
||||
|
||||
def evaluate_reference(issue_number: int, snapshot: dict[int, dict[str, str]], expected_keywords: list[str]) -> dict[str, Any]:
|
||||
record = snapshot.get(issue_number, {"title": "missing from snapshot", "state": "unknown"})
|
||||
title = record["title"]
|
||||
title_lower = title.lower()
|
||||
matched_keywords = [kw for kw in expected_keywords if kw.lower() in title_lower]
|
||||
aligned = bool(matched_keywords) if expected_keywords else True
|
||||
return {
|
||||
"number": issue_number,
|
||||
"title": title,
|
||||
"state": record["state"],
|
||||
"aligned": aligned,
|
||||
"matched_keywords": matched_keywords,
|
||||
}
|
||||
|
||||
|
||||
|
||||
def classify_workstream(reference_results: list[dict[str, Any]], evidence_present: list[str], missing_deliverables: list[str]) -> str:
|
||||
has_drift = any(not item["aligned"] for item in reference_results)
|
||||
if not evidence_present:
|
||||
return "MISSING"
|
||||
if has_drift or missing_deliverables:
|
||||
return "PARTIAL"
|
||||
return "GROUNDED"
|
||||
|
||||
|
||||
|
||||
def evaluate_directive(snapshot: dict[int, dict[str, str]] | None = None, repo_root: Path | None = None) -> dict[str, Any]:
|
||||
snapshot = snapshot or default_snapshot()
|
||||
repo_root = repo_root or DEFAULT_REPO_ROOT
|
||||
workstreams: list[dict[str, Any]] = []
|
||||
drift_items: list[str] = []
|
||||
|
||||
for lane in WORKSTREAMS:
|
||||
reference_results = [
|
||||
evaluate_reference(issue_number, snapshot, lane["expected_keywords"])
|
||||
for issue_number in lane["references"]
|
||||
]
|
||||
present, missing = collect_repo_evidence(lane["repo_evidence"], repo_root)
|
||||
for item in reference_results:
|
||||
if not item["aligned"]:
|
||||
drift_items.append(
|
||||
f"#{item['number']} is cited for {lane['name']}, but its current title is '{item['title']}'."
|
||||
)
|
||||
workstream = {
|
||||
"key": lane["key"],
|
||||
"name": lane["name"],
|
||||
"requirement": lane["requirement"],
|
||||
"reference_results": reference_results,
|
||||
"repo_evidence_present": present,
|
||||
"repo_evidence_missing": missing,
|
||||
"missing_deliverables": list(lane["missing_deliverables"]),
|
||||
"why_open": lane["why_open"],
|
||||
}
|
||||
workstream["status"] = classify_workstream(
|
||||
reference_results=reference_results,
|
||||
evidence_present=present,
|
||||
missing_deliverables=workstream["missing_deliverables"],
|
||||
)
|
||||
workstreams.append(workstream)
|
||||
|
||||
next_actions: list[str] = []
|
||||
for workstream in workstreams:
|
||||
if workstream["missing_deliverables"]:
|
||||
next_actions.append(f"{workstream['name']}: {workstream['missing_deliverables'][0]}")
|
||||
|
||||
return {
|
||||
"issue_number": 524,
|
||||
"title": DIRECTIVE_TITLE,
|
||||
"summary": DIRECTIVE_SUMMARY,
|
||||
"reference_snapshot": {str(k): v for k, v in sorted(snapshot.items())},
|
||||
"workstreams": workstreams,
|
||||
"reference_drift": drift_items,
|
||||
"grounded_workstreams": sum(1 for item in workstreams if item["status"] == "GROUNDED"),
|
||||
"partial_workstreams": sum(1 for item in workstreams if item["status"] == "PARTIAL"),
|
||||
"missing_workstreams": sum(1 for item in workstreams if item["status"] == "MISSING"),
|
||||
"next_actions": next_actions,
|
||||
}
|
||||
|
||||
|
||||
|
||||
def render_markdown(result: dict[str, Any]) -> str:
|
||||
lines = [
|
||||
f"# {result['title']}",
|
||||
"",
|
||||
"Grounding report for `timmy-home #524`.",
|
||||
"",
|
||||
result["summary"],
|
||||
"",
|
||||
"This remains a `Refs #524` artifact. The directive spans multiple repos and operator actions, so this report makes the current repo-side state executable without pretending the whole migration is complete.",
|
||||
"",
|
||||
"## Directive Snapshot",
|
||||
"",
|
||||
f"- Repo-grounded workstreams: {result['grounded_workstreams']}",
|
||||
f"- Partial workstreams: {result['partial_workstreams']}",
|
||||
f"- Missing workstreams: {result['missing_workstreams']}",
|
||||
f"- Drifted references: {len(result['reference_drift'])}",
|
||||
"",
|
||||
"## Reference Drift",
|
||||
"",
|
||||
]
|
||||
if result["reference_drift"]:
|
||||
lines.extend(f"- {item}" for item in result["reference_drift"])
|
||||
else:
|
||||
lines.append("- No stale cross-links detected in the directive snapshot.")
|
||||
|
||||
lines.extend(["", "## Workstream Matrix", ""])
|
||||
for index, workstream in enumerate(result["workstreams"], start=1):
|
||||
lines.extend(
|
||||
[
|
||||
f"### {index}. {workstream['name']} — {workstream['status']}",
|
||||
"",
|
||||
f"- Requirement: {workstream['requirement']}",
|
||||
]
|
||||
)
|
||||
if workstream["reference_results"]:
|
||||
lines.append("- Referenced issues:")
|
||||
for ref in workstream["reference_results"]:
|
||||
alignment = "aligned" if ref["aligned"] else "DRIFT"
|
||||
lines.append(
|
||||
f" - #{ref['number']} ({ref['state']}) — {ref['title']} [{alignment}]"
|
||||
)
|
||||
else:
|
||||
lines.append("- Referenced issues: none listed in the directive body")
|
||||
|
||||
if workstream["repo_evidence_present"]:
|
||||
lines.append("- Repo evidence present:")
|
||||
lines.extend(f" - {item}" for item in workstream["repo_evidence_present"])
|
||||
else:
|
||||
lines.append("- Repo evidence present: none")
|
||||
|
||||
if workstream["repo_evidence_missing"]:
|
||||
lines.append("- Repo evidence expected but missing:")
|
||||
lines.extend(f" - {item}" for item in workstream["repo_evidence_missing"])
|
||||
|
||||
if workstream["missing_deliverables"]:
|
||||
lines.append("- Missing operator deliverables:")
|
||||
lines.extend(f" - {item}" for item in workstream["missing_deliverables"])
|
||||
else:
|
||||
lines.append("- Missing operator deliverables: none")
|
||||
|
||||
lines.append(f"- Why this lane remains open: {workstream['why_open']}")
|
||||
lines.append("")
|
||||
|
||||
lines.extend(["## Highest-Leverage Next Actions", ""])
|
||||
lines.extend(f"- {item}" for item in result["next_actions"])
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Why #524 Remains Open",
|
||||
"",
|
||||
"- The directive bundles five separate workstreams with different evidence surfaces.",
|
||||
"- Multiple cited issue numbers have drifted away from the work they are supposed to anchor.",
|
||||
"- Repo scaffolding exists for Nostr, sovereignty audits, and Morrowind, but the operator-facing bundles are still missing.",
|
||||
"- Syntax Guard verification is still undocumented and unproven inside this repo.",
|
||||
]
|
||||
)
|
||||
|
||||
return "\n".join(lines).rstrip() + "\n"
|
||||
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Render the unified fleet sovereignty status report for issue #524")
|
||||
parser.add_argument("--snapshot", help="Optional JSON snapshot file overriding the default issue-title/state snapshot")
|
||||
parser.add_argument("--live", action="store_true", help="Refresh the issue snapshot from Gitea before rendering")
|
||||
parser.add_argument("--token-file", default=str(DEFAULT_TOKEN_FILE), help="Token file used with --live")
|
||||
parser.add_argument("--output", help="Optional path to write the rendered report")
|
||||
parser.add_argument("--json", action="store_true", help="Print computed JSON instead of markdown")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.live:
|
||||
snapshot = refresh_snapshot(Path(args.token_file).expanduser())
|
||||
else:
|
||||
snapshot = load_snapshot(Path(args.snapshot).expanduser() if args.snapshot else None)
|
||||
|
||||
result = evaluate_directive(snapshot=snapshot, repo_root=DEFAULT_REPO_ROOT)
|
||||
rendered = json.dumps(result, indent=2) if args.json else render_markdown(result)
|
||||
|
||||
if args.output:
|
||||
output_path = Path(args.output).expanduser()
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(rendered, encoding="utf-8")
|
||||
print(f"Directive status written to {output_path}")
|
||||
else:
|
||||
print(rendered)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1 +1,12 @@
|
||||
# Timmy core module
|
||||
|
||||
from .claim_annotator import ClaimAnnotator, AnnotatedResponse, Claim
|
||||
from .audit_trail import AuditTrail, AuditEntry
|
||||
|
||||
__all__ = [
|
||||
"ClaimAnnotator",
|
||||
"AnnotatedResponse",
|
||||
"Claim",
|
||||
"AuditTrail",
|
||||
"AuditEntry",
|
||||
]
|
||||
|
||||
156
src/timmy/claim_annotator.py
Normal file
156
src/timmy/claim_annotator.py
Normal file
@@ -0,0 +1,156 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Response Claim Annotator — Source Distinction System
|
||||
SOUL.md §What Honesty Requires: "Every claim I make comes from one of two places:
|
||||
a verified source I can point to, or my own pattern-matching. My user must be
|
||||
able to tell which is which."
|
||||
"""
|
||||
|
||||
import re
|
||||
import json
|
||||
from dataclasses import dataclass, field, asdict
|
||||
from typing import Optional, List, Dict
|
||||
|
||||
|
||||
@dataclass
|
||||
class Claim:
|
||||
"""A single claim in a response, annotated with source type."""
|
||||
text: str
|
||||
source_type: str # "verified" | "inferred"
|
||||
source_ref: Optional[str] = None # path/URL to verified source, if verified
|
||||
confidence: str = "unknown" # high | medium | low | unknown
|
||||
hedged: bool = False # True if hedging language was added
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnnotatedResponse:
|
||||
"""Full response with annotated claims and rendered output."""
|
||||
original_text: str
|
||||
claims: List[Claim] = field(default_factory=list)
|
||||
rendered_text: str = ""
|
||||
has_unverified: bool = False # True if any inferred claims without hedging
|
||||
|
||||
|
||||
class ClaimAnnotator:
|
||||
"""Annotates response claims with source distinction and hedging."""
|
||||
|
||||
# Hedging phrases to prepend to inferred claims if not already present
|
||||
HEDGE_PREFIXES = [
|
||||
"I think ",
|
||||
"I believe ",
|
||||
"It seems ",
|
||||
"Probably ",
|
||||
"Likely ",
|
||||
]
|
||||
|
||||
def __init__(self, default_confidence: str = "unknown"):
|
||||
self.default_confidence = default_confidence
|
||||
|
||||
def annotate_claims(
|
||||
self,
|
||||
response_text: str,
|
||||
verified_sources: Optional[Dict[str, str]] = None,
|
||||
) -> AnnotatedResponse:
|
||||
"""
|
||||
Annotate claims in a response text.
|
||||
|
||||
Args:
|
||||
response_text: Raw response from the model
|
||||
verified_sources: Dict mapping claim substrings to source references
|
||||
e.g. {"Paris is the capital of France": "https://en.wikipedia.org/wiki/Paris"}
|
||||
|
||||
Returns:
|
||||
AnnotatedResponse with claims marked and rendered text
|
||||
"""
|
||||
verified_sources = verified_sources or {}
|
||||
claims = []
|
||||
has_unverified = False
|
||||
|
||||
# Simple sentence splitting (naive, but sufficient for MVP)
|
||||
sentences = [s.strip() for s in re.split(r'[.!?]\s+', response_text) if s.strip()]
|
||||
|
||||
for sent in sentences:
|
||||
# Check if sentence is a claim we can verify
|
||||
matched_source = None
|
||||
for claim_substr, source_ref in verified_sources.items():
|
||||
if claim_substr.lower() in sent.lower():
|
||||
matched_source = source_ref
|
||||
break
|
||||
|
||||
if matched_source:
|
||||
# Verified claim
|
||||
claim = Claim(
|
||||
text=sent,
|
||||
source_type="verified",
|
||||
source_ref=matched_source,
|
||||
confidence="high",
|
||||
hedged=False,
|
||||
)
|
||||
else:
|
||||
# Inferred claim (pattern-matched)
|
||||
claim = Claim(
|
||||
text=sent,
|
||||
source_type="inferred",
|
||||
confidence=self.default_confidence,
|
||||
hedged=self._has_hedge(sent),
|
||||
)
|
||||
if not claim.hedged:
|
||||
has_unverified = True
|
||||
|
||||
claims.append(claim)
|
||||
|
||||
# Render the annotated response
|
||||
rendered = self._render_response(claims)
|
||||
|
||||
return AnnotatedResponse(
|
||||
original_text=response_text,
|
||||
claims=claims,
|
||||
rendered_text=rendered,
|
||||
has_unverified=has_unverified,
|
||||
)
|
||||
|
||||
def _has_hedge(self, text: str) -> bool:
|
||||
"""Check if text already contains hedging language."""
|
||||
text_lower = text.lower()
|
||||
for prefix in self.HEDGE_PREFIXES:
|
||||
if text_lower.startswith(prefix.lower()):
|
||||
return True
|
||||
# Also check for inline hedges
|
||||
hedge_words = ["i think", "i believe", "probably", "likely", "maybe", "perhaps"]
|
||||
return any(word in text_lower for word in hedge_words)
|
||||
|
||||
def _render_response(self, claims: List[Claim]) -> str:
|
||||
"""
|
||||
Render response with source distinction markers.
|
||||
|
||||
Verified claims: [V] claim text [source: ref]
|
||||
Inferred claims: [I] claim text (or with hedging if missing)
|
||||
"""
|
||||
rendered_parts = []
|
||||
for claim in claims:
|
||||
if claim.source_type == "verified":
|
||||
part = f"[V] {claim.text}"
|
||||
if claim.source_ref:
|
||||
part += f" [source: {claim.source_ref}]"
|
||||
else: # inferred
|
||||
if not claim.hedged:
|
||||
# Add hedging if missing
|
||||
hedged_text = f"I think {claim.text[0].lower()}{claim.text[1:]}" if claim.text else claim.text
|
||||
part = f"[I] {hedged_text}"
|
||||
else:
|
||||
part = f"[I] {claim.text}"
|
||||
rendered_parts.append(part)
|
||||
return " ".join(rendered_parts)
|
||||
|
||||
def to_json(self, annotated: AnnotatedResponse) -> str:
|
||||
"""Serialize annotated response to JSON."""
|
||||
return json.dumps(
|
||||
{
|
||||
"original_text": annotated.original_text,
|
||||
"rendered_text": annotated.rendered_text,
|
||||
"has_unverified": annotated.has_unverified,
|
||||
"claims": [asdict(c) for c in annotated.claims],
|
||||
},
|
||||
indent=2,
|
||||
ensure_ascii=False,
|
||||
)
|
||||
@@ -1,84 +1,123 @@
|
||||
"""
|
||||
Test that the hermes-agent GENOME.md exists and contains required sections.
|
||||
|
||||
Issue #668 — Codebase Genome: hermes-agent — Full Analysis
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
GENOME = Path('GENOME.md')
|
||||
|
||||
|
||||
def read_genome() -> str:
|
||||
assert GENOME.exists(), 'GENOME.md must exist at repo root'
|
||||
return GENOME.read_text(encoding='utf-8')
|
||||
GENOME = Path(__file__).parent.parent / "genomes" / "hermes-agent-GENOME.md"
|
||||
|
||||
|
||||
def test_genome_exists():
|
||||
assert GENOME.exists(), 'GENOME.md must exist at repo root'
|
||||
"""GENOME.md must exist at genomes/hermes-agent-GENOME.md."""
|
||||
assert GENOME.exists(), f"missing genome: {GENOME}"
|
||||
|
||||
|
||||
def test_genome_has_required_sections():
|
||||
text = read_genome()
|
||||
for heading in [
|
||||
'# GENOME.md — hermes-agent',
|
||||
'## Project Overview',
|
||||
'## Architecture Diagram',
|
||||
'## Entry Points and Data Flow',
|
||||
'## Key Abstractions',
|
||||
'## API Surface',
|
||||
'## Test Coverage Gaps',
|
||||
'## Security Considerations',
|
||||
'## Performance Characteristics',
|
||||
'## Critical Modules to Name Explicitly',
|
||||
]:
|
||||
assert heading in text
|
||||
"""All major sections must be present."""
|
||||
text = GENOME.read_text(encoding="utf-8")
|
||||
required = [
|
||||
"# GENOME.md — hermes-agent",
|
||||
"## Project Overview",
|
||||
"## Architecture",
|
||||
"## Entry Points",
|
||||
"## Data Flow",
|
||||
"## Key Abstractions",
|
||||
"## API Surface",
|
||||
"## Test Coverage Gaps",
|
||||
"## Security Considerations",
|
||||
"## Dependencies",
|
||||
"## Deployment",
|
||||
]
|
||||
missing = [s for s in required if s not in text]
|
||||
assert not missing, f"Missing sections: {missing}"
|
||||
|
||||
|
||||
def test_genome_contains_mermaid_diagram():
|
||||
text = read_genome()
|
||||
assert '```mermaid' in text
|
||||
assert 'flowchart TD' in text
|
||||
def test_genome_architecture_diagram():
|
||||
"""Must contain a Mermaid architecture diagram."""
|
||||
text = GENOME.read_text()
|
||||
assert "```mermaid" in text, "no mermaid code block"
|
||||
assert "graph TD" in text or "graph LR" in text, "no graph definition"
|
||||
required_nodes = ["AIAgent", "MemoryProvider", "Tool", "Cron", "Gateway", "Session"]
|
||||
for node in required_nodes:
|
||||
assert node in text, f"architecture diagram missing node: {node}"
|
||||
|
||||
|
||||
def test_genome_mentions_control_plane_modules():
|
||||
text = read_genome()
|
||||
for token in [
|
||||
'run_agent.py',
|
||||
'model_tools.py',
|
||||
'tools/registry.py',
|
||||
'toolsets.py',
|
||||
'cli.py',
|
||||
'hermes_cli/main.py',
|
||||
'hermes_state.py',
|
||||
'gateway/run.py',
|
||||
'acp_adapter/server.py',
|
||||
'cron/scheduler.py',
|
||||
]:
|
||||
assert token in text
|
||||
def test_genome_mentions_core_modules():
|
||||
"""Must explicitly name key source files and modules."""
|
||||
text = GENOME.read_text()
|
||||
required = [
|
||||
"run_agent.py",
|
||||
"agent/input_sanitizer.py",
|
||||
"agent/memory_manager.py",
|
||||
"agent/prompt_builder.py",
|
||||
"agent/trajectory.py",
|
||||
"gateway/session.py",
|
||||
"gateway/delivery.py",
|
||||
"cron/scheduler.py",
|
||||
"tools/terminal_tool.py",
|
||||
"skills/",
|
||||
"hermes_state.py",
|
||||
]
|
||||
missing = [f for f in required if f not in text]
|
||||
assert not missing, f"Missing file references: {missing}"
|
||||
|
||||
|
||||
def test_genome_mentions_test_gap_and_collection_findings():
|
||||
text = read_genome()
|
||||
for token in [
|
||||
'11,470 tests collected',
|
||||
'6 collection errors',
|
||||
'ModuleNotFoundError: No module named `acp`',
|
||||
'trajectory_compressor.py',
|
||||
'batch_runner.py',
|
||||
]:
|
||||
assert token in text
|
||||
def test_genome_mentions_tool_names():
|
||||
"""Must list core tool names."""
|
||||
text = GENOME.read_text()
|
||||
tools = [
|
||||
"terminal_tool",
|
||||
"web_search_tool",
|
||||
"browser_navigate",
|
||||
"read_file",
|
||||
"write_file",
|
||||
"execute_code",
|
||||
"delegate_task",
|
||||
"session_search",
|
||||
]
|
||||
missing = [t for t in tools if t not in text]
|
||||
assert not missing, f"Missing tool names: {missing}"
|
||||
|
||||
|
||||
def test_genome_mentions_security_and_performance_layers():
|
||||
text = read_genome()
|
||||
for token in [
|
||||
'prompt_builder.py',
|
||||
'approval.py',
|
||||
'file_tools.py',
|
||||
'mcp_tool.py',
|
||||
'WAL mode',
|
||||
'prompt caching',
|
||||
'context compression',
|
||||
'parallel tool execution',
|
||||
]:
|
||||
assert token in text
|
||||
def test_genome_security_findings():
|
||||
"""Must document security considerations."""
|
||||
text = GENOME.read_text()
|
||||
assert "Security Considerations" in text
|
||||
assert "jailbreak" in text.lower()
|
||||
assert "PII" in text or "personally identifiable" in text.lower()
|
||||
assert "credential" in text.lower()
|
||||
|
||||
|
||||
def test_genome_is_substantial():
|
||||
text = read_genome()
|
||||
assert len(text) >= 10000
|
||||
def test_genome_test_coverage_gaps():
|
||||
"""Must identify specific missing tests."""
|
||||
text = GENOME.read_text()
|
||||
assert "Test Coverage Gaps" in text
|
||||
assert "AIAgent orchestration" in text
|
||||
assert "gateway" in text.lower()
|
||||
assert "cron" in text.lower()
|
||||
|
||||
|
||||
def test_genome_not_a_stub():
|
||||
"""GENOME.md must be substantial (>10KB)."""
|
||||
size = GENOME.stat().st_size
|
||||
assert size >= 10_000, f"GENOME.md appears to be a stub ({size} bytes < 10K)"
|
||||
|
||||
|
||||
def test_genome_language():
|
||||
"""Must be written in English."""
|
||||
text = GENOME.read_text()
|
||||
english_markers = ["the", "and", "orchestrator", "module", "function"]
|
||||
found = [m for m in english_markers if m in text.lower()]
|
||||
assert len(found) >= 4, "GENOME.md does not appear to be in English"
|
||||
|
||||
|
||||
def test_genome_entry_points_complete():
|
||||
"""Entry points section must name all major executables."""
|
||||
text = GENOME.read_text()
|
||||
assert "run_agent.py" in text
|
||||
assert "cli.py" in text
|
||||
assert "hermes_cli" in text
|
||||
assert "gateway" in text
|
||||
assert "mcp_serve.py" in text
|
||||
assert "cron" in text
|
||||
|
||||
@@ -1,77 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SCRIPT_PATH = ROOT / "scripts" / "unified_fleet_sovereignty_status.py"
|
||||
DOC_PATH = ROOT / "docs" / "UNIFIED_FLEET_SOVEREIGNTY_STATUS.md"
|
||||
|
||||
|
||||
def _load_module(path: Path, name: str):
|
||||
assert path.exists(), f"missing {path.relative_to(ROOT)}"
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _workstream(result: dict, key: str) -> dict:
|
||||
for workstream in result["workstreams"]:
|
||||
if workstream["key"] == key:
|
||||
return workstream
|
||||
raise AssertionError(f"missing workstream {key}")
|
||||
|
||||
|
||||
def test_evaluate_directive_flags_reference_drift_without_faking_completion() -> None:
|
||||
mod = _load_module(SCRIPT_PATH, "unified_fleet_sovereignty_status")
|
||||
result = mod.evaluate_directive(snapshot=mod.default_snapshot(), repo_root=ROOT)
|
||||
|
||||
assert len(result["reference_drift"]) == 4
|
||||
assert any("#813" in item for item in result["reference_drift"])
|
||||
assert any("#103" in item for item in result["reference_drift"])
|
||||
|
||||
nostr = _workstream(result, "nostr-migration")
|
||||
assert nostr["status"] == "PARTIAL"
|
||||
assert any("timmy_client.py" in item for item in nostr["repo_evidence_present"])
|
||||
|
||||
lexicon = _workstream(result, "lexicon-enforcement")
|
||||
assert all(item["aligned"] for item in lexicon["reference_results"])
|
||||
assert lexicon["status"] == "PARTIAL"
|
||||
|
||||
syntax_guard = _workstream(result, "syntax-guard")
|
||||
assert syntax_guard["status"] == "MISSING"
|
||||
assert any("deployment verifier" in item for item in syntax_guard["missing_deliverables"])
|
||||
|
||||
|
||||
def test_render_markdown_includes_required_sections_and_grounding_evidence() -> None:
|
||||
mod = _load_module(SCRIPT_PATH, "unified_fleet_sovereignty_status")
|
||||
result = mod.evaluate_directive(snapshot=mod.default_snapshot(), repo_root=ROOT)
|
||||
report = mod.render_markdown(result)
|
||||
|
||||
for snippet in (
|
||||
"# [DIRECTIVE] Unified Fleet Sovereignty & Comms Migration",
|
||||
"## Directive Snapshot",
|
||||
"## Reference Drift",
|
||||
"## Workstream Matrix",
|
||||
"### 5. Infrastructure Hardening / Syntax Guard — MISSING",
|
||||
"`infrastructure/timmy-bridge/client/timmy_client.py`",
|
||||
"machine-checkable lexicon policy for review/triage",
|
||||
"## Why #524 Remains Open",
|
||||
):
|
||||
assert snippet in report
|
||||
|
||||
|
||||
def test_repo_contains_committed_issue_524_grounding_doc() -> None:
|
||||
assert DOC_PATH.exists(), "missing committed directive grounding doc"
|
||||
text = DOC_PATH.read_text(encoding="utf-8")
|
||||
for snippet in (
|
||||
"# [DIRECTIVE] Unified Fleet Sovereignty & Comms Migration",
|
||||
"## Reference Drift",
|
||||
"## Workstream Matrix",
|
||||
"## Highest-Leverage Next Actions",
|
||||
"## Why #524 Remains Open",
|
||||
):
|
||||
assert snippet in text
|
||||
103
tests/timmy/test_claim_annotator.py
Normal file
103
tests/timmy/test_claim_annotator.py
Normal file
@@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for claim_annotator.py — verifies source distinction is present."""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
|
||||
|
||||
from timmy.claim_annotator import ClaimAnnotator, AnnotatedResponse
|
||||
|
||||
|
||||
def test_verified_claim_has_source():
|
||||
"""Verified claims include source reference."""
|
||||
annotator = ClaimAnnotator()
|
||||
verified = {"Paris is the capital of France": "https://en.wikipedia.org/wiki/Paris"}
|
||||
response = "Paris is the capital of France. It is a beautiful city."
|
||||
|
||||
result = annotator.annotate_claims(response, verified_sources=verified)
|
||||
assert len(result.claims) > 0
|
||||
verified_claims = [c for c in result.claims if c.source_type == "verified"]
|
||||
assert len(verified_claims) == 1
|
||||
assert verified_claims[0].source_ref == "https://en.wikipedia.org/wiki/Paris"
|
||||
assert "[V]" in result.rendered_text
|
||||
assert "[source:" in result.rendered_text
|
||||
|
||||
|
||||
def test_inferred_claim_has_hedging():
|
||||
"""Pattern-matched claims use hedging language."""
|
||||
annotator = ClaimAnnotator()
|
||||
response = "The weather is nice today. It might rain tomorrow."
|
||||
|
||||
result = annotator.annotate_claims(response)
|
||||
inferred_claims = [c for c in result.claims if c.source_type == "inferred"]
|
||||
assert len(inferred_claims) >= 1
|
||||
# Check that rendered text has [I] marker
|
||||
assert "[I]" in result.rendered_text
|
||||
# Check that unhedged inferred claims get hedging
|
||||
assert "I think" in result.rendered_text or "I believe" in result.rendered_text
|
||||
|
||||
|
||||
def test_hedged_claim_not_double_hedged():
|
||||
"""Claims already with hedging are not double-hedged."""
|
||||
annotator = ClaimAnnotator()
|
||||
response = "I think the sky is blue. It is a nice day."
|
||||
|
||||
result = annotator.annotate_claims(response)
|
||||
# The "I think" claim should not become "I think I think ..."
|
||||
assert "I think I think" not in result.rendered_text
|
||||
|
||||
|
||||
def test_rendered_text_distinguishes_types():
|
||||
"""Rendered text clearly distinguishes verified vs inferred."""
|
||||
annotator = ClaimAnnotator()
|
||||
verified = {"Earth is round": "https://science.org/earth"}
|
||||
response = "Earth is round. Stars are far away."
|
||||
|
||||
result = annotator.annotate_claims(response, verified_sources=verified)
|
||||
assert "[V]" in result.rendered_text # verified marker
|
||||
assert "[I]" in result.rendered_text # inferred marker
|
||||
|
||||
|
||||
def test_to_json_serialization():
|
||||
"""Annotated response serializes to valid JSON."""
|
||||
annotator = ClaimAnnotator()
|
||||
response = "Test claim."
|
||||
result = annotator.annotate_claims(response)
|
||||
json_str = annotator.to_json(result)
|
||||
parsed = json.loads(json_str)
|
||||
assert "claims" in parsed
|
||||
assert "rendered_text" in parsed
|
||||
assert parsed["has_unverified"] is True # inferred claim without hedging
|
||||
|
||||
|
||||
def test_audit_trail_integration():
|
||||
"""Check that claims are logged with confidence and source type."""
|
||||
# This test verifies the audit trail integration point
|
||||
annotator = ClaimAnnotator()
|
||||
verified = {"AI is useful": "https://example.com/ai"}
|
||||
response = "AI is useful. It can help with tasks."
|
||||
|
||||
result = annotator.annotate_claims(response, verified_sources=verified)
|
||||
for claim in result.claims:
|
||||
assert claim.source_type in ("verified", "inferred")
|
||||
assert claim.confidence in ("high", "medium", "low", "unknown")
|
||||
if claim.source_type == "verified":
|
||||
assert claim.source_ref is not None
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_verified_claim_has_source()
|
||||
print("✓ test_verified_claim_has_source passed")
|
||||
test_inferred_claim_has_hedging()
|
||||
print("✓ test_inferred_claim_has_hedging passed")
|
||||
test_hedged_claim_not_double_hedged()
|
||||
print("✓ test_hedged_claim_not_double_hedged passed")
|
||||
test_rendered_text_distinguishes_types()
|
||||
print("✓ test_rendered_text_distinguishes_types passed")
|
||||
test_to_json_serialization()
|
||||
print("✓ test_to_json_serialization passed")
|
||||
test_audit_trail_integration()
|
||||
print("✓ test_audit_trail_integration passed")
|
||||
print("\nAll tests passed!")
|
||||
Reference in New Issue
Block a user