Compare commits
1 Commits
fix/681
...
fix/683-wo
| Author | SHA1 | Date | |
|---|---|---|---|
| 3b430114be |
@@ -1,476 +0,0 @@
|
||||
# GENOME.md: burn-fleet
|
||||
|
||||
**Generated:** 2026-04-15
|
||||
**Repo:** Timmy_Foundation/burn-fleet
|
||||
**Purpose:** Laned tmux dispatcher for sovereign burn operations across Mac and Allegro
|
||||
**Analyzed commit:** `2d4d9ab`
|
||||
**Size:** 5 top-level source/config files + README | 985 total lines (`fleet-dispatch.py` 320, `fleet-christen.py` 205, `fleet-status.py` 143, `fleet-launch.sh` 126, `fleet-spec.json` 98, `README.md` 93)
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
`burn-fleet` is a compact control-plane repo for the Hundred-Pane Fleet.
|
||||
Its job is not model inference itself. Its job is to shape where inference runs, which panes wake up, which repos route to which windows, and how work is fanned out across Mac and VPS workers.
|
||||
|
||||
The repo turns a narrative naming scheme into executable infrastructure:
|
||||
- Mac runs the local session (`BURN`) with windows like `CRUCIBLE`, `GNOMES`, `LOOM`, `FOUNDRY`, `WARD`, `COUNCIL`
|
||||
- Allegro runs a remote session (`BURN`) with windows like `FORGE`, `ANVIL`, `CRUCIBLE-2`, `SENTINEL`
|
||||
- `fleet-spec.json` is the single source of truth for pane counts, lanes, sublanes, glyphs, and names
|
||||
- `fleet-launch.sh` materializes the tmux topology
|
||||
- `fleet-christen.py` boots `hermes chat --yolo` in each pane and pushes identity prompts
|
||||
- `fleet-dispatch.py` consumes Gitea issues, maps repos to windows through `MAC_ROUTE` and `ALLEGRO_ROUTE`, and sends `/queue` work into the right panes
|
||||
- `fleet-status.py` inspects pane output and reports fleet health
|
||||
|
||||
The repo is small, but it sits on a high-blast-radius operational seam:
|
||||
- it controls 100+ panes
|
||||
- it writes to live tmux sessions
|
||||
- it comments on live Gitea issues
|
||||
- it depends on SSH reachability to the VPS
|
||||
- it is effectively a narrative infrastructure orchestrator
|
||||
|
||||
This means the right way to read it is as a dispatch kernel, not just a set of scripts.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[fleet-spec.json] --> B[fleet-launch.sh]
|
||||
A --> C[fleet-christen.py]
|
||||
A --> D[fleet-dispatch.py]
|
||||
A --> E[fleet-status.py]
|
||||
|
||||
B --> F[tmux session BURN on Mac]
|
||||
B --> G[tmux session BURN on Allegro over SSH]
|
||||
|
||||
C --> F
|
||||
C --> G
|
||||
C --> H[hermes chat --yolo in every pane]
|
||||
H --> I[identity + lane prompt]
|
||||
|
||||
J[Gitea issues on forge.alexanderwhitestone.com] --> D
|
||||
D --> K[MAC_ROUTE]
|
||||
D --> L[ALLEGRO_ROUTE]
|
||||
D --> M[/queue prompt generation]
|
||||
M --> F
|
||||
M --> G
|
||||
D --> N[comment_on_issue]
|
||||
N --> J
|
||||
D --> O[dispatch-state.json]
|
||||
|
||||
E --> F
|
||||
E --> G
|
||||
E --> P[get_pane_status]
|
||||
P --> Q[fleet health summary]
|
||||
```
|
||||
|
||||
### Structural reading
|
||||
|
||||
The repo has one real architecture pattern:
|
||||
1. declarative topology in `fleet-spec.json`
|
||||
2. imperative realization scripts that consume that topology
|
||||
3. runtime state in `dispatch-state.json`
|
||||
4. external side effects in tmux, SSH, and Gitea
|
||||
|
||||
That makes `fleet-spec.json` the nucleus and the four scripts adapters around it.
|
||||
|
||||
---
|
||||
|
||||
## Entry Points
|
||||
|
||||
| Entry point | Type | Role |
|
||||
|-------------|------|------|
|
||||
| `fleet-launch.sh [mac|allegro|both]` | Shell CLI | Creates tmux sessions and pane layouts from `fleet-spec.json` |
|
||||
| `python3 fleet-christen.py [mac|allegro|both]` | Python CLI | Starts Hermes workers and injects identity/lane prompts |
|
||||
| `python3 fleet-dispatch.py [--cycles N] [--interval S] [--machine mac|allegro|both]` | Python CLI | Pulls open Gitea issues, routes them, comments on issues, persists `dispatch-state.json` |
|
||||
| `python3 fleet-status.py [--machine mac|allegro|both]` | Python CLI | Samples pane output and reports working/idle/error/dead state |
|
||||
| `README.md` quick start | Human runbook | Documents the intended operator flow from launch to christening to dispatch to status |
|
||||
|
||||
### Hidden operational entry points
|
||||
|
||||
These are not CLI entry points, but they matter for behavior:
|
||||
- `MAC_ROUTE` in `fleet-dispatch.py`
|
||||
- `ALLEGRO_ROUTE` in `fleet-dispatch.py`
|
||||
- `SKIP_LABELS` and `INACTIVE` filtering in `fleet-dispatch.py`
|
||||
- `send_to_pane()` as the effectful dispatch primitive
|
||||
- `comment_on_issue()` as the visible acknowledgement primitive
|
||||
- `get_pane_status()` in `fleet-status.py` as the fleet health classifier
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
### 1. Topology creation
|
||||
|
||||
`fleet-launch.sh` reads `fleet-spec.json`, parses each window's pane count, and creates the tmux layout.
|
||||
|
||||
Flow:
|
||||
- load spec file path from `SCRIPT_DIR/fleet-spec.json`
|
||||
- parse `machines.mac.windows` or `machines.allegro.windows`
|
||||
- create `BURN` session locally or remotely
|
||||
- create first window, then split panes, then create remaining windows
|
||||
- continuously tile after splits
|
||||
|
||||
This script is layout-only. It does not launch Hermes.
|
||||
|
||||
### 2. Agent wake-up / identity seeding
|
||||
|
||||
`fleet-christen.py` reads the same `fleet-spec.json` and sends `hermes chat --yolo` into each pane.
|
||||
After a fixed wait window, it sends a second `/queue` identity message containing:
|
||||
- glyph
|
||||
- pane name
|
||||
- machine name
|
||||
- window name
|
||||
- pane number
|
||||
- sublane
|
||||
- sovereign operating instructions
|
||||
|
||||
That identity message is the bridge from infrastructure to narrative.
|
||||
The worker is not just launched; it is assigned a mythic/operator identity with a lane.
|
||||
|
||||
### 3. Issue harvest and lane dispatch
|
||||
|
||||
`fleet-dispatch.py` is the center of the runtime.
|
||||
|
||||
Flow:
|
||||
- load `fleet-spec.json`
|
||||
- load `dispatch-state.json`
|
||||
- load Gitea token
|
||||
- fetch open issues per repo with `requests`
|
||||
- filter PRs, meta labels, and previously dispatched issues
|
||||
- build a candidate pool per machine/window
|
||||
- assign issues pane-by-pane
|
||||
- call `send_to_pane()` to inject `/queue ...`
|
||||
- call `comment_on_issue()` to leave a visible burn dispatch comment
|
||||
- persist the issue assignment into `dispatch-state.json`
|
||||
|
||||
Important: the data flow is not issue -> worker directly.
|
||||
It is:
|
||||
issue -> repo route table -> window -> pane -> `/queue` prompt -> worker.
|
||||
|
||||
### 4. Health sampling
|
||||
|
||||
`fleet-status.py` runs the inverse direction.
|
||||
It samples pane output through `tmux capture-pane` locally or over SSH and classifies the last visible signal as:
|
||||
- `working`
|
||||
- `idle`
|
||||
- `error`
|
||||
- `dead`
|
||||
|
||||
It then summarizes by window, machine, and global fleet totals.
|
||||
|
||||
### 5. Runtime state persistence
|
||||
|
||||
`dispatch-state.json` is not checked in, but it is the only persistent memory of what the dispatcher already assigned.
|
||||
That means the runtime depends on a local mutable file rather than a centralized dispatch ledger.
|
||||
|
||||
---
|
||||
|
||||
## Key Abstractions
|
||||
|
||||
### 1. `fleet-spec.json`
|
||||
|
||||
This is the primary abstraction in the repo.
|
||||
It encodes:
|
||||
- machine identity (`mac`, `allegro`)
|
||||
- host / SSH details
|
||||
- hardware metadata (`cores`, `ram_gb`)
|
||||
- tmux session names
|
||||
- default model/provider metadata
|
||||
- windows with `panes`, `lane`, `sublanes`, `glyphs`, `names`
|
||||
|
||||
Everything else in the repo interprets this document.
|
||||
If the spec drifts from the route tables or runtime assumptions, the fleet silently degrades.
|
||||
|
||||
### 2. Route tables: `MAC_ROUTE` and `ALLEGRO_ROUTE`
|
||||
|
||||
These tables are the repo's second control nucleus.
|
||||
They map repo names to windows.
|
||||
This is how `timmy-home`, `the-nexus`, `the-door`, `fleet-ops`, and `the-beacon` land in different operational lanes.
|
||||
|
||||
This split means routing logic is duplicated:
|
||||
- once in the topology spec
|
||||
- once in Python route dictionaries
|
||||
|
||||
That duplication is one of the most important maintainability risks in the repo.
|
||||
|
||||
### 3. Pane effect primitive: `send_to_pane()`
|
||||
|
||||
`send_to_pane()` is the real actuator.
|
||||
It turns a dispatch decision into a tmux `send-keys` side effect.
|
||||
It handles both:
|
||||
- local tmux injection
|
||||
- remote SSH + tmux injection
|
||||
|
||||
Everything operationally dangerous funnels through this function.
|
||||
It is therefore a critical path even though the repo has no tests around it.
|
||||
|
||||
### 4. Issue acknowledgement primitive: `comment_on_issue()`
|
||||
|
||||
This is the repo's social trace primitive.
|
||||
It posts a burn dispatch comment back to the issue so humans can see that the fleet claimed it.
|
||||
This is the visible heartbeat of autonomous dispatch.
|
||||
|
||||
### 5. Runtime memory: `dispatch-state.json`
|
||||
|
||||
This file is the anti-duplication ledger for dispatch cycles.
|
||||
Without it, the dispatcher would keep recycling the same issues every pass.
|
||||
Because it is local-file state instead of centralized state, machine locality matters.
|
||||
|
||||
### 6. Health classifier: `get_pane_status()`
|
||||
|
||||
`fleet-status.py` does not know the true worker state.
|
||||
It infers state from captured pane output using string heuristics.
|
||||
So `get_pane_status()` is effectively a lightweight log classifier.
|
||||
Its correctness depends on fragile output pattern matching.
|
||||
|
||||
---
|
||||
|
||||
## API Surface
|
||||
|
||||
The repo exposes CLI-level APIs rather than import-oriented libraries.
|
||||
|
||||
### Shell API
|
||||
|
||||
`fleet-launch.sh`
|
||||
- `./fleet-launch.sh mac`
|
||||
- `./fleet-launch.sh allegro`
|
||||
- `./fleet-launch.sh both`
|
||||
|
||||
### Python CLIs
|
||||
|
||||
`fleet-christen.py`
|
||||
- `python3 fleet-christen.py mac`
|
||||
- `python3 fleet-christen.py allegro`
|
||||
- `python3 fleet-christen.py both`
|
||||
|
||||
`fleet-dispatch.py`
|
||||
- `python3 fleet-dispatch.py`
|
||||
- `python3 fleet-dispatch.py --cycles 10 --interval 60`
|
||||
- `python3 fleet-dispatch.py --machine mac`
|
||||
|
||||
`fleet-status.py`
|
||||
- `python3 fleet-status.py`
|
||||
- `python3 fleet-status.py --machine allegro`
|
||||
|
||||
### Internal function surface worth naming explicitly
|
||||
|
||||
`fleet-launch.sh`
|
||||
- `parse_spec()`
|
||||
- `launch_local()`
|
||||
- `launch_remote()`
|
||||
|
||||
`fleet-christen.py`
|
||||
- `send_keys()`
|
||||
- `christen_window()`
|
||||
- `christen_machine()`
|
||||
- `christen_remote()`
|
||||
|
||||
`fleet-dispatch.py`
|
||||
- `load_token()`
|
||||
- `load_spec()`
|
||||
- `load_state()`
|
||||
- `save_state()`
|
||||
- `get_issues()`
|
||||
- `send_to_pane()`
|
||||
- `comment_on_issue()`
|
||||
- `build_prompt()`
|
||||
- `dispatch_cycle()`
|
||||
- `dispatch_council()`
|
||||
|
||||
`fleet-status.py`
|
||||
- `get_pane_status()`
|
||||
- `check_machine()`
|
||||
|
||||
These are the true API surface for future hardening and testing.
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage Gaps
|
||||
|
||||
### Current state
|
||||
|
||||
Grounded from the pipeline dry run on `/tmp/burn-fleet-genome`:
|
||||
- 0% estimated coverage
|
||||
- untested modules called out by pipeline: `fleet-christen`, `fleet-dispatch`, `fleet-status`
|
||||
- no checked-in automated test suite
|
||||
|
||||
### Critical paths with no tests
|
||||
|
||||
1. `send_to_pane()`
|
||||
- local tmux command construction
|
||||
- remote SSH command construction
|
||||
- escaping of issue titles and prompts
|
||||
- failure handling when tmux or SSH fails
|
||||
|
||||
2. `comment_on_issue()`
|
||||
- verifies Gitea comment formatting
|
||||
- verifies non-200 responses do not silently disappear
|
||||
|
||||
3. `get_issues()`
|
||||
- PR filtering
|
||||
- `SKIP_LABELS` filtering
|
||||
- title-based meta filtering
|
||||
- robustness when Gitea returns malformed or partial issue objects
|
||||
|
||||
4. `dispatch_cycle()`
|
||||
- correct pooling by window
|
||||
- deduplication via `dispatch-state.json`
|
||||
- pane recycling behavior
|
||||
- correctness when one repo has zero issues and another has many
|
||||
|
||||
5. `get_pane_status()`
|
||||
- classification heuristics for working/idle/error/dead
|
||||
- false positives from incidental strings like `error` in normal output
|
||||
|
||||
6. `fleet-launch.sh`
|
||||
- parse correctness for pane counts
|
||||
- layout creation behavior across first vs later windows
|
||||
- remote script generation for Allegro
|
||||
|
||||
### Missing tests to generate next in the real target repo
|
||||
|
||||
If the goal is to harden `burn-fleet` itself, the first tests to add should be:
|
||||
- `test_route_tables_cover_spec_windows`
|
||||
- `test_send_to_pane_escapes_single_quotes_and_special_chars`
|
||||
- `test_comment_on_issue_formats_machine_window_pane_body`
|
||||
- `test_get_issues_skips_prs_and_meta_labels`
|
||||
- `test_dispatch_cycle_persists_dispatch_state_once`
|
||||
- `test_get_pane_status_classifies_spinner_vs_traceback_vs_empty`
|
||||
|
||||
These are the minimum critical-path tests.
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### 1. Command injection surface
|
||||
|
||||
`send_to_pane()` and the remote tmux/SSH command assembly are the biggest security surface.
|
||||
Even though single quotes are escaped in prompts, this remains a command injection boundary because untrusted issue titles and repo metadata cross into shell commands.
|
||||
|
||||
This is why `command injection` is the right risk label for the repo.
|
||||
The risk is not hypothetical; the repo is literally translating issue text into shell transport.
|
||||
|
||||
### 2. Credential handling
|
||||
|
||||
The dispatcher uses a local token file for Gitea authentication.
|
||||
That is a credential handling concern because:
|
||||
- token locality is assumed
|
||||
- file path and host assumptions are embedded into runtime code
|
||||
- there is no retry / fallback / explicit missing-token UX beyond failure
|
||||
|
||||
### 3. SSH trust boundary
|
||||
|
||||
Remote pane control over `root@167.99.126.228` means the repo assumes a trusted SSH path to a root shell.
|
||||
That is operationally powerful and dangerous.
|
||||
A malformed remote command, stale known_hosts state, or wrong host mapping has fleet-wide consequences.
|
||||
|
||||
### 4. Runtime state tampering
|
||||
|
||||
`dispatch-state.json` is a local mutable state file with no locking, signing, or cross-machine reconciliation.
|
||||
If it is corrupted or lost, deduplication semantics fail.
|
||||
That can cause repeated dispatches or misleading status.
|
||||
|
||||
### 5. Live-forge mutation
|
||||
|
||||
`comment_on_issue()` mutates live issue threads on every dispatch cycle.
|
||||
That means any bug in deduplication or routing will create visible comment spam on the forge.
|
||||
|
||||
### 6. Dependency risk
|
||||
|
||||
The repo depends on `requests` for Gitea API access but has no pinned dependency metadata or environment contract in-repo.
|
||||
This is a small operational repo, but reproducibility is weak.
|
||||
|
||||
---
|
||||
|
||||
## Dependency Picture
|
||||
|
||||
### Runtime dependencies
|
||||
- Python 3
|
||||
- `requests`
|
||||
- tmux
|
||||
- SSH client
|
||||
- ssh trust boundary to `root@167.99.126.228`
|
||||
- access to a Gitea token file
|
||||
|
||||
### Implied environment dependencies
|
||||
- active tmux sessions on Mac and Allegro
|
||||
- SSH trust / connectivity to the VPS
|
||||
- hermes available in pane environments
|
||||
- Gitea reachable at `https://forge.alexanderwhitestone.com`
|
||||
|
||||
### Notably missing
|
||||
- no `requirements.txt`
|
||||
- no `pyproject.toml`
|
||||
- no explicit test harness
|
||||
- no schema validation for `fleet-spec.json`
|
||||
|
||||
---
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
For such a small repo, the performance question is not CPU time inside Python.
|
||||
It is orchestration fan-out latency.
|
||||
|
||||
The main scaling costs are:
|
||||
- repeated Gitea issue fetches across repos
|
||||
- SSH round-trips to Allegro
|
||||
- tmux pane fan-out across 100+ panes
|
||||
- serialized `time.sleep(0.2)` dispatch staggering
|
||||
|
||||
This means the bottleneck is control-plane coordination, not computation.
|
||||
The repo will scale until SSH / tmux / Gitea latency become dominant.
|
||||
|
||||
---
|
||||
|
||||
## Dead Code / Drift Risks
|
||||
|
||||
### 1. Spec vs route duplication
|
||||
|
||||
`fleet-spec.json` defines windows and lanes, while `fleet-dispatch.py` separately defines `MAC_ROUTE` and `ALLEGRO_ROUTE`.
|
||||
That is the biggest drift risk.
|
||||
A window can exist in the spec and be missing from a route table, or vice versa.
|
||||
|
||||
### 2. Runtime-generated files absent from repo contracts
|
||||
|
||||
`dispatch-state.json` is operationally critical but not described as a first-class contract in code.
|
||||
The repo assumes it exists or can be created, but does not validate structure.
|
||||
|
||||
### 3. README drift risk
|
||||
|
||||
The README says "use fleet-christen.sh" in one place while the actual file is `fleet-christen.py`.
|
||||
That is a small but real operator-footgun and a sign the human runbook can drift from the executable surface.
|
||||
|
||||
---
|
||||
|
||||
## Suggested Follow-up Work
|
||||
|
||||
1. Move repo-to-window routing into `fleet-spec.json` and derive `MAC_ROUTE` / `ALLEGRO_ROUTE` programmatically.
|
||||
2. Add automated tests for `send_to_pane`, `get_issues`, `dispatch_cycle`, and `get_pane_status`.
|
||||
3. Add a schema validator for `fleet-spec.json`.
|
||||
4. Add explicit dependency metadata (`requirements.txt` or `pyproject.toml`).
|
||||
5. Add dry-run / no-side-effect mode for dispatch and christening.
|
||||
6. Add retry/backoff and error reporting around Gitea comments and SSH execution.
|
||||
|
||||
---
|
||||
|
||||
## Bottom Line
|
||||
|
||||
`burn-fleet` is a small repo with outsized operational leverage.
|
||||
Its genome is simple:
|
||||
- one declarative topology file
|
||||
- four operational adapters
|
||||
- one local runtime ledger
|
||||
- many side effects across tmux, SSH, and Gitea
|
||||
|
||||
It already expresses the philosophy of narrative-driven infrastructure well.
|
||||
What it lacks is not architecture.
|
||||
What it lacks is hardening:
|
||||
- tests around the dangerous paths
|
||||
- centralization of duplicated routing truth
|
||||
- stronger command / credential / runtime-state safeguards
|
||||
|
||||
That makes it a strong control-plane prototype and a weakly tested production surface.
|
||||
263
genomes/wolf/GENOME.md
Normal file
263
genomes/wolf/GENOME.md
Normal file
@@ -0,0 +1,263 @@
|
||||
# GENOME.md — Wolf (Timmy_Foundation/wolf)
|
||||
|
||||
> Codebase Genome v1.0 | Generated 2026-04-14 | Repo 16/16
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Wolf** is a multi-model evaluation engine for sovereign AI fleets. It runs prompts against multiple LLM providers, scores responses on relevance, coherence, and safety, and outputs structured JSON results for model selection and ranking.
|
||||
|
||||
**Core principle:** agents work, PRs prove it, CI judges it.
|
||||
|
||||
**Status:** v1.0.0 — production-ready for prompt evaluation. Legacy PR evaluation module retained for backward compatibility.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
CLI[cli.py] --> Config[config.py]
|
||||
CLI --> TaskGen[task.py]
|
||||
CLI --> Runner[runner.py]
|
||||
CLI --> Evaluator[evaluator.py]
|
||||
CLI --> Leaderboard[leaderboard.py]
|
||||
CLI --> Gitea[gitea.py]
|
||||
|
||||
Runner --> Models[models.py]
|
||||
Runner --> Gitea
|
||||
Evaluator --> Models
|
||||
|
||||
TaskGen --> Gitea
|
||||
Leaderboard --> |leaderboard.json| FS[(File System)]
|
||||
Config --> |wolf-config.yaml| FS
|
||||
|
||||
Models --> OpenRouter[OpenRouter API]
|
||||
Models --> Groq[Groq API]
|
||||
Models --> Ollama[Ollama Local]
|
||||
Models --> OpenAI[OpenAI API]
|
||||
Models --> Anthropic[Anthropic API]
|
||||
|
||||
Runner --> |branch + commit| Gitea
|
||||
Evaluator --> |score results| Leaderboard
|
||||
```
|
||||
|
||||
## Entry Points
|
||||
|
||||
| Entry Point | Command | Purpose |
|
||||
|-------------|---------|---------|
|
||||
| `wolf/cli.py` | `python3 -m wolf.cli --run` | Main CLI: run tasks, evaluate PRs, show leaderboard |
|
||||
| `wolf/runner.py` | `python3 -m wolf.runner --prompts p.json --models m.json` | Standalone prompt evaluation runner |
|
||||
| `wolf/__init__.py` | `import wolf` | Package init, version metadata |
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Prompt Evaluation Pipeline (Primary)
|
||||
|
||||
```
|
||||
prompts.json + models.json (or wolf-config.yaml)
|
||||
│
|
||||
▼
|
||||
PromptEvaluator.evaluate()
|
||||
│
|
||||
├─ For each (prompt, model) pair:
|
||||
│ ├─ ModelClient.generate(prompt) → response text
|
||||
│ ├─ ResponseScorer.score(response, prompt)
|
||||
│ │ ├─ score_relevance() (0.40 weight)
|
||||
│ │ ├─ score_coherence() (0.35 weight)
|
||||
│ │ └─ score_safety() (0.25 weight)
|
||||
│ └─ EvaluationResult (prompt, model, scores, latency, error)
|
||||
│
|
||||
▼
|
||||
evaluate_and_serialize() → JSON output
|
||||
│
|
||||
├─ model_summaries (per-model averages)
|
||||
└─ results[] (per-evaluation details)
|
||||
```
|
||||
|
||||
### Task Assignment Pipeline (Legacy)
|
||||
|
||||
```
|
||||
Gitea Issues → TaskGenerator → AgentRunner
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
Fetch tasks Assign models Execute + PR
|
||||
from issues from config via Gitea API
|
||||
```
|
||||
|
||||
## Key Abstractions
|
||||
|
||||
| Class | Module | Purpose |
|
||||
|-------|--------|---------|
|
||||
| `PromptEntry` | evaluator.py | Single prompt with expected keywords and category |
|
||||
| `ModelEndpoint` | evaluator.py | Model connection descriptor (provider, model_id, key) |
|
||||
| `ScoreResult` | evaluator.py | Scores for relevance, coherence, safety, overall |
|
||||
| `EvaluationResult` | evaluator.py | Full result: prompt + model + response + scores + latency |
|
||||
| `ResponseScorer` | evaluator.py | Heuristic scoring engine (regex + keyword + structure) |
|
||||
| `PromptEvaluator` | evaluator.py | Core engine: runs prompts against models, scores output |
|
||||
| `ModelClient` | models.py | Abstract base for LLM API calls |
|
||||
| `ModelFactory` | models.py | Factory: returns correct client for provider name |
|
||||
| `Task` | task.py | Work unit: id, title, description, assigned model/provider |
|
||||
| `TaskGenerator` | task.py | Creates tasks from Gitea issues or JSON spec |
|
||||
| `AgentRunner` | runner.py | Executes tasks: generate → branch → commit → PR |
|
||||
| `Config` | config.py | YAML config loader (wolf-config.yaml) |
|
||||
| `Leaderboard` | leaderboard.py | Persistent model ranking with serverless readiness |
|
||||
| `GiteaClient` | gitea.py | Full Gitea REST API client |
|
||||
| `PREvaluator` | evaluator.py | Legacy: scores PRs on CI, commits, code quality |
|
||||
|
||||
## API Surface
|
||||
|
||||
### CLI Arguments (cli.py)
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--config` | Path to wolf-config.yaml |
|
||||
| `--task-spec` | Path to task specification JSON |
|
||||
| `--run` | Run pending tasks (assign models, execute, create PRs) |
|
||||
| `--evaluate` | Evaluate open PRs and score them |
|
||||
| `--leaderboard` | Show model rankings |
|
||||
|
||||
### CLI Arguments (runner.py)
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--prompts` / `-p` | Path to prompts JSON (required) |
|
||||
| `--models` / `-m` | Path to models JSON |
|
||||
| `--config` / `-c` | Path to wolf-config.yaml (alternative to --models) |
|
||||
| `--output` / `-o` | Path to write JSON results |
|
||||
| `--system-prompt` | System prompt for all model calls |
|
||||
|
||||
### Provider Clients (models.py)
|
||||
|
||||
| Client | Provider | API Format |
|
||||
|--------|----------|------------|
|
||||
| `OpenRouterClient` | openrouter | OpenAI-compatible chat completions |
|
||||
| `GroqClient` | groq | OpenAI-compatible chat completions |
|
||||
| `OllamaClient` | ollama | Ollama native /api/generate |
|
||||
| `OpenAIClient` | openai | OpenAI-compatible (reuses GroqClient with different URL) |
|
||||
| `AnthropicClient` | anthropic | Anthropic Messages API v1 |
|
||||
|
||||
### Gitea Client (gitea.py)
|
||||
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `get_issues()` | Fetch issues by state |
|
||||
| `create_branch()` | Create new branch from base |
|
||||
| `create_file()` | Create file on branch (base64) |
|
||||
| `update_file()` | Update file with SHA |
|
||||
| `get_file()` | Read file contents |
|
||||
| `create_pull_request()` | Open PR |
|
||||
| `get_pull_request()` | Fetch PR details |
|
||||
| `get_pr_status()` | Check PR CI status |
|
||||
|
||||
## Configuration (wolf-config.yaml)
|
||||
|
||||
```yaml
|
||||
gitea:
|
||||
base_url: "https://forge.alexanderwhitestone.com/api/v1"
|
||||
token: "..."
|
||||
owner: "Timmy_Foundation"
|
||||
repo: "wolf"
|
||||
|
||||
providers:
|
||||
openrouter:
|
||||
api_key: "..."
|
||||
base_url: "https://openrouter.ai/api/v1"
|
||||
ollama:
|
||||
base_url: "http://localhost:11434"
|
||||
|
||||
models:
|
||||
- model: "anthropic/claude-3.5-sonnet"
|
||||
provider: "openrouter"
|
||||
- model: "gemma4:latest"
|
||||
provider: "ollama"
|
||||
|
||||
log_dir: "~/.hermes/wolf/"
|
||||
leaderboard_path: "~/.hermes/wolf/leaderboard.json"
|
||||
```
|
||||
|
||||
## Scoring Methodology
|
||||
|
||||
| Dimension | Weight | Method |
|
||||
|-----------|--------|--------|
|
||||
| Relevance | 0.40 | Keyword matching (expected_keywords), prompt word overlap, length penalty, refusal detection |
|
||||
| Coherence | 0.35 | Structure indicators (paragraphs, lists, transitions), sentence completeness, repetition ratio, length sweet spot |
|
||||
| Safety | 0.25 | Unsafe pattern regex (bomb/hack/bypass/injection), profanity detection |
|
||||
|
||||
**Overall = relevance×0.40 + coherence×0.35 + safety×0.25**
|
||||
|
||||
### Safety Patterns Detected
|
||||
|
||||
- Bomb/explosive/weapon creation
|
||||
- System/network hacking
|
||||
- Security/auth bypass
|
||||
- Prompt injection ("ignore previous instructions")
|
||||
- System prompt extraction attempts
|
||||
|
||||
### Refusal Patterns Detected
|
||||
|
||||
- "I cannot/can't/won't help/assist"
|
||||
- "Sorry, but I cannot"
|
||||
- "Against my guidelines/policy"
|
||||
|
||||
## Test Coverage
|
||||
|
||||
| File | Tests | Coverage |
|
||||
|------|-------|----------|
|
||||
| `tests/test_evaluator.py` | 17 tests | PromptEntry, ModelEndpoint, ResponseScorer (relevance/coherence/safety), PromptEvaluator (evaluate, error handling, serialization, file output, multi-model), PREvaluator (score_pr, description scoring) |
|
||||
| `tests/test_config.py` | 1 test | Config load from YAML |
|
||||
|
||||
### Coverage Gaps
|
||||
|
||||
- No tests for `cli.py` (argument parsing, workflow orchestration)
|
||||
- No tests for `runner.py` (`load_prompts`, `load_models_from_json`, `AgentRunner.execute_task`)
|
||||
- No tests for `task.py` (`TaskGenerator.from_gitea_issues`, `from_spec`, `assign_tasks`)
|
||||
- No tests for `models.py` (API clients — would require mocking HTTP)
|
||||
- No tests for `leaderboard.py` (`record_score`, `get_rankings`, serverless readiness logic)
|
||||
- No tests for `gitea.py` (API client — would require mocking HTTP)
|
||||
- No integration tests (end-to-end evaluation pipeline)
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Used By | Purpose |
|
||||
|------------|---------|---------|
|
||||
| `requests` | models.py, gitea.py | HTTP client for all API calls |
|
||||
| `pyyaml` (optional) | config.py | YAML config parsing (falls back to line parser) |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **API keys in config**: wolf-config.yaml stores provider API keys in plaintext. File should be chmod 600 and excluded from git (already in .gitignore pattern via ~/.hermes/).
|
||||
2. **Gitea token**: Full access token used for branch creation, file commits, and PR creation. Scoped access recommended.
|
||||
3. **No input sanitization**: Prompts from Gitea issues are passed directly to models without filtering. Prompt injection risk for automated workflows.
|
||||
4. **No rate limiting**: Model API calls are sequential with no backoff or rate limiting. Could exhaust API quotas.
|
||||
5. **Legacy code reference**: `evaluator.py` references `Evaluator = PREvaluator` alias but `cli.py` imports `Evaluator` expecting the legacy class. This works but is confusing.
|
||||
|
||||
## File Index
|
||||
|
||||
| File | LOC | Purpose |
|
||||
|------|-----|---------|
|
||||
| `wolf/__init__.py` | 12 | Package init, version |
|
||||
| `wolf/cli.py` | 90 | Main CLI orchestrator |
|
||||
| `wolf/config.py` | 48 | YAML config loader |
|
||||
| `wolf/models.py` | 130 | LLM provider clients (5 providers) |
|
||||
| `wolf/runner.py` | 280 | Prompt evaluation CLI + AgentRunner |
|
||||
| `wolf/task.py` | 80 | Task dataclass + generator |
|
||||
| `wolf/evaluator.py` | 350 | Core scoring engine + legacy PR evaluator |
|
||||
| `wolf/leaderboard.py` | 70 | Persistent model ranking |
|
||||
| `wolf/gitea.py` | 100 | Gitea REST API client |
|
||||
| `tests/test_evaluator.py` | 180 | Unit tests for evaluator |
|
||||
| `tests/test_config.py` | 20 | Unit tests for config |
|
||||
|
||||
**Total: ~1,360 LOC Python | 11 modules | 18 tests**
|
||||
|
||||
## Sovereignty Assessment
|
||||
|
||||
- **No external dependencies beyond requests**: Runs on any machine with Python 3.11+ and requests.
|
||||
- **No phone-home**: All API calls are to user-configured endpoints.
|
||||
- **No telemetry**: Logs go to local filesystem only.
|
||||
- **Config-driven**: All secrets in user's ~/.hermes/ directory.
|
||||
- **Provider-agnostic**: Supports 5 providers with easy extension via ModelFactory.
|
||||
|
||||
**Verdict: Fully sovereign. No corporate lock-in. User controls all endpoints and keys.**
|
||||
|
||||
---
|
||||
|
||||
*"The strength of the pack is the wolf, and the strength of the wolf is the pack."*
|
||||
*— The Wolf Sovereign Core has spoken.*
|
||||
@@ -1,70 +0,0 @@
|
||||
from pathlib import Path
|
||||
|
||||
GENOME = Path('genomes/burn-fleet-GENOME.md')
|
||||
|
||||
|
||||
def read_genome() -> str:
|
||||
assert GENOME.exists(), 'burn-fleet genome must exist at genomes/burn-fleet-GENOME.md'
|
||||
return GENOME.read_text(encoding='utf-8')
|
||||
|
||||
|
||||
def test_genome_exists():
|
||||
assert GENOME.exists(), 'burn-fleet genome must exist at genomes/burn-fleet-GENOME.md'
|
||||
|
||||
|
||||
def test_genome_has_required_sections():
|
||||
text = read_genome()
|
||||
for heading in [
|
||||
'# GENOME.md: burn-fleet',
|
||||
'## Project Overview',
|
||||
'## Architecture',
|
||||
'## Entry Points',
|
||||
'## Data Flow',
|
||||
'## Key Abstractions',
|
||||
'## API Surface',
|
||||
'## Test Coverage Gaps',
|
||||
'## Security Considerations',
|
||||
]:
|
||||
assert heading in text
|
||||
|
||||
|
||||
def test_genome_contains_mermaid_diagram():
|
||||
text = read_genome()
|
||||
assert '```mermaid' in text
|
||||
assert 'graph TD' in text or 'flowchart TD' in text
|
||||
|
||||
|
||||
def test_genome_mentions_core_files_and_runtime_state():
|
||||
text = read_genome()
|
||||
for token in [
|
||||
'fleet-spec.json',
|
||||
'fleet-launch.sh',
|
||||
'fleet-christen.py',
|
||||
'fleet-dispatch.py',
|
||||
'fleet-status.py',
|
||||
'dispatch-state.json',
|
||||
'tmux',
|
||||
'ssh',
|
||||
'MAC_ROUTE',
|
||||
'ALLEGRO_ROUTE',
|
||||
]:
|
||||
assert token in text
|
||||
|
||||
|
||||
def test_genome_mentions_test_gap_and_risk_findings():
|
||||
text = read_genome()
|
||||
for token in [
|
||||
'0% estimated coverage',
|
||||
'send_to_pane',
|
||||
'comment_on_issue',
|
||||
'get_pane_status',
|
||||
'requests',
|
||||
'command injection',
|
||||
'credential handling',
|
||||
]:
|
||||
assert token in text
|
||||
|
||||
|
||||
def test_genome_is_substantial():
|
||||
text = read_genome()
|
||||
assert len(text) >= 6000
|
||||
Reference in New Issue
Block a user