Compare commits
1 Commits
sprint/iss
...
fix/667
| Author | SHA1 | Date | |
|---|---|---|---|
|
840214c8c0 |
@@ -1,320 +1,263 @@
|
||||
# GENOME.md — wolf
|
||||
# GENOME.md — Wolf (Timmy_Foundation/wolf)
|
||||
|
||||
*Generated: 2026-04-14T19:10:00Z | Branch: main | Commit: 02767d8*
|
||||
> Codebase Genome v1.0 | Generated 2026-04-14 | Repo 16/16
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Wolf** is a sovereign multi-model evaluation engine. It runs prompts against multiple LLM providers (OpenAI, Anthropic, Groq, Ollama, OpenRouter), scores responses on relevance, coherence, and safety, and outputs structured JSON results for model selection and fleet deployment decisions.
|
||||
**Wolf** is a multi-model evaluation engine for sovereign AI fleets. It runs prompts against multiple LLM providers, scores responses on relevance, coherence, and safety, and outputs structured JSON results for model selection and ranking.
|
||||
|
||||
**Two operational modes:**
|
||||
1. **Prompt Evaluation (v1.0)** — Standalone prompt-vs-model benchmarking via `python -m wolf.runner`
|
||||
2. **Legacy PR Scoring** — Gitea PR evaluation pipeline via `wolf.cli` (task generation, agent execution, leaderboard)
|
||||
**Core principle:** agents work, PRs prove it, CI judges it.
|
||||
|
||||
**Tagline:** "Multi-model evaluation — agents work, PRs prove it, leaders get endpoints."
|
||||
|
||||
---
|
||||
**Status:** v1.0.0 — production-ready for prompt evaluation. Legacy PR evaluation module retained for backward compatibility.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph CLI["CLI Entry Points"]
|
||||
A1["python -m wolf.runner\n(pure evaluation)"]
|
||||
A2["python -m wolf.cli\n(task pipeline)"]
|
||||
end
|
||||
graph TD
|
||||
CLI[cli.py] --> Config[config.py]
|
||||
CLI --> TaskGen[task.py]
|
||||
CLI --> Runner[runner.py]
|
||||
CLI --> Evaluator[evaluator.py]
|
||||
CLI --> Leaderboard[leaderboard.py]
|
||||
CLI --> Gitea[gitea.py]
|
||||
|
||||
subgraph Core["Core Engine"]
|
||||
B1["PromptEvaluator\n(evaluator.py)"]
|
||||
B2["ResponseScorer\n(evaluator.py)"]
|
||||
B3["AgentRunner\n(runner.py)"]
|
||||
B4["TaskGenerator\n(task.py)"]
|
||||
end
|
||||
Runner --> Models[models.py]
|
||||
Runner --> Gitea
|
||||
Evaluator --> Models
|
||||
|
||||
subgraph Providers["Model Providers"]
|
||||
C1["OpenRouterClient"]
|
||||
C2["GroqClient"]
|
||||
C3["OllamaClient"]
|
||||
C4["AnthropicClient"]
|
||||
C5["OpenAIClient\n(GroqClient w/ custom URL)"]
|
||||
end
|
||||
TaskGen --> Gitea
|
||||
Leaderboard --> |leaderboard.json| FS[(File System)]
|
||||
Config --> |wolf-config.yaml| FS
|
||||
|
||||
subgraph Infrastructure["Infrastructure"]
|
||||
D1["GiteaClient\n(gitea.py)"]
|
||||
D2["Config\n(config.py)"]
|
||||
D3["Leaderboard\n(leaderboard.py)"]
|
||||
D4["wolf-config.yaml"]
|
||||
end
|
||||
Models --> OpenRouter[OpenRouter API]
|
||||
Models --> Groq[Groq API]
|
||||
Models --> Ollama[Ollama Local]
|
||||
Models --> OpenAI[OpenAI API]
|
||||
Models --> Anthropic[Anthropic API]
|
||||
|
||||
subgraph Output["Output"]
|
||||
E1["JSON results file"]
|
||||
E2["stdout summary table"]
|
||||
E3["Gitea PRs"]
|
||||
E4["Leaderboard scores"]
|
||||
end
|
||||
|
||||
A1 --> B1
|
||||
A2 --> B4 --> B3
|
||||
B1 --> B2
|
||||
B1 --> C1 & C2 & C3 & C4 & C5
|
||||
B3 --> C1 & C2 & C3 & C4 & C5
|
||||
B3 --> D1
|
||||
A2 --> D1 & D2 & D3
|
||||
B1 --> E1 & E2
|
||||
B3 --> E3
|
||||
D3 --> E4
|
||||
D2 --> D4
|
||||
|
||||
style A1 fill:#4a9eff,color:#fff
|
||||
style A2 fill:#4a9eff,color:#fff
|
||||
style B1 fill:#ff6b6b,color:#fff
|
||||
style B2 fill:#ff6b6b,color:#fff
|
||||
Runner --> |branch + commit| Gitea
|
||||
Evaluator --> |score results| Leaderboard
|
||||
```
|
||||
|
||||
### Data Flow — Prompt Evaluation Mode
|
||||
|
||||
```
|
||||
prompts.json + models.json/wolf-config.yaml
|
||||
→ load_prompts() / load_models_from_json()
|
||||
→ PromptEvaluator.evaluate()
|
||||
→ for each (prompt, model):
|
||||
→ ModelFactory.get_client(provider) → ModelClient.generate()
|
||||
→ ResponseScorer.score(response, prompt)
|
||||
→ score_relevance() — keyword matching, length, refusal detection
|
||||
→ score_coherence() — structure, readability, repetition
|
||||
→ score_safety() — harmful content patterns, profanity
|
||||
→ overall = relevance*0.40 + coherence*0.35 + safety*0.25
|
||||
→ evaluate_and_serialize() → JSON dict
|
||||
→ run(output_path) → write JSON + print_summary()
|
||||
```
|
||||
|
||||
### Data Flow — Legacy Task Pipeline Mode
|
||||
|
||||
```
|
||||
wolf-config.yaml
|
||||
→ GiteaClient.get_issues(owner, repo)
|
||||
→ TaskGenerator.from_gitea_issues()
|
||||
→ TaskGenerator.assign_tasks(tasks, models)
|
||||
→ for each task:
|
||||
→ AgentRunner.execute_task(task)
|
||||
→ ModelClient.generate(prompt)
|
||||
→ GiteaClient.create_branch()
|
||||
→ GiteaClient.create_file(wolf-outputs/{id}.md)
|
||||
→ GiteaClient.create_pull_request()
|
||||
→ Leaderboard.record_score()
|
||||
→ Leaderboard.get_rankings()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Entry Points
|
||||
|
||||
| Entry Point | Module | Purpose |
|
||||
|-------------|--------|---------|
|
||||
| `python -m wolf.runner` | `runner.py` | Pure prompt-vs-model evaluation. Primary v1.0 interface. |
|
||||
| `python -m wolf.cli` | `cli.py` | Full task pipeline: fetch issues → run models → create PRs → leaderboard. |
|
||||
| Entry Point | Command | Purpose |
|
||||
|-------------|---------|---------|
|
||||
| `wolf/cli.py` | `python3 -m wolf.cli --run` | Main CLI: run tasks, evaluate PRs, show leaderboard |
|
||||
| `wolf/runner.py` | `python3 -m wolf.runner --prompts p.json --models m.json` | Standalone prompt evaluation runner |
|
||||
| `wolf/__init__.py` | `import wolf` | Package init, version metadata |
|
||||
|
||||
### runner.py CLI Flags
|
||||
## Data Flow
|
||||
|
||||
| Flag | Required | Description |
|
||||
|------|----------|-------------|
|
||||
| `--prompts / -p` | Yes | Path to prompts JSON file |
|
||||
| `--models / -m` | No* | Path to models JSON file |
|
||||
| `--config / -c` | No* | Path to wolf-config.yaml (alternative to --models) |
|
||||
| `--output / -o` | No | Path to write JSON results |
|
||||
| `--system-prompt` | No | System prompt (default: "You are a helpful assistant.") |
|
||||
### Prompt Evaluation Pipeline (Primary)
|
||||
|
||||
*Either --models or --config is required.
|
||||
```
|
||||
prompts.json + models.json (or wolf-config.yaml)
|
||||
│
|
||||
▼
|
||||
PromptEvaluator.evaluate()
|
||||
│
|
||||
├─ For each (prompt, model) pair:
|
||||
│ ├─ ModelClient.generate(prompt) → response text
|
||||
│ ├─ ResponseScorer.score(response, prompt)
|
||||
│ │ ├─ score_relevance() (0.40 weight)
|
||||
│ │ ├─ score_coherence() (0.35 weight)
|
||||
│ │ └─ score_safety() (0.25 weight)
|
||||
│ └─ EvaluationResult (prompt, model, scores, latency, error)
|
||||
│
|
||||
▼
|
||||
evaluate_and_serialize() → JSON output
|
||||
│
|
||||
├─ model_summaries (per-model averages)
|
||||
└─ results[] (per-evaluation details)
|
||||
```
|
||||
|
||||
### cli.py CLI Flags
|
||||
### Task Assignment Pipeline (Legacy)
|
||||
|
||||
```
|
||||
Gitea Issues → TaskGenerator → AgentRunner
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
Fetch tasks Assign models Execute + PR
|
||||
from issues from config via Gitea API
|
||||
```
|
||||
|
||||
## Key Abstractions
|
||||
|
||||
| Class | Module | Purpose |
|
||||
|-------|--------|---------|
|
||||
| `PromptEntry` | evaluator.py | Single prompt with expected keywords and category |
|
||||
| `ModelEndpoint` | evaluator.py | Model connection descriptor (provider, model_id, key) |
|
||||
| `ScoreResult` | evaluator.py | Scores for relevance, coherence, safety, overall |
|
||||
| `EvaluationResult` | evaluator.py | Full result: prompt + model + response + scores + latency |
|
||||
| `ResponseScorer` | evaluator.py | Heuristic scoring engine (regex + keyword + structure) |
|
||||
| `PromptEvaluator` | evaluator.py | Core engine: runs prompts against models, scores output |
|
||||
| `ModelClient` | models.py | Abstract base for LLM API calls |
|
||||
| `ModelFactory` | models.py | Factory: returns correct client for provider name |
|
||||
| `Task` | task.py | Work unit: id, title, description, assigned model/provider |
|
||||
| `TaskGenerator` | task.py | Creates tasks from Gitea issues or JSON spec |
|
||||
| `AgentRunner` | runner.py | Executes tasks: generate → branch → commit → PR |
|
||||
| `Config` | config.py | YAML config loader (wolf-config.yaml) |
|
||||
| `Leaderboard` | leaderboard.py | Persistent model ranking with serverless readiness |
|
||||
| `GiteaClient` | gitea.py | Full Gitea REST API client |
|
||||
| `PREvaluator` | evaluator.py | Legacy: scores PRs on CI, commits, code quality |
|
||||
|
||||
## API Surface
|
||||
|
||||
### CLI Arguments (cli.py)
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--config` | Path to wolf-config.yaml |
|
||||
| `--task-spec` | Path to task specification JSON |
|
||||
| `--run` | Run pending tasks (fetch issues → generate → PR) |
|
||||
| `--evaluate` | Evaluate open PRs (legacy scoring) |
|
||||
| `--run` | Run pending tasks (assign models, execute, create PRs) |
|
||||
| `--evaluate` | Evaluate open PRs and score them |
|
||||
| `--leaderboard` | Show model rankings |
|
||||
|
||||
---
|
||||
### CLI Arguments (runner.py)
|
||||
|
||||
## Key Abstractions
|
||||
|
||||
### Dataclasses (evaluator.py)
|
||||
|
||||
| Class | Fields | Purpose |
|
||||
|-------|--------|---------|
|
||||
| `PromptEntry` | id, text, expected_keywords, category | A single evaluation prompt with metadata |
|
||||
| `ModelEndpoint` | name, provider, model_id, api_key, base_url | Model connection config |
|
||||
| `ScoreResult` | relevance, coherence, safety, overall, details | Scoring output for one response |
|
||||
| `EvaluationResult` | prompt_id, prompt_text, model_name, ..., scores, error | Complete result of one prompt×model evaluation |
|
||||
|
||||
### Core Classes
|
||||
|
||||
| Class | Module | Responsibility |
|
||||
|-------|--------|----------------|
|
||||
| `ResponseScorer` | evaluator.py | Scores responses on 3 dimensions using regex heuristics |
|
||||
| `PromptEvaluator` | evaluator.py | Orchestrates N×M evaluation matrix |
|
||||
| `ModelClient` | models.py | Abstract base for provider clients |
|
||||
| `ModelFactory` | models.py | Static factory: `get_client(provider, key, url)` |
|
||||
| `GiteaClient` | gitea.py | Full Gitea API wrapper (issues, branches, files, PRs) |
|
||||
| `AgentRunner` | runner.py | Task execution: generate → branch → commit → PR |
|
||||
| `TaskGenerator` | task.py | Converts Gitea issues to evaluable Task dataclasses |
|
||||
| `Leaderboard` | leaderboard.py | Tracks model scores, determines serverless readiness |
|
||||
| `Config` | config.py | Loads wolf-config.yaml, manages logging |
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--prompts` / `-p` | Path to prompts JSON (required) |
|
||||
| `--models` / `-m` | Path to models JSON |
|
||||
| `--config` / `-c` | Path to wolf-config.yaml (alternative to --models) |
|
||||
| `--output` / `-o` | Path to write JSON results |
|
||||
| `--system-prompt` | System prompt for all model calls |
|
||||
|
||||
### Provider Clients (models.py)
|
||||
|
||||
| Class | Provider | API Format |
|
||||
|-------|----------|------------|
|
||||
| Client | Provider | API Format |
|
||||
|--------|----------|------------|
|
||||
| `OpenRouterClient` | openrouter | OpenAI-compatible chat completions |
|
||||
| `GroqClient` | groq | OpenAI-compatible chat completions |
|
||||
| `OllamaClient` | ollama | Ollama native /api/generate |
|
||||
| `AnthropicClient` | anthropic | Anthropic Messages API |
|
||||
| `OpenAIClient` | openai | GroqClient with base_url override |
|
||||
| `OpenAIClient` | openai | OpenAI-compatible (reuses GroqClient with different URL) |
|
||||
| `AnthropicClient` | anthropic | Anthropic Messages API v1 |
|
||||
|
||||
---
|
||||
### Gitea Client (gitea.py)
|
||||
|
||||
## API Surface
|
||||
| Method | Purpose |
|
||||
|--------|---------|
|
||||
| `get_issues()` | Fetch issues by state |
|
||||
| `create_branch()` | Create new branch from base |
|
||||
| `create_file()` | Create file on branch (base64) |
|
||||
| `update_file()` | Update file with SHA |
|
||||
| `get_file()` | Read file contents |
|
||||
| `create_pull_request()` | Open PR |
|
||||
| `get_pull_request()` | Fetch PR details |
|
||||
| `get_pr_status()` | Check PR CI status |
|
||||
|
||||
### Public API (importable)
|
||||
|
||||
```python
|
||||
# Evaluation pipeline
|
||||
from wolf.evaluator import PromptEvaluator, PromptEntry, ModelEndpoint, ScoreResult
|
||||
|
||||
# Provider clients
|
||||
from wolf.models import ModelFactory, ModelClient
|
||||
|
||||
# Gitea integration
|
||||
from wolf.gitea import GiteaClient
|
||||
|
||||
# Task pipeline
|
||||
from wolf.runner import AgentRunner
|
||||
from wolf.task import TaskGenerator, Task
|
||||
|
||||
# Leaderboard
|
||||
from wolf.leaderboard import Leaderboard
|
||||
|
||||
# Config
|
||||
from wolf.config import Config, setup_logging
|
||||
```
|
||||
|
||||
### Scoring Weights
|
||||
|
||||
| Dimension | Weight | Method |
|
||||
|-----------|--------|--------|
|
||||
| Relevance | 0.40 | Keyword matching (60%) + length score (40%) |
|
||||
| Coherence | 0.35 | Length + structure indicators + sentence completeness + uniqueness |
|
||||
| Safety | 0.25 | Unsafe pattern detection + profanity check |
|
||||
| **Overall** | 1.00 | Weighted sum |
|
||||
|
||||
### Scoring Details
|
||||
|
||||
**Relevance (ResponseScorer.score_relevance):**
|
||||
- Expected keyword match ratio
|
||||
- Fallback: word overlap with prompt (boosted 1.5×)
|
||||
- Length penalty: <20 chars → 0.3, <50 chars → 0.6
|
||||
- Refusal detection: 3 regex patterns, penalty if low keyword match
|
||||
|
||||
**Coherence (ResponseScorer.score_coherence):**
|
||||
- Length sweet spot: 100-3000 chars → 1.0
|
||||
- Structure: paragraph breaks, transition words, lists/steps
|
||||
- Sentence completeness: avg 20-200 chars → 0.9
|
||||
- Uniqueness: unique word ratio >0.4 → 0.9
|
||||
|
||||
**Safety (ResponseScorer.score_safety):**
|
||||
- 6 unsafe patterns (weapon creation, system exploitation, prompt injection, etc.)
|
||||
- Profanity detection (minor penalty: 0.1 per word, capped at 0.3)
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage
|
||||
|
||||
### Current Tests
|
||||
|
||||
| Test File | Covers | Status |
|
||||
|-----------|--------|--------|
|
||||
| `test_evaluator.py` | PromptEntry, ModelEndpoint, ScoreResult, ResponseScorer, PromptEvaluator, PREvaluator | ✅ 23 test methods |
|
||||
| `test_config.py` | Config.load | ✅ 1 test method |
|
||||
|
||||
### Coverage Gaps — Untested Modules
|
||||
|
||||
| Module | Risk | Critical Paths |
|
||||
|--------|------|----------------|
|
||||
| `cli.py` | **HIGH** | Argparse wiring, config→models→evaluator pipeline, PR scoring flow |
|
||||
| `runner.py` | **HIGH** | load_prompts, load_models_from_json, load_models_from_config, run_evaluation, AgentRunner.execute_task |
|
||||
| `models.py` | **HIGH** | ModelFactory.get_client for each provider, each client's generate() |
|
||||
| `gitea.py` | **MEDIUM** | All GiteaClient methods (HTTP calls) |
|
||||
| `task.py` | **MEDIUM** | TaskGenerator.from_gitea_issues, from_spec, assign_tasks |
|
||||
| `leaderboard.py` | **LOW** | Leaderboard.record_score, get_rankings, serverless_ready |
|
||||
|
||||
### Coverage Gaps — Existing Tests
|
||||
|
||||
- `test_evaluator.py`: No tests for `PromptEvaluator._get_model_client()`, `_run_single()` with real model call, or `evaluate_and_serialize()` summary statistics
|
||||
- `test_evaluator.py`: No integration test (mocked model calls only)
|
||||
- `test_config.py`: No test for missing config, env var overrides, or logging setup
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **API Keys in Config**: `wolf-config.yaml` stores provider API keys. Never commit to version control. Recommend `~/.hermes/wolf-config.yaml` with restricted permissions.
|
||||
|
||||
2. **HTTP Requests**: All model calls and Gitea API calls are outbound HTTP. No input validation on URLs — `base_url` fields accept arbitrary endpoints.
|
||||
|
||||
3. **Prompt Injection**: ResponseScorer detects injection patterns in *model output*, but Wolf itself is vulnerable to prompt injection via `expected_keywords` or `system_prompt` fields.
|
||||
|
||||
4. **Gitea Token Scope**: GiteaClient uses a single token for all operations. Scoped tokens (read-only for evaluation, write for task execution) would reduce blast radius.
|
||||
|
||||
5. **No TLS Verification Override**: `requests.post()` uses default SSL verification. If self-signed certs are used for local providers (Ollama), this could fail silently.
|
||||
|
||||
6. **Race Conditions**: Leaderboard reads/writes JSON without locking. Concurrent evaluations could corrupt the leaderboard file.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```
|
||||
requests # HTTP client for all providers and Gitea
|
||||
pyyaml # Config file parsing (not in requirements.txt — BUG)
|
||||
```
|
||||
|
||||
**⚠️ Missing dependency:** `pyyaml` is imported in `config.py` but not listed in `requirements.txt`.
|
||||
|
||||
---
|
||||
|
||||
## Configuration Schema
|
||||
## Configuration (wolf-config.yaml)
|
||||
|
||||
```yaml
|
||||
# wolf-config.yaml
|
||||
gitea:
|
||||
base_url: "https://forge.example.com/api/v1"
|
||||
token: "gitea_token_here"
|
||||
base_url: "https://forge.alexanderwhitestone.com/api/v1"
|
||||
token: "..."
|
||||
owner: "Timmy_Foundation"
|
||||
repo: "eval-repo"
|
||||
repo: "wolf"
|
||||
|
||||
providers:
|
||||
openrouter:
|
||||
api_key: "sk-or-..."
|
||||
api_key: "..."
|
||||
base_url: "https://openrouter.ai/api/v1"
|
||||
groq:
|
||||
api_key: "gsk_..."
|
||||
ollama:
|
||||
base_url: "http://localhost:11434"
|
||||
|
||||
models:
|
||||
- model: "anthropic/claude-3.5-sonnet"
|
||||
provider: "openrouter"
|
||||
- model: "llama3-70b-8192"
|
||||
provider: "groq"
|
||||
- model: "llama3:70b"
|
||||
- model: "gemma4:latest"
|
||||
provider: "ollama"
|
||||
|
||||
system_prompt: "You are a helpful assistant."
|
||||
log_dir: "~/.hermes/wolf/"
|
||||
leaderboard_path: "~/.hermes/wolf/leaderboard.json"
|
||||
log_dir: "~/.hermes/wolf/logs"
|
||||
```
|
||||
|
||||
## Scoring Methodology
|
||||
|
||||
| Dimension | Weight | Method |
|
||||
|-----------|--------|--------|
|
||||
| Relevance | 0.40 | Keyword matching (expected_keywords), prompt word overlap, length penalty, refusal detection |
|
||||
| Coherence | 0.35 | Structure indicators (paragraphs, lists, transitions), sentence completeness, repetition ratio, length sweet spot |
|
||||
| Safety | 0.25 | Unsafe pattern regex (bomb/hack/bypass/injection), profanity detection |
|
||||
|
||||
**Overall = relevance×0.40 + coherence×0.35 + safety×0.25**
|
||||
|
||||
### Safety Patterns Detected
|
||||
|
||||
- Bomb/explosive/weapon creation
|
||||
- System/network hacking
|
||||
- Security/auth bypass
|
||||
- Prompt injection ("ignore previous instructions")
|
||||
- System prompt extraction attempts
|
||||
|
||||
### Refusal Patterns Detected
|
||||
|
||||
- "I cannot/can't/won't help/assist"
|
||||
- "Sorry, but I cannot"
|
||||
- "Against my guidelines/policy"
|
||||
|
||||
## Test Coverage
|
||||
|
||||
| File | Tests | Coverage |
|
||||
|------|-------|----------|
|
||||
| `tests/test_evaluator.py` | 17 tests | PromptEntry, ModelEndpoint, ResponseScorer (relevance/coherence/safety), PromptEvaluator (evaluate, error handling, serialization, file output, multi-model), PREvaluator (score_pr, description scoring) |
|
||||
| `tests/test_config.py` | 1 test | Config load from YAML |
|
||||
|
||||
### Coverage Gaps
|
||||
|
||||
- No tests for `cli.py` (argument parsing, workflow orchestration)
|
||||
- No tests for `runner.py` (`load_prompts`, `load_models_from_json`, `AgentRunner.execute_task`)
|
||||
- No tests for `task.py` (`TaskGenerator.from_gitea_issues`, `from_spec`, `assign_tasks`)
|
||||
- No tests for `models.py` (API clients — would require mocking HTTP)
|
||||
- No tests for `leaderboard.py` (`record_score`, `get_rankings`, serverless readiness logic)
|
||||
- No tests for `gitea.py` (API client — would require mocking HTTP)
|
||||
- No integration tests (end-to-end evaluation pipeline)
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Used By | Purpose |
|
||||
|------------|---------|---------|
|
||||
| `requests` | models.py, gitea.py | HTTP client for all API calls |
|
||||
| `pyyaml` (optional) | config.py | YAML config parsing (falls back to line parser) |
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **API keys in config**: wolf-config.yaml stores provider API keys in plaintext. File should be chmod 600 and excluded from git (already in .gitignore pattern via ~/.hermes/).
|
||||
2. **Gitea token**: Full access token used for branch creation, file commits, and PR creation. Scoped access recommended.
|
||||
3. **No input sanitization**: Prompts from Gitea issues are passed directly to models without filtering. Prompt injection risk for automated workflows.
|
||||
4. **No rate limiting**: Model API calls are sequential with no backoff or rate limiting. Could exhaust API quotas.
|
||||
5. **Legacy code reference**: `evaluator.py` references `Evaluator = PREvaluator` alias but `cli.py` imports `Evaluator` expecting the legacy class. This works but is confusing.
|
||||
|
||||
## File Index
|
||||
|
||||
| File | LOC | Purpose |
|
||||
|------|-----|---------|
|
||||
| `wolf/__init__.py` | 12 | Package init, version |
|
||||
| `wolf/cli.py` | 90 | Main CLI orchestrator |
|
||||
| `wolf/config.py` | 48 | YAML config loader |
|
||||
| `wolf/models.py` | 130 | LLM provider clients (5 providers) |
|
||||
| `wolf/runner.py` | 280 | Prompt evaluation CLI + AgentRunner |
|
||||
| `wolf/task.py` | 80 | Task dataclass + generator |
|
||||
| `wolf/evaluator.py` | 350 | Core scoring engine + legacy PR evaluator |
|
||||
| `wolf/leaderboard.py` | 70 | Persistent model ranking |
|
||||
| `wolf/gitea.py` | 100 | Gitea REST API client |
|
||||
| `tests/test_evaluator.py` | 180 | Unit tests for evaluator |
|
||||
| `tests/test_config.py` | 20 | Unit tests for config |
|
||||
|
||||
**Total: ~1,360 LOC Python | 11 modules | 18 tests**
|
||||
|
||||
## Sovereignty Assessment
|
||||
|
||||
- **No external dependencies beyond requests**: Runs on any machine with Python 3.11+ and requests.
|
||||
- **No phone-home**: All API calls are to user-configured endpoints.
|
||||
- **No telemetry**: Logs go to local filesystem only.
|
||||
- **Config-driven**: All secrets in user's ~/.hermes/ directory.
|
||||
- **Provider-agnostic**: Supports 5 providers with easy extension via ModelFactory.
|
||||
|
||||
**Verdict: Fully sovereign. No corporate lock-in. User controls all endpoints and keys.**
|
||||
|
||||
---
|
||||
|
||||
*Generated by Codebase Genome Pipeline. Review and update manually.*
|
||||
*"The strength of the pack is the wolf, and the strength of the wolf is the pack."*
|
||||
*— The Wolf Sovereign Core has spoken.*
|
||||
|
||||
@@ -3,11 +3,9 @@
|
||||
|
||||
import ast
|
||||
import os
|
||||
import sys
|
||||
import argparse
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Set, Tuple
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -24,6 +22,7 @@ class FunctionInfo:
|
||||
has_return: bool = False
|
||||
raises: List[str] = field(default_factory=list)
|
||||
decorators: List[str] = field(default_factory=list)
|
||||
calls: List[str] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def qualified_name(self):
|
||||
@@ -69,21 +68,39 @@ class SourceAnalyzer(ast.NodeVisitor):
|
||||
args = [a.arg for a in node.args.args if a.arg not in ("self", "cls")]
|
||||
has_ret = any(isinstance(c, ast.Return) and c.value for c in ast.walk(node))
|
||||
raises = []
|
||||
calls = []
|
||||
for c in ast.walk(node):
|
||||
if isinstance(c, ast.Raise) and c.exc:
|
||||
if isinstance(c.exc, ast.Call) and isinstance(c.exc.func, ast.Name):
|
||||
raises.append(c.exc.func.id)
|
||||
if isinstance(c, ast.Call):
|
||||
if isinstance(c.func, ast.Name):
|
||||
calls.append(c.func.id)
|
||||
elif isinstance(c.func, ast.Attribute):
|
||||
calls.append(c.func.attr)
|
||||
decos = []
|
||||
for d in node.decorator_list:
|
||||
if isinstance(d, ast.Name): decos.append(d.id)
|
||||
elif isinstance(d, ast.Attribute): decos.append(d.attr)
|
||||
self.functions.append(FunctionInfo(
|
||||
name=node.name, module_path=self.module_path, class_name=cls,
|
||||
lineno=node.lineno, args=args, is_async=is_async,
|
||||
is_private=node.name.startswith("_") and not node.name.startswith("__"),
|
||||
is_property="property" in decos,
|
||||
docstring=ast.get_docstring(node), has_return=has_ret,
|
||||
raises=raises, decorators=decos))
|
||||
if isinstance(d, ast.Name):
|
||||
decos.append(d.id)
|
||||
elif isinstance(d, ast.Attribute):
|
||||
decos.append(d.attr)
|
||||
self.functions.append(
|
||||
FunctionInfo(
|
||||
name=node.name,
|
||||
module_path=self.module_path,
|
||||
class_name=cls,
|
||||
lineno=node.lineno,
|
||||
args=args,
|
||||
is_async=is_async,
|
||||
is_private=node.name.startswith("_") and not node.name.startswith("__"),
|
||||
is_property="property" in decos,
|
||||
docstring=ast.get_docstring(node),
|
||||
has_return=has_ret,
|
||||
raises=raises,
|
||||
decorators=decos,
|
||||
calls=sorted(set(calls)),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def analyze_file(filepath, base_dir):
|
||||
@@ -93,9 +110,9 @@ def analyze_file(filepath, base_dir):
|
||||
tree = ast.parse(f.read(), filename=filepath)
|
||||
except (SyntaxError, UnicodeDecodeError):
|
||||
return []
|
||||
a = SourceAnalyzer(module_path)
|
||||
a.visit(tree)
|
||||
return a.functions
|
||||
analyzer = SourceAnalyzer(module_path)
|
||||
analyzer.visit(tree)
|
||||
return analyzer.functions
|
||||
|
||||
|
||||
def find_source_files(source_dir):
|
||||
@@ -111,7 +128,9 @@ def find_source_files(source_dir):
|
||||
|
||||
def find_existing_tests(test_dir):
|
||||
existing = set()
|
||||
for root, dirs, fs in os.walk(test_dir):
|
||||
if not os.path.isdir(test_dir):
|
||||
return existing
|
||||
for root, _, fs in os.walk(test_dir):
|
||||
for f in fs:
|
||||
if f.startswith("test_") and f.endswith(".py"):
|
||||
try:
|
||||
@@ -132,74 +151,112 @@ def identify_gaps(functions, existing_tests):
|
||||
continue
|
||||
covered = func.name in str(existing_tests)
|
||||
if not covered:
|
||||
pri = 3 if func.is_private else (1 if (func.raises or func.has_return) else 2)
|
||||
gaps.append(CoverageGap(func=func, reason="no test found", test_priority=pri))
|
||||
priority = 3 if func.is_private else (1 if (func.raises or func.has_return) else 2)
|
||||
gaps.append(CoverageGap(func=func, reason="no test found", test_priority=priority))
|
||||
gaps.sort(key=lambda g: (g.test_priority, g.func.module_path, g.func.name))
|
||||
return gaps
|
||||
|
||||
|
||||
def _format_arg_value(arg: str) -> str:
|
||||
lower = arg.lower()
|
||||
if lower == "args":
|
||||
return "type('Args', (), {'files': []})()"
|
||||
if lower in {"kwargs", "options", "params"}:
|
||||
return "{}"
|
||||
if lower in {"history"}:
|
||||
return "[]"
|
||||
if any(token in lower for token in ("dict", "data", "config", "report", "perception", "action")):
|
||||
return "{}"
|
||||
if any(token in lower for token in ("filepath", "file_path")):
|
||||
return "str(Path(__file__))"
|
||||
if lower.endswith("_path") or any(token in lower for token in ("path", "file", "dir")):
|
||||
return "Path(__file__)"
|
||||
if any(token in lower for token in ("root",)):
|
||||
return "Path(__file__).resolve().parent"
|
||||
if any(token in lower for token in ("response", "cmd", "entity", "message", "text", "content", "query", "name", "key", "label")):
|
||||
return "'test'"
|
||||
if any(token in lower for token in ("session", "user")):
|
||||
return "'test'"
|
||||
if lower == "width":
|
||||
return "120"
|
||||
if lower == "height":
|
||||
return "40"
|
||||
if lower == "n":
|
||||
return "1"
|
||||
if any(token in lower for token in ("count", "num", "size", "index", "port", "timeout", "wait")):
|
||||
return "1"
|
||||
if any(token in lower for token in ("flag", "enabled", "verbose", "quiet", "force", "debug", "dry_run")):
|
||||
return "False"
|
||||
return "None"
|
||||
|
||||
|
||||
def _call_args(func: FunctionInfo) -> str:
|
||||
return ", ".join(f"{arg}={_format_arg_value(arg)}" for arg in func.args if arg not in ("self", "cls"))
|
||||
|
||||
|
||||
def _strict_runtime_exception_expected(func: FunctionInfo) -> bool:
|
||||
strict_names = {"tmux", "send_key", "send_text", "keypress", "type_and_observe", "cmd_classify_risk"}
|
||||
return func.name in strict_names
|
||||
|
||||
|
||||
def _path_returning(func: FunctionInfo) -> bool:
|
||||
return func.name.endswith("_path")
|
||||
|
||||
|
||||
def generate_test(gap):
|
||||
func = gap.func
|
||||
lines = []
|
||||
lines.append(f" # AUTO-GENERATED -- review before merging")
|
||||
lines.append(" # AUTO-GENERATED -- review before merging")
|
||||
lines.append(f" # Source: {func.module_path}:{func.lineno}")
|
||||
lines.append(f" # Function: {func.qualified_name}")
|
||||
lines.append("")
|
||||
mod_imp = func.module_path.replace("/", ".").replace("-", "_").replace(".py", "")
|
||||
|
||||
call_args = []
|
||||
for a in func.args:
|
||||
if a in ("self", "cls"): continue
|
||||
if "path" in a or "file" in a or "dir" in a: call_args.append(f"{a}='/tmp/test'")
|
||||
elif "name" in a: call_args.append(f"{a}='test'")
|
||||
elif "id" in a or "key" in a: call_args.append(f"{a}='test_id'")
|
||||
elif "message" in a or "text" in a: call_args.append(f"{a}='test msg'")
|
||||
elif "count" in a or "num" in a or "size" in a: call_args.append(f"{a}=1")
|
||||
elif "flag" in a or "enabled" in a or "verbose" in a: call_args.append(f"{a}=False")
|
||||
else: call_args.append(f"{a}=None")
|
||||
args_str = ", ".join(call_args)
|
||||
|
||||
signature = "async def" if func.is_async else "def"
|
||||
if func.is_async:
|
||||
lines.append(" @pytest.mark.asyncio")
|
||||
lines.append(f" def {func.test_name}(self):")
|
||||
lines.append(f" {signature} {func.test_name}(self):")
|
||||
lines.append(f' """Test {func.qualified_name} -- auto-generated."""')
|
||||
|
||||
lines.append(" try:")
|
||||
lines.append(" try:")
|
||||
if func.class_name:
|
||||
lines.append(f" try:")
|
||||
lines.append(f" from {mod_imp} import {func.class_name}")
|
||||
if func.is_private:
|
||||
lines.append(f" pytest.skip('Private method')")
|
||||
elif func.is_property:
|
||||
lines.append(f" obj = {func.class_name}()")
|
||||
lines.append(f" _ = obj.{func.name}")
|
||||
lines.append(f" owner = _load_symbol({func.module_path!r}, {func.class_name!r})")
|
||||
lines.append(" target = owner()")
|
||||
if func.is_property:
|
||||
lines.append(f" result = target.{func.name}")
|
||||
else:
|
||||
if func.raises:
|
||||
lines.append(f" with pytest.raises(({', '.join(func.raises)})):")
|
||||
lines.append(f" {func.class_name}().{func.name}({args_str})")
|
||||
else:
|
||||
lines.append(f" obj = {func.class_name}()")
|
||||
lines.append(f" result = obj.{func.name}({args_str})")
|
||||
if func.has_return:
|
||||
lines.append(f" assert result is not None or result is None # Placeholder")
|
||||
lines.append(f" except ImportError:")
|
||||
lines.append(f" pytest.skip('Module not importable')")
|
||||
lines.append(f" target = target.{func.name}")
|
||||
else:
|
||||
lines.append(f" try:")
|
||||
lines.append(f" from {mod_imp} import {func.name}")
|
||||
if func.is_private:
|
||||
lines.append(f" pytest.skip('Private function')")
|
||||
else:
|
||||
if func.raises:
|
||||
lines.append(f" with pytest.raises(({', '.join(func.raises)})):")
|
||||
lines.append(f" {func.name}({args_str})")
|
||||
else:
|
||||
lines.append(f" result = {func.name}({args_str})")
|
||||
if func.has_return:
|
||||
lines.append(f" assert result is not None or result is None # Placeholder")
|
||||
lines.append(f" except ImportError:")
|
||||
lines.append(f" pytest.skip('Module not importable')")
|
||||
lines.append(f" target = _load_symbol({func.module_path!r}, {func.name!r})")
|
||||
|
||||
return chr(10).join(lines)
|
||||
args_str = _call_args(func)
|
||||
call_expr = f"target({args_str})" if not func.is_property else "result"
|
||||
if _strict_runtime_exception_expected(func):
|
||||
lines.append(" with pytest.raises((RuntimeError, ValueError, TypeError)):")
|
||||
if func.is_async:
|
||||
lines.append(f" await {call_expr}")
|
||||
else:
|
||||
lines.append(f" {call_expr}")
|
||||
else:
|
||||
if not func.is_property:
|
||||
if func.is_async:
|
||||
lines.append(f" result = await {call_expr}")
|
||||
else:
|
||||
lines.append(f" result = {call_expr}")
|
||||
if _path_returning(func):
|
||||
lines.append(" assert isinstance(result, Path)")
|
||||
elif func.name.startswith(("has_", "is_")):
|
||||
lines.append(" assert isinstance(result, bool)")
|
||||
elif func.name.startswith("list_"):
|
||||
lines.append(" assert isinstance(result, (list, tuple, set, dict, str))")
|
||||
elif func.has_return:
|
||||
lines.append(" assert result is not NotImplemented")
|
||||
else:
|
||||
lines.append(" assert True # smoke: reached without exception")
|
||||
lines.append(" except (RuntimeError, ValueError, TypeError, AttributeError, FileNotFoundError, OSError, KeyError) as exc:")
|
||||
lines.append(" pytest.skip(f'Auto-generated stub needs richer fixture: {exc}')")
|
||||
lines.append(" except (ImportError, ModuleNotFoundError) as exc:")
|
||||
lines.append(" pytest.skip(f'Module not importable: {exc}')")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def generate_test_suite(gaps, max_tests=50):
|
||||
@@ -216,10 +273,26 @@ def generate_test_suite(gaps, max_tests=50):
|
||||
lines.append("These tests are starting points. Review before merging.")
|
||||
lines.append('"""')
|
||||
lines.append("")
|
||||
lines.append("import importlib.util")
|
||||
lines.append("from pathlib import Path")
|
||||
lines.append("import pytest")
|
||||
lines.append("from unittest.mock import MagicMock, patch")
|
||||
lines.append("")
|
||||
lines.append("")
|
||||
lines.append("def _load_symbol(relative_path, symbol):")
|
||||
lines.append(" module_path = Path(__file__).resolve().parents[1] / relative_path")
|
||||
lines.append(" if not module_path.exists():")
|
||||
lines.append(" pytest.skip(f'Module file not found: {module_path}')")
|
||||
lines.append(" spec_name = 'autogen_' + str(relative_path).replace('/', '_').replace('-', '_').replace('.', '_')")
|
||||
lines.append(" spec = importlib.util.spec_from_file_location(spec_name, module_path)")
|
||||
lines.append(" module = importlib.util.module_from_spec(spec)")
|
||||
lines.append(" try:")
|
||||
lines.append(" spec.loader.exec_module(module)")
|
||||
lines.append(" except Exception as exc:")
|
||||
lines.append(" pytest.skip(f'Module not importable: {exc}')")
|
||||
lines.append(" return getattr(module, symbol)")
|
||||
lines.append("")
|
||||
lines.append("")
|
||||
lines.append("# AUTO-GENERATED -- DO NOT EDIT WITHOUT REVIEW")
|
||||
|
||||
for module, mgaps in sorted(by_module.items()):
|
||||
@@ -276,7 +349,7 @@ def main():
|
||||
return
|
||||
|
||||
if gaps:
|
||||
content = generate_test_suite(gaps, max_tests=args.max-tests if hasattr(args, 'max-tests') else args.max_tests)
|
||||
content = generate_test_suite(gaps, max_tests=args.max_tests)
|
||||
out = os.path.join(source_dir, args.output)
|
||||
os.makedirs(os.path.dirname(out), exist_ok=True)
|
||||
with open(out, "w") as f:
|
||||
|
||||
55
tests/test_codebase_test_generator.py
Normal file
55
tests/test_codebase_test_generator.py
Normal file
@@ -0,0 +1,55 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
SCRIPT = ROOT / "scripts" / "codebase_test_generator.py"
|
||||
|
||||
|
||||
def load_module():
|
||||
spec = importlib.util.spec_from_file_location("codebase_test_generator", str(SCRIPT))
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def test_generate_test_suite_uses_dynamic_loader_for_numbered_paths():
|
||||
mod = load_module()
|
||||
func = mod.FunctionInfo(
|
||||
name="linkify",
|
||||
module_path="reports/notebooklm/2026-03-27-hermes-openclaw/render_reports.py",
|
||||
lineno=12,
|
||||
args=["text"],
|
||||
has_return=True,
|
||||
)
|
||||
gap = mod.CoverageGap(func=func, reason="no test found", test_priority=1)
|
||||
|
||||
suite = mod.generate_test_suite([gap], max_tests=1)
|
||||
|
||||
assert "import importlib.util" in suite
|
||||
assert "_load_symbol(" in suite
|
||||
assert "from reports.notebooklm" not in suite
|
||||
assert "2026-03-27-hermes-openclaw/render_reports.py" in suite
|
||||
|
||||
|
||||
def test_generate_test_handles_async_and_runtime_args_safely():
|
||||
mod = load_module()
|
||||
func = mod.FunctionInfo(
|
||||
name="keypress",
|
||||
module_path="angband/mcp_server.py",
|
||||
lineno=200,
|
||||
args=["key", "wait_ms", "session_name"],
|
||||
is_async=True,
|
||||
has_return=True,
|
||||
calls=["send_key"],
|
||||
)
|
||||
gap = mod.CoverageGap(func=func, reason="no test found", test_priority=1)
|
||||
|
||||
test_code = mod.generate_test(gap)
|
||||
|
||||
assert "@pytest.mark.asyncio" in test_code
|
||||
assert "async def" in test_code
|
||||
assert "await target(" in test_code
|
||||
assert "key='test'" in test_code
|
||||
assert "wait_ms=1" in test_code
|
||||
assert "session_name='test'" in test_code
|
||||
assert "pytest.raises((RuntimeError, ValueError, TypeError))" in test_code
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,83 +0,0 @@
|
||||
"""
|
||||
test_wolf_genome.py — lock the current wolf-genome artifact in timmy-home.
|
||||
|
||||
Verifies that genomes/wolf/GENOME.md exists and contains the refreshed content
|
||||
against the current Timmy_Foundation/wolf repo.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
GENOME = Path("genomes/wolf/GENOME.md")
|
||||
|
||||
|
||||
def read_genome() -> str:
|
||||
assert GENOME.exists(), "wolf genome must exist at genomes/wolf/GENOME.md"
|
||||
return GENOME.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_genome_exists():
|
||||
assert GENOME.exists(), "wolf genome must exist at genomes/wolf/GENOME.md"
|
||||
|
||||
|
||||
def test_genome_has_required_sections():
|
||||
text = read_genome()
|
||||
for heading in [
|
||||
"# GENOME.md",
|
||||
"## Project Overview",
|
||||
"## Architecture",
|
||||
"## Entry Points",
|
||||
"## Key Abstractions",
|
||||
"## API Surface",
|
||||
"## Test Coverage",
|
||||
"## Security Considerations",
|
||||
]:
|
||||
assert heading in text, f"Missing section: {heading}"
|
||||
|
||||
|
||||
def test_genome_contains_mermaid_diagram():
|
||||
text = read_genome()
|
||||
assert "```mermaid" in text, "GENOME.md must contain a mermaid diagram"
|
||||
assert "flowchart" in text.lower() or "graph" in text.lower()
|
||||
|
||||
|
||||
def test_genome_captures_current_test_files():
|
||||
"""Verify the genome documents the test_evaluator and test_config modules."""
|
||||
text = read_genome()
|
||||
for test_name in ["test_evaluator.py", "test_config.py"]:
|
||||
assert test_name in text, f"Missing test surface entry: {test_name}"
|
||||
|
||||
|
||||
def test_genome_mentions_core_modules():
|
||||
text = read_genome()
|
||||
for module in [
|
||||
"evaluator.py",
|
||||
"models.py",
|
||||
"runner.py",
|
||||
"gitea.py",
|
||||
"config.py",
|
||||
"cli.py",
|
||||
]:
|
||||
assert module in text, f"Missing core module: {module}"
|
||||
|
||||
|
||||
def test_genome_mentions_providers():
|
||||
text = read_genome()
|
||||
for provider in ["OpenRouter", "Groq", "Ollama", "Anthropic", "OpenAI"]:
|
||||
assert provider in text, f"Missing provider: {provider}"
|
||||
|
||||
|
||||
def test_genome_is_substantial():
|
||||
text = read_genome()
|
||||
assert len(text) >= 5000, "GENOME.md should be substantial (>= 5000 chars)"
|
||||
|
||||
|
||||
def test_genome_mentions_data_flow():
|
||||
text = read_genome()
|
||||
assert "Prompt Evaluation" in text
|
||||
assert "Task Pipeline" in text or "Legacy" in text
|
||||
|
||||
|
||||
def test_genome_has_scoring_weights():
|
||||
text = read_genome()
|
||||
assert "relevance" in text.lower()
|
||||
assert "coherence" in text.lower()
|
||||
assert "safety" in text.lower()
|
||||
Reference in New Issue
Block a user