feat: cross-repo dependency graph builder (#93)

GENOME.md

@@ -1,239 +0,0 @@
-# GENOME.md — compounding-intelligence
-
-*Auto-generated codebase genome. See timmy-home#676.*
-
----
-
-## Project Overview
-
-**What:** A system that turns 1B+ daily agent tokens into durable, compounding fleet intelligence.
-
-**Why:** Every agent session starts at zero. The same mistakes get made repeatedly — the same HTTP 405 is rediscovered as a branch protection issue, the same token path is searched for from scratch. Intelligence evaporates when the session ends.
-
-**How:** Three pipelines form a compounding loop:
-
-```
-SESSION ENDS → HARVESTER → KNOWLEDGE STORE → BOOTSTRAPPER → NEW SESSION STARTS SMARTER
-                              ↓
-                         MEASURER → Prove it's working
-```
-
-**Status:** Early stage. Template and test scaffolding exist. Core pipeline scripts (harvester.py, bootstrapper.py, measurer.py, session_reader.py) are planned but not yet implemented. The knowledge extraction prompt is complete and validated.
-
----
-
-## Architecture
-
-```mermaid
-graph TD
-    A[Session Transcript<br/>.jsonl] --> B[Harvester]
-    B --> C{Extract Knowledge}
-    C --> D[knowledge/index.json]
-    C --> E[knowledge/global/*.md]
-    C --> F[knowledge/repos/{repo}.md]
-    C --> G[knowledge/agents/{agent}.md]
-    D --> H[Bootstrapper]
-    H --> I[Bootstrap Context<br/>2k token injection]
-    I --> J[New Session<br/>starts smarter]
-    J --> A
-    D --> K[Measurer]
-    K --> L[metrics/dashboard.md]
-    K --> M[Velocity / Hit Rate<br/>Error Reduction]
-```
-
-### Pipeline 1: Harvester
-
-**Status:** Prompt designed. Script not implemented.
-
-Reads finished session transcripts (JSONL). Uses `templates/harvest-prompt.md` to extract durable knowledge into five categories:
-
-| Category | Description | Example |
-|----------|-------------|---------|
-| `fact` | Concrete, verifiable information | "Repository X has 5 files" |
-| `pitfall` | Errors encountered, wrong assumptions | "Token is at ~/.config/gitea/token, not env var" |
-| `pattern` | Successful action sequences | "Deploy: test → build → push → webhook" |
-| `tool-quirk` | Environment-specific behaviors | "URL format requires trailing slash" |
-| `question` | Identified but unanswered | "Need optimal batch size for harvesting" |
-
-Output schema per knowledge item:
-```json
-{
-  "fact": "One sentence description",
-  "category": "fact|pitfall|pattern|tool-quirk|question",
-  "repo": "repo-name or 'global'",
-  "confidence": 0.0-1.0
-}
-```
-
-### Pipeline 2: Bootstrapper
-
-**Status:** Not implemented.
-
-Queries knowledge store before session start. Assembles a compact 2k-token context from relevant facts. Injects into session startup so the agent begins with full situational awareness.
-
-### Pipeline 3: Measurer
-
-**Status:** Not implemented.
-
-Tracks compounding metrics: knowledge velocity (facts/day), error reduction (%), hit rate (knowledge used / knowledge available), task completion improvement.
-
----
-
-## Directory Structure
-
-```
-compounding-intelligence/
-├── README.md                           # Project overview and architecture
-├── GENOME.md                           # This file (codebase genome)
-├── knowledge/                          # [PLANNED] Knowledge store
-│   ├── index.json                      # Machine-readable fact index
-│   ├── global/                         # Cross-repo knowledge
-│   ├── repos/{repo}.md                 # Per-repo knowledge
-│   └── agents/{agent}.md               # Agent-type notes
-├── scripts/
-│   ├── test_harvest_prompt.py          # Basic prompt validation (2.5KB)
-│   └── test_harvest_prompt_comprehensive.py  # Full prompt structure test (6.8KB)
-├── templates/
-│   └── harvest-prompt.md               # Knowledge extraction prompt (3.5KB)
-├── test_sessions/
-│   ├── session_success.jsonl           # Happy path test data
-│   ├── session_failure.jsonl           # Failure path test data
-│   ├── session_partial.jsonl           # Incomplete session test data
-│   ├── session_patterns.jsonl          # Pattern extraction test data
-│   └── session_questions.jsonl         # Question identification test data
-└── metrics/                            # [PLANNED] Compounding metrics
-    └── dashboard.md
-```
-
----
-
-## Entry Points and Data Flow
-
-### Entry Point 1: Knowledge Extraction (Harvester)
-
-```
-Input:  Session transcript (JSONL)
-        ↓
-        templates/harvest-prompt.md (LLM prompt)
-        ↓
-        Knowledge items (JSON array)
-        ↓
-Output: knowledge/index.json + per-repo/per-agent markdown files
-```
-
-### Entry Point 2: Session Bootstrap (Bootstrapper)
-
-```
-Input:  Session context (repo, agent type, task type)
-        ↓
-        knowledge/index.json (query relevant facts)
-        ↓
-        2k-token bootstrap context
-        ↓
-Output: Injected into session startup
-```
-
-### Entry Point 3: Measurement (Measurer)
-
-```
-Input:  knowledge/index.json + session history
-        ↓
-        Velocity, hit rate, error reduction calculations
-        ↓
-Output: metrics/dashboard.md
-```
-
----
-
-## Key Abstractions
-
-### Knowledge Item
-The atomic unit. One sentence, one category, one confidence score. Designed to be small enough that 1000 items fit in a 2k-token bootstrap context.
-
-### Knowledge Store
-A directory structure that mirrors the fleet's mental model:
-- `global/` — knowledge that applies everywhere (tool quirks, environment facts)
-- `repos/` — knowledge specific to each repo
-- `agents/` — knowledge specific to each agent type
-
-### Confidence Score
-0.0–1.0 scale. Defines how certain the harvester is about each extracted fact:
-- 0.9–1.0: Explicitly stated with verification
-- 0.7–0.8: Clearly implied by multiple data points
-- 0.5–0.6: Suggested but not fully verified
-- 0.3–0.4: Inferred from limited data
-- 0.1–0.2: Speculative or uncertain
-
-### Bootstrap Context
-The 2k-token injection that a new session receives. Assembled from the most relevant knowledge items for the current task, filtered by confidence > 0.7, deduplicated, and compressed.
-
----
-
-## API Surface
-
-### Internal (scripts not yet implemented)
-
-| Script | Input | Output | Status |
-|--------|-------|--------|--------|
-| `harvester.py` | Session JSONL path | Knowledge items JSON | PLANNED |
-| `bootstrapper.py` | Repo + agent type | 2k-token context string | PLANNED |
-| `measurer.py` | Knowledge store path | Metrics JSON | PLANNED |
-| `session_reader.py` | Session JSONL path | Parsed transcript | PLANNED |
-
-### Prompt (templates/harvest-prompt.md)
-
-The extraction prompt is the core "API." It takes a session transcript and returns structured JSON. It defines:
-- Five extraction categories
-- Output format (JSON array of knowledge items)
-- Confidence scoring rubric
-- Constraints (no hallucination, specificity, relevance, brevity)
-- Example input/output pair
-
----
-
-## Test Coverage
-
-### What Exists
-
-| File | Tests | Coverage |
-|------|-------|----------|
-| `scripts/test_harvest_prompt.py` | 2 tests | Prompt file existence, sample transcript |
-| `scripts/test_harvest_prompt_comprehensive.py` | 5 tests | Prompt structure, categories, fields, confidence scoring, size limits |
-| `test_sessions/*.jsonl` | 5 sessions | Success, failure, partial, patterns, questions |
-
-### What's Missing
-
-1. **Harvester integration test** — Does the prompt actually extract correct knowledge from real transcripts?
-2. **Bootstrapper test** — Does it assemble relevant context correctly?
-3. **Knowledge store test** — Does the index.json maintain consistency?
-4. **Confidence calibration test** — Do high-confidence facts actually prove true in later sessions?
-5. **Deduplication test** — Are duplicate facts across sessions handled?
-6. **Staleness test** — How does the system handle outdated knowledge?
-
----
-
-## Security Considerations
-
-1. **No secrets in knowledge store** — The harvester must filter out API keys, tokens, and credentials from extracted facts. The prompt constraints mention this but there is no automated guard.
-
-2. **Knowledge poisoning** — A malicious or corrupted session could inject false facts. Confidence scoring partially mitigates this, but there is no verification step.
-
-3. **Access control** — The knowledge store has no access control. Any process that can read the directory can read all facts. In a multi-tenant setup, this is a concern.
-
-4. **Transcript privacy** — Session transcripts may contain user data. The harvester must not extract personally identifiable information into the knowledge store.
-
----
-
-## The 100x Path (from README)
-
-```
-Month 1:  15,000 facts, sessions 20% faster
-Month 2:  45,000 facts, sessions 40% faster, first-try success up 30%
-Month 3:  90,000 facts, fleet measurably smarter per token
-```
-
-Each new session is better than the last. The intelligence compounds.
-
----
-
-*Generated by codebase-genome pipeline. Ref: timmy-home#676.*

scripts/dependency_graph.py

@@ -0,0 +1,249 @@
+#!/usr/bin/env python3
+"""
+Cross-Repo Dependency Graph Builder
+
+Scans repos for import/require/reference patterns and builds a directed
+dependency graph. Detects circular dependencies. Outputs DOT and Mermaid.
+
+Usage:
+  python3 scripts/dependency_graph.py /path/to/repos/
+  python3 scripts/dependency_graph.py --repos repo1,repo2,repo3 --format mermaid
+  python3 scripts/dependency_graph.py --repos-dir /path/to/ --format dot --output deps.dot
+
+Patterns detected:
+  - Python: import X, from X import Y
+  - JavaScript: require("X"), import ... from "X"
+  - Go: import "X"
+  - Ansible: include_role, import_role
+  - Docker/Compose: image: X, depends_on
+  - Config references: repo-name in YAML/TOML/JSON
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+from collections import defaultdict
+from pathlib import Path
+
+
+# Known repo names for matching
+KNOWN_REPOS = [
+    "hermes-agent", "timmy-config", "timmy-home", "the-nexus", "the-door",
+    "the-beacon", "fleet-ops", "burn-fleet", "timmy-dispatch", "turboquant",
+    "compounding-intelligence", "the-playground", "second-son-of-timmy",
+    "ai-safety-review", "the-echo-pattern", "timmy-academy", "wolf",
+    "the-testament",
+]
+
+
+def normalize_repo_name(name: str) -> str:
+    """Normalize a repo name for comparison."""
+    return name.lower().replace("_", "-").replace(".git", "").strip()
+
+
+def scan_file_for_deps(filepath: str, content: str, own_repo: str) -> set:
+    """Scan a file's content for references to other repos."""
+    deps = set()
+    own_norm = normalize_repo_name(own_repo)
+
+    for repo in KNOWN_REPOS:
+        repo_norm = normalize_repo_name(repo)
+        if repo_norm == own_norm:
+            continue
+
+        # Direct name references
+        patterns = [
+            repo,  # exact name
+            repo.replace("-", "_"),  # underscore variant
+            repo.replace("-", ""),  # no separator
+            f"/{repo}/",  # path reference
+            f'"{repo}"',  # quoted
+            f"'{repo}'",  # single quoted
+            f"Timmy_Foundation/{repo}",  # full Gitea path
+            f"Timmy_Foundation.{repo}",  # Python module path
+        ]
+
+        for pattern in patterns:
+            if pattern in content:
+                deps.add(repo)
+                break
+
+    return deps
+
+
+def scan_repo(repo_path: str, repo_name: str = None) -> dict:
+    """Scan a repo directory for dependencies."""
+    path = Path(repo_path)
+    if not path.is_dir():
+        return {"error": f"Not a directory: {repo_path}"}
+
+    if not repo_name:
+        repo_name = path.name
+
+    deps = set()
+    files_scanned = 0
+    exts = {".py", ".js", ".ts", ".go", ".yaml", ".yml", ".toml", ".json",
+            ".md", ".sh", ".bash", ".Dockerfile", ".tf", ".hcl"}
+
+    for fpath in path.rglob("*"):
+        if not fpath.is_file():
+            continue
+        if fpath.suffix not in exts:
+            continue
+        # Skip common non-source dirs
+        parts = fpath.parts
+        if any(p in (".git", "node_modules", "__pycache__", ".venv", "venv",
+                      "vendor", "dist", "build", ".tox") for p in parts):
+            continue
+
+        try:
+            content = fpath.read_text(errors="ignore")
+        except:
+            continue
+
+        file_deps = scan_file_for_deps(str(fpath), content, repo_name)
+        deps.update(file_deps)
+        files_scanned += 1
+
+    return {
+        "repo": repo_name,
+        "dependencies": sorted(deps),
+        "files_scanned": files_scanned,
+    }
+
+
+def detect_cycles(graph: dict) -> list:
+    """Detect circular dependencies using DFS."""
+    cycles = []
+    visited = set()
+    rec_stack = set()
+
+    def dfs(node, path):
+        visited.add(node)
+        rec_stack.add(node)
+
+        for neighbor in graph.get(node, {}).get("dependencies", []):
+            if neighbor not in visited:
+                result = dfs(neighbor, path + [neighbor])
+                if result:
+                    return result
+            elif neighbor in rec_stack:
+                cycle_start = path.index(neighbor)
+                return path[cycle_start:] + [neighbor]
+
+        rec_stack.remove(node)
+        return None
+
+    for node in graph:
+        if node not in visited:
+            cycle = dfs(node, [node])
+            if cycle:
+                cycles.append(cycle)
+
+    return cycles
+
+
+def to_dot(graph: dict) -> str:
+    """Generate DOT format output."""
+    lines = ["digraph dependencies {"]
+    lines.append("  rankdir=LR;")
+    lines.append("  node [shape=box, style=filled, fillcolor="#1a1a2e", fontcolor="#e6edf3"];")
+    lines.append("  edge [color="#4a4a6a"];")
+    lines.append("")
+
+    for repo, data in sorted(graph.items()):
+        dep_count = len(data.get("dependencies", []))
+        fill = "#2d1b69" if dep_count > 2 else "#16213e"
+        lines.append(f'  "{repo}" [fillcolor="{fill}"];')
+        for dep in data.get("dependencies", []):
+            lines.append(f'  "{repo}" -> "{dep}";')
+
+    lines.append("}")
+    return "\n".join(lines)
+
+
+def to_mermaid(graph: dict) -> str:
+    """Generate Mermaid format output."""
+    lines = ["graph LR"]
+
+    for repo, data in sorted(graph.items()):
+        for dep in data.get("dependencies", []):
+            lines.append(f"    {repo.replace('-','_')} --> {dep.replace('-','_')}")
+
+    # Add node labels
+    lines.append("")
+    for repo in sorted(graph.keys()):
+        lines.append(f"    {repo.replace('-','_')}[{repo}]")
+
+    return "\n".join(lines)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Build cross-repo dependency graph")
+    parser.add_argument("repos_dir", nargs="?", help="Directory containing repos")
+    parser.add_argument("--repos", help="Comma-separated list of repo paths")
+    parser.add_argument("--format", choices=["dot", "mermaid", "json"], default="json")
+    parser.add_argument("--output", "-o", help="Output file (default: stdout)")
+    parser.add_argument("--cycles-only", action="store_true", help="Only report cycles")
+    args = parser.parse_args()
+
+    results = {}
+    repo_paths = []
+
+    if args.repos:
+        repo_paths = [p.strip() for p in args.repos.split(",")]
+    elif args.repos_dir:
+        base = Path(args.repos_dir)
+        repo_paths = [str(p) for p in base.iterdir() if p.is_dir() and not p.name.startswith(".")]
+    else:
+        parser.print_help()
+        sys.exit(1)
+
+    for rpath in repo_paths:
+        name = Path(rpath).name
+        print(f"Scanning {name}...", file=sys.stderr)
+        result = scan_repo(rpath, name)
+        if "error" not in result:
+            results[name] = result
+
+    # Detect cycles
+    cycles = detect_cycles(results)
+
+    if args.cycles_only:
+        if cycles:
+            print("CIRCULAR DEPENDENCIES DETECTED:")
+            for cycle in cycles:
+                print(f"  {' -> '.join(cycle)}")
+            sys.exit(1)
+        else:
+            print("No circular dependencies found.")
+            sys.exit(0)
+
+    # Output
+    output = {}
+    if args.format == "dot":
+        output = to_dot(results)
+    elif args.format == "mermaid":
+        output = to_mermaid(results)
+    else:
+        output = json.dumps({
+            "repos": results,
+            "cycles": cycles,
+            "summary": {
+                "total_repos": len(results),
+                "total_deps": sum(len(r["dependencies"]) for r in results.values()),
+                "cycles_found": len(cycles),
+            }
+        }, indent=2)
+
+    if args.output:
+        Path(args.output).write_text(output)
+        print(f"Written to {args.output}", file=sys.stderr)
+    else:
+        print(output)
+
+
+if __name__ == "__main__":
+    main()