test: add tests for issue body parser (#90)

GENOME.md

@@ -1,239 +0,0 @@
-# GENOME.md — compounding-intelligence
-
-*Auto-generated codebase genome. See timmy-home#676.*
-
----
-
-## Project Overview
-
-**What:** A system that turns 1B+ daily agent tokens into durable, compounding fleet intelligence.
-
-**Why:** Every agent session starts at zero. The same mistakes get made repeatedly — the same HTTP 405 is rediscovered as a branch protection issue, the same token path is searched for from scratch. Intelligence evaporates when the session ends.
-
-**How:** Three pipelines form a compounding loop:
-
-```
-SESSION ENDS → HARVESTER → KNOWLEDGE STORE → BOOTSTRAPPER → NEW SESSION STARTS SMARTER
-                              ↓
-                         MEASURER → Prove it's working
-```
-
-**Status:** Early stage. Template and test scaffolding exist. Core pipeline scripts (harvester.py, bootstrapper.py, measurer.py, session_reader.py) are planned but not yet implemented. The knowledge extraction prompt is complete and validated.
-
----
-
-## Architecture
-
-```mermaid
-graph TD
-    A[Session Transcript<br/>.jsonl] --> B[Harvester]
-    B --> C{Extract Knowledge}
-    C --> D[knowledge/index.json]
-    C --> E[knowledge/global/*.md]
-    C --> F[knowledge/repos/{repo}.md]
-    C --> G[knowledge/agents/{agent}.md]
-    D --> H[Bootstrapper]
-    H --> I[Bootstrap Context<br/>2k token injection]
-    I --> J[New Session<br/>starts smarter]
-    J --> A
-    D --> K[Measurer]
-    K --> L[metrics/dashboard.md]
-    K --> M[Velocity / Hit Rate<br/>Error Reduction]
-```
-
-### Pipeline 1: Harvester
-
-**Status:** Prompt designed. Script not implemented.
-
-Reads finished session transcripts (JSONL). Uses `templates/harvest-prompt.md` to extract durable knowledge into five categories:
-
-| Category | Description | Example |
-|----------|-------------|---------|
-| `fact` | Concrete, verifiable information | "Repository X has 5 files" |
-| `pitfall` | Errors encountered, wrong assumptions | "Token is at ~/.config/gitea/token, not env var" |
-| `pattern` | Successful action sequences | "Deploy: test → build → push → webhook" |
-| `tool-quirk` | Environment-specific behaviors | "URL format requires trailing slash" |
-| `question` | Identified but unanswered | "Need optimal batch size for harvesting" |
-
-Output schema per knowledge item:
-```json
-{
-  "fact": "One sentence description",
-  "category": "fact|pitfall|pattern|tool-quirk|question",
-  "repo": "repo-name or 'global'",
-  "confidence": 0.0-1.0
-}
-```
-
-### Pipeline 2: Bootstrapper
-
-**Status:** Not implemented.
-
-Queries knowledge store before session start. Assembles a compact 2k-token context from relevant facts. Injects into session startup so the agent begins with full situational awareness.
-
-### Pipeline 3: Measurer
-
-**Status:** Not implemented.
-
-Tracks compounding metrics: knowledge velocity (facts/day), error reduction (%), hit rate (knowledge used / knowledge available), task completion improvement.
-
----
-
-## Directory Structure
-
-```
-compounding-intelligence/
-├── README.md                           # Project overview and architecture
-├── GENOME.md                           # This file (codebase genome)
-├── knowledge/                          # [PLANNED] Knowledge store
-│   ├── index.json                      # Machine-readable fact index
-│   ├── global/                         # Cross-repo knowledge
-│   ├── repos/{repo}.md                 # Per-repo knowledge
-│   └── agents/{agent}.md               # Agent-type notes
-├── scripts/
-│   ├── test_harvest_prompt.py          # Basic prompt validation (2.5KB)
-│   └── test_harvest_prompt_comprehensive.py  # Full prompt structure test (6.8KB)
-├── templates/
-│   └── harvest-prompt.md               # Knowledge extraction prompt (3.5KB)
-├── test_sessions/
-│   ├── session_success.jsonl           # Happy path test data
-│   ├── session_failure.jsonl           # Failure path test data
-│   ├── session_partial.jsonl           # Incomplete session test data
-│   ├── session_patterns.jsonl          # Pattern extraction test data
-│   └── session_questions.jsonl         # Question identification test data
-└── metrics/                            # [PLANNED] Compounding metrics
-    └── dashboard.md
-```
-
----
-
-## Entry Points and Data Flow
-
-### Entry Point 1: Knowledge Extraction (Harvester)
-
-```
-Input:  Session transcript (JSONL)
-        ↓
-        templates/harvest-prompt.md (LLM prompt)
-        ↓
-        Knowledge items (JSON array)
-        ↓
-Output: knowledge/index.json + per-repo/per-agent markdown files
-```
-
-### Entry Point 2: Session Bootstrap (Bootstrapper)
-
-```
-Input:  Session context (repo, agent type, task type)
-        ↓
-        knowledge/index.json (query relevant facts)
-        ↓
-        2k-token bootstrap context
-        ↓
-Output: Injected into session startup
-```
-
-### Entry Point 3: Measurement (Measurer)
-
-```
-Input:  knowledge/index.json + session history
-        ↓
-        Velocity, hit rate, error reduction calculations
-        ↓
-Output: metrics/dashboard.md
-```
-
----
-
-## Key Abstractions
-
-### Knowledge Item
-The atomic unit. One sentence, one category, one confidence score. Designed to be small enough that 1000 items fit in a 2k-token bootstrap context.
-
-### Knowledge Store
-A directory structure that mirrors the fleet's mental model:
-- `global/` — knowledge that applies everywhere (tool quirks, environment facts)
-- `repos/` — knowledge specific to each repo
-- `agents/` — knowledge specific to each agent type
-
-### Confidence Score
-0.0–1.0 scale. Defines how certain the harvester is about each extracted fact:
-- 0.9–1.0: Explicitly stated with verification
-- 0.7–0.8: Clearly implied by multiple data points
-- 0.5–0.6: Suggested but not fully verified
-- 0.3–0.4: Inferred from limited data
-- 0.1–0.2: Speculative or uncertain
-
-### Bootstrap Context
-The 2k-token injection that a new session receives. Assembled from the most relevant knowledge items for the current task, filtered by confidence > 0.7, deduplicated, and compressed.
-
----
-
-## API Surface
-
-### Internal (scripts not yet implemented)
-
-| Script | Input | Output | Status |
-|--------|-------|--------|--------|
-| `harvester.py` | Session JSONL path | Knowledge items JSON | PLANNED |
-| `bootstrapper.py` | Repo + agent type | 2k-token context string | PLANNED |
-| `measurer.py` | Knowledge store path | Metrics JSON | PLANNED |
-| `session_reader.py` | Session JSONL path | Parsed transcript | PLANNED |
-
-### Prompt (templates/harvest-prompt.md)
-
-The extraction prompt is the core "API." It takes a session transcript and returns structured JSON. It defines:
-- Five extraction categories
-- Output format (JSON array of knowledge items)
-- Confidence scoring rubric
-- Constraints (no hallucination, specificity, relevance, brevity)
-- Example input/output pair
-
----
-
-## Test Coverage
-
-### What Exists
-
-| File | Tests | Coverage |
-|------|-------|----------|
-| `scripts/test_harvest_prompt.py` | 2 tests | Prompt file existence, sample transcript |
-| `scripts/test_harvest_prompt_comprehensive.py` | 5 tests | Prompt structure, categories, fields, confidence scoring, size limits |
-| `test_sessions/*.jsonl` | 5 sessions | Success, failure, partial, patterns, questions |
-
-### What's Missing
-
-1. **Harvester integration test** — Does the prompt actually extract correct knowledge from real transcripts?
-2. **Bootstrapper test** — Does it assemble relevant context correctly?
-3. **Knowledge store test** — Does the index.json maintain consistency?
-4. **Confidence calibration test** — Do high-confidence facts actually prove true in later sessions?
-5. **Deduplication test** — Are duplicate facts across sessions handled?
-6. **Staleness test** — How does the system handle outdated knowledge?
-
----
-
-## Security Considerations
-
-1. **No secrets in knowledge store** — The harvester must filter out API keys, tokens, and credentials from extracted facts. The prompt constraints mention this but there is no automated guard.
-
-2. **Knowledge poisoning** — A malicious or corrupted session could inject false facts. Confidence scoring partially mitigates this, but there is no verification step.
-
-3. **Access control** — The knowledge store has no access control. Any process that can read the directory can read all facts. In a multi-tenant setup, this is a concern.
-
-4. **Transcript privacy** — Session transcripts may contain user data. The harvester must not extract personally identifiable information into the knowledge store.
-
----
-
-## The 100x Path (from README)
-
-```
-Month 1:  15,000 facts, sessions 20% faster
-Month 2:  45,000 facts, sessions 40% faster, first-try success up 30%
-Month 3:  90,000 facts, fleet measurably smarter per token
-```
-
-Each new session is better than the last. The intelligence compounds.
-
----
-
-*Generated by codebase-genome pipeline. Ref: timmy-home#676.*

scripts/gitea_issue_parser.py

@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+"""
+Gitea Issue Body Parser
+
+Extracts structured data from Gitea issue markdown bodies:
+- Title
+- Context section
+- Acceptance criteria (checkboxes)
+- Labels
+- Epic/parent references
+
+Usage:
+  python3 scripts/gitea_issue_parser.py <issue_body.txt
+  python3 scripts/gitea_issue_parser.py --url https://forge.../api/v1/repos/.../issues/123
+  echo "issue body" | python3 scripts/gitea_issue_parser.py --stdin
+
+Output: JSON with {title, context, criteria[], labels[], epic_ref}
+"""
+
+import argparse
+import json
+import re
+import sys
+from typing import Optional
+
+
+def parse_issue_body(body: str, title: str = "", labels: list = None) -> dict:
+    """Parse a Gitea issue body into structured JSON."""
+    result = {
+        "title": title,
+        "context": "",
+        "criteria": [],
+        "labels": labels or [],
+        "epic_ref": None,
+        "sections": {},
+    }
+
+    if not body:
+        return result
+
+    # Extract epic/parent reference from title or body
+    epic_pattern = r"#(\d+)"
+    title_refs = re.findall(epic_pattern, title)
+    body_refs = re.findall(epic_pattern, body[:200])  # Check early body refs
+
+    # Look for "Closes #N" or "Part of #N" or "Epic: #N"
+    close_match = re.search(r"(?:Closes?|Fixes?|Resolves?)\s+#(\d+)", body, re.IGNORECASE)
+    part_match = re.search(r"(?:Part of|Epic|Parent|Blocks?)\s+#(\d+)", body, re.IGNORECASE)
+
+    if close_match:
+        result["epic_ref"] = f"#{close_match.group(1)}"
+    elif part_match:
+        result["epic_ref"] = f"#{part_match.group(1)}"
+    elif title_refs:
+        result["epic_ref"] = f"#{title_refs[0]}"
+    elif body_refs:
+        result["epic_ref"] = f"#{body_refs[0]}"
+
+    # Split into sections by ## headers
+    section_pattern = r"^##\s+(.+)$"
+    lines = body.split("\n")
+    current_section = None
+    current_content = []
+
+    for line in lines:
+        header_match = re.match(section_pattern, line)
+        if header_match:
+            # Save previous section
+            if current_section:
+                result["sections"][current_section] = "\n".join(current_content).strip()
+            current_section = header_match.group(1).strip().lower()
+            current_content = []
+        else:
+            current_content.append(line)
+
+    # Save last section
+    if current_section:
+        result["sections"][current_section] = "\n".join(current_content).strip()
+
+    # Extract context
+    for key in ["context", "background", "description", "problem"]:
+        if key in result["sections"]:
+            result["context"] = result["sections"][key]
+            break
+
+    # Extract acceptance criteria (checkboxes)
+    criteria_section = None
+    for key in ["acceptance criteria", "acceptance_criteria", "criteria", "requirements", "definition of done"]:
+        if key in result["sections"]:
+            criteria_section = result["sections"][key]
+            break
+
+    if criteria_section:
+        checkbox_pattern = r"-\s*\[[ xX]?\]\s*(.+)"
+        for match in re.finditer(checkbox_pattern, criteria_section):
+            result["criteria"].append(match.group(1).strip())
+
+        # Also try plain numbered/bulleted lists if no checkboxes found
+        if not result["criteria"]:
+            list_pattern = r"^\s*(?:\d+\.|-|\*)\s+(.+)"
+            for match in re.finditer(list_pattern, criteria_section, re.MULTILINE):
+                result["criteria"].append(match.group(1).strip())
+
+    # If no sectioned criteria found, scan whole body for checkboxes
+    if not result["criteria"]:
+        for match in re.finditer(r"-\s*\[[ xX]?\]\s*(.+)", body):
+            result["criteria"].append(match.group(1).strip())
+
+    return result
+
+
+def parse_from_url(api_url: str, token: str = None) -> dict:
+    """Parse an issue from a Gitea API URL."""
+    import urllib.request
+
+    headers = {}
+    if token:
+        headers["Authorization"] = f"token {token}"
+
+    req = urllib.request.Request(api_url, headers=headers)
+    resp = json.loads(urllib.request.urlopen(req, timeout=30).read())
+
+    title = resp.get("title", "")
+    body = resp.get("body", "")
+    labels = [l["name"] for l in resp.get("labels", [])]
+
+    return parse_issue_body(body, title, labels)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Parse Gitea issue body into structured JSON")
+    parser.add_argument("input", nargs="?", help="Issue body file (or - for stdin)")
+    parser.add_argument("--url", help="Gitea API URL for the issue")
+    parser.add_argument("--stdin", action="store_true", help="Read from stdin")
+    parser.add_argument("--token", help="Gitea API token (or set GITEA_TOKEN env var)")
+    parser.add_argument("--title", default="", help="Issue title (for epic ref extraction)")
+    parser.add_argument("--labels", nargs="*", default=[], help="Issue labels")
+    parser.add_argument("--pretty", action="store_true", help="Pretty-print JSON output")
+    args = parser.parse_args()
+
+    import os
+    token = args.token or os.environ.get("GITEA_TOKEN")
+
+    if args.url:
+        result = parse_from_url(args.url, token)
+    elif args.stdin or (args.input and args.input == "-"):
+        body = sys.stdin.read()
+        result = parse_issue_body(body, args.title, args.labels)
+    elif args.input:
+        with open(args.input) as f:
+            body = f.read()
+        result = parse_issue_body(body, args.title, args.labels)
+    else:
+        parser.print_help()
+        sys.exit(1)
+
+    indent = 2 if args.pretty else None
+    print(json.dumps(result, indent=indent))
+
+
+if __name__ == "__main__":
+    main()

scripts/test_gitea_issue_parser.py

@@ -0,0 +1,111 @@
+#!/usr/bin/env python3
+"""Tests for gitea_issue_parser."""
+
+import json
+import sys
+import os
+
+sys.path.insert(0, os.path.dirname(__file__))
+from gitea_issue_parser import parse_issue_body
+
+
+def test_basic_structure():
+    body = """## Context
+This is the background.
+
+## Acceptance Criteria
+- [ ] First criterion
+- [x] Second criterion (already done)
+- [ ] Third criterion
+
+## Labels
+`pipeline`, `extraction`
+"""
+    result = parse_issue_body(body, "Test Issue", ["pipeline", "extraction"])
+    assert result["title"] == "Test Issue"
+    assert "background" in result["context"].lower()
+    assert len(result["criteria"]) == 3
+    assert "First criterion" in result["criteria"]
+    assert result["labels"] == ["pipeline", "extraction"]
+    print("PASS: test_basic_structure")
+
+
+def test_epic_ref():
+    body = "Closes #645\n\nSome description."
+    result = parse_issue_body(body, "feat: thing (#688)")
+    assert result["epic_ref"] == "#645"
+    print("PASS: test_epic_ref")
+
+
+def test_epic_ref_from_title():
+    body = "Some description without close ref."
+    result = parse_issue_body(body, "feat: scene descriptions (#645)")
+    assert result["epic_ref"] == "#645"
+    print("PASS: test_epic_ref_from_title")
+
+
+def test_no_checkboxes():
+    body = """## Requirements
+1. First thing
+2. Second thing
+3. Third thing
+"""
+    result = parse_issue_body(body)
+    assert len(result["criteria"]) == 3
+    print("PASS: test_no_checkboxes")
+
+
+def test_empty_body():
+    result = parse_issue_body("", "Empty Issue")
+    assert result["title"] == "Empty Issue"
+    assert result["criteria"] == []
+    assert result["context"] == ""
+    print("PASS: test_empty_body")
+
+
+def test_real_issue_format():
+    body = """Closes #681
+
+## Changes
+
+Add `#!/usr/bin/env python3` shebang to 6 Python scripts.
+
+## Verification
+
+All 6 files confirmed missing shebangs before fix.
+
+## Impact
+
+Scripts can now be executed directly.
+"""
+    result = parse_issue_body(body, "fix: add python3 shebangs (#685)")
+    assert result["epic_ref"] == "#681"
+    assert "shebang" in result["context"].lower()
+    print("PASS: test_real_issue_format")
+
+
+def test_all_sections_captured():
+    body = """## Context
+Background info.
+
+## Acceptance Criteria
+- [ ] Do thing
+
+## Labels
+`test`
+"""
+    result = parse_issue_body(body)
+    assert "context" in result["sections"]
+    assert "acceptance criteria" in result["sections"]
+    print("PASS: test_all_sections_captured")
+
+
+if __name__ == "__main__":
+    test_basic_structure()
+    test_epic_ref()
+    test_epic_ref_from_title()
+    test_no_checkboxes()
+    test_empty_body()
+    test_real_issue_format()
+    test_all_sections_captured()
+    print("\nAll tests passed.")