fix: docs: verify epic slice for #582 on main (closes #789) (closes #790)

docs/issue-582-verification.md

@@ -1,63 +1,73 @@
 # Issue #582 Verification — Parent-Epic Orchestration Slice
 
-**Date:** 2026-04-14
-**Status:** ✅ Verified on `main`
-**Refs:** #582, #789, #795
+**Date:** 2026-04-20
+**Status:** Slice already present on `main`; epic remains open for full archive consumption.
 
-## Summary
+## What #582 asked for
 
-The parent-epic orchestration slice for the Know Thy Father multimodal pipeline is **already implemented on `main`**. This document records the verification evidence.
+A single orchestration script that stitches the five Know Thy Father phases together
+into one reviewable plan — not a replacement for individual scripts, but a spine
+that future passes can run, resume, and verify.
 
-## What Exists
+## What exists on `main`
 
-### Core Orchestrator
-- **File:** `scripts/know_thy_father/epic_pipeline.py`
-- **Function:** Builds and executes a 5-phase pipeline plan; reports status snapshots
-- **Phases:**
-  1. Media Indexing (`scripts/know_thy_father/index_media.py`)
-  2. Multimodal Analysis (`scripts/twitter_archive/analyze_media.py`)
-  3. Holographic Synthesis (`scripts/know_thy_father/synthesize_kernels.py`)
-  4. Cross-Reference Audit (`scripts/know_thy_father/crossref_audit.py`)
-  5. Processing Log (`twitter-archive/know-thy-father/tracker.py`)
+| Artifact | Path | Present |
+|----------|------|---------|
+| Epic pipeline runner | `scripts/know_thy_father/epic_pipeline.py` | ✅ |
+| Pipeline documentation | `docs/KNOW_THY_FATHER_MULTIMODAL_PIPELINE.md` | ✅ |
+| Phase 1 — Media Indexing | `scripts/know_thy_father/index_media.py` | ✅ |
+| Phase 2 — Multimodal Analysis | `scripts/twitter_archive/analyze_media.py` | ✅ |
+| Phase 3 — Holographic Synthesis | `scripts/know_thy_father/synthesize_kernels.py` | ✅ |
+| Phase 4 — Cross-Reference Audit | `scripts/know_thy_father/crossref_audit.py` | ✅ |
+| Phase 5 — Processing Log | `twitter-archive/know-thy-father/tracker.py` | ✅ |
 
-### Supporting Scripts (All Present)
-| Script | Purpose |
-|--------|---------|
-| `scripts/know_thy_father/index_media.py` | Phase 1 — scan tweets, emit media manifest |
-| `scripts/twitter_archive/analyze_media.py` | Phase 2 — batch multimodal analysis |
-| `scripts/know_thy_father/synthesize_kernels.py` | Phase 3 — kernels → Father's Ledger |
-| `scripts/know_thy_father/crossref_audit.py` | Phase 4 — cross-reference against SOUL.md |
-| `twitter-archive/know-thy-father/tracker.py` | Phase 5 — processing log / status report |
-
-### Test Coverage
-- `tests/test_know_thy_father_pipeline.py` — orchestrator structure
-- `tests/test_know_thy_father_index.py` — Phase 1 logic
-- `tests/test_know_thy_father_synthesis.py` — Phase 3 logic
-- `tests/test_know_thy_father_crossref.py` — Phase 4 logic
-- `tests/twitter_archive/test_ktf_tracker.py` — Phase 5 tracker
-- `tests/twitter_archive/test_analyze_media.py` — Phase 2 analysis
-
-## Why `Refs #582`, Not `Closes #582`
-
-The repo-side operational slice is complete, but the parent epic (#582) also encompasses:
-- Full archive consumption (not yet finished)
-- Downstream memory integration (pending)
-
-Therefore #582 remains open; this evidence trail closes the verification sub-issues (#789, #795).
-
-## Verification Commands
+## Runner capabilities (all implemented)
 
 ```bash
-# Orchestrator slice tests (10 tests)
-python3 -m pytest tests/test_issue_582_verification.py -q
+# Print the orchestrated plan
+python3 scripts/know_thy_father/epic_pipeline.py
 
-# Full Know Thy Father suite (71 tests)
-python3 -m pytest \
-  tests/test_know_thy_father_pipeline.py \
-  tests/test_know_thy_father_index.py \
-  tests/test_know_thy_father_synthesis.py \
-  tests/test_know_thy_father_crossref.py \
-  tests/twitter_archive/test_ktf_tracker.py \
-  tests/twitter_archive/test_analyze_media.py \
-  -q
+# JSON status snapshot of scripts + known artifact paths
+python3 scripts/know_thy_father/epic_pipeline.py --status --json
+
+# Execute one concrete step
+python3 scripts/know_thy_father/epic_pipeline.py --run-step phase2_multimodal_analysis --batch-size 10
 ```
+
+## Test coverage
+
+The following test suites confirm the orchestration slice is intact:
+
+- `tests/test_know_thy_father_pipeline.py` — pipeline plan structure, status snapshot, doc presence
+- `tests/test_know_thy_father_index.py` — Phase 1 media indexing logic
+- `tests/test_know_thy_father_synthesis.py` — Phase 3 kernel synthesis
+- `tests/test_know_thy_father_crossref.py` — Phase 4 cross-reference audit
+- `tests/twitter_archive/test_ktf_tracker.py` — Phase 5 processing tracker
+- `tests/twitter_archive/test_analyze_media.py` — Phase 2 multimodal analysis
+
+Run all with:
+
+```bash
+python3 -m pytest tests/test_know_thy_father_pipeline.py tests/test_know_thy_father_index.py tests/test_know_thy_father_synthesis.py tests/test_know_thy_father_crossref.py tests/twitter_archive/test_ktf_tracker.py tests/twitter_archive/test_analyze_media.py -q
+```
+
+## Why Refs #582, not Closes #582
+
+The **repo-side orchestration slice** is fully implemented on `main`. However, the
+parent epic itself remains open because:
+
+1. The local Twitter archive has not been fully consumed through all five phases.
+2. Downstream memory/fact-store integration is not yet wired end-to-end.
+3. The processing log (`PROCESSING_LOG.md`) reflects halted progress that has not resumed.
+
+This PR adds durable verification evidence without overstating closure.
+
+## Historical trail
+
+- Parent-epic PR that landed the orchestration slice: [closed on main]
+- This verification document: added by #789, superseded by this PR #790.
+
+## Linked issues
+
+- Refs #582 (parent epic — remains open)
+- Closes #789 (verification task — closed by this PR)

tests/test_issue_582_verification.py

@@ -1,94 +1,146 @@
-"""Verification tests for issue #582 — parent-epic orchestration slice on main.
+"""Durable verification that the Issue #582 parent-epic orchestration slice exists on main.
 
-These 10 tests confirm that the Know Thy Father epic pipeline orchestrator
-and all five phase scripts are present and structurally correct on main.
+These tests confirm:
+  1. The epic pipeline runner script is present and importable.
+  2. The pipeline documentation is committed.
+  3. All five phase scripts exist at their expected paths.
+  4. The pipeline plan exposes the correct five phases in order.
+  5. Each plan step references the correct underlying script.
+  6. The status snapshot reports script_exists=True for all phases.
+  7. The status snapshot includes expected artifact output paths.
+  8. The runner can produce a JSON-serialisable plan.
+  9. The runner can produce a JSON-serialisable status snapshot.
+ 10. The verification document itself is present.
 
-Refs #582 | Closes #789 | Closes #795
+Refs #582.  Closes #789.
 """
 
 import importlib.util
+import json
+import unittest
 from pathlib import Path
 
-import pytest
-
 
 ROOT = Path(__file__).resolve().parent.parent
 EPIC_PIPELINE = ROOT / "scripts" / "know_thy_father" / "epic_pipeline.py"
+PIPELINE_DOC = ROOT / "docs" / "KNOW_THY_FATHER_MULTIMODAL_PIPELINE.md"
+VERIFICATION_DOC = ROOT / "docs" / "issue-582-verification.md"
+
+EXPECTED_PHASES = [
+    "phase1_media_indexing",
+    "phase2_multimodal_analysis",
+    "phase3_holographic_synthesis",
+    "phase4_cross_reference_audit",
+    "phase5_processing_log",
+]
+
+EXPECTED_SCRIPTS = {
+    "phase1_media_indexing": "scripts/know_thy_father/index_media.py",
+    "phase2_multimodal_analysis": "scripts/twitter_archive/analyze_media.py",
+    "phase3_holographic_synthesis": "scripts/know_thy_father/synthesize_kernels.py",
+    "phase4_cross_reference_audit": "scripts/know_thy_father/crossref_audit.py",
+    "phase5_processing_log": "twitter-archive/know-thy-father/tracker.py",
+}
+
+EXPECTED_OUTPUTS = {
+    "phase1_media_indexing": ["twitter-archive/know-thy-father/media_manifest.jsonl"],
+    "phase3_holographic_synthesis": ["twitter-archive/knowledge/fathers_ledger.jsonl"],
+    "phase5_processing_log": ["twitter-archive/know-thy-father/REPORT.md"],
+}
 
 
-def _load(name: str, path: Path):
-    spec = importlib.util.spec_from_file_location(name, path)
-    assert spec and spec.loader, f"cannot load {path}"
+def _load_epic_module():
+    spec = importlib.util.spec_from_file_location("ktf_epic_pipeline", EPIC_PIPELINE)
+    assert spec and spec.loader, "Cannot load epic_pipeline module spec"
     mod = importlib.util.module_from_spec(spec)
     spec.loader.exec_module(mod)
     return mod
 
 
-class TestEpicPipelineExists:
-    def test_epic_pipeline_script_exists(self):
-        """epic_pipeline.py must be committed on main."""
-        assert EPIC_PIPELINE.exists(), "scripts/know_thy_father/epic_pipeline.py missing"
+class TestIssue582Verification(unittest.TestCase):
+    """10-test suite proving the #582 orchestration slice is on main."""
 
+    # -- existence checks --------------------------------------------------
 
-class TestEpicPipelineAPI:
-    @pytest.fixture(autouse=True)
-    def _mod(self):
-        self.mod = _load("epic_pipeline", EPIC_PIPELINE)
+    def test_01_epic_pipeline_script_exists(self):
+        """The orchestration runner is committed."""
+        self.assertTrue(EPIC_PIPELINE.exists(), f"missing {EPIC_PIPELINE.relative_to(ROOT)}")
 
-    def test_has_build_pipeline_plan(self):
-        assert hasattr(self.mod, "build_pipeline_plan")
+    def test_02_pipeline_documentation_exists(self):
+        """The multimodal pipeline doc is committed."""
+        self.assertTrue(PIPELINE_DOC.exists(), "missing KNOW_THY_FATHER_MULTIMODAL_PIPELINE.md")
 
-    def test_has_build_status_snapshot(self):
-        assert hasattr(self.mod, "build_status_snapshot")
+    def test_03_all_phase_scripts_exist_on_disk(self):
+        """Every script referenced by the pipeline exists in the repo."""
+        for phase_id, script_rel in EXPECTED_SCRIPTS.items():
+            path = ROOT / script_rel
+            self.assertTrue(path.exists(), f"{phase_id}: missing {script_rel}")
 
+    # -- plan structure ----------------------------------------------------
 
-class TestPipelinePlanStructure:
-    @pytest.fixture(autouse=True)
-    def _mod(self):
-        self.mod = _load("epic_pipeline", EPIC_PIPELINE)
-
-    def test_plan_has_five_phases(self):
-        plan = self.mod.build_pipeline_plan(batch_size=10)
-        assert len(plan) == 5
-
-    def test_phase_ids_in_order(self):
-        plan = self.mod.build_pipeline_plan(batch_size=10)
+    def test_04_pipeline_plan_has_five_phases_in_order(self):
+        mod = _load_epic_module()
+        plan = mod.build_pipeline_plan(batch_size=10)
         ids = [step["id"] for step in plan]
-        assert ids == [
-            "phase1_media_indexing",
-            "phase2_multimodal_analysis",
-            "phase3_holographic_synthesis",
-            "phase4_cross_reference_audit",
-            "phase5_processing_log",
-        ]
+        self.assertEqual(ids, EXPECTED_PHASES)
+
+    def test_05_plan_commands_reference_correct_scripts(self):
+        mod = _load_epic_module()
+        plan = mod.build_pipeline_plan(batch_size=10)
+        for step in plan:
+            expected_script = EXPECTED_SCRIPTS[step["id"]]
+            self.assertIn(
+                expected_script,
+                step["command"],
+                f"{step['id']} command missing {expected_script}",
+            )
+
+    # -- status snapshot ---------------------------------------------------
+
+    def test_06_status_snapshot_all_scripts_exist(self):
+        mod = _load_epic_module()
+        status = mod.build_status_snapshot(ROOT)
+        for phase_id in EXPECTED_PHASES:
+            self.assertIn(phase_id, status)
+            self.assertTrue(
+                status[phase_id]["script_exists"],
+                f"{phase_id} script_exists should be True",
+            )
+
+    def test_07_status_snapshot_reports_expected_outputs(self):
+        mod = _load_epic_module()
+        status = mod.build_status_snapshot(ROOT)
+        for phase_id, expected_paths in EXPECTED_OUTPUTS.items():
+            actual_paths = [o["path"] for o in status[phase_id]["outputs"]]
+            for p in expected_paths:
+                self.assertIn(p, actual_paths, f"{phase_id} missing output path {p}")
+
+    # -- JSON serialisation ------------------------------------------------
+
+    def test_08_plan_is_json_serialisable(self):
+        mod = _load_epic_module()
+        plan = mod.build_pipeline_plan(batch_size=10)
+        dumped = json.dumps(plan)
+        restored = json.loads(dumped)
+        self.assertEqual(len(restored), 5)
+
+    def test_09_status_snapshot_is_json_serialisable(self):
+        mod = _load_epic_module()
+        status = mod.build_status_snapshot(ROOT)
+        dumped = json.dumps(status)
+        restored = json.loads(dumped)
+        for phase_id in EXPECTED_PHASES:
+            self.assertIn(phase_id, restored)
+
+    # -- verification doc --------------------------------------------------
+
+    def test_10_verification_document_exists(self):
+        """This verification trail is committed."""
+        self.assertTrue(
+            VERIFICATION_DOC.exists(),
+            "missing docs/issue-582-verification.md",
+        )
 
 
-class TestPhaseScriptsExist:
-    def test_all_four_phase_scripts_exist(self):
-        expected = [
-            "scripts/know_thy_father/index_media.py",
-            "scripts/twitter_archive/analyze_media.py",
-            "scripts/know_thy_father/synthesize_kernels.py",
-            "scripts/know_thy_father/crossref_audit.py",
-        ]
-        for rel in expected:
-            assert (ROOT / rel).exists(), f"{rel} missing"
-
-    def test_tracker_script_exists(self):
-        candidates = [
-            ROOT / "twitter-archive" / "know-thy-father" / "tracker.py",
-            ROOT / "scripts" / "know_thy_father" / "tracker.py",
-        ]
-        assert any(c.exists() for c in candidates), "tracker.py not found"
-
-
-class TestPipelineDocs:
-    def test_pipeline_design_doc_exists(self):
-        assert (ROOT / "docs" / "KNOW_THY_FATHER_MULTIMODAL_PIPELINE.md").exists()
-
-    def test_verification_doc_exists(self):
-        assert (ROOT / "docs" / "issue-582-verification.md").exists()
-
-    def test_verification_doc_mentions_epic(self):
-        text = (ROOT / "docs" / "issue-582-verification.md").read_text()
-        assert "#582" in text
+if __name__ == "__main__":
+    unittest.main()