#!/usr/bin/env python3
"""path_claim_check.py -- does every in-namespace path this archive names exist
at this revision?

WHY THIS EXISTS
  `citation_resolve_check.py` grades `<path>:<line>` citations: it can resolve
  them because a line number makes the claim decidable. A path named WITHOUT a
  line -- "`recoveries/foo.md` (10,232 B)", "the record `docs/bar.md`" -- is the
  same kind of claim with one fewer field, and nothing here graded it. The
  defect is on the record: checkins/2026-10-19-volt.md records a report that
  "named `check.sh` and `files.sha256`" and never wrote them, and the ledger's
  rule is that a reader must be able to go and look. A path a reader cannot
  open is a citation that fails the only test a stranger can run.

  The measurement that made this tool possible also bounded it. Over the whole
  corpus, most distinct path tokens are not candidates for "this archive holds
  that file" at all: they begin with a component this repo does not track --
  scratch trees (`after/verify.txt`), host artifacts (`/root/crucible/...`),
  other repositories, served routes and URLs. Grading those as missing would be
  hundreds of false defects, so they are COUNTED and never graded. What is left
  is small enough for a reader: every MISSING path at the revision the ledger
  was last reviewed against is in `tools/path_claim_baseline.tsv` with the class
  that explains it, and `--fail-on-missing` fails on any the ledger does not
  name -- so a revision that moves the corpus reds the gate until a reader has
  looked. The census print is what says which revision that was.

  This docstring deliberately carries no census figures. The first draft did,
  and the leading-slash fix in the matcher moved every one of them (1,024
  distinct tokens became 1,036) -- the defect class this archive grades as a
  stale claim. The tool prints its own census on every run; that print is the
  only place those numbers live.

WHAT IT MEASURES -- one verdict per DISTINCT path (unit: distinct paths)
  EXISTS    the path is in the tree at --rev.
  DOCREL    the path is NOT in the tree at the repo root, but it resolves
            relative to the document that cites it -- an artifact's own
            `tools/fetch.py` named beside its `REPORT.md`. Printed with the
            path it resolves to and the cite that resolved it, so a reader can
            go and look at the base the tool used. Counted as resolved, not as
            a defect: the file it names exists.
  MISSING   the path's first component IS a tracked top-level entry, so the
            prose is naming something inside this repository, and neither base
            holds it -- not the tree at --rev, and not the citing document's
            own directory. This is the graded class.
  EXTERNAL  the first component is not tracked here (scratch tree, host path,
            URL, other repo). OUT OF SCOPE: the path may be perfectly true and
            this tool cannot see it. Counted, never graded -- the same
            vocabulary `citation_resolve_check.py` uses for EXTERNAL_PATH.
  TEMPLATE  the path carries a placeholder (`YYYY`, `<name>`, `{n}`), so it is
            a pattern rather than a claim about one file.

  Occurrences are counted beside distinct paths, because one wrong path cited
  16 times is one defect and 16 chances for a reader to trip on it.

WHY THERE ARE TWO BASES, AND WHY THE ROOT ONE WINS
  The corpus writes most paths relative to the repository root, so the root base
  is the contract and it is tried first, always. But an artifact inside
  `wizard-work/` writes its own inventory the way a reader sitting in that
  directory would: `tools/fetch.py` beside `REPORT.md` is the artifact's fetch
  script. Graded at the root alone, exactly those citations read MISSING while
  the files sat in that artifact's own `tools/` directory (the live case:
  `wizard-work/sphinx/longctx-needle-grounding-2026-10-22/REPORT.md:49-52`) --
  a tool reporting a base mismatch as a defect teaches writers to stop naming
  files, which is the worse failure. So the second base exists, it is used ONLY
  when the root base missed, and the verdict it produces is a different word
  (DOCREL) that carries the resolved path and the resolving cite. A root-
  relative citation is never rewritten by it, and a token that resolves under
  neither base is still MISSING.

A FENCED BLOCK IS QUOTED MATERIAL, NOT THIS DOCUMENT'S PROSE -- the rule
  `queue_index.py`, `checkin_check.py` and `citation_resolve_check.py` already
  apply to headings and citations. A fenced listing that quotes a path is not
  the containing document asserting that path exists. Skipped lines are printed
  on every run, never silently dropped.

THE BASELINE, AND WHY THE FIRST CENSUS IS ITS LEDGER
  `<path>\\t<CLASS>\\t<reason>`, one row per reviewed missing path, class from a
  closed set (PLANTED, NOT_ADDED, RENAMED, SHORT_NAME, FOREIGN_TREE, EXAMPLE,
  TARGET).
  A row is a stated reading, not an exemption: `--fail-on-missing` fails only on
  a MISSING path the ledger does not name, and every tolerated row is still
  PRINTED with its class on every run, so a reader sees the tolerated rows
  rather than trusting that there are none. A ledger row whose path now exists
  is reported CLEARED, and a row naming a path the census no longer produces is
  reported as a dead row -- the ledger cannot rot invisibly.

  TARGET is the seventh class, added 2026-10-21 for the case the first six did
  not cover: a path named in an IMPERATIVE sentence -- "Post
  `checkins/2026-10-21-spark.md` (Did / Blocked / Next / Ask)" -- which asks its
  owner to create the file rather than asserting that one exists. `git log --all
  --diff-filter=A -- checkins/2026-10-21-spark.md` returns 0 commits, so no ref
  ever held it, and the artifact is genuinely owed; but a repository whose
  deliverables cannot be named without reddening the census teaches writers to
  stop naming them, which is the worse failure. The row must name the asking
  file and line and the zero-commit reading, so it is a citation a reader can
  check rather than a word that closes a red. A claim that something EXISTS is
  still MISSING until the file does.

EXIT CODES
  0  the census ran. Without --fail-on-missing a MISSING path is not a fault.
  1  --fail-on-missing and at least one MISSING path is not in the baseline, or
     names a class the ledger does not tolerate.
  2  could not grade: not a git work tree, a revision that does not resolve, a
     corpus with no markdown, a NAMED baseline that cannot be read or names an
     unknown class. Never 0, and never a silent empty census. A repo that
     carries no ledger of its own is not this: `--baseline` defaults to
     tools/path_claim_baseline.tsv resolved against --repo, and a tree without
     that file is graded with no ledger after one note on stderr.

USAGE
  python3 path_claim_check.py [--repo DIR] [--rev HEAD] [--corpus PATH...]
      [--json OUT] [--only missing|exists|docrel|external|template|all]
      [--context] [--fail-on-missing] [--baseline PATH]
      --baseline defaults to tools/path_claim_baseline.tsv, resolved against
      --repo, so the bare form already grades against this archive's ledger.
  python3 path_claim_check.py --selftest      # 27 cells
  python3 path_claim_check.py --repo . --fail-on-missing
  python3 path_claim_check.py --repo . --only docrel --context
"""
import argparse
import collections
import json
import os
import posixpath
import re
import subprocess
import sys

DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# The ledger this archive carries, REPO-relative: resolved against --repo, not
# against the caller's cwd, so a bare run against a clone reads that clone's
# ledger. Until 2026-10-21 this was advertised in USAGE and never applied --
# `--baseline` had no default, so a bare invocation graded against an empty
# ledger and, under --fail-on-missing, returned rc 1 for an archive whose own
# rows excuse seven of the ten MISSING paths. A default a reader can cite and
# the code does not apply is the worst kind: it makes the tool lie in the
# direction of a defect.
DEFAULT_BASELINE = "tools/path_claim_baseline.tsv"

# The extensions this archive actually names. A closed list, for the reason
# citation_resolve_check.py gives: an open `\\.[a-z]+` swallows clock times,
# decimals and domain names.
EXT = (r"md|markdown|py|json|txt|js|mjs|cjs|sh|bash|yml|yaml|html|css|csv|tsv|"
       r"jsonl|sql|db|sqlite|age|wav|mp3|mp4|gz|tgz|toml|cfg|ini|log|out|bak")

# At least one '/' is REQUIRED: a bare filename ("index.html") is not a claim
# about a location, and the archive names dozens of those for pages it does not
# hold. The trailing guard stops `x.md` from matching the front of `x.md5`.
PATH = re.compile(r"(?P<path>/?(?:[A-Za-z0-9_.@+-]+/)+[A-Za-z0-9_.@+-]+\.(?:%s))"
                  r"(?![A-Za-z0-9])" % EXT)
FENCE = re.compile(r"^\s*(```|~~~)")
HOST = re.compile(r"^(?:/|~/|\.\./)")
TEMPLATE = re.compile(r"YYYY|<[^>]*>|\{[^}]*\}|NNN")
STRIP = "`*'\"()[]<>,;:."

# The closed class set a baseline row may name. Each is a reason a MISSING path
# is not a live defect; adding one is a deliberate edit to this tuple.
# EXAMPLE was added after the tool's own construction record landed and the next
# census found 3 new MISSING paths -- two quoted metasyntactic illustrations
# ("recoveries/foo.md", "the record docs/bar.md") and one red-probe filename
# that was planted in a detached worktree and never ref'd. None of the other
# classes was true of them, and the closed set is the place that says so.
# TARGET was added 2026-10-21 by the same rule: the census's next tick found
# `checkins/2026-10-21-spark.md`, cited by replies/2026-10-21-to-spark-06.md:26
# in an imperative ("Post ... (Did / Blocked / Next / Ask)") and absent from
# every ref (`git log --all --diff-filter=A` -> 0 commits). It is not a broken
# citation; it is a deliverable somebody owes, which is a different thing.
BASELINE_CLASSES = ("PLANTED", "NOT_ADDED", "RENAMED", "SHORT_NAME",
                    "FOREIGN_TREE", "EXAMPLE", "TARGET")
# DOCREL sits between the two graded outcomes on purpose: it is a resolution,
# reported beside EXISTS rather than folded into it, because the base it used is
# part of the claim and a reader has to be able to see that.
VERDICTS = ("EXISTS", "DOCREL", "MISSING", "EXTERNAL", "TEMPLATE")
# Which resolution wins when one path is cited from more than one document: a
# root hit beats a document-relative hit beats no hit. EXTERNAL/TEMPLATE never
# meet the others (they are a property of the token, not of the tree).
RESOLUTION_RANK = {"MISSING": 0, "DOCREL": 1, "EXISTS": 2}


class CannotGrade(Exception):
    """An input this tool refuses to grade: the caller turns it into rc 2."""


def git(repo, *args):
    return subprocess.run(["git", "-C", repo, *args], capture_output=True)


def precondition(repo, rev):
    """None when `repo` is a git work tree carrying `rev`, else why not.

    Every reader below swallows git's status, so an unreadable repo or a
    mistyped revision would come back as an empty tree and print a clean census
    of nothing -- the silent-pass class selftest_sweep.py's malformed axis
    exists to catch.
    """
    if not os.path.isdir(repo):
        return "cannot read repo %s: no such directory" % repo
    p = git(repo, "rev-parse", "--is-inside-work-tree")
    if p.returncode != 0 or p.stdout.strip() != b"true":
        return "cannot read repo %s: not a git work tree" % repo
    q = git(repo, "rev-parse", "--verify", rev + "^{commit}")
    if q.returncode != 0:
        why = q.stderr.decode("utf-8", "replace").strip().splitlines()
        return "cannot read %s at %s: %s" % (repo, rev, why[-1] if why else "git rev-parse failed")
    return None


def clean(token):
    """Strip the punctuation a path wears when it sits in prose or in a span."""
    p = token.strip(STRIP)
    while p.startswith("./"):
        p = p[2:].strip(STRIP)
    return p


def classify(path, tracked, tops):
    """The verdict at the repo-root base alone. DOCREL needs a second base."""
    if TEMPLATE.search(path):
        return "TEMPLATE"
    if HOST.match(path) or path.split("/")[0] not in tops:
        return "EXTERNAL"
    return "EXISTS" if path in tracked else "MISSING"


def doc_relative(path, where, tracked):
    """The tracked path this document's own directory resolves `path` to, or None.

    `where` is the citing document's repo-relative name; a document at the root
    has no second base distinct from the first, so it returns None and root
    documents keep the single-base contract. A candidate that climbs out of the
    tree is not a base this tool will stand on.
    """
    if not where or "/" not in where:
        return None
    cand = posixpath.normpath(posixpath.join(posixpath.dirname(where), path))
    if cand == ".." or cand.startswith("../"):
        return None
    return cand if cand in tracked else None


def fence_mask(text):
    """1-based line numbers that are a fence marker or sit inside a fence."""
    out, in_fence = set(), False
    for i, raw in enumerate(text.split("\n"), 1):
        if FENCE.match(raw):
            out.add(i)
            in_fence = not in_fence
            continue
        if in_fence:
            out.add(i)
    return out


def scan_text(text, where, tracked, tops):
    """Rows for one document -> ({path: row}, counter).

    `where` is the document's name, used for the first-cite handle and as the
    second base: a token the root base misses is tried beside the document
    (`doc_relative`) before it is called MISSING. Line numbers are the
    document's own, so a row can be read back at --rev.
    """
    rows, stats = {}, collections.Counter()
    mask = fence_mask(text)
    stats["fenced_lines"] = len(mask)
    for i, line in enumerate(text.split("\n"), 1):
        if i in mask:
            continue
        for m in PATH.finditer(line):
            p = clean(m.group("path"))
            if not p:
                continue
            stats["occurrences"] += 1
            verdict = classify(p, tracked, tops)
            resolved = None
            if verdict == "MISSING":
                resolved = doc_relative(p, where, tracked)
                if resolved:
                    verdict = "DOCREL"
            row = rows.get(p)
            if row is None:
                row = rows[p] = {"path": p, "mentions": 0,
                                 "first": "%s:%d" % (where, i),
                                 "context": line.strip()[:180],
                                 "verdict": verdict,
                                 "resolved": resolved,
                                 "resolved_from": ("%s:%d" % (where, i)
                                                   if resolved else None),
                                 "unresolved": 0}
            row["mentions"] += 1
            if verdict == "MISSING":
                # One resolving cite does not erase the cites that did not
                # resolve at their own base; the count travels with the row.
                row["unresolved"] += 1
            if (RESOLUTION_RANK.get(verdict, -1) >
                    RESOLUTION_RANK.get(row["verdict"], -1)):
                row["verdict"] = verdict
                if resolved:
                    row["resolved"] = resolved
                    row["resolved_from"] = "%s:%d" % (where, i)
    return rows, stats


def scan(repo, rev, corpus, store):
    """Every document's rows, merged by path -- the same scan_text() the fixtures
    are graded by, so no second matcher exists to drift from this one."""
    all_rows, stats = {}, collections.Counter()
    for f in corpus:
        text = store.read(f).decode("utf-8", "replace")
        rows, st = scan_text(text, f, store.tree, store.tops)
        stats.update(st)
        for p, row in rows.items():
            if p in all_rows:
                cur = all_rows[p]
                cur["mentions"] += row["mentions"]
                cur["unresolved"] += row["unresolved"]
                if (RESOLUTION_RANK.get(row["verdict"], -1) >
                        RESOLUTION_RANK.get(cur["verdict"], -1)):
                    cur["verdict"] = row["verdict"]
                    cur["resolved"] = row.get("resolved")
                    cur["resolved_from"] = row.get("resolved_from")
            else:
                all_rows[p] = row
    return all_rows, stats


class Store:
    def __init__(self, repo, rev):
        self.repo, self.rev = repo, rev
        self.tree = set(git(repo, "ls-tree", "-r", "--name-only", rev)
                        .stdout.decode("utf-8", "replace").splitlines())
        self.tops = {p.split("/")[0] for p in self.tree}

    def read(self, path):
        p = git(self.repo, "show", "%s:%s" % (self.rev, path))
        if p.returncode != 0:
            raise CannotGrade("cannot read %s at %s" % (path, self.rev))
        return p.stdout


def load_baseline(path):
    """`<path>\\t<CLASS>\\t<reason>` -> {path: (class, reason)}. Unreadable is rc 2."""
    try:
        with open(path, "r", encoding="utf-8") as fh:
            text = fh.read()
    except OSError as exc:
        raise CannotGrade("cannot read baseline %s: %s" % (path, exc))
    except UnicodeDecodeError as exc:
        raise CannotGrade("cannot read baseline %s: not UTF-8: %s" % (path, exc))
    out = {}
    for n, line in enumerate(text.splitlines(), 1):
        s = line.rstrip("\r")
        if not s.strip() or s.lstrip().startswith("#"):
            continue
        parts = s.split("\t")
        if len(parts) != 3:
            raise CannotGrade("cannot read baseline %s:%d: not "
                              "<path>\\t<CLASS>\\t<reason>: %r" % (path, n, s))
        name, cls, reason = (x.strip() for x in parts)
        if not name or not reason:
            raise CannotGrade("cannot read baseline %s:%d: empty path or reason: %r"
                              % (path, n, s))
        if cls not in BASELINE_CLASSES:
            raise CannotGrade("cannot read baseline %s:%d: unknown class %r "
                              "(known: %s)" % (path, n, cls, ", ".join(BASELINE_CLASSES)))
        out[name] = (cls, reason)
    return out


def apply_baseline(rows, ledger):
    """Attach the ledger to each MISSING row; report what moved on both sides."""
    missing = {p: r for p, r in rows.items() if r["verdict"] == "MISSING"}
    for p, row in missing.items():
        if p in ledger:
            row["baseline"] = {"known": True, "class": ledger[p][0],
                               "reason": ledger[p][1], "new": False}
        else:
            row["baseline"] = {"known": False, "class": None, "reason": None, "new": True}
    cleared = sorted(p for p in ledger if p in rows and rows[p]["verdict"] == "EXISTS")
    # A row the second base now resolves is cleared too, but it says so: the
    # path is true at the citing document's directory, not at the repo root, and
    # a reader who deletes the row should know which base made it go.
    cleared_docrel = sorted(p for p in ledger
                            if p in rows and rows[p]["verdict"] == "DOCREL")
    dead = sorted(p for p in ledger if p not in rows)
    return {"entries": len(ledger), "matched": sum(1 for r in missing.values()
                                                   if r["baseline"]["known"]),
            "new": sorted(p for p, r in missing.items() if r["baseline"]["new"]),
            "cleared": cleared, "cleared_docrel": cleared_docrel, "dead": dead,
            "tolerated": collections.Counter(r["baseline"]["class"]
                                             for r in missing.values()
                                             if r["baseline"]["known"])}


def report(rows, stats, rev, sha, corpus, ledger, path, only, context):
    counts = collections.Counter(r["verdict"] for r in rows.values())
    print("path_claim_check: %s at %s (%s)" % ("repo", rev, sha[:7]))
    print("matcher: /?(?:[A-Za-z0-9_.@+-]+/)+[A-Za-z0-9_.@+-]+\\.(closed ext list), "
          "at least one '/' required")
    print("  graded only when the first component is a tracked top-level entry; "
          "fenced blocks skipped")
    print("corpus: %d markdown file(s)" % len(corpus))
    print("paths: %d distinct / %d occurrence(s) (unit: distinct paths; "
          "occurrences beside)" % (len(rows), stats["occurrences"]))
    for v in VERDICTS:
        print("  %-9s %4d distinct" % (v, counts[v]))
    print("fenced skipped: %d line(s)" % stats["fenced_lines"])
    if ledger is not None:
        print("baseline %s: %d entries, %d matched, %d new, %d cleared, %d dead row(s)"
              % (path, ledger["entries"], ledger["matched"], len(ledger["new"]),
                 len(ledger["cleared"]), len(ledger["dead"])))
        for cls, n in sorted(ledger["tolerated"].items()):
            print("  tolerated %-13s %d" % (cls, n))
        for p in ledger["cleared"]:
            print("  CLEARED  %s -- ledger row, now in the tree; the row can go" % p)
        for p in ledger["cleared_docrel"]:
            row = rows[p]
            print("  CLEARED_DOCREL %s -- ledger row, now resolves at %s from %s; "
                  "the row can go" % (p, row.get("resolved"), row.get("resolved_from")))
        for p in ledger["dead"]:
            print("  DEAD_ROW %s -- ledger row, no longer in the census; the row can go" % p)

    sel = {"missing": lambda r: r["verdict"] == "MISSING",
           "exists": lambda r: r["verdict"] == "EXISTS",
           "docrel": lambda r: r["verdict"] == "DOCREL",
           "external": lambda r: r["verdict"] == "EXTERNAL",
           "template": lambda r: r["verdict"] == "TEMPLATE",
           "all": lambda r: True}[only]
    show = sorted((r for r in rows.values() if sel(r)),
                  key=lambda r: (-r["mentions"], r["path"]))
    print("--- %s: %d distinct path(s) (showing first 300) ---" % (only, len(show)))
    for r in show[:300]:
        tag = ""
        if r["verdict"] == "MISSING" and r.get("baseline"):
            b = r["baseline"]
            tag = ("  [%s: %s]" % (b["class"], b["reason"])) if b["known"] \
                else "  [NEW -- not in the baseline]"
        elif r["verdict"] == "DOCREL":
            tag = ("  [DOCREL: resolves at %s, from %s; %d cite(s) did not resolve "
                   "at their own base]"
                   % (r.get("resolved"), r.get("resolved_from"), r.get("unresolved", 0)))
        print("  %3dx  %-58s first %s%s" % (r["mentions"], r["path"], r["first"], tag))
        if context:
            print("        ...%s..." % r["context"])


def resolve_baseline(repo, path):
    """The ledger to grade against, or None when this repo carries none.

    `path` is repo-relative by contract (DEFAULT_BASELINE), so it is joined onto
    `repo` before the existence test: a bare run from another cwd must read the
    tree it is grading, never a file beside the caller. An absolute path is
    taken as given. A caller who NAMED a ledger that is not there gets the path
    back unchanged, so load_baseline refuses it with rc 2 -- a mistyped ledger
    must never read as "no ledger". A repo that simply carries none (a fixture
    tree) is a stated reading: one note on stderr, no ledger, and the census
    still prints, because the count is the finding.
    """
    if not path:
        return None
    cand = path if os.path.isabs(path) else os.path.join(repo, path)
    if os.path.exists(cand):
        return cand
    if path == DEFAULT_BASELINE:
        print("note: no ledger at %s -- every MISSING path counts as new" % cand,
              file=sys.stderr)
        return None
    return cand


def main():
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("--repo", default=DEFAULT_REPO)
    ap.add_argument("--rev", default="HEAD")
    ap.add_argument("--corpus", nargs="*")
    ap.add_argument("--json")
    ap.add_argument("--only", choices=["missing", "exists", "docrel", "external",
                                       "template", "all"], default="missing")
    ap.add_argument("--context", action="store_true")
    ap.add_argument("--fail-on-missing", action="store_true")
    ap.add_argument("--baseline", default=DEFAULT_BASELINE,
                    help="ledger to grade MISSING rows against (default: "
                         "%s, resolved against --repo)" % DEFAULT_BASELINE)
    ap.add_argument("--selftest", action="store_true")
    a = ap.parse_args()

    if a.selftest:
        return selftest()

    problem = precondition(a.repo, a.rev)
    if problem:
        print(problem, file=sys.stderr)
        return 2
    try:
        store = Store(a.repo, a.rev)
        sha = git(a.repo, "rev-parse", a.rev).stdout.decode().strip()
        corpus = a.corpus if a.corpus is not None else \
            sorted(p for p in store.tree if p.endswith((".md", ".markdown")))
        if not corpus:
            raise CannotGrade("cannot grade: no markdown corpus at %s in %s"
                              % (a.rev, a.repo))
        ledger = resolve_baseline(a.repo, a.baseline)
        ledger = load_baseline(ledger) if ledger else None
        rows, stats = scan(a.repo, a.rev, corpus, store)
    except CannotGrade as exc:
        print(str(exc), file=sys.stderr)
        return 2
    led = apply_baseline(rows, ledger) if ledger is not None else None
    if led is None:
        # No ledger: every MISSING path is new, and the count is the finding.
        led_view = None
    else:
        led_view = led
    report(rows, stats, a.rev, sha, corpus, led_view, a.baseline, a.only, a.context)

    missing = [r for r in rows.values() if r["verdict"] == "MISSING"]
    new = len(missing) if led is None else len(led["new"])
    print("\npath_claim_check: %d distinct paths, %d missing, %d new against the baseline"
          % (len(rows), len(missing), new))
    if a.json:
        try:
            with open(a.json, "w") as fh:
                json.dump({"repo": a.repo, "rev": a.rev, "sha": sha, "corpus": corpus,
                           "stats": dict(stats), "baseline": led,
                           "paths": sorted(rows.values(), key=lambda r: r["path"])},
                          fh, indent=1)
        except OSError as exc:
            print("cannot write --json " + a.json + ": " + str(exc), file=sys.stderr)
            return 2
        print("wrote %s" % a.json)
    if a.fail_on_missing and new:
        print("FAIL: %d missing path(s) the baseline does not name" % new)
        return 1
    return 0


# --------------------------------------------------------------- self-test
# The fixtures are graded by the same scan_text() the corpus is, so a cell that
# passes here is a statement about the shipped matcher, not about a copy of it.
TRACKED = {"docs/real.md", "tools/t.py", "notes/n.md", "WORK-QUEUE.md"}
TOPS = {"docs", "tools", "notes"}


def _verdicts(text):
    rows, _ = scan_text(text, "fixture.md", TRACKED, TOPS)
    return {p: r["verdict"] for p, r in rows.items()}


def _rows(text):
    return scan_text(text, "fixture.md", TRACKED, TOPS)[0]


def selftest():
    n = bad = 0

    def cell(name, ok, detail=""):
        nonlocal n, bad
        n += 1
        bad += 0 if ok else 1
        print("%s %s%s" % ("PASS" if ok else "FAIL", name,
                           "" if ok else "  <- " + detail))

    cell("c1 a tracked path is EXISTS",
         _verdicts("see docs/real.md for the table") == {"docs/real.md": "EXISTS"},
         str(_verdicts("see docs/real.md for the table")))
    cell("c2 an in-namespace path not in the tree is MISSING",
         _verdicts("the record `docs/absent.md` was written") == {"docs/absent.md": "MISSING"},
         str(_verdicts("the record `docs/absent.md` was written")))
    cell("c3 a component this repo does not track is EXTERNAL, not MISSING",
         _verdicts("kept at crucible/q51/helm-selftest-q51.py") ==
         {"crucible/q51/helm-selftest-q51.py": "EXTERNAL"},
         str(_verdicts("kept at crucible/q51/helm-selftest-q51.py")))
    cell("c4 an absolute host path is EXTERNAL",
         _verdicts("under /root/crucible/q39/gate.py today") ==
         {"/root/crucible/q39/gate.py": "EXTERNAL"},
         str(_verdicts("under /root/crucible/q39/gate.py today")))
    cell("c5 a placeholder path is TEMPLATE, not MISSING",
         _verdicts("one file per day: checkins/YYYY-MM-DD-volt.md") ==
         {"checkins/YYYY-MM-DD-volt.md": "TEMPLATE"},
         str(_verdicts("one file per day: checkins/YYYY-MM-DD-volt.md")))
    cell("c6 a fenced path is quoted material, not a claim",
         _verdicts("```\ndocs/absent.md\n```") == {},
         str(_verdicts("```\ndocs/absent.md\n```")))
    cell("c7 the fence closes: the same path in prose is graded again",
         _verdicts("```\ndocs/absent.md\n```\nsee docs/absent.md") ==
         {"docs/absent.md": "MISSING"},
         str(_verdicts("```\ndocs/absent.md\n```\nsee docs/absent.md")))
    cell("c8 a bare filename with no directory is not a path claim",
         _verdicts("the page index.html and blocks/index.html") ==
         {"blocks/index.html": "EXISTS" if "blocks/index.html" in TRACKED else "EXTERNAL"},
         str(_verdicts("the page index.html and blocks/index.html")))
    cell("c9 trailing punctuation and backticks are stripped",
         list(_verdicts("`docs/absent.md`.").keys()) == ["docs/absent.md"],
         str(_verdicts("`docs/absent.md`.")))
    r = _rows("docs/absent.md once, then docs/absent.md again")
    cell("c10 occurrences aggregate onto one row with its first cite",
         len(r) == 1 and r["docs/absent.md"]["mentions"] == 2
         and r["docs/absent.md"]["first"] == "fixture.md:1", str(r))

    # --- the CLI contract, on a real fixture repo -------------------------
    import tempfile
    td = tempfile.mkdtemp(prefix="pathclaim-")
    repo = os.path.join(td, "repo")
    os.makedirs(os.path.join(repo, "docs"))
    os.makedirs(os.path.join(repo, "tools"))
    with open(os.path.join(repo, "docs", "real.md"), "w") as fh:
        fh.write("real\n")
    with open(os.path.join(repo, "tools", "t.py"), "w") as fh:
        fh.write("# t\n")
    with open(os.path.join(repo, "WORK-QUEUE.md"), "w") as fh:
        fh.write("the record `docs/absent.md` and the real docs/real.md\n")
    subprocess.run(["git", "init", "-q", repo], capture_output=True)
    subprocess.run(["git", "-C", repo, "add", "-A"], capture_output=True)
    subprocess.run(["git", "-C", repo, "-c", "user.email=t@t", "-c", "user.name=t",
                    "commit", "-qm", "fixture"], capture_output=True)

    def run(*args):
        p = subprocess.run([sys.executable, os.path.abspath(__file__), *args],
                           capture_output=True, text=True)
        return p.returncode, p.stdout + p.stderr

    rc, out = run("--repo", repo)
    cell("c11 the census alone is rc 0 even with a missing path",
         rc == 0 and "1 missing" in out, "rc=%d %s" % (rc, out[-200:]))
    rc, out = run("--repo", repo, "--fail-on-missing")
    cell("c12 --fail-on-missing is rc 1 on an unbaselined missing path",
         rc == 1 and "FAIL: 1 missing path" in out, "rc=%d %s" % (rc, out[-200:]))
    bl = os.path.join(td, "baseline.tsv")
    with open(bl, "w") as fh:
        fh.write("# fixture\n" + "docs/absent.md\tPLANTED\tfixture row\n")
    rc, out = run("--repo", repo, "--fail-on-missing", "--baseline", bl)
    cell("c13 a baselined missing path is tolerated, and still printed",
         rc == 0 and "PLANTED" in out and "1 matched" in out,
         "rc=%d %s" % (rc, out[-300:]))
    with open(os.path.join(td, "cleared.tsv"), "w") as fh:
        fh.write("docs/real.md\tRENAMED\tfixture: the row is now in the tree\n")
    rc, out = run("--repo", repo, "--baseline", os.path.join(td, "cleared.tsv"))
    cell("c14 a ledger row whose path now exists is CLEARED and named",
         rc == 0 and "CLEARED" in out and "docs/real.md" in out,
         "rc=%d %s" % (rc, out[-300:]))
    rc, out = run("--repo", td)
    cell("c15 a directory that is not a git work tree is rc 2",
         rc == 2 and "not a git work tree" in out, "rc=%d %s" % (rc, out[-200:]))
    rc, out = run("--repo", repo, "--rev", "nosuchrev")
    cell("c16 a revision that does not resolve is rc 2",
         rc == 2 and "cannot read" in out, "rc=%d %s" % (rc, out[-200:]))
    empty = os.path.join(td, "empty")
    os.makedirs(empty)
    with open(os.path.join(empty, "a.txt"), "w") as fh:
        fh.write("no markdown here\n")
    subprocess.run(["git", "init", "-q", empty], capture_output=True)
    subprocess.run(["git", "-C", empty, "add", "-A"], capture_output=True)
    subprocess.run(["git", "-C", empty, "-c", "user.email=t@t", "-c", "user.name=t",
                    "commit", "-qm", "fixture"], capture_output=True)
    rc, out = run("--repo", empty)
    cell("c17 a corpus with no markdown is rc 2, never an empty clean census",
         rc == 2 and "no markdown corpus" in out, "rc=%d %s" % (rc, out[-200:]))

    # --- the ledger's closed set is enforced, not decorative ----------------
    badcls = os.path.join(td, "badclass.tsv")
    with open(badcls, "w") as fh:
        fh.write("docs/absent.md\tMADE_UP\tfixture: a class the tool does not know\n")
    rc, out = run("--repo", repo, "--fail-on-missing", "--baseline", badcls)
    cell("c18 a ledger row naming a class outside the closed set is rc 2",
         rc == 2 and "unknown class" in out and "MADE_UP" in out,
         "rc=%d %s" % (rc, out[-300:]))
    excls = os.path.join(td, "example.tsv")
    with open(excls, "w") as fh:
        fh.write("docs/absent.md\tEXAMPLE\tfixture: a metasyntactic illustration\n")
    rc, out = run("--repo", repo, "--fail-on-missing", "--baseline", excls)
    cell("c19 the EXAMPLE class is accepted and its row still printed",
         rc == 0 and "tolerated EXAMPLE" in out and "1 matched" in out,
         "rc=%d %s" % (rc, out[-300:]))
    tgtcls = os.path.join(td, "target.tsv")
    with open(tgtcls, "w") as fh:
        fh.write("docs/absent.md\tTARGET\tfixture: an owed deliverable\n")
    rc, out = run("--repo", repo, "--fail-on-missing", "--baseline", tgtcls)
    cell("c21 the TARGET class is accepted and its row still printed",
         rc == 0 and "tolerated TARGET" in out and "1 matched" in out,
         "rc=%d %s" % (rc, out[-300:]))

    # --- the default ledger is the one the tree carries ---------------------
    # Until 2026-10-21 `--baseline` had NO default while USAGE advertised one:
    # a bare run graded against an empty ledger, so `--fail-on-missing` returned
    # rc 1 for a tree whose own rows excused the path. This pins bare == named.
    with open(os.path.join(repo, "tools", "path_claim_baseline.tsv"), "w") as fh:
        fh.write("# fixture: the ledger this tree carries\n"
                 "docs/absent.md\tPLANTED\tfixture row, carried by the tree\n")
    subprocess.run(["git", "-C", repo, "add", "-A"], capture_output=True)
    subprocess.run(["git", "-C", repo, "-c", "user.email=t@t", "-c", "user.name=t",
                    "commit", "-qm", "ledger"], capture_output=True)

    def summary(text):
        return [ln for ln in text.splitlines() if ln.startswith("path_claim_check:")]

    bare_rc, bare_out = run("--repo", repo, "--fail-on-missing")
    named_rc, named_out = run("--repo", repo, "--fail-on-missing", "--baseline",
                              os.path.join(repo, "tools", "path_claim_baseline.tsv"))
    cell("c20 the repo's own ledger is the default: bare run == named run",
         bare_rc == named_rc == 0 and summary(bare_out) == summary(named_out)
         and "1 matched" in bare_out,
         "bare rc=%d %r | named rc=%d %r"
         % (bare_rc, summary(bare_out), named_rc, summary(named_out)))

    # --- the second base: DOCREL, and the root base still winning -------------
    # These are statements about the shipped scan_text()/scan(), not about a
    # copy: the merge is exercised through the same function the corpus uses.
    ART = "wizard-work/art/REPORT.md"
    art_tracked = {"docs/real.md", "tools/t.py", "wizard-work/art/tools/fetch.py"}

    def _doc(text, where, tracked):
        rows, _ = scan_text(text, where, tracked,
                            {"docs", "tools", "notes", "wizard-work"})
        return rows

    r = _doc("inventory: `tools/fetch.py` beside this report", ART, art_tracked)
    cell("c22 a path the root base misses but the document's own directory holds is DOCREL",
         r.get("tools/fetch.py", {}).get("verdict") == "DOCREL"
         and r["tools/fetch.py"].get("resolved") == "wizard-work/art/tools/fetch.py"
         and r["tools/fetch.py"].get("resolved_from") == ART + ":1", str(r))
    r = _doc("inventory: `tools/fetch.py` at the record", "WORK-QUEUE.md", art_tracked)
    cell("c23 the second base is not available to a document at the repo root",
         r.get("tools/fetch.py", {}).get("verdict") == "MISSING", str(r))
    both = art_tracked | {"tools/fetch.py"}
    r = _doc("inventory: `tools/fetch.py`", ART, both)
    cell("c24 a root-relative hit is never rewritten by the second base",
         r.get("tools/fetch.py", {}).get("verdict") == "EXISTS"
         and r["tools/fetch.py"].get("resolved") is None, str(r))
    r = _doc("inventory: `tools/nope.py`", ART, art_tracked)
    cell("c25 a candidate that resolves under neither base is still MISSING",
         r.get("tools/nope.py", {}).get("verdict") == "MISSING", str(r))

    class _FakeStore:
        def __init__(self, tree, docs):
            self.tree, self.docs = set(tree), docs
            self.tops = {p.split("/")[0] for p in self.tree}

        def read(self, path):
            return self.docs[path].encode()

    merged, _ = scan(".", "HEAD", ["WORK-QUEUE.md", ART],
                     _FakeStore(art_tracked,
                                {"WORK-QUEUE.md": "quotes `tools/fetch.py`\n",
                                 ART: "inventory: `tools/fetch.py`\n"}))
    row = merged.get("tools/fetch.py", {})
    cell("c26 one path cited from two documents keeps the resolving base and counts both",
         row.get("verdict") == "DOCREL" and row.get("mentions") == 2
         and row.get("unresolved") == 1 and row.get("resolved_from") == ART + ":1",
         str(row))

    # --- DOCREL end to end, through the CLI ----------------------------------
    os.makedirs(os.path.join(repo, "wizard-work", "art", "tools"))
    with open(os.path.join(repo, "wizard-work", "art", "tools", "fetch.py"), "w") as fh:
        fh.write("# fetch\n")
    with open(os.path.join(repo, "wizard-work", "art", "REPORT.md"), "w") as fh:
        fh.write("## Evidence chain\n\n- `tools/fetch.py` -- fetch + normalise\n")
    subprocess.run(["git", "-C", repo, "add", "-A"], capture_output=True)
    subprocess.run(["git", "-C", repo, "-c", "user.email=t@t", "-c", "user.name=t",
                    "commit", "-qm", "artifact"], capture_output=True)
    rc, out = run("--repo", repo, "--fail-on-missing")
    rc2, out2 = run("--repo", repo, "--only", "docrel")
    cell("c27 the CLI reports a document-relative citation as DOCREL and does not fail on it",
         rc == 0 and "DOCREL       1 distinct" in out
         and "0 new against the baseline" in out
         and rc2 == 0 and "1 distinct path(s)" in out2 and "tools/fetch.py" in out2,
         "rc=%d %r | only rc=%d %r" % (rc, out[-300:], rc2, out2[-300:]))

    print("selftest: %d cells, %d passed, %d failed" % (n, n - bad, bad))
    return 0 if bad == 0 else 1


if __name__ == "__main__":
    sys.exit(main())