#!/usr/bin/env python3
"""queue_index.py -- the work queue's table of contents, and the gate that it is current.

WHY THIS EXISTS
  `WORK-QUEUE.md` is 1.2 MB and 9,953 lines long at `39312c2`. It has no table of contents.
  Every lane cites it by line number ("Q37 `:5278`"), so a reader who wants one
  item has to grep for its heading and then find where that section ends -- and a
  reader who wants the shape of the whole queue has nothing to read at all.

  The shape is not obvious even after a grep. Measured on 2026-10-17 at HEAD
  (`ece5228`): 49 `## ` sections carry 45 distinct ids, because four ids (Q37,
  Q38, Q39, Q43) own a second section each -- a `closure` or `claimed` block
  appended under a fresh heading. And 9 further ids (Q41, Q42, Q46, Q47, Q50,
  Q51, Q52, Q53, Q63) have NO heading of their own: they are opened as bold
  records nested inside somebody else's section. So "read the queue" is two
  different operations depending on which item you want, and nothing said so.

  THE RECORD RULE (Q63, 2026-10-17). An id is *opened by a record, not mentioned
  by prose*: a heading, a bold lead that opens with the id (`**Qnn -- ...**` /
  `**Qnn opened -- ...**`), or a line whose own lead is a `Status:` marker. Before
  that rule the detector admitted every id token that had no heading, so a tick
  quoting `--outside = Q63/Q77/Q99` opened three ids, an artifact path
  (`out/Q54-READBACK.txt`) opened Q54, and the range `Q31-Q62` opened Q62. The
  first *mention* also used to be reported as the record, which is wrong twice:
  Q53 is first named in prose well before it is opened (:7246 prose vs :7292
  record) and Q63 is first named inside the very tick that quotes it (:9909) but
  opened at :9918. The table below now carries the record line, the first-mention
  line and the form that admitted the id, so the difference is visible instead of
  implied.

  An id is `Q` and one or more digits, no width (Q103, 2026-10-27). The pattern
  read `Q` plus one or two digits, so from Q100 up an id was not a section and
  not even a token: a
  `## Q100 -- ...` heading parsed as body text of the section before it and that
  section's byte count absorbed the whole tail behind it. Three-digit ids are now
  sections, tokens and record ids alike, with no ceiling restored at three digits
  -- a fixed width is the defect, so a three-digit ceiling would only postpone it.
  The trailing word boundary is what stops `Q100` matching inside `Q1000`.

  TIER TWO HAS BOUNDS NOW (Q103). A `**Qnn -- ...**` record that opens an id with
  no heading of its own is an item too: its block runs from the record line to
  the line before the next item opens, so `Q98`'s row stopped carrying the 40,957
  bytes of `Q100`-`Q103` and each of those ids has a row and an extent.

WHAT IT MEASURES -- structural facts only, one row per item -- a `## ` section,
  or (Q103) a record block that opens an id with no heading of its own
  id, line, end_line, role, title, lines, bytes, first `Status:` marker.
    role=opening       the first section carrying that id
    role=continuation  a later section for an id already seen (a closure or
                       claim block); its `role_detail` names the first word
    lines/bytes        the section's own extent, header line to last line
    status             the first `Status:` line in the section, VERBATIM and
                       truncated. NOT adjudicated: whether an item is open is
                       `tools/queue_claim_check.py`'s verdict, not this tool's.
                       Two instruments disagreeing about closure is a bug this
                       tool declines to introduce.

  Plus the two legs that make the structure visible:
    nested   ids opened by a record inside a section that has no heading of their
             own: the record's line, the first line that mentions it, the section
             that contains it, and the shape/form of the record line
    outside  ids referenced by the prose corpus (checkins/ replies/ recoveries/
             audits/ access/ decisions/ docs/ and the top-level documents) but
             absent from the queue entirely -- `tools/` is excluded on purpose:
             its fixtures carry synthetic ids by construction, and this tool's
             own selftest is one of them.

  The `outside` leg is REPORTED, not embedded: `--outside` prints it, and the
  rendered document does not contain it. It cannot. That set is a function of
  the working tree's prose, not of the queue, so baking it into a document that
  `--check` byte-compares against a fresh render means any checkin that quotes
  an id turns the queue's gate RED for a reason that has nothing to do with the
  queue. Measured on 2026-10-17: `--outside` read Q63 from `recoveries/`, then
  Q63+Q77+Q99 once a checkin described the test that names Q77 -- same queue,
  same sha256, three different documents. A table of contents must be a
  function of the file it indexes.

  Headings inside a fenced block are never sections; a heading whose id is
  followed by `closure --` or `§9 follow-on --` is still that id's section.

WHAT IT DOES NOT DO
  It does not grade the work, does not adjudicate closure, and does not read
  git history. It answers "what is in this file and where", and it keeps the
  generated copy of that answer honest: `--write` renders `docs/queue-index.md`
  with the parse pinned inside a machine block, and `--check` re-renders the
  whole document -- prose table included, compared byte for byte -- and goes RED
  the moment the two disagree. (The pin alone was not enough: the first
  whole-gate run found the table above the pin drifted while the pin still
  matched, because the document had begun listing itself in its own corpus.
  The rendered file is excluded from that corpus, which makes `--write` a fixed
  point, and the bytes are now compared whole.) The document is therefore a
  function of `WORK-QUEUE.md` alone: the same queue renders the same bytes
  whatever the prose corpus says, which is what makes `--check` a statement
  about the queue's currency and not about the last person to write a checkin.

  `--check` exit codes: 0 current, 1 stale (with the delta printed), 2 the
  document carries no parseable pinned block, 3 the queue or the document could
  not be read at all (missing path, or bytes that are not UTF-8). An unreadable
  input is a precondition failure, never a stale index and never a traceback:
  `--queue` at a missing path or at a binary used to raise FileNotFoundError /
  UnicodeDecodeError out of `run()` with rc 1, which reads like "stale" to every
  caller that only checks the exit code.

  READ-ONLY except `--json PATH` and `--write PATH`.

USAGE
  python3 tools/queue_index.py --index                  # 49 sections, one screen each side
  python3 tools/queue_index.py --outside                # ids the prose corpus names and the queue does not
  python3 tools/queue_index.py --json out.json
  python3 tools/queue_index.py --write docs/queue-index.md
  python3 tools/queue_index.py --check docs/queue-index.md
  python3 tools/queue_index.py --selftest               # fixture corpus, 80 cells

  Re-run `--write` after any append to the queue. The document carries the
  queue's sha256, so a stale copy is a RED check, never a silent lie.
"""

import argparse
import hashlib
import json
import os
import re
import subprocess
import sys
import tempfile

QUEUE_DEFAULT = "WORK-QUEUE.md"
DOC_PATH = "docs/queue-index.md"
BEGIN = "<!-- queue-index:begin -->"
END = "<!-- queue-index:end -->"

# THE ID CLASS (Q103, 2026-10-27). An id is `Q` and one or more digits -- no
# width. It was `\d{1,2}`, so from Q100 up an item was not a section and not even
# a token: `## Q100 -- ...` parsed as body text of whatever section preceded it
# and the eight-figure byte count of a whole tail accrued to its neighbour. The
# trailing `\b` is load-bearing in the other direction: without it `Q100` is a
# prefix-match inside `Q1000` and `Q1000` would be indexed as the item Q100.
# A width ceiling is deliberately NOT restored: `\d{1,3}` reproduces this exact
# defect one digit later, which is the shape of the bug, not a fix for it.
HEADING = re.compile(r"^##\s+(Q\d+)\b(.*)$")
ID_TOKEN = re.compile(r"\bQ\d+\b")
STATUS = re.compile(r"^\s*\*{0,2}Status:\s*(.*)$")
FENCE = re.compile(r"^(```|~~~)")

# A record opens with its id; a mention merely carries it. `BULLET` strips the
# markdown that can precede a record without being part of it, and `OPEN_SEP` is
# what may follow the id in a bold lead that is *opening* an item (Q46's
# "**Q46 (Helm lane) — OPEN ...**" and Q63's "**Q63 opened — ...**" both count;
# a "**Q7, measured:**" bold run does not).
BULLET = re.compile(r"^[\s>|]*(?:[-*+]\s+)?")
OPEN_SEP = re.compile(r"\s*(?:—|–|-|:|\(|\bopened\b|\bopen\b|\bOPEN\b|\bOPENED\b)")

ROLE_WORDS = ("closure", "claimed", "§", "status")


def record_form(line, qid):
    """The form in which `line` opens `qid`, or None if it only mentions it.

    The counting rule is settled in `WORK-QUEUE.md` at Q63 (2026-10-17): an id is
    *opened by a record, not mentioned by prose*. Three forms count --

      HEADING      `## Qnn ...` (a section; never nested)
      BOLD-LEAD    after bullet/bold markers the line opens `**Qnn` and the id is
                   followed by a separator or the word open/opened
      STATUS-LEAD  the line's own lead is a `Status:` marker (bold or plain) and
                   it names the id

    A bare token inside a sentence, a tick quoting tool output (`= Q63/Q77/Q99`),
    an artifact filename (`out/Q54-READBACK.txt`) or a range (`Q31–Q62`) matches
    none of them, which is the point: those were opening ids before this rule.
    """
    s = line.strip()
    m = HEADING.match(s)
    if m and m.group(1) == qid:
        return "HEADING"
    body = BULLET.sub("", s)
    if qid not in ID_TOKEN.findall(body):
        return None
    if body.startswith("**Status:") or body.startswith("Status:"):
        return "STATUS-LEAD"
    if body.startswith("**"):
        inner = body[2:].lstrip("*").lstrip()
        m = ID_TOKEN.match(inner)
        if m and m.group(0) == qid and OPEN_SEP.match(inner[m.end():]):
            return "BOLD-LEAD"
    return None


def record_detail(line, qid):
    """The first word after a record's bold lead -- `DONE`, `CLOSED`, `opened`.

    The heading loop's `role_detail` already names the first word after a
    continuation heading's dash; this is the same field for a continuation
    record, so the table says *why* a second block exists without a reader
    having to open it.
    """
    body = BULLET.sub("", line.strip())
    if not body.startswith("**"):
        return ""
    inner = body[2:].lstrip("*").lstrip()
    if inner.startswith(qid):
        inner = inner[len(qid):]
    for word in inner.lstrip("—-–: ").split():
        return word.strip("*().,;:").strip("*")
    return ""


def split_lines(text):
    """Lines without their terminators, plus the byte length each line occupies."""
    lines = text.split("\n")
    trailing_newline = text.endswith("\n")
    if trailing_newline:
        lines = lines[:-1]
    return lines, trailing_newline


def line_bytes(line, is_last, trailing_newline):
    if is_last and not trailing_newline:
        return len(line.encode("utf-8"))
    return len(line.encode("utf-8")) + 1


def status_value(raw):
    """The value of a `Status:` marker, in either emphasis form.

    `**Status: value**` puts the value *inside* the bold run; `**Status:** value`
    closes the bold *before* it. The `STATUS` pattern consumes the optional
    leading `**`, so cutting at the first `**` is right for the first form and
    returns "" for the second -- which published a stated status as `--`.
    Q13:1129 (`**Status:** 1 of 8 has a decision ...`) was the live case.
    """
    raw = raw.strip()
    while raw.startswith("**"):
        raw = raw[2:].strip()
    if "**" in raw:
        raw = raw.split("**", 1)[0].strip()
    return raw


def fence_mask(lines):
    """1-based line numbers to skip: fence delimiters and every line they hide.

    A fenced block is quoted material, not document structure -- headings inside
    it are not sections, and id tokens inside it are not records. Stated once
    here and reused by parse(), rather than restated verbatim in each scan loop.
    """
    out = set()
    in_fence = False
    for i, ln in enumerate(lines, 1):
        if FENCE.match(ln.strip()):
            in_fence = not in_fence
            out.add(i)
        elif in_fence:
            out.add(i)
    return out


def parse(text):
    """Return the structural parse: items, nested ids, id map, byte accounting.

    An *item* is a heading section OR a record block (Q103, 2026-10-27). An id
    opened by a record -- `**Q100 -- OPEN ...**` inside somebody else's section --
    is an item with its own extent, from its record line to the line before the
    next item opens, instead of being counted as prose inside its neighbour. The
    two tiers are one table here and `kind` says which: before this, a record
    opening an id with no heading handed its bytes to the section around it (Q98's
    row carried 40,957 bytes of four later items) and an id from Q100 up was not
    even a token.
    """
    lines, trailing_newline = split_lines(text)
    n = len(lines)
    sizes = [line_bytes(ln, i == n - 1, trailing_newline) for i, ln in enumerate(lines)]
    fenced = fence_mask(lines)

    headings = []           # (line_no, id, rest)
    for i, ln in enumerate(lines, 1):
        if i in fenced:
            continue
        m = HEADING.match(ln)
        if m:
            headings.append((i, m.group(1), m.group(2).strip()))
    heading_ids = {h[1] for h in headings}

    # every id token in the queue, counted as a mention, plus the first line that
    # opens it as a record. Fenced lines are skipped: an id in a code block is
    # quoted text, not a record.
    #
    # Q63 (2026-10-17): the first *mention* is not the record. Q53 is first named
    # in prose 46 lines before its `**Q53 opened -- ...**` line, and Q63 is named
    # inside a tick quoting tool output at :9909 but opened at :9918. So a nested
    # id is admitted by its first record-form line, and an id that never has one
    # (Q54/Q55/Q56/Q58 as artifact filenames, Q62 inside `Q31-Q62`, Q77/Q99 inside
    # a quoted `--outside` list) is not an item at all.
    mentioned = {}
    for i, ln in enumerate(lines, 1):
        if i in fenced:
            continue
        for m in ID_TOKEN.finditer(ln):
            qid = m.group(0)
            hit = mentioned.get(qid)
            if hit is None:
                hit = mentioned[qid] = {"line": i, "refs": 0, "record": None, "form": None}
            hit["refs"] += 1
            if hit["record"] is None:
                form = record_form(ln, qid)
                if form:
                    hit["record"] = i
                    hit["form"] = form

    # tier two: ids opened by a record and owning no heading of their own
    records = [(mentioned[qid]["record"], qid, mentioned[qid]["form"])
               for qid in sorted(mentioned, key=lambda s: int(s[1:]))
               if qid not in heading_ids and mentioned[qid]["record"] is not None]

    # tier two, later records (Q103, 2026-10-28): an id's SECOND block-initial
    # record -- a closure, a correction -- is an item too, with its own bounds.
    # Before this, only the first record line per id cut anything, so every later
    # closure block was charged to whichever item preceded it: measured on this
    # queue before the fix, `**Q103 -- DONE ...**` (30 lines appended at the foot)
    # grew Q104's row from 5,990 to 14,007 bytes, and Q100's own DONE block at
    # :12915 sat inside Q101's row. That is the same swallow the id class had,
    # one level down.
    #
    # The admission rule is deliberately narrow, because `record_form` alone is
    # too loose here: 171 lines in this queue carry a record form for some id and
    # 49 ids have more than one, but most of the extras are bullet status lines
    # (`- **Q16:** unchanged at ...`) that are prose inside an item and must not
    # split it. A later record is a boundary only when it BEGINS a block: the
    # line opens with `**` at column 0, and the line above it is blank or a `---`
    # rule or an HTML comment. Measured: 52 lines qualify, every one of them a
    # standalone block in the file.
    def begins_block(i):
        prev = lines[i - 2].strip() if i >= 2 else ""
        return prev == "" or prev == "---" or prev.startswith("<!--")

    first_records = {(r[0], r[1]) for r in records}
    later = []
    for i, ln in enumerate(lines, 1):
        if i in fenced or not ln.startswith("**") or not begins_block(i):
            continue
        for qid in sorted(set(ID_TOKEN.findall(ln)), key=lambda s: int(s[1:])):
            if (i, qid) in first_records or record_form(ln, qid) != "BOLD-LEAD":
                continue
            later.append((i, qid, "BOLD-LEAD"))
            break

    # Every item opens somewhere and owns the lines up to the next opening: a
    # heading, or a record that opens an id with no heading, or a later
    # block-initial record for an id that is already an item. Nothing else cuts a
    # section, so a bold *mention* inside a section does not split it.
    boundaries = sorted({h[0] for h in headings} | {r[0] for r in records}
                        | {r[0] for r in later})

    def end_of(line_no):
        nxt = [b for b in boundaries if b > line_no]
        return (nxt[0] - 1) if nxt else n

    items = []
    seen = {}
    for line_no, qid, rest in headings:
        role = "opening" if qid not in seen else "continuation"
        detail = ""
        if role == "continuation":
            words = rest.lstrip("—- ").split()
            detail = words[0].strip("*(") if words else ""
        seen.setdefault(qid, line_no)
        items.append({"id": qid, "line": line_no, "kind": "heading", "role": role,
                      "role_detail": detail, "title": clip(rest.lstrip("—- ").strip(), 200)})
    for line_no, qid, form in records:
        raw = lines[line_no - 1]
        seen.setdefault(qid, line_no)
        items.append({"id": qid, "line": line_no, "kind": "record", "role": "record",
                      "role_detail": form,
                      "title": clip(BULLET.sub("", raw).strip(), 200)})
    for line_no, qid, form in later:
        raw = lines[line_no - 1]
        role = "continuation" if qid in seen else "record"
        seen.setdefault(qid, line_no)
        items.append({"id": qid, "line": line_no, "kind": "record", "role": role,
                      "role_detail": record_detail(raw, qid) if role == "continuation"
                      else form,
                      "title": clip(BULLET.sub("", raw).strip(), 200)})
    items.sort(key=lambda it: it["line"])

    for it in items:
        end = end_of(it["line"])
        status = None
        status_line = None
        for off, ln in enumerate(lines[it["line"]:end], start=it["line"] + 1):
            m = STATUS.match(ln)
            if m:
                status = status_value(m.group(1)) or None
                status_line = off
                break
        it["end_line"] = end
        it["lines"] = end - it["line"] + 1
        it["bytes"] = sum(sizes[it["line"] - 1:end])
        it["status"] = clip(status, 200) if status else None
        it["status_line"] = status_line

    sections = [it for it in items if it["kind"] == "heading"]
    blocks = [it for it in items if it["kind"] == "record"]

    ids: dict = {}
    for s in sections:
        ids.setdefault(s["id"], []).append(s["line"])

    # `nested` carries what a bounds row cannot: the record's form, how often the
    # id is named, the first line that names it at all, and the item that opens
    # before it (its parent). A block is the same id with bounds; both are
    # reported because a reader needs the line and the shape.
    nested = []
    for it in blocks:
        # one row per id, not per block: a later closure block for the same id is
        # an item with bounds (see `items`), but the table below is the id's
        # record -- its form and how often the file names it -- so the opening
        # block stands for it (Q103, 2026-10-28).
        if it["role"] != "record":
            continue
        qid = it["id"]
        rec = mentioned[qid]
        idx = [k for k, cand in enumerate(items) if cand["line"] == it["line"]][0]
        owner = items[idx - 1]["id"] if idx else None
        raw = lines[it["line"] - 1]
        stripped = raw.lstrip()
        if stripped.startswith("#"):
            shape = "HEADING"
        elif stripped.startswith(("-", "*", "|")):
            shape = "BULLET"
        elif stripped.startswith("**"):
            shape = "BOLD"
        else:
            shape = "PLAIN"
        if shape not in ("HEADING", "BOLD") and "**" in raw:
            shape += "+BOLD"
        nested.append({"id": qid, "line": it["line"], "section": owner, "shape": shape,
                       "refs": rec["refs"], "form": rec["form"],
                       "first_mention": rec["line"]})

    first = headings[0][0] if headings else n + 1
    preamble_end = first - 1
    payload = {
        "queue_lines": n,
        "queue_bytes": sum(sizes),
        "queue_sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(),
        "preamble": {"line": 1, "end_line": preamble_end,
                     "bytes": sum(sizes[:preamble_end])},
        "items": items,
        "sections": sections,
        "blocks": blocks,
        "ids_with_section": sorted(ids, key=lambda s: int(s[1:])),
        "ids_multi_section": {k: v for k, v in sorted(
            ids.items(), key=lambda kv: int(kv[0][1:])) if len(v) > 1},
        "nested": nested,
    }
    return payload


def outside_ids(repo, queue_path, exclude=(), exclude_tools=True):
    """Ids referenced by the prose corpus that the queue never mentions.

    `exclude` is the rendered document itself: a generated file that names ids
    would otherwise join the corpus that generates it, so `--write` would not be
    a fixed point -- measured once, the first whole-gate run went red on exactly
    that (the document added itself to Q63's file list).
    """
    try:
        out = subprocess.check_output(["git", "-C", repo, "ls-files"], text=True,
                                      stderr=subprocess.DEVNULL)
    except (subprocess.CalledProcessError, FileNotFoundError):
        return None
    skip = {os.path.realpath(p) for p in exclude if p}
    files = [f for f in out.split("\n") if f.endswith(".md")]
    if exclude_tools:
        files = [f for f in files if not f.startswith("tools/")]
    # The generated document is never part of the corpus it summarizes, whatever
    # path it is written to -- otherwise a render into any other filename would
    # count the committed copy and the output would cease to be a fixed point.
    files = [f for f in files if f != DOC_PATH]
    queue_abs = os.path.normpath(os.path.join(repo, queue_path))
    queue_text = open(queue_abs, encoding="utf-8", errors="replace").read()
    in_queue = set(ID_TOKEN.findall(queue_text))
    hits = {}
    read = 0
    for f in files:
        path = os.path.join(repo, f)
        if os.path.realpath(path) in skip:
            continue
        try:
            txt = open(path, encoding="utf-8", errors="replace").read()
        except OSError:
            continue
        read += 1
        for m in ID_TOKEN.finditer(txt):
            hits.setdefault(m.group(0), set()).add(f)
    return {k: sorted(v) for k, v in sorted(
        hits.items(), key=lambda kv: int(kv[0][1:])) if k not in in_queue}, read


def summary(payload):
    secs = payload["sections"]
    blocks = payload.get("blocks", [])
    cont = [s for s in secs if s["role"] == "continuation"]
    return ("index: %d items (%d sections, %d record blocks), %d ids, "
            "%d continuation sections, %d nested ids, %d lines / %d bytes"
            % (len(payload["items"]), len(secs), len(blocks),
               len(payload["ids_with_section"]), len(cont), len(payload["nested"]),
               payload["queue_lines"], payload["queue_bytes"]))


def clip(text, limit):
    """Cut at a word boundary and say so; a silent mid-word cut reads as data loss."""
    if len(text) <= limit:
        return text
    cut = text[:limit]
    if " " in cut[max(0, limit - 20):]:
        cut = cut.rsplit(" ", 1)[0]
    return cut.rstrip(" ,;:—-") + " …"


def nested_prose(nested):
    """The 'what it exposes' clause about record-opened ids.

    `build_doc` used to index `nested[0]` unconditionally, so `--write` died
    with IndexError -- leaving a zero-byte document and rc 1, the code that
    means "graded a fault" -- on any queue whose ids all carry headings (a fresh
    queue, a fixture, a partial extract). Found 2026-10-18 by the fixture in
    `tools/queue_append_audit.py --selftest`. The non-empty branch reproduced
    that fix's sentence byte for byte so the render did not move.

    Q103 (2026-10-27): the sentence used to open "A further **N ids**", which
    double-counted -- the clause before it already said those N ids are items
    with rows and extents, so the paragraph read as 2N. The ids in `blocks` and
    the ids in `nested` are the same set now that a record block is an item (16
    and 16 in this tree, set-equal); the sentence says so instead of implying a
    second population, and the table below is described as the second view of
    the same ids it is.
    """
    if not nested:
        return (" No id is opened as a record without a heading of its own, so the "
                "table below is empty.")
    return (" The same **%d ids** are listed again in the table below, which carries "
            "what a bounds row cannot -- the form that opened each one and how often the "
            "file names it: grepping `^## %s` finds nothing while %s is a live item the "
            "file names %d times."
            % (len(nested), nested[0]["id"], nested[0]["id"], nested[0]["refs"]))


def build_doc(payload):
    lines = []
    lines.append("# The work queue's table of contents")
    lines.append("")
    lines.append("*Generated by `tools/queue_index.py --write`; do not hand-edit. `python3 "
                 "tools/queue_index.py --check docs/queue-index.md` re-derives the parse and "
                 "goes RED when this document and `WORK-QUEUE.md` disagree, which is what an "
                 "append to the queue produces. Regenerate, commit, move on.*")
    lines.append("")
    lines.append("**What this is.** One row per item of `WORK-QUEUE.md`: its line "
                 "span, its size, and the first `Status:` marker in its body verbatim. "
                 "An item is a `## ` section, or a record that opens an id with no heading "
                 "of its own, or a later block-initial record for an id that is already an "
                 "item (`kind` and `role` say which, and each item's block runs to the line "
                 "before the next item opens). Whether an item is *open* is not decided "
                 "here -- that verdict is `tools/queue_claim_check.py`'s, and this document "
                 "does not overrule it.")
    lines.append("")
    blocks = payload.get("blocks", [])
    open_blocks = [b for b in blocks if b["role"] == "record"]
    later_blocks = [b for b in blocks if b["role"] != "record"]
    lines.append("**What it exposes.** The queue is two-tier. %d sections carry %d ids because "
                 "%d ids own a second, continuation section; %d further ids are opened by a "
                 "record and own no heading at all, and they have rows and extents here -- "
                 "%d record blocks in all, %d of them a later closure or correction for an id "
                 "already opened.%s"
                 % (len(payload["sections"]), len(payload["ids_with_section"]),
                    len(payload["ids_multi_section"]), len(open_blocks), len(blocks),
                    len(later_blocks), nested_prose(payload["nested"])))
    lines.append("")
    lines.append("| # | id | kind | lines | bytes | role | title | first `Status:` |")
    lines.append("|---|---|---|---|---|---|---|---|")
    for i, s in enumerate(payload["items"], 1):
        status = (s["status"] or "--").replace("|", "\\|")
        title = s["title"].replace("|", "\\|")
        if s["role"] == "opening":
            role = "opening"
        elif s["role"] == "record":
            role = "record: %s" % (s["role_detail"] or "?")
        else:
            role = "cont: %s" % (s["role_detail"] or "?")
        lines.append("| %d | %s | %s | %d-%d | %d | %s | %s | %s |"
                     % (i, s["id"], s["kind"], s["line"], s["end_line"], s["bytes"],
                        role, title, status))
    lines.append("")
    lines.append("## Ids with no `## ` heading")
    lines.append("")
    lines.append("Opened as records inside another section. `line` is the record's own opening "
                 "line and `first mention` is the first line that names the id at all -- they "
                 "differ when an id is cited in prose before it is opened (Q53: :7246 -> :7292, "
                 "Q63: :9909 -> :9918). `form` is the record form that admits it.")
    lines.append("")
    lines.append("| id | record line | first mention | refs | inside section | shape | form |")
    lines.append("|---|---|---|---|---|---|---|")
    for r in payload["nested"]:
        lines.append("| %s | %d | %d | %d | %s | %s | %s |"
                     % (r["id"], r["line"], r["first_mention"], r["refs"],
                        r["section"] or "--", r["shape"], r["form"]))
    lines.append("")
    lines.append("**Not in this document.** Ids the prose corpus names that the queue has never "
                 "mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the "
                 "top-level documents; `tools/` excluded). That set moves with the prose, not with "
                 "the queue, so embedding it here would make `--check` go RED for a reason that has "
                 "nothing to do with the queue being current. It is reported on demand: "
                 "`python3 tools/queue_index.py --outside`.")
    lines.append("")
    lines.append("## The pinned parse")
    lines.append("")
    lines.append("The block below is the parse this document renders, byte for byte. "
                 "`--check` compares a fresh parse against it.")
    lines.append("")
    lines.append(BEGIN)
    lines.append("```json")
    lines.append(json.dumps(payload, indent=1, sort_keys=True))
    lines.append("```")
    lines.append(END)
    lines.append("")
    return "\n".join(lines)


def extract_pinned(text):
    if BEGIN not in text or END not in text:
        return None
    body = text.split(BEGIN, 1)[1].split(END, 1)[0]
    m = re.search(r"```json\n(.*?)\n```", body, re.S)
    if not m:
        return None
    try:
        return json.loads(m.group(1))
    except ValueError:
        return None


def delta(pinned, fresh):
    """A short, specific account of how a pinned parse and a fresh one differ."""
    out = []
    p_it = {(s["id"], s["line"]): s for s in pinned.get("items", pinned.get("sections", []))}
    f_it = {(s["id"], s["line"]): s for s in fresh.get("items", [])}
    added = [k for k in f_it if k not in p_it]
    removed = [k for k in p_it if k not in f_it]
    moved = [k for k in f_it if k in p_it and
             (f_it[k]["end_line"], f_it[k]["bytes"], f_it[k]["status"])
             != (p_it[k]["end_line"], p_it[k]["bytes"], p_it[k]["status"])]
    # Q103: an item is a heading section or a record block, and the account names
    # which -- a block that moved is not reported as a section.
    def kind_of(table, key):
        return table[key].get("kind", "heading")
    add_h = [k for k in added if kind_of(f_it, k) == "heading"]
    add_b = [k for k in added if kind_of(f_it, k) != "heading"]
    if add_h:
        out.append("%d new section(s): %s" % (len(add_h), ", ".join("%s@%d" % k for k in add_h)))
    if add_b:
        out.append("%d new record block(s): %s"
                   % (len(add_b), ", ".join("%s@%d" % k for k in add_b)))
    gone_h = [k for k in removed if kind_of(p_it, k) == "heading"]
    gone_b = [k for k in removed if kind_of(p_it, k) != "heading"]
    if gone_h:
        out.append("%d gone section(s): %s" % (len(gone_h), ", ".join("%s@%d" % k for k in gone_h)))
    if gone_b:
        out.append("%d gone record block(s): %s"
                   % (len(gone_b), ", ".join("%s@%d" % k for k in gone_b)))
    if moved:
        out.append("%d item(s) changed extent or status, e.g. %s@%d %d->%d lines"
                   % (len(moved), moved[0][0], moved[0][1],
                      p_it[moved[0]]["end_line"] - p_it[moved[0]]["line"] + 1,
                      f_it[moved[0]]["end_line"] - f_it[moved[0]]["line"] + 1))
    p_nest = [r["id"] for r in pinned.get("nested", [])]
    f_nest = [r["id"] for r in fresh.get("nested", [])]
    if p_nest != f_nest:
        out.append("nested id set changed: +%s -%s"
                   % (",".join(sorted(set(f_nest) - set(p_nest), key=lambda s: int(s[1:]))),
                      ",".join(sorted(set(p_nest) - set(f_nest), key=lambda s: int(s[1:])))))
    if pinned.get("queue_sha256") != fresh.get("queue_sha256"):
        out.append("queue sha256 %s -> %s" % (str(pinned.get("queue_sha256"))[:12],
                                              str(fresh.get("queue_sha256"))[:12]))
    return "; ".join(out) if out else "no structural difference (byte-level drift only)"


def run(args):
    repo = args.repo
    queue_path = os.path.join(repo, args.queue)
    # A read failure is an exit code, not a traceback. `--queue` pointed at a
    # missing path or at bytes that are not UTF-8 raised FileNotFoundError /
    # UnicodeDecodeError straight out of run(), so the operator got a stack and
    # rc 1 while every other reader in this repo (`queue_claim_check.py:558-563`)
    # says `cannot read queue:` and exits 2. The same contract is owed to the
    # --check document: a gate that cannot read its input has verified nothing,
    # and must not report that as a stale index (rc 1) or a crash.
    try:
        with open(queue_path, encoding="utf-8") as fh:
            text = fh.read()
    except (OSError, UnicodeDecodeError) as exc:
        print("cannot read queue: %s" % exc, file=sys.stderr)
        return 2
    payload = parse(text)

    if args.check:
        try:
            doc = open(args.check, encoding="utf-8").read()
        except (OSError, UnicodeDecodeError) as exc:
            print("cannot read index document: %s" % exc, file=sys.stderr)
            return 2
        pinned = extract_pinned(doc)
        if pinned is None:
            print("UNVERIFIABLE: no parseable pinned block in %s" % args.check)
            return 2
        render = build_doc(payload)
        if render != doc:
            # The rendered prose is graded, not only the pin. A pin-only gate
            # stayed green while the table above it had drifted; that was the
            # first whole-gate run's finding, so the bytes are compared whole.
            rl = render.split("\n")
            dl = doc.split("\n")
            first = next((i for i in range(min(len(rl), len(dl))) if rl[i] != dl[i]),
                         min(len(rl), len(dl)))
            print("STALE: rendered document differs at line %d" % (first + 1))
            if first < len(dl):
                print("  document: %s" % dl[first][:160])
            if first < len(rl):
                print("  render:   %s" % rl[first][:160])
            if pinned is not None:
                print("  pin:      %s" % delta(pinned, payload))
            print("regenerate with: python3 tools/queue_index.py --repo %s --write %s"
                  % (repo, args.check))
            return 1
        print("index current: %s" % summary(payload))
        return 0

    # `outside` is None outside a git work tree (no corpus to speak of); --index
    # and --json still work there, which is what makes the tool usable on an
    # extracted copy of the queue -- a bare `unpack NoneType` was a crash until
    # this line, found by verifying the tool against `git show <rev>:...` blobs.
    res = outside_ids(repo, args.queue, exclude=(args.write, args.check))
    outside, corpus = res if res else ({}, 0)
    if args.outside:
        print("outside the queue: %d id(s) named by the %d tracked *.md files read "
              "(`tools/` excluded), which `WORK-QUEUE.md` never mentions" % (len(outside or {}), corpus or 0))
        for qid, files in (outside or {}).items():
            print("  %-5s %s" % (qid, ", ".join(files[:4])
                                 + (" +%d" % (len(files) - 4) if len(files) > 4 else "")))
        return 0

    if args.write:
        try:
            with open(args.write, "w", encoding="utf-8") as fh:
                fh.write(build_doc(payload))
        except OSError as exc:
            print("cannot write --write " + args.write + ": " + str(exc), file=sys.stderr)
            return 2
        print("wrote %s (%d bytes)" % (args.write, os.path.getsize(args.write)))
        return 0

    if args.json:
        try:
            with open(args.json, "w", encoding="utf-8") as fh:
                json.dump(payload, fh, indent=2, sort_keys=True)
                fh.write("\n")
        except OSError as exc:
            print("cannot write --json " + args.json + ": " + str(exc), file=sys.stderr)
            return 2

    if args.index:
        print("%-4s %-5s %-7s %-11s %-7s %-10s %s"
              % ("#", "id", "kind", "lines", "bytes", "role", "title"))
        for i, s in enumerate(payload["items"], 1):
            print("%-4d %-5s %-7s %-11s %-7d %-10s %s"
                  % (i, s["id"], s["kind"], "%d-%d" % (s["line"], s["end_line"]), s["bytes"],
                     s["role"][:10], clip(s["title"], 72)))
        print()
        for r in payload["nested"]:
            print("nested %-4s line %-5d refs %-3d inside %-4s %s"
                  % (r["id"], r["line"], r["refs"], r["section"] or "--", r["shape"]))
        if outside:
            print("outside the queue: %s" % ", ".join(outside))
        print()

    print(summary(payload))
    return 0


# --------------------------------------------------------------------------
# selftest: a fixture corpus, one cell per stated rule
# --------------------------------------------------------------------------

FIXTURE = """# Work queue

Preface text.

## Q1 — First item *(lane)*

**Status: DONE at `abc1234`.** Body.

```
## Q9 — not a section, this is fenced
```

## Q2 — Second item

Body with no status marker.

## Q1 — closure — verified

**Status: OPEN.** A continuation section for Q1.

## Q3 §9 follow-on — odd shape

Prose that names Q41 before its record.
- **Q41 — OPEN (pipe, 2026-10-13, unclaimed).** A nested record.
- prose mentioning Q42 in passing.
- **Q43 opened — a bold lead record for the fixture.**
- a bullet carrying **Q44** mid-line, which is not a record.
- a tick quoting `--outside` = Q77/Q99 as prose.
**Status: OPEN.** The status lead opens Q45 on this line.
"""


def selftest():
    cells = [0]
    bad = [0]

    def check(name, got, want):
        cells[0] += 1
        if got != want:
            bad[0] += 1
            print("FAIL %s\n  got  %r\n  want %r" % (name, got, want))

    p = parse(FIXTURE)
    secs = p["sections"]
    check("four sections parsed", len(secs), 4)
    check("ids in order", [s["id"] for s in secs], ["Q1", "Q2", "Q1", "Q3"])
    check("fenced heading is not a section", "Q9" in p["ids_with_section"], False)
    check("second Q1 is a continuation", secs[2]["role"], "continuation")
    check("continuation detail is the role word", secs[2]["role_detail"], "closure")
    check("odd heading shape still parses", secs[3]["id"], "Q3")
    check("multi-section map names Q1", p["ids_multi_section"], {"Q1": [5, 17]})
    check("status marker extracted verbatim", secs[0]["status"], "DONE at `abc1234`.")
    check("status line number", secs[0]["status_line"], 7)
    check("section with no marker is None", secs[1]["status"], None)
    check("continuation status is its own", secs[2]["status"], "OPEN.")
    # `**Status:** value` -- the bold run closes *before* the value. Cutting at
    # the first `**` returned "" and the index published a stated status as
    # `--` (Q13:1129 was the live case in the real queue).
    lead = parse("## Q1 — a\n\n**Status:** 1 of 8 decided.\n\n"
                 "## Q2 — b\n\n**Status:** **OPEN.** tail.\n\n"
                 "## Q3 — c\n\n**Status:**\n")
    check("status whose bold closes before its value",
          lead["sections"][0]["status"], "1 of 8 decided.")
    check("that marker still reports its own line",
          lead["sections"][0]["status_line"], 3)
    check("status lead with a bold value still extracts",
          lead["sections"][1]["status"], "OPEN.")
    check("a marker with no value at all stays None",
          lead["sections"][2]["status"], None)
    check("nested ids found", [r["id"] for r in p["nested"]], ["Q41", "Q43", "Q45"])
    check("nested owner is the containing section", p["nested"][0]["section"], "Q3")
    check("nested shape is bullet carrying a bold lead", p["nested"][0]["shape"], "BULLET+BOLD")
    check("bold lead is a record form", p["nested"][0]["form"], "BOLD-LEAD")
    check("status lead is a record form", p["nested"][2]["form"], "STATUS-LEAD")
    check("record line follows the first mention of the same id",
          (p["nested"][0]["line"], p["nested"][0]["first_mention"]), (24, 23))
    check("prose mention is not a record (Q42)", "Q42" in [r["id"] for r in p["nested"]], False)
    check("mid-line bold mention is not a record (Q44)",
          "Q44" in [r["id"] for r in p["nested"]], False)
    check("quoted tool output opens nothing (Q77/Q99)",
          {"Q77", "Q99"} & {r["id"] for r in p["nested"]}, set())
    check("mentions are still counted as refs", p["nested"][0]["refs"], 2)
    check("items tile the file with no gap",
          all(p["items"][i]["end_line"] + 1 == p["items"][i + 1]["line"]
              for i in range(len(p["items"]) - 1)), True)
    check("a record block carves its own bounds out of its neighbour",
          [(b["id"], b["line"], b["end_line"]) for b in p["blocks"]],
          [("Q41", 24, 25), ("Q43", 26, 28), ("Q45", 29, 29)])
    check("the section that used to swallow them now stops before the block",
          (secs[3]["id"], secs[3]["line"], secs[3]["end_line"]), ("Q3", 21, 23))
    check("last item ends at the last line", p["items"][-1]["end_line"], p["queue_lines"])
    check("preamble ends before the first heading", p["preamble"]["end_line"], 4)
    check("byte accounting is exact",
          p["preamble"]["bytes"] + sum(s["bytes"] for s in p["items"]), p["queue_bytes"])
    # a block between two headings keeps both headings as sections and takes the
    # lines between them: the case that made `sections tile` a false statement
    gap = parse("## Q99 — a\n\nbody\n\n- **Q102 — OPEN (pipe).**\n\nprose\n\n## Q100 — b\n")
    check("a block between two headings does not dissolve them",
          [s["id"] for s in gap["sections"]], ["Q99", "Q100"])
    check("the section before the block stops at it",
          [(s["id"], s["line"], s["end_line"]) for s in gap["sections"][:1]],
          [("Q99", 1, 4)])
    check("the block owns the lines up to the next heading",
          [(b["id"], b["line"], b["end_line"]) for b in gap["blocks"]],
          [("Q102", 5, 8)])
    check("items still tile across a block boundary",
          [(i["id"], i["line"], i["end_line"]) for i in gap["items"]],
          [("Q99", 1, 4), ("Q102", 5, 8), ("Q100", 9, 9)])

    # --- a second record block for the same id is an item too (Q103) --------
    # A closure block used to be charged to whichever item preceded it: measured
    # on the real queue, appending `**Q103 -- DONE ...**` grew Q104's row from
    # 5,990 to 14,007 bytes. These cells pin the rule and its two refusals.
    sec = parse("## Q99 — a\n\nbody\n\n"
                "**Q102 — OPEN (pipe).**\n\nthe opening block\n\n---\n\n"
                "**Q102 — DONE at `abc1234`.**\n\nthe closure block\n")
    check("a second block-initial record is its own item",
          [(i["id"], i["line"], i["role"], i["role_detail"]) for i in sec["items"]],
          [("Q99", 1, "opening", ""), ("Q102", 5, "record", "BOLD-LEAD"),
           ("Q102", 11, "continuation", "DONE")])
    check("the opening block stops before the closure block",
          [(b["id"], b["line"], b["end_line"]) for b in sec["blocks"]],
          [("Q102", 5, 10), ("Q102", 11, 13)])
    # slice, not [-1]: under a mutation that cuts no block this must read as a
    # failed cell, not as a traceback out of the selftest (measured: it did).
    check("the closure block owns the tail",
          [(b["line"], b["end_line"]) for b in sec["blocks"]][-1:], [(11, 13)])
    check("the nested table keeps one row for the id, not one per block",
          [(r["id"], r["line"]) for r in sec["nested"]], [("Q102", 5)])
    # Refusal 1: a bullet-led record inside a section is prose in a list, not a
    # boundary. `- **Q102 -- DONE**` must not split the section that holds it --
    # and here Q102 owns a heading, so the bullet is not its first record either.
    bullet = parse("## Q99 — a\n\nbody\n\n## Q102 — d\n\nprose\n\n"
                   "- **Q102 — DONE at `abc1234`.**\n\nmore body\n")
    check("a bullet-led record does not cut the section",
          [(i["id"], i["line"], i["end_line"]) for i in bullet["items"]],
          [("Q99", 1, 4), ("Q102", 5, 11)])
    check("...and it adds no block row",
          [(r["id"], r["form"]) for r in bullet["nested"]], [])
    # Refusal 2: column 0 is not enough either -- the line must begin a block,
    # i.e. the line above is blank, a `---` rule or an HTML comment.
    glued = parse("## Q99 — a\n\nbody\n\n## Q102 — d\n\nprose runs on\n"
                  "**Q102 — DONE.**\n\ntail\n")
    check("a column-0 record glued to prose does not cut the section",
          [(i["id"], i["line"], i["end_line"]) for i in glued["items"]],
          [("Q99", 1, 4), ("Q102", 5, 10)])

    check("bytes are the file's own length", p["queue_bytes"], len(FIXTURE.encode()))
    check("sha256 is the file's own", p["queue_sha256"],
          hashlib.sha256(FIXTURE.encode()).hexdigest())

    # --- the id class: three digits are an id (Q103, 2026-10-27) -------------
    # The fixture above stops at Q99, exactly like the real queue did: `\d{1,2}`
    # made every item from Q100 up body text of the section before it. These
    # cells fail on the pre-fix pattern, which is why they are here.
    three = parse("## Q99 — a\n\nbody\n\n## Q100 — b\n\nQ101 is named here.\n\n"
                  "## Q101 — c\n\n- **Q102 — OPEN (pipe).**\n")
    check("three-digit heading is a section",
          [s["id"] for s in three["sections"]], ["Q99", "Q100", "Q101"])
    check("three-digit heading is an id",
          three["ids_with_section"], ["Q99", "Q100", "Q101"])
    pairs = [(a["end_line"], b["line"]) for a, b in
             zip(three["sections"], three["sections"][1:])]
    check("three-digit section bounds do not overlap the one before",
          pairs[:1], [(4, 5)])
    check("sections still tile across the id boundary",
          all(three["sections"][i]["end_line"] + 1 == three["sections"][i + 1]["line"]
              for i in range(len(three["sections"]) - 1)), True)
    check("a three-digit bold record is a nested id (form kept)",
          [(r["id"], r["form"]) for r in three["nested"]], [("Q102", "BOLD-LEAD")])
    # No width ceiling: a fourth digit is a new id, never a truncation of a
    # three-digit one. `\d{1,3}` would fail both cells below -- the ceiling is
    # the bug one digit later, so it is not restored.
    four = parse("## Q1000 — d\n")
    check("four-digit heading is its own id", four["ids_with_section"], ["Q1000"])
    check("a longer id is one token, not a prefix match",
          ID_TOKEN.findall("see Q1000 and Q100, plus Q1000x"), ["Q1000", "Q100"])
    check("an id glued to a word is not an id",
          ID_TOKEN.findall("FAQ100 / xQ100 / Q100y / Q"), [])
    check("the old id class could not see Q100; this one does (RED -> GREEN)",
          (re.compile(r"\bQ\d{1,2}\b").findall("## Q100 — b"),
           HEADING.match("## Q100 — b") is not None), ([], True))

    # --- the gate, round trip and both directions of falsification ----------
    root = tempfile.mkdtemp(prefix="queue-index-selftest-")
    repo = os.path.join(root, "repo")
    os.makedirs(os.path.join(repo, "docs"))
    queue = os.path.join(repo, "WORK-QUEUE.md")
    doc_path = os.path.join(repo, DOC_PATH)
    subprocess.check_call(["git", "-C", repo, "init", "-q", "-b", "main"])
    subprocess.check_call(["git", "-C", repo, "config", "user.email", "s@invalid"])
    subprocess.check_call(["git", "-C", repo, "config", "user.name", "s"])
    with open(queue, "w", encoding="utf-8") as fh:
        fh.write(FIXTURE)
    subprocess.check_call(["git", "-C", repo, "add", "-A"])
    subprocess.check_call(["git", "-C", repo, "commit", "-q", "-m", "fixture"])

    import contextlib
    import io
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=doc_path, check=None, outside=False))
    check("--write exits 0", rc, 0)
    check("document written", os.path.exists(doc_path), True)
    doc = open(doc_path, encoding="utf-8").read()
    check("document carries the pinned block", extract_pinned(doc) is not None, True)

    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=None, check=doc_path, outside=False))
    check("--check on its own output exits 0", rc, 0)
    check("--check says current", "index current:" in buf.getvalue(), True)

    # --- the document is a function of the queue alone -----------------------
    # A prose file that names an id the queue never mentions must not move a
    # byte of the document, and must not turn --check RED. Baking the corpus
    # table into the document did both: same queue, three different documents,
    # and a checkin quoting an id reddened a gate about the queue's currency.
    with open(os.path.join(repo, "notes.md"), "w", encoding="utf-8") as fh:
        fh.write("# notes\n\nQ63 is named only here; Q1 is a queue id.\n")
    subprocess.check_call(["git", "-C", repo, "add", "notes.md"])
    before = open(doc_path, encoding="utf-8").read()
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=doc_path, check=None, outside=False))
    check("re-render beside a prose mention exits 0", rc, 0)
    check("a prose mention does not change the document",
          open(doc_path, encoding="utf-8").read(), before)
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=None, check=doc_path, outside=False))
    check("--check stays green when prose names an id", rc, 0)

    # --- the outside leg is reported, not embedded ---------------------------
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=None, check=None, outside=True))
    out = buf.getvalue()
    check("--outside exits 0", rc, 0)
    check("--outside counts the corpus it actually read",
          "1 id(s) named by the 2 tracked *.md files" in out, True)
    check("--outside names Q63 and the file that carries it",
          "  Q63   notes.md" in out, True)

    # --- a tree that is not a git repo must still index -----------------------
    # The `outside` leg needs git; --index does not. A verifier that extracts
    # `git show <rev>:WORK-QUEUE.md` into a temp dir has no .git, and this used to
    # crash on `outside, corpus = outside_ids(...)` returning None.
    plain = os.path.join(root, "plain")
    os.makedirs(plain)
    with open(os.path.join(plain, "WORK-QUEUE.md"), "w", encoding="utf-8") as fh:
        fh.write(FIXTURE)
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        rc = run(argparse.Namespace(repo=plain, queue="WORK-QUEUE.md", index=True,
                                    json=None, write=None, check=None, outside=False))
    check("--index outside a git work tree exits 0", rc, 0)
    check("--index outside a git work tree still summarizes",
          "index: 7 items (4 sections, 3 record blocks)" in buf.getvalue(), True)

    # falsify 1: an append moves the last section's extent
    with open(queue, "a", encoding="utf-8") as fh:
        fh.write("- an appended line.\n")
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=None, check=doc_path, outside=False))
    check("--check goes RED on an append", rc, 1)
    check("RED names the moved section", "changed extent or status" in buf.getvalue(), True)

    # falsify 2: a new heading is a new section
    with open(queue, "a", encoding="utf-8") as fh:
        fh.write("\n## Q7 — a brand new item\n\n**Status: OPEN.**\n")
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=None, check=doc_path, outside=False))
    check("--check goes RED on a new heading", rc, 1)
    check("RED names the new section", "new section(s): Q7" in buf.getvalue(), True)

    # falsify 3: a document with no block cannot be verified
    open(doc_path, "w", encoding="utf-8").write("# empty\n")
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=None, check=doc_path, outside=False))
    check("no pin -> exit 2", rc, 2)
    check("no pin -> says so", "UNVERIFIABLE" in buf.getvalue(), True)

    # --- a read error is an exit code, not a traceback -----------------------
    # `--queue` at a missing path or at bytes that are not UTF-8 used to raise
    # straight out of run() (FileNotFoundError / UnicodeDecodeError, rc 1). The
    # repo's other reader exits 2 with `cannot read queue:`; this tool now does
    # the same, and owes the --check document the same answer: an unreadable
    # gate input is rc 2, never the rc 1 that means "stale".
    bad_q = os.path.join(root, "bad-utf8.md")
    with open(bad_q, "wb") as fh:
        fh.write(b"\xff\xfe\x80\n")
    err = io.StringIO()
    with contextlib.redirect_stderr(err):
        rc = run(argparse.Namespace(repo=root, queue="missing-queue.md", index=False,
                                    json=None, write=None, check=None, outside=False))
    check("missing queue -> exit 2", rc, 2)
    check("missing queue -> says cannot read queue",
          err.getvalue().startswith("cannot read queue:"), True)
    err = io.StringIO()
    with contextlib.redirect_stderr(err):
        rc = run(argparse.Namespace(repo=root, queue="bad-utf8.md", index=False,
                                    json=None, write=None, check=None, outside=False))
    check("non-UTF-8 queue -> exit 2", rc, 2)
    check("non-UTF-8 queue -> says cannot read queue",
          "cannot read queue:" in err.getvalue(), True)
    err = io.StringIO()
    with contextlib.redirect_stderr(err):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=None,
                                    check=os.path.join(repo, "missing-index.md"),
                                    outside=False))
    check("missing index document -> exit 2", rc, 2)
    check("missing index document -> names the document",
          "cannot read index document:" in err.getvalue(), True)
    err = io.StringIO()
    with contextlib.redirect_stderr(err):
        rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
                                    json=None, write=None, check=bad_q, outside=False))
    check("non-UTF-8 index document -> exit 2", rc, 2)
    check("non-UTF-8 index document -> names the document",
          "cannot read index document:" in err.getvalue(), True)

    print("selftest: %d cells, %d passed, %d failed" % (cells[0], cells[0] - bad[0], bad[0]))
    return 1 if bad[0] else 0


def main(argv=None):
    ap = argparse.ArgumentParser(
        description="queue_index.py -- the work queue's table of contents, and the gate "
                    "that it is current.")
    ap.add_argument("--repo", default=".")
    ap.add_argument("--queue", default=QUEUE_DEFAULT)
    ap.add_argument("--index", action="store_true", help="print the section table")
    ap.add_argument("--json", metavar="PATH", help="write the parse here")
    ap.add_argument("--write", metavar="PATH", help="render the index document here")
    ap.add_argument("--check", metavar="PATH", help="re-derive and compare a rendered index")
    ap.add_argument("--outside", action="store_true",
                    help="report ids the prose corpus names that the queue never mentions")
    ap.add_argument("--selftest", action="store_true", help="fixture corpus, 80 cells")
    args = ap.parse_args(argv)
    if args.selftest:
        return selftest()
    return run(args)


if __name__ == "__main__":
    sys.exit(main())