#!/usr/bin/env python3 """queue_index.py -- the work queue's table of contents, and the gate that it is current. WHY THIS EXISTS `WORK-QUEUE.md` is 1.2 MB and 9,953 lines long at `39312c2`. It has no table of contents. Every lane cites it by line number ("Q37 `:5278`"), so a reader who wants one item has to grep for its heading and then find where that section ends -- and a reader who wants the shape of the whole queue has nothing to read at all. The shape is not obvious even after a grep. Measured on 2026-10-17 at HEAD (`ece5228`): 49 `## ` sections carry 45 distinct ids, because four ids (Q37, Q38, Q39, Q43) own a second section each -- a `closure` or `claimed` block appended under a fresh heading. And 9 further ids (Q41, Q42, Q46, Q47, Q50, Q51, Q52, Q53, Q63) have NO heading of their own: they are opened as bold records nested inside somebody else's section. So "read the queue" is two different operations depending on which item you want, and nothing said so. THE RECORD RULE (Q63, 2026-10-17). An id is *opened by a record, not mentioned by prose*: a heading, a bold lead that opens with the id (`**Qnn -- ...**` / `**Qnn opened -- ...**`), or a line whose own lead is a `Status:` marker. Before that rule the detector admitted every id token that had no heading, so a tick quoting `--outside = Q63/Q77/Q99` opened three ids, an artifact path (`out/Q54-READBACK.txt`) opened Q54, and the range `Q31-Q62` opened Q62. The first *mention* also used to be reported as the record, which is wrong twice: Q53 is first named in prose well before it is opened (:7246 prose vs :7292 record) and Q63 is first named inside the very tick that quotes it (:9909) but opened at :9918. The table below now carries the record line, the first-mention line and the form that admitted the id, so the difference is visible instead of implied. An id is `Q` and one or more digits, no width (Q103, 2026-10-27). The pattern read `Q` plus one or two digits, so from Q100 up an id was not a section and not even a token: a `## Q100 -- ...` heading parsed as body text of the section before it and that section's byte count absorbed the whole tail behind it. Three-digit ids are now sections, tokens and record ids alike, with no ceiling restored at three digits -- a fixed width is the defect, so a three-digit ceiling would only postpone it. The trailing word boundary is what stops `Q100` matching inside `Q1000`. TIER TWO HAS BOUNDS NOW (Q103). A `**Qnn -- ...**` record that opens an id with no heading of its own is an item too: its block runs from the record line to the line before the next item opens, so `Q98`'s row stopped carrying the 40,957 bytes of `Q100`-`Q103` and each of those ids has a row and an extent. WHAT IT MEASURES -- structural facts only, one row per item -- a `## ` section, or (Q103) a record block that opens an id with no heading of its own id, line, end_line, role, title, lines, bytes, first `Status:` marker. role=opening the first section carrying that id role=continuation a later section for an id already seen (a closure or claim block); its `role_detail` names the first word lines/bytes the section's own extent, header line to last line status the first `Status:` line in the section, VERBATIM and truncated. NOT adjudicated: whether an item is open is `tools/queue_claim_check.py`'s verdict, not this tool's. Two instruments disagreeing about closure is a bug this tool declines to introduce. Plus the two legs that make the structure visible: nested ids opened by a record inside a section that has no heading of their own: the record's line, the first line that mentions it, the section that contains it, and the shape/form of the record line outside ids referenced by the prose corpus (checkins/ replies/ recoveries/ audits/ access/ decisions/ docs/ and the top-level documents) but absent from the queue entirely -- `tools/` is excluded on purpose: its fixtures carry synthetic ids by construction, and this tool's own selftest is one of them. The `outside` leg is REPORTED, not embedded: `--outside` prints it, and the rendered document does not contain it. It cannot. That set is a function of the working tree's prose, not of the queue, so baking it into a document that `--check` byte-compares against a fresh render means any checkin that quotes an id turns the queue's gate RED for a reason that has nothing to do with the queue. Measured on 2026-10-17: `--outside` read Q63 from `recoveries/`, then Q63+Q77+Q99 once a checkin described the test that names Q77 -- same queue, same sha256, three different documents. A table of contents must be a function of the file it indexes. Headings inside a fenced block are never sections; a heading whose id is followed by `closure --` or `§9 follow-on --` is still that id's section. WHAT IT DOES NOT DO It does not grade the work, does not adjudicate closure, and does not read git history. It answers "what is in this file and where", and it keeps the generated copy of that answer honest: `--write` renders `docs/queue-index.md` with the parse pinned inside a machine block, and `--check` re-renders the whole document -- prose table included, compared byte for byte -- and goes RED the moment the two disagree. (The pin alone was not enough: the first whole-gate run found the table above the pin drifted while the pin still matched, because the document had begun listing itself in its own corpus. The rendered file is excluded from that corpus, which makes `--write` a fixed point, and the bytes are now compared whole.) The document is therefore a function of `WORK-QUEUE.md` alone: the same queue renders the same bytes whatever the prose corpus says, which is what makes `--check` a statement about the queue's currency and not about the last person to write a checkin. `--check` exit codes: 0 current, 1 stale (with the delta printed), 2 the document carries no parseable pinned block, 3 the queue or the document could not be read at all (missing path, or bytes that are not UTF-8). An unreadable input is a precondition failure, never a stale index and never a traceback: `--queue` at a missing path or at a binary used to raise FileNotFoundError / UnicodeDecodeError out of `run()` with rc 1, which reads like "stale" to every caller that only checks the exit code. READ-ONLY except `--json PATH` and `--write PATH`. USAGE python3 tools/queue_index.py --index # 49 sections, one screen each side python3 tools/queue_index.py --outside # ids the prose corpus names and the queue does not python3 tools/queue_index.py --json out.json python3 tools/queue_index.py --write docs/queue-index.md python3 tools/queue_index.py --check docs/queue-index.md python3 tools/queue_index.py --selftest # fixture corpus, 73 cells Re-run `--write` after any append to the queue. The document carries the queue's sha256, so a stale copy is a RED check, never a silent lie. """ import argparse import hashlib import json import os import re import subprocess import sys import tempfile QUEUE_DEFAULT = "WORK-QUEUE.md" DOC_PATH = "docs/queue-index.md" BEGIN = "" END = "" # THE ID CLASS (Q103, 2026-10-27). An id is `Q` and one or more digits -- no # width. It was `\d{1,2}`, so from Q100 up an item was not a section and not even # a token: `## Q100 -- ...` parsed as body text of whatever section preceded it # and the eight-figure byte count of a whole tail accrued to its neighbour. The # trailing `\b` is load-bearing in the other direction: without it `Q100` is a # prefix-match inside `Q1000` and `Q1000` would be indexed as the item Q100. # A width ceiling is deliberately NOT restored: `\d{1,3}` reproduces this exact # defect one digit later, which is the shape of the bug, not a fix for it. HEADING = re.compile(r"^##\s+(Q\d+)\b(.*)$") ID_TOKEN = re.compile(r"\bQ\d+\b") STATUS = re.compile(r"^\s*\*{0,2}Status:\s*(.*)$") FENCE = re.compile(r"^(```|~~~)") # A record opens with its id; a mention merely carries it. `BULLET` strips the # markdown that can precede a record without being part of it, and `OPEN_SEP` is # what may follow the id in a bold lead that is *opening* an item (Q46's # "**Q46 (Helm lane) — OPEN ...**" and Q63's "**Q63 opened — ...**" both count; # a "**Q7, measured:**" bold run does not). BULLET = re.compile(r"^[\s>|]*(?:[-*+]\s+)?") OPEN_SEP = re.compile(r"\s*(?:—|–|-|:|\(|\bopened\b|\bopen\b|\bOPEN\b|\bOPENED\b)") ROLE_WORDS = ("closure", "claimed", "§", "status") def record_form(line, qid): """The form in which `line` opens `qid`, or None if it only mentions it. The counting rule is settled in `WORK-QUEUE.md` at Q63 (2026-10-17): an id is *opened by a record, not mentioned by prose*. Three forms count -- HEADING `## Qnn ...` (a section; never nested) BOLD-LEAD after bullet/bold markers the line opens `**Qnn` and the id is followed by a separator or the word open/opened STATUS-LEAD the line's own lead is a `Status:` marker (bold or plain) and it names the id A bare token inside a sentence, a tick quoting tool output (`= Q63/Q77/Q99`), an artifact filename (`out/Q54-READBACK.txt`) or a range (`Q31–Q62`) matches none of them, which is the point: those were opening ids before this rule. """ s = line.strip() m = HEADING.match(s) if m and m.group(1) == qid: return "HEADING" body = BULLET.sub("", s) if qid not in ID_TOKEN.findall(body): return None if body.startswith("**Status:") or body.startswith("Status:"): return "STATUS-LEAD" if body.startswith("**"): inner = body[2:].lstrip("*").lstrip() m = ID_TOKEN.match(inner) if m and m.group(0) == qid and OPEN_SEP.match(inner[m.end():]): return "BOLD-LEAD" return None def split_lines(text): """Lines without their terminators, plus the byte length each line occupies.""" lines = text.split("\n") trailing_newline = text.endswith("\n") if trailing_newline: lines = lines[:-1] return lines, trailing_newline def line_bytes(line, is_last, trailing_newline): if is_last and not trailing_newline: return len(line.encode("utf-8")) return len(line.encode("utf-8")) + 1 def status_value(raw): """The value of a `Status:` marker, in either emphasis form. `**Status: value**` puts the value *inside* the bold run; `**Status:** value` closes the bold *before* it. The `STATUS` pattern consumes the optional leading `**`, so cutting at the first `**` is right for the first form and returns "" for the second -- which published a stated status as `--`. Q13:1129 (`**Status:** 1 of 8 has a decision ...`) was the live case. """ raw = raw.strip() while raw.startswith("**"): raw = raw[2:].strip() if "**" in raw: raw = raw.split("**", 1)[0].strip() return raw def fence_mask(lines): """1-based line numbers to skip: fence delimiters and every line they hide. A fenced block is quoted material, not document structure -- headings inside it are not sections, and id tokens inside it are not records. Stated once here and reused by parse(), rather than restated verbatim in each scan loop. """ out = set() in_fence = False for i, ln in enumerate(lines, 1): if FENCE.match(ln.strip()): in_fence = not in_fence out.add(i) elif in_fence: out.add(i) return out def parse(text): """Return the structural parse: items, nested ids, id map, byte accounting. An *item* is a heading section OR a record block (Q103, 2026-10-27). An id opened by a record -- `**Q100 -- OPEN ...**` inside somebody else's section -- is an item with its own extent, from its record line to the line before the next item opens, instead of being counted as prose inside its neighbour. The two tiers are one table here and `kind` says which: before this, a record opening an id with no heading handed its bytes to the section around it (Q98's row carried 40,957 bytes of four later items) and an id from Q100 up was not even a token. """ lines, trailing_newline = split_lines(text) n = len(lines) sizes = [line_bytes(ln, i == n - 1, trailing_newline) for i, ln in enumerate(lines)] fenced = fence_mask(lines) headings = [] # (line_no, id, rest) for i, ln in enumerate(lines, 1): if i in fenced: continue m = HEADING.match(ln) if m: headings.append((i, m.group(1), m.group(2).strip())) heading_ids = {h[1] for h in headings} # every id token in the queue, counted as a mention, plus the first line that # opens it as a record. Fenced lines are skipped: an id in a code block is # quoted text, not a record. # # Q63 (2026-10-17): the first *mention* is not the record. Q53 is first named # in prose 46 lines before its `**Q53 opened -- ...**` line, and Q63 is named # inside a tick quoting tool output at :9909 but opened at :9918. So a nested # id is admitted by its first record-form line, and an id that never has one # (Q54/Q55/Q56/Q58 as artifact filenames, Q62 inside `Q31-Q62`, Q77/Q99 inside # a quoted `--outside` list) is not an item at all. mentioned = {} for i, ln in enumerate(lines, 1): if i in fenced: continue for m in ID_TOKEN.finditer(ln): qid = m.group(0) hit = mentioned.get(qid) if hit is None: hit = mentioned[qid] = {"line": i, "refs": 0, "record": None, "form": None} hit["refs"] += 1 if hit["record"] is None: form = record_form(ln, qid) if form: hit["record"] = i hit["form"] = form # tier two: ids opened by a record and owning no heading of their own records = [(mentioned[qid]["record"], qid, mentioned[qid]["form"]) for qid in sorted(mentioned, key=lambda s: int(s[1:])) if qid not in heading_ids and mentioned[qid]["record"] is not None] # Every item opens somewhere and owns the lines up to the next opening: a # heading, or a record that opens an id with no heading. Nothing else cuts a # section, so a bold *mention* inside a section does not split it. boundaries = sorted({h[0] for h in headings} | {r[0] for r in records}) def end_of(line_no): nxt = [b for b in boundaries if b > line_no] return (nxt[0] - 1) if nxt else n items = [] seen = {} for line_no, qid, rest in headings: role = "opening" if qid not in seen else "continuation" detail = "" if role == "continuation": words = rest.lstrip("—- ").split() detail = words[0].strip("*(") if words else "" seen.setdefault(qid, line_no) items.append({"id": qid, "line": line_no, "kind": "heading", "role": role, "role_detail": detail, "title": clip(rest.lstrip("—- ").strip(), 200)}) for line_no, qid, form in records: raw = lines[line_no - 1] items.append({"id": qid, "line": line_no, "kind": "record", "role": "record", "role_detail": form, "title": clip(BULLET.sub("", raw).strip(), 200)}) items.sort(key=lambda it: it["line"]) for it in items: end = end_of(it["line"]) status = None status_line = None for off, ln in enumerate(lines[it["line"]:end], start=it["line"] + 1): m = STATUS.match(ln) if m: status = status_value(m.group(1)) or None status_line = off break it["end_line"] = end it["lines"] = end - it["line"] + 1 it["bytes"] = sum(sizes[it["line"] - 1:end]) it["status"] = clip(status, 200) if status else None it["status_line"] = status_line sections = [it for it in items if it["kind"] == "heading"] blocks = [it for it in items if it["kind"] == "record"] ids: dict = {} for s in sections: ids.setdefault(s["id"], []).append(s["line"]) # `nested` carries what a bounds row cannot: the record's form, how often the # id is named, the first line that names it at all, and the item that opens # before it (its parent). A block is the same id with bounds; both are # reported because a reader needs the line and the shape. nested = [] for it in blocks: qid = it["id"] rec = mentioned[qid] idx = [k for k, cand in enumerate(items) if cand["line"] == it["line"]][0] owner = items[idx - 1]["id"] if idx else None raw = lines[it["line"] - 1] stripped = raw.lstrip() if stripped.startswith("#"): shape = "HEADING" elif stripped.startswith(("-", "*", "|")): shape = "BULLET" elif stripped.startswith("**"): shape = "BOLD" else: shape = "PLAIN" if shape not in ("HEADING", "BOLD") and "**" in raw: shape += "+BOLD" nested.append({"id": qid, "line": it["line"], "section": owner, "shape": shape, "refs": rec["refs"], "form": rec["form"], "first_mention": rec["line"]}) first = headings[0][0] if headings else n + 1 preamble_end = first - 1 payload = { "queue_lines": n, "queue_bytes": sum(sizes), "queue_sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(), "preamble": {"line": 1, "end_line": preamble_end, "bytes": sum(sizes[:preamble_end])}, "items": items, "sections": sections, "blocks": blocks, "ids_with_section": sorted(ids, key=lambda s: int(s[1:])), "ids_multi_section": {k: v for k, v in sorted( ids.items(), key=lambda kv: int(kv[0][1:])) if len(v) > 1}, "nested": nested, } return payload def outside_ids(repo, queue_path, exclude=(), exclude_tools=True): """Ids referenced by the prose corpus that the queue never mentions. `exclude` is the rendered document itself: a generated file that names ids would otherwise join the corpus that generates it, so `--write` would not be a fixed point -- measured once, the first whole-gate run went red on exactly that (the document added itself to Q63's file list). """ try: out = subprocess.check_output(["git", "-C", repo, "ls-files"], text=True, stderr=subprocess.DEVNULL) except (subprocess.CalledProcessError, FileNotFoundError): return None skip = {os.path.realpath(p) for p in exclude if p} files = [f for f in out.split("\n") if f.endswith(".md")] if exclude_tools: files = [f for f in files if not f.startswith("tools/")] # The generated document is never part of the corpus it summarizes, whatever # path it is written to -- otherwise a render into any other filename would # count the committed copy and the output would cease to be a fixed point. files = [f for f in files if f != DOC_PATH] queue_abs = os.path.normpath(os.path.join(repo, queue_path)) queue_text = open(queue_abs, encoding="utf-8", errors="replace").read() in_queue = set(ID_TOKEN.findall(queue_text)) hits = {} read = 0 for f in files: path = os.path.join(repo, f) if os.path.realpath(path) in skip: continue try: txt = open(path, encoding="utf-8", errors="replace").read() except OSError: continue read += 1 for m in ID_TOKEN.finditer(txt): hits.setdefault(m.group(0), set()).add(f) return {k: sorted(v) for k, v in sorted( hits.items(), key=lambda kv: int(kv[0][1:])) if k not in in_queue}, read def summary(payload): secs = payload["sections"] blocks = payload.get("blocks", []) cont = [s for s in secs if s["role"] == "continuation"] return ("index: %d items (%d sections, %d record blocks), %d ids, " "%d continuation sections, %d nested ids, %d lines / %d bytes" % (len(payload["items"]), len(secs), len(blocks), len(payload["ids_with_section"]), len(cont), len(payload["nested"]), payload["queue_lines"], payload["queue_bytes"])) def clip(text, limit): """Cut at a word boundary and say so; a silent mid-word cut reads as data loss.""" if len(text) <= limit: return text cut = text[:limit] if " " in cut[max(0, limit - 20):]: cut = cut.rsplit(" ", 1)[0] return cut.rstrip(" ,;:—-") + " …" def nested_prose(nested): """The 'what it exposes' clause about record-opened ids. `build_doc` used to index `nested[0]` unconditionally, so `--write` died with IndexError -- leaving a zero-byte document and rc 1, the code that means "graded a fault" -- on any queue whose ids all carry headings (a fresh queue, a fixture, a partial extract). Found 2026-10-18 by the fixture in `tools/queue_append_audit.py --selftest`. The non-empty branch reproduced that fix's sentence byte for byte so the render did not move. Q103 (2026-10-27): the sentence used to open "A further **N ids**", which double-counted -- the clause before it already said those N ids are items with rows and extents, so the paragraph read as 2N. The ids in `blocks` and the ids in `nested` are the same set now that a record block is an item (16 and 16 in this tree, set-equal); the sentence says so instead of implying a second population, and the table below is described as the second view of the same ids it is. """ if not nested: return (" No id is opened as a record without a heading of its own, so the " "table below is empty.") return (" The same **%d ids** are listed again in the table below, which carries " "what a bounds row cannot -- the form that opened each one and how often the " "file names it: grepping `^## %s` finds nothing while %s is a live item the " "file names %d times." % (len(nested), nested[0]["id"], nested[0]["id"], nested[0]["refs"])) def build_doc(payload): lines = [] lines.append("# The work queue's table of contents") lines.append("") lines.append("*Generated by `tools/queue_index.py --write`; do not hand-edit. `python3 " "tools/queue_index.py --check docs/queue-index.md` re-derives the parse and " "goes RED when this document and `WORK-QUEUE.md` disagree, which is what an " "append to the queue produces. Regenerate, commit, move on.*") lines.append("") lines.append("**What this is.** One row per item of `WORK-QUEUE.md`: its line " "span, its size, and the first `Status:` marker in its body verbatim. " "An item is a `## ` section, or a record that opens an id with no heading " "of its own (`kind` says which, and such an id's block runs to the line " "before the next item opens). Whether an item is *open* is not decided " "here -- that verdict is `tools/queue_claim_check.py`'s, and this document " "does not overrule it.") lines.append("") lines.append("**What it exposes.** The queue is two-tier. %d sections carry %d ids because " "%d ids own a second, continuation section; %d further ids are opened by a " "record and own no heading at all, and they have rows and extents here.%s" % (len(payload["sections"]), len(payload["ids_with_section"]), len(payload["ids_multi_section"]), len(payload.get("blocks", [])), nested_prose(payload["nested"]))) lines.append("") lines.append("| # | id | kind | lines | bytes | role | title | first `Status:` |") lines.append("|---|---|---|---|---|---|---|---|") for i, s in enumerate(payload["items"], 1): status = (s["status"] or "--").replace("|", "\\|") title = s["title"].replace("|", "\\|") if s["role"] == "opening": role = "opening" elif s["role"] == "record": role = "record: %s" % (s["role_detail"] or "?") else: role = "cont: %s" % (s["role_detail"] or "?") lines.append("| %d | %s | %s | %d-%d | %d | %s | %s | %s |" % (i, s["id"], s["kind"], s["line"], s["end_line"], s["bytes"], role, title, status)) lines.append("") lines.append("## Ids with no `## ` heading") lines.append("") lines.append("Opened as records inside another section. `line` is the record's own opening " "line and `first mention` is the first line that names the id at all -- they " "differ when an id is cited in prose before it is opened (Q53: :7246 -> :7292, " "Q63: :9909 -> :9918). `form` is the record form that admits it.") lines.append("") lines.append("| id | record line | first mention | refs | inside section | shape | form |") lines.append("|---|---|---|---|---|---|---|") for r in payload["nested"]: lines.append("| %s | %d | %d | %d | %s | %s | %s |" % (r["id"], r["line"], r["first_mention"], r["refs"], r["section"] or "--", r["shape"], r["form"])) lines.append("") lines.append("**Not in this document.** Ids the prose corpus names that the queue has never " "mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the " "top-level documents; `tools/` excluded). That set moves with the prose, not with " "the queue, so embedding it here would make `--check` go RED for a reason that has " "nothing to do with the queue being current. It is reported on demand: " "`python3 tools/queue_index.py --outside`.") lines.append("") lines.append("## The pinned parse") lines.append("") lines.append("The block below is the parse this document renders, byte for byte. " "`--check` compares a fresh parse against it.") lines.append("") lines.append(BEGIN) lines.append("```json") lines.append(json.dumps(payload, indent=1, sort_keys=True)) lines.append("```") lines.append(END) lines.append("") return "\n".join(lines) def extract_pinned(text): if BEGIN not in text or END not in text: return None body = text.split(BEGIN, 1)[1].split(END, 1)[0] m = re.search(r"```json\n(.*?)\n```", body, re.S) if not m: return None try: return json.loads(m.group(1)) except ValueError: return None def delta(pinned, fresh): """A short, specific account of how a pinned parse and a fresh one differ.""" out = [] p_it = {(s["id"], s["line"]): s for s in pinned.get("items", pinned.get("sections", []))} f_it = {(s["id"], s["line"]): s for s in fresh.get("items", [])} added = [k for k in f_it if k not in p_it] removed = [k for k in p_it if k not in f_it] moved = [k for k in f_it if k in p_it and (f_it[k]["end_line"], f_it[k]["bytes"], f_it[k]["status"]) != (p_it[k]["end_line"], p_it[k]["bytes"], p_it[k]["status"])] # Q103: an item is a heading section or a record block, and the account names # which -- a block that moved is not reported as a section. def kind_of(table, key): return table[key].get("kind", "heading") add_h = [k for k in added if kind_of(f_it, k) == "heading"] add_b = [k for k in added if kind_of(f_it, k) != "heading"] if add_h: out.append("%d new section(s): %s" % (len(add_h), ", ".join("%s@%d" % k for k in add_h))) if add_b: out.append("%d new record block(s): %s" % (len(add_b), ", ".join("%s@%d" % k for k in add_b))) gone_h = [k for k in removed if kind_of(p_it, k) == "heading"] gone_b = [k for k in removed if kind_of(p_it, k) != "heading"] if gone_h: out.append("%d gone section(s): %s" % (len(gone_h), ", ".join("%s@%d" % k for k in gone_h))) if gone_b: out.append("%d gone record block(s): %s" % (len(gone_b), ", ".join("%s@%d" % k for k in gone_b))) if moved: out.append("%d item(s) changed extent or status, e.g. %s@%d %d->%d lines" % (len(moved), moved[0][0], moved[0][1], p_it[moved[0]]["end_line"] - p_it[moved[0]]["line"] + 1, f_it[moved[0]]["end_line"] - f_it[moved[0]]["line"] + 1)) p_nest = [r["id"] for r in pinned.get("nested", [])] f_nest = [r["id"] for r in fresh.get("nested", [])] if p_nest != f_nest: out.append("nested id set changed: +%s -%s" % (",".join(sorted(set(f_nest) - set(p_nest), key=lambda s: int(s[1:]))), ",".join(sorted(set(p_nest) - set(f_nest), key=lambda s: int(s[1:]))))) if pinned.get("queue_sha256") != fresh.get("queue_sha256"): out.append("queue sha256 %s -> %s" % (str(pinned.get("queue_sha256"))[:12], str(fresh.get("queue_sha256"))[:12])) return "; ".join(out) if out else "no structural difference (byte-level drift only)" def run(args): repo = args.repo queue_path = os.path.join(repo, args.queue) # A read failure is an exit code, not a traceback. `--queue` pointed at a # missing path or at bytes that are not UTF-8 raised FileNotFoundError / # UnicodeDecodeError straight out of run(), so the operator got a stack and # rc 1 while every other reader in this repo (`queue_claim_check.py:558-563`) # says `cannot read queue:` and exits 2. The same contract is owed to the # --check document: a gate that cannot read its input has verified nothing, # and must not report that as a stale index (rc 1) or a crash. try: with open(queue_path, encoding="utf-8") as fh: text = fh.read() except (OSError, UnicodeDecodeError) as exc: print("cannot read queue: %s" % exc, file=sys.stderr) return 2 payload = parse(text) if args.check: try: doc = open(args.check, encoding="utf-8").read() except (OSError, UnicodeDecodeError) as exc: print("cannot read index document: %s" % exc, file=sys.stderr) return 2 pinned = extract_pinned(doc) if pinned is None: print("UNVERIFIABLE: no parseable pinned block in %s" % args.check) return 2 render = build_doc(payload) if render != doc: # The rendered prose is graded, not only the pin. A pin-only gate # stayed green while the table above it had drifted; that was the # first whole-gate run's finding, so the bytes are compared whole. rl = render.split("\n") dl = doc.split("\n") first = next((i for i in range(min(len(rl), len(dl))) if rl[i] != dl[i]), min(len(rl), len(dl))) print("STALE: rendered document differs at line %d" % (first + 1)) if first < len(dl): print(" document: %s" % dl[first][:160]) if first < len(rl): print(" render: %s" % rl[first][:160]) if pinned is not None: print(" pin: %s" % delta(pinned, payload)) print("regenerate with: python3 tools/queue_index.py --repo %s --write %s" % (repo, args.check)) return 1 print("index current: %s" % summary(payload)) return 0 # `outside` is None outside a git work tree (no corpus to speak of); --index # and --json still work there, which is what makes the tool usable on an # extracted copy of the queue -- a bare `unpack NoneType` was a crash until # this line, found by verifying the tool against `git show :...` blobs. res = outside_ids(repo, args.queue, exclude=(args.write, args.check)) outside, corpus = res if res else ({}, 0) if args.outside: print("outside the queue: %d id(s) named by the %d tracked *.md files read " "(`tools/` excluded), which `WORK-QUEUE.md` never mentions" % (len(outside or {}), corpus or 0)) for qid, files in (outside or {}).items(): print(" %-5s %s" % (qid, ", ".join(files[:4]) + (" +%d" % (len(files) - 4) if len(files) > 4 else ""))) return 0 if args.write: try: with open(args.write, "w", encoding="utf-8") as fh: fh.write(build_doc(payload)) except OSError as exc: print("cannot write --write " + args.write + ": " + str(exc), file=sys.stderr) return 2 print("wrote %s (%d bytes)" % (args.write, os.path.getsize(args.write))) return 0 if args.json: try: with open(args.json, "w", encoding="utf-8") as fh: json.dump(payload, fh, indent=2, sort_keys=True) fh.write("\n") except OSError as exc: print("cannot write --json " + args.json + ": " + str(exc), file=sys.stderr) return 2 if args.index: print("%-4s %-5s %-7s %-11s %-7s %-10s %s" % ("#", "id", "kind", "lines", "bytes", "role", "title")) for i, s in enumerate(payload["items"], 1): print("%-4d %-5s %-7s %-11s %-7d %-10s %s" % (i, s["id"], s["kind"], "%d-%d" % (s["line"], s["end_line"]), s["bytes"], s["role"][:10], clip(s["title"], 72))) print() for r in payload["nested"]: print("nested %-4s line %-5d refs %-3d inside %-4s %s" % (r["id"], r["line"], r["refs"], r["section"] or "--", r["shape"])) if outside: print("outside the queue: %s" % ", ".join(outside)) print() print(summary(payload)) return 0 # -------------------------------------------------------------------------- # selftest: a fixture corpus, one cell per stated rule # -------------------------------------------------------------------------- FIXTURE = """# Work queue Preface text. ## Q1 — First item *(lane)* **Status: DONE at `abc1234`.** Body. ``` ## Q9 — not a section, this is fenced ``` ## Q2 — Second item Body with no status marker. ## Q1 — closure — verified **Status: OPEN.** A continuation section for Q1. ## Q3 §9 follow-on — odd shape Prose that names Q41 before its record. - **Q41 — OPEN (pipe, 2026-10-13, unclaimed).** A nested record. - prose mentioning Q42 in passing. - **Q43 opened — a bold lead record for the fixture.** - a bullet carrying **Q44** mid-line, which is not a record. - a tick quoting `--outside` = Q77/Q99 as prose. **Status: OPEN.** The status lead opens Q45 on this line. """ def selftest(): cells = [0] bad = [0] def check(name, got, want): cells[0] += 1 if got != want: bad[0] += 1 print("FAIL %s\n got %r\n want %r" % (name, got, want)) p = parse(FIXTURE) secs = p["sections"] check("four sections parsed", len(secs), 4) check("ids in order", [s["id"] for s in secs], ["Q1", "Q2", "Q1", "Q3"]) check("fenced heading is not a section", "Q9" in p["ids_with_section"], False) check("second Q1 is a continuation", secs[2]["role"], "continuation") check("continuation detail is the role word", secs[2]["role_detail"], "closure") check("odd heading shape still parses", secs[3]["id"], "Q3") check("multi-section map names Q1", p["ids_multi_section"], {"Q1": [5, 17]}) check("status marker extracted verbatim", secs[0]["status"], "DONE at `abc1234`.") check("status line number", secs[0]["status_line"], 7) check("section with no marker is None", secs[1]["status"], None) check("continuation status is its own", secs[2]["status"], "OPEN.") # `**Status:** value` -- the bold run closes *before* the value. Cutting at # the first `**` returned "" and the index published a stated status as # `--` (Q13:1129 was the live case in the real queue). lead = parse("## Q1 — a\n\n**Status:** 1 of 8 decided.\n\n" "## Q2 — b\n\n**Status:** **OPEN.** tail.\n\n" "## Q3 — c\n\n**Status:**\n") check("status whose bold closes before its value", lead["sections"][0]["status"], "1 of 8 decided.") check("that marker still reports its own line", lead["sections"][0]["status_line"], 3) check("status lead with a bold value still extracts", lead["sections"][1]["status"], "OPEN.") check("a marker with no value at all stays None", lead["sections"][2]["status"], None) check("nested ids found", [r["id"] for r in p["nested"]], ["Q41", "Q43", "Q45"]) check("nested owner is the containing section", p["nested"][0]["section"], "Q3") check("nested shape is bullet carrying a bold lead", p["nested"][0]["shape"], "BULLET+BOLD") check("bold lead is a record form", p["nested"][0]["form"], "BOLD-LEAD") check("status lead is a record form", p["nested"][2]["form"], "STATUS-LEAD") check("record line follows the first mention of the same id", (p["nested"][0]["line"], p["nested"][0]["first_mention"]), (24, 23)) check("prose mention is not a record (Q42)", "Q42" in [r["id"] for r in p["nested"]], False) check("mid-line bold mention is not a record (Q44)", "Q44" in [r["id"] for r in p["nested"]], False) check("quoted tool output opens nothing (Q77/Q99)", {"Q77", "Q99"} & {r["id"] for r in p["nested"]}, set()) check("mentions are still counted as refs", p["nested"][0]["refs"], 2) check("items tile the file with no gap", all(p["items"][i]["end_line"] + 1 == p["items"][i + 1]["line"] for i in range(len(p["items"]) - 1)), True) check("a record block carves its own bounds out of its neighbour", [(b["id"], b["line"], b["end_line"]) for b in p["blocks"]], [("Q41", 24, 25), ("Q43", 26, 28), ("Q45", 29, 29)]) check("the section that used to swallow them now stops before the block", (secs[3]["id"], secs[3]["line"], secs[3]["end_line"]), ("Q3", 21, 23)) check("last item ends at the last line", p["items"][-1]["end_line"], p["queue_lines"]) check("preamble ends before the first heading", p["preamble"]["end_line"], 4) check("byte accounting is exact", p["preamble"]["bytes"] + sum(s["bytes"] for s in p["items"]), p["queue_bytes"]) # a block between two headings keeps both headings as sections and takes the # lines between them: the case that made `sections tile` a false statement gap = parse("## Q99 — a\n\nbody\n\n- **Q102 — OPEN (pipe).**\n\nprose\n\n## Q100 — b\n") check("a block between two headings does not dissolve them", [s["id"] for s in gap["sections"]], ["Q99", "Q100"]) check("the section before the block stops at it", [(s["id"], s["line"], s["end_line"]) for s in gap["sections"][:1]], [("Q99", 1, 4)]) check("the block owns the lines up to the next heading", [(b["id"], b["line"], b["end_line"]) for b in gap["blocks"]], [("Q102", 5, 8)]) check("items still tile across a block boundary", [(i["id"], i["line"], i["end_line"]) for i in gap["items"]], [("Q99", 1, 4), ("Q102", 5, 8), ("Q100", 9, 9)]) check("bytes are the file's own length", p["queue_bytes"], len(FIXTURE.encode())) check("sha256 is the file's own", p["queue_sha256"], hashlib.sha256(FIXTURE.encode()).hexdigest()) # --- the id class: three digits are an id (Q103, 2026-10-27) ------------- # The fixture above stops at Q99, exactly like the real queue did: `\d{1,2}` # made every item from Q100 up body text of the section before it. These # cells fail on the pre-fix pattern, which is why they are here. three = parse("## Q99 — a\n\nbody\n\n## Q100 — b\n\nQ101 is named here.\n\n" "## Q101 — c\n\n- **Q102 — OPEN (pipe).**\n") check("three-digit heading is a section", [s["id"] for s in three["sections"]], ["Q99", "Q100", "Q101"]) check("three-digit heading is an id", three["ids_with_section"], ["Q99", "Q100", "Q101"]) pairs = [(a["end_line"], b["line"]) for a, b in zip(three["sections"], three["sections"][1:])] check("three-digit section bounds do not overlap the one before", pairs[:1], [(4, 5)]) check("sections still tile across the id boundary", all(three["sections"][i]["end_line"] + 1 == three["sections"][i + 1]["line"] for i in range(len(three["sections"]) - 1)), True) check("a three-digit bold record is a nested id (form kept)", [(r["id"], r["form"]) for r in three["nested"]], [("Q102", "BOLD-LEAD")]) # No width ceiling: a fourth digit is a new id, never a truncation of a # three-digit one. `\d{1,3}` would fail both cells below -- the ceiling is # the bug one digit later, so it is not restored. four = parse("## Q1000 — d\n") check("four-digit heading is its own id", four["ids_with_section"], ["Q1000"]) check("a longer id is one token, not a prefix match", ID_TOKEN.findall("see Q1000 and Q100, plus Q1000x"), ["Q1000", "Q100"]) check("an id glued to a word is not an id", ID_TOKEN.findall("FAQ100 / xQ100 / Q100y / Q"), []) check("the old id class could not see Q100; this one does (RED -> GREEN)", (re.compile(r"\bQ\d{1,2}\b").findall("## Q100 — b"), HEADING.match("## Q100 — b") is not None), ([], True)) # --- the gate, round trip and both directions of falsification ---------- root = tempfile.mkdtemp(prefix="queue-index-selftest-") repo = os.path.join(root, "repo") os.makedirs(os.path.join(repo, "docs")) queue = os.path.join(repo, "WORK-QUEUE.md") doc_path = os.path.join(repo, DOC_PATH) subprocess.check_call(["git", "-C", repo, "init", "-q", "-b", "main"]) subprocess.check_call(["git", "-C", repo, "config", "user.email", "s@invalid"]) subprocess.check_call(["git", "-C", repo, "config", "user.name", "s"]) with open(queue, "w", encoding="utf-8") as fh: fh.write(FIXTURE) subprocess.check_call(["git", "-C", repo, "add", "-A"]) subprocess.check_call(["git", "-C", repo, "commit", "-q", "-m", "fixture"]) import contextlib import io buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=doc_path, check=None, outside=False)) check("--write exits 0", rc, 0) check("document written", os.path.exists(doc_path), True) doc = open(doc_path, encoding="utf-8").read() check("document carries the pinned block", extract_pinned(doc) is not None, True) buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=None, check=doc_path, outside=False)) check("--check on its own output exits 0", rc, 0) check("--check says current", "index current:" in buf.getvalue(), True) # --- the document is a function of the queue alone ----------------------- # A prose file that names an id the queue never mentions must not move a # byte of the document, and must not turn --check RED. Baking the corpus # table into the document did both: same queue, three different documents, # and a checkin quoting an id reddened a gate about the queue's currency. with open(os.path.join(repo, "notes.md"), "w", encoding="utf-8") as fh: fh.write("# notes\n\nQ63 is named only here; Q1 is a queue id.\n") subprocess.check_call(["git", "-C", repo, "add", "notes.md"]) before = open(doc_path, encoding="utf-8").read() buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=doc_path, check=None, outside=False)) check("re-render beside a prose mention exits 0", rc, 0) check("a prose mention does not change the document", open(doc_path, encoding="utf-8").read(), before) buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=None, check=doc_path, outside=False)) check("--check stays green when prose names an id", rc, 0) # --- the outside leg is reported, not embedded --------------------------- buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=None, check=None, outside=True)) out = buf.getvalue() check("--outside exits 0", rc, 0) check("--outside counts the corpus it actually read", "1 id(s) named by the 2 tracked *.md files" in out, True) check("--outside names Q63 and the file that carries it", " Q63 notes.md" in out, True) # --- a tree that is not a git repo must still index ----------------------- # The `outside` leg needs git; --index does not. A verifier that extracts # `git show :WORK-QUEUE.md` into a temp dir has no .git, and this used to # crash on `outside, corpus = outside_ids(...)` returning None. plain = os.path.join(root, "plain") os.makedirs(plain) with open(os.path.join(plain, "WORK-QUEUE.md"), "w", encoding="utf-8") as fh: fh.write(FIXTURE) buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(argparse.Namespace(repo=plain, queue="WORK-QUEUE.md", index=True, json=None, write=None, check=None, outside=False)) check("--index outside a git work tree exits 0", rc, 0) check("--index outside a git work tree still summarizes", "index: 7 items (4 sections, 3 record blocks)" in buf.getvalue(), True) # falsify 1: an append moves the last section's extent with open(queue, "a", encoding="utf-8") as fh: fh.write("- an appended line.\n") buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=None, check=doc_path, outside=False)) check("--check goes RED on an append", rc, 1) check("RED names the moved section", "changed extent or status" in buf.getvalue(), True) # falsify 2: a new heading is a new section with open(queue, "a", encoding="utf-8") as fh: fh.write("\n## Q7 — a brand new item\n\n**Status: OPEN.**\n") buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=None, check=doc_path, outside=False)) check("--check goes RED on a new heading", rc, 1) check("RED names the new section", "new section(s): Q7" in buf.getvalue(), True) # falsify 3: a document with no block cannot be verified open(doc_path, "w", encoding="utf-8").write("# empty\n") buf = io.StringIO() with contextlib.redirect_stdout(buf): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=None, check=doc_path, outside=False)) check("no pin -> exit 2", rc, 2) check("no pin -> says so", "UNVERIFIABLE" in buf.getvalue(), True) # --- a read error is an exit code, not a traceback ----------------------- # `--queue` at a missing path or at bytes that are not UTF-8 used to raise # straight out of run() (FileNotFoundError / UnicodeDecodeError, rc 1). The # repo's other reader exits 2 with `cannot read queue:`; this tool now does # the same, and owes the --check document the same answer: an unreadable # gate input is rc 2, never the rc 1 that means "stale". bad_q = os.path.join(root, "bad-utf8.md") with open(bad_q, "wb") as fh: fh.write(b"\xff\xfe\x80\n") err = io.StringIO() with contextlib.redirect_stderr(err): rc = run(argparse.Namespace(repo=root, queue="missing-queue.md", index=False, json=None, write=None, check=None, outside=False)) check("missing queue -> exit 2", rc, 2) check("missing queue -> says cannot read queue", err.getvalue().startswith("cannot read queue:"), True) err = io.StringIO() with contextlib.redirect_stderr(err): rc = run(argparse.Namespace(repo=root, queue="bad-utf8.md", index=False, json=None, write=None, check=None, outside=False)) check("non-UTF-8 queue -> exit 2", rc, 2) check("non-UTF-8 queue -> says cannot read queue", "cannot read queue:" in err.getvalue(), True) err = io.StringIO() with contextlib.redirect_stderr(err): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=None, check=os.path.join(repo, "missing-index.md"), outside=False)) check("missing index document -> exit 2", rc, 2) check("missing index document -> names the document", "cannot read index document:" in err.getvalue(), True) err = io.StringIO() with contextlib.redirect_stderr(err): rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False, json=None, write=None, check=bad_q, outside=False)) check("non-UTF-8 index document -> exit 2", rc, 2) check("non-UTF-8 index document -> names the document", "cannot read index document:" in err.getvalue(), True) print("selftest: %d cells, %d passed, %d failed" % (cells[0], cells[0] - bad[0], bad[0])) return 1 if bad[0] else 0 def main(argv=None): ap = argparse.ArgumentParser( description="queue_index.py -- the work queue's table of contents, and the gate " "that it is current.") ap.add_argument("--repo", default=".") ap.add_argument("--queue", default=QUEUE_DEFAULT) ap.add_argument("--index", action="store_true", help="print the section table") ap.add_argument("--json", metavar="PATH", help="write the parse here") ap.add_argument("--write", metavar="PATH", help="render the index document here") ap.add_argument("--check", metavar="PATH", help="re-derive and compare a rendered index") ap.add_argument("--outside", action="store_true", help="report ids the prose corpus names that the queue never mentions") ap.add_argument("--selftest", action="store_true", help="fixture corpus, 73 cells") args = ap.parse_args(argv) if args.selftest: return selftest() return run(args) if __name__ == "__main__": sys.exit(main())