1152 lines
57 KiB
Python
1152 lines
57 KiB
Python
#!/usr/bin/env python3
|
||
"""queue_index.py -- the work queue's table of contents, and the gate that it is current.
|
||
|
||
WHY THIS EXISTS
|
||
`WORK-QUEUE.md` is 1.2 MB and 9,953 lines long at `39312c2`. It has no table of contents.
|
||
Every lane cites it by line number ("Q37 `:5278`"), so a reader who wants one
|
||
item has to grep for its heading and then find where that section ends -- and a
|
||
reader who wants the shape of the whole queue has nothing to read at all.
|
||
|
||
The shape is not obvious even after a grep. Measured on 2026-10-17 at HEAD
|
||
(`ece5228`): 49 `## ` sections carry 45 distinct ids, because four ids (Q37,
|
||
Q38, Q39, Q43) own a second section each -- a `closure` or `claimed` block
|
||
appended under a fresh heading. And 9 further ids (Q41, Q42, Q46, Q47, Q50,
|
||
Q51, Q52, Q53, Q63) have NO heading of their own: they are opened as bold
|
||
records nested inside somebody else's section. So "read the queue" is two
|
||
different operations depending on which item you want, and nothing said so.
|
||
|
||
THE RECORD RULE (Q63, 2026-10-17). An id is *opened by a record, not mentioned
|
||
by prose*: a heading, a bold lead that opens with the id (`**Qnn -- ...**` /
|
||
`**Qnn opened -- ...**`), or a line whose own lead is a `Status:` marker. Before
|
||
that rule the detector admitted every id token that had no heading, so a tick
|
||
quoting `--outside = Q63/Q77/Q99` opened three ids, an artifact path
|
||
(`out/Q54-READBACK.txt`) opened Q54, and the range `Q31-Q62` opened Q62. The
|
||
first *mention* also used to be reported as the record, which is wrong twice:
|
||
Q53 is first named in prose well before it is opened (:7246 prose vs :7292
|
||
record) and Q63 is first named inside the very tick that quotes it (:9909) but
|
||
opened at :9918. The table below now carries the record line, the first-mention
|
||
line and the form that admitted the id, so the difference is visible instead of
|
||
implied.
|
||
|
||
An id is `Q` and one or more digits, no width (Q103, 2026-10-27). The pattern
|
||
read `Q` plus one or two digits, so from Q100 up an id was not a section and
|
||
not even a token: a
|
||
`## Q100 -- ...` heading parsed as body text of the section before it and that
|
||
section's byte count absorbed the whole tail behind it. Three-digit ids are now
|
||
sections, tokens and record ids alike, with no ceiling restored at three digits
|
||
-- a fixed width is the defect, so a three-digit ceiling would only postpone it.
|
||
The trailing word boundary is what stops `Q100` matching inside `Q1000`.
|
||
|
||
TIER TWO HAS BOUNDS NOW (Q103). A `**Qnn -- ...**` record that opens an id with
|
||
no heading of its own is an item too: its block runs from the record line to
|
||
the line before the next item opens, so `Q98`'s row stopped carrying the 40,957
|
||
bytes of `Q100`-`Q103` and each of those ids has a row and an extent.
|
||
|
||
WHAT IT MEASURES -- structural facts only, one row per item -- a `## ` section,
|
||
or (Q103) a record block that opens an id with no heading of its own
|
||
id, line, end_line, role, title, lines, bytes, first `Status:` marker.
|
||
role=opening the first section carrying that id
|
||
role=continuation a later section for an id already seen (a closure or
|
||
claim block); its `role_detail` names the first word
|
||
lines/bytes the section's own extent, header line to last line
|
||
status the first `Status:` line in the section, VERBATIM and
|
||
truncated. NOT adjudicated: whether an item is open is
|
||
`tools/queue_claim_check.py`'s verdict, not this tool's.
|
||
Two instruments disagreeing about closure is a bug this
|
||
tool declines to introduce.
|
||
|
||
Plus the two legs that make the structure visible:
|
||
nested ids opened by a record inside a section that has no heading of their
|
||
own: the record's line, the first line that mentions it, the section
|
||
that contains it, and the shape/form of the record line
|
||
outside ids referenced by the prose corpus (checkins/ replies/ recoveries/
|
||
audits/ access/ decisions/ docs/ and the top-level documents) but
|
||
absent from the queue entirely -- `tools/` is excluded on purpose:
|
||
its fixtures carry synthetic ids by construction, and this tool's
|
||
own selftest is one of them.
|
||
|
||
The `outside` leg is REPORTED, not embedded: `--outside` prints it, and the
|
||
rendered document does not contain it. It cannot. That set is a function of
|
||
the working tree's prose, not of the queue, so baking it into a document that
|
||
`--check` byte-compares against a fresh render means any checkin that quotes
|
||
an id turns the queue's gate RED for a reason that has nothing to do with the
|
||
queue. Measured on 2026-10-17: `--outside` read Q63 from `recoveries/`, then
|
||
Q63+Q77+Q99 once a checkin described the test that names Q77 -- same queue,
|
||
same sha256, three different documents. A table of contents must be a
|
||
function of the file it indexes.
|
||
|
||
Headings inside a fenced block are never sections; a heading whose id is
|
||
followed by `closure --` or `§9 follow-on --` is still that id's section.
|
||
|
||
WHAT IT DOES NOT DO
|
||
It does not grade the work, does not adjudicate closure, and does not read
|
||
git history. It answers "what is in this file and where", and it keeps the
|
||
generated copy of that answer honest: `--write` renders `docs/queue-index.md`
|
||
with the parse pinned inside a machine block, and `--check` re-renders the
|
||
whole document -- prose table included, compared byte for byte -- and goes RED
|
||
the moment the two disagree. (The pin alone was not enough: the first
|
||
whole-gate run found the table above the pin drifted while the pin still
|
||
matched, because the document had begun listing itself in its own corpus.
|
||
The rendered file is excluded from that corpus, which makes `--write` a fixed
|
||
point, and the bytes are now compared whole.) The document is therefore a
|
||
function of `WORK-QUEUE.md` alone: the same queue renders the same bytes
|
||
whatever the prose corpus says, which is what makes `--check` a statement
|
||
about the queue's currency and not about the last person to write a checkin.
|
||
|
||
`--check` exit codes: 0 current, 1 stale (with the delta printed), 2 the
|
||
document carries no parseable pinned block, 3 the queue or the document could
|
||
not be read at all (missing path, or bytes that are not UTF-8). An unreadable
|
||
input is a precondition failure, never a stale index and never a traceback:
|
||
`--queue` at a missing path or at a binary used to raise FileNotFoundError /
|
||
UnicodeDecodeError out of `run()` with rc 1, which reads like "stale" to every
|
||
caller that only checks the exit code.
|
||
|
||
READ-ONLY except `--json PATH` and `--write PATH`.
|
||
|
||
USAGE
|
||
python3 tools/queue_index.py --index # 49 sections, one screen each side
|
||
python3 tools/queue_index.py --outside # ids the prose corpus names and the queue does not
|
||
python3 tools/queue_index.py --json out.json
|
||
python3 tools/queue_index.py --write docs/queue-index.md
|
||
python3 tools/queue_index.py --check docs/queue-index.md
|
||
python3 tools/queue_index.py --selftest # fixture corpus, 80 cells
|
||
|
||
Re-run `--write` after any append to the queue. The document carries the
|
||
queue's sha256, so a stale copy is a RED check, never a silent lie.
|
||
"""
|
||
|
||
import argparse
|
||
import hashlib
|
||
import json
|
||
import os
|
||
import re
|
||
import subprocess
|
||
import sys
|
||
import tempfile
|
||
|
||
QUEUE_DEFAULT = "WORK-QUEUE.md"
|
||
DOC_PATH = "docs/queue-index.md"
|
||
BEGIN = "<!-- queue-index:begin -->"
|
||
END = "<!-- queue-index:end -->"
|
||
|
||
# THE ID CLASS (Q103, 2026-10-27). An id is `Q` and one or more digits -- no
|
||
# width. It was `\d{1,2}`, so from Q100 up an item was not a section and not even
|
||
# a token: `## Q100 -- ...` parsed as body text of whatever section preceded it
|
||
# and the eight-figure byte count of a whole tail accrued to its neighbour. The
|
||
# trailing `\b` is load-bearing in the other direction: without it `Q100` is a
|
||
# prefix-match inside `Q1000` and `Q1000` would be indexed as the item Q100.
|
||
# A width ceiling is deliberately NOT restored: `\d{1,3}` reproduces this exact
|
||
# defect one digit later, which is the shape of the bug, not a fix for it.
|
||
HEADING = re.compile(r"^##\s+(Q\d+)\b(.*)$")
|
||
ID_TOKEN = re.compile(r"\bQ\d+\b")
|
||
STATUS = re.compile(r"^\s*\*{0,2}Status:\s*(.*)$")
|
||
FENCE = re.compile(r"^(```|~~~)")
|
||
|
||
# A record opens with its id; a mention merely carries it. `BULLET` strips the
|
||
# markdown that can precede a record without being part of it, and `OPEN_SEP` is
|
||
# what may follow the id in a bold lead that is *opening* an item (Q46's
|
||
# "**Q46 (Helm lane) — OPEN ...**" and Q63's "**Q63 opened — ...**" both count;
|
||
# a "**Q7, measured:**" bold run does not).
|
||
BULLET = re.compile(r"^[\s>|]*(?:[-*+]\s+)?")
|
||
OPEN_SEP = re.compile(r"\s*(?:—|–|-|:|\(|\bopened\b|\bopen\b|\bOPEN\b|\bOPENED\b)")
|
||
|
||
ROLE_WORDS = ("closure", "claimed", "§", "status")
|
||
|
||
|
||
def record_form(line, qid):
|
||
"""The form in which `line` opens `qid`, or None if it only mentions it.
|
||
|
||
The counting rule is settled in `WORK-QUEUE.md` at Q63 (2026-10-17): an id is
|
||
*opened by a record, not mentioned by prose*. Three forms count --
|
||
|
||
HEADING `## Qnn ...` (a section; never nested)
|
||
BOLD-LEAD after bullet/bold markers the line opens `**Qnn` and the id is
|
||
followed by a separator or the word open/opened
|
||
STATUS-LEAD the line's own lead is a `Status:` marker (bold or plain) and
|
||
it names the id
|
||
|
||
A bare token inside a sentence, a tick quoting tool output (`= Q63/Q77/Q99`),
|
||
an artifact filename (`out/Q54-READBACK.txt`) or a range (`Q31–Q62`) matches
|
||
none of them, which is the point: those were opening ids before this rule.
|
||
"""
|
||
s = line.strip()
|
||
m = HEADING.match(s)
|
||
if m and m.group(1) == qid:
|
||
return "HEADING"
|
||
body = BULLET.sub("", s)
|
||
if qid not in ID_TOKEN.findall(body):
|
||
return None
|
||
if body.startswith("**Status:") or body.startswith("Status:"):
|
||
return "STATUS-LEAD"
|
||
if body.startswith("**"):
|
||
inner = body[2:].lstrip("*").lstrip()
|
||
m = ID_TOKEN.match(inner)
|
||
if m and m.group(0) == qid and OPEN_SEP.match(inner[m.end():]):
|
||
return "BOLD-LEAD"
|
||
return None
|
||
|
||
|
||
def record_detail(line, qid):
|
||
"""The first word after a record's bold lead -- `DONE`, `CLOSED`, `opened`.
|
||
|
||
The heading loop's `role_detail` already names the first word after a
|
||
continuation heading's dash; this is the same field for a continuation
|
||
record, so the table says *why* a second block exists without a reader
|
||
having to open it.
|
||
"""
|
||
body = BULLET.sub("", line.strip())
|
||
if not body.startswith("**"):
|
||
return ""
|
||
inner = body[2:].lstrip("*").lstrip()
|
||
if inner.startswith(qid):
|
||
inner = inner[len(qid):]
|
||
for word in inner.lstrip("—-–: ").split():
|
||
return word.strip("*().,;:").strip("*")
|
||
return ""
|
||
|
||
|
||
def split_lines(text):
|
||
"""Lines without their terminators, plus the byte length each line occupies."""
|
||
lines = text.split("\n")
|
||
trailing_newline = text.endswith("\n")
|
||
if trailing_newline:
|
||
lines = lines[:-1]
|
||
return lines, trailing_newline
|
||
|
||
|
||
def line_bytes(line, is_last, trailing_newline):
|
||
if is_last and not trailing_newline:
|
||
return len(line.encode("utf-8"))
|
||
return len(line.encode("utf-8")) + 1
|
||
|
||
|
||
def status_value(raw):
|
||
"""The value of a `Status:` marker, in either emphasis form.
|
||
|
||
`**Status: value**` puts the value *inside* the bold run; `**Status:** value`
|
||
closes the bold *before* it. The `STATUS` pattern consumes the optional
|
||
leading `**`, so cutting at the first `**` is right for the first form and
|
||
returns "" for the second -- which published a stated status as `--`.
|
||
Q13:1129 (`**Status:** 1 of 8 has a decision ...`) was the live case.
|
||
"""
|
||
raw = raw.strip()
|
||
while raw.startswith("**"):
|
||
raw = raw[2:].strip()
|
||
if "**" in raw:
|
||
raw = raw.split("**", 1)[0].strip()
|
||
return raw
|
||
|
||
|
||
def fence_mask(lines):
|
||
"""1-based line numbers to skip: fence delimiters and every line they hide.
|
||
|
||
A fenced block is quoted material, not document structure -- headings inside
|
||
it are not sections, and id tokens inside it are not records. Stated once
|
||
here and reused by parse(), rather than restated verbatim in each scan loop.
|
||
"""
|
||
out = set()
|
||
in_fence = False
|
||
for i, ln in enumerate(lines, 1):
|
||
if FENCE.match(ln.strip()):
|
||
in_fence = not in_fence
|
||
out.add(i)
|
||
elif in_fence:
|
||
out.add(i)
|
||
return out
|
||
|
||
|
||
def parse(text):
|
||
"""Return the structural parse: items, nested ids, id map, byte accounting.
|
||
|
||
An *item* is a heading section OR a record block (Q103, 2026-10-27). An id
|
||
opened by a record -- `**Q100 -- OPEN ...**` inside somebody else's section --
|
||
is an item with its own extent, from its record line to the line before the
|
||
next item opens, instead of being counted as prose inside its neighbour. The
|
||
two tiers are one table here and `kind` says which: before this, a record
|
||
opening an id with no heading handed its bytes to the section around it (Q98's
|
||
row carried 40,957 bytes of four later items) and an id from Q100 up was not
|
||
even a token.
|
||
"""
|
||
lines, trailing_newline = split_lines(text)
|
||
n = len(lines)
|
||
sizes = [line_bytes(ln, i == n - 1, trailing_newline) for i, ln in enumerate(lines)]
|
||
fenced = fence_mask(lines)
|
||
|
||
headings = [] # (line_no, id, rest)
|
||
for i, ln in enumerate(lines, 1):
|
||
if i in fenced:
|
||
continue
|
||
m = HEADING.match(ln)
|
||
if m:
|
||
headings.append((i, m.group(1), m.group(2).strip()))
|
||
heading_ids = {h[1] for h in headings}
|
||
|
||
# every id token in the queue, counted as a mention, plus the first line that
|
||
# opens it as a record. Fenced lines are skipped: an id in a code block is
|
||
# quoted text, not a record.
|
||
#
|
||
# Q63 (2026-10-17): the first *mention* is not the record. Q53 is first named
|
||
# in prose 46 lines before its `**Q53 opened -- ...**` line, and Q63 is named
|
||
# inside a tick quoting tool output at :9909 but opened at :9918. So a nested
|
||
# id is admitted by its first record-form line, and an id that never has one
|
||
# (Q54/Q55/Q56/Q58 as artifact filenames, Q62 inside `Q31-Q62`, Q77/Q99 inside
|
||
# a quoted `--outside` list) is not an item at all.
|
||
mentioned = {}
|
||
for i, ln in enumerate(lines, 1):
|
||
if i in fenced:
|
||
continue
|
||
for m in ID_TOKEN.finditer(ln):
|
||
qid = m.group(0)
|
||
hit = mentioned.get(qid)
|
||
if hit is None:
|
||
hit = mentioned[qid] = {"line": i, "refs": 0, "record": None, "form": None}
|
||
hit["refs"] += 1
|
||
if hit["record"] is None:
|
||
form = record_form(ln, qid)
|
||
if form:
|
||
hit["record"] = i
|
||
hit["form"] = form
|
||
|
||
# tier two: ids opened by a record and owning no heading of their own
|
||
records = [(mentioned[qid]["record"], qid, mentioned[qid]["form"])
|
||
for qid in sorted(mentioned, key=lambda s: int(s[1:]))
|
||
if qid not in heading_ids and mentioned[qid]["record"] is not None]
|
||
|
||
# tier two, later records (Q103, 2026-10-28): an id's SECOND block-initial
|
||
# record -- a closure, a correction -- is an item too, with its own bounds.
|
||
# Before this, only the first record line per id cut anything, so every later
|
||
# closure block was charged to whichever item preceded it: measured on this
|
||
# queue before the fix, `**Q103 -- DONE ...**` (30 lines appended at the foot)
|
||
# grew Q104's row from 5,990 to 14,007 bytes, and Q100's own DONE block at
|
||
# :12915 sat inside Q101's row. That is the same swallow the id class had,
|
||
# one level down.
|
||
#
|
||
# The admission rule is deliberately narrow, because `record_form` alone is
|
||
# too loose here: 171 lines in this queue carry a record form for some id and
|
||
# 49 ids have more than one, but most of the extras are bullet status lines
|
||
# (`- **Q16:** unchanged at ...`) that are prose inside an item and must not
|
||
# split it. A later record is a boundary only when it BEGINS a block: the
|
||
# line opens with `**` at column 0, and the line above it is blank or a `---`
|
||
# rule or an HTML comment. Measured: 52 lines qualify, every one of them a
|
||
# standalone block in the file.
|
||
def begins_block(i):
|
||
prev = lines[i - 2].strip() if i >= 2 else ""
|
||
return prev == "" or prev == "---" or prev.startswith("<!--")
|
||
|
||
first_records = {(r[0], r[1]) for r in records}
|
||
later = []
|
||
for i, ln in enumerate(lines, 1):
|
||
if i in fenced or not ln.startswith("**") or not begins_block(i):
|
||
continue
|
||
for qid in sorted(set(ID_TOKEN.findall(ln)), key=lambda s: int(s[1:])):
|
||
if (i, qid) in first_records or record_form(ln, qid) != "BOLD-LEAD":
|
||
continue
|
||
later.append((i, qid, "BOLD-LEAD"))
|
||
break
|
||
|
||
# Every item opens somewhere and owns the lines up to the next opening: a
|
||
# heading, or a record that opens an id with no heading, or a later
|
||
# block-initial record for an id that is already an item. Nothing else cuts a
|
||
# section, so a bold *mention* inside a section does not split it.
|
||
boundaries = sorted({h[0] for h in headings} | {r[0] for r in records}
|
||
| {r[0] for r in later})
|
||
|
||
def end_of(line_no):
|
||
nxt = [b for b in boundaries if b > line_no]
|
||
return (nxt[0] - 1) if nxt else n
|
||
|
||
items = []
|
||
seen = {}
|
||
for line_no, qid, rest in headings:
|
||
role = "opening" if qid not in seen else "continuation"
|
||
detail = ""
|
||
if role == "continuation":
|
||
words = rest.lstrip("—- ").split()
|
||
detail = words[0].strip("*(") if words else ""
|
||
seen.setdefault(qid, line_no)
|
||
items.append({"id": qid, "line": line_no, "kind": "heading", "role": role,
|
||
"role_detail": detail, "title": clip(rest.lstrip("—- ").strip(), 200)})
|
||
for line_no, qid, form in records:
|
||
raw = lines[line_no - 1]
|
||
seen.setdefault(qid, line_no)
|
||
items.append({"id": qid, "line": line_no, "kind": "record", "role": "record",
|
||
"role_detail": form,
|
||
"title": clip(BULLET.sub("", raw).strip(), 200)})
|
||
for line_no, qid, form in later:
|
||
raw = lines[line_no - 1]
|
||
role = "continuation" if qid in seen else "record"
|
||
seen.setdefault(qid, line_no)
|
||
items.append({"id": qid, "line": line_no, "kind": "record", "role": role,
|
||
"role_detail": record_detail(raw, qid) if role == "continuation"
|
||
else form,
|
||
"title": clip(BULLET.sub("", raw).strip(), 200)})
|
||
items.sort(key=lambda it: it["line"])
|
||
|
||
for it in items:
|
||
end = end_of(it["line"])
|
||
status = None
|
||
status_line = None
|
||
for off, ln in enumerate(lines[it["line"]:end], start=it["line"] + 1):
|
||
m = STATUS.match(ln)
|
||
if m:
|
||
status = status_value(m.group(1)) or None
|
||
status_line = off
|
||
break
|
||
it["end_line"] = end
|
||
it["lines"] = end - it["line"] + 1
|
||
it["bytes"] = sum(sizes[it["line"] - 1:end])
|
||
it["status"] = clip(status, 200) if status else None
|
||
it["status_line"] = status_line
|
||
|
||
sections = [it for it in items if it["kind"] == "heading"]
|
||
blocks = [it for it in items if it["kind"] == "record"]
|
||
|
||
ids: dict = {}
|
||
for s in sections:
|
||
ids.setdefault(s["id"], []).append(s["line"])
|
||
|
||
# `nested` carries what a bounds row cannot: the record's form, how often the
|
||
# id is named, the first line that names it at all, and the item that opens
|
||
# before it (its parent). A block is the same id with bounds; both are
|
||
# reported because a reader needs the line and the shape.
|
||
nested = []
|
||
for it in blocks:
|
||
# one row per id, not per block: a later closure block for the same id is
|
||
# an item with bounds (see `items`), but the table below is the id's
|
||
# record -- its form and how often the file names it -- so the opening
|
||
# block stands for it (Q103, 2026-10-28).
|
||
if it["role"] != "record":
|
||
continue
|
||
qid = it["id"]
|
||
rec = mentioned[qid]
|
||
idx = [k for k, cand in enumerate(items) if cand["line"] == it["line"]][0]
|
||
owner = items[idx - 1]["id"] if idx else None
|
||
raw = lines[it["line"] - 1]
|
||
stripped = raw.lstrip()
|
||
if stripped.startswith("#"):
|
||
shape = "HEADING"
|
||
elif stripped.startswith(("-", "*", "|")):
|
||
shape = "BULLET"
|
||
elif stripped.startswith("**"):
|
||
shape = "BOLD"
|
||
else:
|
||
shape = "PLAIN"
|
||
if shape not in ("HEADING", "BOLD") and "**" in raw:
|
||
shape += "+BOLD"
|
||
nested.append({"id": qid, "line": it["line"], "section": owner, "shape": shape,
|
||
"refs": rec["refs"], "form": rec["form"],
|
||
"first_mention": rec["line"]})
|
||
|
||
first = headings[0][0] if headings else n + 1
|
||
preamble_end = first - 1
|
||
payload = {
|
||
"queue_lines": n,
|
||
"queue_bytes": sum(sizes),
|
||
"queue_sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(),
|
||
"preamble": {"line": 1, "end_line": preamble_end,
|
||
"bytes": sum(sizes[:preamble_end])},
|
||
"items": items,
|
||
"sections": sections,
|
||
"blocks": blocks,
|
||
"ids_with_section": sorted(ids, key=lambda s: int(s[1:])),
|
||
"ids_multi_section": {k: v for k, v in sorted(
|
||
ids.items(), key=lambda kv: int(kv[0][1:])) if len(v) > 1},
|
||
"nested": nested,
|
||
}
|
||
return payload
|
||
|
||
|
||
def outside_ids(repo, queue_path, exclude=(), exclude_tools=True):
|
||
"""Ids referenced by the prose corpus that the queue never mentions.
|
||
|
||
`exclude` is the rendered document itself: a generated file that names ids
|
||
would otherwise join the corpus that generates it, so `--write` would not be
|
||
a fixed point -- measured once, the first whole-gate run went red on exactly
|
||
that (the document added itself to Q63's file list).
|
||
"""
|
||
try:
|
||
out = subprocess.check_output(["git", "-C", repo, "ls-files"], text=True,
|
||
stderr=subprocess.DEVNULL)
|
||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||
return None
|
||
skip = {os.path.realpath(p) for p in exclude if p}
|
||
files = [f for f in out.split("\n") if f.endswith(".md")]
|
||
if exclude_tools:
|
||
files = [f for f in files if not f.startswith("tools/")]
|
||
# The generated document is never part of the corpus it summarizes, whatever
|
||
# path it is written to -- otherwise a render into any other filename would
|
||
# count the committed copy and the output would cease to be a fixed point.
|
||
files = [f for f in files if f != DOC_PATH]
|
||
queue_abs = os.path.normpath(os.path.join(repo, queue_path))
|
||
queue_text = open(queue_abs, encoding="utf-8", errors="replace").read()
|
||
in_queue = set(ID_TOKEN.findall(queue_text))
|
||
hits = {}
|
||
read = 0
|
||
for f in files:
|
||
path = os.path.join(repo, f)
|
||
if os.path.realpath(path) in skip:
|
||
continue
|
||
try:
|
||
txt = open(path, encoding="utf-8", errors="replace").read()
|
||
except OSError:
|
||
continue
|
||
read += 1
|
||
for m in ID_TOKEN.finditer(txt):
|
||
hits.setdefault(m.group(0), set()).add(f)
|
||
return {k: sorted(v) for k, v in sorted(
|
||
hits.items(), key=lambda kv: int(kv[0][1:])) if k not in in_queue}, read
|
||
|
||
|
||
def summary(payload):
|
||
secs = payload["sections"]
|
||
blocks = payload.get("blocks", [])
|
||
cont = [s for s in secs if s["role"] == "continuation"]
|
||
return ("index: %d items (%d sections, %d record blocks), %d ids, "
|
||
"%d continuation sections, %d nested ids, %d lines / %d bytes"
|
||
% (len(payload["items"]), len(secs), len(blocks),
|
||
len(payload["ids_with_section"]), len(cont), len(payload["nested"]),
|
||
payload["queue_lines"], payload["queue_bytes"]))
|
||
|
||
|
||
def clip(text, limit):
|
||
"""Cut at a word boundary and say so; a silent mid-word cut reads as data loss."""
|
||
if len(text) <= limit:
|
||
return text
|
||
cut = text[:limit]
|
||
if " " in cut[max(0, limit - 20):]:
|
||
cut = cut.rsplit(" ", 1)[0]
|
||
return cut.rstrip(" ,;:—-") + " …"
|
||
|
||
|
||
def nested_prose(nested):
|
||
"""The 'what it exposes' clause about record-opened ids.
|
||
|
||
`build_doc` used to index `nested[0]` unconditionally, so `--write` died
|
||
with IndexError -- leaving a zero-byte document and rc 1, the code that
|
||
means "graded a fault" -- on any queue whose ids all carry headings (a fresh
|
||
queue, a fixture, a partial extract). Found 2026-10-18 by the fixture in
|
||
`tools/queue_append_audit.py --selftest`. The non-empty branch reproduced
|
||
that fix's sentence byte for byte so the render did not move.
|
||
|
||
Q103 (2026-10-27): the sentence used to open "A further **N ids**", which
|
||
double-counted -- the clause before it already said those N ids are items
|
||
with rows and extents, so the paragraph read as 2N. The ids in `blocks` and
|
||
the ids in `nested` are the same set now that a record block is an item (16
|
||
and 16 in this tree, set-equal); the sentence says so instead of implying a
|
||
second population, and the table below is described as the second view of
|
||
the same ids it is.
|
||
"""
|
||
if not nested:
|
||
return (" No id is opened as a record without a heading of its own, so the "
|
||
"table below is empty.")
|
||
return (" The same **%d ids** are listed again in the table below, which carries "
|
||
"what a bounds row cannot -- the form that opened each one and how often the "
|
||
"file names it: grepping `^## %s` finds nothing while %s is a live item the "
|
||
"file names %d times."
|
||
% (len(nested), nested[0]["id"], nested[0]["id"], nested[0]["refs"]))
|
||
|
||
|
||
def build_doc(payload):
|
||
lines = []
|
||
lines.append("# The work queue's table of contents")
|
||
lines.append("")
|
||
lines.append("*Generated by `tools/queue_index.py --write`; do not hand-edit. `python3 "
|
||
"tools/queue_index.py --check docs/queue-index.md` re-derives the parse and "
|
||
"goes RED when this document and `WORK-QUEUE.md` disagree, which is what an "
|
||
"append to the queue produces. Regenerate, commit, move on.*")
|
||
lines.append("")
|
||
lines.append("**What this is.** One row per item of `WORK-QUEUE.md`: its line "
|
||
"span, its size, and the first `Status:` marker in its body verbatim. "
|
||
"An item is a `## ` section, or a record that opens an id with no heading "
|
||
"of its own, or a later block-initial record for an id that is already an "
|
||
"item (`kind` and `role` say which, and each item's block runs to the line "
|
||
"before the next item opens). Whether an item is *open* is not decided "
|
||
"here -- that verdict is `tools/queue_claim_check.py`'s, and this document "
|
||
"does not overrule it.")
|
||
lines.append("")
|
||
blocks = payload.get("blocks", [])
|
||
open_blocks = [b for b in blocks if b["role"] == "record"]
|
||
later_blocks = [b for b in blocks if b["role"] != "record"]
|
||
lines.append("**What it exposes.** The queue is two-tier. %d sections carry %d ids because "
|
||
"%d ids own a second, continuation section; %d further ids are opened by a "
|
||
"record and own no heading at all, and they have rows and extents here -- "
|
||
"%d record blocks in all, %d of them a later closure or correction for an id "
|
||
"already opened.%s"
|
||
% (len(payload["sections"]), len(payload["ids_with_section"]),
|
||
len(payload["ids_multi_section"]), len(open_blocks), len(blocks),
|
||
len(later_blocks), nested_prose(payload["nested"])))
|
||
lines.append("")
|
||
lines.append("| # | id | kind | lines | bytes | role | title | first `Status:` |")
|
||
lines.append("|---|---|---|---|---|---|---|---|")
|
||
for i, s in enumerate(payload["items"], 1):
|
||
status = (s["status"] or "--").replace("|", "\\|")
|
||
title = s["title"].replace("|", "\\|")
|
||
if s["role"] == "opening":
|
||
role = "opening"
|
||
elif s["role"] == "record":
|
||
role = "record: %s" % (s["role_detail"] or "?")
|
||
else:
|
||
role = "cont: %s" % (s["role_detail"] or "?")
|
||
lines.append("| %d | %s | %s | %d-%d | %d | %s | %s | %s |"
|
||
% (i, s["id"], s["kind"], s["line"], s["end_line"], s["bytes"],
|
||
role, title, status))
|
||
lines.append("")
|
||
lines.append("## Ids with no `## ` heading")
|
||
lines.append("")
|
||
lines.append("Opened as records inside another section. `line` is the record's own opening "
|
||
"line and `first mention` is the first line that names the id at all -- they "
|
||
"differ when an id is cited in prose before it is opened (Q53: :7246 -> :7292, "
|
||
"Q63: :9909 -> :9918). `form` is the record form that admits it.")
|
||
lines.append("")
|
||
lines.append("| id | record line | first mention | refs | inside section | shape | form |")
|
||
lines.append("|---|---|---|---|---|---|---|")
|
||
for r in payload["nested"]:
|
||
lines.append("| %s | %d | %d | %d | %s | %s | %s |"
|
||
% (r["id"], r["line"], r["first_mention"], r["refs"],
|
||
r["section"] or "--", r["shape"], r["form"]))
|
||
lines.append("")
|
||
lines.append("**Not in this document.** Ids the prose corpus names that the queue has never "
|
||
"mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the "
|
||
"top-level documents; `tools/` excluded). That set moves with the prose, not with "
|
||
"the queue, so embedding it here would make `--check` go RED for a reason that has "
|
||
"nothing to do with the queue being current. It is reported on demand: "
|
||
"`python3 tools/queue_index.py --outside`.")
|
||
lines.append("")
|
||
lines.append("## The pinned parse")
|
||
lines.append("")
|
||
lines.append("The block below is the parse this document renders, byte for byte. "
|
||
"`--check` compares a fresh parse against it.")
|
||
lines.append("")
|
||
lines.append(BEGIN)
|
||
lines.append("```json")
|
||
lines.append(json.dumps(payload, indent=1, sort_keys=True))
|
||
lines.append("```")
|
||
lines.append(END)
|
||
lines.append("")
|
||
return "\n".join(lines)
|
||
|
||
|
||
def extract_pinned(text):
|
||
if BEGIN not in text or END not in text:
|
||
return None
|
||
body = text.split(BEGIN, 1)[1].split(END, 1)[0]
|
||
m = re.search(r"```json\n(.*?)\n```", body, re.S)
|
||
if not m:
|
||
return None
|
||
try:
|
||
return json.loads(m.group(1))
|
||
except ValueError:
|
||
return None
|
||
|
||
|
||
def delta(pinned, fresh):
|
||
"""A short, specific account of how a pinned parse and a fresh one differ."""
|
||
out = []
|
||
p_it = {(s["id"], s["line"]): s for s in pinned.get("items", pinned.get("sections", []))}
|
||
f_it = {(s["id"], s["line"]): s for s in fresh.get("items", [])}
|
||
added = [k for k in f_it if k not in p_it]
|
||
removed = [k for k in p_it if k not in f_it]
|
||
moved = [k for k in f_it if k in p_it and
|
||
(f_it[k]["end_line"], f_it[k]["bytes"], f_it[k]["status"])
|
||
!= (p_it[k]["end_line"], p_it[k]["bytes"], p_it[k]["status"])]
|
||
# Q103: an item is a heading section or a record block, and the account names
|
||
# which -- a block that moved is not reported as a section.
|
||
def kind_of(table, key):
|
||
return table[key].get("kind", "heading")
|
||
add_h = [k for k in added if kind_of(f_it, k) == "heading"]
|
||
add_b = [k for k in added if kind_of(f_it, k) != "heading"]
|
||
if add_h:
|
||
out.append("%d new section(s): %s" % (len(add_h), ", ".join("%s@%d" % k for k in add_h)))
|
||
if add_b:
|
||
out.append("%d new record block(s): %s"
|
||
% (len(add_b), ", ".join("%s@%d" % k for k in add_b)))
|
||
gone_h = [k for k in removed if kind_of(p_it, k) == "heading"]
|
||
gone_b = [k for k in removed if kind_of(p_it, k) != "heading"]
|
||
if gone_h:
|
||
out.append("%d gone section(s): %s" % (len(gone_h), ", ".join("%s@%d" % k for k in gone_h)))
|
||
if gone_b:
|
||
out.append("%d gone record block(s): %s"
|
||
% (len(gone_b), ", ".join("%s@%d" % k for k in gone_b)))
|
||
if moved:
|
||
out.append("%d item(s) changed extent or status, e.g. %s@%d %d->%d lines"
|
||
% (len(moved), moved[0][0], moved[0][1],
|
||
p_it[moved[0]]["end_line"] - p_it[moved[0]]["line"] + 1,
|
||
f_it[moved[0]]["end_line"] - f_it[moved[0]]["line"] + 1))
|
||
p_nest = [r["id"] for r in pinned.get("nested", [])]
|
||
f_nest = [r["id"] for r in fresh.get("nested", [])]
|
||
if p_nest != f_nest:
|
||
out.append("nested id set changed: +%s -%s"
|
||
% (",".join(sorted(set(f_nest) - set(p_nest), key=lambda s: int(s[1:]))),
|
||
",".join(sorted(set(p_nest) - set(f_nest), key=lambda s: int(s[1:])))))
|
||
if pinned.get("queue_sha256") != fresh.get("queue_sha256"):
|
||
out.append("queue sha256 %s -> %s" % (str(pinned.get("queue_sha256"))[:12],
|
||
str(fresh.get("queue_sha256"))[:12]))
|
||
return "; ".join(out) if out else "no structural difference (byte-level drift only)"
|
||
|
||
|
||
def run(args):
|
||
repo = args.repo
|
||
queue_path = os.path.join(repo, args.queue)
|
||
# A read failure is an exit code, not a traceback. `--queue` pointed at a
|
||
# missing path or at bytes that are not UTF-8 raised FileNotFoundError /
|
||
# UnicodeDecodeError straight out of run(), so the operator got a stack and
|
||
# rc 1 while every other reader in this repo (`queue_claim_check.py:558-563`)
|
||
# says `cannot read queue:` and exits 2. The same contract is owed to the
|
||
# --check document: a gate that cannot read its input has verified nothing,
|
||
# and must not report that as a stale index (rc 1) or a crash.
|
||
try:
|
||
with open(queue_path, encoding="utf-8") as fh:
|
||
text = fh.read()
|
||
except (OSError, UnicodeDecodeError) as exc:
|
||
print("cannot read queue: %s" % exc, file=sys.stderr)
|
||
return 2
|
||
payload = parse(text)
|
||
|
||
if args.check:
|
||
try:
|
||
doc = open(args.check, encoding="utf-8").read()
|
||
except (OSError, UnicodeDecodeError) as exc:
|
||
print("cannot read index document: %s" % exc, file=sys.stderr)
|
||
return 2
|
||
pinned = extract_pinned(doc)
|
||
if pinned is None:
|
||
print("UNVERIFIABLE: no parseable pinned block in %s" % args.check)
|
||
return 2
|
||
render = build_doc(payload)
|
||
if render != doc:
|
||
# The rendered prose is graded, not only the pin. A pin-only gate
|
||
# stayed green while the table above it had drifted; that was the
|
||
# first whole-gate run's finding, so the bytes are compared whole.
|
||
rl = render.split("\n")
|
||
dl = doc.split("\n")
|
||
first = next((i for i in range(min(len(rl), len(dl))) if rl[i] != dl[i]),
|
||
min(len(rl), len(dl)))
|
||
print("STALE: rendered document differs at line %d" % (first + 1))
|
||
if first < len(dl):
|
||
print(" document: %s" % dl[first][:160])
|
||
if first < len(rl):
|
||
print(" render: %s" % rl[first][:160])
|
||
if pinned is not None:
|
||
print(" pin: %s" % delta(pinned, payload))
|
||
print("regenerate with: python3 tools/queue_index.py --repo %s --write %s"
|
||
% (repo, args.check))
|
||
return 1
|
||
print("index current: %s" % summary(payload))
|
||
return 0
|
||
|
||
# `outside` is None outside a git work tree (no corpus to speak of); --index
|
||
# and --json still work there, which is what makes the tool usable on an
|
||
# extracted copy of the queue -- a bare `unpack NoneType` was a crash until
|
||
# this line, found by verifying the tool against `git show <rev>:...` blobs.
|
||
res = outside_ids(repo, args.queue, exclude=(args.write, args.check))
|
||
outside, corpus = res if res else ({}, 0)
|
||
if args.outside:
|
||
print("outside the queue: %d id(s) named by the %d tracked *.md files read "
|
||
"(`tools/` excluded), which `WORK-QUEUE.md` never mentions" % (len(outside or {}), corpus or 0))
|
||
for qid, files in (outside or {}).items():
|
||
print(" %-5s %s" % (qid, ", ".join(files[:4])
|
||
+ (" +%d" % (len(files) - 4) if len(files) > 4 else "")))
|
||
return 0
|
||
|
||
if args.write:
|
||
try:
|
||
with open(args.write, "w", encoding="utf-8") as fh:
|
||
fh.write(build_doc(payload))
|
||
except OSError as exc:
|
||
print("cannot write --write " + args.write + ": " + str(exc), file=sys.stderr)
|
||
return 2
|
||
print("wrote %s (%d bytes)" % (args.write, os.path.getsize(args.write)))
|
||
return 0
|
||
|
||
if args.json:
|
||
try:
|
||
with open(args.json, "w", encoding="utf-8") as fh:
|
||
json.dump(payload, fh, indent=2, sort_keys=True)
|
||
fh.write("\n")
|
||
except OSError as exc:
|
||
print("cannot write --json " + args.json + ": " + str(exc), file=sys.stderr)
|
||
return 2
|
||
|
||
if args.index:
|
||
print("%-4s %-5s %-7s %-11s %-7s %-10s %s"
|
||
% ("#", "id", "kind", "lines", "bytes", "role", "title"))
|
||
for i, s in enumerate(payload["items"], 1):
|
||
print("%-4d %-5s %-7s %-11s %-7d %-10s %s"
|
||
% (i, s["id"], s["kind"], "%d-%d" % (s["line"], s["end_line"]), s["bytes"],
|
||
s["role"][:10], clip(s["title"], 72)))
|
||
print()
|
||
for r in payload["nested"]:
|
||
print("nested %-4s line %-5d refs %-3d inside %-4s %s"
|
||
% (r["id"], r["line"], r["refs"], r["section"] or "--", r["shape"]))
|
||
if outside:
|
||
print("outside the queue: %s" % ", ".join(outside))
|
||
print()
|
||
|
||
print(summary(payload))
|
||
return 0
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# selftest: a fixture corpus, one cell per stated rule
|
||
# --------------------------------------------------------------------------
|
||
|
||
FIXTURE = """# Work queue
|
||
|
||
Preface text.
|
||
|
||
## Q1 — First item *(lane)*
|
||
|
||
**Status: DONE at `abc1234`.** Body.
|
||
|
||
```
|
||
## Q9 — not a section, this is fenced
|
||
```
|
||
|
||
## Q2 — Second item
|
||
|
||
Body with no status marker.
|
||
|
||
## Q1 — closure — verified
|
||
|
||
**Status: OPEN.** A continuation section for Q1.
|
||
|
||
## Q3 §9 follow-on — odd shape
|
||
|
||
Prose that names Q41 before its record.
|
||
- **Q41 — OPEN (pipe, 2026-10-13, unclaimed).** A nested record.
|
||
- prose mentioning Q42 in passing.
|
||
- **Q43 opened — a bold lead record for the fixture.**
|
||
- a bullet carrying **Q44** mid-line, which is not a record.
|
||
- a tick quoting `--outside` = Q77/Q99 as prose.
|
||
**Status: OPEN.** The status lead opens Q45 on this line.
|
||
"""
|
||
|
||
|
||
def selftest():
|
||
cells = [0]
|
||
bad = [0]
|
||
|
||
def check(name, got, want):
|
||
cells[0] += 1
|
||
if got != want:
|
||
bad[0] += 1
|
||
print("FAIL %s\n got %r\n want %r" % (name, got, want))
|
||
|
||
p = parse(FIXTURE)
|
||
secs = p["sections"]
|
||
check("four sections parsed", len(secs), 4)
|
||
check("ids in order", [s["id"] for s in secs], ["Q1", "Q2", "Q1", "Q3"])
|
||
check("fenced heading is not a section", "Q9" in p["ids_with_section"], False)
|
||
check("second Q1 is a continuation", secs[2]["role"], "continuation")
|
||
check("continuation detail is the role word", secs[2]["role_detail"], "closure")
|
||
check("odd heading shape still parses", secs[3]["id"], "Q3")
|
||
check("multi-section map names Q1", p["ids_multi_section"], {"Q1": [5, 17]})
|
||
check("status marker extracted verbatim", secs[0]["status"], "DONE at `abc1234`.")
|
||
check("status line number", secs[0]["status_line"], 7)
|
||
check("section with no marker is None", secs[1]["status"], None)
|
||
check("continuation status is its own", secs[2]["status"], "OPEN.")
|
||
# `**Status:** value` -- the bold run closes *before* the value. Cutting at
|
||
# the first `**` returned "" and the index published a stated status as
|
||
# `--` (Q13:1129 was the live case in the real queue).
|
||
lead = parse("## Q1 — a\n\n**Status:** 1 of 8 decided.\n\n"
|
||
"## Q2 — b\n\n**Status:** **OPEN.** tail.\n\n"
|
||
"## Q3 — c\n\n**Status:**\n")
|
||
check("status whose bold closes before its value",
|
||
lead["sections"][0]["status"], "1 of 8 decided.")
|
||
check("that marker still reports its own line",
|
||
lead["sections"][0]["status_line"], 3)
|
||
check("status lead with a bold value still extracts",
|
||
lead["sections"][1]["status"], "OPEN.")
|
||
check("a marker with no value at all stays None",
|
||
lead["sections"][2]["status"], None)
|
||
check("nested ids found", [r["id"] for r in p["nested"]], ["Q41", "Q43", "Q45"])
|
||
check("nested owner is the containing section", p["nested"][0]["section"], "Q3")
|
||
check("nested shape is bullet carrying a bold lead", p["nested"][0]["shape"], "BULLET+BOLD")
|
||
check("bold lead is a record form", p["nested"][0]["form"], "BOLD-LEAD")
|
||
check("status lead is a record form", p["nested"][2]["form"], "STATUS-LEAD")
|
||
check("record line follows the first mention of the same id",
|
||
(p["nested"][0]["line"], p["nested"][0]["first_mention"]), (24, 23))
|
||
check("prose mention is not a record (Q42)", "Q42" in [r["id"] for r in p["nested"]], False)
|
||
check("mid-line bold mention is not a record (Q44)",
|
||
"Q44" in [r["id"] for r in p["nested"]], False)
|
||
check("quoted tool output opens nothing (Q77/Q99)",
|
||
{"Q77", "Q99"} & {r["id"] for r in p["nested"]}, set())
|
||
check("mentions are still counted as refs", p["nested"][0]["refs"], 2)
|
||
check("items tile the file with no gap",
|
||
all(p["items"][i]["end_line"] + 1 == p["items"][i + 1]["line"]
|
||
for i in range(len(p["items"]) - 1)), True)
|
||
check("a record block carves its own bounds out of its neighbour",
|
||
[(b["id"], b["line"], b["end_line"]) for b in p["blocks"]],
|
||
[("Q41", 24, 25), ("Q43", 26, 28), ("Q45", 29, 29)])
|
||
check("the section that used to swallow them now stops before the block",
|
||
(secs[3]["id"], secs[3]["line"], secs[3]["end_line"]), ("Q3", 21, 23))
|
||
check("last item ends at the last line", p["items"][-1]["end_line"], p["queue_lines"])
|
||
check("preamble ends before the first heading", p["preamble"]["end_line"], 4)
|
||
check("byte accounting is exact",
|
||
p["preamble"]["bytes"] + sum(s["bytes"] for s in p["items"]), p["queue_bytes"])
|
||
# a block between two headings keeps both headings as sections and takes the
|
||
# lines between them: the case that made `sections tile` a false statement
|
||
gap = parse("## Q99 — a\n\nbody\n\n- **Q102 — OPEN (pipe).**\n\nprose\n\n## Q100 — b\n")
|
||
check("a block between two headings does not dissolve them",
|
||
[s["id"] for s in gap["sections"]], ["Q99", "Q100"])
|
||
check("the section before the block stops at it",
|
||
[(s["id"], s["line"], s["end_line"]) for s in gap["sections"][:1]],
|
||
[("Q99", 1, 4)])
|
||
check("the block owns the lines up to the next heading",
|
||
[(b["id"], b["line"], b["end_line"]) for b in gap["blocks"]],
|
||
[("Q102", 5, 8)])
|
||
check("items still tile across a block boundary",
|
||
[(i["id"], i["line"], i["end_line"]) for i in gap["items"]],
|
||
[("Q99", 1, 4), ("Q102", 5, 8), ("Q100", 9, 9)])
|
||
|
||
# --- a second record block for the same id is an item too (Q103) --------
|
||
# A closure block used to be charged to whichever item preceded it: measured
|
||
# on the real queue, appending `**Q103 -- DONE ...**` grew Q104's row from
|
||
# 5,990 to 14,007 bytes. These cells pin the rule and its two refusals.
|
||
sec = parse("## Q99 — a\n\nbody\n\n"
|
||
"**Q102 — OPEN (pipe).**\n\nthe opening block\n\n---\n\n"
|
||
"**Q102 — DONE at `abc1234`.**\n\nthe closure block\n")
|
||
check("a second block-initial record is its own item",
|
||
[(i["id"], i["line"], i["role"], i["role_detail"]) for i in sec["items"]],
|
||
[("Q99", 1, "opening", ""), ("Q102", 5, "record", "BOLD-LEAD"),
|
||
("Q102", 11, "continuation", "DONE")])
|
||
check("the opening block stops before the closure block",
|
||
[(b["id"], b["line"], b["end_line"]) for b in sec["blocks"]],
|
||
[("Q102", 5, 10), ("Q102", 11, 13)])
|
||
# slice, not [-1]: under a mutation that cuts no block this must read as a
|
||
# failed cell, not as a traceback out of the selftest (measured: it did).
|
||
check("the closure block owns the tail",
|
||
[(b["line"], b["end_line"]) for b in sec["blocks"]][-1:], [(11, 13)])
|
||
check("the nested table keeps one row for the id, not one per block",
|
||
[(r["id"], r["line"]) for r in sec["nested"]], [("Q102", 5)])
|
||
# Refusal 1: a bullet-led record inside a section is prose in a list, not a
|
||
# boundary. `- **Q102 -- DONE**` must not split the section that holds it --
|
||
# and here Q102 owns a heading, so the bullet is not its first record either.
|
||
bullet = parse("## Q99 — a\n\nbody\n\n## Q102 — d\n\nprose\n\n"
|
||
"- **Q102 — DONE at `abc1234`.**\n\nmore body\n")
|
||
check("a bullet-led record does not cut the section",
|
||
[(i["id"], i["line"], i["end_line"]) for i in bullet["items"]],
|
||
[("Q99", 1, 4), ("Q102", 5, 11)])
|
||
check("...and it adds no block row",
|
||
[(r["id"], r["form"]) for r in bullet["nested"]], [])
|
||
# Refusal 2: column 0 is not enough either -- the line must begin a block,
|
||
# i.e. the line above is blank, a `---` rule or an HTML comment.
|
||
glued = parse("## Q99 — a\n\nbody\n\n## Q102 — d\n\nprose runs on\n"
|
||
"**Q102 — DONE.**\n\ntail\n")
|
||
check("a column-0 record glued to prose does not cut the section",
|
||
[(i["id"], i["line"], i["end_line"]) for i in glued["items"]],
|
||
[("Q99", 1, 4), ("Q102", 5, 10)])
|
||
|
||
check("bytes are the file's own length", p["queue_bytes"], len(FIXTURE.encode()))
|
||
check("sha256 is the file's own", p["queue_sha256"],
|
||
hashlib.sha256(FIXTURE.encode()).hexdigest())
|
||
|
||
# --- the id class: three digits are an id (Q103, 2026-10-27) -------------
|
||
# The fixture above stops at Q99, exactly like the real queue did: `\d{1,2}`
|
||
# made every item from Q100 up body text of the section before it. These
|
||
# cells fail on the pre-fix pattern, which is why they are here.
|
||
three = parse("## Q99 — a\n\nbody\n\n## Q100 — b\n\nQ101 is named here.\n\n"
|
||
"## Q101 — c\n\n- **Q102 — OPEN (pipe).**\n")
|
||
check("three-digit heading is a section",
|
||
[s["id"] for s in three["sections"]], ["Q99", "Q100", "Q101"])
|
||
check("three-digit heading is an id",
|
||
three["ids_with_section"], ["Q99", "Q100", "Q101"])
|
||
pairs = [(a["end_line"], b["line"]) for a, b in
|
||
zip(three["sections"], three["sections"][1:])]
|
||
check("three-digit section bounds do not overlap the one before",
|
||
pairs[:1], [(4, 5)])
|
||
check("sections still tile across the id boundary",
|
||
all(three["sections"][i]["end_line"] + 1 == three["sections"][i + 1]["line"]
|
||
for i in range(len(three["sections"]) - 1)), True)
|
||
check("a three-digit bold record is a nested id (form kept)",
|
||
[(r["id"], r["form"]) for r in three["nested"]], [("Q102", "BOLD-LEAD")])
|
||
# No width ceiling: a fourth digit is a new id, never a truncation of a
|
||
# three-digit one. `\d{1,3}` would fail both cells below -- the ceiling is
|
||
# the bug one digit later, so it is not restored.
|
||
four = parse("## Q1000 — d\n")
|
||
check("four-digit heading is its own id", four["ids_with_section"], ["Q1000"])
|
||
check("a longer id is one token, not a prefix match",
|
||
ID_TOKEN.findall("see Q1000 and Q100, plus Q1000x"), ["Q1000", "Q100"])
|
||
check("an id glued to a word is not an id",
|
||
ID_TOKEN.findall("FAQ100 / xQ100 / Q100y / Q"), [])
|
||
check("the old id class could not see Q100; this one does (RED -> GREEN)",
|
||
(re.compile(r"\bQ\d{1,2}\b").findall("## Q100 — b"),
|
||
HEADING.match("## Q100 — b") is not None), ([], True))
|
||
|
||
# --- the gate, round trip and both directions of falsification ----------
|
||
root = tempfile.mkdtemp(prefix="queue-index-selftest-")
|
||
repo = os.path.join(root, "repo")
|
||
os.makedirs(os.path.join(repo, "docs"))
|
||
queue = os.path.join(repo, "WORK-QUEUE.md")
|
||
doc_path = os.path.join(repo, DOC_PATH)
|
||
subprocess.check_call(["git", "-C", repo, "init", "-q", "-b", "main"])
|
||
subprocess.check_call(["git", "-C", repo, "config", "user.email", "s@invalid"])
|
||
subprocess.check_call(["git", "-C", repo, "config", "user.name", "s"])
|
||
with open(queue, "w", encoding="utf-8") as fh:
|
||
fh.write(FIXTURE)
|
||
subprocess.check_call(["git", "-C", repo, "add", "-A"])
|
||
subprocess.check_call(["git", "-C", repo, "commit", "-q", "-m", "fixture"])
|
||
|
||
import contextlib
|
||
import io
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=doc_path, check=None, outside=False))
|
||
check("--write exits 0", rc, 0)
|
||
check("document written", os.path.exists(doc_path), True)
|
||
doc = open(doc_path, encoding="utf-8").read()
|
||
check("document carries the pinned block", extract_pinned(doc) is not None, True)
|
||
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("--check on its own output exits 0", rc, 0)
|
||
check("--check says current", "index current:" in buf.getvalue(), True)
|
||
|
||
# --- the document is a function of the queue alone -----------------------
|
||
# A prose file that names an id the queue never mentions must not move a
|
||
# byte of the document, and must not turn --check RED. Baking the corpus
|
||
# table into the document did both: same queue, three different documents,
|
||
# and a checkin quoting an id reddened a gate about the queue's currency.
|
||
with open(os.path.join(repo, "notes.md"), "w", encoding="utf-8") as fh:
|
||
fh.write("# notes\n\nQ63 is named only here; Q1 is a queue id.\n")
|
||
subprocess.check_call(["git", "-C", repo, "add", "notes.md"])
|
||
before = open(doc_path, encoding="utf-8").read()
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=doc_path, check=None, outside=False))
|
||
check("re-render beside a prose mention exits 0", rc, 0)
|
||
check("a prose mention does not change the document",
|
||
open(doc_path, encoding="utf-8").read(), before)
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("--check stays green when prose names an id", rc, 0)
|
||
|
||
# --- the outside leg is reported, not embedded ---------------------------
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=None, outside=True))
|
||
out = buf.getvalue()
|
||
check("--outside exits 0", rc, 0)
|
||
check("--outside counts the corpus it actually read",
|
||
"1 id(s) named by the 2 tracked *.md files" in out, True)
|
||
check("--outside names Q63 and the file that carries it",
|
||
" Q63 notes.md" in out, True)
|
||
|
||
# --- a tree that is not a git repo must still index -----------------------
|
||
# The `outside` leg needs git; --index does not. A verifier that extracts
|
||
# `git show <rev>:WORK-QUEUE.md` into a temp dir has no .git, and this used to
|
||
# crash on `outside, corpus = outside_ids(...)` returning None.
|
||
plain = os.path.join(root, "plain")
|
||
os.makedirs(plain)
|
||
with open(os.path.join(plain, "WORK-QUEUE.md"), "w", encoding="utf-8") as fh:
|
||
fh.write(FIXTURE)
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=plain, queue="WORK-QUEUE.md", index=True,
|
||
json=None, write=None, check=None, outside=False))
|
||
check("--index outside a git work tree exits 0", rc, 0)
|
||
check("--index outside a git work tree still summarizes",
|
||
"index: 7 items (4 sections, 3 record blocks)" in buf.getvalue(), True)
|
||
|
||
# falsify 1: an append moves the last section's extent
|
||
with open(queue, "a", encoding="utf-8") as fh:
|
||
fh.write("- an appended line.\n")
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("--check goes RED on an append", rc, 1)
|
||
check("RED names the moved section", "changed extent or status" in buf.getvalue(), True)
|
||
|
||
# falsify 2: a new heading is a new section
|
||
with open(queue, "a", encoding="utf-8") as fh:
|
||
fh.write("\n## Q7 — a brand new item\n\n**Status: OPEN.**\n")
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("--check goes RED on a new heading", rc, 1)
|
||
check("RED names the new section", "new section(s): Q7" in buf.getvalue(), True)
|
||
|
||
# falsify 3: a document with no block cannot be verified
|
||
open(doc_path, "w", encoding="utf-8").write("# empty\n")
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("no pin -> exit 2", rc, 2)
|
||
check("no pin -> says so", "UNVERIFIABLE" in buf.getvalue(), True)
|
||
|
||
# --- a read error is an exit code, not a traceback -----------------------
|
||
# `--queue` at a missing path or at bytes that are not UTF-8 used to raise
|
||
# straight out of run() (FileNotFoundError / UnicodeDecodeError, rc 1). The
|
||
# repo's other reader exits 2 with `cannot read queue:`; this tool now does
|
||
# the same, and owes the --check document the same answer: an unreadable
|
||
# gate input is rc 2, never the rc 1 that means "stale".
|
||
bad_q = os.path.join(root, "bad-utf8.md")
|
||
with open(bad_q, "wb") as fh:
|
||
fh.write(b"\xff\xfe\x80\n")
|
||
err = io.StringIO()
|
||
with contextlib.redirect_stderr(err):
|
||
rc = run(argparse.Namespace(repo=root, queue="missing-queue.md", index=False,
|
||
json=None, write=None, check=None, outside=False))
|
||
check("missing queue -> exit 2", rc, 2)
|
||
check("missing queue -> says cannot read queue",
|
||
err.getvalue().startswith("cannot read queue:"), True)
|
||
err = io.StringIO()
|
||
with contextlib.redirect_stderr(err):
|
||
rc = run(argparse.Namespace(repo=root, queue="bad-utf8.md", index=False,
|
||
json=None, write=None, check=None, outside=False))
|
||
check("non-UTF-8 queue -> exit 2", rc, 2)
|
||
check("non-UTF-8 queue -> says cannot read queue",
|
||
"cannot read queue:" in err.getvalue(), True)
|
||
err = io.StringIO()
|
||
with contextlib.redirect_stderr(err):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None,
|
||
check=os.path.join(repo, "missing-index.md"),
|
||
outside=False))
|
||
check("missing index document -> exit 2", rc, 2)
|
||
check("missing index document -> names the document",
|
||
"cannot read index document:" in err.getvalue(), True)
|
||
err = io.StringIO()
|
||
with contextlib.redirect_stderr(err):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=bad_q, outside=False))
|
||
check("non-UTF-8 index document -> exit 2", rc, 2)
|
||
check("non-UTF-8 index document -> names the document",
|
||
"cannot read index document:" in err.getvalue(), True)
|
||
|
||
print("selftest: %d cells, %d passed, %d failed" % (cells[0], cells[0] - bad[0], bad[0]))
|
||
return 1 if bad[0] else 0
|
||
|
||
|
||
def main(argv=None):
|
||
ap = argparse.ArgumentParser(
|
||
description="queue_index.py -- the work queue's table of contents, and the gate "
|
||
"that it is current.")
|
||
ap.add_argument("--repo", default=".")
|
||
ap.add_argument("--queue", default=QUEUE_DEFAULT)
|
||
ap.add_argument("--index", action="store_true", help="print the section table")
|
||
ap.add_argument("--json", metavar="PATH", help="write the parse here")
|
||
ap.add_argument("--write", metavar="PATH", help="render the index document here")
|
||
ap.add_argument("--check", metavar="PATH", help="re-derive and compare a rendered index")
|
||
ap.add_argument("--outside", action="store_true",
|
||
help="report ids the prose corpus names that the queue never mentions")
|
||
ap.add_argument("--selftest", action="store_true", help="fixture corpus, 80 cells")
|
||
args = ap.parse_args(argv)
|
||
if args.selftest:
|
||
return selftest()
|
||
return run(args)
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|