777 lines
36 KiB
Python
777 lines
36 KiB
Python
#!/usr/bin/env python3
|
||
"""queue_index.py -- the work queue's table of contents, and the gate that it is current.
|
||
|
||
WHY THIS EXISTS
|
||
`WORK-QUEUE.md` is 1.2 MB and 9,901 lines long. It has no table of contents.
|
||
Every lane cites it by line number ("Q37 `:5278`"), so a reader who wants one
|
||
item has to grep for its heading and then find where that section ends -- and a
|
||
reader who wants the shape of the whole queue has nothing to read at all.
|
||
|
||
The shape is not obvious even after a grep. Measured on 2026-10-17 at HEAD
|
||
(`ece5228`): 49 `## ` sections carry 45 distinct ids, because four ids (Q37,
|
||
Q38, Q39, Q43) own a second section each -- a `closure` or `claimed` block
|
||
appended under a fresh heading. And 9 further ids (Q41, Q42, Q46, Q47, Q50,
|
||
Q51, Q52, Q53, Q63) have NO heading of their own: they are opened as bold
|
||
records nested inside somebody else's section. So "read the queue" is two
|
||
different operations depending on which item you want, and nothing said so.
|
||
|
||
THE RECORD RULE (Q63, 2026-10-17). An id is *opened by a record, not mentioned
|
||
by prose*: a heading, a bold lead that opens with the id (`**Qnn -- ...**` /
|
||
`**Qnn opened -- ...**`), or a line whose own lead is a `Status:` marker. Before
|
||
that rule the detector admitted every id token that had no heading, so a tick
|
||
quoting `--outside = Q63/Q77/Q99` opened three ids, an artifact path
|
||
(`out/Q54-READBACK.txt`) opened Q54, and the range `Q31-Q62` opened Q62. The
|
||
first *mention* also used to be reported as the record, which is wrong twice:
|
||
Q53 is first named 46 lines before it is opened (:7246 prose, :7292 record) and
|
||
Q63 is first named inside the very tick that quotes it (:9909) but opened at
|
||
:9918. The table below now carries the record line, the first-mention line and
|
||
the form that admitted the id, so the difference is visible instead of implied.
|
||
|
||
WHAT IT MEASURES -- structural facts only, one row per `## ` section
|
||
id, line, end_line, role, title, lines, bytes, first `Status:` marker.
|
||
role=opening the first section carrying that id
|
||
role=continuation a later section for an id already seen (a closure or
|
||
claim block); its `role_detail` names the first word
|
||
lines/bytes the section's own extent, header line to last line
|
||
status the first `Status:` line in the section, VERBATIM and
|
||
truncated. NOT adjudicated: whether an item is open is
|
||
`tools/queue_claim_check.py`'s verdict, not this tool's.
|
||
Two instruments disagreeing about closure is a bug this
|
||
tool declines to introduce.
|
||
|
||
Plus the two legs that make the structure visible:
|
||
nested ids opened by a record inside a section that has no heading of their
|
||
own: the record's line, the first line that mentions it, the section
|
||
that contains it, and the shape/form of the record line
|
||
outside ids referenced by the prose corpus (checkins/ replies/ recoveries/
|
||
audits/ access/ decisions/ docs/ and the top-level documents) but
|
||
absent from the queue entirely -- `tools/` is excluded on purpose:
|
||
its fixtures carry synthetic ids by construction, and this tool's
|
||
own selftest is one of them.
|
||
|
||
The `outside` leg is REPORTED, not embedded: `--outside` prints it, and the
|
||
rendered document does not contain it. It cannot. That set is a function of
|
||
the working tree's prose, not of the queue, so baking it into a document that
|
||
`--check` byte-compares against a fresh render means any checkin that quotes
|
||
an id turns the queue's gate RED for a reason that has nothing to do with the
|
||
queue. Measured on 2026-10-17: `--outside` read Q63 from `recoveries/`, then
|
||
Q63+Q77+Q99 once a checkin described the test that names Q77 -- same queue,
|
||
same sha256, three different documents. A table of contents must be a
|
||
function of the file it indexes.
|
||
|
||
Headings inside a fenced block are never sections; a heading whose id is
|
||
followed by `closure --` or `§9 follow-on --` is still that id's section.
|
||
|
||
WHAT IT DOES NOT DO
|
||
It does not grade the work, does not adjudicate closure, and does not read
|
||
git history. It answers "what is in this file and where", and it keeps the
|
||
generated copy of that answer honest: `--write` renders `docs/queue-index.md`
|
||
with the parse pinned inside a machine block, and `--check` re-renders the
|
||
whole document -- prose table included, compared byte for byte -- and goes RED
|
||
the moment the two disagree. (The pin alone was not enough: the first
|
||
whole-gate run found the table above the pin drifted while the pin still
|
||
matched, because the document had begun listing itself in its own corpus.
|
||
The rendered file is excluded from that corpus, which makes `--write` a fixed
|
||
point, and the bytes are now compared whole.) The document is therefore a
|
||
function of `WORK-QUEUE.md` alone: the same queue renders the same bytes
|
||
whatever the prose corpus says, which is what makes `--check` a statement
|
||
about the queue's currency and not about the last person to write a checkin.
|
||
|
||
`--check` exit codes: 0 current, 1 stale (with the delta printed), 2 the
|
||
document carries no parseable pinned block.
|
||
|
||
READ-ONLY except `--json PATH` and `--write PATH`.
|
||
|
||
USAGE
|
||
python3 tools/queue_index.py --index # 49 sections, one screen each side
|
||
python3 tools/queue_index.py --outside # ids the prose corpus names and the queue does not
|
||
python3 tools/queue_index.py --json out.json
|
||
python3 tools/queue_index.py --write docs/queue-index.md
|
||
python3 tools/queue_index.py --check docs/queue-index.md
|
||
python3 tools/queue_index.py --selftest # fixture corpus, 46 cells
|
||
|
||
Re-run `--write` after any append to the queue. The document carries the
|
||
queue's sha256, so a stale copy is a RED check, never a silent lie.
|
||
"""
|
||
|
||
import argparse
|
||
import hashlib
|
||
import json
|
||
import os
|
||
import re
|
||
import subprocess
|
||
import sys
|
||
import tempfile
|
||
|
||
QUEUE_DEFAULT = "WORK-QUEUE.md"
|
||
DOC_PATH = "docs/queue-index.md"
|
||
BEGIN = "<!-- queue-index:begin -->"
|
||
END = "<!-- queue-index:end -->"
|
||
|
||
HEADING = re.compile(r"^##\s+(Q\d{1,2})\b(.*)$")
|
||
ID_TOKEN = re.compile(r"\bQ\d{1,2}\b")
|
||
STATUS = re.compile(r"^\s*\*{0,2}Status:\s*(.*)$")
|
||
FENCE = re.compile(r"^(```|~~~)")
|
||
|
||
# A record opens with its id; a mention merely carries it. `BULLET` strips the
|
||
# markdown that can precede a record without being part of it, and `OPEN_SEP` is
|
||
# what may follow the id in a bold lead that is *opening* an item (Q46's
|
||
# "**Q46 (Helm lane) — OPEN ...**" and Q63's "**Q63 opened — ...**" both count;
|
||
# a "**Q7, measured:**" bold run does not).
|
||
BULLET = re.compile(r"^[\s>|]*(?:[-*+]\s+)?")
|
||
OPEN_SEP = re.compile(r"\s*(?:—|–|-|:|\(|\bopened\b|\bopen\b|\bOPEN\b|\bOPENED\b)")
|
||
|
||
ROLE_WORDS = ("closure", "claimed", "§", "status")
|
||
|
||
|
||
def record_form(line, qid):
|
||
"""The form in which `line` opens `qid`, or None if it only mentions it.
|
||
|
||
The counting rule is settled in `WORK-QUEUE.md` at Q63 (2026-10-17): an id is
|
||
*opened by a record, not mentioned by prose*. Three forms count --
|
||
|
||
HEADING `## Qnn ...` (a section; never nested)
|
||
BOLD-LEAD after bullet/bold markers the line opens `**Qnn` and the id is
|
||
followed by a separator or the word open/opened
|
||
STATUS-LEAD the line's own lead is a `Status:` marker (bold or plain) and
|
||
it names the id
|
||
|
||
A bare token inside a sentence, a tick quoting tool output (`= Q63/Q77/Q99`),
|
||
an artifact filename (`out/Q54-READBACK.txt`) or a range (`Q31–Q62`) matches
|
||
none of them, which is the point: those were opening ids before this rule.
|
||
"""
|
||
s = line.strip()
|
||
m = HEADING.match(s)
|
||
if m and m.group(1) == qid:
|
||
return "HEADING"
|
||
body = BULLET.sub("", s)
|
||
if qid not in ID_TOKEN.findall(body):
|
||
return None
|
||
if body.startswith("**Status:") or body.startswith("Status:"):
|
||
return "STATUS-LEAD"
|
||
if body.startswith("**"):
|
||
inner = body[2:].lstrip("*").lstrip()
|
||
m = ID_TOKEN.match(inner)
|
||
if m and m.group(0) == qid and OPEN_SEP.match(inner[m.end():]):
|
||
return "BOLD-LEAD"
|
||
return None
|
||
|
||
|
||
def split_lines(text):
|
||
"""Lines without their terminators, plus the byte length each line occupies."""
|
||
lines = text.split("\n")
|
||
trailing_newline = text.endswith("\n")
|
||
if trailing_newline:
|
||
lines = lines[:-1]
|
||
return lines, trailing_newline
|
||
|
||
|
||
def line_bytes(line, is_last, trailing_newline):
|
||
if is_last and not trailing_newline:
|
||
return len(line.encode("utf-8"))
|
||
return len(line.encode("utf-8")) + 1
|
||
|
||
|
||
def parse(text):
|
||
"""Return the structural parse: sections, nested ids, id map, byte accounting."""
|
||
lines, trailing_newline = split_lines(text)
|
||
n = len(lines)
|
||
sizes = [line_bytes(ln, i == n - 1, trailing_newline) for i, ln in enumerate(lines)]
|
||
|
||
headings = [] # (line_no, id, rest)
|
||
in_fence = False
|
||
for i, ln in enumerate(lines, 1):
|
||
if FENCE.match(ln.strip()):
|
||
in_fence = not in_fence
|
||
continue
|
||
if in_fence:
|
||
continue
|
||
m = HEADING.match(ln)
|
||
if m:
|
||
headings.append((i, m.group(1), m.group(2).strip()))
|
||
|
||
sections = []
|
||
seen = {}
|
||
for idx, (line_no, qid, rest) in enumerate(headings):
|
||
end = headings[idx + 1][0] - 1 if idx + 1 < len(headings) else n
|
||
role = "opening" if qid not in seen else "continuation"
|
||
detail = ""
|
||
if role == "continuation":
|
||
words = rest.lstrip("—- ").split()
|
||
detail = words[0].strip("*(") if words else ""
|
||
seen.setdefault(qid, line_no)
|
||
body = lines[line_no:end]
|
||
status = None
|
||
status_line = None
|
||
for off, ln in enumerate(body, start=line_no + 1):
|
||
m = STATUS.match(ln)
|
||
if m:
|
||
raw = m.group(1).strip()
|
||
if "**" in raw: # a bold status marker closes mid-line
|
||
raw = raw.split("**", 1)[0]
|
||
status = raw.strip()
|
||
status_line = off
|
||
break
|
||
title = rest.lstrip("—- ").strip()
|
||
sections.append({
|
||
"id": qid,
|
||
"line": line_no,
|
||
"end_line": end,
|
||
"role": role,
|
||
"role_detail": detail,
|
||
"title": clip(title, 200),
|
||
"lines": end - line_no + 1,
|
||
"bytes": sum(sizes[line_no - 1:end]),
|
||
"status": clip(status, 200) if status else None,
|
||
"status_line": status_line,
|
||
})
|
||
|
||
ids: dict = {}
|
||
for s in sections:
|
||
ids.setdefault(s["id"], []).append(s["line"])
|
||
|
||
# every id token in the queue, counted as a mention, plus the first line that
|
||
# opens it as a record. Fenced lines are skipped: an id in a code block is
|
||
# quoted text, not a record.
|
||
#
|
||
# Q63 (2026-10-17): the first *mention* is not the record. Q53 is first named
|
||
# in prose 46 lines before its `**Q53 opened -- ...**` line, and Q63 is named
|
||
# inside a tick quoting tool output at :9909 but opened at :9918. So a nested
|
||
# id is admitted by its first record-form line, and an id that never has one
|
||
# (Q54/Q55/Q56/Q58 as artifact filenames, Q62 inside `Q31-Q62`, Q77/Q99 inside
|
||
# a quoted `--outside` list) is not an item at all.
|
||
mentioned = {}
|
||
in_fence = False
|
||
for i, ln in enumerate(lines, 1):
|
||
if FENCE.match(ln.strip()):
|
||
in_fence = not in_fence
|
||
continue
|
||
if in_fence:
|
||
continue
|
||
for m in ID_TOKEN.finditer(ln):
|
||
qid = m.group(0)
|
||
hit = mentioned.get(qid)
|
||
if hit is None:
|
||
hit = mentioned[qid] = {"line": i, "refs": 0, "record": None, "form": None}
|
||
hit["refs"] += 1
|
||
if hit["record"] is None:
|
||
form = record_form(ln, qid)
|
||
if form:
|
||
hit["record"] = i
|
||
hit["form"] = form
|
||
nested = []
|
||
for qid in sorted(mentioned, key=lambda s: int(s[1:])):
|
||
if qid in ids:
|
||
continue
|
||
rec = mentioned[qid]
|
||
if rec["record"] is None:
|
||
continue # a mention, not a record -- not an item
|
||
line_no = rec["record"]
|
||
raw = lines[line_no - 1]
|
||
owner = None
|
||
for s in sections:
|
||
if s["line"] <= line_no <= s["end_line"]:
|
||
owner = s["id"]
|
||
break
|
||
stripped = raw.lstrip()
|
||
if stripped.startswith("#"):
|
||
shape = "HEADING"
|
||
elif stripped.startswith(("-", "*", "|")):
|
||
shape = "BULLET"
|
||
elif stripped.startswith("**"):
|
||
shape = "BOLD"
|
||
else:
|
||
shape = "PLAIN"
|
||
if shape not in ("HEADING", "BOLD") and "**" in raw:
|
||
shape += "+BOLD"
|
||
nested.append({"id": qid, "line": line_no, "section": owner, "shape": shape,
|
||
"refs": rec["refs"], "form": rec["form"],
|
||
"first_mention": rec["line"]})
|
||
|
||
first = headings[0][0] if headings else n + 1
|
||
preamble_end = first - 1
|
||
payload = {
|
||
"queue_lines": n,
|
||
"queue_bytes": sum(sizes),
|
||
"queue_sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(),
|
||
"preamble": {"line": 1, "end_line": preamble_end,
|
||
"bytes": sum(sizes[:preamble_end])},
|
||
"sections": sections,
|
||
"ids_with_section": sorted(ids, key=lambda s: int(s[1:])),
|
||
"ids_multi_section": {k: v for k, v in sorted(
|
||
ids.items(), key=lambda kv: int(kv[0][1:])) if len(v) > 1},
|
||
"nested": nested,
|
||
}
|
||
return payload
|
||
|
||
|
||
def outside_ids(repo, queue_path, exclude=(), exclude_tools=True):
|
||
"""Ids referenced by the prose corpus that the queue never mentions.
|
||
|
||
`exclude` is the rendered document itself: a generated file that names ids
|
||
would otherwise join the corpus that generates it, so `--write` would not be
|
||
a fixed point -- measured once, the first whole-gate run went red on exactly
|
||
that (the document added itself to Q63's file list).
|
||
"""
|
||
try:
|
||
out = subprocess.check_output(["git", "-C", repo, "ls-files"], text=True,
|
||
stderr=subprocess.DEVNULL)
|
||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||
return None
|
||
skip = {os.path.realpath(p) for p in exclude if p}
|
||
files = [f for f in out.split("\n") if f.endswith(".md")]
|
||
if exclude_tools:
|
||
files = [f for f in files if not f.startswith("tools/")]
|
||
# The generated document is never part of the corpus it summarizes, whatever
|
||
# path it is written to -- otherwise a render into any other filename would
|
||
# count the committed copy and the output would cease to be a fixed point.
|
||
files = [f for f in files if f != DOC_PATH]
|
||
queue_abs = os.path.normpath(os.path.join(repo, queue_path))
|
||
queue_text = open(queue_abs, encoding="utf-8", errors="replace").read()
|
||
in_queue = set(ID_TOKEN.findall(queue_text))
|
||
hits = {}
|
||
read = 0
|
||
for f in files:
|
||
path = os.path.join(repo, f)
|
||
if os.path.realpath(path) in skip:
|
||
continue
|
||
try:
|
||
txt = open(path, encoding="utf-8", errors="replace").read()
|
||
except OSError:
|
||
continue
|
||
read += 1
|
||
for m in ID_TOKEN.finditer(txt):
|
||
hits.setdefault(m.group(0), set()).add(f)
|
||
return {k: sorted(v) for k, v in sorted(
|
||
hits.items(), key=lambda kv: int(kv[0][1:])) if k not in in_queue}, read
|
||
|
||
|
||
def summary(payload):
|
||
secs = payload["sections"]
|
||
cont = [s for s in secs if s["role"] == "continuation"]
|
||
return ("index: %d sections, %d ids, %d continuation sections, %d nested ids, "
|
||
"%d lines / %d bytes" % (len(secs), len(payload["ids_with_section"]),
|
||
len(cont), len(payload["nested"]),
|
||
payload["queue_lines"], payload["queue_bytes"]))
|
||
|
||
|
||
def clip(text, limit):
|
||
"""Cut at a word boundary and say so; a silent mid-word cut reads as data loss."""
|
||
if len(text) <= limit:
|
||
return text
|
||
cut = text[:limit]
|
||
if " " in cut[max(0, limit - 20):]:
|
||
cut = cut.rsplit(" ", 1)[0]
|
||
return cut.rstrip(" ,;:—-") + " …"
|
||
|
||
|
||
def build_doc(payload):
|
||
lines = []
|
||
lines.append("# The work queue's table of contents")
|
||
lines.append("")
|
||
lines.append("*Generated by `tools/queue_index.py --write`; do not hand-edit. `python3 "
|
||
"tools/queue_index.py --check docs/queue-index.md` re-derives the parse and "
|
||
"goes RED when this document and `WORK-QUEUE.md` disagree, which is what an "
|
||
"append to the queue produces. Regenerate, commit, move on.*")
|
||
lines.append("")
|
||
lines.append("**What this is.** One row per `## ` section of `WORK-QUEUE.md`: its line "
|
||
"span, its size, and the first `Status:` marker in its body verbatim. "
|
||
"Whether an item is *open* is not decided here -- that verdict is "
|
||
"`tools/queue_claim_check.py`'s, and this document does not overrule it.")
|
||
lines.append("")
|
||
lines.append("**What it exposes.** The queue is two-tier. %d sections carry %d ids because "
|
||
"%d ids own a second, continuation section. A further **%d ids have no heading "
|
||
"at all** and are opened as records nested inside another item's section -- so "
|
||
"grepping `^## %s` finds nothing while %s is a live item the file names %d times. "
|
||
"Those are listed in their own table below."
|
||
% (len(payload["sections"]), len(payload["ids_with_section"]),
|
||
len(payload["ids_multi_section"]), len(payload["nested"]),
|
||
payload["nested"][0]["id"], payload["nested"][0]["id"],
|
||
payload["nested"][0]["refs"]))
|
||
lines.append("")
|
||
lines.append("| # | id | lines | bytes | role | title | first `Status:` |")
|
||
lines.append("|---|---|---|---|---|---|---|")
|
||
for i, s in enumerate(payload["sections"], 1):
|
||
status = (s["status"] or "--").replace("|", "\\|")
|
||
title = s["title"].replace("|", "\\|")
|
||
role = s["role"] if s["role"] == "opening" else "cont: %s" % (s["role_detail"] or "?")
|
||
lines.append("| %d | %s | %d-%d | %d | %s | %s | %s |"
|
||
% (i, s["id"], s["line"], s["end_line"], s["bytes"], role, title, status))
|
||
lines.append("")
|
||
lines.append("## Ids with no `## ` heading")
|
||
lines.append("")
|
||
lines.append("Opened as records inside another section. `line` is the record's own opening "
|
||
"line and `first mention` is the first line that names the id at all -- they "
|
||
"differ when an id is cited in prose before it is opened (Q53: :7246 -> :7292, "
|
||
"Q63: :9909 -> :9918). `form` is the record form that admits it.")
|
||
lines.append("")
|
||
lines.append("| id | record line | first mention | refs | inside section | shape | form |")
|
||
lines.append("|---|---|---|---|---|---|---|")
|
||
for r in payload["nested"]:
|
||
lines.append("| %s | %d | %d | %d | %s | %s | %s |"
|
||
% (r["id"], r["line"], r["first_mention"], r["refs"],
|
||
r["section"] or "--", r["shape"], r["form"]))
|
||
lines.append("")
|
||
lines.append("**Not in this document.** Ids the prose corpus names that the queue has never "
|
||
"mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the "
|
||
"top-level documents; `tools/` excluded). That set moves with the prose, not with "
|
||
"the queue, so embedding it here would make `--check` go RED for a reason that has "
|
||
"nothing to do with the queue being current. It is reported on demand: "
|
||
"`python3 tools/queue_index.py --outside`.")
|
||
lines.append("")
|
||
lines.append("## The pinned parse")
|
||
lines.append("")
|
||
lines.append("The block below is the parse this document renders, byte for byte. "
|
||
"`--check` compares a fresh parse against it.")
|
||
lines.append("")
|
||
lines.append(BEGIN)
|
||
lines.append("```json")
|
||
lines.append(json.dumps(payload, indent=1, sort_keys=True))
|
||
lines.append("```")
|
||
lines.append(END)
|
||
lines.append("")
|
||
return "\n".join(lines)
|
||
|
||
|
||
def extract_pinned(text):
|
||
if BEGIN not in text or END not in text:
|
||
return None
|
||
body = text.split(BEGIN, 1)[1].split(END, 1)[0]
|
||
m = re.search(r"```json\n(.*?)\n```", body, re.S)
|
||
if not m:
|
||
return None
|
||
try:
|
||
return json.loads(m.group(1))
|
||
except ValueError:
|
||
return None
|
||
|
||
|
||
def delta(pinned, fresh):
|
||
"""A short, specific account of how a pinned parse and a fresh one differ."""
|
||
out = []
|
||
p_sec = {(s["id"], s["line"]): s for s in pinned.get("sections", [])}
|
||
f_sec = {(s["id"], s["line"]): s for s in fresh.get("sections", [])}
|
||
added = [k for k in f_sec if k not in p_sec]
|
||
removed = [k for k in p_sec if k not in f_sec]
|
||
moved = [k for k in f_sec if k in p_sec and
|
||
(f_sec[k]["end_line"], f_sec[k]["bytes"], f_sec[k]["status"])
|
||
!= (p_sec[k]["end_line"], p_sec[k]["bytes"], p_sec[k]["status"])]
|
||
if added:
|
||
out.append("%d new section(s): %s" % (len(added), ", ".join("%s@%d" % k for k in added)))
|
||
if removed:
|
||
out.append("%d gone section(s): %s" % (len(removed), ", ".join("%s@%d" % k for k in removed)))
|
||
if moved:
|
||
out.append("%d section(s) changed extent or status, e.g. %s@%d %d->%d lines"
|
||
% (len(moved), moved[0][0], moved[0][1],
|
||
p_sec[moved[0]]["end_line"] - p_sec[moved[0]]["line"] + 1,
|
||
f_sec[moved[0]]["end_line"] - f_sec[moved[0]]["line"] + 1))
|
||
p_nest = [r["id"] for r in pinned.get("nested", [])]
|
||
f_nest = [r["id"] for r in fresh.get("nested", [])]
|
||
if p_nest != f_nest:
|
||
out.append("nested id set changed: +%s -%s"
|
||
% (",".join(sorted(set(f_nest) - set(p_nest), key=lambda s: int(s[1:]))),
|
||
",".join(sorted(set(p_nest) - set(f_nest), key=lambda s: int(s[1:])))))
|
||
if pinned.get("queue_sha256") != fresh.get("queue_sha256"):
|
||
out.append("queue sha256 %s -> %s" % (str(pinned.get("queue_sha256"))[:12],
|
||
str(fresh.get("queue_sha256"))[:12]))
|
||
return "; ".join(out) if out else "no structural difference (byte-level drift only)"
|
||
|
||
|
||
def run(args):
|
||
repo = args.repo
|
||
queue_path = os.path.join(repo, args.queue)
|
||
with open(queue_path, encoding="utf-8") as fh:
|
||
text = fh.read()
|
||
payload = parse(text)
|
||
|
||
if args.check:
|
||
doc = open(args.check, encoding="utf-8").read()
|
||
pinned = extract_pinned(doc)
|
||
if pinned is None:
|
||
print("UNVERIFIABLE: no parseable pinned block in %s" % args.check)
|
||
return 2
|
||
render = build_doc(payload)
|
||
if render != doc:
|
||
# The rendered prose is graded, not only the pin. A pin-only gate
|
||
# stayed green while the table above it had drifted; that was the
|
||
# first whole-gate run's finding, so the bytes are compared whole.
|
||
rl = render.split("\n")
|
||
dl = doc.split("\n")
|
||
first = next((i for i in range(min(len(rl), len(dl))) if rl[i] != dl[i]),
|
||
min(len(rl), len(dl)))
|
||
print("STALE: rendered document differs at line %d" % (first + 1))
|
||
if first < len(dl):
|
||
print(" document: %s" % dl[first][:160])
|
||
if first < len(rl):
|
||
print(" render: %s" % rl[first][:160])
|
||
if pinned is not None:
|
||
print(" pin: %s" % delta(pinned, payload))
|
||
print("regenerate with: python3 tools/queue_index.py --repo %s --write %s"
|
||
% (repo, args.check))
|
||
return 1
|
||
print("index current: %s" % summary(payload))
|
||
return 0
|
||
|
||
# `outside` is None outside a git work tree (no corpus to speak of); --index
|
||
# and --json still work there, which is what makes the tool usable on an
|
||
# extracted copy of the queue -- a bare `unpack NoneType` was a crash until
|
||
# this line, found by verifying the tool against `git show <rev>:...` blobs.
|
||
res = outside_ids(repo, args.queue, exclude=(args.write, args.check))
|
||
outside, corpus = res if res else ({}, 0)
|
||
if args.outside:
|
||
print("outside the queue: %d id(s) named by the %d tracked *.md files read "
|
||
"(`tools/` excluded), which `WORK-QUEUE.md` never mentions" % (len(outside or {}), corpus or 0))
|
||
for qid, files in (outside or {}).items():
|
||
print(" %-5s %s" % (qid, ", ".join(files[:4])
|
||
+ (" +%d" % (len(files) - 4) if len(files) > 4 else "")))
|
||
return 0
|
||
|
||
if args.write:
|
||
with open(args.write, "w", encoding="utf-8") as fh:
|
||
fh.write(build_doc(payload))
|
||
print("wrote %s (%d bytes)" % (args.write, os.path.getsize(args.write)))
|
||
return 0
|
||
|
||
if args.json:
|
||
with open(args.json, "w", encoding="utf-8") as fh:
|
||
json.dump(payload, fh, indent=2, sort_keys=True)
|
||
fh.write("\n")
|
||
|
||
if args.index:
|
||
print("%-4s %-5s %-11s %-7s %-13s %-22s %s"
|
||
% ("#", "id", "lines", "bytes", "role", "status", "title"))
|
||
for i, s in enumerate(payload["sections"], 1):
|
||
print("%-4d %-5s %-11s %-7d %-13s %-22s %s"
|
||
% (i, s["id"], "%d-%d" % (s["line"], s["end_line"]), s["bytes"],
|
||
s["role"][:13], clip((s["status"] or "--"), 22), clip(s["title"], 72)))
|
||
print()
|
||
for r in payload["nested"]:
|
||
print("nested %-4s line %-5d refs %-3d inside %-4s %s"
|
||
% (r["id"], r["line"], r["refs"], r["section"] or "--", r["shape"]))
|
||
if outside:
|
||
print("outside the queue: %s" % ", ".join(outside))
|
||
print()
|
||
|
||
print(summary(payload))
|
||
return 0
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# selftest: a fixture corpus, one cell per stated rule
|
||
# --------------------------------------------------------------------------
|
||
|
||
FIXTURE = """# Work queue
|
||
|
||
Preface text.
|
||
|
||
## Q1 — First item *(lane)*
|
||
|
||
**Status: DONE at `abc1234`.** Body.
|
||
|
||
```
|
||
## Q9 — not a section, this is fenced
|
||
```
|
||
|
||
## Q2 — Second item
|
||
|
||
Body with no status marker.
|
||
|
||
## Q1 — closure — verified
|
||
|
||
**Status: OPEN.** A continuation section for Q1.
|
||
|
||
## Q3 §9 follow-on — odd shape
|
||
|
||
Prose that names Q41 before its record.
|
||
- **Q41 — OPEN (pipe, 2026-10-13, unclaimed).** A nested record.
|
||
- prose mentioning Q42 in passing.
|
||
- **Q43 opened — a bold lead record for the fixture.**
|
||
- a bullet carrying **Q44** mid-line, which is not a record.
|
||
- a tick quoting `--outside` = Q77/Q99 as prose.
|
||
**Status: OPEN.** The status lead opens Q45 on this line.
|
||
"""
|
||
|
||
|
||
def selftest():
|
||
cells = [0]
|
||
bad = [0]
|
||
|
||
def check(name, got, want):
|
||
cells[0] += 1
|
||
if got != want:
|
||
bad[0] += 1
|
||
print("FAIL %s\n got %r\n want %r" % (name, got, want))
|
||
|
||
p = parse(FIXTURE)
|
||
secs = p["sections"]
|
||
check("four sections parsed", len(secs), 4)
|
||
check("ids in order", [s["id"] for s in secs], ["Q1", "Q2", "Q1", "Q3"])
|
||
check("fenced heading is not a section", "Q9" in p["ids_with_section"], False)
|
||
check("second Q1 is a continuation", secs[2]["role"], "continuation")
|
||
check("continuation detail is the role word", secs[2]["role_detail"], "closure")
|
||
check("odd heading shape still parses", secs[3]["id"], "Q3")
|
||
check("multi-section map names Q1", p["ids_multi_section"], {"Q1": [5, 17]})
|
||
check("status marker extracted verbatim", secs[0]["status"], "DONE at `abc1234`.")
|
||
check("status line number", secs[0]["status_line"], 7)
|
||
check("section with no marker is None", secs[1]["status"], None)
|
||
check("continuation status is its own", secs[2]["status"], "OPEN.")
|
||
check("nested ids found", [r["id"] for r in p["nested"]], ["Q41", "Q43", "Q45"])
|
||
check("nested owner is the containing section", p["nested"][0]["section"], "Q3")
|
||
check("nested shape is bullet carrying a bold lead", p["nested"][0]["shape"], "BULLET+BOLD")
|
||
check("bold lead is a record form", p["nested"][0]["form"], "BOLD-LEAD")
|
||
check("status lead is a record form", p["nested"][2]["form"], "STATUS-LEAD")
|
||
check("record line follows the first mention of the same id",
|
||
(p["nested"][0]["line"], p["nested"][0]["first_mention"]), (24, 23))
|
||
check("prose mention is not a record (Q42)", "Q42" in [r["id"] for r in p["nested"]], False)
|
||
check("mid-line bold mention is not a record (Q44)",
|
||
"Q44" in [r["id"] for r in p["nested"]], False)
|
||
check("quoted tool output opens nothing (Q77/Q99)",
|
||
{"Q77", "Q99"} & {r["id"] for r in p["nested"]}, set())
|
||
check("mentions are still counted as refs", p["nested"][0]["refs"], 2)
|
||
check("sections tile the file with no gap",
|
||
all(secs[i]["end_line"] + 1 == secs[i + 1]["line"] for i in range(len(secs) - 1)), True)
|
||
check("last section ends at the last line", secs[-1]["end_line"], p["queue_lines"])
|
||
check("preamble ends before the first heading", p["preamble"]["end_line"], 4)
|
||
check("byte accounting is exact",
|
||
p["preamble"]["bytes"] + sum(s["bytes"] for s in secs), p["queue_bytes"])
|
||
check("bytes are the file's own length", p["queue_bytes"], len(FIXTURE.encode()))
|
||
check("sha256 is the file's own", p["queue_sha256"],
|
||
hashlib.sha256(FIXTURE.encode()).hexdigest())
|
||
|
||
# --- the gate, round trip and both directions of falsification ----------
|
||
root = tempfile.mkdtemp(prefix="queue-index-selftest-")
|
||
repo = os.path.join(root, "repo")
|
||
os.makedirs(os.path.join(repo, "docs"))
|
||
queue = os.path.join(repo, "WORK-QUEUE.md")
|
||
doc_path = os.path.join(repo, DOC_PATH)
|
||
subprocess.check_call(["git", "-C", repo, "init", "-q", "-b", "main"])
|
||
subprocess.check_call(["git", "-C", repo, "config", "user.email", "s@invalid"])
|
||
subprocess.check_call(["git", "-C", repo, "config", "user.name", "s"])
|
||
with open(queue, "w", encoding="utf-8") as fh:
|
||
fh.write(FIXTURE)
|
||
subprocess.check_call(["git", "-C", repo, "add", "-A"])
|
||
subprocess.check_call(["git", "-C", repo, "commit", "-q", "-m", "fixture"])
|
||
|
||
import contextlib
|
||
import io
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=doc_path, check=None, outside=False))
|
||
check("--write exits 0", rc, 0)
|
||
check("document written", os.path.exists(doc_path), True)
|
||
doc = open(doc_path, encoding="utf-8").read()
|
||
check("document carries the pinned block", extract_pinned(doc) is not None, True)
|
||
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("--check on its own output exits 0", rc, 0)
|
||
check("--check says current", "index current:" in buf.getvalue(), True)
|
||
|
||
# --- the document is a function of the queue alone -----------------------
|
||
# A prose file that names an id the queue never mentions must not move a
|
||
# byte of the document, and must not turn --check RED. Baking the corpus
|
||
# table into the document did both: same queue, three different documents,
|
||
# and a checkin quoting an id reddened a gate about the queue's currency.
|
||
with open(os.path.join(repo, "notes.md"), "w", encoding="utf-8") as fh:
|
||
fh.write("# notes\n\nQ63 is named only here; Q1 is a queue id.\n")
|
||
subprocess.check_call(["git", "-C", repo, "add", "notes.md"])
|
||
before = open(doc_path, encoding="utf-8").read()
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=doc_path, check=None, outside=False))
|
||
check("re-render beside a prose mention exits 0", rc, 0)
|
||
check("a prose mention does not change the document",
|
||
open(doc_path, encoding="utf-8").read(), before)
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("--check stays green when prose names an id", rc, 0)
|
||
|
||
# --- the outside leg is reported, not embedded ---------------------------
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=None, outside=True))
|
||
out = buf.getvalue()
|
||
check("--outside exits 0", rc, 0)
|
||
check("--outside counts the corpus it actually read",
|
||
"1 id(s) named by the 2 tracked *.md files" in out, True)
|
||
check("--outside names Q63 and the file that carries it",
|
||
" Q63 notes.md" in out, True)
|
||
|
||
# --- a tree that is not a git repo must still index -----------------------
|
||
# The `outside` leg needs git; --index does not. A verifier that extracts
|
||
# `git show <rev>:WORK-QUEUE.md` into a temp dir has no .git, and this used to
|
||
# crash on `outside, corpus = outside_ids(...)` returning None.
|
||
plain = os.path.join(root, "plain")
|
||
os.makedirs(plain)
|
||
with open(os.path.join(plain, "WORK-QUEUE.md"), "w", encoding="utf-8") as fh:
|
||
fh.write(FIXTURE)
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=plain, queue="WORK-QUEUE.md", index=True,
|
||
json=None, write=None, check=None, outside=False))
|
||
check("--index outside a git work tree exits 0", rc, 0)
|
||
check("--index outside a git work tree still summarizes",
|
||
"index: 4 sections" in buf.getvalue(), True)
|
||
|
||
# falsify 1: an append moves the last section's extent
|
||
with open(queue, "a", encoding="utf-8") as fh:
|
||
fh.write("- an appended line.\n")
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("--check goes RED on an append", rc, 1)
|
||
check("RED names the moved section", "changed extent or status" in buf.getvalue(), True)
|
||
|
||
# falsify 2: a new heading is a new section
|
||
with open(queue, "a", encoding="utf-8") as fh:
|
||
fh.write("\n## Q7 — a brand new item\n\n**Status: OPEN.**\n")
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("--check goes RED on a new heading", rc, 1)
|
||
check("RED names the new section", "new section(s): Q7" in buf.getvalue(), True)
|
||
|
||
# falsify 3: a document with no block cannot be verified
|
||
open(doc_path, "w", encoding="utf-8").write("# empty\n")
|
||
buf = io.StringIO()
|
||
with contextlib.redirect_stdout(buf):
|
||
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
|
||
json=None, write=None, check=doc_path, outside=False))
|
||
check("no pin -> exit 2", rc, 2)
|
||
check("no pin -> says so", "UNVERIFIABLE" in buf.getvalue(), True)
|
||
|
||
print("selftest: %d cells, %d passed, %d failed" % (cells[0], cells[0] - bad[0], bad[0]))
|
||
return 1 if bad[0] else 0
|
||
|
||
|
||
def main(argv=None):
|
||
ap = argparse.ArgumentParser(
|
||
description="queue_index.py -- the work queue's table of contents, and the gate "
|
||
"that it is current.")
|
||
ap.add_argument("--repo", default=".")
|
||
ap.add_argument("--queue", default=QUEUE_DEFAULT)
|
||
ap.add_argument("--index", action="store_true", help="print the section table")
|
||
ap.add_argument("--json", metavar="PATH", help="write the parse here")
|
||
ap.add_argument("--write", metavar="PATH", help="render the index document here")
|
||
ap.add_argument("--check", metavar="PATH", help="re-derive and compare a rendered index")
|
||
ap.add_argument("--outside", action="store_true",
|
||
help="report ids the prose corpus names that the queue never mentions")
|
||
ap.add_argument("--selftest", action="store_true", help="fixture corpus, 46 cells")
|
||
args = ap.parse_args(argv)
|
||
if args.selftest:
|
||
return selftest()
|
||
return run(args)
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|