#!/usr/bin/env python3 """queue_claim_check.py — the claim check for `WORK-QUEUE.md`, shipped with its tests. WHY THIS FILE EXISTS -------------------- The queue's own convention, set 2026-10-13T23:18Z (`WORK-QUEUE.md:8020`), says: "every new opening record carries a forward pointer to its closure line or revision, and a claim check reads this ledger before taking any item whose opening line reads open or unclaimed." Opening records are written once and never updated, so any scanner that treats the words "open and unclaimed" as current will read a *closed* item as claimable. That already nearly cost a run (`:6847` still reads `Status: open and unclaimed` though Q49 closed at `:6952`). Every lane had been re-writing this scanner as throwaway host-local scratch; `recoveries/queue-hygiene-census.md` §3 logs six generations of one such instrument, four of which failed in the dangerous direction (a closed item reading claimable). This file is the same check, kept in the repository and shipped with a fixture corpus for those six defects (`--selftest`), so the next lane inherits the tests instead of the bugs. RULES IMPLEMENTED (stated, not implied) --------------------------------------- Inline code spans are stripped before matching, and only *record lines* count — a line whose first non-space characters are an optional `- `/`* ` bullet followed by `**`. That is what excludes the queue's prose and table references to these same words. A FENCED BLOCK IS QUOTED MATERIAL, NOT A RECORD (2026-10-19). `queue_index.py` and `checkin_check.py` already refuse to read structure out of a fence, on the rule that a quoted heading is not a heading, and `citation_resolve_check.py` applies the same rule to citations. This file had no fence state, so a tick that quoted an example record inside a ``` block opened a phantom item: measured on a 19-line fixture carrying one fenced `**Q99 — OPEN …**` and one fenced `**Status: OPEN …**`, the census printed two OPEN records no writer ever opened (`before/fence-fixture.txt` in the tick's artifact). The live queue at HEAD has no record inside a fence — 0 of its fenced lines lead `**` — so the rule moves no verdict there; it is enforced against the next one. What is set aside is printed on every run, never silently dropped, and an unclosed fence fails OPEN — nothing is masked, every line is graded, and `FENCE_UNCLOSED` names the line — because masking to EOF would hide the newest records, and those are the ones a claim check exists to surface. The legacy BQ unit is a transcription of another tool and is left byte-for-byte; its fence delta is printed beside it, not applied. OPENING RECORD — a record line carrying either (a) `Status:` followed by `open`/`OPEN`, or (b) an item id `Q` followed within 60 characters by an `—`/`–`/`--` separator and then `OPEN`/`open`/`Opened`/`accepted`. Its item id is the leading `**Q` of the line if present; otherwise the id is inherited from the nearest preceding `## Q` header. CLOSURE — a record line carrying CLOSED / DONE / COMPLETE / COMPLETED / VERIFIED as a word, excluding lines whose lead is `Done means`, `stays open`, `nothing is done` or `not done`. It is anchored to an item by (in order): its own leading `**Q`; else any `Q` token on the line; else the nearest preceding `## Q` header. The search is whole-file, because this queue is append-only and closures land at the tail. A closure line that anchors an opening record's item => that item is CLOSED. An opening record with no anchored closure => OPEN. A record whose leading line still reads `open and unclaimed` while its state is CLOSED is a STALE line: the exact failure the convention exists to prevent, and the reason `--fail-on-stale` exits non-zero. - **The unit.** `closure_candidates` counts **anchored closure statements (ACS)**: a record line leading `**Q`/`**Status:` with a closure word in the first `CLOSURE_WINDOW` (120) characters, code spans stripped, exclusion phrases tested at the lead, anchored to an item id. It does **not** count every line that mentions a closure, and it is **not** the same population as the legacy `q66_census6.py` figure. See UNITS below before citing a number. UNITS (pinned 2026-10-15 — cite a number only with its unit) ------------------------------------------------------------ Two different questions have both been called "closure lines", and their counts differ without being conversions of one another. Measured twice, at the revision Run 37 read (`29a4872`, queue 1,170,483 B / sha256 `fbe51445a89f81ab…`) and at `28ca8e5` (1,193,681 B / `c3b4a290dc64bd24…`) — identical on both: * **ACS — anchored closure statement** (this file's `closure_candidates`) = **73**. * **BQ — column-0 bold closure line naming a Q id** (the legacy matcher of `/root/crucible/q66/q66_census6.py:54-62`, host-local, Run 37) = **85**. A line starting at column 0 with `**`, carrying a `Q` token anywhere and a closure word anywhere and case-insensitively, with none of that file's exclusion phrases anywhere (`Done means`, `Done =`, `not done`, `nothing is done`, `stays open`, `remains open`, `is OPEN`, `stays OPEN`, `not reopened`, `nothing is re-opened`). Headline figures pinned at `abe08a9` (WORK-QUEUE.md 1,229,541 B / 9,953 lines, `python3 tools/queue_claim_check.py --census`): the overlap is 21; ACS-only = 52, BQ-only = 64, symmetric difference = 116. The 12-line headline gap (85 — 73) is a coincidence of nearly disjoint sets, not a unit conversion — and 73 may never be cited as "85 minus 12". The five axes, counted: The pin is checked, not trusted. `--census` re-reads every figure in this paragraph and in the five axis bullets, compares them with the queue it just read, and prints `moved:` with each figure that no longer holds. A pinned census is a claim about a revision; printed with no comparison beside it, it reads as a claim about the present — which is how a reader ends up quoting a figure the tool no longer prints while every command exits 0. * 46 BQ-only lines fail ACS's record-lead rule (an addendum / status / verification **prose** line, which BQ takes because it is column-0 bold); * 39 ACS-only lines carry no `Q` token (ACS inherits the container id; BQ requires the token on the line, which is why it drops `**Status: DONE …` rows); * 18 BQ-only lines carry a **lowercase** closure word (ACS is case-sensitive); * 12 ACS-only lines are indented or bulleted (BQ requires column 0); * 1 ACS-only line carries a BQ exclusion phrase (`:4035`, "stays open") that ACS tests only at the lead. `--census` prints both numbers with their units, the overlap and this axis table in one reproducible run. KNOWN BLIND SPOTS (measured, not hand-waved) -------------------------------------------- - Item ids are labels, not keys: this file's own record carries a collision (career-lane apply queue `:7292` also reads `Q53`; the queue's own reply 62 notes it). Binding is by **section**: within a duplicated id, a closure binds to the last record of that id opened at or before the closure, so neither namesake can close the other. Until 2026-10-19 the tool took the id alone as sufficient, graded `:7303` CLOSED on Mercury's `:9869`, and printed `open set: Q37:5278` while the ledger's own blocks asserted `:7303` open; the row still reads `COLLISION` because the labelling defect is the ledger's. - The state is only as good as the closure line's anchoring. Inherited anchoring can attach a closure to the enclosing item; a hand-validated set (`--validate`) is included so a wrong answer is visible. - Nothing here judges whether a closure is *earned*, only that it exists. USAGE ----- python3 tools/queue_claim_check.py # census of ./WORK-QUEUE.md python3 tools/queue_claim_check.py --queue PATH --json OUT.json python3 tools/queue_claim_check.py --validate # + the hand-read answers python3 tools/queue_claim_check.py --git # resolve closure revisions python3 tools/queue_claim_check.py --selftest # 24 fixtures, exit 0/1 python3 tools/queue_claim_check.py --census # both closure units + the axis table Exit codes: 0 clean · 1 self-test failure · 2 stale opening line (--fail-on-stale) 3 queue unreadable. No file is written unless --json names one. """ import argparse import contextlib import io import json import os import re import subprocess import sys import tempfile # -------------------------------------------------------------------------- # matching primitives # -------------------------------------------------------------------------- CODE_SPAN = re.compile(r'`[^`]*`') RECORD_PREFIX = re.compile(r'^[ \t]*(?:[-*]\s+)?\*\*') ITEM_LEAD = re.compile(r'^[ \t]*(?:[-*]\s+)?\*\*Q(\d+)\b') HEADER_ITEM = re.compile(r'^#{1,3}\s+Q(\d+)\b') Q_TOKEN = re.compile(r'\bQ(\d+)\b') OPEN_STATUS = re.compile(r'Status:\s*\**\s*(?:open|OPEN)\b') OPEN_Q = re.compile( r'Q\d+\b[^*\n]{0,60}?(?:\u2014|\u2013|--)\s*\**\s*(?:OPEN|open|Opened|accepted)\b') CLOSURE_WORD = re.compile(r'\b(CLOSED|DONE|COMPLETE|COMPLETED|VERIFIED)\b') CLOSURE_EXCLUDE = re.compile(r'^\s*(?:[-*]\s+)?\*\*\s*(?:Done means|stays open|nothing is done|not done)\b', re.I) CLOSURE_EXCLUDE_ANY = re.compile(r'stays open|nothing is done', re.I) CONTAINER_HEAD = re.compile(r'^\s*(?:[-*]\s+)?\*\*Q(\d+)\b') CONTAINER_TITLE_MAX = 100 CONTAINER_SEP = re.compile(r'(?:\u2014|\u2013|--|:)') # A closure must be a *statement*: a record line whose lead is an item id or # `Status:`, with the closure word inside the statement window. That is what # refuses the queue's prose lines that mention a closure in passing # (`:6631` "- **No item state changed this tick:** ... no DONE line", `:8430` # "- **No item closed, no new item opened.** ... remains DONE at 32ad55b"). CLOSURE_LEAD = re.compile(r'^\s*(?:[-*]\s+)?\*\*(?:Q\d+\b|Status:)') CLOSURE_WINDOW = 120 STALE_LITERAL = re.compile(r'Status:\s*\**\s*open and unclaimed', re.I) SHA7 = re.compile(r'`([0-9a-f]{7,63})[0-9a-f\u2026]*`') # Hand-read ground truth: twelve answers read off the queue by hand before any # instrument runs (queue-hygiene-census.md §2). Two independent lanes now. HAND_READ = { # Read by hand off WORK-QUEUE.md at `7fa6261` (2026-10-19). :8530 was 'OPEN' # here until this revision; it closed legitimately at :9869 on 2026-10-16 and # the hand-read answer had gone stale, which is what `--validate` exists to # show. :7303 is the collided sibling and is OPEN in its own words. 2264: 'CLOSED', 2684: 'CLOSED', 2898: 'CLOSED', 6348: 'CLOSED', 6419: 'CLOSED', 6542: 'CLOSED', 6847: 'CLOSED', 6990: 'CLOSED', 7106: 'CLOSED', 7167: 'CLOSED', 7303: 'OPEN', 8530: 'CLOSED', } def strip_code(line): return CODE_SPAN.sub('', line) # A FENCED BLOCK IS QUOTED MATERIAL, NOT A RECORD (2026-10-19). `queue_index.py` # (:207, :266) and `checkin_check.py` (:251) already refuse to read structure out # of a fence; this scanner had no fence state, so a quoted example record opened a # phantom item. Line numbers stay absolute -- the mask is a set of 1-based line # numbers, so every citation into the queue keeps its meaning. FENCE = re.compile(r'^\s*(?:`{3,}|~{3,})') def fence_mask(lines): """(fenced 1-based line numbers, block count, opening line if never closed). Markdown's rule is that an unclosed fence runs to the end of the document. For a CLAIM CHECK that rule is the dangerous direction -- masked text cannot open a record, so a stray fence would hide the newest items, which are exactly the ones a claim check exists to surface. So an unclosed fence fails OPEN: nothing is masked, every line is graded, and the opening line is returned and printed as FENCE_UNCLOSED. Quoted material graded as a record is a visible row (the operator sees the diagnostic and the row); a record silently masked is an item nobody sees at all. """ out = set() in_fence = False opener = None opened_at = None blocks = 0 for n, ln in enumerate(lines, 1): m = FENCE.match(ln) if m: mark = m.group(0).strip()[0] if not in_fence: in_fence, opener, opened_at = True, mark, n blocks += 1 out.add(n) continue if mark == opener: in_fence, opener = False, None out.add(n) continue if in_fence: out.add(n) if in_fence: # malformed document: grade everything return set(), blocks, opened_at return out, blocks, None def is_record_line(line): return bool(RECORD_PREFIX.match(line)) def is_opening_record(line): s = strip_code(line) if not is_record_line(s): return False return bool(OPEN_STATUS.search(s) or OPEN_Q.search(s)) def is_closure_line(line): s = strip_code(line) if not CLOSURE_LEAD.match(s): return False if CLOSURE_EXCLUDE.match(s): return False return bool(CLOSURE_WORD.search(s[:CLOSURE_WINDOW])) def lead_item(line): m = ITEM_LEAD.match(line) return int(m.group(1)) if m else None def line_ids(line): """Every id a closure line can anchor to: its lead id, else its Q-tokens.""" lead = lead_item(line) if lead is not None: return {lead} return {int(x) for x in Q_TOKEN.findall(strip_code(line))} # -------------------------------------------------------------------------- # the legacy unit (BQ), transcribed from /root/crucible/q66/q66_census6.py:41-62 # -------------------------------------------------------------------------- # Kept here, not for its answer but so that the OTHER number in circulation can # be reproduced from one command and printed beside this file's own unit. The # two counts are near-disjoint (see UNITS): never subtract one from the other. LEGACY_CLOSE_WORD = re.compile(r'\b(CLOSED|DONE|COMPLETE|VERIFIED)\b', re.I) LEGACY_NOT_CLOSURE = re.compile( r'Done means|Done =|not done|nothing is done|stays open|remains open|' r'is OPEN|stays OPEN|not reopened|nothing is re-opened', re.I) def legacy_closures(text): """Line numbers the BQ unit counts: column-0 `**`, a Q token, a closure word anywhere (case-insensitive), no legacy exclusion phrase.""" out = [] for i, ln in enumerate(text.split('\n'), 1): if not ln.startswith('**') or LEGACY_NOT_CLOSURE.search(ln): continue if re.search(r'\bQ(\d+)\b', ln) and LEGACY_CLOSE_WORD.search(ln): out.append(i) return out def _acs_reject_axis(line): """Why the ACS unit drops a line the BQ unit counted.""" s = strip_code(line) if not CLOSURE_LEAD.match(s): return 'lead: not a `**Q`/`**Status:` record line (BQ takes any column-0 `**`)' if CLOSURE_EXCLUDE.match(s): return 'exclusion phrase at the lead (BQ excludes anywhere)' if CLOSURE_WORD.search(s[:CLOSURE_WINDOW]): return 'other' if CLOSURE_WORD.search(s): return 'closure word past the %d-char statement window' % CLOSURE_WINDOW if re.search(r'\b(CLOSED|DONE|COMPLETE|VERIFIED)\b', s, re.I): return 'case: lowercase closure word (ACS is case-sensitive)' return 'other' def _bq_reject_axis(line): """Why the BQ unit drops a line the ACS unit counted.""" if not line.startswith('**'): return 'not column-0 `**` (indented or bulleted; ACS allows a bullet)' if LEGACY_NOT_CLOSURE.search(line): return 'BQ exclusion phrase anywhere on the line' if not re.search(r'\bQ(\d+)\b', line): return 'no Q token on the line (ACS inherits the container id)' if not LEGACY_CLOSE_WORD.search(line): return 'BQ closure word missing' return 'other' # -------------------------------------------------------------------------- # the docstring's pinned census -- prose vs the queue that is actually read # -------------------------------------------------------------------------- # The UNITS section above states a census for revision abe08a9. Those figures # were true at that revision, and they are kept for the reader who wants to # know why the two units are not convertible. What nothing said until # 2026-10-20 is that the live queue has moved past them: BQ 85 -> 86, BQ-only # 64 -> 65, symmetric difference 116 -> 117, lead axis 46 -> 47, while every # command exited 0. A pinned census printed with no comparison reads as a # claim about the present, so `--census` re-reads the prose and names every # figure the queue it just read no longer reproduces. It reports; it never # changes the exit code, because a queue that grows is not a fault. DOC_PIN_RE = re.compile(r'pinned at `([0-9a-f]{7,40})`') DOC_BOLD_RE = re.compile(r'=\s*\*\*(\d+)\*\*') DOC_AXES_RE = re.compile( r'the overlap is\s+(\d+);\s*ACS-only = (\d+), BQ-only = (\d+),\s*' r'symmetric difference = (\d+)') DOC_AXIS_RE = ( ('BQ-only/lead', re.compile(r'(\d+) BQ-only lines fail')), ('BQ-only/lowercase', re.compile(r'(\d+) BQ-only lines carry a \*\*lowercase\*\*')), ('ACS-only/no-token', re.compile(r'(\d+) ACS-only lines carry no')), ('ACS-only/indented', re.compile(r'(\d+) ACS-only lines are indented')), ('ACS-only/exclusion', re.compile(r'(\d+) ACS-only line carries a BQ exclusion')), ) DOC_FIGURE_NAMES = ('ACS', 'BQ', 'overlap', 'ACS-only', 'BQ-only', 'symmetric difference', 'BQ-only/lead', 'BQ-only/lowercase', 'ACS-only/no-token', 'ACS-only/indented', 'ACS-only/exclusion') # The axis labels are matched by their exact text, because a renamed axis is a # figure nobody checks any more. `computed_figures` below asserts the mapping by # counting on these two strings; if an axis is reworded the count becomes 0 and # the drift line names it -- visible, not silent. AXIS_LEAD = ('lead: not a `**Q`/`**Status:` record line ' '(BQ takes any column-0 `**`)') AXIS_LOWER = 'case: lowercase closure word (ACS is case-sensitive)' AXIS_NOTOKEN = 'no Q token on the line (ACS inherits the container id)' AXIS_INDENT = 'not column-0 `**` (indented or bulleted; ACS allows a bullet)' AXIS_EXCL = 'BQ exclusion phrase anywhere on the line' def doc_figures(doc=None): """The pinned census as the prose states it: {figure: int}, plus {'pin': rev}. A figure the prose can no longer be read for is *absent* from the result and never defaulted -- a cell asserts the figure count, so a number cannot vanish quietly by being reworded. """ if doc is None: doc = __doc__ or '' out = {} m = DOC_PIN_RE.search(doc) if m: out['pin'] = m.group(1) bold = DOC_BOLD_RE.findall(doc) if len(bold) >= 2: out['ACS'], out['BQ'] = int(bold[0]), int(bold[1]) m = DOC_AXES_RE.search(doc) if m: (out['overlap'], out['ACS-only'], out['BQ-only'], out['symmetric difference']) = (int(x) for x in m.groups()) for name, rx in DOC_AXIS_RE: m = rx.search(doc) if m: out[name] = int(m.group(1)) return out def computed_figures(lines, acs, bq): """The same names, measured on the queue text that was just read.""" acs_counts = {} for n in bq - acs: a = _acs_reject_axis(lines[n - 1]) acs_counts[a] = acs_counts.get(a, 0) + 1 bq_counts = {} for n in acs - bq: a = _bq_reject_axis(lines[n - 1]) bq_counts[a] = bq_counts.get(a, 0) + 1 return { 'ACS': len(acs), 'BQ': len(bq), 'overlap': len(acs & bq), 'ACS-only': len(acs - bq), 'BQ-only': len(bq - acs), 'symmetric difference': len(acs ^ bq), 'BQ-only/lead': acs_counts.get(AXIS_LEAD, 0), 'BQ-only/lowercase': acs_counts.get(AXIS_LOWER, 0), 'ACS-only/no-token': bq_counts.get(AXIS_NOTOKEN, 0), 'ACS-only/indented': bq_counts.get(AXIS_INDENT, 0), 'ACS-only/exclusion': bq_counts.get(AXIS_EXCL, 0), } def doc_drift(stated, computed): """[(name, stated or None, computed)] for every pinned figure that moved.""" return [(name, stated.get(name), computed[name]) for name in DOC_FIGURE_NAMES if stated.get(name) != computed[name]] # -------------------------------------------------------------------------- # the census # -------------------------------------------------------------------------- def item_title_container(line): """The id of a `**Q — title**` container line, or None. A container is a *title statement*: bold, leading item id, a separator inside the bold span, and the bold span closing within CONTAINER_TITLE_MAX characters. Measured reasons, both from the live queue: `:5597` (`- **Q41 — the non-magazine excerpt class *and* placement** *(archive verification; ...`) is Q41's container and must count, while `:4457` (`**Q32 record precision — two of the thirteen new assertions cannot fail, ...`) is a prose record that must not — counting it stole Q33's own closure lines (`:4505`) for Q32. """ m = CONTAINER_HEAD.match(line) if not m: return None rest = line[m.end():] close = rest.find('**') if close == -1 or close > CONTAINER_TITLE_MAX: return None if not CONTAINER_SEP.search(rest[:close]): return None return int(m.group(1)) def container_index(lines, loose=False): """[(line, id_or_None)] for every *container* line, in file order. A container is a header of any level, or a bold record line that leads with an item id. Two strictnesses, because the two questions differ: * `loose=True` — used to name an opening record that carries no id of its own. Any bold line leading with `Q` counts; the item whose block the record sits in is the nearest such marker above it. * `loose=False` — used to anchor a closure. Here the marker must be a *title statement* (see `item_title_container`), because a long bold prose record that merely starts with an id steals the anchor: `:4457` (`**Q32 record precision — two of the thirteen new assertions cannot fail, ...`) otherwise took `:4505`, Q33's own closure, for Q32. """ out = [] for n, l in enumerate(lines, 1): if l.startswith('#'): m = HEADER_ITEM.match(l) out.append((n, int(m.group(1)) if m else None)) elif loose and CONTAINER_HEAD.match(l): m = ITEM_LEAD.match(l) out.append((n, int(m.group(1)) if m else None)) else: cid = item_title_container(l) if cid is not None: out.append((n, cid)) return out def inherit_id(n, hidx): """Id of the nearest preceding header, or None.""" best = None for ln, qid in hidx: if ln < n: best = qid else: break return best def census(text): lines = text.splitlines() fenced, n_blocks, unclosed = fence_mask(lines) hidx = container_index(lines, loose=False) lidx = container_index(lines, loose=True) closures = [] # (line, idset, text) unanchored = [] # closure statements that name no item for n, l in enumerate(lines, 1): if n in fenced: # quoted material, not a statement continue if not is_closure_line(l): continue ids = line_ids(l) if not ids: inh = inherit_id(n, hidx) if inh is None: unanchored.append(n) continue ids = {inh} closures.append((n, ids, l)) records = [] for n, l in enumerate(lines, 1): if n in fenced: # quoted material, not a record continue if not is_opening_record(l): continue lead = lead_item(l) inherited = False if lead is None: lead = inherit_id(n, lidx) inherited = True anchored = [(cn, cl) for (cn, ids, cl) in closures if lead is not None and lead in ids] # State is decided by a closure recorded at or after the opening record. # Earlier closures are reported as evidence but never set the state: the # queue's record order is not the work's order (Q25's DONE line sits # 96 lines *above* its own opening record). closing = [c for c in anchored if c[0] >= n] state = 'CLOSED' if closing else 'OPEN' stale = bool(STALE_LITERAL.search(strip_code(l))) and state == 'CLOSED' records.append({ 'line': n, 'item': None if lead is None else 'Q%d' % lead, 'inherited_id': inherited, 'state': state, 'closure_line': closing[0][0] if closing else None, 'closure_lines_after': [c[0] for c in closing], 'closure_lines_before': [c[0] for c in anchored if c[0] < n], 'closure_text': closing[0][1].strip() if closing else None, 'stale': stale, 'text': l.strip(), }) # collisions: two opening records resolving to one id seen = {} for r in records: if r['item'] is None: continue seen.setdefault(r['item'], []).append(r['line']) for r in records: r['collision'] = bool(r['item'] and len(seen.get(r['item'], [])) > 1) # A duplicate label cannot bind a closure by id alone. Until 2026-10-19 this # file took `lead in ids` as sufficient and let EVERY record carrying the # label inherit the same closure -- so one item's DONE closed its namesake. # On the live queue that graded the career-lane apply queue (:7303, held for # Alexander) CLOSED on Mercury's :9869 and dropped it from `open set:`, while # four standing ledger blocks assert it open. A record owns its own section, # so a closure binds to the last record of that id opened at or before it -- # never to a sibling that ended before the closure was written. groups = {} for r in records: if r['item'] is not None: groups.setdefault(r['item'], []).append(r) for item, recs in groups.items(): if len(recs) < 2: continue recs.sort(key=lambda r: r['line']) qid = int(item[1:]) for i, r in enumerate(recs): lo = r['line'] hi = recs[i + 1]['line'] if i + 1 < len(recs) else None owned = [(c[0], c[2]) for c in closures if qid in c[1] and c[0] >= lo and (hi is None or c[0] < hi)] r['closure_line'] = owned[0][0] if owned else None r['closure_lines_after'] = [c[0] for c in owned] r['closure_text'] = owned[0][1].strip() if owned else None r['state'] = 'CLOSED' if owned else 'OPEN' r['stale'] = (bool(STALE_LITERAL.search(strip_code(lines[lo - 1]))) and r['state'] == 'CLOSED') raw_literal = len(re.findall(r'Status:\s*\**\s*open and unclaimed', text, re.I)) return { 'lines': len(lines), 'bytes': len(text.encode()), 'records': records, 'closure_candidates': len(closures), 'closure_unanchored': unanchored, 'literal_anywhere': raw_literal, 'fenced_lines': len(fenced), 'fenced_blocks': n_blocks, 'fenced_unclosed': unclosed, } def annotate_revisions(rec, git_dir): """First 7-hex backticked token on the closure line, resolved with git.""" out = [] txt = rec.get('closure_text') or '' for tok in SHA7.findall(txt): try: p = subprocess.run(['git', 'cat-file', '-t', tok], cwd=git_dir, capture_output=True, text=True, timeout=20) kind = p.stdout.strip() or p.stderr.strip() ok = p.returncode == 0 except Exception as exc: # pragma: no cover kind, ok = str(exc), False out.append({'token': tok, 'resolves': ok, 'kind': kind}) return out # -------------------------------------------------------------------------- # fixtures — the six defects logged in recoveries/queue-hygiene-census.md §3 # -------------------------------------------------------------------------- FIXTURES = [ ('v1 header-walk only: an opening record nested inside another section', '## Q10 — outer section\n\nDone means: nothing here.\n\n' '- **Q99 — OPEN (nested, not a `##` header).** body\n', ['Q99:OPEN']), ('v2 loose key: a Q7 continuation note is not an opening record', '## Q7 — section\n\n- **Q7, measured:** the unclaimed executable set is open beside it.\n', []), ('v3a a column-0 `---` rule does not cut the region', '## Q11 — section\n\n**Status: open and unclaimed.**\n\n---\n\n' '**Status: DONE — closed below the rule.**\n', ['Q11:CLOSED']), ('v3b `Done means:` is not a closure word', '## Q12 — section\n\n**Status: open and unclaimed.**\n\n**Done means:** one row per space.\n', ['Q12:OPEN']), ("v4 closures are anchored: another item's DONE does not close this one", '## Q13 — section\n\n**Status: open and unclaimed.**\n\n## Q14 — other\n\n' '**Q14 — OPEN (lane).**\n\n**Q14 — DONE at 4010b96.**\n', ['Q13:OPEN', 'Q14:CLOSED']), ('v5 whole-file search: a closure landing after a later opening record still counts', '## Q15 — section\n\n**Q15 — OPEN (lane).**\n\n## Q16 — other\n\n**Q16 — OPEN (lane).**\n\n' '**Q15 — DONE at 4010b96.**\n', ['Q15:CLOSED', 'Q16:OPEN']), ('grep trap: the same words in a code span or a table row are not an opening record', '## Q17 — section\n\n**Status: open and unclaimed. Opened by the pipe.**\n\n' '**Status: DONE — closed.**\n\n' '- (1) Three lines carry the literal `Status: open and unclaimed` (`:1`, `:2`).\n' '\n| `Status: open and unclaimed` | `:2264` | Q21 | DONE |\n', ['Q17:CLOSED']), ('unnumbered opening record inherits the enclosing `## Q` header', '## Q20 — section\n\n**Status: OPEN — held for Alexander.**\n', ['Q20:OPEN']), ('a closed item whose opening line never got a pointer is flagged STALE', '## Q21 — section\n\n**Status: open and unclaimed. Opened by the pipe.**\n\n' '**Status: DONE — `recoveries/x.md`, 2026-10-10.**\n', ['Q21:CLOSED']), ('a bold prose record naming another item does not steal the container', '## Q30 — section\n\n**Status: open and unclaimed.**\n\n' '- **Q31 record precision — two of the thirteen new assertions cannot fail, and two totals are not\n' ' enumerable as stated**, so the record is corrected in place rather than restated.\n\n' '**Status: DONE — closed by the pipe.**\n', ['Q30:CLOSED']), # The live defect this cell was written for (2026-10-19): the career-lane # apply queue (:7303) and Mercury's record (:8530) both read `Q53`, so # Mercury's `:9869` DONE closed both and the tool printed `open set: Q37:5278` # against four ledger blocks asserting :7303 open. Pre-fix this cell reads # ['Q40:CLOSED', 'Q41:OPEN', 'Q40:CLOSED']. ("collision: a duplicate label does not let one record's closure close the other", '## Q40 — first use of the label\n\n**Status: OPEN — held for Alexander.**\n\n' '## Q41 — an unrelated section between them\n\n**Q41 — OPEN (lane).**\n\n' '## Q40 — second use of the same label\n\n**Status: OPEN — queued.**\n\n' '**Q40 — DONE at 4010b96.**\n', ['Q40:OPEN', 'Q41:OPEN', 'Q40:CLOSED']), # The live defect these three cells were written for (2026-10-19): this file # had no fence state, so a tick quoting an example record inside a fenced # block opened a phantom item. Measured pre-fix: the first cell reads # ['Q60:CLOSED', 'Q99:CLOSED'] -- a quoted item in the open set -- and the # second reads two colliding Q61 rows (the fenced `**Status:` inherited the # header). The third is identical on both sides by design: an unclosed fence # fails OPEN, so the cell pins the FENCE_UNCLOSED line, not the masking. ('fence: a record inside a ``` block is quoted material, not a record', '## Q60 — section\n\n**Q60 — OPEN (lane).**\n\n**Q60 — DONE at 4010b96.**\n\n' 'Prose quoting a tick:\n\n```\n**Q99 — OPEN (pipe).**\n' '**Q99 — DONE at 4010b96.**\n```\n', ['Q60:CLOSED']), ('fence: a tilde fence is a fence, and a fenced Status line inherits nothing', '## Q61 — section\n\n**Q61 — OPEN (lane).**\n\n~~~\n' '**Status: OPEN — queued.**\n~~~\n', ['Q61:OPEN']), ('fence: an unclosed fence fails open — every line graded, the line named', '## Q62 — section\n\n**Q62 — OPEN (lane).**\n\n```\n**Q63 — OPEN (lane).**\n', ['Q62:OPEN', 'Q63:OPEN']), ] UNIT_FIXTURES = [ # name, text, ACS count, BQ count — the four axes that separate the units. ('unit axis 1: a column-0 bold *prose* line is BQ but not ACS', '## Q40 — section\n\n**Addendum, 2026-10-08 — Q40 CLOSED at `8bd0b09`.**\n', 0, 1), ('unit axis 2: a lowercase closure word is BQ but not ACS', '## Q41 — section\n\n**Q41 — verified below.**\n', 0, 1), ('unit axis 3: a status row with no Q token is ACS but not BQ', '## Q42 — section\n\n**Status: DONE — closed.**\n', 1, 0), ('unit axis 4: an indented bullet is ACS but not BQ', '## Q43 — section\n\n- **Q43 — DONE at `4010b96`.**\n', 1, 0), ] STALE_EXPECT = { 'v3a a column-0 `---` rule does not cut the region': 1, 'grep trap: the same words in a code span or a table row are not an opening record': 1, 'a closed item whose opening line never got a pointer is flagged STALE': 1, 'a bold prose record naming another item does not steal the container': 1, } # The unclosed-fence cell pins the DIAGNOSTIC, not just the masking: the value is # the line the fence opened on, which the report must print. UNCLOSED_EXPECT = { 'fence: an unclosed fence fails open — every line graded, the line named': 5, } def fixture_signature(text): res = census(text) sig = ['%s:%s' % (r['item'] or 'NONE', r['state']) for r in res['records']] return sig, len([r for r in res['records'] if r['stale']]), res['fenced_unclosed'] def selftest(verbose=True): failures = [] for name, text, expect in FIXTURES: got, stale, unclosed = fixture_signature(text) ok = got == expect want_stale = STALE_EXPECT.get(name, 0) if stale != want_stale: ok = False want_unclosed = UNCLOSED_EXPECT.get(name) if want_unclosed is not None and unclosed != want_unclosed: ok = False if verbose: print('%s %s' % ('PASS' if ok else 'FAIL', name)) if not ok: print(' expected records %s (stale %d, unclosed %s)\n' ' got records %s (stale %d, unclosed %s)' % (expect, want_stale, want_unclosed, got, stale, unclosed)) if not ok: failures.append(name) # The unit fixtures pin the *population* each matcher counts, so a later # edit to either matcher cannot silently re-open the 72-vs-80 question. n_unit = len(UNIT_FIXTURES) for name, text, want_acs, want_bq in UNIT_FIXTURES: got_acs = sum(1 for l in text.splitlines() if is_closure_line(l)) got_bq = len(legacy_closures(text)) ok = (got_acs, got_bq) == (want_acs, want_bq) if verbose: print('%s %s' % ('PASS' if ok else 'FAIL', name)) if not ok: print(' expected ACS=%d BQ=%d\n got ACS=%d BQ=%d' % (want_acs, want_bq, got_acs, got_bq)) if not ok: failures.append(name) # The pinned census in this file's UNITS section is prose, and prose is not # executed: it was correct at `abe08a9` and had drifted 1 on four figures at # HEAD while every command exited 0. Four cells grade the comparison the # `--census` block prints -- the reader is the live docstring, so mutating a # figure in it is the only way to see the detector go red. n_doc = 0 live = doc_figures() doc_cells = [] # 1. Every figure is readable, and the pin names a revision. doc_cells.append(( 'doc pin 1: the UNITS section states all %d figures and names its revision' % len(DOC_FIGURE_NAMES), live.get('pin') is not None and all(k in live for k in DOC_FIGURE_NAMES), 'pin=%r missing=%r' % (live.get('pin'), [k for k in DOC_FIGURE_NAMES if k not in live]))) # 2. A changed figure is caught, and only that figure. mutated = (__doc__ or '').replace('= **85**', '= **84**') drift = doc_drift(doc_figures(mutated), live) doc_cells.append(( 'doc pin 2: one figure changed in the prose is the one figure reported', [d[0] for d in drift] == ['BQ'] and drift[0][1] == 84 and drift[0][2] == 85, 'drift=%r' % (drift,))) # 3. Positive control: identical dicts report nothing, so "moved: none" is a # reachable answer rather than an artifact of never comparing. doc_cells.append(( 'doc pin 3: an unchanged census reports no drift (positive control)', doc_drift(dict(live), dict(live)) == [], 'drift=%r' % (doc_drift(dict(live), dict(live)),))) # 4. A figure reworded out of reach is reported as unreadable, not dropped. reworded = (__doc__ or '').replace('* 46 BQ-only lines fail', '* forty-six BQ-only lines fail') gone = doc_drift(doc_figures(reworded), live) doc_cells.append(( 'doc pin 4: a reworded figure is reported unreadable, not silently dropped', ('BQ-only/lead', None, 46) in gone, 'drift=%r' % (gone,))) for name, ok, detail in doc_cells: n_doc += 1 if verbose: print('%s %s' % ('PASS' if ok else 'FAIL', name)) if not ok: print(' %s' % detail) if not ok: failures.append(name) # A read failure is rc 2, not a traceback. `--queue` at a missing path or at # bytes that are not UTF-8 escaped main() as FileNotFoundError / # UnicodeDecodeError; the first was already caught, the second was not, and # rc 1 is what this corpus uses for "a check failed". Two cells, because the # exit code is the whole contract an operator sees. n_read = 0 with tempfile.TemporaryDirectory() as td: bad = os.path.join(td, 'not-utf8.md') with open(bad, 'wb') as fh: fh.write(b'\xff\xfe\x80\n') for name, path, want in ( ('read cell 1: a missing queue path exits 2, not a traceback', os.path.join(td, 'nope.md'), 'No such file'), ('read cell 2: a queue that is not UTF-8 exits 2, not a traceback', bad, 'utf-8')): n_read += 1 err = io.StringIO() with contextlib.redirect_stderr(err): rc = main(['--queue', path]) msg = err.getvalue() ok = rc == 2 and 'cannot read queue:' in msg if ok: ok = want.lower() in msg.lower() if verbose: print('%s %s' % ('PASS' if ok else 'FAIL', name)) if not ok: print(' expected rc 2 and `cannot read queue: ...%s...`\n' ' got rc %d, stderr %r' % (want, rc, msg.strip()[:160])) if not ok: failures.append(name) if verbose: total = len(FIXTURES) + n_unit + n_doc + n_read print('\nself-test: %d fixtures (%d claim-check, %d unit, %d doc-pin, %d read-error), ' '%d passed, %d failed' % (total, len(FIXTURES), n_unit, n_doc, n_read, total - len(failures), len(failures))) return failures # -------------------------------------------------------------------------- # main # -------------------------------------------------------------------------- def main(argv=None): ap = argparse.ArgumentParser(description='Claim check for WORK-QUEUE.md') ap.add_argument('--queue', default=None, help='path to WORK-QUEUE.md') ap.add_argument('--json', default=None, help='write the census as JSON here') ap.add_argument('--git', action='store_true', help='resolve closure revisions with git') ap.add_argument('--validate', action='store_true', help='check against the hand-read answers') ap.add_argument('--selftest', action='store_true', help='run the fixture corpus and exit') ap.add_argument('--fail-on-stale', action='store_true', help='exit 2 if any opening record is stale') ap.add_argument('--explain', type=int, nargs='+', metavar='LINE', help='show how the given line numbers resolve, then exit') ap.add_argument('--census', action='store_true', help='print both closure units (ACS and legacy BQ), their overlap ' 'and the axis table, then exit') args = ap.parse_args(argv) if args.selftest: return 1 if selftest() else 0 root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) path = args.queue or os.path.join(root, 'WORK-QUEUE.md') try: text = open(path, encoding='utf-8').read() except (OSError, UnicodeDecodeError) as exc: # UnicodeDecodeError is not an OSError: a binary or mis-encoded file at # --queue used to escape main() as a traceback with rc 1, which is the # code the fixture corpus uses for "a check failed". A read failure is # rc 2 with the same message, whatever the reason. print('cannot read queue: %s' % exc, file=sys.stderr) return 2 res = census(text) git_dir = os.path.dirname(os.path.abspath(path)) if args.explain: lines = text.splitlines() fenced, _b, _u = fence_mask(lines) hidx = container_index(lines, loose=False) lidx = container_index(lines, loose=True) for ln in args.explain: if not 1 <= ln <= len(lines): print(':%d out of range' % ln) continue l = lines[ln - 1] ids = line_ids(l) inh = inherit_id(ln, hidx) inl = inherit_id(ln, lidx) print(':%d container(strict)=%s container(loose)=%s' % ( ln, ('Q%s' % inh) if inh else '(none)', ('Q%s' % inl) if inl else '(none)')) print(' opening=%s closure=%s fenced=%s ids=%s' % (is_opening_record(l), is_closure_line(l), ln in fenced, sorted('Q%d' % i for i in ids) if ids else '(inherit)')) print(' %s' % l.strip()[:160]) return 0 if args.git: for r in res['records']: r['revisions'] = annotate_revisions(r, git_dir) print('queue %s — %d lines, %s bytes' % (path, res['lines'], format(res['bytes'], ','))) print('fenced skipped: %d line(s) in %d block(s) — a fenced block is quoted ' 'material, not a record' % (res['fenced_lines'], res['fenced_blocks'])) if res['fenced_unclosed']: print('FENCE_UNCLOSED at :%d — the fence never closes, so NOTHING is masked: ' 'every line below it is graded, and quoted material may appear as a record ' 'rather than a record disappearing' % res['fenced_unclosed']) print('%-6s %-38s %-7s %-9s %s' % ('line', 'item', 'state', 'closure', 'note')) for r in res['records']: note = [] if r['stale']: note.append('STALE-opening-line') if r['collision']: note.append('COLLISION') if r['inherited_id']: note.append('id-inherited') if r['closure_lines_before']: note.append('closure-before-opening ' + ','.join(':%d' % x for x in r['closure_lines_before'])) if len(r['closure_lines_after']) > 1: note.append('closures ' + ','.join(':%d' % x for x in r['closure_lines_after'])) if args.git and r.get('revisions'): note.append('revs ' + ','.join( '%s:%s' % (v['token'], 'ok' if v['resolves'] else 'NOT-A-GIT-OBJECT') for v in r['revisions'])) print('%-6d %-38s %-7s %-9s %s' % ( r['line'], (r['item'] or '(unnumbered)')[:38], r['state'], str(r['closure_line'] or '-'), ' '.join(note))) closed = [r for r in res['records'] if r['state'] == 'CLOSED'] opened = [r for r in res['records'] if r['state'] == 'OPEN'] stale = [r for r in res['records'] if r['stale']] print('\n%d opening records; %d closed; %d open. %d anchored closure statements ' '(unit ACS — a record line leading `**Q`/`**Status:` with a closure word ' 'in the first %d chars, exclusions tested at the lead; %d closure statements ' 'name no item and are ignored, not guessed at).' % (len(res['records']), len(closed), len(opened), res['closure_candidates'], CLOSURE_WINDOW, len(res['closure_unanchored']))) print('open set: %s' % ', '.join( '%s:%d' % (r['item'] or '(unnumbered)', r['line']) for r in opened)) # Three different numbers have been printed as if they were one: the raw # occurrences of the literal (which include code spans and table rows), the # lines it stands on once code spans are stripped, and how many of those are # opening records. Name each, or the count is cited with the wrong unit. _lines = text.splitlines() _lit_stripped = [n for n, l in enumerate(_lines, 1) if STALE_LITERAL.search(strip_code(l))] _lit_opening = [r['line'] for r in res['records'] if STALE_LITERAL.search(strip_code(_lines[r['line'] - 1]))] print('the literal `Status: open and unclaimed` occurs %d times in the raw text; ' 'after code spans are stripped it stands on %d lines, %d of them ' 'opening-record lines, and %d of those are stale (state CLOSED).' % (res['literal_anywhere'], len(_lit_stripped), len(_lit_opening), len(stale))) if args.census: lines = text.splitlines() fenced, _blocks, _unclosed = fence_mask(lines) acs = {n for n, l in enumerate(lines, 1) if n not in fenced and is_closure_line(l)} bq = set(legacy_closures(text)) print('\n--- closure-unit census (cite a number only with its unit) ---') print('ACS anchored closure statement (this file) = %d' % len(acs)) print(' record line leading `**Q`/`**Status:`, closure word in the first') print(' %d chars, code spans stripped, exclusions tested at the lead,' % CLOSURE_WINDOW) print(' anchored to its own id or the nearest preceding container; fenced') print(' lines excluded (%d set aside).' % len(fenced)) print('BQ column-0 bold closure line naming a Q id (legacy q66_census6.py:54-62) = %d' % len(bq)) print(' line starting at column 0 with `**`, a Q token anywhere, a') print(' closure word anywhere and case-insensitively, no legacy exclusion phrase.') print(' Transcribed byte-for-byte and NOT fenced: %d of its lines stand inside a ' 'fence here, a delta printed and not applied.' % len(bq & fenced)) print('overlap A&B = %d ; ACS-only = %d ; BQ-only = %d ; symmetric difference = %d' % (len(acs & bq), len(acs - bq), len(bq - acs), len(acs ^ bq))) print('NOT a unit conversion: the difference between the two headline numbers is') print('not the delta that shows here, and neither may be cited without its unit.') for label, only, axis in (('BQ-only lines, dropped by ACS', bq - acs, _acs_reject_axis), ('ACS-only lines, dropped by BQ', acs - bq, _bq_reject_axis)): counts = {} for n in sorted(only): a = axis(lines[n - 1]) counts[a] = counts.get(a, 0) + 1 print('%s (%d):' % (label, len(only))) for k in sorted(counts): print(' %-78s %d' % (k, counts[k])) # The prose above states this census for one revision. Print what it # says beside what this queue measures, so a figure that has moved is # named on the run instead of waiting for a reader to notice. stated = doc_figures() here = computed_figures(lines, acs, bq) moved = doc_drift(stated, here) print('\n--- docstring pin (prose figures vs the queue just read) ---') print('pinned at `%s`; %d figure(s) read from the UNITS section, %d compared' % (stated.get('pin', 'UNREADABLE'), len(stated) - (1 if 'pin' in stated else 0), len(DOC_FIGURE_NAMES))) if moved: print('moved: ' + ', '.join( '%s %s -> %d' % (n, 'unreadable' if h is None else h, w) for n, h, w in moved)) print('the prose is a claim about that revision; these are the figures this ' 'queue prints now') else: print('moved: none -- every pinned figure still holds on this queue') return 0 rc = 0 if args.validate: bad = [] for r in res['records']: exp = HAND_READ.get(r['line']) if exp and exp != r['state']: bad.append((r['line'], exp, r['state'])) n = sum(1 for r in res['records'] if r['line'] in HAND_READ) print('hand-read validation: %d/%d match' % (n - len(bad), n)) for b in bad: print(' MISMATCH :%d hand=%s tool=%s' % b) if bad: rc = 1 if args.json: with open(args.json, 'w', encoding='utf-8') as fh: json.dump(res, fh, indent=2, sort_keys=True) print('wrote %s' % args.json) if args.fail_on_stale and stale: rc = 2 return rc if __name__ == '__main__': sys.exit(main())