volt
75168620fe
The queue's other three text parsers already refuse to read structure out of a
fence (queue_index.py:207/:266, checkin_check.py:251, citation_resolve_check.py
after efe7ec6). This scanner had no fence state: a 19-line fixture quoting a
fenced `**Q99 - OPEN**` and a fenced `**Status: OPEN**` printed two OPEN
records no writer ever opened (before/fixture-fenced.txt).
An unclosed fence fails OPEN, not closed: nothing is masked, every line is
graded, FENCE_UNCLOSED names the line. Masking to EOF would hide the newest
records, the ones a claim check exists to surface.
Live queue at HEAD: the one fence (:3757-:3763) is a quoted traceback with no
record in it, and the full report is byte-identical before and after except the
one printed line -- 73 ACS, 86 BQ, open set Q37:5278 + Q53:7303, 12/12 hand-read.
898 lines
41 KiB
Python
898 lines
41 KiB
Python
#!/usr/bin/env python3
|
||
"""queue_claim_check.py — the claim check for `WORK-QUEUE.md`, shipped with its tests.
|
||
|
||
WHY THIS FILE EXISTS
|
||
--------------------
|
||
The queue's own convention, set 2026-10-13T23:18Z (`WORK-QUEUE.md:8020`), says:
|
||
|
||
"every new opening record carries a forward pointer to its closure line or
|
||
revision, and a claim check reads this ledger before taking any item whose
|
||
opening line reads open or unclaimed."
|
||
|
||
Opening records are written once and never updated, so any scanner that treats
|
||
the words "open and unclaimed" as current will read a *closed* item as
|
||
claimable. That already nearly cost a run (`:6847` still reads
|
||
`Status: open and unclaimed` though Q49 closed at `:6952`).
|
||
|
||
Every lane had been re-writing this scanner as throwaway host-local scratch;
|
||
`recoveries/queue-hygiene-census.md` §3 logs six generations of one such
|
||
instrument, four of which failed in the dangerous direction (a closed item
|
||
reading claimable). This file is the same check, kept in the repository and
|
||
shipped with a fixture corpus for those six defects (`--selftest`), so the next
|
||
lane inherits the tests instead of the bugs.
|
||
|
||
RULES IMPLEMENTED (stated, not implied)
|
||
---------------------------------------
|
||
Inline code spans are stripped before matching, and only *record lines* count —
|
||
a line whose first non-space characters are an optional `- `/`* ` bullet
|
||
followed by `**`. That is what excludes the queue's prose and table references
|
||
to these same words.
|
||
|
||
A FENCED BLOCK IS QUOTED MATERIAL, NOT A RECORD (2026-10-19). `queue_index.py`
|
||
and `checkin_check.py` already refuse to read structure out of a fence, on the
|
||
rule that a quoted heading is not a heading, and `citation_resolve_check.py`
|
||
applies the same rule to citations. This file had no fence state, so a tick that
|
||
quoted an example record inside a ``` block opened a phantom item: measured on a
|
||
19-line fixture carrying one fenced `**Q99 — OPEN …**` and one fenced
|
||
`**Status: OPEN …**`, the census printed two OPEN records no writer ever opened
|
||
(`before/fence-fixture.txt` in the tick's artifact). The live queue at HEAD has
|
||
no record inside a fence — 0 of its fenced lines lead `**` — so the rule moves no
|
||
verdict there; it is enforced against the next one. What is set aside is printed
|
||
on every run, never silently dropped, and an unclosed fence fails OPEN — nothing
|
||
is masked, every line is graded, and `FENCE_UNCLOSED` names the line — because
|
||
masking to EOF would hide the newest records, and those are the ones a claim
|
||
check exists to surface. The legacy BQ unit is a transcription of another tool
|
||
and is left byte-for-byte; its fence delta is printed beside it, not applied.
|
||
|
||
OPENING RECORD — a record line carrying either
|
||
(a) `Status:` followed by `open`/`OPEN`, or
|
||
(b) an item id `Q<n>` followed within 60 characters by an `—`/`–`/`--`
|
||
separator and then `OPEN`/`open`/`Opened`/`accepted`.
|
||
Its item id is the leading `**Q<n>` of the line if present; otherwise the id is
|
||
inherited from the nearest preceding `## Q<n>` header.
|
||
|
||
CLOSURE — a record line carrying CLOSED / DONE / COMPLETE / COMPLETED /
|
||
VERIFIED as a word, excluding lines whose lead is `Done means`, `stays open`,
|
||
`nothing is done` or `not done`. It is anchored to an item by (in order): its
|
||
own leading `**Q<n>`; else any `Q<n>` token on the line; else the nearest
|
||
preceding `## Q<n>` header. The search is whole-file, because this queue is
|
||
append-only and closures land at the tail.
|
||
|
||
A closure line that anchors an opening record's item => that item is CLOSED.
|
||
An opening record with no anchored closure => OPEN. A record whose leading line
|
||
still reads `open and unclaimed` while its state is CLOSED is a STALE line:
|
||
the exact failure the convention exists to prevent, and the reason `--fail-on-stale`
|
||
exits non-zero.
|
||
|
||
- **The unit.** `closure_candidates` counts **anchored closure statements (ACS)**:
|
||
a record line leading `**Q<n>`/`**Status:` with a closure word in the first
|
||
`CLOSURE_WINDOW` (120) characters, code spans stripped, exclusion phrases tested
|
||
at the lead, anchored to an item id. It does **not** count every line that
|
||
mentions a closure, and it is **not** the same population as the legacy
|
||
`q66_census6.py` figure. See UNITS below before citing a number.
|
||
|
||
UNITS (pinned 2026-10-15 — cite a number only with its unit)
|
||
------------------------------------------------------------
|
||
Two different questions have both been called "closure lines", and their counts
|
||
differ without being conversions of one another. Measured twice, at the revision
|
||
Run 37 read (`29a4872`, queue 1,170,483 B / sha256 `fbe51445a89f81ab…`) and at
|
||
`28ca8e5` (1,193,681 B / `c3b4a290dc64bd24…`) — identical on both:
|
||
|
||
* **ACS — anchored closure statement** (this file's `closure_candidates`) = **73**.
|
||
* **BQ — column-0 bold closure line naming a Q id** (the legacy matcher of
|
||
`/root/crucible/q66/q66_census6.py:54-62`, host-local, Run 37) = **85**.
|
||
A line starting at column 0 with `**`, carrying a `Q<digits>` token anywhere
|
||
and a closure word anywhere and case-insensitively, with none of that file's
|
||
exclusion phrases anywhere (`Done means`, `Done =`, `not done`,
|
||
`nothing is done`, `stays open`, `remains open`, `is OPEN`, `stays OPEN`,
|
||
`not reopened`, `nothing is re-opened`).
|
||
|
||
Headline figures re-measured at HEAD abe08a9 (WORK-QUEUE.md 1,229,541 B /
|
||
9,953 lines, `python3 tools/queue_claim_check.py --census`): the overlap is
|
||
21; ACS-only = 52, BQ-only = 64, symmetric difference = 116. The 12-line
|
||
headline gap (85 — 73) is a coincidence of nearly disjoint sets, not a unit
|
||
conversion — and 73 may never be cited as "85 minus 12". The five axes,
|
||
counted:
|
||
|
||
* 46 BQ-only lines fail ACS's record-lead rule (an addendum / status /
|
||
verification **prose** line, which BQ takes because it is column-0 bold);
|
||
* 39 ACS-only lines carry no `Q<digits>` token (ACS inherits the container id;
|
||
BQ requires the token on the line, which is why it drops `**Status: DONE …` rows);
|
||
* 18 BQ-only lines carry a **lowercase** closure word (ACS is case-sensitive);
|
||
* 12 ACS-only lines are indented or bulleted (BQ requires column 0);
|
||
* 1 ACS-only line carries a BQ exclusion phrase (`:4035`, "stays open") that ACS
|
||
tests only at the lead.
|
||
|
||
`--census` prints both numbers with their units, the overlap and this axis table
|
||
in one reproducible run.
|
||
|
||
KNOWN BLIND SPOTS (measured, not hand-waved)
|
||
--------------------------------------------
|
||
- Item ids are labels, not keys: this file's own record carries a collision
|
||
(career-lane apply queue `:7292` also reads `Q53`; the queue's own reply 62
|
||
notes it). Binding is by **section**: within a duplicated id, a closure binds
|
||
to the last record of that id opened at or before the closure, so neither
|
||
namesake can close the other. Until 2026-10-19 the tool took the id alone as
|
||
sufficient, graded `:7303` CLOSED on Mercury's `:9869`, and printed
|
||
`open set: Q37:5278` while the ledger's own blocks asserted `:7303` open; the
|
||
row still reads `COLLISION` because the labelling defect is the ledger's.
|
||
- The state is only as good as the closure line's anchoring. Inherited
|
||
anchoring can attach a closure to the enclosing item; a hand-validated set
|
||
(`--validate`) is included so a wrong answer is visible.
|
||
- Nothing here judges whether a closure is *earned*, only that it exists.
|
||
|
||
USAGE
|
||
-----
|
||
python3 tools/queue_claim_check.py # census of ./WORK-QUEUE.md
|
||
python3 tools/queue_claim_check.py --queue PATH --json OUT.json
|
||
python3 tools/queue_claim_check.py --validate # + the hand-read answers
|
||
python3 tools/queue_claim_check.py --git # resolve closure revisions
|
||
python3 tools/queue_claim_check.py --selftest # 20 fixtures, exit 0/1
|
||
python3 tools/queue_claim_check.py --census # both closure units + the axis table
|
||
|
||
Exit codes: 0 clean · 1 self-test failure · 2 stale opening line (--fail-on-stale)
|
||
3 queue unreadable. No file is written unless --json names one.
|
||
"""
|
||
|
||
import argparse
|
||
import contextlib
|
||
import io
|
||
import json
|
||
import os
|
||
import re
|
||
import subprocess
|
||
import sys
|
||
import tempfile
|
||
|
||
# --------------------------------------------------------------------------
|
||
# matching primitives
|
||
# --------------------------------------------------------------------------
|
||
|
||
CODE_SPAN = re.compile(r'`[^`]*`')
|
||
RECORD_PREFIX = re.compile(r'^[ \t]*(?:[-*]\s+)?\*\*')
|
||
ITEM_LEAD = re.compile(r'^[ \t]*(?:[-*]\s+)?\*\*Q(\d+)\b')
|
||
HEADER_ITEM = re.compile(r'^#{1,3}\s+Q(\d+)\b')
|
||
Q_TOKEN = re.compile(r'\bQ(\d+)\b')
|
||
|
||
OPEN_STATUS = re.compile(r'Status:\s*\**\s*(?:open|OPEN)\b')
|
||
OPEN_Q = re.compile(
|
||
r'Q\d+\b[^*\n]{0,60}?(?:\u2014|\u2013|--)\s*\**\s*(?:OPEN|open|Opened|accepted)\b')
|
||
|
||
CLOSURE_WORD = re.compile(r'\b(CLOSED|DONE|COMPLETE|COMPLETED|VERIFIED)\b')
|
||
CLOSURE_EXCLUDE = re.compile(r'^\s*(?:[-*]\s+)?\*\*\s*(?:Done means|stays open|nothing is done|not done)\b',
|
||
re.I)
|
||
CLOSURE_EXCLUDE_ANY = re.compile(r'stays open|nothing is done', re.I)
|
||
|
||
CONTAINER_HEAD = re.compile(r'^\s*(?:[-*]\s+)?\*\*Q(\d+)\b')
|
||
CONTAINER_TITLE_MAX = 100
|
||
CONTAINER_SEP = re.compile(r'(?:\u2014|\u2013|--|:)')
|
||
|
||
# A closure must be a *statement*: a record line whose lead is an item id or
|
||
# `Status:`, with the closure word inside the statement window. That is what
|
||
# refuses the queue's prose lines that mention a closure in passing
|
||
# (`:6631` "- **No item state changed this tick:** ... no DONE line", `:8430`
|
||
# "- **No item closed, no new item opened.** ... remains DONE at 32ad55b").
|
||
CLOSURE_LEAD = re.compile(r'^\s*(?:[-*]\s+)?\*\*(?:Q\d+\b|Status:)')
|
||
CLOSURE_WINDOW = 120
|
||
|
||
STALE_LITERAL = re.compile(r'Status:\s*\**\s*open and unclaimed', re.I)
|
||
SHA7 = re.compile(r'`([0-9a-f]{7,63})[0-9a-f\u2026]*`')
|
||
|
||
# Hand-read ground truth: twelve answers read off the queue by hand before any
|
||
# instrument runs (queue-hygiene-census.md §2). Two independent lanes now.
|
||
HAND_READ = {
|
||
# Read by hand off WORK-QUEUE.md at `7fa6261` (2026-10-19). :8530 was 'OPEN'
|
||
# here until this revision; it closed legitimately at :9869 on 2026-10-16 and
|
||
# the hand-read answer had gone stale, which is what `--validate` exists to
|
||
# show. :7303 is the collided sibling and is OPEN in its own words.
|
||
2264: 'CLOSED', 2684: 'CLOSED', 2898: 'CLOSED', 6348: 'CLOSED',
|
||
6419: 'CLOSED', 6542: 'CLOSED', 6847: 'CLOSED', 6990: 'CLOSED',
|
||
7106: 'CLOSED', 7167: 'CLOSED', 7303: 'OPEN', 8530: 'CLOSED',
|
||
}
|
||
|
||
|
||
def strip_code(line):
|
||
return CODE_SPAN.sub('', line)
|
||
|
||
|
||
# A FENCED BLOCK IS QUOTED MATERIAL, NOT A RECORD (2026-10-19). `queue_index.py`
|
||
# (:207, :266) and `checkin_check.py` (:251) already refuse to read structure out
|
||
# of a fence; this scanner had no fence state, so a quoted example record opened a
|
||
# phantom item. Line numbers stay absolute -- the mask is a set of 1-based line
|
||
# numbers, so every citation into the queue keeps its meaning.
|
||
FENCE = re.compile(r'^\s*(?:`{3,}|~{3,})')
|
||
|
||
|
||
def fence_mask(lines):
|
||
"""(fenced 1-based line numbers, block count, opening line if never closed).
|
||
|
||
Markdown's rule is that an unclosed fence runs to the end of the document.
|
||
For a CLAIM CHECK that rule is the dangerous direction -- masked text cannot
|
||
open a record, so a stray fence would hide the newest items, which are
|
||
exactly the ones a claim check exists to surface. So an unclosed fence fails
|
||
OPEN: nothing is masked, every line is graded, and the opening line is
|
||
returned and printed as FENCE_UNCLOSED. Quoted material graded as a record is
|
||
a visible row (the operator sees the diagnostic and the row); a record
|
||
silently masked is an item nobody sees at all.
|
||
"""
|
||
out = set()
|
||
in_fence = False
|
||
opener = None
|
||
opened_at = None
|
||
blocks = 0
|
||
for n, ln in enumerate(lines, 1):
|
||
m = FENCE.match(ln)
|
||
if m:
|
||
mark = m.group(0).strip()[0]
|
||
if not in_fence:
|
||
in_fence, opener, opened_at = True, mark, n
|
||
blocks += 1
|
||
out.add(n)
|
||
continue
|
||
if mark == opener:
|
||
in_fence, opener = False, None
|
||
out.add(n)
|
||
continue
|
||
if in_fence:
|
||
out.add(n)
|
||
if in_fence: # malformed document: grade everything
|
||
return set(), blocks, opened_at
|
||
return out, blocks, None
|
||
|
||
|
||
def is_record_line(line):
|
||
return bool(RECORD_PREFIX.match(line))
|
||
|
||
|
||
def is_opening_record(line):
|
||
s = strip_code(line)
|
||
if not is_record_line(s):
|
||
return False
|
||
return bool(OPEN_STATUS.search(s) or OPEN_Q.search(s))
|
||
|
||
|
||
def is_closure_line(line):
|
||
s = strip_code(line)
|
||
if not CLOSURE_LEAD.match(s):
|
||
return False
|
||
if CLOSURE_EXCLUDE.match(s):
|
||
return False
|
||
return bool(CLOSURE_WORD.search(s[:CLOSURE_WINDOW]))
|
||
|
||
|
||
def lead_item(line):
|
||
m = ITEM_LEAD.match(line)
|
||
return int(m.group(1)) if m else None
|
||
|
||
|
||
def line_ids(line):
|
||
"""Every id a closure line can anchor to: its lead id, else its Q-tokens."""
|
||
lead = lead_item(line)
|
||
if lead is not None:
|
||
return {lead}
|
||
return {int(x) for x in Q_TOKEN.findall(strip_code(line))}
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# the legacy unit (BQ), transcribed from /root/crucible/q66/q66_census6.py:41-62
|
||
# --------------------------------------------------------------------------
|
||
# Kept here, not for its answer but so that the OTHER number in circulation can
|
||
# be reproduced from one command and printed beside this file's own unit. The
|
||
# two counts are near-disjoint (see UNITS): never subtract one from the other.
|
||
|
||
LEGACY_CLOSE_WORD = re.compile(r'\b(CLOSED|DONE|COMPLETE|VERIFIED)\b', re.I)
|
||
LEGACY_NOT_CLOSURE = re.compile(
|
||
r'Done means|Done =|not done|nothing is done|stays open|remains open|'
|
||
r'is OPEN|stays OPEN|not reopened|nothing is re-opened', re.I)
|
||
|
||
|
||
def legacy_closures(text):
|
||
"""Line numbers the BQ unit counts: column-0 `**`, a Q<digits> token, a
|
||
closure word anywhere (case-insensitive), no legacy exclusion phrase."""
|
||
out = []
|
||
for i, ln in enumerate(text.split('\n'), 1):
|
||
if not ln.startswith('**') or LEGACY_NOT_CLOSURE.search(ln):
|
||
continue
|
||
if re.search(r'\bQ(\d+)\b', ln) and LEGACY_CLOSE_WORD.search(ln):
|
||
out.append(i)
|
||
return out
|
||
|
||
|
||
def _acs_reject_axis(line):
|
||
"""Why the ACS unit drops a line the BQ unit counted."""
|
||
s = strip_code(line)
|
||
if not CLOSURE_LEAD.match(s):
|
||
return 'lead: not a `**Q<n>`/`**Status:` record line (BQ takes any column-0 `**`)'
|
||
if CLOSURE_EXCLUDE.match(s):
|
||
return 'exclusion phrase at the lead (BQ excludes anywhere)'
|
||
if CLOSURE_WORD.search(s[:CLOSURE_WINDOW]):
|
||
return 'other'
|
||
if CLOSURE_WORD.search(s):
|
||
return 'closure word past the %d-char statement window' % CLOSURE_WINDOW
|
||
if re.search(r'\b(CLOSED|DONE|COMPLETE|VERIFIED)\b', s, re.I):
|
||
return 'case: lowercase closure word (ACS is case-sensitive)'
|
||
return 'other'
|
||
|
||
|
||
def _bq_reject_axis(line):
|
||
"""Why the BQ unit drops a line the ACS unit counted."""
|
||
if not line.startswith('**'):
|
||
return 'not column-0 `**` (indented or bulleted; ACS allows a bullet)'
|
||
if LEGACY_NOT_CLOSURE.search(line):
|
||
return 'BQ exclusion phrase anywhere on the line'
|
||
if not re.search(r'\bQ(\d+)\b', line):
|
||
return 'no Q<digits> token on the line (ACS inherits the container id)'
|
||
if not LEGACY_CLOSE_WORD.search(line):
|
||
return 'BQ closure word missing'
|
||
return 'other'
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# the census
|
||
# --------------------------------------------------------------------------
|
||
|
||
def item_title_container(line):
|
||
"""The id of a `**Q<n> — title**` container line, or None.
|
||
|
||
A container is a *title statement*: bold, leading item id, a separator
|
||
inside the bold span, and the bold span closing within CONTAINER_TITLE_MAX
|
||
characters. Measured reasons, both from the live queue: `:5597`
|
||
(`- **Q41 — the non-magazine excerpt class *and* placement** *(archive
|
||
verification; ...`) is Q41's container and must count, while `:4457`
|
||
(`**Q32 record precision — two of the thirteen new assertions cannot fail,
|
||
...`) is a prose record that must not — counting it stole Q33's own closure
|
||
lines (`:4505`) for Q32.
|
||
"""
|
||
m = CONTAINER_HEAD.match(line)
|
||
if not m:
|
||
return None
|
||
rest = line[m.end():]
|
||
close = rest.find('**')
|
||
if close == -1 or close > CONTAINER_TITLE_MAX:
|
||
return None
|
||
if not CONTAINER_SEP.search(rest[:close]):
|
||
return None
|
||
return int(m.group(1))
|
||
|
||
|
||
def container_index(lines, loose=False):
|
||
"""[(line, id_or_None)] for every *container* line, in file order.
|
||
|
||
A container is a header of any level, or a bold record line that leads with
|
||
an item id. Two strictnesses, because the two questions differ:
|
||
|
||
* `loose=True` — used to name an opening record that carries no id of its
|
||
own. Any bold line leading with `Q<n>` counts; the item whose block the
|
||
record sits in is the nearest such marker above it.
|
||
* `loose=False` — used to anchor a closure. Here the marker must be a
|
||
*title statement* (see `item_title_container`), because a long bold prose
|
||
record that merely starts with an id steals the anchor: `:4457`
|
||
(`**Q32 record precision — two of the thirteen new assertions cannot
|
||
fail, ...`) otherwise took `:4505`, Q33's own closure, for Q32.
|
||
"""
|
||
out = []
|
||
for n, l in enumerate(lines, 1):
|
||
if l.startswith('#'):
|
||
m = HEADER_ITEM.match(l)
|
||
out.append((n, int(m.group(1)) if m else None))
|
||
elif loose and CONTAINER_HEAD.match(l):
|
||
m = ITEM_LEAD.match(l)
|
||
out.append((n, int(m.group(1)) if m else None))
|
||
else:
|
||
cid = item_title_container(l)
|
||
if cid is not None:
|
||
out.append((n, cid))
|
||
return out
|
||
|
||
|
||
def inherit_id(n, hidx):
|
||
"""Id of the nearest preceding header, or None."""
|
||
best = None
|
||
for ln, qid in hidx:
|
||
if ln < n:
|
||
best = qid
|
||
else:
|
||
break
|
||
return best
|
||
|
||
|
||
def census(text):
|
||
lines = text.splitlines()
|
||
fenced, n_blocks, unclosed = fence_mask(lines)
|
||
hidx = container_index(lines, loose=False)
|
||
lidx = container_index(lines, loose=True)
|
||
|
||
closures = [] # (line, idset, text)
|
||
unanchored = [] # closure statements that name no item
|
||
for n, l in enumerate(lines, 1):
|
||
if n in fenced: # quoted material, not a statement
|
||
continue
|
||
if not is_closure_line(l):
|
||
continue
|
||
ids = line_ids(l)
|
||
if not ids:
|
||
inh = inherit_id(n, hidx)
|
||
if inh is None:
|
||
unanchored.append(n)
|
||
continue
|
||
ids = {inh}
|
||
closures.append((n, ids, l))
|
||
|
||
records = []
|
||
for n, l in enumerate(lines, 1):
|
||
if n in fenced: # quoted material, not a record
|
||
continue
|
||
if not is_opening_record(l):
|
||
continue
|
||
lead = lead_item(l)
|
||
inherited = False
|
||
if lead is None:
|
||
lead = inherit_id(n, lidx)
|
||
inherited = True
|
||
anchored = [(cn, cl) for (cn, ids, cl) in closures if lead is not None and lead in ids]
|
||
# State is decided by a closure recorded at or after the opening record.
|
||
# Earlier closures are reported as evidence but never set the state: the
|
||
# queue's record order is not the work's order (Q25's DONE line sits
|
||
# 96 lines *above* its own opening record).
|
||
closing = [c for c in anchored if c[0] >= n]
|
||
state = 'CLOSED' if closing else 'OPEN'
|
||
stale = bool(STALE_LITERAL.search(strip_code(l))) and state == 'CLOSED'
|
||
records.append({
|
||
'line': n,
|
||
'item': None if lead is None else 'Q%d' % lead,
|
||
'inherited_id': inherited,
|
||
'state': state,
|
||
'closure_line': closing[0][0] if closing else None,
|
||
'closure_lines_after': [c[0] for c in closing],
|
||
'closure_lines_before': [c[0] for c in anchored if c[0] < n],
|
||
'closure_text': closing[0][1].strip() if closing else None,
|
||
'stale': stale,
|
||
'text': l.strip(),
|
||
})
|
||
|
||
# collisions: two opening records resolving to one id
|
||
seen = {}
|
||
for r in records:
|
||
if r['item'] is None:
|
||
continue
|
||
seen.setdefault(r['item'], []).append(r['line'])
|
||
for r in records:
|
||
r['collision'] = bool(r['item'] and len(seen.get(r['item'], [])) > 1)
|
||
|
||
# A duplicate label cannot bind a closure by id alone. Until 2026-10-19 this
|
||
# file took `lead in ids` as sufficient and let EVERY record carrying the
|
||
# label inherit the same closure -- so one item's DONE closed its namesake.
|
||
# On the live queue that graded the career-lane apply queue (:7303, held for
|
||
# Alexander) CLOSED on Mercury's :9869 and dropped it from `open set:`, while
|
||
# four standing ledger blocks assert it open. A record owns its own section,
|
||
# so a closure binds to the last record of that id opened at or before it --
|
||
# never to a sibling that ended before the closure was written.
|
||
groups = {}
|
||
for r in records:
|
||
if r['item'] is not None:
|
||
groups.setdefault(r['item'], []).append(r)
|
||
for item, recs in groups.items():
|
||
if len(recs) < 2:
|
||
continue
|
||
recs.sort(key=lambda r: r['line'])
|
||
qid = int(item[1:])
|
||
for i, r in enumerate(recs):
|
||
lo = r['line']
|
||
hi = recs[i + 1]['line'] if i + 1 < len(recs) else None
|
||
owned = [(c[0], c[2]) for c in closures
|
||
if qid in c[1] and c[0] >= lo and (hi is None or c[0] < hi)]
|
||
r['closure_line'] = owned[0][0] if owned else None
|
||
r['closure_lines_after'] = [c[0] for c in owned]
|
||
r['closure_text'] = owned[0][1].strip() if owned else None
|
||
r['state'] = 'CLOSED' if owned else 'OPEN'
|
||
r['stale'] = (bool(STALE_LITERAL.search(strip_code(lines[lo - 1])))
|
||
and r['state'] == 'CLOSED')
|
||
|
||
raw_literal = len(re.findall(r'Status:\s*\**\s*open and unclaimed', text, re.I))
|
||
return {
|
||
'lines': len(lines),
|
||
'bytes': len(text.encode()),
|
||
'records': records,
|
||
'closure_candidates': len(closures),
|
||
'closure_unanchored': unanchored,
|
||
'literal_anywhere': raw_literal,
|
||
'fenced_lines': len(fenced),
|
||
'fenced_blocks': n_blocks,
|
||
'fenced_unclosed': unclosed,
|
||
}
|
||
|
||
|
||
def annotate_revisions(rec, git_dir):
|
||
"""First 7-hex backticked token on the closure line, resolved with git."""
|
||
out = []
|
||
txt = rec.get('closure_text') or ''
|
||
for tok in SHA7.findall(txt):
|
||
try:
|
||
p = subprocess.run(['git', 'cat-file', '-t', tok], cwd=git_dir,
|
||
capture_output=True, text=True, timeout=20)
|
||
kind = p.stdout.strip() or p.stderr.strip()
|
||
ok = p.returncode == 0
|
||
except Exception as exc: # pragma: no cover
|
||
kind, ok = str(exc), False
|
||
out.append({'token': tok, 'resolves': ok, 'kind': kind})
|
||
return out
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# fixtures — the six defects logged in recoveries/queue-hygiene-census.md §3
|
||
# --------------------------------------------------------------------------
|
||
|
||
FIXTURES = [
|
||
('v1 header-walk only: an opening record nested inside another section',
|
||
'## Q10 — outer section\n\nDone means: nothing here.\n\n'
|
||
'- **Q99 — OPEN (nested, not a `##` header).** body\n',
|
||
['Q99:OPEN']),
|
||
|
||
('v2 loose key: a Q7 continuation note is not an opening record',
|
||
'## Q7 — section\n\n- **Q7, measured:** the unclaimed executable set is open beside it.\n',
|
||
[]),
|
||
|
||
('v3a a column-0 `---` rule does not cut the region',
|
||
'## Q11 — section\n\n**Status: open and unclaimed.**\n\n---\n\n'
|
||
'**Status: DONE — closed below the rule.**\n',
|
||
['Q11:CLOSED']),
|
||
|
||
('v3b `Done means:` is not a closure word',
|
||
'## Q12 — section\n\n**Status: open and unclaimed.**\n\n**Done means:** one row per space.\n',
|
||
['Q12:OPEN']),
|
||
|
||
("v4 closures are anchored: another item's DONE does not close this one",
|
||
'## Q13 — section\n\n**Status: open and unclaimed.**\n\n## Q14 — other\n\n'
|
||
'**Q14 — OPEN (lane).**\n\n**Q14 — DONE at 4010b96.**\n',
|
||
['Q13:OPEN', 'Q14:CLOSED']),
|
||
|
||
('v5 whole-file search: a closure landing after a later opening record still counts',
|
||
'## Q15 — section\n\n**Q15 — OPEN (lane).**\n\n## Q16 — other\n\n**Q16 — OPEN (lane).**\n\n'
|
||
'**Q15 — DONE at 4010b96.**\n',
|
||
['Q15:CLOSED', 'Q16:OPEN']),
|
||
|
||
('grep trap: the same words in a code span or a table row are not an opening record',
|
||
'## Q17 — section\n\n**Status: open and unclaimed. Opened by the pipe.**\n\n'
|
||
'**Status: DONE — closed.**\n\n'
|
||
'- (1) Three lines carry the literal `Status: open and unclaimed` (`:1`, `:2`).\n'
|
||
'\n| `Status: open and unclaimed` | `:2264` | Q21 | DONE |\n',
|
||
['Q17:CLOSED']),
|
||
|
||
('unnumbered opening record inherits the enclosing `## Q<n>` header',
|
||
'## Q20 — section\n\n**Status: OPEN — held for Alexander.**\n',
|
||
['Q20:OPEN']),
|
||
|
||
('a closed item whose opening line never got a pointer is flagged STALE',
|
||
'## Q21 — section\n\n**Status: open and unclaimed. Opened by the pipe.**\n\n'
|
||
'**Status: DONE — `recoveries/x.md`, 2026-10-10.**\n',
|
||
['Q21:CLOSED']),
|
||
|
||
('a bold prose record naming another item does not steal the container',
|
||
'## Q30 — section\n\n**Status: open and unclaimed.**\n\n'
|
||
'- **Q31 record precision — two of the thirteen new assertions cannot fail, and two totals are not\n'
|
||
' enumerable as stated**, so the record is corrected in place rather than restated.\n\n'
|
||
'**Status: DONE — closed by the pipe.**\n',
|
||
['Q30:CLOSED']),
|
||
|
||
# The live defect this cell was written for (2026-10-19): the career-lane
|
||
# apply queue (:7303) and Mercury's record (:8530) both read `Q53`, so
|
||
# Mercury's `:9869` DONE closed both and the tool printed `open set: Q37:5278`
|
||
# against four ledger blocks asserting :7303 open. Pre-fix this cell reads
|
||
# ['Q40:CLOSED', 'Q41:OPEN', 'Q40:CLOSED'].
|
||
("collision: a duplicate label does not let one record's closure close the other",
|
||
'## Q40 — first use of the label\n\n**Status: OPEN — held for Alexander.**\n\n'
|
||
'## Q41 — an unrelated section between them\n\n**Q41 — OPEN (lane).**\n\n'
|
||
'## Q40 — second use of the same label\n\n**Status: OPEN — queued.**\n\n'
|
||
'**Q40 — DONE at 4010b96.**\n',
|
||
['Q40:OPEN', 'Q41:OPEN', 'Q40:CLOSED']),
|
||
|
||
# The live defect these three cells were written for (2026-10-19): this file
|
||
# had no fence state, so a tick quoting an example record inside a fenced
|
||
# block opened a phantom item. Pre-fix the first cell reads
|
||
# ['Q60:CLOSED', 'Q99:OPEN', 'Q99:CLOSED'] and the second reads two Q61
|
||
# records (the fenced `**Status:` inherited the header, then collided).
|
||
('fence: a record inside a ``` block is quoted material, not a record',
|
||
'## Q60 — section\n\n**Q60 — OPEN (lane).**\n\n**Q60 — DONE at 4010b96.**\n\n'
|
||
'Prose quoting a tick:\n\n```\n**Q99 — OPEN (pipe).**\n'
|
||
'**Q99 — DONE at 4010b96.**\n```\n',
|
||
['Q60:CLOSED']),
|
||
|
||
('fence: a tilde fence is a fence, and a fenced Status line inherits nothing',
|
||
'## Q61 — section\n\n**Q61 — OPEN (lane).**\n\n~~~\n'
|
||
'**Status: OPEN — queued.**\n~~~\n',
|
||
['Q61:OPEN']),
|
||
|
||
('fence: an unclosed fence fails open — every line graded, the line named',
|
||
'## Q62 — section\n\n**Q62 — OPEN (lane).**\n\n```\n**Q63 — OPEN (lane).**\n',
|
||
['Q62:OPEN', 'Q63:OPEN']),
|
||
]
|
||
|
||
UNIT_FIXTURES = [
|
||
# name, text, ACS count, BQ count — the four axes that separate the units.
|
||
('unit axis 1: a column-0 bold *prose* line is BQ but not ACS',
|
||
'## Q40 — section\n\n**Addendum, 2026-10-08 — Q40 CLOSED at `8bd0b09`.**\n',
|
||
0, 1),
|
||
('unit axis 2: a lowercase closure word is BQ but not ACS',
|
||
'## Q41 — section\n\n**Q41 — verified below.**\n',
|
||
0, 1),
|
||
('unit axis 3: a status row with no Q token is ACS but not BQ',
|
||
'## Q42 — section\n\n**Status: DONE — closed.**\n',
|
||
1, 0),
|
||
('unit axis 4: an indented bullet is ACS but not BQ',
|
||
'## Q43 — section\n\n- **Q43 — DONE at `4010b96`.**\n',
|
||
1, 0),
|
||
]
|
||
|
||
STALE_EXPECT = {
|
||
'v3a a column-0 `---` rule does not cut the region': 1,
|
||
'grep trap: the same words in a code span or a table row are not an opening record': 1,
|
||
'a closed item whose opening line never got a pointer is flagged STALE': 1,
|
||
'a bold prose record naming another item does not steal the container': 1,
|
||
}
|
||
|
||
# The unclosed-fence cell pins the DIAGNOSTIC, not just the masking: the value is
|
||
# the line the fence opened on, which the report must print.
|
||
UNCLOSED_EXPECT = {
|
||
'fence: an unclosed fence fails open — every line graded, the line named': 5,
|
||
}
|
||
|
||
|
||
def fixture_signature(text):
|
||
res = census(text)
|
||
sig = ['%s:%s' % (r['item'] or 'NONE', r['state']) for r in res['records']]
|
||
return sig, len([r for r in res['records'] if r['stale']]), res['fenced_unclosed']
|
||
|
||
|
||
def selftest(verbose=True):
|
||
failures = []
|
||
for name, text, expect in FIXTURES:
|
||
got, stale, unclosed = fixture_signature(text)
|
||
ok = got == expect
|
||
want_stale = STALE_EXPECT.get(name, 0)
|
||
if stale != want_stale:
|
||
ok = False
|
||
want_unclosed = UNCLOSED_EXPECT.get(name)
|
||
if want_unclosed is not None and unclosed != want_unclosed:
|
||
ok = False
|
||
if verbose:
|
||
print('%s %s' % ('PASS' if ok else 'FAIL', name))
|
||
if not ok:
|
||
print(' expected records %s (stale %d, unclosed %s)\n'
|
||
' got records %s (stale %d, unclosed %s)'
|
||
% (expect, want_stale, want_unclosed, got, stale, unclosed))
|
||
if not ok:
|
||
failures.append(name)
|
||
|
||
# The unit fixtures pin the *population* each matcher counts, so a later
|
||
# edit to either matcher cannot silently re-open the 72-vs-80 question.
|
||
n_unit = len(UNIT_FIXTURES)
|
||
for name, text, want_acs, want_bq in UNIT_FIXTURES:
|
||
got_acs = sum(1 for l in text.splitlines() if is_closure_line(l))
|
||
got_bq = len(legacy_closures(text))
|
||
ok = (got_acs, got_bq) == (want_acs, want_bq)
|
||
if verbose:
|
||
print('%s %s' % ('PASS' if ok else 'FAIL', name))
|
||
if not ok:
|
||
print(' expected ACS=%d BQ=%d\n got ACS=%d BQ=%d'
|
||
% (want_acs, want_bq, got_acs, got_bq))
|
||
if not ok:
|
||
failures.append(name)
|
||
|
||
# A read failure is rc 2, not a traceback. `--queue` at a missing path or at
|
||
# bytes that are not UTF-8 escaped main() as FileNotFoundError /
|
||
# UnicodeDecodeError; the first was already caught, the second was not, and
|
||
# rc 1 is what this corpus uses for "a check failed". Two cells, because the
|
||
# exit code is the whole contract an operator sees.
|
||
n_read = 0
|
||
with tempfile.TemporaryDirectory() as td:
|
||
bad = os.path.join(td, 'not-utf8.md')
|
||
with open(bad, 'wb') as fh:
|
||
fh.write(b'\xff\xfe\x80\n')
|
||
for name, path, want in (
|
||
('read cell 1: a missing queue path exits 2, not a traceback',
|
||
os.path.join(td, 'nope.md'), 'No such file'),
|
||
('read cell 2: a queue that is not UTF-8 exits 2, not a traceback',
|
||
bad, 'utf-8')):
|
||
n_read += 1
|
||
err = io.StringIO()
|
||
with contextlib.redirect_stderr(err):
|
||
rc = main(['--queue', path])
|
||
msg = err.getvalue()
|
||
ok = rc == 2 and 'cannot read queue:' in msg
|
||
if ok:
|
||
ok = want.lower() in msg.lower()
|
||
if verbose:
|
||
print('%s %s' % ('PASS' if ok else 'FAIL', name))
|
||
if not ok:
|
||
print(' expected rc 2 and `cannot read queue: ...%s...`\n'
|
||
' got rc %d, stderr %r' % (want, rc, msg.strip()[:160]))
|
||
if not ok:
|
||
failures.append(name)
|
||
|
||
if verbose:
|
||
total = len(FIXTURES) + n_unit + n_read
|
||
print('\nself-test: %d fixtures (%d claim-check, %d unit, %d read-error), '
|
||
'%d passed, %d failed'
|
||
% (total, len(FIXTURES), n_unit, n_read,
|
||
total - len(failures), len(failures)))
|
||
return failures
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# main
|
||
# --------------------------------------------------------------------------
|
||
|
||
def main(argv=None):
|
||
ap = argparse.ArgumentParser(description='Claim check for WORK-QUEUE.md')
|
||
ap.add_argument('--queue', default=None, help='path to WORK-QUEUE.md')
|
||
ap.add_argument('--json', default=None, help='write the census as JSON here')
|
||
ap.add_argument('--git', action='store_true', help='resolve closure revisions with git')
|
||
ap.add_argument('--validate', action='store_true', help='check against the hand-read answers')
|
||
ap.add_argument('--selftest', action='store_true', help='run the fixture corpus and exit')
|
||
ap.add_argument('--fail-on-stale', action='store_true',
|
||
help='exit 2 if any opening record is stale')
|
||
ap.add_argument('--explain', type=int, nargs='+', metavar='LINE',
|
||
help='show how the given line numbers resolve, then exit')
|
||
ap.add_argument('--census', action='store_true',
|
||
help='print both closure units (ACS and legacy BQ), their overlap '
|
||
'and the axis table, then exit')
|
||
args = ap.parse_args(argv)
|
||
|
||
if args.selftest:
|
||
return 1 if selftest() else 0
|
||
|
||
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||
path = args.queue or os.path.join(root, 'WORK-QUEUE.md')
|
||
try:
|
||
text = open(path, encoding='utf-8').read()
|
||
except (OSError, UnicodeDecodeError) as exc:
|
||
# UnicodeDecodeError is not an OSError: a binary or mis-encoded file at
|
||
# --queue used to escape main() as a traceback with rc 1, which is the
|
||
# code the fixture corpus uses for "a check failed". A read failure is
|
||
# rc 2 with the same message, whatever the reason.
|
||
print('cannot read queue: %s' % exc, file=sys.stderr)
|
||
return 2
|
||
|
||
res = census(text)
|
||
git_dir = os.path.dirname(os.path.abspath(path))
|
||
|
||
if args.explain:
|
||
lines = text.splitlines()
|
||
fenced, _b, _u = fence_mask(lines)
|
||
hidx = container_index(lines, loose=False)
|
||
lidx = container_index(lines, loose=True)
|
||
for ln in args.explain:
|
||
if not 1 <= ln <= len(lines):
|
||
print(':%d out of range' % ln)
|
||
continue
|
||
l = lines[ln - 1]
|
||
ids = line_ids(l)
|
||
inh = inherit_id(ln, hidx)
|
||
inl = inherit_id(ln, lidx)
|
||
print(':%d container(strict)=%s container(loose)=%s' % (
|
||
ln, ('Q%s' % inh) if inh else '(none)', ('Q%s' % inl) if inl else '(none)'))
|
||
print(' opening=%s closure=%s fenced=%s ids=%s' %
|
||
(is_opening_record(l), is_closure_line(l), ln in fenced,
|
||
sorted('Q%d' % i for i in ids) if ids else '(inherit)'))
|
||
print(' %s' % l.strip()[:160])
|
||
return 0
|
||
|
||
if args.git:
|
||
for r in res['records']:
|
||
r['revisions'] = annotate_revisions(r, git_dir)
|
||
|
||
print('queue %s — %d lines, %s bytes' % (path, res['lines'], format(res['bytes'], ',')))
|
||
print('fenced skipped: %d line(s) in %d block(s) — a fenced block is quoted '
|
||
'material, not a record' % (res['fenced_lines'], res['fenced_blocks']))
|
||
if res['fenced_unclosed']:
|
||
print('FENCE_UNCLOSED at :%d — the fence never closes, so NOTHING is masked: '
|
||
'every line below it is graded, and quoted material may appear as a record '
|
||
'rather than a record disappearing' % res['fenced_unclosed'])
|
||
print('%-6s %-38s %-7s %-9s %s' % ('line', 'item', 'state', 'closure', 'note'))
|
||
for r in res['records']:
|
||
note = []
|
||
if r['stale']:
|
||
note.append('STALE-opening-line')
|
||
if r['collision']:
|
||
note.append('COLLISION')
|
||
if r['inherited_id']:
|
||
note.append('id-inherited')
|
||
if r['closure_lines_before']:
|
||
note.append('closure-before-opening ' +
|
||
','.join(':%d' % x for x in r['closure_lines_before']))
|
||
if len(r['closure_lines_after']) > 1:
|
||
note.append('closures ' + ','.join(':%d' % x for x in r['closure_lines_after']))
|
||
if args.git and r.get('revisions'):
|
||
note.append('revs ' + ','.join(
|
||
'%s:%s' % (v['token'], 'ok' if v['resolves'] else 'NOT-A-GIT-OBJECT')
|
||
for v in r['revisions']))
|
||
print('%-6d %-38s %-7s %-9s %s' % (
|
||
r['line'],
|
||
(r['item'] or '(unnumbered)')[:38],
|
||
r['state'],
|
||
str(r['closure_line'] or '-'),
|
||
' '.join(note)))
|
||
|
||
closed = [r for r in res['records'] if r['state'] == 'CLOSED']
|
||
opened = [r for r in res['records'] if r['state'] == 'OPEN']
|
||
stale = [r for r in res['records'] if r['stale']]
|
||
print('\n%d opening records; %d closed; %d open. %d anchored closure statements '
|
||
'(unit ACS — a record line leading `**Q<n>`/`**Status:` with a closure word '
|
||
'in the first %d chars, exclusions tested at the lead; %d closure statements '
|
||
'name no item and are ignored, not guessed at).'
|
||
% (len(res['records']), len(closed), len(opened), res['closure_candidates'],
|
||
CLOSURE_WINDOW, len(res['closure_unanchored'])))
|
||
print('open set: %s' % ', '.join(
|
||
'%s:%d' % (r['item'] or '(unnumbered)', r['line']) for r in opened))
|
||
# Three different numbers have been printed as if they were one: the raw
|
||
# occurrences of the literal (which include code spans and table rows), the
|
||
# lines it stands on once code spans are stripped, and how many of those are
|
||
# opening records. Name each, or the count is cited with the wrong unit.
|
||
_lines = text.splitlines()
|
||
_lit_stripped = [n for n, l in enumerate(_lines, 1) if STALE_LITERAL.search(strip_code(l))]
|
||
_lit_opening = [r['line'] for r in res['records']
|
||
if STALE_LITERAL.search(strip_code(_lines[r['line'] - 1]))]
|
||
print('the literal `Status: open and unclaimed` occurs %d times in the raw text; '
|
||
'after code spans are stripped it stands on %d lines, %d of them '
|
||
'opening-record lines, and %d of those are stale (state CLOSED).'
|
||
% (res['literal_anywhere'], len(_lit_stripped), len(_lit_opening), len(stale)))
|
||
|
||
if args.census:
|
||
lines = text.splitlines()
|
||
fenced, _blocks, _unclosed = fence_mask(lines)
|
||
acs = {n for n, l in enumerate(lines, 1) if n not in fenced and is_closure_line(l)}
|
||
bq = set(legacy_closures(text))
|
||
print('\n--- closure-unit census (cite a number only with its unit) ---')
|
||
print('ACS anchored closure statement (this file) = %d' % len(acs))
|
||
print(' record line leading `**Q<n>`/`**Status:`, closure word in the first')
|
||
print(' %d chars, code spans stripped, exclusions tested at the lead,' % CLOSURE_WINDOW)
|
||
print(' anchored to its own id or the nearest preceding container; fenced')
|
||
print(' lines excluded (%d set aside).' % len(fenced))
|
||
print('BQ column-0 bold closure line naming a Q id (legacy q66_census6.py:54-62) = %d'
|
||
% len(bq))
|
||
print(' line starting at column 0 with `**`, a Q<digits> token anywhere, a')
|
||
print(' closure word anywhere and case-insensitively, no legacy exclusion phrase.')
|
||
print(' Transcribed byte-for-byte and NOT fenced: %d of its lines stand inside a '
|
||
'fence here, a delta printed and not applied.' % len(bq & fenced))
|
||
print('overlap A&B = %d ; ACS-only = %d ; BQ-only = %d ; symmetric difference = %d'
|
||
% (len(acs & bq), len(acs - bq), len(bq - acs), len(acs ^ bq)))
|
||
print('NOT a unit conversion: the difference between the two headline numbers is')
|
||
print('not the delta that shows here, and neither may be cited without its unit.')
|
||
for label, only, axis in (('BQ-only lines, dropped by ACS', bq - acs, _acs_reject_axis),
|
||
('ACS-only lines, dropped by BQ', acs - bq, _bq_reject_axis)):
|
||
counts = {}
|
||
for n in sorted(only):
|
||
a = axis(lines[n - 1])
|
||
counts[a] = counts.get(a, 0) + 1
|
||
print('%s (%d):' % (label, len(only)))
|
||
for k in sorted(counts):
|
||
print(' %-78s %d' % (k, counts[k]))
|
||
return 0
|
||
|
||
rc = 0
|
||
if args.validate:
|
||
bad = []
|
||
for r in res['records']:
|
||
exp = HAND_READ.get(r['line'])
|
||
if exp and exp != r['state']:
|
||
bad.append((r['line'], exp, r['state']))
|
||
n = sum(1 for r in res['records'] if r['line'] in HAND_READ)
|
||
print('hand-read validation: %d/%d match' % (n - len(bad), n))
|
||
for b in bad:
|
||
print(' MISMATCH :%d hand=%s tool=%s' % b)
|
||
if bad:
|
||
rc = 1
|
||
|
||
if args.json:
|
||
with open(args.json, 'w', encoding='utf-8') as fh:
|
||
json.dump(res, fh, indent=2, sort_keys=True)
|
||
print('wrote %s' % args.json)
|
||
|
||
if args.fail_on_stale and stale:
|
||
rc = 2
|
||
return rc
|
||
|
||
|
||
if __name__ == '__main__':
|
||
sys.exit(main())
|