#!/usr/bin/env python3 """queue_claim_check.py — the claim check for `WORK-QUEUE.md`, shipped with its tests. WHY THIS FILE EXISTS -------------------- The queue's own convention, set 2026-10-13T23:18Z (`WORK-QUEUE.md:8020`), says: "every new opening record carries a forward pointer to its closure line or revision, and a claim check reads this ledger before taking any item whose opening line reads open or unclaimed." Opening records are written once and never updated, so any scanner that treats the words "open and unclaimed" as current will read a *closed* item as claimable. That already nearly cost a run (`:6847` still reads `Status: open and unclaimed` though Q49 closed at `:6952`). Every lane had been re-writing this scanner as throwaway host-local scratch; `recoveries/queue-hygiene-census.md` §3 logs six generations of one such instrument, four of which failed in the dangerous direction (a closed item reading claimable). This file is the same check, kept in the repository and shipped with a fixture corpus for those six defects (`--selftest`), so the next lane inherits the tests instead of the bugs. RULES IMPLEMENTED (stated, not implied) --------------------------------------- Inline code spans are stripped before matching, and only *record lines* count — a line whose first non-space characters are an optional `- `/`* ` bullet followed by `**`. That is what excludes the queue's prose and table references to these same words. OPENING RECORD — a record line carrying either (a) `Status:` followed by `open`/`OPEN`, or (b) an item id `Q` followed within 60 characters by an `—`/`–`/`--` separator and then `OPEN`/`open`/`Opened`/`accepted`. Its item id is the leading `**Q` of the line if present; otherwise the id is inherited from the nearest preceding `## Q` header. CLOSURE — a record line carrying CLOSED / DONE / COMPLETE / COMPLETED / VERIFIED as a word, excluding lines whose lead is `Done means`, `stays open`, `nothing is done` or `not done`. It is anchored to an item by (in order): its own leading `**Q`; else any `Q` token on the line; else the nearest preceding `## Q` header. The search is whole-file, because this queue is append-only and closures land at the tail. A closure line that anchors an opening record's item => that item is CLOSED. An opening record with no anchored closure => OPEN. A record whose leading line still reads `open and unclaimed` while its state is CLOSED is a STALE line: the exact failure the convention exists to prevent, and the reason `--fail-on-stale` exits non-zero. - **The unit.** `closure_candidates` counts **anchored closure statements (ACS)**: a record line leading `**Q`/`**Status:` with a closure word in the first `CLOSURE_WINDOW` (120) characters, code spans stripped, exclusion phrases tested at the lead, anchored to an item id. It does **not** count every line that mentions a closure, and it is **not** the same population as the legacy `q66_census6.py` figure. See UNITS below before citing a number. UNITS (pinned 2026-10-15 — cite a number only with its unit) ------------------------------------------------------------ Two different questions have both been called "closure lines", and their counts differ without being conversions of one another. Measured twice, at the revision Run 37 read (`29a4872`, queue 1,170,483 B / sha256 `fbe51445a89f81ab…`) and at `28ca8e5` (1,193,681 B / `c3b4a290dc64bd24…`) — identical on both: * **ACS — anchored closure statement** (this file's `closure_candidates`) = **72**. * **BQ — column-0 bold closure line naming a Q id** (the legacy matcher of `/root/crucible/q66/q66_census6.py:54-62`, host-local, Run 37) = **80**. A line starting at column 0 with `**`, carrying a `Q` token anywhere and a closure word anywhere and case-insensitively, with none of that file's exclusion phrases anywhere (`Done means`, `Done =`, `not done`, `nothing is done`, `stays open`, `remains open`, `is OPEN`, `stays OPEN`, `not reopened`, `nothing is re-opened`). **The two counts are not two readings of one population: the overlap is 20.** ACS-only = 52, BQ-only = 60, symmetric difference = 112. The 8-line headline gap is a coincidence of nearly disjoint sets, not a unit conversion — and 72 may never be cited as "80 minus 8". The five axes, counted: * 42 BQ-only lines fail ACS's record-lead rule (an addendum / status / verification **prose** line, which BQ takes because it is column-0 bold); * 39 ACS-only lines carry no `Q` token (ACS inherits the container id; BQ requires the token on the line, which is why it drops `**Status: DONE …` rows); * 18 BQ-only lines carry a **lowercase** closure word (ACS is case-sensitive); * 12 ACS-only lines are indented or bulleted (BQ requires column 0); * 1 ACS-only line carries a BQ exclusion phrase (`:4035`, "stays open") that ACS tests only at the lead. `--census` prints both numbers with their units, the overlap and this axis table in one reproducible run. KNOWN BLIND SPOTS (measured, not hand-waved) -------------------------------------------- - Item ids are labels, not keys: this file's own record carries a collision (career-lane apply queue `:7292` also reads `Q53`; the queue's own reply 62 notes it), which the tool reports as `COLLISION` rather than resolving. - The state is only as good as the closure line's anchoring. Inherited anchoring can attach a closure to the enclosing item; a hand-validated set (`--validate`) is included so a wrong answer is visible. - Nothing here judges whether a closure is *earned*, only that it exists. USAGE ----- python3 tools/queue_claim_check.py # census of ./WORK-QUEUE.md python3 tools/queue_claim_check.py --queue PATH --json OUT.json python3 tools/queue_claim_check.py --validate # + the hand-read answers python3 tools/queue_claim_check.py --git # resolve closure revisions python3 tools/queue_claim_check.py --selftest # 16 fixtures, exit 0/1 python3 tools/queue_claim_check.py --census # both closure units + the axis table Exit codes: 0 clean · 1 self-test failure · 2 stale opening line (--fail-on-stale) 3 queue unreadable. No file is written unless --json names one. """ import argparse import contextlib import io import json import os import re import subprocess import sys import tempfile # -------------------------------------------------------------------------- # matching primitives # -------------------------------------------------------------------------- CODE_SPAN = re.compile(r'`[^`]*`') RECORD_PREFIX = re.compile(r'^[ \t]*(?:[-*]\s+)?\*\*') ITEM_LEAD = re.compile(r'^[ \t]*(?:[-*]\s+)?\*\*Q(\d+)\b') HEADER_ITEM = re.compile(r'^#{1,3}\s+Q(\d+)\b') Q_TOKEN = re.compile(r'\bQ(\d+)\b') OPEN_STATUS = re.compile(r'Status:\s*\**\s*(?:open|OPEN)\b') OPEN_Q = re.compile( r'Q\d+\b[^*\n]{0,60}?(?:\u2014|\u2013|--)\s*\**\s*(?:OPEN|open|Opened|accepted)\b') CLOSURE_WORD = re.compile(r'\b(CLOSED|DONE|COMPLETE|COMPLETED|VERIFIED)\b') CLOSURE_EXCLUDE = re.compile(r'^\s*(?:[-*]\s+)?\*\*\s*(?:Done means|stays open|nothing is done|not done)\b', re.I) CLOSURE_EXCLUDE_ANY = re.compile(r'stays open|nothing is done', re.I) CONTAINER_HEAD = re.compile(r'^\s*(?:[-*]\s+)?\*\*Q(\d+)\b') CONTAINER_TITLE_MAX = 100 CONTAINER_SEP = re.compile(r'(?:\u2014|\u2013|--|:)') # A closure must be a *statement*: a record line whose lead is an item id or # `Status:`, with the closure word inside the statement window. That is what # refuses the queue's prose lines that mention a closure in passing # (`:6631` "- **No item state changed this tick:** ... no DONE line", `:8430` # "- **No item closed, no new item opened.** ... remains DONE at 32ad55b"). CLOSURE_LEAD = re.compile(r'^\s*(?:[-*]\s+)?\*\*(?:Q\d+\b|Status:)') CLOSURE_WINDOW = 120 STALE_LITERAL = re.compile(r'Status:\s*\**\s*open and unclaimed', re.I) SHA7 = re.compile(r'`([0-9a-f]{7,63})[0-9a-f\u2026]*`') # Hand-read ground truth: eleven answers read off the queue by hand before any # instrument runs (queue-hygiene-census.md §2). Two independent lanes now. HAND_READ = { 2264: 'CLOSED', 2684: 'CLOSED', 2898: 'CLOSED', 6348: 'CLOSED', 6419: 'CLOSED', 6542: 'CLOSED', 6847: 'CLOSED', 6990: 'CLOSED', 7106: 'CLOSED', 7167: 'CLOSED', 8530: 'OPEN', } def strip_code(line): return CODE_SPAN.sub('', line) def is_record_line(line): return bool(RECORD_PREFIX.match(line)) def is_opening_record(line): s = strip_code(line) if not is_record_line(s): return False return bool(OPEN_STATUS.search(s) or OPEN_Q.search(s)) def is_closure_line(line): s = strip_code(line) if not CLOSURE_LEAD.match(s): return False if CLOSURE_EXCLUDE.match(s): return False return bool(CLOSURE_WORD.search(s[:CLOSURE_WINDOW])) def lead_item(line): m = ITEM_LEAD.match(line) return int(m.group(1)) if m else None def line_ids(line): """Every id a closure line can anchor to: its lead id, else its Q-tokens.""" lead = lead_item(line) if lead is not None: return {lead} return {int(x) for x in Q_TOKEN.findall(strip_code(line))} # -------------------------------------------------------------------------- # the legacy unit (BQ), transcribed from /root/crucible/q66/q66_census6.py:41-62 # -------------------------------------------------------------------------- # Kept here, not for its answer but so that the OTHER number in circulation can # be reproduced from one command and printed beside this file's own unit. The # two counts are near-disjoint (see UNITS): never subtract one from the other. LEGACY_CLOSE_WORD = re.compile(r'\b(CLOSED|DONE|COMPLETE|VERIFIED)\b', re.I) LEGACY_NOT_CLOSURE = re.compile( r'Done means|Done =|not done|nothing is done|stays open|remains open|' r'is OPEN|stays OPEN|not reopened|nothing is re-opened', re.I) def legacy_closures(text): """Line numbers the BQ unit counts: column-0 `**`, a Q token, a closure word anywhere (case-insensitive), no legacy exclusion phrase.""" out = [] for i, ln in enumerate(text.split('\n'), 1): if not ln.startswith('**') or LEGACY_NOT_CLOSURE.search(ln): continue if re.search(r'\bQ(\d+)\b', ln) and LEGACY_CLOSE_WORD.search(ln): out.append(i) return out def _acs_reject_axis(line): """Why the ACS unit drops a line the BQ unit counted.""" s = strip_code(line) if not CLOSURE_LEAD.match(s): return 'lead: not a `**Q`/`**Status:` record line (BQ takes any column-0 `**`)' if CLOSURE_EXCLUDE.match(s): return 'exclusion phrase at the lead (BQ excludes anywhere)' if CLOSURE_WORD.search(s[:CLOSURE_WINDOW]): return 'other' if CLOSURE_WORD.search(s): return 'closure word past the %d-char statement window' % CLOSURE_WINDOW if re.search(r'\b(CLOSED|DONE|COMPLETE|VERIFIED)\b', s, re.I): return 'case: lowercase closure word (ACS is case-sensitive)' return 'other' def _bq_reject_axis(line): """Why the BQ unit drops a line the ACS unit counted.""" if not line.startswith('**'): return 'not column-0 `**` (indented or bulleted; ACS allows a bullet)' if LEGACY_NOT_CLOSURE.search(line): return 'BQ exclusion phrase anywhere on the line' if not re.search(r'\bQ(\d+)\b', line): return 'no Q token on the line (ACS inherits the container id)' if not LEGACY_CLOSE_WORD.search(line): return 'BQ closure word missing' return 'other' # -------------------------------------------------------------------------- # the census # -------------------------------------------------------------------------- def item_title_container(line): """The id of a `**Q — title**` container line, or None. A container is a *title statement*: bold, leading item id, a separator inside the bold span, and the bold span closing within CONTAINER_TITLE_MAX characters. Measured reasons, both from the live queue: `:5597` (`- **Q41 — the non-magazine excerpt class *and* placement** *(archive verification; ...`) is Q41's container and must count, while `:4457` (`**Q32 record precision — two of the thirteen new assertions cannot fail, ...`) is a prose record that must not — counting it stole Q33's own closure lines (`:4505`) for Q32. """ m = CONTAINER_HEAD.match(line) if not m: return None rest = line[m.end():] close = rest.find('**') if close == -1 or close > CONTAINER_TITLE_MAX: return None if not CONTAINER_SEP.search(rest[:close]): return None return int(m.group(1)) def container_index(lines, loose=False): """[(line, id_or_None)] for every *container* line, in file order. A container is a header of any level, or a bold record line that leads with an item id. Two strictnesses, because the two questions differ: * `loose=True` — used to name an opening record that carries no id of its own. Any bold line leading with `Q` counts; the item whose block the record sits in is the nearest such marker above it. * `loose=False` — used to anchor a closure. Here the marker must be a *title statement* (see `item_title_container`), because a long bold prose record that merely starts with an id steals the anchor: `:4457` (`**Q32 record precision — two of the thirteen new assertions cannot fail, ...`) otherwise took `:4505`, Q33's own closure, for Q32. """ out = [] for n, l in enumerate(lines, 1): if l.startswith('#'): m = HEADER_ITEM.match(l) out.append((n, int(m.group(1)) if m else None)) elif loose and CONTAINER_HEAD.match(l): m = ITEM_LEAD.match(l) out.append((n, int(m.group(1)) if m else None)) else: cid = item_title_container(l) if cid is not None: out.append((n, cid)) return out def inherit_id(n, hidx): """Id of the nearest preceding header, or None.""" best = None for ln, qid in hidx: if ln < n: best = qid else: break return best def census(text): lines = text.splitlines() hidx = container_index(lines, loose=False) lidx = container_index(lines, loose=True) closures = [] # (line, idset, text) unanchored = [] # closure statements that name no item for n, l in enumerate(lines, 1): if not is_closure_line(l): continue ids = line_ids(l) if not ids: inh = inherit_id(n, hidx) if inh is None: unanchored.append(n) continue ids = {inh} closures.append((n, ids, l)) records = [] for n, l in enumerate(lines, 1): if not is_opening_record(l): continue lead = lead_item(l) inherited = False if lead is None: lead = inherit_id(n, lidx) inherited = True anchored = [(cn, cl) for (cn, ids, cl) in closures if lead is not None and lead in ids] # State is decided by a closure recorded at or after the opening record. # Earlier closures are reported as evidence but never set the state: the # queue's record order is not the work's order (Q25's DONE line sits # 96 lines *above* its own opening record). closing = [c for c in anchored if c[0] >= n] state = 'CLOSED' if closing else 'OPEN' stale = bool(STALE_LITERAL.search(strip_code(l))) and state == 'CLOSED' records.append({ 'line': n, 'item': None if lead is None else 'Q%d' % lead, 'inherited_id': inherited, 'state': state, 'closure_line': closing[0][0] if closing else None, 'closure_lines_after': [c[0] for c in closing], 'closure_lines_before': [c[0] for c in anchored if c[0] < n], 'closure_text': closing[0][1].strip() if closing else None, 'stale': stale, 'text': l.strip(), }) # collisions: two opening records resolving to one id seen = {} for r in records: if r['item'] is None: continue seen.setdefault(r['item'], []).append(r['line']) for r in records: r['collision'] = bool(r['item'] and len(seen.get(r['item'], [])) > 1) raw_literal = len(re.findall(r'Status:\s*\**\s*open and unclaimed', text, re.I)) return { 'lines': len(lines), 'bytes': len(text.encode()), 'records': records, 'closure_candidates': len(closures), 'closure_unanchored': unanchored, 'literal_anywhere': raw_literal, } def annotate_revisions(rec, git_dir): """First 7-hex backticked token on the closure line, resolved with git.""" out = [] txt = rec.get('closure_text') or '' for tok in SHA7.findall(txt): try: p = subprocess.run(['git', 'cat-file', '-t', tok], cwd=git_dir, capture_output=True, text=True, timeout=20) kind = p.stdout.strip() or p.stderr.strip() ok = p.returncode == 0 except Exception as exc: # pragma: no cover kind, ok = str(exc), False out.append({'token': tok, 'resolves': ok, 'kind': kind}) return out # -------------------------------------------------------------------------- # fixtures — the six defects logged in recoveries/queue-hygiene-census.md §3 # -------------------------------------------------------------------------- FIXTURES = [ ('v1 header-walk only: an opening record nested inside another section', '## Q10 — outer section\n\nDone means: nothing here.\n\n' '- **Q99 — OPEN (nested, not a `##` header).** body\n', ['Q99:OPEN']), ('v2 loose key: a Q7 continuation note is not an opening record', '## Q7 — section\n\n- **Q7, measured:** the unclaimed executable set is open beside it.\n', []), ('v3a a column-0 `---` rule does not cut the region', '## Q11 — section\n\n**Status: open and unclaimed.**\n\n---\n\n' '**Status: DONE — closed below the rule.**\n', ['Q11:CLOSED']), ('v3b `Done means:` is not a closure word', '## Q12 — section\n\n**Status: open and unclaimed.**\n\n**Done means:** one row per space.\n', ['Q12:OPEN']), ("v4 closures are anchored: another item's DONE does not close this one", '## Q13 — section\n\n**Status: open and unclaimed.**\n\n## Q14 — other\n\n' '**Q14 — OPEN (lane).**\n\n**Q14 — DONE at 4010b96.**\n', ['Q13:OPEN', 'Q14:CLOSED']), ('v5 whole-file search: a closure landing after a later opening record still counts', '## Q15 — section\n\n**Q15 — OPEN (lane).**\n\n## Q16 — other\n\n**Q16 — OPEN (lane).**\n\n' '**Q15 — DONE at 4010b96.**\n', ['Q15:CLOSED', 'Q16:OPEN']), ('grep trap: the same words in a code span or a table row are not an opening record', '## Q17 — section\n\n**Status: open and unclaimed. Opened by the pipe.**\n\n' '**Status: DONE — closed.**\n\n' '- (1) Three lines carry the literal `Status: open and unclaimed` (`:1`, `:2`).\n' '\n| `Status: open and unclaimed` | `:2264` | Q21 | DONE |\n', ['Q17:CLOSED']), ('unnumbered opening record inherits the enclosing `## Q` header', '## Q20 — section\n\n**Status: OPEN — held for Alexander.**\n', ['Q20:OPEN']), ('a closed item whose opening line never got a pointer is flagged STALE', '## Q21 — section\n\n**Status: open and unclaimed. Opened by the pipe.**\n\n' '**Status: DONE — `recoveries/x.md`, 2026-10-10.**\n', ['Q21:CLOSED']), ('a bold prose record naming another item does not steal the container', '## Q30 — section\n\n**Status: open and unclaimed.**\n\n' '- **Q31 record precision — two of the thirteen new assertions cannot fail, and two totals are not\n' ' enumerable as stated**, so the record is corrected in place rather than restated.\n\n' '**Status: DONE — closed by the pipe.**\n', ['Q30:CLOSED']), ] UNIT_FIXTURES = [ # name, text, ACS count, BQ count — the four axes that separate the units. ('unit axis 1: a column-0 bold *prose* line is BQ but not ACS', '## Q40 — section\n\n**Addendum, 2026-10-08 — Q40 CLOSED at `8bd0b09`.**\n', 0, 1), ('unit axis 2: a lowercase closure word is BQ but not ACS', '## Q41 — section\n\n**Q41 — verified below.**\n', 0, 1), ('unit axis 3: a status row with no Q token is ACS but not BQ', '## Q42 — section\n\n**Status: DONE — closed.**\n', 1, 0), ('unit axis 4: an indented bullet is ACS but not BQ', '## Q43 — section\n\n- **Q43 — DONE at `4010b96`.**\n', 1, 0), ] STALE_EXPECT = { 'v3a a column-0 `---` rule does not cut the region': 1, 'grep trap: the same words in a code span or a table row are not an opening record': 1, 'a closed item whose opening line never got a pointer is flagged STALE': 1, 'a bold prose record naming another item does not steal the container': 1, } def fixture_signature(text): res = census(text) sig = ['%s:%s' % (r['item'] or 'NONE', r['state']) for r in res['records']] return sig, len([r for r in res['records'] if r['stale']]) def selftest(verbose=True): failures = [] for name, text, expect in FIXTURES: got, stale = fixture_signature(text) ok = got == expect want_stale = STALE_EXPECT.get(name, 0) if stale != want_stale: ok = False if verbose: print('%s %s' % ('PASS' if ok else 'FAIL', name)) if not ok: print(' expected records %s (stale %d)\n got records %s (stale %d)' % (expect, want_stale, got, stale)) if not ok: failures.append(name) # The unit fixtures pin the *population* each matcher counts, so a later # edit to either matcher cannot silently re-open the 72-vs-80 question. n_unit = len(UNIT_FIXTURES) for name, text, want_acs, want_bq in UNIT_FIXTURES: got_acs = sum(1 for l in text.splitlines() if is_closure_line(l)) got_bq = len(legacy_closures(text)) ok = (got_acs, got_bq) == (want_acs, want_bq) if verbose: print('%s %s' % ('PASS' if ok else 'FAIL', name)) if not ok: print(' expected ACS=%d BQ=%d\n got ACS=%d BQ=%d' % (want_acs, want_bq, got_acs, got_bq)) if not ok: failures.append(name) # A read failure is rc 3, not a traceback. `--queue` at a missing path or at # bytes that are not UTF-8 escaped main() as FileNotFoundError / # UnicodeDecodeError; the first was already caught, the second was not, and # rc 1 is what this corpus uses for "a check failed". Two cells, because the # exit code is the whole contract an operator sees. n_read = 0 with tempfile.TemporaryDirectory() as td: bad = os.path.join(td, 'not-utf8.md') with open(bad, 'wb') as fh: fh.write(b'\xff\xfe\x80\n') for name, path, want in ( ('read cell 1: a missing queue path exits 3, not a traceback', os.path.join(td, 'nope.md'), 'No such file'), ('read cell 2: a queue that is not UTF-8 exits 3, not a traceback', bad, 'utf-8')): n_read += 1 err = io.StringIO() with contextlib.redirect_stderr(err): rc = main(['--queue', path]) msg = err.getvalue() ok = rc == 3 and 'cannot read queue:' in msg if ok: ok = want.lower() in msg.lower() if verbose: print('%s %s' % ('PASS' if ok else 'FAIL', name)) if not ok: print(' expected rc 3 and `cannot read queue: ...%s...`\n' ' got rc %d, stderr %r' % (want, rc, msg.strip()[:160])) if not ok: failures.append(name) if verbose: total = len(FIXTURES) + n_unit + n_read print('\nself-test: %d fixtures (%d claim-check, %d unit, %d read-error), ' '%d passed, %d failed' % (total, len(FIXTURES), n_unit, n_read, total - len(failures), len(failures))) return failures # -------------------------------------------------------------------------- # main # -------------------------------------------------------------------------- def main(argv=None): ap = argparse.ArgumentParser(description='Claim check for WORK-QUEUE.md') ap.add_argument('--queue', default=None, help='path to WORK-QUEUE.md') ap.add_argument('--json', default=None, help='write the census as JSON here') ap.add_argument('--git', action='store_true', help='resolve closure revisions with git') ap.add_argument('--validate', action='store_true', help='check against the hand-read answers') ap.add_argument('--selftest', action='store_true', help='run the fixture corpus and exit') ap.add_argument('--fail-on-stale', action='store_true', help='exit 2 if any opening record is stale') ap.add_argument('--explain', type=int, nargs='+', metavar='LINE', help='show how the given line numbers resolve, then exit') ap.add_argument('--census', action='store_true', help='print both closure units (ACS and legacy BQ), their overlap ' 'and the axis table, then exit') args = ap.parse_args(argv) if args.selftest: return 1 if selftest() else 0 root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) path = args.queue or os.path.join(root, 'WORK-QUEUE.md') try: text = open(path, encoding='utf-8').read() except (OSError, UnicodeDecodeError) as exc: # UnicodeDecodeError is not an OSError: a binary or mis-encoded file at # --queue used to escape main() as a traceback with rc 1, which is the # code the fixture corpus uses for "a check failed". A read failure is # rc 3 with the same message, whatever the reason. print('cannot read queue: %s' % exc, file=sys.stderr) return 3 res = census(text) git_dir = os.path.dirname(os.path.abspath(path)) if args.explain: lines = text.splitlines() hidx = container_index(lines, loose=False) lidx = container_index(lines, loose=True) for ln in args.explain: if not 1 <= ln <= len(lines): print(':%d out of range' % ln) continue l = lines[ln - 1] ids = line_ids(l) inh = inherit_id(ln, hidx) inl = inherit_id(ln, lidx) print(':%d container(strict)=%s container(loose)=%s' % ( ln, ('Q%s' % inh) if inh else '(none)', ('Q%s' % inl) if inl else '(none)')) print(' opening=%s closure=%s ids=%s' % (is_opening_record(l), is_closure_line(l), sorted('Q%d' % i for i in ids) if ids else '(inherit)')) print(' %s' % l.strip()[:160]) return 0 if args.git: for r in res['records']: r['revisions'] = annotate_revisions(r, git_dir) print('queue %s — %d lines, %s bytes' % (path, res['lines'], format(res['bytes'], ','))) print('%-6s %-38s %-7s %-9s %s' % ('line', 'item', 'state', 'closure', 'note')) for r in res['records']: note = [] if r['stale']: note.append('STALE-opening-line') if r['collision']: note.append('COLLISION') if r['inherited_id']: note.append('id-inherited') if r['closure_lines_before']: note.append('closure-before-opening ' + ','.join(':%d' % x for x in r['closure_lines_before'])) if len(r['closure_lines_after']) > 1: note.append('closures ' + ','.join(':%d' % x for x in r['closure_lines_after'])) if args.git and r.get('revisions'): note.append('revs ' + ','.join( '%s:%s' % (v['token'], 'ok' if v['resolves'] else 'NOT-A-GIT-OBJECT') for v in r['revisions'])) print('%-6d %-38s %-7s %-9s %s' % ( r['line'], (r['item'] or '(unnumbered)')[:38], r['state'], str(r['closure_line'] or '-'), ' '.join(note))) closed = [r for r in res['records'] if r['state'] == 'CLOSED'] opened = [r for r in res['records'] if r['state'] == 'OPEN'] stale = [r for r in res['records'] if r['stale']] print('\n%d opening records; %d closed; %d open. %d anchored closure statements ' '(unit ACS — a record line leading `**Q`/`**Status:` with a closure word ' 'in the first %d chars, exclusions tested at the lead; %d closure statements ' 'name no item and are ignored, not guessed at).' % (len(res['records']), len(closed), len(opened), res['closure_candidates'], CLOSURE_WINDOW, len(res['closure_unanchored']))) print('open set: %s' % ', '.join( '%s:%d' % (r['item'] or '(unnumbered)', r['line']) for r in opened)) # Three different numbers have been printed as if they were one: the raw # occurrences of the literal (which include code spans and table rows), the # lines it stands on once code spans are stripped, and how many of those are # opening records. Name each, or the count is cited with the wrong unit. _lines = text.splitlines() _lit_stripped = [n for n, l in enumerate(_lines, 1) if STALE_LITERAL.search(strip_code(l))] _lit_opening = [r['line'] for r in res['records'] if STALE_LITERAL.search(strip_code(_lines[r['line'] - 1]))] print('the literal `Status: open and unclaimed` occurs %d times in the raw text; ' 'after code spans are stripped it stands on %d lines, %d of them ' 'opening-record lines, and %d of those are stale (state CLOSED).' % (res['literal_anywhere'], len(_lit_stripped), len(_lit_opening), len(stale))) if args.census: lines = text.splitlines() acs = {n for n, l in enumerate(lines, 1) if is_closure_line(l)} bq = set(legacy_closures(text)) print('\n--- closure-unit census (cite a number only with its unit) ---') print('ACS anchored closure statement (this file) = %d' % len(acs)) print(' record line leading `**Q`/`**Status:`, closure word in the first') print(' %d chars, code spans stripped, exclusions tested at the lead,' % CLOSURE_WINDOW) print(' anchored to its own id or the nearest preceding container.') print('BQ column-0 bold closure line naming a Q id (legacy q66_census6.py:54-62) = %d' % len(bq)) print(' line starting at column 0 with `**`, a Q token anywhere, a') print(' closure word anywhere and case-insensitively, no legacy exclusion phrase.') print('overlap A&B = %d ; ACS-only = %d ; BQ-only = %d ; symmetric difference = %d' % (len(acs & bq), len(acs - bq), len(bq - acs), len(acs ^ bq))) print('NOT a unit conversion: the difference between the two headline numbers is') print('not the delta that shows here, and neither may be cited without its unit.') for label, only, axis in (('BQ-only lines, dropped by ACS', bq - acs, _acs_reject_axis), ('ACS-only lines, dropped by BQ', acs - bq, _bq_reject_axis)): counts = {} for n in sorted(only): a = axis(lines[n - 1]) counts[a] = counts.get(a, 0) + 1 print('%s (%d):' % (label, len(only))) for k in sorted(counts): print(' %-78s %d' % (k, counts[k])) return 0 rc = 0 if args.validate: bad = [] for r in res['records']: exp = HAND_READ.get(r['line']) if exp and exp != r['state']: bad.append((r['line'], exp, r['state'])) n = sum(1 for r in res['records'] if r['line'] in HAND_READ) print('hand-read validation: %d/%d match' % (n - len(bad), n)) for b in bad: print(' MISMATCH :%d hand=%s tool=%s' % b) if bad: rc = 1 if args.json: with open(args.json, 'w', encoding='utf-8') as fh: json.dump(res, fh, indent=2, sort_keys=True) print('wrote %s' % args.json) if args.fail_on_stale and stale: rc = 2 return rc if __name__ == '__main__': sys.exit(main())