troupe-standup/tools/queue_claim_check.py

#!/usr/bin/env python3
"""queue_claim_check.py — the claim check for `WORK-QUEUE.md`, shipped with its tests.

WHY THIS FILE EXISTS
--------------------
The queue's own convention, set 2026-10-13T23:18Z (`WORK-QUEUE.md:8020`), says:

    "every new opening record carries a forward pointer to its closure line or
     revision, and a claim check reads this ledger before taking any item whose
     opening line reads open or unclaimed."

Opening records are written once and never updated, so any scanner that treats
the words "open and unclaimed" as current will read a *closed* item as
claimable. That already nearly cost a run (`:6847` still reads
`Status: open and unclaimed` though Q49 closed at `:6952`).

Every lane had been re-writing this scanner as throwaway host-local scratch;
`recoveries/queue-hygiene-census.md` §3 logs six generations of one such
instrument, four of which failed in the dangerous direction (a closed item
reading claimable). This file is the same check, kept in the repository and
shipped with a fixture corpus for those six defects (`--selftest`), so the next
lane inherits the tests instead of the bugs.

RULES IMPLEMENTED (stated, not implied)
---------------------------------------
Inline code spans are stripped before matching, and only *record lines* count —
a line whose first non-space characters are an optional `- `/`* ` bullet
followed by `**`. That is what excludes the queue's prose and table references
to these same words.

OPENING RECORD — a record line carrying either
  (a) `Status:` followed by `open`/`OPEN`, or
  (b) an item id `Q<n>` followed within 60 characters by an `—`/`–`/`--`
      separator and then `OPEN`/`open`/`Opened`/`accepted`.
Its item id is the leading `**Q<n>` of the line if present; otherwise the id is
inherited from the nearest preceding `## Q<n>` header.

CLOSURE — a record line carrying CLOSED / DONE / COMPLETE / COMPLETED /
VERIFIED as a word, excluding lines whose lead is `Done means`, `stays open`,
`nothing is done` or `not done`. It is anchored to an item by (in order): its
own leading `**Q<n>`; else any `Q<n>` token on the line; else the nearest
preceding `## Q<n>` header. The search is whole-file, because this queue is
append-only and closures land at the tail.

A closure line that anchors an opening record's item => that item is CLOSED.
An opening record with no anchored closure => OPEN. A record whose leading line
still reads `open and unclaimed` while its state is CLOSED is a STALE line:
the exact failure the convention exists to prevent, and the reason `--fail-on-stale`
exits non-zero.

- **The unit.** `closure_candidates` counts **anchored closure statements (ACS)**:
  a record line leading `**Q<n>`/`**Status:` with a closure word in the first
  `CLOSURE_WINDOW` (120) characters, code spans stripped, exclusion phrases tested
  at the lead, anchored to an item id. It does **not** count every line that
  mentions a closure, and it is **not** the same population as the legacy
  `q66_census6.py` figure. See UNITS below before citing a number.

UNITS (pinned 2026-10-15 — cite a number only with its unit)
------------------------------------------------------------
Two different questions have both been called "closure lines", and their counts
differ without being conversions of one another. Measured twice, at the revision
Run 37 read (`29a4872`, queue 1,170,483 B / sha256 `fbe51445a89f81ab…`) and at
`28ca8e5` (1,193,681 B / `c3b4a290dc64bd24…`) — identical on both:

* **ACS — anchored closure statement** (this file's `closure_candidates`) = **72**.
* **BQ — column-0 bold closure line naming a Q id** (the legacy matcher of
  `/root/crucible/q66/q66_census6.py:54-62`, host-local, Run 37) = **80**.
  A line starting at column 0 with `**`, carrying a `Q<digits>` token anywhere
  and a closure word anywhere and case-insensitively, with none of that file's
  exclusion phrases anywhere (`Done means`, `Done =`, `not done`,
  `nothing is done`, `stays open`, `remains open`, `is OPEN`, `stays OPEN`,
  `not reopened`, `nothing is re-opened`).

**The two counts are not two readings of one population: the overlap is 20.**
ACS-only = 52, BQ-only = 60, symmetric difference = 112. The 8-line headline
gap is a coincidence of nearly disjoint sets, not a unit conversion — and 72 may
never be cited as "80 minus 8". The five axes, counted:

* 42 BQ-only lines fail ACS's record-lead rule (an addendum / status /
  verification **prose** line, which BQ takes because it is column-0 bold);
* 39 ACS-only lines carry no `Q<digits>` token (ACS inherits the container id;
  BQ requires the token on the line, which is why it drops `**Status: DONE …` rows);
* 18 BQ-only lines carry a **lowercase** closure word (ACS is case-sensitive);
* 12 ACS-only lines are indented or bulleted (BQ requires column 0);
* 1 ACS-only line carries a BQ exclusion phrase (`:4035`, "stays open") that ACS
  tests only at the lead.

`--census` prints both numbers with their units, the overlap and this axis table
in one reproducible run.

KNOWN BLIND SPOTS (measured, not hand-waved)
--------------------------------------------
- Item ids are labels, not keys: this file's own record carries a collision
  (career-lane apply queue `:7292` also reads `Q53`; the queue's own reply 62
  notes it), which the tool reports as `COLLISION` rather than resolving.
- The state is only as good as the closure line's anchoring. Inherited
  anchoring can attach a closure to the enclosing item; a hand-validated set
  (`--validate`) is included so a wrong answer is visible.
- Nothing here judges whether a closure is *earned*, only that it exists.

USAGE
-----
    python3 tools/queue_claim_check.py                      # census of ./WORK-QUEUE.md
    python3 tools/queue_claim_check.py --queue PATH --json OUT.json
    python3 tools/queue_claim_check.py --validate           # + the hand-read answers
    python3 tools/queue_claim_check.py --git                # resolve closure revisions
    python3 tools/queue_claim_check.py --selftest           # fixture corpus, exit 0/1
    python3 tools/queue_claim_check.py --census             # both closure units + the axis table

Exit codes: 0 clean · 1 self-test failure · 2 stale opening line (--fail-on-stale)
            3 queue unreadable.  No file is written unless --json names one.
"""

import argparse
import json
import os
import re
import subprocess
import sys

# --------------------------------------------------------------------------
# matching primitives
# --------------------------------------------------------------------------

CODE_SPAN = re.compile(r'`[^`]*`')
RECORD_PREFIX = re.compile(r'^[ \t]*(?:[-*]\s+)?\*\*')
ITEM_LEAD = re.compile(r'^[ \t]*(?:[-*]\s+)?\*\*Q(\d+)\b')
HEADER_ITEM = re.compile(r'^#{1,3}\s+Q(\d+)\b')
Q_TOKEN = re.compile(r'\bQ(\d+)\b')

OPEN_STATUS = re.compile(r'Status:\s*\**\s*(?:open|OPEN)\b')
OPEN_Q = re.compile(
    r'Q\d+\b[^*\n]{0,60}?(?:\u2014|\u2013|--)\s*\**\s*(?:OPEN|open|Opened|accepted)\b')

CLOSURE_WORD = re.compile(r'\b(CLOSED|DONE|COMPLETE|COMPLETED|VERIFIED)\b')
CLOSURE_EXCLUDE = re.compile(r'^\s*(?:[-*]\s+)?\*\*\s*(?:Done means|stays open|nothing is done|not done)\b',
                             re.I)
CLOSURE_EXCLUDE_ANY = re.compile(r'stays open|nothing is done', re.I)

CONTAINER_HEAD = re.compile(r'^\s*(?:[-*]\s+)?\*\*Q(\d+)\b')
CONTAINER_TITLE_MAX = 100
CONTAINER_SEP = re.compile(r'(?:\u2014|\u2013|--|:)')

# A closure must be a *statement*: a record line whose lead is an item id or
# `Status:`, with the closure word inside the statement window. That is what
# refuses the queue's prose lines that mention a closure in passing
# (`:6631` "- **No item state changed this tick:** ... no DONE line", `:8430`
# "- **No item closed, no new item opened.** ... remains DONE at 32ad55b").
CLOSURE_LEAD = re.compile(r'^\s*(?:[-*]\s+)?\*\*(?:Q\d+\b|Status:)')
CLOSURE_WINDOW = 120

STALE_LITERAL = re.compile(r'Status:\s*\**\s*open and unclaimed', re.I)
SHA7 = re.compile(r'`([0-9a-f]{7,63})[0-9a-f\u2026]*`')

# Hand-read ground truth: eleven answers read off the queue by hand before any
# instrument runs (queue-hygiene-census.md §2). Two independent lanes now.
HAND_READ = {
    2264: 'CLOSED', 2684: 'CLOSED', 2898: 'CLOSED', 6348: 'CLOSED',
    6419: 'CLOSED', 6542: 'CLOSED', 6847: 'CLOSED', 6990: 'CLOSED',
    7106: 'CLOSED', 7167: 'CLOSED', 8530: 'OPEN',
}


def strip_code(line):
    return CODE_SPAN.sub('', line)


def is_record_line(line):
    return bool(RECORD_PREFIX.match(line))


def is_opening_record(line):
    s = strip_code(line)
    if not is_record_line(s):
        return False
    return bool(OPEN_STATUS.search(s) or OPEN_Q.search(s))


def is_closure_line(line):
    s = strip_code(line)
    if not CLOSURE_LEAD.match(s):
        return False
    if CLOSURE_EXCLUDE.match(s):
        return False
    return bool(CLOSURE_WORD.search(s[:CLOSURE_WINDOW]))


def lead_item(line):
    m = ITEM_LEAD.match(line)
    return int(m.group(1)) if m else None


def line_ids(line):
    """Every id a closure line can anchor to: its lead id, else its Q-tokens."""
    lead = lead_item(line)
    if lead is not None:
        return {lead}
    return {int(x) for x in Q_TOKEN.findall(strip_code(line))}


# --------------------------------------------------------------------------
# the legacy unit (BQ), transcribed from /root/crucible/q66/q66_census6.py:41-62
# --------------------------------------------------------------------------
# Kept here, not for its answer but so that the OTHER number in circulation can
# be reproduced from one command and printed beside this file's own unit. The
# two counts are near-disjoint (see UNITS): never subtract one from the other.

LEGACY_CLOSE_WORD = re.compile(r'\b(CLOSED|DONE|COMPLETE|VERIFIED)\b', re.I)
LEGACY_NOT_CLOSURE = re.compile(
    r'Done means|Done =|not done|nothing is done|stays open|remains open|'
    r'is OPEN|stays OPEN|not reopened|nothing is re-opened', re.I)


def legacy_closures(text):
    """Line numbers the BQ unit counts: column-0 `**`, a Q<digits> token, a
    closure word anywhere (case-insensitive), no legacy exclusion phrase."""
    out = []
    for i, ln in enumerate(text.split('\n'), 1):
        if not ln.startswith('**') or LEGACY_NOT_CLOSURE.search(ln):
            continue
        if re.search(r'\bQ(\d+)\b', ln) and LEGACY_CLOSE_WORD.search(ln):
            out.append(i)
    return out


def _acs_reject_axis(line):
    """Why the ACS unit drops a line the BQ unit counted."""
    s = strip_code(line)
    if not CLOSURE_LEAD.match(s):
        return 'lead: not a `**Q<n>`/`**Status:` record line (BQ takes any column-0 `**`)'
    if CLOSURE_EXCLUDE.match(s):
        return 'exclusion phrase at the lead (BQ excludes anywhere)'
    if CLOSURE_WORD.search(s[:CLOSURE_WINDOW]):
        return 'other'
    if CLOSURE_WORD.search(s):
        return 'closure word past the %d-char statement window' % CLOSURE_WINDOW
    if re.search(r'\b(CLOSED|DONE|COMPLETE|VERIFIED)\b', s, re.I):
        return 'case: lowercase closure word (ACS is case-sensitive)'
    return 'other'


def _bq_reject_axis(line):
    """Why the BQ unit drops a line the ACS unit counted."""
    if not line.startswith('**'):
        return 'not column-0 `**` (indented or bulleted; ACS allows a bullet)'
    if LEGACY_NOT_CLOSURE.search(line):
        return 'BQ exclusion phrase anywhere on the line'
    if not re.search(r'\bQ(\d+)\b', line):
        return 'no Q<digits> token on the line (ACS inherits the container id)'
    if not LEGACY_CLOSE_WORD.search(line):
        return 'BQ closure word missing'
    return 'other'


# --------------------------------------------------------------------------
# the census
# --------------------------------------------------------------------------

def item_title_container(line):
    """The id of a `**Q<n> — title**` container line, or None.

    A container is a *title statement*: bold, leading item id, a separator
    inside the bold span, and the bold span closing within CONTAINER_TITLE_MAX
    characters. Measured reasons, both from the live queue: `:5597`
    (`- **Q41 — the non-magazine excerpt class *and* placement** *(archive
    verification; ...`) is Q41's container and must count, while `:4457`
    (`**Q32 record precision — two of the thirteen new assertions cannot fail,
    ...`) is a prose record that must not — counting it stole Q33's own closure
    lines (`:4505`) for Q32.
    """
    m = CONTAINER_HEAD.match(line)
    if not m:
        return None
    rest = line[m.end():]
    close = rest.find('**')
    if close == -1 or close > CONTAINER_TITLE_MAX:
        return None
    if not CONTAINER_SEP.search(rest[:close]):
        return None
    return int(m.group(1))


def container_index(lines, loose=False):
    """[(line, id_or_None)] for every *container* line, in file order.

    A container is a header of any level, or a bold record line that leads with
    an item id. Two strictnesses, because the two questions differ:

    * `loose=True` — used to name an opening record that carries no id of its
      own. Any bold line leading with `Q<n>` counts; the item whose block the
      record sits in is the nearest such marker above it.
    * `loose=False` — used to anchor a closure. Here the marker must be a
      *title statement* (see `item_title_container`), because a long bold prose
      record that merely starts with an id steals the anchor: `:4457`
      (`**Q32 record precision — two of the thirteen new assertions cannot
      fail, ...`) otherwise took `:4505`, Q33's own closure, for Q32.
    """
    out = []
    for n, l in enumerate(lines, 1):
        if l.startswith('#'):
            m = HEADER_ITEM.match(l)
            out.append((n, int(m.group(1)) if m else None))
        elif loose and CONTAINER_HEAD.match(l):
            m = ITEM_LEAD.match(l)
            out.append((n, int(m.group(1)) if m else None))
        else:
            cid = item_title_container(l)
            if cid is not None:
                out.append((n, cid))
    return out


def inherit_id(n, hidx):
    """Id of the nearest preceding header, or None."""
    best = None
    for ln, qid in hidx:
        if ln < n:
            best = qid
        else:
            break
    return best


def census(text):
    lines = text.splitlines()
    hidx = container_index(lines, loose=False)
    lidx = container_index(lines, loose=True)

    closures = []          # (line, idset, text)
    unanchored = []        # closure statements that name no item
    for n, l in enumerate(lines, 1):
        if not is_closure_line(l):
            continue
        ids = line_ids(l)
        if not ids:
            inh = inherit_id(n, hidx)
            if inh is None:
                unanchored.append(n)
                continue
            ids = {inh}
        closures.append((n, ids, l))

    records = []
    for n, l in enumerate(lines, 1):
        if not is_opening_record(l):
            continue
        lead = lead_item(l)
        inherited = False
        if lead is None:
            lead = inherit_id(n, lidx)
            inherited = True
        anchored = [(cn, cl) for (cn, ids, cl) in closures if lead is not None and lead in ids]
        # State is decided by a closure recorded at or after the opening record.
        # Earlier closures are reported as evidence but never set the state: the
        # queue's record order is not the work's order (Q25's DONE line sits
        # 96 lines *above* its own opening record).
        closing = [c for c in anchored if c[0] >= n]
        state = 'CLOSED' if closing else 'OPEN'
        stale = bool(STALE_LITERAL.search(strip_code(l))) and state == 'CLOSED'
        records.append({
            'line': n,
            'item': None if lead is None else 'Q%d' % lead,
            'inherited_id': inherited,
            'state': state,
            'closure_line': closing[0][0] if closing else None,
            'closure_lines_after': [c[0] for c in closing],
            'closure_lines_before': [c[0] for c in anchored if c[0] < n],
            'closure_text': closing[0][1].strip() if closing else None,
            'stale': stale,
            'text': l.strip(),
        })

    # collisions: two opening records resolving to one id
    seen = {}
    for r in records:
        if r['item'] is None:
            continue
        seen.setdefault(r['item'], []).append(r['line'])
    for r in records:
        r['collision'] = bool(r['item'] and len(seen.get(r['item'], [])) > 1)

    raw_literal = len(re.findall(r'Status:\s*\**\s*open and unclaimed', text, re.I))
    return {
        'lines': len(lines),
        'bytes': len(text.encode()),
        'records': records,
        'closure_candidates': len(closures),
        'closure_unanchored': unanchored,
        'literal_anywhere': raw_literal,
    }


def annotate_revisions(rec, git_dir):
    """First 7-hex backticked token on the closure line, resolved with git."""
    out = []
    txt = rec.get('closure_text') or ''
    for tok in SHA7.findall(txt):
        try:
            p = subprocess.run(['git', 'cat-file', '-t', tok], cwd=git_dir,
                               capture_output=True, text=True, timeout=20)
            kind = p.stdout.strip() or p.stderr.strip()
            ok = p.returncode == 0
        except Exception as exc:                       # pragma: no cover
            kind, ok = str(exc), False
        out.append({'token': tok, 'resolves': ok, 'kind': kind})
    return out


# --------------------------------------------------------------------------
# fixtures — the six defects logged in recoveries/queue-hygiene-census.md §3
# --------------------------------------------------------------------------

FIXTURES = [
    ('v1 header-walk only: an opening record nested inside another section',
     '## Q10 — outer section\n\nDone means: nothing here.\n\n'
     '- **Q99 — OPEN (nested, not a `##` header).** body\n',
     ['Q99:OPEN']),

    ('v2 loose key: a Q7 continuation note is not an opening record',
     '## Q7 — section\n\n- **Q7, measured:** the unclaimed executable set is open beside it.\n',
     []),

    ('v3a a column-0 `---` rule does not cut the region',
     '## Q11 — section\n\n**Status: open and unclaimed.**\n\n---\n\n'
     '**Status: DONE — closed below the rule.**\n',
     ['Q11:CLOSED']),

    ('v3b `Done means:` is not a closure word',
     '## Q12 — section\n\n**Status: open and unclaimed.**\n\n**Done means:** one row per space.\n',
     ['Q12:OPEN']),

    ("v4 closures are anchored: another item's DONE does not close this one",
     '## Q13 — section\n\n**Status: open and unclaimed.**\n\n## Q14 — other\n\n'
     '**Q14 — OPEN (lane).**\n\n**Q14 — DONE at 4010b96.**\n',
     ['Q13:OPEN', 'Q14:CLOSED']),

    ('v5 whole-file search: a closure landing after a later opening record still counts',
     '## Q15 — section\n\n**Q15 — OPEN (lane).**\n\n## Q16 — other\n\n**Q16 — OPEN (lane).**\n\n'
     '**Q15 — DONE at 4010b96.**\n',
     ['Q15:CLOSED', 'Q16:OPEN']),

    ('grep trap: the same words in a code span or a table row are not an opening record',
     '## Q17 — section\n\n**Status: open and unclaimed. Opened by the pipe.**\n\n'
     '**Status: DONE — closed.**\n\n'
     '- (1) Three lines carry the literal `Status: open and unclaimed` (`:1`, `:2`).\n'
     '\n| `Status: open and unclaimed` | `:2264` | Q21 | DONE |\n',
     ['Q17:CLOSED']),

    ('unnumbered opening record inherits the enclosing `## Q<n>` header',
     '## Q20 — section\n\n**Status: OPEN — held for Alexander.**\n',
     ['Q20:OPEN']),

    ('a closed item whose opening line never got a pointer is flagged STALE',
     '## Q21 — section\n\n**Status: open and unclaimed. Opened by the pipe.**\n\n'
     '**Status: DONE — `recoveries/x.md`, 2026-10-10.**\n',
     ['Q21:CLOSED']),

    ('a bold prose record naming another item does not steal the container',
     '## Q30 — section\n\n**Status: open and unclaimed.**\n\n'
     '- **Q31 record precision — two of the thirteen new assertions cannot fail, and two totals are not\n'
     '  enumerable as stated**, so the record is corrected in place rather than restated.\n\n'
     '**Status: DONE — closed by the pipe.**\n',
     ['Q30:CLOSED']),
]

UNIT_FIXTURES = [
    # name, text, ACS count, BQ count — the four axes that separate the units.
    ('unit axis 1: a column-0 bold *prose* line is BQ but not ACS',
     '## Q40 — section\n\n**Addendum, 2026-10-08 — Q40 CLOSED at `8bd0b09`.**\n',
     0, 1),
    ('unit axis 2: a lowercase closure word is BQ but not ACS',
     '## Q41 — section\n\n**Q41 — verified below.**\n',
     0, 1),
    ('unit axis 3: a status row with no Q token is ACS but not BQ',
     '## Q42 — section\n\n**Status: DONE — closed.**\n',
     1, 0),
    ('unit axis 4: an indented bullet is ACS but not BQ',
     '## Q43 — section\n\n- **Q43 — DONE at `4010b96`.**\n',
     1, 0),
]

STALE_EXPECT = {
    'v3a a column-0 `---` rule does not cut the region': 1,
    'grep trap: the same words in a code span or a table row are not an opening record': 1,
    'a closed item whose opening line never got a pointer is flagged STALE': 1,
    'a bold prose record naming another item does not steal the container': 1,
}


def fixture_signature(text):
    res = census(text)
    sig = ['%s:%s' % (r['item'] or 'NONE', r['state']) for r in res['records']]
    return sig, len([r for r in res['records'] if r['stale']])


def selftest(verbose=True):
    failures = []
    for name, text, expect in FIXTURES:
        got, stale = fixture_signature(text)
        ok = got == expect
        want_stale = STALE_EXPECT.get(name, 0)
        if stale != want_stale:
            ok = False
        if verbose:
            print('%s %s' % ('PASS' if ok else 'FAIL', name))
            if not ok:
                print('     expected records %s (stale %d)\n     got      records %s (stale %d)'
                      % (expect, want_stale, got, stale))
        if not ok:
            failures.append(name)

    # The unit fixtures pin the *population* each matcher counts, so a later
    # edit to either matcher cannot silently re-open the 72-vs-80 question.
    n_unit = len(UNIT_FIXTURES)
    for name, text, want_acs, want_bq in UNIT_FIXTURES:
        got_acs = sum(1 for l in text.splitlines() if is_closure_line(l))
        got_bq = len(legacy_closures(text))
        ok = (got_acs, got_bq) == (want_acs, want_bq)
        if verbose:
            print('%s %s' % ('PASS' if ok else 'FAIL', name))
            if not ok:
                print('     expected ACS=%d BQ=%d\n     got      ACS=%d BQ=%d'
                      % (want_acs, want_bq, got_acs, got_bq))
        if not ok:
            failures.append(name)

    if verbose:
        total = len(FIXTURES) + n_unit
        print('\nself-test: %d fixtures (%d claim-check, %d unit), %d passed, %d failed'
              % (total, len(FIXTURES), n_unit, total - len(failures), len(failures)))
    return failures


# --------------------------------------------------------------------------
# main
# --------------------------------------------------------------------------

def main(argv=None):
    ap = argparse.ArgumentParser(description='Claim check for WORK-QUEUE.md')
    ap.add_argument('--queue', default=None, help='path to WORK-QUEUE.md')
    ap.add_argument('--json', default=None, help='write the census as JSON here')
    ap.add_argument('--git', action='store_true', help='resolve closure revisions with git')
    ap.add_argument('--validate', action='store_true', help='check against the hand-read answers')
    ap.add_argument('--selftest', action='store_true', help='run the fixture corpus and exit')
    ap.add_argument('--fail-on-stale', action='store_true',
                    help='exit 2 if any opening record is stale')
    ap.add_argument('--explain', type=int, nargs='+', metavar='LINE',
                    help='show how the given line numbers resolve, then exit')
    ap.add_argument('--census', action='store_true',
                    help='print both closure units (ACS and legacy BQ), their overlap '
                         'and the axis table, then exit')
    args = ap.parse_args(argv)

    if args.selftest:
        return 1 if selftest() else 0

    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
    path = args.queue or os.path.join(root, 'WORK-QUEUE.md')
    try:
        text = open(path, encoding='utf-8').read()
    except OSError as exc:
        print('cannot read queue: %s' % exc, file=sys.stderr)
        return 3

    res = census(text)
    git_dir = os.path.dirname(os.path.abspath(path))

    if args.explain:
        lines = text.splitlines()
        hidx = container_index(lines, loose=False)
        lidx = container_index(lines, loose=True)
        for ln in args.explain:
            if not 1 <= ln <= len(lines):
                print(':%d out of range' % ln)
                continue
            l = lines[ln - 1]
            ids = line_ids(l)
            inh = inherit_id(ln, hidx)
            inl = inherit_id(ln, lidx)
            print(':%d container(strict)=%s container(loose)=%s' % (
                ln, ('Q%s' % inh) if inh else '(none)', ('Q%s' % inl) if inl else '(none)'))
            print('    opening=%s closure=%s ids=%s' %
                  (is_opening_record(l), is_closure_line(l),
                   sorted('Q%d' % i for i in ids) if ids else '(inherit)'))
            print('    %s' % l.strip()[:160])
        return 0

    if args.git:
        for r in res['records']:
            r['revisions'] = annotate_revisions(r, git_dir)

    print('queue %s — %d lines, %s bytes' % (path, res['lines'], format(res['bytes'], ',')))
    print('%-6s %-38s %-7s %-9s %s' % ('line', 'item', 'state', 'closure', 'note'))
    for r in res['records']:
        note = []
        if r['stale']:
            note.append('STALE-opening-line')
        if r['collision']:
            note.append('COLLISION')
        if r['inherited_id']:
            note.append('id-inherited')
        if r['closure_lines_before']:
            note.append('closure-before-opening ' +
                        ','.join(':%d' % x for x in r['closure_lines_before']))
        if len(r['closure_lines_after']) > 1:
            note.append('closures ' + ','.join(':%d' % x for x in r['closure_lines_after']))
        if args.git and r.get('revisions'):
            note.append('revs ' + ','.join(
                '%s:%s' % (v['token'], 'ok' if v['resolves'] else 'NOT-A-GIT-OBJECT')
                for v in r['revisions']))
        print('%-6d %-38s %-7s %-9s %s' % (
            r['line'],
            (r['item'] or '(unnumbered)')[:38],
            r['state'],
            str(r['closure_line'] or '-'),
            ' '.join(note)))

    closed = [r for r in res['records'] if r['state'] == 'CLOSED']
    opened = [r for r in res['records'] if r['state'] == 'OPEN']
    stale = [r for r in res['records'] if r['stale']]
    print('\n%d opening records; %d closed; %d open. %d anchored closure statements '
          '(unit ACS — a record line leading `**Q<n>`/`**Status:` with a closure word '
          'in the first %d chars, exclusions tested at the lead; %d closure statements '
          'name no item and are ignored, not guessed at).'
          % (len(res['records']), len(closed), len(opened), res['closure_candidates'],
             CLOSURE_WINDOW, len(res['closure_unanchored'])))
    print('open set: %s' % ', '.join(
        '%s:%d' % (r['item'] or '(unnumbered)', r['line']) for r in opened))
    # Three different numbers have been printed as if they were one: the raw
    # occurrences of the literal (which include code spans and table rows), the
    # lines it stands on once code spans are stripped, and how many of those are
    # opening records. Name each, or the count is cited with the wrong unit.
    _lines = text.splitlines()
    _lit_stripped = [n for n, l in enumerate(_lines, 1) if STALE_LITERAL.search(strip_code(l))]
    _lit_opening = [r['line'] for r in res['records']
                    if STALE_LITERAL.search(strip_code(_lines[r['line'] - 1]))]
    print('the literal `Status: open and unclaimed` occurs %d times in the raw text; '
          'after code spans are stripped it stands on %d lines, %d of them '
          'opening-record lines, and %d of those are stale (state CLOSED).'
          % (res['literal_anywhere'], len(_lit_stripped), len(_lit_opening), len(stale)))

    if args.census:
        lines = text.splitlines()
        acs = {n for n, l in enumerate(lines, 1) if is_closure_line(l)}
        bq = set(legacy_closures(text))
        print('\n--- closure-unit census (cite a number only with its unit) ---')
        print('ACS anchored closure statement (this file) = %d' % len(acs))
        print('    record line leading `**Q<n>`/`**Status:`, closure word in the first')
        print('    %d chars, code spans stripped, exclusions tested at the lead,' % CLOSURE_WINDOW)
        print('    anchored to its own id or the nearest preceding container.')
        print('BQ  column-0 bold closure line naming a Q id (legacy q66_census6.py:54-62) = %d'
              % len(bq))
        print('    line starting at column 0 with `**`, a Q<digits> token anywhere, a')
        print('    closure word anywhere and case-insensitively, no legacy exclusion phrase.')
        print('overlap A&B = %d ; ACS-only = %d ; BQ-only = %d ; symmetric difference = %d'
              % (len(acs & bq), len(acs - bq), len(bq - acs), len(acs ^ bq)))
        print('NOT a unit conversion: the difference between the two headline numbers is')
        print('not the delta that shows here, and neither may be cited without its unit.')
        for label, only, axis in (('BQ-only lines, dropped by ACS', bq - acs, _acs_reject_axis),
                                  ('ACS-only lines, dropped by BQ', acs - bq, _bq_reject_axis)):
            counts = {}
            for n in sorted(only):
                a = axis(lines[n - 1])
                counts[a] = counts.get(a, 0) + 1
            print('%s (%d):' % (label, len(only)))
            for k in sorted(counts):
                print('   %-78s %d' % (k, counts[k]))
        return 0

    rc = 0
    if args.validate:
        bad = []
        for r in res['records']:
            exp = HAND_READ.get(r['line'])
            if exp and exp != r['state']:
                bad.append((r['line'], exp, r['state']))
        n = sum(1 for r in res['records'] if r['line'] in HAND_READ)
        print('hand-read validation: %d/%d match' % (n - len(bad), n))
        for b in bad:
            print('  MISMATCH :%d hand=%s tool=%s' % b)
        if bad:
            rc = 1

    if args.json:
        with open(args.json, 'w', encoding='utf-8') as fh:
            json.dump(res, fh, indent=2, sort_keys=True)
        print('wrote %s' % args.json)

    if args.fail_on_stale and stale:
        rc = 2
    return rc


if __name__ == '__main__':
    sys.exit(main())