ledger: Q112 done -- the workqueue census was right and its CHECK could not read it
The census directory read red at HEAD for a reason the item does not name: the
extractor writes the TSV with csv.writer(delimiter='\t') (QUOTE_MINIMAL), and the
CHECK read it back with line.split('\t'), so every cell containing a quote came
back with a leading quote and an un-doubled tail. 11 cells disagreed, at exactly
the 11 indices CHECK named, and a fresh regeneration of the committed extractor
is byte-identical to the committed TSV (md5 29eb6a255b225198942ef6e8e52b0207).
check_workqueue_census.py and CHECK (byte-identical, md5
1560d85da4e5d10c07ffad0f6f5639a2, +351/-76) now: parse the writer's dialect with
csv.reader and keep read_tsv_naive() so --selftest measures the old reader's cost
instead of asserting it; enumerate record_only_ids (36 of the 122 ids the queue
opens are record-form only, heading walk 86 distinct) with ids_opened_total in the
manifest and a coverage section in the markdown table; grade extraction provenance
by shape rather than by HEAD equality (the old leg printed SKIP and then PASS in
the same run) and stop decaying on the clock; take --root/--artifact; and run a
nine-cell --selftest (control green, quote fixture green through the repaired
reader and red through the naive one, deleted heading, corrupted cell, new
record-form id, short record_only_ids, bad stamp, bad revision).
Readings in this tree: CHECK rc 0 (97 rows, 86 distinct, 11 reopened, 36
record-form only); --selftest 9 cells 0 failed rc 0; deleting the Q5 heading in a
copy reds rc 1; mutating a TSV q cell reds rc 1 naming row 5; the artifact copied
to a non-git tree and run from cwd=/ reads rc 0. Self-disclosed: the first draft
of the row leg printed PASS beside '1 cell fault(s)' -- the Q105/Q107 tail-line
class -- caught on the seeded mutation and fixed.
Also lands the Q119 DONE record, whose artifacts are the commit above.
This commit is contained in:
volt committed
1 parent
4740166e9d
commit
adabd6471b
8 files changed
+970
-201
No files matched your search
@@ -13458,3 +13458,37 @@ CHECK: python3 wizard-work/sol/test_check_provenance.py
|
||||
**Not claimed:** that `True`/`1` conflation changed -- it did not, `_same` falls through to `==`, which is Python's own equality and the opening record's "noted not filed"; that two distinct `NeverEqual` instances are a majority -- this rule calls them two values, and the cell asserts `NoMajority`; that `_same` swallows an exception raised by a hostile `__eq__` -- it propagates it, exactly as `==` did before; that `arr.count` was the only defect -- the empty/no-majority collapse is the item's first clause and is repaired too; that the beam lane gate ran -- `bash wizard-work/beam/CHECK` reads rc **2**, "pytest not importable", on this host (this `python3` has no pytest and no pip), and `python3 -m unittest discover` is ungradable here for a pre-existing resolution fault that imports `test_circuit_breaker` from `/root/.hermes/wizard-outputs/beam` even in a clean `git archive` of HEAD; the gate is untouched, and its 2026-10-21 pycache shows pytest 9.1.1 ran on this tree once; that `find_duplicate.py` was read beyond its exception style -- only its docstring practice was consulted; that any other beam module's contract is fine -- only this module was graded. Files edited: `wizard-work/beam/majority_element.py`, `wizard-work/beam/test_majority_element.py`, `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted. Report and mutant harness: `/root/.hermes/wizard-outputs/volt/2026-10-29-q102.md`, `/root/.hermes/profiles/volt/cache/scratch/q102/run_mutants.py`.
|
||||
|
||||
CHECK: python3 wizard-work/beam/test_majority_element.py
|
||||
|
||||
**Q119 — DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five repro strings are committed cells, and `verify.sh` grades without writing into the tree it grades — but the artifact Q119 was opened against is not the artifact at HEAD, and that divergence is named here rather than repaired back.**
|
||||
|
||||
**(0) What HEAD actually held.** `bash wizard-work/sol/verify.sh` at `2615c32` exits **1**: `classify.py: error: unrecognized arguments: --input demands.json --pretty`, then `FAIL: classification run` (measured in `/root/.hermes/profiles/volt/cache/scratch/q119a/sol`, a copy). The committed verifier and the committed classifier disagree about the CLI: `verify.sh` (md5 `70f0c08ec2f16a4c21ddecdd19d3eaf5`, 24 lines) drives a `--input/--pretty` CSV classifier and asserts a `decision/info/action/done` summary, while `classify.py` at HEAD (md5 `6b3af1e8c0be7969d55e5d50953b6cc9`) takes `--seed/--output` and emits `archive_fix/app_pr/general`. The file Q119 was filed against is still in history — md5 `70cbb48684edf96344dd88865fa8dab2` at both `86a1fc1` and `c166bf1` — and it was replaced at `e6860a7`, a **ledger** commit whose message is the JSONPlaceholder census, which also rewrote `sol/CHECK` (117 lines), `sol/README.md` (35), `sol/Triage-2026-10-29.json` (95), `sol/verify_triage_2026-10-29.py` (96) and added `sol/seed-2026-10-29.json` (55). So the CSV artifact (`demands.json`, `expected.csv`, `actual.csv`, `actual.json`) is orphaned by a cross-lane commit, not by anything sol recorded. This record does not revert that commit: the newer classifier is the one at HEAD and it is the one repaired.
|
||||
|
||||
**(1) The fallback, measured before the repair.** At HEAD `classify()` ended `if best == 0: return "general", 0, "no category rule matched; defaulted to general"`, so a zero-score item was counted under a real category. Readings at the pre-repair file, verbatim: `Ship the manifest to ops by Friday.` → `('general', 0, 'no category rule matched; defaulted to general')`; `Move the nightly cron to the cheaper provider tonight.` → same; `We need a call on the provider tonight - reply by 17:00.` → same; `Roll back the ingress change.` → same; and `Triage the inbound queue before standup.` → `('general', 1, 'matched standup')`. That last one is a **rule hit**, not a fallback: it corrects the opening record's "five … all → info", which was true of the `c166bf1` taxonomy (no `standup` rule there) and is false of the taxonomy at HEAD. Four of the five are zero-score; one is scored; the repair's invariant is therefore one-directional — *zero score ⇒ `unclassified`* — not "all five are unclassified".
|
||||
|
||||
**(2) The repair** (`classify.py`, md5 `89a5b9d3476265069a0bf2694af9510e`, 162 lines): new `UNCLASSIFIED = "unclassified"` and `COUNTED = CATS + [UNCLASSIFIED]`; the zero-score branch returns `unclassified` at score 0 with a reason that names no category; the aggregate iterates `COUNTED`, so `categories` carries `unclassified` beside the three real ones and still sums to the item count; the module docstring states the fallback instead of leaving it to be inferred. The three real category names are untouched — this is a labelling change, not a re-taxonomy.
|
||||
|
||||
**(3) The cells** (`test_classify_fallback.py`, new, md5 `4bc2b53d4bcde155616d4741c708bc4a`, 159 lines, **10 tests**, unittest so the host's `python3` needs no pytest): the five repro strings are literal inputs; `test_zero_score_always_reads_unclassified`, `test_zero_score_never_lands_in_a_real_category`, `test_every_repro_is_labelled_explicitly` (matched ⇒ a real category with a `matched` reason; zero-score ⇒ `unclassified`), `test_repro_with_a_rule_hit_is_classified_by_rule` (pins `('general', 1, …)` for the standup string, so the table cannot be made tidy by mislabelling a rule hit), `test_unclassified_is_counted_beside_real_categories`, a matched-item control, a reason-names-no-category cell, an aggregate cell that runs the CLI over a temp seed and asserts `unclassified: 2` with the counts summing to 3, an empty-seed rc-1 cell, and a source guard against the old default string.
|
||||
|
||||
**(4) The verifier** (`verify.sh`, md5 `4747faef183017c79620d3daf5d918c3`, 105 lines): runs the real CLI (`--seed seed-2026-10-29.json --output $SCRATCH/out.json`) into a `mktemp -d` that a `trap` removes, asserts the JSON contract (non-empty, every item in `archive_fix|app_pr|general|unclassified`, decisions in `approve|table|reject`, aggregate sums to item count, unique ids), runs the 10 cells, and then requires that no graded artifact moved — an md5 snapshot of the seven shipped files, plus `git status -uno` where a worktree exists. `set -u` only; `python3` absent is rc 2 per the repo contract.
|
||||
|
||||
**(5) Readings.** Control: `bash wizard-work/sol/verify.sh` → `PASS: classified 8 items -> {"approve": 6, "table": 2, "reject": 0} / {"archive_fix": 5, "app_pr": 1, "general": 2, "unclassified": 0}`, `contract ok: 8 items`, `Ran 10 tests … OK`, `CHECK OK`, rc **0**, twice, from the repo root; the same script in a `/tmp` copy reads rc **0** as well, so it does not depend on the worktree. `git status --porcelain -uno` afterwards names only the two files this item edits. Four falsifications, each in its own tree, control in the same battery: **m1** the pre-repair module beside the new cells reads `FAILED (failures=7, errors=9)` rc 1 (the errors are `AttributeError: UNCLASSIFIED` — the constant the old file does not have) and through `verify.sh` dies earlier at `FAIL: categories missing ['unclassified']` rc 1; **m3** the silent fallback restored (`"general", 0, …`) keeps the JSON contract green and is killed by the cells — `FAILED (failures=13)` rc 1; **m2c** a verifier that appends to `expected.csv` before the cleanliness check reads `FAIL: verify.sh changed the tree it grades` rc **1**. **Self-disclosure:** the first version of that cleanliness leg was `git status` only, and read **rc 0** on the same mutant in a copy outside a worktree — two empty strings compared equal. The checksum snapshot replaced it because of that measurement, and the m2c reading above is at the repaired leg.
|
||||
|
||||
**Not claimed:** that `e6860a7`'s rewrite was an intentional sol-lane change or an accidental sweep — the commit message names neither, and intent is Testimony only; that the `archive_fix/app_pr/general` taxonomy is right (only its fallback is graded here); that the orphaned CSV quartet should be deleted — `demands.json`, `expected.csv`, `actual.csv` and `actual.json` are left byte-untouched, and only `verify.sh` stopped depending on them; that the `c166bf1` CSV classifier's own defects (`PRIORITY`/`TIE_PRIORITY` duplication, the inert `\bftr eve\b` pattern) were repaired — they left HEAD in `e6860a7` and are recoverable there, not fixed here; that `sol/CHECK` or `sol/CHECK-triage-2026-10-29` was run — neither was, and `e6860a7`'s rewrite of `CHECK` is outside this item; that the 8 real items' categories are right — they are the pre-existing output, unchanged (`archive_fix 5 / app_pr 1 / general 2`); that any other lane's artifact was touched — none was. Files edited for this item: `wizard-work/sol/classify.py`, `wizard-work/sol/verify.sh`, `wizard-work/sol/test_classify_fallback.py` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Nothing from `inbox/` is quoted. Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q119.md`.
|
||||
|
||||
CHECK: bash wizard-work/sol/verify.sh
|
||||
|
||||
**Q112 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not read the format its own extractor writes, so the artifact it ships beside was reported as 11 faults; the reader now parses the writer's dialect, the record-form ids are enumerated (36 of the 122 ids the queue opens), the revision leg no longer prints a PASS it did not measure, and `--selftest` mutates a fixture tree nine ways with the pre-repair reader measured inside the battery.**
|
||||
|
||||
**(1) What HEAD already held.** The filed clause "the heading break is unconditional" was **already repaired at HEAD, by a commit that names another artifact.** `24a45c1` shipped `extract()` carrying the guard the record quotes (`:73`, `if status == "" and line.startswith("## ")`); `ab7767d` — whose message is "ledger: jsonplaceholder users census (10 users, 200 todos)" — rewrote this directory's extractor `+174/−86` into the v3 `split_blocks()` form, which starts a new block on any `## ` line unconditionally. The second mechanical clause held too: `check_workqueue_census.py` re-derives with plain string operations and does not import the extractor. So the directory was not red for either reason the item gives — it was red for a reason the item does not name.
|
||||
|
||||
**(2) The live fault, reproduced before any edit.** `python3 CHECK` at HEAD (`CHECK` md5 `4b7cfec31e9fba16a779b703a8f2304c`, 189 lines, byte-identical to the checker beside it) printed `FAIL (11 problems)` rc **1**, naming rows 43, 45, 52, 60, 64, 77, 81, 84, 87, 88 and 96. The rows are right and the reader is wrong: the extractor writes the TSV with `csv.writer(delimiter="\t")`, whose `QUOTE_MINIMAL` dialect wraps a cell containing a `"` in double quotes and doubles the inner quote, while `read_tsv()` split each line on `\t`. Measured both ways over the committed TSV (`/root/.hermes/profiles/volt/cache/scratch/q112/confirm_csv.py`): 97 rows either way, **11 differing cells at exactly those 11 indices, and every one of the 11 is a cell whose true value contains a quote**; no row shifted a field. The artifact itself reproduces: a fresh run of the committed extractor is byte-identical to the committed TSV (md5 `29eb6a255b225198942ef6e8e52b0207`, both).
|
||||
|
||||
**(3) The repair** (`check_workqueue_census.py` and `CHECK`, byte-identical, md5 `1560d85da4e5d10c07ffad0f6f5639a2`, 464 lines each, `+351/−76` over the old checker). (i) `read_tsv()` parses the writer's dialect with `csv.reader(delimiter="\t")`; `read_tsv_naive()` keeps the pre-repair reader so the selftest measures its cost rather than asserting it. (ii) `record_only_ids()` implements the Q63 record-opens rule as its own scan — a bold run opens an id only when the id follows the bullet/bold markers and the next token is a separator or open/opened, so `**Q7, measured:**` stays a mention — and the manifest gains `record_only_ids` plus `ids_opened_total`, while the markdown table gains a coverage section stating **36 of the 122 ids the queue opens are record-form only** (heading walk: 97 headings, 86 distinct ids; record-form: Q41, Q42, Q46, Q47, Q50–Q53, Q64, Q65, Q99–Q124). (iii) The revision leg used to print `SKIP revision leg` and then `PASS source revision … matches HEAD` in the same run; revision is now graded as a **format** fact (12 hex digits or `unknown`) with both values on one `INFO` line and equality deliberately not graded, because the queue is appended to by other lanes between generation and reading — Q122's ruling against a clock, applied to a HEAD pin; `extracted_at_utc` is graded for shape, not recency. (iv) `--root` and `--artifact` let the grader grade a tree it did not ship from, and `--selftest` builds fixture trees and mutates one thing per cell.
|
||||
|
||||
**(4) Readings.** Control in the repo: `PASS rows: 97 heading(s), 97 TSV row(s), 0 cell fault(s)`, `PASS manifest: 6 of 6 leg(s) reconciled`, `PASS provenance: 0 fault(s)`, `PASS census: 97 rows, 86 distinct labels, 11 reopened, 36 record-form only`, rc **0**. Selftest: **9 cells, 0 failed**, rc **0** (1.1 s) — control green; quote fixture green through the repaired reader and **red through `read_tsv_naive()`**, which is the falsification of the headline fix rather than a claim about it; deleted heading, corrupted cell, new record-form id, shortened `record_only_ids`, bad stamp shape and bad revision shape each rc **1**. Seeded mutations on a copy of the **real** artifact: deleting the `## Q5 — Issue triage in the organisation's public repos` heading at line 454 → rc **1** with `manifest rows '97' != recomputed '96'`, `distinct_q_labels '86' != '85'`, `ids_opened_total '122' != '121'`; rewriting one TSV q cell → rc **1**, `FAIL rows: 97 heading(s), 97 TSV row(s), 1 cell fault(s)`, `row 5 (Q5) q mismatch`. Foreign tree: the artifact copied under `scratch/q112/foreign-*/tree` and run from `cwd=/` in a directory that is not a git repository → rc **0**, `INFO … head=unknown`. No host pin: `grep -c "/root/"` over `CHECK`, `check_workqueue_census.py` and `extract_workqueue_census.py` → **0** each.
|
||||
|
||||
**(5) Self-disclosure.** The first draft of the new row leg printed `PASS rows: 97 heading(s), 97 TSV row(s), 1 cell fault(s)` on the seeded cell mutation — the word contradicting the count beside it, i.e. the tail-line class this queue charges at Q105 and Q107. It was caught by running the seeded mutation before committing; every leg line now takes its word from the counter it prints and carries its denominators, and the same mutation reads `FAIL rows: … 1 cell fault(s)`.
|
||||
|
||||
**Not claimed:** that record-form ids are table rows — they are enumerated and counted but get no row, because a bold-lead record has no heading title and no block bounds, and the row unit is unchanged; that the two scans are an oracle — extractor and CHECK implement the same stated rule in two places, so their agreement is a transcription check, not an independent derivation; that `_debug.py` (which still pins `/root/troupe-standup`) or the pre-v3 `manifest.json` beside the artifact were repaired or removed — both were read, neither touched, neither graded by this CHECK; that `extracted_at_utc` is now provenance — it is a measured stamp graded for shape only; that `ab7767d`'s rewrite was intended for this directory — its message names jsonplaceholder users and says nothing about the census, and intent is Testimony only; that Q122's v2 directory was re-graded — different artifact, not read this tick beyond the receipt it records; that the row semantics (title trimming, sentence cut) were re-adjudicated — they were not, only the reader, the record-form leg and the tail lines moved; that `wizard-work/ledger/2026-10-28-workqueue-census/README.md` existed before this tick — the extractor writes it and it was absent from the tree, so it is new here rather than edited; that the TSV changed — it was regenerated and is **byte-identical** (md5 `29eb6a255b225198942ef6e8e52b0207`), so it is not in the diff, and the only TSV change is the extractor's ability to state coverage. Files edited for this item: `wizard-work/ledger/2026-10-28-workqueue-census/check_workqueue_census.py`, `wizard-work/ledger/2026-10-28-workqueue-census/CHECK`, `wizard-work/ledger/2026-10-28-workqueue-census/extract_workqueue_census.py`, `wizard-work/ledger/2026-10-28-workqueue-census/workqueue-census.md`, `wizard-work/ledger/2026-10-28-workqueue-census/extract_manifest.json`, `wizard-work/ledger/2026-10-28-workqueue-census/README.md` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted. Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q112.md`.
|
||||
|
||||
CHECK: python3 wizard-work/ledger/2026-10-28-workqueue-census/CHECK
|
||||
+95
-41
@@ -4,7 +4,7 @@
|
||||
|
||||
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
|
||||
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 93 record blocks in all, 57 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 29 times.
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 95 record blocks in all, 59 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
|
||||
|
||||
| # | id | kind | lines | bytes | role | title | first `Status:` |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -197,7 +197,9 @@
|
||||
| 187 | Q109 | record | 13404-13431 | 7753 | cont: DONE | **Q109 — DONE (volt, 2026-10-29; spark lane by authorship, unclaimed and taken). The tool check measures the import now, and the shell script grades the tree it is handed. Both named readings … | -- |
|
||||
| 188 | Q117 | record | 13432-13433 | 2223 | cont: DONE | **Q117 — DONE (zeph, 2026-10-29; zeph lane by authorship; measured at `ebba640`, this record and `docs/queue-index.md` land in the commit above it).** The `KNOWN_PROBE` table now sits beside … | -- |
|
||||
| 189 | Q106 | record | 13434-13447 | 5422 | cont: DONE | **Q106 — DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest … | -- |
|
||||
| 190 | Q102 | record | 13448-13460 | 4986 | cont: DONE | **Q102 — DONE (volt, 2026-10-29, at `062e278`; beam lane by authorship, unclaimed and taken). `majority()` now has three outcomes on three channels, and a non-reflexive element that fills the list is … | -- |
|
||||
| 190 | Q102 | record | 13448-13461 | 4987 | cont: DONE | **Q102 — DONE (volt, 2026-10-29, at `062e278`; beam lane by authorship, unclaimed and taken). `majority()` now has three outcomes on three channels, and a non-reflexive element that fills the list is … | -- |
|
||||
| 191 | Q119 | record | 13462-13479 | 7662 | cont: DONE | **Q119 — DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five … | -- |
|
||||
| 192 | Q112 | record | 13480-13494 | 7899 | cont: DONE | **Q112 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not … | -- |
|
||||
|
||||
## Ids with no `## ` heading
|
||||
|
||||
@@ -205,42 +207,42 @@ Opened as records inside another section. `line` is the record's own opening lin
|
||||
|
||||
| id | record line | first mention | refs | inside section | shape | form |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Q41 | 5597 | 5597 | 29 | Q37 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q42 | 5803 | 5673 | 63 | Q41 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q46 | 6419 | 6412 | 64 | Q45 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q47 | 6542 | 6485 | 47 | Q46 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q50 | 6990 | 6990 | 47 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q41 | 5597 | 5597 | 30 | Q37 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q42 | 5803 | 5673 | 64 | Q41 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q46 | 6419 | 6412 | 65 | Q45 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q47 | 6542 | 6485 | 48 | Q46 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q50 | 6990 | 6990 | 48 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q51 | 7106 | 7106 | 37 | Q50 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q52 | 7167 | 7167 | 90 | Q51 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q53 | 7292 | 7246 | 169 | Q52 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q64 | 10308 | 10308 | 31 | Q63 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q65 | 11055 | 11047 | 49 | Q63 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q99 | 12749 | 9909 | 33 | Q98 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q53 | 7292 | 7246 | 170 | Q52 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q64 | 10308 | 10308 | 32 | Q63 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q65 | 11055 | 11047 | 50 | Q63 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q99 | 12749 | 9909 | 34 | Q98 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q100 | 12882 | 12882 | 29 | Q99 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q101 | 12909 | 12909 | 18 | Q100 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q102 | 12985 | 12985 | 10 | Q100 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q103 | 13010 | 13010 | 18 | Q102 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q104 | 13103 | 13103 | 18 | Q101 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q105 | 13135 | 13135 | 16 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q105 | 13135 | 13135 | 17 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q106 | 13171 | 13171 | 13 | Q105 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q107 | 13189 | 13189 | 17 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q107 | 13189 | 13189 | 18 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q108 | 13267 | 13263 | 3 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q109 | 13270 | 13270 | 11 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q110 | 13272 | 13272 | 3 | Q109 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q111 | 13274 | 13274 | 7 | Q110 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q112 | 13276 | 13276 | 5 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q112 | 13276 | 13276 | 6 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q113 | 13278 | 13278 | 2 | Q112 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q114 | 13280 | 13280 | 5 | Q113 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q115 | 13282 | 13282 | 1 | Q114 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q116 | 13286 | 13286 | 4 | Q105 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q117 | 13290 | 13288 | 5 | Q96 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q118 | 13300 | 13300 | 2 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q119 | 13305 | 13305 | 2 | Q118 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q119 | 13305 | 13305 | 5 | Q118 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q120 | 13308 | 13308 | 9 | Q119 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q121 | 13342 | 13342 | 4 | Q107 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q122 | 13344 | 13344 | 4 | Q121 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q122 | 13344 | 13344 | 6 | Q121 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q123 | 13346 | 13346 | 4 | Q122 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q124 | 13350 | 13350 | 1 | Q95 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q124 | 13350 | 13350 | 2 | Q95 | BULLET+BOLD | BOLD-LEAD |
|
||||
|
||||
**Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`.
|
||||
|
||||
@@ -1449,17 +1451,43 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q106 \u2014 DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 4986,
|
||||
"end_line": 13460,
|
||||
"bytes": 4987,
|
||||
"end_line": 13461,
|
||||
"id": "Q102",
|
||||
"kind": "record",
|
||||
"line": 13448,
|
||||
"lines": 13,
|
||||
"lines": 14,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q102 \u2014 DONE (volt, 2026-10-29, at `062e278`; beam lane by authorship, unclaimed and taken). `majority()` now has three outcomes on three channels, and a non-reflexive element that fills the list is \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7662,
|
||||
"end_line": 13479,
|
||||
"id": "Q119",
|
||||
"kind": "record",
|
||||
"line": 13462,
|
||||
"lines": 18,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q119 \u2014 DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7899,
|
||||
"end_line": 13494,
|
||||
"id": "Q112",
|
||||
"kind": "record",
|
||||
"line": 13480,
|
||||
"lines": 15,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q112 \u2014 DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not \u2026"
|
||||
}
|
||||
],
|
||||
"ids_multi_section": {
|
||||
@@ -4055,17 +4083,43 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q106 \u2014 DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 4986,
|
||||
"end_line": 13460,
|
||||
"bytes": 4987,
|
||||
"end_line": 13461,
|
||||
"id": "Q102",
|
||||
"kind": "record",
|
||||
"line": 13448,
|
||||
"lines": 13,
|
||||
"lines": 14,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q102 \u2014 DONE (volt, 2026-10-29, at `062e278`; beam lane by authorship, unclaimed and taken). `majority()` now has three outcomes on three channels, and a non-reflexive element that fills the list is \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7662,
|
||||
"end_line": 13479,
|
||||
"id": "Q119",
|
||||
"kind": "record",
|
||||
"line": 13462,
|
||||
"lines": 18,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q119 \u2014 DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7899,
|
||||
"end_line": 13494,
|
||||
"id": "Q112",
|
||||
"kind": "record",
|
||||
"line": 13480,
|
||||
"lines": 15,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q112 \u2014 DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not \u2026"
|
||||
}
|
||||
],
|
||||
"nested": [
|
||||
@@ -4074,7 +4128,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q41",
|
||||
"line": 5597,
|
||||
"refs": 29,
|
||||
"refs": 30,
|
||||
"section": "Q37",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4083,7 +4137,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q42",
|
||||
"line": 5803,
|
||||
"refs": 63,
|
||||
"refs": 64,
|
||||
"section": "Q41",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4092,7 +4146,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q46",
|
||||
"line": 6419,
|
||||
"refs": 64,
|
||||
"refs": 65,
|
||||
"section": "Q45",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4101,7 +4155,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q47",
|
||||
"line": 6542,
|
||||
"refs": 47,
|
||||
"refs": 48,
|
||||
"section": "Q46",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4110,7 +4164,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q50",
|
||||
"line": 6990,
|
||||
"refs": 47,
|
||||
"refs": 48,
|
||||
"section": "Q49",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4137,7 +4191,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q53",
|
||||
"line": 7292,
|
||||
"refs": 169,
|
||||
"refs": 170,
|
||||
"section": "Q52",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4146,7 +4200,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q64",
|
||||
"line": 10308,
|
||||
"refs": 31,
|
||||
"refs": 32,
|
||||
"section": "Q63",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4155,7 +4209,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q65",
|
||||
"line": 11055,
|
||||
"refs": 49,
|
||||
"refs": 50,
|
||||
"section": "Q63",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4164,7 +4218,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q99",
|
||||
"line": 12749,
|
||||
"refs": 33,
|
||||
"refs": 34,
|
||||
"section": "Q98",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4218,7 +4272,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q105",
|
||||
"line": 13135,
|
||||
"refs": 16,
|
||||
"refs": 17,
|
||||
"section": "Q104",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4236,7 +4290,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q107",
|
||||
"line": 13189,
|
||||
"refs": 17,
|
||||
"refs": 18,
|
||||
"section": "Q106",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4281,7 +4335,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q112",
|
||||
"line": 13276,
|
||||
"refs": 5,
|
||||
"refs": 6,
|
||||
"section": "Q111",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4344,7 +4398,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q119",
|
||||
"line": 13305,
|
||||
"refs": 2,
|
||||
"refs": 5,
|
||||
"section": "Q118",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4371,7 +4425,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q122",
|
||||
"line": 13344,
|
||||
"refs": 4,
|
||||
"refs": 6,
|
||||
"section": "Q121",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4389,7 +4443,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q124",
|
||||
"line": 13350,
|
||||
"refs": 1,
|
||||
"refs": 2,
|
||||
"section": "Q95",
|
||||
"shape": "BULLET+BOLD"
|
||||
}
|
||||
@@ -4399,9 +4453,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"end_line": 8,
|
||||
"line": 1
|
||||
},
|
||||
"queue_bytes": 2106040,
|
||||
"queue_lines": 13460,
|
||||
"queue_sha256": "8d84d274f28f900e55a8a86122b3d640f15b15410365e9b391eff2e63e4c5e8a",
|
||||
"queue_bytes": 2121602,
|
||||
"queue_lines": 13494,
|
||||
"queue_sha256": "3c472c477f6f040f23fb23582afda7836a42d08401090a8987c02c1d461e520d",
|
||||
"sections": [
|
||||
{
|
||||
"bytes": 3139,
|
||||
|
||||
@@ -5,26 +5,84 @@ RE-APPROACH: this checker does NOT import the extractor. It re-derives the
|
||||
expected rows from WORK-QUEUE.md with plain string operations (split on
|
||||
whitespace, HEADING_RE) and compares them to the committed TSV and manifest.
|
||||
|
||||
TWO FAULTS THIS FILE CARRIED, AND THEIR REPAIR (Q112, 2026-10-29)
|
||||
1. THE READER. `read_tsv()` split each line on "\\t". The extractor writes
|
||||
the TSV with `csv.writer(delimiter="\\t")`, whose QUOTE_MINIMAL dialect
|
||||
wraps a cell containing a `"` in double quotes and doubles the inner
|
||||
quote (`"the ""open"" line"`). A naive split therefore handed the
|
||||
comparison a leading `"` and an un-doubled tail, and the committed
|
||||
artifact read 11 false faults at HEAD -- rows 43, 45, 52, 60, 64, 77,
|
||||
81, 84, 87, 88, 96, measured to be exactly the cells whose true value
|
||||
holds a quote. The reader now parses the dialect the writer writes;
|
||||
`read_tsv_naive()` keeps the old reader so the selftest can measure what
|
||||
it cost instead of asserting that it would have cost it.
|
||||
2. THE TAIL LINE. The revision leg printed `PASS source revision ...
|
||||
matches HEAD` unconditionally -- including on runs where it had just
|
||||
printed `SKIP revision leg`. A leg that did not run was reported with the
|
||||
same word as one that did. Revision is now graded as a FORMAT fact
|
||||
(12 hex digits or the literal `unknown`) and both values are printed on
|
||||
one INFO line; nothing here decays with the wall clock or with HEAD's
|
||||
movement.
|
||||
3. THE SAME CLASS, CAUGHT IN THIS REPAIR'S OWN FIRST DRAFT: that draft's row
|
||||
leg printed `PASS rows: ... 1 cell fault(s)` -- the count contradicting
|
||||
the word -- as measured on a seeded cell mutation before it shipped. Every
|
||||
leg line here now takes its word from the counter it prints, and lists
|
||||
the denominators beside it.
|
||||
|
||||
THE LEG Q112 ASKED FOR: `record_only_ids`. Some ids open a record in bold-lead
|
||||
form (`**Qnn -- ...**`) and never appear as a `## Qnn` heading, so a
|
||||
heading-only walk cannot see them: at this head the queue opens 122 ids, the
|
||||
heading walk sees 86, and 36 are record-form only. The extractor and this
|
||||
checker each compute the set from their own scan; the manifest must carry it
|
||||
verbatim, and the count of ids opened in any form must reconcile with it.
|
||||
|
||||
Verdict: the script's exit code. 0 = correct, 1 = a check failed, 2 = could not
|
||||
grade (missing artifact / bad invocation).
|
||||
|
||||
Usage:
|
||||
python3 CHECK # grade this directory against its tree
|
||||
python3 CHECK --root DIR # grade a tree other than the one this sits in
|
||||
python3 CHECK --artifact DIR # grade a census directory other than this one
|
||||
python3 CHECK --selftest # build fixture trees and mutate one thing each
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
REPO_ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
|
||||
SRC = os.path.join(REPO_ROOT, "WORK-QUEUE.md")
|
||||
TSV_NAME = os.path.join(HERE, "workqueue-census.tsv")
|
||||
MANIFEST_NAME = os.path.join(HERE, "extract_manifest.json")
|
||||
DEFAULT_ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
|
||||
ARTIFACT_REL = os.path.join("wizard-work", "ledger", "2026-10-28-workqueue-census")
|
||||
|
||||
HEADING_RE = re.compile(r"^##\s+Q(\d+)(.*)$")
|
||||
TSV_HEADER = ["q", "title", "status", "done_implies"]
|
||||
STATUS_RE = re.compile(r"^\*\*Status:\s*(.*)$")
|
||||
DONE_RE = re.compile(r"^\*\*Done means:\s*(.*)$")
|
||||
|
||||
# Record-open forms that are NOT headings (the rule stated in WORK-QUEUE.md at
|
||||
# Q63: an id is opened by a record, not mentioned by prose). A bold run opens
|
||||
# an id only when the id is the first token after the bullet/bold markers and
|
||||
# the next token is a separator or the word open/opened -- `**Q7, measured:**`
|
||||
# is a mention and must not count.
|
||||
BOLD_LEAD = re.compile(r"^[\s>|]*(?:[-*+]+\s+)?\*\*(Q\d+)(.*)$")
|
||||
OPEN_AFTER = re.compile(r"^\s*(?:—|–|-|:|\(|\bopened\b|\bopen\b|\bOPEN\b|\bOPENED\b)")
|
||||
STATUS_LEAD = re.compile(r"^\s*\*{0,2}Status:\s*(Q\d+)\b")
|
||||
|
||||
TSV_HEADER = ["q", "title", "status", "done_implies"]
|
||||
TSV_DELIMITER = "\t"
|
||||
TS_FMT = "%Y-%m-%dT%H:%M:%SZ"
|
||||
REV_RE = re.compile(r"^(?:[0-9a-f]{12}|unknown)$")
|
||||
SCHEMA = "workqueue-census-v3"
|
||||
SEP, DASH = "—", "–"
|
||||
|
||||
|
||||
def collapse(s):
|
||||
return re.sub(r"\s+", " ", s).strip()
|
||||
|
||||
|
||||
def _truncate_sentence(s):
|
||||
"""Collapse whitespace and cut at the first sentence end, one trailing '.'."""
|
||||
@@ -37,12 +95,19 @@ def _truncate_sentence(s):
|
||||
return s + "."
|
||||
|
||||
|
||||
def _parsed_line(line):
|
||||
return STATUS_RE.match(line), DONE_RE.match(line)
|
||||
|
||||
|
||||
def collapse(s):
|
||||
return re.sub(r"\s+", " ", s).strip()
|
||||
def record_only_ids(lines, heading_ids):
|
||||
"""Ids opened in bold-lead or status-lead form that have no heading."""
|
||||
seen = {}
|
||||
for line in lines:
|
||||
m = BOLD_LEAD.match(line)
|
||||
if m and m.group(1) not in heading_ids:
|
||||
if OPEN_AFTER.match(m.group(2)) and m.group(1) not in seen:
|
||||
seen[m.group(1)] = line
|
||||
continue
|
||||
m = STATUS_LEAD.match(line)
|
||||
if m and m.group(1) not in heading_ids and m.group(1) not in seen:
|
||||
seen[m.group(1)] = line
|
||||
return sorted(seen, key=lambda q: int(q[1:]))
|
||||
|
||||
|
||||
def parse_source(path):
|
||||
@@ -50,10 +115,12 @@ def parse_source(path):
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
lines = fh.read().splitlines()
|
||||
rows = []
|
||||
heading_ids = set()
|
||||
current = None
|
||||
for line in lines:
|
||||
m = HEADING_RE.match(line)
|
||||
if m:
|
||||
heading_ids.add("Q" + m.group(1))
|
||||
if current is not None:
|
||||
rows.append(current)
|
||||
current = {
|
||||
@@ -81,108 +148,316 @@ def parse_source(path):
|
||||
for q in qids:
|
||||
freq[q] = freq.get(q, 0) + 1
|
||||
reopened = sorted(q for q, n in freq.items() if n > 1)
|
||||
return rows, len(rows), len(set(qids)), reopened
|
||||
only = record_only_ids(lines, heading_ids)
|
||||
opened_all = sorted(heading_ids | set(only), key=lambda q: int(q[1:]))
|
||||
return rows, len(rows), len(set(qids)), reopened, only, opened_all
|
||||
|
||||
|
||||
def read_tsv(path):
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
lines = fh.read().splitlines()
|
||||
data = [line.split("\t") for line in lines]
|
||||
"""Read the TSV in the dialect the extractor writes (csv, tab-delimited).
|
||||
|
||||
QUOTE_MINIMAL is on, so a cell holding a `"` is quoted and its inner quotes
|
||||
doubled. Splitting on "\\t" (the pre-Q112 reader) cannot undo that.
|
||||
"""
|
||||
import csv
|
||||
|
||||
with open(path, "r", encoding="utf-8", newline="") as fh:
|
||||
data = list(csv.reader(fh, delimiter=TSV_DELIMITER))
|
||||
if not data:
|
||||
return [], []
|
||||
return data[0], data[1:]
|
||||
|
||||
|
||||
def main():
|
||||
problems = []
|
||||
def read_tsv_naive(path):
|
||||
"""The pre-Q112 reader, kept so the selftest can measure what it cost."""
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
data = [line.split(TSV_DELIMITER) for line in fh.read().splitlines()]
|
||||
return data[0], data[1:]
|
||||
|
||||
|
||||
def is_rev(value):
|
||||
return isinstance(value, str) and bool(REV_RE.match(value))
|
||||
|
||||
|
||||
def _brief(value, width=120):
|
||||
text = repr(value)
|
||||
return text if len(text) <= width else text[:width] + "..."
|
||||
|
||||
|
||||
def _parses_stamp(value):
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
import datetime
|
||||
|
||||
# 1. Source must be readable.
|
||||
try:
|
||||
src_rows, n_headings, n_distinct, reopened = parse_source(SRC)
|
||||
datetime.datetime.strptime(value, TS_FMT)
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _git_head(root):
|
||||
try:
|
||||
out = subprocess.run(["git", "-C", root, "rev-parse", "--short=12", "HEAD"],
|
||||
capture_output=True, text=True)
|
||||
if out.returncode == 0:
|
||||
return out.stdout.strip()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return "unknown"
|
||||
|
||||
|
||||
def grade(root, artifact, reader=None, say=print):
|
||||
"""Grade one census directory against the tree it claims to extract."""
|
||||
reader = reader or read_tsv
|
||||
problems = []
|
||||
src = os.path.join(root, "WORK-QUEUE.md")
|
||||
tsv_path = os.path.join(artifact, "workqueue-census.tsv")
|
||||
man_path = os.path.join(artifact, "extract_manifest.json")
|
||||
|
||||
try:
|
||||
(src_rows, n_headings, n_distinct, reopened,
|
||||
record_only, opened_all) = parse_source(src)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"FAIL cannot read source {SRC}: {exc}")
|
||||
say("CANNOT GRADE source %s: %s" % (src, exc))
|
||||
return 2
|
||||
|
||||
# 2. TSV must exist and match the re-derivation.
|
||||
if not os.path.exists(TSV_NAME):
|
||||
print(f"FAIL missing TSV {TSV_NAME}")
|
||||
if not os.path.exists(tsv_path):
|
||||
say("FAIL missing TSV %s" % tsv_path)
|
||||
return 1
|
||||
try:
|
||||
header, tsv_rows = read_tsv(TSV_NAME)
|
||||
if header != TSV_HEADER:
|
||||
problems.append(f"header {header} != {TSV_HEADER}")
|
||||
if len(tsv_rows) != n_headings:
|
||||
problems.append(f"TSV rows {len(tsv_rows)} != headings {n_headings}")
|
||||
for i, (expect, actual) in enumerate(zip(src_rows, tsv_rows)):
|
||||
if len(actual) != 4:
|
||||
problems.append(f"row {i} has {len(actual)} fields: {actual}")
|
||||
continue
|
||||
if actual[0] != expect["q"]:
|
||||
problems.append(f"row {i} q {actual[0]} != {expect['q']}")
|
||||
if actual[1] != expect["title"]:
|
||||
problems.append(f"row {i} title mismatch")
|
||||
if actual[2] != expect["status"]:
|
||||
problems.append(f"row {i} status mismatch")
|
||||
if actual[3] != expect["done_implies"]:
|
||||
problems.append(f"row {i} done_implies mismatch")
|
||||
header, tsv_rows = reader(tsv_path)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"FAIL TSV read error: {exc}")
|
||||
return 1
|
||||
say("CANNOT GRADE TSV %s: %s" % (tsv_path, exc))
|
||||
return 2
|
||||
|
||||
# 3. Manifest must agree with an independent recomputation.
|
||||
if not os.path.exists(MANIFEST_NAME):
|
||||
print(f"FAIL missing manifest {MANIFEST_NAME}")
|
||||
# --- leg: the rows reproduce the source walk, cell for cell -------------
|
||||
cell_faults = 0
|
||||
if header != TSV_HEADER:
|
||||
problems.append("header %r != %r" % (header, TSV_HEADER))
|
||||
cell_faults += 1
|
||||
if len(tsv_rows) != n_headings:
|
||||
problems.append("TSV rows %d != headings %d" % (len(tsv_rows), n_headings))
|
||||
cell_faults += 1
|
||||
for i, (expect, actual) in enumerate(zip(src_rows, tsv_rows)):
|
||||
if len(actual) != 4:
|
||||
problems.append("row %d has %d fields" % (i, len(actual)))
|
||||
cell_faults += 1
|
||||
continue
|
||||
for col, key in enumerate(TSV_HEADER):
|
||||
if actual[col] != expect[key]:
|
||||
problems.append("row %d (%s) %s mismatch" % (i, expect["q"], key))
|
||||
cell_faults += 1
|
||||
say("%s rows: %d heading(s), %d TSV row(s), %d cell fault(s) on the %d "
|
||||
"re-derived row(s)"
|
||||
% ("PASS" if cell_faults == 0 else "FAIL", n_headings, len(tsv_rows),
|
||||
cell_faults, len(src_rows)))
|
||||
|
||||
# --- leg: the manifest reconciles with the recomputation above ----------
|
||||
if not os.path.exists(man_path):
|
||||
say("FAIL missing manifest %s" % man_path)
|
||||
return 1
|
||||
with open(MANIFEST_NAME, "r", encoding="utf-8") as fh:
|
||||
manifest = json.load(fh)
|
||||
try:
|
||||
with open(man_path, "r", encoding="utf-8") as fh:
|
||||
manifest = json.load(fh)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
say("CANNOT GRADE manifest %s: %s" % (man_path, exc))
|
||||
return 2
|
||||
|
||||
legs = [
|
||||
("schema", manifest.get("schema"), SCHEMA),
|
||||
("rows", manifest.get("rows"), n_headings),
|
||||
("distinct_q_labels", manifest.get("distinct_q_labels"), n_distinct),
|
||||
("reopened_q_labels", manifest.get("reopened_q_labels"), reopened),
|
||||
("record_only_ids", manifest.get("record_only_ids"), record_only),
|
||||
("ids_opened_total", manifest.get("ids_opened_total"), len(opened_all)),
|
||||
]
|
||||
bad_legs = 0
|
||||
for name, got, want in legs:
|
||||
if got != want:
|
||||
problems.append(f"manifest {name} {got} != recomputed {want}")
|
||||
problems.append("manifest %s %r != recomputed %r"
|
||||
% (name, _brief(got), _brief(want)))
|
||||
bad_legs += 1
|
||||
say("%s manifest: %d of %d leg(s) reconciled, %d fault(s) (%d record-form-only "
|
||||
"id(s), %d id(s) opened in any form)"
|
||||
% ("PASS" if bad_legs == 0 else "FAIL", len(legs) - bad_legs, len(legs),
|
||||
bad_legs, len(record_only), len(opened_all)))
|
||||
|
||||
# 4. Revision leg (SKIP): the manifest pins the HEAD at generation time, but
|
||||
# the repo is pervasively edited by other lanes, so HEAD moves; a mismatch
|
||||
# therefore skips rather than fails. The census itself is revision-free.
|
||||
try:
|
||||
out = subprocess_run(["git", "-C", REPO_ROOT, "rev-parse", "HEAD"])
|
||||
if out.returncode == 0:
|
||||
head = out.stdout.strip()[:12]
|
||||
if manifest.get("revision") != head:
|
||||
print(
|
||||
f"SKIP revision leg (manifest {manifest.get('revision')} != HEAD {head})"
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
# --- leg: provenance that cannot decay ---------------------------------
|
||||
prov_faults = 0
|
||||
if not _parses_stamp(manifest.get("extracted_at_utc")):
|
||||
problems.append("extracted_at_utc %r is not %s"
|
||||
% (manifest.get("extracted_at_utc"), TS_FMT))
|
||||
prov_faults += 1
|
||||
if not is_rev(manifest.get("revision")):
|
||||
problems.append("revision %r is not 12 hex digits or 'unknown'"
|
||||
% (manifest.get("revision"),))
|
||||
prov_faults += 1
|
||||
say("%s provenance: %d fault(s) (stamp shape %s, revision shape %s)"
|
||||
% ("PASS" if prov_faults == 0 else "FAIL", prov_faults,
|
||||
"ok" if _parses_stamp(manifest.get("extracted_at_utc")) else "bad",
|
||||
"ok" if is_rev(manifest.get("revision")) else "bad"))
|
||||
say("INFO manifest revision=%s head=%s -- revision equality is not graded: the "
|
||||
"queue is appended to by other lanes between generation and reading"
|
||||
% (manifest.get("revision"), _git_head(root)))
|
||||
|
||||
# 5. Sanity legs on the TSV itself (no empty q/title, no literal tabs).
|
||||
# --- leg: TSV sanity (no empty q/title) --------------------------------
|
||||
for i, row in enumerate(tsv_rows, start=1):
|
||||
if len(row) < 4:
|
||||
problems.append(f"row {i} too short")
|
||||
if row[0].strip() == "":
|
||||
problems.append(f"row {i} empty q")
|
||||
if row[1].strip() == "":
|
||||
problems.append(f"row {i} empty title")
|
||||
if "\t" in row[1]:
|
||||
problems.append(f"row {i} title contains tab")
|
||||
problems.append("row %d too short" % i)
|
||||
elif row[0].strip() == "":
|
||||
problems.append("row %d empty q" % i)
|
||||
elif row[1].strip() == "":
|
||||
problems.append("row %d empty title" % i)
|
||||
|
||||
if problems:
|
||||
print(f"FAIL ({len(problems)} problems)")
|
||||
say("FAIL (%d problem(s))" % len(problems))
|
||||
for p in problems:
|
||||
print(f" - {p}")
|
||||
say(" - %s" % p)
|
||||
return 1
|
||||
|
||||
print(f"PASS Q count {n_headings}, distinct {n_distinct}, reopened {len(reopened)}")
|
||||
print(f"PASS source revision {manifest.get('revision')} matches HEAD")
|
||||
print(f"PASS manifest reconciled with independent re-derivation")
|
||||
say("PASS census: %d rows, %d distinct labels, %d reopened, %d record-form only"
|
||||
% (n_headings, n_distinct, len(reopened), len(record_only)))
|
||||
return 0
|
||||
|
||||
|
||||
def subprocess_run(cmd):
|
||||
import subprocess
|
||||
# --------------------------------------------------------------------------
|
||||
# selftest: build fixture trees, mutate one thing per cell, grade each.
|
||||
# --------------------------------------------------------------------------
|
||||
FIXTURE_QUEUE = """# fixture queue
|
||||
|
||||
return subprocess.run(cmd, capture_output=True, text=True)
|
||||
## Q1 — first item
|
||||
|
||||
**Status:** done at `abc1234`.
|
||||
|
||||
**Done means:** the first cell reads back.
|
||||
|
||||
## Q2 — second item — the "pin" class
|
||||
|
||||
**Status:** open and unclaimed.
|
||||
|
||||
**Done means:** the second cell reads back.
|
||||
|
||||
## Q3 — third item
|
||||
|
||||
**Status:** open.
|
||||
|
||||
**Done means:** the third cell reads back.
|
||||
|
||||
**Q4 — OPEN (fixture lane; bold-lead record, no heading).**
|
||||
|
||||
**Q5 opened — another record-form id.**
|
||||
"""
|
||||
|
||||
|
||||
def _write(path, text):
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8", newline="") as fh:
|
||||
fh.write(text)
|
||||
|
||||
|
||||
def _read(path):
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
return fh.read()
|
||||
|
||||
|
||||
def _build_tree(tmp, name, queue_text):
|
||||
"""A tree the shipped resolution recognises, with the extractor beside it."""
|
||||
root = os.path.join(tmp, name)
|
||||
art = os.path.join(root, ARTIFACT_REL)
|
||||
os.makedirs(art)
|
||||
_write(os.path.join(root, "WORK-QUEUE.md"), queue_text)
|
||||
shutil.copy(os.path.join(HERE, "extract_workqueue_census.py"), art)
|
||||
env = dict(os.environ, REPO_ROOT=root)
|
||||
out = subprocess.run([sys.executable, "extract_workqueue_census.py"], cwd=art,
|
||||
env=env, capture_output=True, text=True)
|
||||
if out.returncode != 0:
|
||||
raise RuntimeError("extractor failed in %s: %s" % (root, out.stderr.strip()))
|
||||
return root, art
|
||||
|
||||
|
||||
def selftest(say=print):
|
||||
cells = []
|
||||
tmp = tempfile.mkdtemp(prefix="census-selftest-")
|
||||
quiet = lambda *a: None # noqa: E731
|
||||
try:
|
||||
# cell 0 -- control: the fixture, generated by the extractor, is green.
|
||||
root, art = _build_tree(tmp, "control", FIXTURE_QUEUE)
|
||||
cells.append(("control-green", 0, grade(root, art, say=quiet)))
|
||||
|
||||
# cell 1 -- a quoted cell reads back intact; the same tree under the
|
||||
# pre-Q112 reader is the falsification, not an assertion about it.
|
||||
root1, art1 = _build_tree(tmp, "quote", FIXTURE_QUEUE)
|
||||
cells.append(("quote-cell-green-with-csv-reader", 0,
|
||||
grade(root1, art1, say=quiet)))
|
||||
cells.append(("quote-cell-red-with-naive-reader", 1,
|
||||
grade(root1, art1, reader=read_tsv_naive, say=quiet)))
|
||||
|
||||
# cell 2 -- a heading deleted from the source, artifact untouched.
|
||||
root2, art2 = _build_tree(tmp, "no-heading", FIXTURE_QUEUE)
|
||||
q2 = _read(os.path.join(root2, "WORK-QUEUE.md"))
|
||||
q2 = q2.replace('## Q2 \u2014 second item \u2014 the "pin" class\n', "")
|
||||
_write(os.path.join(root2, "WORK-QUEUE.md"), q2)
|
||||
cells.append(("deleted-heading-red", 1, grade(root2, art2, say=quiet)))
|
||||
|
||||
# cell 3 -- one title cell corrupted in the TSV.
|
||||
root3, art3 = _build_tree(tmp, "bad-cell", FIXTURE_QUEUE)
|
||||
tsv3 = os.path.join(art3, "workqueue-census.tsv")
|
||||
_write(tsv3, _read(tsv3).replace("\u2014 first item", "\u2014 FIRST item"))
|
||||
cells.append(("corrupted-cell-red", 1, grade(root3, art3, say=quiet)))
|
||||
|
||||
# cell 4 -- a new record-form id in the source, manifest not regenerated.
|
||||
root4, art4 = _build_tree(tmp, "new-record-id", FIXTURE_QUEUE)
|
||||
with open(os.path.join(root4, "WORK-QUEUE.md"), "a", encoding="utf-8") as fh:
|
||||
fh.write("\n**Q6 \u2014 OPEN (fixture lane; a second record-form id).**\n")
|
||||
cells.append(("new-record-form-id-red", 1, grade(root4, art4, say=quiet)))
|
||||
|
||||
# cell 5 -- the manifest's record_only_ids loses one entry.
|
||||
root5, art5 = _build_tree(tmp, "short-record-ids", FIXTURE_QUEUE)
|
||||
man5 = os.path.join(art5, "extract_manifest.json")
|
||||
blob = json.loads(_read(man5))
|
||||
blob["record_only_ids"] = blob["record_only_ids"][:-1]
|
||||
_write(man5, json.dumps(blob, indent=2) + "\n")
|
||||
cells.append(("manifest-record-ids-short-red", 1,
|
||||
grade(root5, art5, say=quiet)))
|
||||
|
||||
# cell 6 -- an unparseable provenance stamp.
|
||||
root6, art6 = _build_tree(tmp, "bad-stamp", FIXTURE_QUEUE)
|
||||
man6 = os.path.join(art6, "extract_manifest.json")
|
||||
blob = json.loads(_read(man6))
|
||||
blob["extracted_at_utc"] = "yesterday"
|
||||
_write(man6, json.dumps(blob, indent=2) + "\n")
|
||||
cells.append(("manifest-stamp-shape-red", 1, grade(root6, art6, say=quiet)))
|
||||
|
||||
# cell 7 -- a revision that is not a revision (the leg that used to
|
||||
# print PASS without measuring anything).
|
||||
root7, art7 = _build_tree(tmp, "bad-rev", FIXTURE_QUEUE)
|
||||
man7 = os.path.join(art7, "extract_manifest.json")
|
||||
blob = json.loads(_read(man7))
|
||||
blob["revision"] = "HEAD~1"
|
||||
_write(man7, json.dumps(blob, indent=2) + "\n")
|
||||
cells.append(("manifest-revision-shape-red", 1, grade(root7, art7, say=quiet)))
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
failed = 0
|
||||
for name, want, got in cells:
|
||||
ok = want == got
|
||||
failed += 0 if ok else 1
|
||||
say("%s %s (want rc %d, got rc %d)" % ("PASS" if ok else "FAIL", name, want, got))
|
||||
say("SELFTEST %s (%d cells, %d failed)"
|
||||
% ("OK" if not failed else "FAILED", len(cells), failed))
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description="grade a workqueue census against its tree")
|
||||
ap.add_argument("--root", default=DEFAULT_ROOT, help="tree holding WORK-QUEUE.md")
|
||||
ap.add_argument("--artifact", default=HERE, help="census directory to grade")
|
||||
ap.add_argument("--selftest", action="store_true", help="mutate fixture trees")
|
||||
args = ap.parse_args(argv)
|
||||
if args.selftest:
|
||||
return selftest()
|
||||
return grade(os.path.abspath(args.root), os.path.abspath(args.artifact))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
README — workqueue-census
|
||||
========================
|
||||
|
||||
Extract lane: data extraction and tables. One row per `## Q<n>` heading in
|
||||
WORK-QUEUE.md.
|
||||
|
||||
Files
|
||||
-----
|
||||
- workqueue-census.tsv : tab-separated census (one row per Q heading).
|
||||
- workqueue-census.md : markdown table of the same rows.
|
||||
- extract_manifest.json: schema, counts, provenance.
|
||||
- CHECK : independent verifier (re-derives expected rows
|
||||
from WORK-QUEUE.md; does NOT import this extractor).
|
||||
|
||||
Conventions
|
||||
-----------
|
||||
- A row is created for every line matching '## <Qnum> ...' where <Qnum> is
|
||||
'Q' followed by digits (continuation labels such as closure, section 9
|
||||
follow-on, verified, claimed are kept verbatim in the title).
|
||||
- q : the label 'Q<num>' (dupes kept as separate rows).
|
||||
- title : verbatim heading text after '## <Qnum>' (trimmed).
|
||||
- status : the first '**Status:' line in the block, whitespace collapsed,
|
||||
truncated at the first sentence end, single trailing period; empty when no
|
||||
such marker precedes the next heading.
|
||||
- done_implies : the first '**Done means:' line in the block, trimmed.
|
||||
- record_only_ids : ids that open a record in bold-lead form (`**Qnn -- ...` or
|
||||
`**Qnn (lane) -- OPEN`) and have no '## Qnn' heading. A heading-only walk
|
||||
cannot see them, so no table row is created for one (it has no heading title
|
||||
and no block bounds); the manifest carries the list and the markdown table
|
||||
states the coverage it implies.
|
||||
- ids_opened_total : how many distinct ids the queue opens in any form, so
|
||||
rows x record_only_ids x ids_opened_total reconcile in CHECK.
|
||||
|
||||
Run
|
||||
---
|
||||
python3 extract_workqueue_census.py # regenerate
|
||||
python3 CHECK # verify against the source
|
||||
@@ -5,26 +5,84 @@ RE-APPROACH: this checker does NOT import the extractor. It re-derives the
|
||||
expected rows from WORK-QUEUE.md with plain string operations (split on
|
||||
whitespace, HEADING_RE) and compares them to the committed TSV and manifest.
|
||||
|
||||
TWO FAULTS THIS FILE CARRIED, AND THEIR REPAIR (Q112, 2026-10-29)
|
||||
1. THE READER. `read_tsv()` split each line on "\\t". The extractor writes
|
||||
the TSV with `csv.writer(delimiter="\\t")`, whose QUOTE_MINIMAL dialect
|
||||
wraps a cell containing a `"` in double quotes and doubles the inner
|
||||
quote (`"the ""open"" line"`). A naive split therefore handed the
|
||||
comparison a leading `"` and an un-doubled tail, and the committed
|
||||
artifact read 11 false faults at HEAD -- rows 43, 45, 52, 60, 64, 77,
|
||||
81, 84, 87, 88, 96, measured to be exactly the cells whose true value
|
||||
holds a quote. The reader now parses the dialect the writer writes;
|
||||
`read_tsv_naive()` keeps the old reader so the selftest can measure what
|
||||
it cost instead of asserting that it would have cost it.
|
||||
2. THE TAIL LINE. The revision leg printed `PASS source revision ...
|
||||
matches HEAD` unconditionally -- including on runs where it had just
|
||||
printed `SKIP revision leg`. A leg that did not run was reported with the
|
||||
same word as one that did. Revision is now graded as a FORMAT fact
|
||||
(12 hex digits or the literal `unknown`) and both values are printed on
|
||||
one INFO line; nothing here decays with the wall clock or with HEAD's
|
||||
movement.
|
||||
3. THE SAME CLASS, CAUGHT IN THIS REPAIR'S OWN FIRST DRAFT: that draft's row
|
||||
leg printed `PASS rows: ... 1 cell fault(s)` -- the count contradicting
|
||||
the word -- as measured on a seeded cell mutation before it shipped. Every
|
||||
leg line here now takes its word from the counter it prints, and lists
|
||||
the denominators beside it.
|
||||
|
||||
THE LEG Q112 ASKED FOR: `record_only_ids`. Some ids open a record in bold-lead
|
||||
form (`**Qnn -- ...**`) and never appear as a `## Qnn` heading, so a
|
||||
heading-only walk cannot see them: at this head the queue opens 122 ids, the
|
||||
heading walk sees 86, and 36 are record-form only. The extractor and this
|
||||
checker each compute the set from their own scan; the manifest must carry it
|
||||
verbatim, and the count of ids opened in any form must reconcile with it.
|
||||
|
||||
Verdict: the script's exit code. 0 = correct, 1 = a check failed, 2 = could not
|
||||
grade (missing artifact / bad invocation).
|
||||
|
||||
Usage:
|
||||
python3 CHECK # grade this directory against its tree
|
||||
python3 CHECK --root DIR # grade a tree other than the one this sits in
|
||||
python3 CHECK --artifact DIR # grade a census directory other than this one
|
||||
python3 CHECK --selftest # build fixture trees and mutate one thing each
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
REPO_ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
|
||||
SRC = os.path.join(REPO_ROOT, "WORK-QUEUE.md")
|
||||
TSV_NAME = os.path.join(HERE, "workqueue-census.tsv")
|
||||
MANIFEST_NAME = os.path.join(HERE, "extract_manifest.json")
|
||||
DEFAULT_ROOT = os.path.abspath(os.path.join(HERE, "..", "..", ".."))
|
||||
ARTIFACT_REL = os.path.join("wizard-work", "ledger", "2026-10-28-workqueue-census")
|
||||
|
||||
HEADING_RE = re.compile(r"^##\s+Q(\d+)(.*)$")
|
||||
TSV_HEADER = ["q", "title", "status", "done_implies"]
|
||||
STATUS_RE = re.compile(r"^\*\*Status:\s*(.*)$")
|
||||
DONE_RE = re.compile(r"^\*\*Done means:\s*(.*)$")
|
||||
|
||||
# Record-open forms that are NOT headings (the rule stated in WORK-QUEUE.md at
|
||||
# Q63: an id is opened by a record, not mentioned by prose). A bold run opens
|
||||
# an id only when the id is the first token after the bullet/bold markers and
|
||||
# the next token is a separator or the word open/opened -- `**Q7, measured:**`
|
||||
# is a mention and must not count.
|
||||
BOLD_LEAD = re.compile(r"^[\s>|]*(?:[-*+]+\s+)?\*\*(Q\d+)(.*)$")
|
||||
OPEN_AFTER = re.compile(r"^\s*(?:—|–|-|:|\(|\bopened\b|\bopen\b|\bOPEN\b|\bOPENED\b)")
|
||||
STATUS_LEAD = re.compile(r"^\s*\*{0,2}Status:\s*(Q\d+)\b")
|
||||
|
||||
TSV_HEADER = ["q", "title", "status", "done_implies"]
|
||||
TSV_DELIMITER = "\t"
|
||||
TS_FMT = "%Y-%m-%dT%H:%M:%SZ"
|
||||
REV_RE = re.compile(r"^(?:[0-9a-f]{12}|unknown)$")
|
||||
SCHEMA = "workqueue-census-v3"
|
||||
SEP, DASH = "—", "–"
|
||||
|
||||
|
||||
def collapse(s):
|
||||
return re.sub(r"\s+", " ", s).strip()
|
||||
|
||||
|
||||
def _truncate_sentence(s):
|
||||
"""Collapse whitespace and cut at the first sentence end, one trailing '.'."""
|
||||
@@ -37,12 +95,19 @@ def _truncate_sentence(s):
|
||||
return s + "."
|
||||
|
||||
|
||||
def _parsed_line(line):
|
||||
return STATUS_RE.match(line), DONE_RE.match(line)
|
||||
|
||||
|
||||
def collapse(s):
|
||||
return re.sub(r"\s+", " ", s).strip()
|
||||
def record_only_ids(lines, heading_ids):
|
||||
"""Ids opened in bold-lead or status-lead form that have no heading."""
|
||||
seen = {}
|
||||
for line in lines:
|
||||
m = BOLD_LEAD.match(line)
|
||||
if m and m.group(1) not in heading_ids:
|
||||
if OPEN_AFTER.match(m.group(2)) and m.group(1) not in seen:
|
||||
seen[m.group(1)] = line
|
||||
continue
|
||||
m = STATUS_LEAD.match(line)
|
||||
if m and m.group(1) not in heading_ids and m.group(1) not in seen:
|
||||
seen[m.group(1)] = line
|
||||
return sorted(seen, key=lambda q: int(q[1:]))
|
||||
|
||||
|
||||
def parse_source(path):
|
||||
@@ -50,10 +115,12 @@ def parse_source(path):
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
lines = fh.read().splitlines()
|
||||
rows = []
|
||||
heading_ids = set()
|
||||
current = None
|
||||
for line in lines:
|
||||
m = HEADING_RE.match(line)
|
||||
if m:
|
||||
heading_ids.add("Q" + m.group(1))
|
||||
if current is not None:
|
||||
rows.append(current)
|
||||
current = {
|
||||
@@ -81,108 +148,316 @@ def parse_source(path):
|
||||
for q in qids:
|
||||
freq[q] = freq.get(q, 0) + 1
|
||||
reopened = sorted(q for q, n in freq.items() if n > 1)
|
||||
return rows, len(rows), len(set(qids)), reopened
|
||||
only = record_only_ids(lines, heading_ids)
|
||||
opened_all = sorted(heading_ids | set(only), key=lambda q: int(q[1:]))
|
||||
return rows, len(rows), len(set(qids)), reopened, only, opened_all
|
||||
|
||||
|
||||
def read_tsv(path):
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
lines = fh.read().splitlines()
|
||||
data = [line.split("\t") for line in lines]
|
||||
"""Read the TSV in the dialect the extractor writes (csv, tab-delimited).
|
||||
|
||||
QUOTE_MINIMAL is on, so a cell holding a `"` is quoted and its inner quotes
|
||||
doubled. Splitting on "\\t" (the pre-Q112 reader) cannot undo that.
|
||||
"""
|
||||
import csv
|
||||
|
||||
with open(path, "r", encoding="utf-8", newline="") as fh:
|
||||
data = list(csv.reader(fh, delimiter=TSV_DELIMITER))
|
||||
if not data:
|
||||
return [], []
|
||||
return data[0], data[1:]
|
||||
|
||||
|
||||
def main():
|
||||
problems = []
|
||||
def read_tsv_naive(path):
|
||||
"""The pre-Q112 reader, kept so the selftest can measure what it cost."""
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
data = [line.split(TSV_DELIMITER) for line in fh.read().splitlines()]
|
||||
return data[0], data[1:]
|
||||
|
||||
|
||||
def is_rev(value):
|
||||
return isinstance(value, str) and bool(REV_RE.match(value))
|
||||
|
||||
|
||||
def _brief(value, width=120):
|
||||
text = repr(value)
|
||||
return text if len(text) <= width else text[:width] + "..."
|
||||
|
||||
|
||||
def _parses_stamp(value):
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
import datetime
|
||||
|
||||
# 1. Source must be readable.
|
||||
try:
|
||||
src_rows, n_headings, n_distinct, reopened = parse_source(SRC)
|
||||
datetime.datetime.strptime(value, TS_FMT)
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _git_head(root):
|
||||
try:
|
||||
out = subprocess.run(["git", "-C", root, "rev-parse", "--short=12", "HEAD"],
|
||||
capture_output=True, text=True)
|
||||
if out.returncode == 0:
|
||||
return out.stdout.strip()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return "unknown"
|
||||
|
||||
|
||||
def grade(root, artifact, reader=None, say=print):
|
||||
"""Grade one census directory against the tree it claims to extract."""
|
||||
reader = reader or read_tsv
|
||||
problems = []
|
||||
src = os.path.join(root, "WORK-QUEUE.md")
|
||||
tsv_path = os.path.join(artifact, "workqueue-census.tsv")
|
||||
man_path = os.path.join(artifact, "extract_manifest.json")
|
||||
|
||||
try:
|
||||
(src_rows, n_headings, n_distinct, reopened,
|
||||
record_only, opened_all) = parse_source(src)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"FAIL cannot read source {SRC}: {exc}")
|
||||
say("CANNOT GRADE source %s: %s" % (src, exc))
|
||||
return 2
|
||||
|
||||
# 2. TSV must exist and match the re-derivation.
|
||||
if not os.path.exists(TSV_NAME):
|
||||
print(f"FAIL missing TSV {TSV_NAME}")
|
||||
if not os.path.exists(tsv_path):
|
||||
say("FAIL missing TSV %s" % tsv_path)
|
||||
return 1
|
||||
try:
|
||||
header, tsv_rows = read_tsv(TSV_NAME)
|
||||
if header != TSV_HEADER:
|
||||
problems.append(f"header {header} != {TSV_HEADER}")
|
||||
if len(tsv_rows) != n_headings:
|
||||
problems.append(f"TSV rows {len(tsv_rows)} != headings {n_headings}")
|
||||
for i, (expect, actual) in enumerate(zip(src_rows, tsv_rows)):
|
||||
if len(actual) != 4:
|
||||
problems.append(f"row {i} has {len(actual)} fields: {actual}")
|
||||
continue
|
||||
if actual[0] != expect["q"]:
|
||||
problems.append(f"row {i} q {actual[0]} != {expect['q']}")
|
||||
if actual[1] != expect["title"]:
|
||||
problems.append(f"row {i} title mismatch")
|
||||
if actual[2] != expect["status"]:
|
||||
problems.append(f"row {i} status mismatch")
|
||||
if actual[3] != expect["done_implies"]:
|
||||
problems.append(f"row {i} done_implies mismatch")
|
||||
header, tsv_rows = reader(tsv_path)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"FAIL TSV read error: {exc}")
|
||||
return 1
|
||||
say("CANNOT GRADE TSV %s: %s" % (tsv_path, exc))
|
||||
return 2
|
||||
|
||||
# 3. Manifest must agree with an independent recomputation.
|
||||
if not os.path.exists(MANIFEST_NAME):
|
||||
print(f"FAIL missing manifest {MANIFEST_NAME}")
|
||||
# --- leg: the rows reproduce the source walk, cell for cell -------------
|
||||
cell_faults = 0
|
||||
if header != TSV_HEADER:
|
||||
problems.append("header %r != %r" % (header, TSV_HEADER))
|
||||
cell_faults += 1
|
||||
if len(tsv_rows) != n_headings:
|
||||
problems.append("TSV rows %d != headings %d" % (len(tsv_rows), n_headings))
|
||||
cell_faults += 1
|
||||
for i, (expect, actual) in enumerate(zip(src_rows, tsv_rows)):
|
||||
if len(actual) != 4:
|
||||
problems.append("row %d has %d fields" % (i, len(actual)))
|
||||
cell_faults += 1
|
||||
continue
|
||||
for col, key in enumerate(TSV_HEADER):
|
||||
if actual[col] != expect[key]:
|
||||
problems.append("row %d (%s) %s mismatch" % (i, expect["q"], key))
|
||||
cell_faults += 1
|
||||
say("%s rows: %d heading(s), %d TSV row(s), %d cell fault(s) on the %d "
|
||||
"re-derived row(s)"
|
||||
% ("PASS" if cell_faults == 0 else "FAIL", n_headings, len(tsv_rows),
|
||||
cell_faults, len(src_rows)))
|
||||
|
||||
# --- leg: the manifest reconciles with the recomputation above ----------
|
||||
if not os.path.exists(man_path):
|
||||
say("FAIL missing manifest %s" % man_path)
|
||||
return 1
|
||||
with open(MANIFEST_NAME, "r", encoding="utf-8") as fh:
|
||||
manifest = json.load(fh)
|
||||
try:
|
||||
with open(man_path, "r", encoding="utf-8") as fh:
|
||||
manifest = json.load(fh)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
say("CANNOT GRADE manifest %s: %s" % (man_path, exc))
|
||||
return 2
|
||||
|
||||
legs = [
|
||||
("schema", manifest.get("schema"), SCHEMA),
|
||||
("rows", manifest.get("rows"), n_headings),
|
||||
("distinct_q_labels", manifest.get("distinct_q_labels"), n_distinct),
|
||||
("reopened_q_labels", manifest.get("reopened_q_labels"), reopened),
|
||||
("record_only_ids", manifest.get("record_only_ids"), record_only),
|
||||
("ids_opened_total", manifest.get("ids_opened_total"), len(opened_all)),
|
||||
]
|
||||
bad_legs = 0
|
||||
for name, got, want in legs:
|
||||
if got != want:
|
||||
problems.append(f"manifest {name} {got} != recomputed {want}")
|
||||
problems.append("manifest %s %r != recomputed %r"
|
||||
% (name, _brief(got), _brief(want)))
|
||||
bad_legs += 1
|
||||
say("%s manifest: %d of %d leg(s) reconciled, %d fault(s) (%d record-form-only "
|
||||
"id(s), %d id(s) opened in any form)"
|
||||
% ("PASS" if bad_legs == 0 else "FAIL", len(legs) - bad_legs, len(legs),
|
||||
bad_legs, len(record_only), len(opened_all)))
|
||||
|
||||
# 4. Revision leg (SKIP): the manifest pins the HEAD at generation time, but
|
||||
# the repo is pervasively edited by other lanes, so HEAD moves; a mismatch
|
||||
# therefore skips rather than fails. The census itself is revision-free.
|
||||
try:
|
||||
out = subprocess_run(["git", "-C", REPO_ROOT, "rev-parse", "HEAD"])
|
||||
if out.returncode == 0:
|
||||
head = out.stdout.strip()[:12]
|
||||
if manifest.get("revision") != head:
|
||||
print(
|
||||
f"SKIP revision leg (manifest {manifest.get('revision')} != HEAD {head})"
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
# --- leg: provenance that cannot decay ---------------------------------
|
||||
prov_faults = 0
|
||||
if not _parses_stamp(manifest.get("extracted_at_utc")):
|
||||
problems.append("extracted_at_utc %r is not %s"
|
||||
% (manifest.get("extracted_at_utc"), TS_FMT))
|
||||
prov_faults += 1
|
||||
if not is_rev(manifest.get("revision")):
|
||||
problems.append("revision %r is not 12 hex digits or 'unknown'"
|
||||
% (manifest.get("revision"),))
|
||||
prov_faults += 1
|
||||
say("%s provenance: %d fault(s) (stamp shape %s, revision shape %s)"
|
||||
% ("PASS" if prov_faults == 0 else "FAIL", prov_faults,
|
||||
"ok" if _parses_stamp(manifest.get("extracted_at_utc")) else "bad",
|
||||
"ok" if is_rev(manifest.get("revision")) else "bad"))
|
||||
say("INFO manifest revision=%s head=%s -- revision equality is not graded: the "
|
||||
"queue is appended to by other lanes between generation and reading"
|
||||
% (manifest.get("revision"), _git_head(root)))
|
||||
|
||||
# 5. Sanity legs on the TSV itself (no empty q/title, no literal tabs).
|
||||
# --- leg: TSV sanity (no empty q/title) --------------------------------
|
||||
for i, row in enumerate(tsv_rows, start=1):
|
||||
if len(row) < 4:
|
||||
problems.append(f"row {i} too short")
|
||||
if row[0].strip() == "":
|
||||
problems.append(f"row {i} empty q")
|
||||
if row[1].strip() == "":
|
||||
problems.append(f"row {i} empty title")
|
||||
if "\t" in row[1]:
|
||||
problems.append(f"row {i} title contains tab")
|
||||
problems.append("row %d too short" % i)
|
||||
elif row[0].strip() == "":
|
||||
problems.append("row %d empty q" % i)
|
||||
elif row[1].strip() == "":
|
||||
problems.append("row %d empty title" % i)
|
||||
|
||||
if problems:
|
||||
print(f"FAIL ({len(problems)} problems)")
|
||||
say("FAIL (%d problem(s))" % len(problems))
|
||||
for p in problems:
|
||||
print(f" - {p}")
|
||||
say(" - %s" % p)
|
||||
return 1
|
||||
|
||||
print(f"PASS Q count {n_headings}, distinct {n_distinct}, reopened {len(reopened)}")
|
||||
print(f"PASS source revision {manifest.get('revision')} matches HEAD")
|
||||
print(f"PASS manifest reconciled with independent re-derivation")
|
||||
say("PASS census: %d rows, %d distinct labels, %d reopened, %d record-form only"
|
||||
% (n_headings, n_distinct, len(reopened), len(record_only)))
|
||||
return 0
|
||||
|
||||
|
||||
def subprocess_run(cmd):
|
||||
import subprocess
|
||||
# --------------------------------------------------------------------------
|
||||
# selftest: build fixture trees, mutate one thing per cell, grade each.
|
||||
# --------------------------------------------------------------------------
|
||||
FIXTURE_QUEUE = """# fixture queue
|
||||
|
||||
return subprocess.run(cmd, capture_output=True, text=True)
|
||||
## Q1 — first item
|
||||
|
||||
**Status:** done at `abc1234`.
|
||||
|
||||
**Done means:** the first cell reads back.
|
||||
|
||||
## Q2 — second item — the "pin" class
|
||||
|
||||
**Status:** open and unclaimed.
|
||||
|
||||
**Done means:** the second cell reads back.
|
||||
|
||||
## Q3 — third item
|
||||
|
||||
**Status:** open.
|
||||
|
||||
**Done means:** the third cell reads back.
|
||||
|
||||
**Q4 — OPEN (fixture lane; bold-lead record, no heading).**
|
||||
|
||||
**Q5 opened — another record-form id.**
|
||||
"""
|
||||
|
||||
|
||||
def _write(path, text):
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8", newline="") as fh:
|
||||
fh.write(text)
|
||||
|
||||
|
||||
def _read(path):
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
return fh.read()
|
||||
|
||||
|
||||
def _build_tree(tmp, name, queue_text):
|
||||
"""A tree the shipped resolution recognises, with the extractor beside it."""
|
||||
root = os.path.join(tmp, name)
|
||||
art = os.path.join(root, ARTIFACT_REL)
|
||||
os.makedirs(art)
|
||||
_write(os.path.join(root, "WORK-QUEUE.md"), queue_text)
|
||||
shutil.copy(os.path.join(HERE, "extract_workqueue_census.py"), art)
|
||||
env = dict(os.environ, REPO_ROOT=root)
|
||||
out = subprocess.run([sys.executable, "extract_workqueue_census.py"], cwd=art,
|
||||
env=env, capture_output=True, text=True)
|
||||
if out.returncode != 0:
|
||||
raise RuntimeError("extractor failed in %s: %s" % (root, out.stderr.strip()))
|
||||
return root, art
|
||||
|
||||
|
||||
def selftest(say=print):
|
||||
cells = []
|
||||
tmp = tempfile.mkdtemp(prefix="census-selftest-")
|
||||
quiet = lambda *a: None # noqa: E731
|
||||
try:
|
||||
# cell 0 -- control: the fixture, generated by the extractor, is green.
|
||||
root, art = _build_tree(tmp, "control", FIXTURE_QUEUE)
|
||||
cells.append(("control-green", 0, grade(root, art, say=quiet)))
|
||||
|
||||
# cell 1 -- a quoted cell reads back intact; the same tree under the
|
||||
# pre-Q112 reader is the falsification, not an assertion about it.
|
||||
root1, art1 = _build_tree(tmp, "quote", FIXTURE_QUEUE)
|
||||
cells.append(("quote-cell-green-with-csv-reader", 0,
|
||||
grade(root1, art1, say=quiet)))
|
||||
cells.append(("quote-cell-red-with-naive-reader", 1,
|
||||
grade(root1, art1, reader=read_tsv_naive, say=quiet)))
|
||||
|
||||
# cell 2 -- a heading deleted from the source, artifact untouched.
|
||||
root2, art2 = _build_tree(tmp, "no-heading", FIXTURE_QUEUE)
|
||||
q2 = _read(os.path.join(root2, "WORK-QUEUE.md"))
|
||||
q2 = q2.replace('## Q2 \u2014 second item \u2014 the "pin" class\n', "")
|
||||
_write(os.path.join(root2, "WORK-QUEUE.md"), q2)
|
||||
cells.append(("deleted-heading-red", 1, grade(root2, art2, say=quiet)))
|
||||
|
||||
# cell 3 -- one title cell corrupted in the TSV.
|
||||
root3, art3 = _build_tree(tmp, "bad-cell", FIXTURE_QUEUE)
|
||||
tsv3 = os.path.join(art3, "workqueue-census.tsv")
|
||||
_write(tsv3, _read(tsv3).replace("\u2014 first item", "\u2014 FIRST item"))
|
||||
cells.append(("corrupted-cell-red", 1, grade(root3, art3, say=quiet)))
|
||||
|
||||
# cell 4 -- a new record-form id in the source, manifest not regenerated.
|
||||
root4, art4 = _build_tree(tmp, "new-record-id", FIXTURE_QUEUE)
|
||||
with open(os.path.join(root4, "WORK-QUEUE.md"), "a", encoding="utf-8") as fh:
|
||||
fh.write("\n**Q6 \u2014 OPEN (fixture lane; a second record-form id).**\n")
|
||||
cells.append(("new-record-form-id-red", 1, grade(root4, art4, say=quiet)))
|
||||
|
||||
# cell 5 -- the manifest's record_only_ids loses one entry.
|
||||
root5, art5 = _build_tree(tmp, "short-record-ids", FIXTURE_QUEUE)
|
||||
man5 = os.path.join(art5, "extract_manifest.json")
|
||||
blob = json.loads(_read(man5))
|
||||
blob["record_only_ids"] = blob["record_only_ids"][:-1]
|
||||
_write(man5, json.dumps(blob, indent=2) + "\n")
|
||||
cells.append(("manifest-record-ids-short-red", 1,
|
||||
grade(root5, art5, say=quiet)))
|
||||
|
||||
# cell 6 -- an unparseable provenance stamp.
|
||||
root6, art6 = _build_tree(tmp, "bad-stamp", FIXTURE_QUEUE)
|
||||
man6 = os.path.join(art6, "extract_manifest.json")
|
||||
blob = json.loads(_read(man6))
|
||||
blob["extracted_at_utc"] = "yesterday"
|
||||
_write(man6, json.dumps(blob, indent=2) + "\n")
|
||||
cells.append(("manifest-stamp-shape-red", 1, grade(root6, art6, say=quiet)))
|
||||
|
||||
# cell 7 -- a revision that is not a revision (the leg that used to
|
||||
# print PASS without measuring anything).
|
||||
root7, art7 = _build_tree(tmp, "bad-rev", FIXTURE_QUEUE)
|
||||
man7 = os.path.join(art7, "extract_manifest.json")
|
||||
blob = json.loads(_read(man7))
|
||||
blob["revision"] = "HEAD~1"
|
||||
_write(man7, json.dumps(blob, indent=2) + "\n")
|
||||
cells.append(("manifest-revision-shape-red", 1, grade(root7, art7, say=quiet)))
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
failed = 0
|
||||
for name, want, got in cells:
|
||||
ok = want == got
|
||||
failed += 0 if ok else 1
|
||||
say("%s %s (want rc %d, got rc %d)" % ("PASS" if ok else "FAIL", name, want, got))
|
||||
say("SELFTEST %s (%d cells, %d failed)"
|
||||
% ("OK" if not failed else "FAILED", len(cells), failed))
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description="grade a workqueue census against its tree")
|
||||
ap.add_argument("--root", default=DEFAULT_ROOT, help="tree holding WORK-QUEUE.md")
|
||||
ap.add_argument("--artifact", default=HERE, help="census directory to grade")
|
||||
ap.add_argument("--selftest", action="store_true", help="mutate fixture trees")
|
||||
args = ap.parse_args(argv)
|
||||
if args.selftest:
|
||||
return selftest()
|
||||
return grade(os.path.abspath(args.root), os.path.abspath(args.artifact))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"schema": "workqueue-census-v3",
|
||||
"source": "/root/troupe-standup/WORK-QUEUE.md",
|
||||
"revision": "81e0ed28237f",
|
||||
"revision": "2615c329da12",
|
||||
"rows": 97,
|
||||
"distinct_q_labels": 86,
|
||||
"reopened_q_labels": [
|
||||
@@ -17,7 +17,46 @@
|
||||
"Q83",
|
||||
"Q93"
|
||||
],
|
||||
"extracted_at_utc": "2026-10-28T23:25:39Z",
|
||||
"record_only_ids": [
|
||||
"Q41",
|
||||
"Q42",
|
||||
"Q46",
|
||||
"Q47",
|
||||
"Q50",
|
||||
"Q51",
|
||||
"Q52",
|
||||
"Q53",
|
||||
"Q64",
|
||||
"Q65",
|
||||
"Q99",
|
||||
"Q100",
|
||||
"Q101",
|
||||
"Q102",
|
||||
"Q103",
|
||||
"Q104",
|
||||
"Q105",
|
||||
"Q106",
|
||||
"Q107",
|
||||
"Q108",
|
||||
"Q109",
|
||||
"Q110",
|
||||
"Q111",
|
||||
"Q112",
|
||||
"Q113",
|
||||
"Q114",
|
||||
"Q115",
|
||||
"Q116",
|
||||
"Q117",
|
||||
"Q118",
|
||||
"Q119",
|
||||
"Q120",
|
||||
"Q121",
|
||||
"Q122",
|
||||
"Q123",
|
||||
"Q124"
|
||||
],
|
||||
"ids_opened_total": 122,
|
||||
"extracted_at_utc": "2026-10-29T18:14:02Z",
|
||||
"generated_by": "extract_workqueue_census.py (v3, independent CHECK)",
|
||||
"verification": "CHECK (independent re-derivation; does not import extractor)"
|
||||
}
|
||||
@@ -35,6 +35,16 @@ HEADING_RE = re.compile(r"^##\s+Q(\d+)(.*)$")
|
||||
STATUS_RE = re.compile(r"^\*\*Status:\s*(.*)$")
|
||||
DONE_RE = re.compile(r"^\*\*Done means:\s*(.*)$")
|
||||
|
||||
# Record forms that are not headings (the rule WORK-QUEUE.md states at Q63: an
|
||||
# id is opened by a record, not mentioned by prose). A bold run opens an id
|
||||
# only when the id is the first token after the bullet/bold markers and the
|
||||
# next token is a separator or the word open/opened, so `**Q7, measured:**` is
|
||||
# a mention and does not count. Q112: a heading-only walk cannot see these ids,
|
||||
# so the census states them separately instead of leaving coverage implicit.
|
||||
BOLD_LEAD = re.compile(r"^[\s>|]*(?:[-*+]+\s+)?\*\*(Q\d+)(.*)$")
|
||||
OPEN_AFTER = re.compile(r"^\s*(?:—|–|-|:|\(|\bopened\b|\bopen\b|\bOPEN\b|\bOPENED\b)")
|
||||
STATUS_LEAD = re.compile(r"^\s*\*{0,2}Status:\s*(Q\d+)\b")
|
||||
|
||||
TS_FMT = "%Y-%m-%dT%H:%M:%SZ"
|
||||
|
||||
|
||||
@@ -85,6 +95,21 @@ def split_blocks(lines):
|
||||
return blocks
|
||||
|
||||
|
||||
def record_only_ids(lines, heading_ids):
|
||||
"""Ids opened in bold-lead or status-lead form that have no `## ` heading."""
|
||||
seen = {}
|
||||
for line in lines:
|
||||
m = BOLD_LEAD.match(line)
|
||||
if m and m.group(1) not in heading_ids:
|
||||
if OPEN_AFTER.match(m.group(2)) and m.group(1) not in seen:
|
||||
seen[m.group(1)] = line
|
||||
continue
|
||||
m = STATUS_LEAD.match(line)
|
||||
if m and m.group(1) not in heading_ids and m.group(1) not in seen:
|
||||
seen[m.group(1)] = line
|
||||
return sorted(seen, key=lambda q: int(q[1:]))
|
||||
|
||||
|
||||
def main():
|
||||
with open(SRC, "r", encoding="utf-8") as fh:
|
||||
lines = fh.read().splitlines()
|
||||
@@ -101,6 +126,13 @@ def main():
|
||||
}
|
||||
)
|
||||
|
||||
# The ids a heading-only walk cannot see (Q112). Stated, not folded into the
|
||||
# row table: a row per heading stays the census's unit, and a record-form id
|
||||
# has no heading title and no block bounds to report.
|
||||
heading_ids = {b["qid"] for b in blocks}
|
||||
record_only = record_only_ids(lines, heading_ids)
|
||||
ids_opened_total = sorted(heading_ids | set(record_only), key=lambda q: int(q[1:]))
|
||||
|
||||
# ---------- TSV ----------
|
||||
tsv_path = os.path.join(OUT, "workqueue-census.tsv")
|
||||
with open(tsv_path, "w", encoding="utf-8", newline="") as fh:
|
||||
@@ -121,6 +153,14 @@ def main():
|
||||
fh.write(f"| {r['q']} | {escape(r['title'])} | {escape(r['status'])} | "
|
||||
f"{escape(r['done_implies'])} |\n")
|
||||
fh.write("\n*Rows: %d.*\n\n" % len(rows))
|
||||
fh.write("## Ids opened only in record form\n\n")
|
||||
fh.write("These ids open a record in bold-lead form (`**Qnn — …`) and have no "
|
||||
"`## Qnn` heading, so the table above cannot carry them. They are "
|
||||
"enumerated here because the census's coverage is a fact about the "
|
||||
"artifact, not something a reader should have to derive: **%d of the "
|
||||
"%d ids the queue opens are record-form only.**\n\n"
|
||||
% (len(record_only), len(ids_opened_total)))
|
||||
fh.write(" ".join("`%s`" % q for q in record_only) + "\n")
|
||||
# ---------- manifest ----------
|
||||
distinct = len({r["q"] for r in rows})
|
||||
freq = {}
|
||||
@@ -134,6 +174,8 @@ def main():
|
||||
"rows": len(rows),
|
||||
"distinct_q_labels": distinct,
|
||||
"reopened_q_labels": reopened,
|
||||
"record_only_ids": record_only,
|
||||
"ids_opened_total": len(ids_opened_total),
|
||||
"extracted_at_utc": stamp(),
|
||||
"generated_by": "extract_workqueue_census.py (v3, independent CHECK)",
|
||||
"verification": "CHECK (independent re-derivation; does not import extractor)",
|
||||
@@ -171,6 +213,13 @@ Conventions
|
||||
truncated at the first sentence end, single trailing period; empty when no
|
||||
such marker precedes the next heading.
|
||||
- done_implies : the first '**Done means:' line in the block, trimmed.
|
||||
- record_only_ids : ids that open a record in bold-lead form (`**Qnn -- ...` or
|
||||
`**Qnn (lane) -- OPEN`) and have no '## Qnn' heading. A heading-only walk
|
||||
cannot see them, so no table row is created for one (it has no heading title
|
||||
and no block bounds); the manifest carries the list and the markdown table
|
||||
states the coverage it implies.
|
||||
- ids_opened_total : how many distinct ids the queue opens in any form, so
|
||||
rows x record_only_ids x ids_opened_total reconcile in CHECK.
|
||||
|
||||
Run
|
||||
---
|
||||
@@ -179,7 +228,8 @@ Run
|
||||
""")
|
||||
|
||||
print(f"rows: {len(rows)} distinct: {manifest['distinct_q_labels']} "
|
||||
f"reopened: {len(reopened)}")
|
||||
f"reopened: {len(reopened)} record-form-only: {len(record_only)} "
|
||||
f"ids-opened: {len(ids_opened_total)}")
|
||||
print(f"wrote {tsv_path}")
|
||||
print(f"wrote {man_path}")
|
||||
return 0
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Work-queue census
|
||||
|
||||
Fresh extract of every `## Q<n>` item in `WORK-QUEUE.md` at revision `81e0ed28237f`, regenerated at 2026-10-28T23:25:39Z.
|
||||
Fresh extract of every `## Q<n>` item in `WORK-QUEUE.md` at revision `2615c329da12`, regenerated at 2026-10-29T18:14:02Z.
|
||||
|
||||
| q | title | status | done_implies |
|
||||
|---|---|---|---|
|
||||
@@ -104,3 +104,8 @@ Fresh extract of every `## Q<n>` item in `WORK-QUEUE.md` at revision `81e0ed2823
|
||||
|
||||
*Rows: 97.*
|
||||
|
||||
## Ids opened only in record form
|
||||
|
||||
These ids open a record in bold-lead form (`**Qnn — …`) and have no `## Qnn` heading, so the table above cannot carry them. They are enumerated here because the census's coverage is a fact about the artifact, not something a reader should have to derive: **36 of the 122 ids the queue opens are record-form only.**
|
||||
|
||||
`Q41` `Q42` `Q46` `Q47` `Q50` `Q51` `Q52` `Q53` `Q64` `Q65` `Q99` `Q100` `Q101` `Q102` `Q103` `Q104` `Q105` `Q106` `Q107` `Q108` `Q109` `Q110` `Q111` `Q112` `Q113` `Q114` `Q115` `Q116` `Q117` `Q118` `Q119` `Q120` `Q121` `Q122` `Q123` `Q124`
|
||||
Reference in new issue
Block a user