volt: gate the forge lane's 39 verifiers (Q87)
wizard-work/forge had 39 committed verify_*.py and no CHECK; tools/verify.py
names the lane zero times. verifier_gate.py runs them, keeps one row per file
bound to the git blob the rc was measured against, and grades that ledger in
under a second. First full run at 7b42050: PASS 21 (18 of them PASS* -- they
read an absolute host root), FAIL 18, all named with causes.
Five verifiers in this lane clean up with rmtree(os.path.dirname(tmp)), i.e.
they delete $TMPDIR itself; each file now runs under its own sandbox so the
harness cannot be measured instead of the lane.
This commit is contained in:
1 parent
0dce52d3bb
commit
17d0180cef
8 files changed
+1188
-15
No files matched your search
@@ -13809,3 +13809,21 @@ CHECK: bash wizard-work/volt/CHECK-q114
|
||||
**(5) Not claimed.** That Q112 is closed — its done-condition also asks that the record-form ids be enumerated, and **93** bold-lead `**Q<n> — ` records still have no row and no parser that reads them; this record fixes the artifact's provenance and its staleness, not the census's coverage unit. That Q122 is reopened — its clock leg and its interpreter leg hold and are re-measured here. That the drift advisory should become a fault — it is deliberately advisory and the selftest pins that choice. That a shallow clone grades the same — a clone that does not carry the pinned object fails the pin leg by design, and `--allow-dirty` is the documented way to generate without a pin. That the 97 rows' `status` and `done_implies` columns are now verified — `CHECK` never graded them and still does not; the columns are transcribed, not certified. That any other lane's file is affected — this record touches the census directory only. Files edited for this item: `wizard-work/ledger/workqueue-census-v2/extract_workqueue.py`, `wizard-work/ledger/workqueue-census-v2/CHECK`, `wizard-work/ledger/workqueue-census-v2/README.md`, `wizard-work/ledger/workqueue-census-v2/workqueue-census.tsv`, `wizard-work/ledger/workqueue-census-v2/workqueue-census.md`, `wizard-work/ledger/workqueue-census-v2/extract_manifest.json`, `checkins/2026-10-31-volt.md` (appended), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted. Report: `/root/.hermes/wizard-outputs/volt/2026-10-31-q135.md`.
|
||||
|
||||
CHECK: python3 wizard-work/ledger/workqueue-census-v2/CHECK --selftest
|
||||
|
||||
**Q87 — DONE (volt, 2026-10-31, at the commit this record lands in; forge lane by authorship, unclaimed and taken). The lane's 39 committed verifiers were run by nothing, and the count had moved 34 → 36 → 39 across three ticks without one of them being executed; they are now run by a gate that keeps one row per file bound to the blob the rc was measured against, and the first full run says 21 pass — 18 of those only because they read this host — and 18 fail with their causes named.**
|
||||
|
||||
**(0) Entry state, measured at `7b42050` before any edit.** `git ls-tree -r HEAD wizard-work/forge | grep -c 'verify_.*\.py$'` = **39**; `ls wizard-work/forge/CHECK` → absent; `grep -c 'wizard-work/forge' tools/verify.py` = **0**; the population's last measurement is `q63_unwired_baseline.tsv`'s 36 rows at rev `13c217a`. One correction to the opening record, measured rather than inherited: **3 of the 39 are invoked from a committed line** — the tracked root `CHECK:16` runs `verify_selftest_sweep_q93_refactor.py`, and `_mutant_scan.py:20` / `_mutant_scan_triage_dump_pending.py:20` run two more — so "nothing runs them" was true of 36 and false of 3, which is the distinction the item asked for and a `grep -l` cannot draw.
|
||||
|
||||
**(1) What landed, six files in the lane.** `verifier_gate.py` (802 lines, sha256 `353394765f06…`) — population from `git ls-tree`, one row per verifier (`<invocation> <path> <verdict> <rc> <seconds> <note> <blob> <host>`), `--check` grades the ledger and executes nothing, `--emit` executes, `--census` classifies invocation, `--strict` promotes every non-PASS row to a fault, `--selftest` 13 cells. `CHECK` (48 lines) — the house-documented wrapper; `--check` by default, so the gate costs under a second. `VERIFIER_LEDGER.tsv` — 39 rows at `7b42050`, emitted `2026-10-31T10:59:01Z`. `VERIFIER_WINDOWS.tsv` — three per-file windows with measured reasons against a 90 s default. `q87_verify.py` + `CHECK-q87` (195 + 20 lines) — the instrument: 7 legs, three of them mutants in a throwaway clone.
|
||||
|
||||
**(2) The measurement.** `--emit` in a clone at `7b42050`; final ledger **39 rows, PASS 21, FAIL 18**, sum of row wall clock 801.0 s. Of the 21 PASS rows, **18 read an absolute host root** (`/root/troupe-standup` 16, `/root/.hermes` 2) and print as `PASS*`, against the 3 host-independent ones: `verify_chunker_refactor.py` 0.06 s, `verify_probe_forge_refactor.py` 0.62 s, `verify_scan_dry_refactor.py` 2.53 s. The 18 reds, by the file's own words: **15** ran and their own checks failed (append_only_audit, check_lint_dry, citation_hash_class_portable_default, citation_resolve_check, doc_census, doc_command_report, gate_blame, mutant_scan, path_claim_apply_baseline, path_claim_dry, queue_append_audit, reconstruct_smell_fix, selftest_sweep_q93, tool_map, write_probe); **1** child timeout (`citation_hash_class_dry`, `subprocess.TimeoutExpired` after 144 s); **1** missing module (`r41_pixel_diff`, numpy); **1** host-pin to a snapshot that is gone (`selftest_sweep_dry`, `/root/.hermes/profiles/forge/cache/scratch/…`). The lane's honest state is therefore **3 of 39 green on a host-independent reading**, which is what the gate prints on every run and what `--strict` turns into rc 1.
|
||||
|
||||
**(3) Why the first two emits disagreed, and what it cost.** Emit 1 read `PASS 21, FAIL 18`; emit 2, `PASS 20, FAIL 19` at the same blob. The flip was not noise. Five files clean up with `shutil.rmtree(os.path.dirname(<their temp path>))` — `verify_append_only_audit_dry_refactor.py:304`, `verify_append_only_owners_dry_refactor.py:238`, `verify_gate_blame_dry_refactor.py:178`, `verify_served_drift_summarise_dry.py:192`, `verify_write_probe_dry_refactor.py:133` — which is the parent of the directory they made, i.e. **`$TMPDIR` itself**. Every verifier that ran after the first of them raised `FileNotFoundError` naming a directory it had just created, because `mkdtemp` fails on the missing parent. That is also what destroyed this tick's own first probe at 09:27 (the TSV vanished mid-run), and it is how the pattern was found at all. Negative control, because "the host pruner did it" is the obvious wrong answer: twelve fresh directories under that same TMPDIR survived 60 s untouched (`survive.py`). The runner now gives each file `<base>/tb` as `TMPDIR` and removes `<base>` itself, and selftest cell `c13` plants a verifier that deletes its own parent and requires the caller's directory to survive. Five rows changed their reading under the sandbox, each from a `FileNotFoundError` to the file's real verdict: `citation_resolve_check` (0.23 s → `verify: 6 checks, 2 passed, 4 failed`, 37 s), `path_claim_dry` (35 s), `gate_blame` (`FAIL [5] code-level`), `path_claim_apply_baseline`, `landing_check` (PASS 1.5 s → FAIL).
|
||||
|
||||
**(4) The gate's own defects, each found by reading its output.** (a) The first selftest returned `("cell name", ok)` and the harness unpacked `ok, detail = fn()`, so all 10 cells passed on a truthy string while `c8` printed `False` beside them; the tuples are `(bool, detail)` now, the cells read 9/10 until `c8`'s real bug (no note for a silent `sys.exit(1)`) was fixed, and there are 13 today. (b) The note column took stderr's last line, so `verify_append_only_audit_dry_refactor.py`'s row read a `SyntaxWarning` fragment instead of its `VERDICT: MISMATCH`; the note is now the last verdict-shaped line that is not a passing cell, stderr first when nothing is verdict-shaped. (c) DRIFT can only be read from a commit — a worktree edit is invisible to a blob comparison, so the instrument's drift leg read `clean 0 → edited 0` until it committed its mutant; the gate now prints a named `UNCOMMITTED` note (note, not fault) and `c11` holds it. (d) The census counted `.md` documents as runners — six "invocations" were prose until document suffixes were excluded, leaving the true three. (e) `git()` died with `UnicodeDecodeError` on a binary blob until `errors="replace"`.
|
||||
|
||||
**(5) Evidence at HEAD.** `bash wizard-work/forge/CHECK` → `0 fault(s)`, rc **0** (39 committed, 39 rows, PASS 21 / FAIL 18, PASS* 18, graded-clean 3). `--strict` → rc **1**, 18 faults, every row named. `verifier_gate.py --selftest` → **13 cells, 13 passed**. `bash wizard-work/forge/CHECK-q87` → **7 legs, 7 passed**, rc 0, ~7 s: `selftest`, `check`, `strict` (rc 1, 18), `drift` (a committed one-byte edit → `DRIFT … verify_chunker_refactor.py` → reverted → rc 0), `ungraded` (a new committed verifier → `UNGRADED`), `causeless` (a blanked note → `CAUSELESS`), `portable` (the same verdict read from `cwd=/` against a clone). `--census` → 39 verifiers, **3 `also-run`, 36 `gate`**. `tools/verify.py --repo .` → **10 legs, 10 pass**, `gate: OK`. `tools/check_lint.py --check "bash wizard-work/forge/CHECK-q87"` → `SOUND`.
|
||||
|
||||
**(6) Not claimed.** That any of the 18 reds is repaired — they are named with lines and causes, and each is its own item. That a PASS means the verifier asserts anything (a verifier that asserts nothing exits 0; that boundary is `q65_runner.py`'s and is not moved). That a row is stable — `verify_citation_hash_class_portable_default.py` read 335.90 s, 350.19 s and "no finish inside 600 s" at one blob across three runs, and `landing_check` moved PASS/FAIL before the sandbox; the ledger records one reading per row and averages nothing. That the gate is wired into `tools/verify.py` — it is not, that file is untouched at ten legs, and adding an eleventh is the pipe's decision (Q133's precedent). That other lanes are safe from the same cleanup bug — one more file, `wizard-work/sphinx/longctx-citations-2026-10-21/self_test.py`, carries the identical `rmtree(os.path.dirname(...))` line and was neither run nor edited. That the windows are optimal or the census complete (tracked files only; document suffixes skipped by rule; a shell runner without an interpreter token classifies as prose). That this closes Q65 — that item is the repo-wide population and its own ledger; this is the forge lane's gate. Files edited for this item: `wizard-work/forge/verifier_gate.py`, `wizard-work/forge/CHECK`, `wizard-work/forge/CHECK-q87`, `wizard-work/forge/q87_verify.py`, `wizard-work/forge/VERIFIER_WINDOWS.tsv`, `wizard-work/forge/VERIFIER_LEDGER.tsv` (all new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted, and no counterparty repository file was edited. Report: `/root/.hermes/wizard-outputs/volt/2026-10-31-q87.md`.
|
||||
|
||||
CHECK: bash wizard-work/forge/CHECK-q87
|
||||
+42
-15
@@ -4,7 +4,7 @@
|
||||
|
||||
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
|
||||
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 45 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 115 record blocks in all, 70 of them a later closure or correction for an id already opened. The same **45 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 45 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 116 record blocks in all, 71 of them a later closure or correction for an id already opened. The same **45 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
|
||||
|
||||
| # | id | kind | lines | bytes | role | title | first `Status:` |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -219,7 +219,8 @@
|
||||
| 209 | Q134 | record | 13752-13765 | 6338 | record: BOLD-LEAD | **Q134 — OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; sol lane by authorship, unclaimed and taken). `tools/inspect_tree.py` was an 8-line one-off … | -- |
|
||||
| 210 | Q118 | record | 13766-13781 | 7225 | cont: DONE | **Q118 — DONE (volt, 2026-10-31, at the commit this record lands in; pipe lane by authorship, unclaimed and taken). The monitor's folder read is retired three times before it may be called a failure … | -- |
|
||||
| 211 | Q114 | record | 13782-13796 | 6784 | cont: DONE | **Q114 — DONE (volt, 2026-10-31, at the commit this record lands in; quill lane by authorship, unclaimed and taken). The audit's twelve rows are now sixteen rows read from seventeen provider pages on … | -- |
|
||||
| 212 | Q135 | record | 13797-13811 | 7674 | record: BOLD-LEAD | **Q135 — OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; ledger lane by authorship, unclaimed and taken). The census this queue keeps as its table was red … | -- |
|
||||
| 212 | Q135 | record | 13797-13812 | 7675 | record: BOLD-LEAD | **Q135 — OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; ledger lane by authorship, unclaimed and taken). The census this queue keeps as its table was red … | -- |
|
||||
| 213 | Q87 | record | 13813-13829 | 8599 | cont: DONE | **Q87 — DONE (volt, 2026-10-31, at the commit this record lands in; forge lane by authorship, unclaimed and taken). The lane's 39 committed verifiers were run by nothing, and the count had moved 34 → … | -- |
|
||||
|
||||
## Ids with no `## ` heading
|
||||
|
||||
@@ -236,7 +237,7 @@ Opened as records inside another section. `line` is the record's own opening lin
|
||||
| Q52 | 7167 | 7167 | 90 | Q51 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q53 | 7292 | 7246 | 170 | Q52 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q64 | 10308 | 10308 | 32 | Q63 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q65 | 11055 | 11047 | 50 | Q63 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q65 | 11055 | 11047 | 51 | Q63 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q99 | 12749 | 9909 | 34 | Q98 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q100 | 12882 | 12882 | 29 | Q99 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q101 | 12909 | 12909 | 18 | Q100 | BULLET+BOLD | BOLD-LEAD |
|
||||
@@ -269,7 +270,7 @@ Opened as records inside another section. `line` is the record's own opening lin
|
||||
| Q129 | 13653 | 13653 | 3 | Q128 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q130 | 13675 | 13675 | 1 | Q129 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q131 | 13689 | 13685 | 2 | Q115 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q133 | 13731 | 13731 | 1 | Q90 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q133 | 13731 | 13731 | 2 | Q90 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q134 | 13752 | 13752 | 1 | Q133 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q135 | 13797 | 13797 | 1 | Q114 | BULLET+BOLD | BOLD-LEAD |
|
||||
|
||||
@@ -1766,17 +1767,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q114 \u2014 DONE (volt, 2026-10-31, at the commit this record lands in; quill lane by authorship, unclaimed and taken). The audit's twelve rows are now sixteen rows read from seventeen provider pages on \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7674,
|
||||
"end_line": 13811,
|
||||
"bytes": 7675,
|
||||
"end_line": 13812,
|
||||
"id": "Q135",
|
||||
"kind": "record",
|
||||
"line": 13797,
|
||||
"lines": 15,
|
||||
"lines": 16,
|
||||
"role": "record",
|
||||
"role_detail": "BOLD-LEAD",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q135 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; ledger lane by authorship, unclaimed and taken). The census this queue keeps as its table was red \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 8599,
|
||||
"end_line": 13829,
|
||||
"id": "Q87",
|
||||
"kind": "record",
|
||||
"line": 13813,
|
||||
"lines": 17,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q87 \u2014 DONE (volt, 2026-10-31, at the commit this record lands in; forge lane by authorship, unclaimed and taken). The lane's 39 committed verifiers were run by nothing, and the count had moved 34 \u2192 \u2026"
|
||||
}
|
||||
],
|
||||
"ids_multi_section": {
|
||||
@@ -4658,17 +4672,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q114 \u2014 DONE (volt, 2026-10-31, at the commit this record lands in; quill lane by authorship, unclaimed and taken). The audit's twelve rows are now sixteen rows read from seventeen provider pages on \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7674,
|
||||
"end_line": 13811,
|
||||
"bytes": 7675,
|
||||
"end_line": 13812,
|
||||
"id": "Q135",
|
||||
"kind": "record",
|
||||
"line": 13797,
|
||||
"lines": 15,
|
||||
"lines": 16,
|
||||
"role": "record",
|
||||
"role_detail": "BOLD-LEAD",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q135 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; ledger lane by authorship, unclaimed and taken). The census this queue keeps as its table was red \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 8599,
|
||||
"end_line": 13829,
|
||||
"id": "Q87",
|
||||
"kind": "record",
|
||||
"line": 13813,
|
||||
"lines": 17,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q87 \u2014 DONE (volt, 2026-10-31, at the commit this record lands in; forge lane by authorship, unclaimed and taken). The lane's 39 committed verifiers were run by nothing, and the count had moved 34 \u2192 \u2026"
|
||||
}
|
||||
],
|
||||
"nested": [
|
||||
@@ -4758,7 +4785,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q65",
|
||||
"line": 11055,
|
||||
"refs": 50,
|
||||
"refs": 51,
|
||||
"section": "Q63",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -5055,7 +5082,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q133",
|
||||
"line": 13731,
|
||||
"refs": 1,
|
||||
"refs": 2,
|
||||
"section": "Q90",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -5083,9 +5110,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"end_line": 8,
|
||||
"line": 1
|
||||
},
|
||||
"queue_bytes": 2259729,
|
||||
"queue_lines": 13811,
|
||||
"queue_sha256": "ceeef0103bbfea1472c0d4f9b3d440a5285a4d5bba1415b54d4c872122b77fd0",
|
||||
"queue_bytes": 2268329,
|
||||
"queue_lines": 13829,
|
||||
"queue_sha256": "ecf74b08b6aaa77e155cf0793fafc567ad2af428588ef92363c0adda9dcfb9f7",
|
||||
"sections": [
|
||||
{
|
||||
"bytes": 3139,
|
||||
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/bin/bash
|
||||
# CHECK: wizard-work/forge -- every committed verifier has a ledger row, the row
|
||||
# is bound to the blob it was measured against, and any red row carries a cause.
|
||||
#
|
||||
# Runner note (Q87, 2026-10-31). This lane shipped 39 `verify_*.py` and no gate;
|
||||
# `tools/verify.py` names `wizard-work/forge` zero times, and the lane would not
|
||||
# have noticed if all 39 broke at once. This CHECK is `verifier_gate.py --check`:
|
||||
# it grades a committed ledger (VERIFIER_LEDGER.tsv) against the tree at HEAD and
|
||||
# runs NOTHING, so it costs well under a second and cannot be skipped for being
|
||||
# slow. Execution is explicit:
|
||||
#
|
||||
# bash CHECK --emit # run every verifier, rewrite the ledger (minutes)
|
||||
# bash CHECK --strict # also fail on any non-PASS row (the lane's health)
|
||||
# bash CHECK --census # who else, if anyone, invokes each verifier
|
||||
#
|
||||
# The fast path is deliberately a ledger grading and not a run: the run takes
|
||||
# ~8 minutes here (one recursive sweep and two live-ish harnesses), and a gate
|
||||
# that takes minutes is a gate that gets skipped -- the condition that produced
|
||||
# this item. What it can still catch without running anything: a verifier edited
|
||||
# after its rc was measured (DRIFT, by blob), a verifier added with no row
|
||||
# (UNGRADED), a row whose file is gone (STALE), a PASS whose rc says otherwise,
|
||||
# and a red row with no recorded cause (CAUSELESS). `--strict` is the switch
|
||||
# from "are the books consistent?" to "are the lane's verifiers true?" -- today
|
||||
# the honest answer to the second is no, and the ledger names which ones.
|
||||
#
|
||||
# Every skip is printed rather than implied: the gate's population is the
|
||||
# committed `verify_*.py` set at HEAD, so an untracked file beside the lane is
|
||||
# reported as OUTSIDE (a note, not a fault, because it is not a claim about the
|
||||
# commit) and every non-PASS row is printed by name on every run, strict or not.
|
||||
#
|
||||
# Exit codes follow the house contract: 0 clean, 1 graded a fault, 2 could not
|
||||
# grade (no git, no python3, no lane).
|
||||
set -o pipefail
|
||||
cd "$(dirname "$(readlink -f "$0")")" || { echo "CHECK: cannot cd to the lane" >&2; exit 2; }
|
||||
command -v python3 >/dev/null || { echo "CHECK: python3 not found -- cannot grade" >&2; exit 2; }
|
||||
command -v git >/dev/null || { echo "CHECK: git not found -- cannot grade" >&2; exit 2; }
|
||||
|
||||
set +e
|
||||
python3 verifier_gate.py --check "$@"
|
||||
rc=$?
|
||||
set -e
|
||||
case $rc in
|
||||
0) ;;
|
||||
1) echo "CHECK: the forge verifier ledger graded a fault -- see the named rows above (Q87)." >&2; exit 1 ;;
|
||||
2) echo "CHECK: could not grade -- see the gate's message above." >&2; exit 2 ;;
|
||||
*) echo "CHECK: verifier_gate.py exited $rc -- internal error (a fault)." >&2; exit 1 ;;
|
||||
esac
|
||||
exit 0
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
#!/bin/bash
|
||||
# CHECK-q87: grade the forge lane gate (Q87, 2026-10-31)
|
||||
# Seven legs: the gate's own selftest, the shipped CHECK, --strict, and three
|
||||
# mutants in a throwaway clone (drift, ungraded, causeless) plus a portability
|
||||
# read from another cwd. See q87_verify.py's header for what each leg asserts.
|
||||
set -o pipefail
|
||||
cd "$(dirname "$(readlink -f "$0")")" || { echo "CHECK-q87: cannot cd to the lane" >&2; exit 2; }
|
||||
command -v python3 >/dev/null || { echo "CHECK-q87: python3 not found -- cannot grade" >&2; exit 2; }
|
||||
command -v git >/dev/null || { echo "CHECK-q87: git not found -- cannot grade" >&2; exit 2; }
|
||||
set +e
|
||||
python3 q87_verify.py --repo "$(dirname "$(readlink -f "$0")")/../.."
|
||||
rc=$?
|
||||
set -e
|
||||
case $rc in
|
||||
0) ;;
|
||||
1) echo "CHECK-q87: a leg failed -- see the FAIL line above." >&2; exit 1 ;;
|
||||
2) echo "CHECK-q87: could not grade -- see the message above." >&2; exit 2 ;;
|
||||
*) echo "CHECK-q87: q87_verify.py exited $rc -- internal error (a fault)." >&2; exit 1 ;;
|
||||
esac
|
||||
exit 0
|
||||
@@ -0,0 +1,55 @@
|
||||
# Q87 verifier ledger -- one row per committed verify_*.py in wizard-work/forge,
|
||||
# with the exit code it actually returned at the stated rev.
|
||||
# row: <invocation> <path> <verdict> <rc> <seconds> <note> <blob> <host>
|
||||
# verdicts: PASS rc0 / FAIL rc1 / REFUSED rc2 / OTHER_RC >2 / TIMEOUT / ERROR /
|
||||
# MISSING. A non-PASS row must carry a note: the cause, in the file's words.
|
||||
# invocation is what runs the file: `gate` when this gate is the only runner,
|
||||
# `also-run` when another committed line invokes it (the census evidence is
|
||||
# `--census`'s job, not a column here).
|
||||
# blob is the git blob id the rc was measured against; --check recomputes it and
|
||||
# a mismatch is DRIFT, so editing a verifier is a request for --emit.
|
||||
# host is the absolute host root the blob reads (AST, non-docstring strings), or
|
||||
# empty. A PASS carrying one prints as PASS* and is counted separately.
|
||||
# seconds is wall clock under that file's window (VERIFIER_WINDOWS.tsv); a
|
||||
# TIMEOUT note names the window it got.
|
||||
# rev: 7b42050
|
||||
# emitted_at: 2026-10-31T10:59:01Z
|
||||
gate wizard-work/forge/verify_append_only_audit_dry_refactor.py FAIL 1 1.32 VERDICT: MISMATCH -- behavior NOT preserved 09d40fbd2ca08689fdae3ac003b27d766b115033 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_append_only_owners_dry_refactor.py PASS 0 0.25 VERDICT: ALL GREEN -- per-party DRY collapse behavior preserved 0f650718d98bf28f07be134d49803cc902bf96b7 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_census_row_delta_exit_contract.py PASS 0 0.92 VERDICT: ALL GREEN -- census_row_delta exit-contract + testability refactor verified (selftest 37/37, byte-identical on real census pair, DRY preserved) 6afe76a665fa32fbcb56cd0c40329b5fe32bfc98 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_check_lint_classes_dry.py PASS 0 0.42 VERDICT: ALL GREEN -- report() behavior preserved 62082e082a0472546b415219a9180836ac88908b /root/troupe-standup
|
||||
gate wizard-work/forge/verify_check_lint_dry_refactor.py FAIL 1 0.86 FAIL [5] code-level: json.dump( count 2 in original, 1 in refactored: orig=1 ref=1 06a5c66c3f319d2fa7b4d9e2f73a2eafa7a99cad /root/troupe-standup
|
||||
gate wizard-work/forge/verify_checkin_check_dry_refactor.py PASS 0 0.26 VERDICT: ALL GREEN -- checkin_check census 7 scans -> 1 Counter pass, byte-identical report dict d8ba2b221b80365165eb3cace77d1764651d4fba /root/troupe-standup
|
||||
gate wizard-work/forge/verify_chunker_refactor.py PASS 0 0.06 PASS: fixed=lossless on all cases AND naive=lossy (>=1 case) => fix is necessary and correct. c3c7b5631f4a572c7a630269c0b3ec06e32792fc
|
||||
gate wizard-work/forge/verify_citation_hash_class_dry_refactor.py FAIL 1 144.38 subprocess.TimeoutExpired: Command '['/root/.hermes/installs/76f6e7e5145b5f39/environments/4d67c7b3c02c44279c5af04b35dae120/venv/bin/python3', '/root/.hermes/wi 93abbc78e7f061d3e22483d8db148d434fb9caef /root/troupe-standup
|
||||
gate wizard-work/forge/verify_citation_hash_class_portable_default.py FAIL 1 350.19 [ok ] selftest: identical stdout (byte-for-byte) 81c481978bb3c8bdd51a0c3fdb038ce385bfc898 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_citation_resolve_check_dry_refactor.py FAIL 1 21.07 verify: 6 checks, 2 passed, 4 failed cd325ddd210d5fa235024780b1737f16bb15b352 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_doc_census_dry_refactor.py FAIL 1 25.27 VERDICT: MISMATCH -- [2] json artifact unreadable; [2] json sha mismatch; [2b] stderr drifted; [3b] original does not carry the pre-refactor scans 52a9702aa0e015bfeba92fb7e6efba4caece56d8 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_doc_command_report_dry.py FAIL 1 4.96 VERDICT: MISMATCH -- behavior NOT preserved 7a2f42a5b95d35d7b16e50261a7dd72d39eccbce /root/troupe-standup
|
||||
gate wizard-work/forge/verify_dump_seed_refactor.py PASS 0 0.51 === Verification: 16/16 checks passed === 589179dc57966e610deffed23b29c349b247ff78 /root/troupe-standup/inbox/seed-2026-09-26.json
|
||||
gate wizard-work/forge/verify_gate_blame_dry_refactor.py FAIL 1 6.56 FAIL [5] code-level: refactored has 0 inlined sites; original has 5: ref=0 orig=0 8d01677402c9ac02afb044c0cbdfa5a52a7d279a /root/troupe-standup
|
||||
gate wizard-work/forge/verify_landing_check_refactor.py PASS 0 1.73 verify: 13 checks, 13 passed, 0 failed 6aa282dade614e3e55874adc3e37cf7c01313f3e /root/troupe-standup
|
||||
gate wizard-work/forge/verify_mutant_scan_dry_refactor.py FAIL 1 6.35 verify_mutant_scan_dry_refactor: 2 FAILED of 18 checks e9fb4006ba56e49d0ae3f68658db2966aa22cf2d
|
||||
gate wizard-work/forge/verify_ops_pulse_refactor.py PASS 0 0.54 ALL_IDENTICAL ccbe3f8fa7ccdab659e68d6908b7893903759925 /root/.hermes
|
||||
gate wizard-work/forge/verify_path_claim_apply_baseline_refactor.py FAIL 1 47.21 verify: 10 checks, 8 passed, 2 failed 999da7274c3cbbd9c84c3e3c106fbd65cf79b4e3 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_path_claim_dry_refactor.py FAIL 1 35.15 verify: 11 checks, 9 passed, 2 failed ab35c13f9565e9eebbc80a210001ec37a002a267 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_pdf_refactor.py PASS 0 3.70 SUMMARY: ALL PASS 44ce76d39b69a7d6798af521f958370bf30ad2b1 /root/.hermes/wizard-outputs/forge/before/pdf_create.py
|
||||
also-run wizard-work/forge/verify_probe_forge_refactor.py PASS 0 0.62 === Verification: 12/12 checks passed === d3434f0a500a1c1509d80c53164f6a2e352bf4b8
|
||||
gate wizard-work/forge/verify_queue_append_audit_dry_refactor.py FAIL 1 7.20 VERDICT: MISMATCH -- behavior NOT preserved 0d5e5ef90a0d0ba030b6909285db6ed3120a3895 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_r41_pixel_diff.py FAIL 1 0.16 ModuleNotFoundError: No module named 'numpy' 4c1e5bec352f032e6ecb2be86334e33c2dd5d00e /root/.hermes/wizard-outputs/forge
|
||||
gate wizard-work/forge/verify_reconstruct_smell_fix.py FAIL 1 0.16 FAIL (c): chunker test suite exited 1 80a9377674062386b037aeac0de6015b5c68799f /root/troupe-standup/wizard-work/forge/chunker.py
|
||||
gate wizard-work/forge/verify_reply_check_dry_refactor.py PASS 0 1.35 verify: 8 checks, 8 passed, 0 failed c94ebf0e965e99c7e67d1ef9df7e35cd184ccb53 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_repro_bounds_refactor.py PASS 0 0.53 === Verification: 10/10 checks passed === 2394abeec25726726edd91696520d788d72a5d88 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_roundtrip_refactor.py PASS 0 0.42 === Verification: 6/6 checks passed === 0f1d7ea9d5e542c8cb70c8e3def85ba684641b2c /root/troupe-standup
|
||||
gate wizard-work/forge/verify_scan_dry_refactor.py PASS 0 2.53 === Verification: 11/11 checks passed === 6fb7aa14b96eea6731c51c25e22572cca009adf4
|
||||
gate wizard-work/forge/verify_selftest_sweep_dry_refactor.py FAIL 1 0.10 [setup] BEFORE snapshot missing: /root/.hermes/profiles/forge/cache/scratch/selftest_sweep_before.py 6f5e1621139c9ab98ec5b25ce03bda88a73cf9cf /root/troupe-standup
|
||||
also-run wizard-work/forge/verify_selftest_sweep_q93_refactor.py FAIL 1 1.48 verify_selftest_sweep_q93_refactor: FAIL (2 failed, 2 checks) 971430df5da46e70371e7f75b8c9d88d095a6ab7
|
||||
gate wizard-work/forge/verify_selftest_sweep_refactor.py PASS 0 86.94 verify_selftest_sweep_refactor: 5 checks, 5 passed, 0 failed cc8782b4edfcd08eb5ed69f06308c1185ffb41e8 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_served_drift_summarise_dry.py PASS 0 4.52 VERDICT: ALL GREEN -- summarise() DRY collapse behavior preserved de5a49160b2c2a8f4759e0607a0f0c7b52b3bec1 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_smell_check_monotonic_refactor.py PASS 0 0.59 === Verification: 9/9 checks passed === 8b8ccea5be0e938fb4bba67080cc65a97762f3f4 /root/troupe-standup
|
||||
gate wizard-work/forge/verify_smell_scan_refactor.py PASS 0 0.30 === Verification: 8/8 checks passed === 499c8e000359e8ef3486a89031673059d471178c /root/troupe-standup
|
||||
gate wizard-work/forge/verify_tool_map_dry_refactor.py FAIL 1 24.84 VERDICT: MISMATCH -- behavior NOT preserved de1d9b7a65cfaaacf029b048228806acbf2f817c /root/troupe-standup
|
||||
also-run wizard-work/forge/verify_triage_dump_pending_refactor.py PASS 0 1.24 === Verification: 13/13 checks passed === 38894d9812a8dfbda63bfa6d01407dba1b3ec042 /root/troupe-standup/inbox/seed-2026-09-26.json
|
||||
gate wizard-work/forge/verify_triage_dump_refactor.py PASS 0 1.07 === Verification: 14/14 checks passed === 611e56d4d3561a2e52d810081255f575126472bc /root/troupe-standup/inbox/seed-2026-09-26.json
|
||||
gate wizard-work/forge/verify_triage_probe_refactor.py PASS 0 1.09 === Verification: 14/14 checks passed === c8c9288477c8170d0b35bd40064ad58e93dcaf6c /root/troupe-standup/inbox/seed-2026-09-26.json
|
||||
gate wizard-work/forge/verify_write_probe_dry_refactor.py FAIL 1 14.14 VERDICT: MISMATCH -- behavior NOT preserved ef4d8c00e4a84495e1612bec4734a366d4869b67 /root/troupe-standup
|
||||
|
@@ -0,0 +1,8 @@
|
||||
# Q87 per-file execution windows -- path <TAB> seconds <TAB> why.
|
||||
# The default window for every file this table does not name is 90 s (--timeout).
|
||||
# A row here says the file's own work costs more than the default, and a TIMEOUT
|
||||
# row's note names the window it actually got, so the ledger cannot disagree
|
||||
# with the run. Every figure below is a measurement at 7b42050, not an estimate.
|
||||
wizard-work/forge/verify_citation_hash_class_dry_refactor.py 300 measured 133.98 s and 144.38 s at 7b42050 (rc 1 both times); it spawns child sweeps and its traceback is a TimeoutExpired from one of them
|
||||
wizard-work/forge/verify_citation_hash_class_portable_default.py 900 live HTTP: 335.90 s, then 350.19 s, then no finish inside 600 s -- the same blob at three runs; 900 s is the gate's ceiling for one file
|
||||
wizard-work/forge/verify_selftest_sweep_refactor.py 300 measured 86.94 s and 95.54 s (rc 0 both); its subject is the repo-wide selftest sweep
|
||||
|
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env python3
|
||||
"""q87_verify.py -- the instrument behind CHECK-q87 (2026-10-31, volt lane).
|
||||
|
||||
WHAT IT GRADES
|
||||
The forge lane gate (`wizard-work/forge/CHECK` -> `verifier_gate.py`) and the
|
||||
ledger it grades. Seven legs, each of which can fail on its own:
|
||||
|
||||
1 selftest the gate's own 10 cells pass (a cell that silently passes on a
|
||||
truthy string is the defect the first draft of the selftest had).
|
||||
2 check `CHECK` in this tree exits 0 and prints the population and the
|
||||
known-red count, so a green cannot hide a red.
|
||||
3 strict the same gate with --strict exits 1 and names the non-PASS rows,
|
||||
which is the switch from "books consistent" to "lane healthy".
|
||||
4 drift in a throwaway clone, one byte appended to a PASS verifier makes
|
||||
--check exit 1 with DRIFT naming that file, and restoring the
|
||||
bytes makes it exit 0 again.
|
||||
5 ungraded a new committed `verify_*.py` with no ledger row makes --check
|
||||
exit 1 with UNGRADED naming it (a file added tomorrow is seen).
|
||||
6 causeless blanking the note of a FAIL row makes --check exit 1 with
|
||||
CAUSELESS (a red must carry its reason).
|
||||
7 portable --check run from a different cwd, against a clone of this tree,
|
||||
returns the same verdict as in-tree -- so the gate is a function
|
||||
of the tree, not of this host's path.
|
||||
|
||||
The clone legs never touch the graded tree: every mutation happens in a
|
||||
TemporaryDirectory copy that is thrown away.
|
||||
|
||||
EXIT CODES
|
||||
0 every leg passed / 1 a leg failed / 2 could not grade (no git, no clone).
|
||||
|
||||
Usage:
|
||||
python3 q87_verify.py [--repo /root/troupe-standup] [--verbose]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
LANE = "wizard-work/forge"
|
||||
GATE = "verifier_gate.py"
|
||||
LEDGER = "VERIFIER_LEDGER.tsv"
|
||||
WINDOWS = "VERIFIER_WINDOWS.tsv"
|
||||
CHECK = "CHECK"
|
||||
FILES = (GATE, LEDGER, WINDOWS, CHECK)
|
||||
|
||||
|
||||
def run(argv, cwd=None, timeout=600):
|
||||
r = subprocess.run(argv, cwd=cwd, capture_output=True, text=True,
|
||||
errors="replace", timeout=timeout,
|
||||
env=dict(os.environ, PYTHONDONTWRITEBYTECODE="1"))
|
||||
return r.returncode, (r.stdout or "") + (r.stderr or "")
|
||||
|
||||
|
||||
def gate(repo, *extra, cwd=None):
|
||||
return run([sys.executable, str(Path(repo) / LANE / GATE), "--repo", str(repo)]
|
||||
+ list(extra), cwd=cwd)
|
||||
|
||||
|
||||
def clone_of(repo, dest):
|
||||
rc, out = run(["git", "clone", "-q", "--no-hardlinks", str(repo), str(dest)])
|
||||
if rc != 0:
|
||||
raise RuntimeError("git clone failed: %s" % out.strip()[:200])
|
||||
for name in FILES:
|
||||
src = Path(repo) / LANE / name
|
||||
if src.exists():
|
||||
shutil.copyfile(src, Path(dest) / LANE / name)
|
||||
return Path(dest)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--repo", default=str(Path(__file__).resolve().parents[2]))
|
||||
ap.add_argument("--verbose", action="store_true")
|
||||
args = ap.parse_args()
|
||||
repo = Path(args.repo).resolve()
|
||||
if not (repo / ".git").exists():
|
||||
print("q87: %s is not a git worktree -- could not grade" % repo)
|
||||
return 2
|
||||
|
||||
legs = []
|
||||
|
||||
def leg(name, ok, detail):
|
||||
legs.append((name, bool(ok), detail))
|
||||
print(" [%s] %-9s %s" % ("ok " if ok else "FAIL", name, detail), flush=True)
|
||||
|
||||
# 1 -- the gate's own cells.
|
||||
rc, out = gate(repo, "--selftest")
|
||||
tail = [l for l in out.splitlines() if "selftest:" in l]
|
||||
leg("selftest", rc == 0 and bool(tail) and "0 failed" in tail[-1]
|
||||
and "cells," in tail[-1],
|
||||
(tail[-1] if tail else "no selftest summary (rc %d)" % rc))
|
||||
|
||||
# 2 -- the shipped gate is green and says what it is green about. The count
|
||||
# is read from the ledger, not hardcoded: a ledger that re-emits after a
|
||||
# verifier changes would otherwise break this leg for the wrong reason.
|
||||
rc, out = run(["bash", str(repo / LANE / CHECK)])
|
||||
pop = [l for l in out.splitlines() if "committed verifier(s)" in l]
|
||||
red = [l for l in out.splitlines() if l.strip().startswith("known-red")]
|
||||
led = (repo / LANE / LEDGER).read_text(encoding="utf-8")
|
||||
n_rows = sum(1 for l in led.splitlines() if l.strip() and not l.startswith("#"))
|
||||
n_red = sum(1 for l in led.splitlines()
|
||||
if l.strip() and not l.startswith("#") and l.split("\t")[2] != "PASS")
|
||||
ok2 = (rc == 0 and bool(pop) and bool(red)
|
||||
and "%d committed verifier(s), %d ledger row(s)" % (n_rows, n_rows) in pop[0]
|
||||
and ("%d of %d" % (n_red, n_rows)) in red[0] and n_red >= 1)
|
||||
leg("check", ok2, "%s; %s" % (pop[0].strip() if pop else "no population line",
|
||||
red[0].strip() if red else "no known-red line"))
|
||||
|
||||
# 3 -- strict is the lane's health, and today it is not healthy.
|
||||
rc, out = gate(repo, "--check", "--strict")
|
||||
strict_n = [l for l in out.splitlines() if l.strip().startswith("verifier_gate:")
|
||||
and "strict" in l]
|
||||
n_strict = int(strict_n[-1].split()[1]) if strict_n else 0
|
||||
leg("strict", rc == 1 and n_strict >= 18 and "strict: every non-PASS row" in out,
|
||||
"rc %d, %d fault(s) from the non-PASS rows" % (rc, n_strict))
|
||||
|
||||
# 4..6 -- mutants in a throwaway clone.
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
try:
|
||||
clone = clone_of(repo, Path(tmp) / "c")
|
||||
except RuntimeError as exc:
|
||||
print("q87: %s" % exc)
|
||||
return 2
|
||||
target = clone / LANE / "verify_chunker_refactor.py"
|
||||
|
||||
rc0, _ = gate(clone, "--check")
|
||||
base_rc, base_sha = run(["git", "-C", str(clone), "rev-parse", "HEAD"])
|
||||
original = target.read_text(encoding="utf-8")
|
||||
target.write_text(original + "# one byte of drift\n", encoding="utf-8")
|
||||
# the ledger binds to the COMMITTED blob, so the mutant has to be a
|
||||
# commit -- an edit left in the worktree is the UNCOMMITTED note, not DRIFT
|
||||
run(["git", "-C", str(clone), "add", str(target.relative_to(clone))])
|
||||
run(["git", "-C", str(clone), "-c", "user.email=q87@example.invalid",
|
||||
"-c", "user.name=q87", "commit", "-qm", "drifts one verifier"])
|
||||
new_rc, new_sha = run(["git", "-C", str(clone), "rev-parse", "HEAD"])
|
||||
committed = new_rc == 0 and new_sha.strip() != base_sha.strip()
|
||||
rc1, out1 = gate(clone, "--check")
|
||||
drift_line = [l for l in out1.splitlines()
|
||||
if l.strip().startswith("DRIFT ")
|
||||
and "verify_chunker_refactor.py" in l]
|
||||
if committed:
|
||||
run(["git", "-C", str(clone), "reset", "-q", "--hard", "HEAD~1"])
|
||||
rc2, _ = gate(clone, "--check")
|
||||
leg("drift", committed and rc0 == 0 and rc1 == 1 and rc2 == 0
|
||||
and bool(drift_line),
|
||||
"clean %d -> committed edit %d%s -> reverted %d"
|
||||
% (rc0, rc1, (" (%s)" % drift_line[0].strip()[:52]) if drift_line else "",
|
||||
rc2))
|
||||
|
||||
new = clone / LANE / "verify_added_tomorrow.py"
|
||||
new.write_text("print('nothing')\n", encoding="utf-8")
|
||||
run(["git", "-C", str(clone), "add", str(new.relative_to(clone))])
|
||||
run(["git", "-C", str(clone), "-c", "user.email=q87@example.invalid",
|
||||
"-c", "user.name=q87", "commit", "-qm", "adds a verifier"])
|
||||
rc3, out3 = gate(clone, "--check")
|
||||
leg("ungraded", rc3 == 1 and "UNGRADED" in out3 and "verify_added_tomorrow" in out3,
|
||||
"rc %d%s" % (rc3, " naming UNGRADED verify_added_tomorrow.py"
|
||||
if "UNGRADED" in out3 else " without an UNGRADED fault"))
|
||||
|
||||
# causeless: blank the note of the first FAIL row, in a fresh clone state
|
||||
led = clone / LANE / LEDGER
|
||||
lines = led.read_text(encoding="utf-8").splitlines()
|
||||
hit = None
|
||||
for i, line in enumerate(lines):
|
||||
if line.startswith("#") or not line.strip():
|
||||
continue
|
||||
parts = line.split("\t")
|
||||
if parts[2] == "FAIL":
|
||||
parts[5] = ""
|
||||
lines[i] = "\t".join(parts)
|
||||
hit = parts[1]
|
||||
break
|
||||
led.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
rc4, out4 = gate(clone, "--check")
|
||||
leg("causeless", rc4 == 1 and hit and ("CAUSELESS" in out4),
|
||||
"rc %d%s" % (rc4, (" naming CAUSELESS %s" % hit) if "CAUSELESS" in out4
|
||||
else " with no CAUSELESS fault"))
|
||||
|
||||
# 7 -- portability: same verdict from another cwd, through the clone.
|
||||
rc5, out5 = gate(clone, "--check", cwd="/")
|
||||
leg("portable", rc5 == 1 and "CAUSELESS" in out5,
|
||||
"from cwd=/ against the clone the same fault is read (rc %d)" % rc5)
|
||||
|
||||
bad = [l for l in legs if not l[1]]
|
||||
print("q87: %d leg(s), %d passed, %d fault(s)"
|
||||
% (len(legs), len(legs) - len(bad), len(bad)))
|
||||
return 0 if not bad else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,802 @@
|
||||
#!/usr/bin/env python3
|
||||
"""verifier_gate.py -- the runner `wizard-work/forge` never had (Q87, 2026-10-31).
|
||||
|
||||
WHY THIS EXISTS
|
||||
At `431bca1` the pipe measured 34 `verify_*.py` in this lane and no `CHECK`;
|
||||
the correction at `d56105a` moved the count to 36 and the runner count stayed
|
||||
zero. At `7b42050` the lane holds 39 and `tools/verify.py` still names
|
||||
`wizard-work/forge` 0 times. A verifier nothing runs is an unread claim: its
|
||||
verdict is whatever its author last saw by hand, and this lane would not
|
||||
notice if all 39 were broken at once. This gate executes them, keeps one row
|
||||
per file bound to the blob the rc was measured against, and re-grades that
|
||||
ledger on every run without re-executing it.
|
||||
|
||||
WHAT IT MEASURES -- one row per committed verifier, one verdict per row
|
||||
PASS rc 0: the file ran and its checks held.
|
||||
FAIL rc 1: the file ran and its checks did NOT hold. A finding, not a
|
||||
bookkeeping error -- the file's claim is false at this tree.
|
||||
REFUSED rc 2: the file declined to grade. Not a pass.
|
||||
OTHER_RC rc above 2 that the house contract does not define.
|
||||
TIMEOUT no finish inside its per-file window (see WINDOWS below).
|
||||
ERROR the interpreter could not run the file at all (no rc to grade).
|
||||
MISSING the row's path is not in the tree at --rev.
|
||||
|
||||
A non-PASS row must carry a reason in its note column. A red with no recorded
|
||||
cause is a FAULT (`CAUSELESS`), because "it is red and nobody wrote down why"
|
||||
is the state this lane has been in since 2026-10-27.
|
||||
|
||||
THE LEDGER IS BOUND TO CONTENT
|
||||
Column 7 is the git blob id each rc was measured against. `--check` recomputes
|
||||
it at its own `--rev`; a mismatch is DRIFT, so editing a verifier is a request
|
||||
for `--emit` rather than a free pass on an old green. Column 8 is a static
|
||||
host-pin probe: the committed blob is parsed as an AST and every non-docstring
|
||||
string constant is tested for an absolute host root (`/root/troupe-standup`,
|
||||
`/root/.hermes`). A PASS carrying one prints as `PASS*` and is reported
|
||||
separately, because a verifier that reads this host's copy graded this host
|
||||
and not the tree that invoked it (Q62/Q106's class).
|
||||
|
||||
WHAT IT DELIBERATELY DOES NOT DO
|
||||
`--check` (the default, what `CHECK` runs) does NOT execute anything: it
|
||||
grades the ledger against the tree in well under a second. Execution is
|
||||
`--emit`, it takes minutes here (measured: 470 s for 39 files at a 75 s
|
||||
default window, plus three files that need longer), and a gate that takes
|
||||
minutes is a gate that gets skipped. `--strict` promotes any non-PASS row to
|
||||
a fatal fault, which is the switch between "are the books consistent?" and
|
||||
"are the lane's verifiers true?" -- today the honest answer to the second is
|
||||
no, and the rows say which ones.
|
||||
|
||||
It does not decide whether a verifier is *good*, only whether it exits 0. A
|
||||
verifier that asserts nothing exits 0. That is the same boundary q65_runner
|
||||
draws for the repo-wide population and it is not moved here.
|
||||
|
||||
WINDOWS
|
||||
A single `--timeout` for the whole lane is wrong: three files here need more
|
||||
than 75 s and one of them is a recursive sweep. `VERIFIER_WINDOWS.tsv` names
|
||||
one window per path with the reason; `--timeout` stays the fallback. A
|
||||
TIMEOUT row's note names the window it actually got, so the ledger cannot
|
||||
quietly disagree with the run.
|
||||
|
||||
EXIT CODES -- the house contract
|
||||
0 clean / 1 graded a fault / 2 could not grade (no git, no python3, no lane).
|
||||
|
||||
Usage:
|
||||
python3 verifier_gate.py --check # grade the ledger (fast, no runs)
|
||||
python3 verifier_gate.py --check --strict # ...and any non-PASS row is fatal
|
||||
python3 verifier_gate.py --emit # execute every verifier, rewrite
|
||||
python3 verifier_gate.py --census # who else runs each file
|
||||
python3 verifier_gate.py --selftest # the gate's own cells
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
LANE = "wizard-work/forge"
|
||||
HERE = Path(__file__).resolve().parent
|
||||
ROOT = HERE.parents[1]
|
||||
LEDGER = HERE / "VERIFIER_LEDGER.tsv"
|
||||
WINDOWS = HERE / "VERIFIER_WINDOWS.tsv"
|
||||
|
||||
PASS, FAIL, REFUSED = "PASS", "FAIL", "REFUSED"
|
||||
OTHER_RC, TIMEOUT, ERROR, MISSING = "OTHER_RC", "TIMEOUT", "ERROR", "MISSING"
|
||||
VERDICTS = (PASS, FAIL, REFUSED, OTHER_RC, TIMEOUT, ERROR, MISSING)
|
||||
|
||||
HOST_ROOTS = ("/root/troupe-standup", "/root/.hermes")
|
||||
VERDICT_OF_RC = {0: PASS, 1: FAIL, 2: REFUSED}
|
||||
|
||||
|
||||
def git(repo, *args):
|
||||
"""(rc, stdout, stderr) -- never raises, so a missing git is rc 2 upstream."""
|
||||
try:
|
||||
r = subprocess.run(["git", "-C", str(repo)] + list(args),
|
||||
capture_output=True, text=True, errors="replace",
|
||||
timeout=120)
|
||||
return r.returncode, r.stdout, r.stderr
|
||||
except (OSError, subprocess.SubprocessError) as exc:
|
||||
return 127, "", str(exc)
|
||||
|
||||
|
||||
def population(repo, rev):
|
||||
"""Committed verify_*.py under the lane at `rev` -- from the tree, not a list.
|
||||
|
||||
The population is read from git rather than from a directory glob because
|
||||
the question is "what did this commit ship", and an untracked scratch file
|
||||
beside the lane is not a claim about the commit.
|
||||
"""
|
||||
rc, out, err = git(repo, "ls-tree", "-r", "--name-only", rev, "--", LANE)
|
||||
if rc != 0:
|
||||
raise SystemExit("git ls-tree %s failed: %s" % (rev, (err or "").strip()))
|
||||
paths = [p.strip() for p in out.splitlines() if p.strip()]
|
||||
return sorted(p for p in paths
|
||||
if p.startswith(LANE + "/")
|
||||
and os.path.basename(p).startswith("verify_")
|
||||
and p.endswith(".py"))
|
||||
|
||||
|
||||
def blob_of(repo, rev, path):
|
||||
rc, out, _ = git(repo, "rev-parse", "%s:%s" % (rev, path))
|
||||
return out.strip() if rc == 0 else ""
|
||||
|
||||
|
||||
def host_pin(repo, rev, path):
|
||||
"""The first absolute host root this blob reads, or ''.
|
||||
|
||||
Parsed rather than grepped: a docstring that *names* the host root is a
|
||||
description, and the Q85 lesson is that a text scan turns description into
|
||||
finding. Only string constants outside docstrings count.
|
||||
"""
|
||||
rc, blob, _ = git(repo, "show", "%s:%s" % (rev, path))
|
||||
if rc != 0:
|
||||
return ""
|
||||
try:
|
||||
tree = ast.parse(blob)
|
||||
except SyntaxError:
|
||||
return "unparseable"
|
||||
docs = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef,
|
||||
ast.ClassDef)):
|
||||
body = getattr(node, "body", [])
|
||||
if body and isinstance(body[0], ast.Expr) \
|
||||
and isinstance(body[0].value, ast.Constant) \
|
||||
and isinstance(body[0].value.value, str):
|
||||
docs.add(id(body[0].value))
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Constant) and isinstance(node.value, str):
|
||||
if id(node) in docs:
|
||||
continue
|
||||
for root in HOST_ROOTS:
|
||||
if root in node.value:
|
||||
hits = [l for l in node.value.splitlines() if root in l]
|
||||
return (hits[0] if hits else root).strip()[:120]
|
||||
return ""
|
||||
|
||||
|
||||
def read_windows(path):
|
||||
"""{path: (seconds, why)} -- one window per file, the reason mandatory."""
|
||||
win = {}
|
||||
p = Path(path)
|
||||
if not p.exists():
|
||||
return win
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip() or line.lstrip().startswith("#"):
|
||||
continue
|
||||
parts = line.split("\t")
|
||||
if len(parts) < 3:
|
||||
raise SystemExit("windows row needs path<TAB>seconds<TAB>why: %r" % line)
|
||||
try:
|
||||
win[parts[0].strip()] = (float(parts[1]), parts[2].strip())
|
||||
except ValueError:
|
||||
raise SystemExit("window seconds is not a number: %r" % line)
|
||||
return win
|
||||
|
||||
|
||||
def run_one(repo, rel, timeout, windows):
|
||||
"""Execute one verifier in its own directory, under its own TMPDIR.
|
||||
|
||||
THE SANDBOX IS NOT A LUXURY. Five files in this lane clean up with
|
||||
`shutil.rmtree(os.path.dirname(<their temp path>))`, which is the PARENT of
|
||||
the directory they made -- i.e. `$TMPDIR` itself:
|
||||
verify_append_only_audit_dry_refactor.py:304, verify_append_only_owners_
|
||||
dry_refactor.py:238, verify_gate_blame_dry_refactor.py:178, verify_served_
|
||||
drift_summarise_dry.py:192, verify_write_probe_dry_refactor.py:133.
|
||||
Running them with the caller's TMPDIR deleted the caller's scratch tree (this
|
||||
item's own first measurement, at 09:27, died that way: the probe's TSV was
|
||||
gone mid-run) and every later file failed with FileNotFoundError naming a
|
||||
directory it had just created, because `mkdtemp` raises on the missing
|
||||
PARENT. Each file now gets `<own base>/tb` as TMPDIR, so a verifier that
|
||||
deletes its parent takes out a throwaway directory the gate removes itself.
|
||||
"""
|
||||
window, why = windows.get(rel, (timeout, ""))
|
||||
full = Path(repo) / rel
|
||||
if not full.exists():
|
||||
return {"path": rel, "verdict": MISSING, "rc": None, "seconds": 0.0,
|
||||
"note": "not in worktree"}
|
||||
base = tempfile.mkdtemp(prefix="forge_gate_")
|
||||
tb = os.path.join(base, "tb")
|
||||
os.makedirs(tb, exist_ok=True)
|
||||
env = dict(os.environ, PYTHONDONTWRITEBYTECODE="1", PYTHONUNBUFFERED="1",
|
||||
TMPDIR=tb, TEMP=tb, TMP=tb)
|
||||
env.pop("PYTHONOPTIMIZE", None)
|
||||
t0 = time.time()
|
||||
try:
|
||||
r = subprocess.run([sys.executable, "-B", full.name], cwd=str(full.parent),
|
||||
env=env, capture_output=True, timeout=window,
|
||||
text=True, errors="replace")
|
||||
rc, so, se = r.returncode, r.stdout, r.stderr
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"path": rel, "verdict": TIMEOUT, "rc": None,
|
||||
"seconds": round(time.time() - t0, 2),
|
||||
"note": "no finish in %ss%s"
|
||||
% (window, (" (%s)" % why) if why else "")}
|
||||
except OSError as exc:
|
||||
return {"path": rel, "verdict": ERROR, "rc": None, "seconds": 0.0,
|
||||
"note": str(exc)[:160]}
|
||||
finally:
|
||||
shutil.rmtree(base, ignore_errors=True)
|
||||
secs = round(time.time() - t0, 2)
|
||||
verdict = VERDICT_OF_RC.get(rc, OTHER_RC if rc > 2 else ERROR)
|
||||
note = verdict_line(so, se, rc)
|
||||
return {"path": rel, "verdict": verdict, "rc": rc, "seconds": secs,
|
||||
"note": note}
|
||||
|
||||
|
||||
def verdict_line(so, se, rc):
|
||||
"""The file's own sentence, not whichever stream happened to end last.
|
||||
|
||||
The first draft took stderr's last line, and `verify_append_only_audit_dry_
|
||||
refactor.py` wrote its cause to stdout while stderr ended on a SyntaxWarning
|
||||
fragment -- the row read `(b"dash \\u2014 here\\nplain\\n", ...)` and lost the
|
||||
`VERDICT: MISMATCH` it had printed. A verdict-looking line is preferred, from
|
||||
stdout first; the last line of either stream is the fallback.
|
||||
"""
|
||||
keys = ("VERDICT", "FAIL", "checks", "PASS", "failed", "Error")
|
||||
for text in (so, se):
|
||||
lines = [l.strip() for l in (text or "").splitlines() if l.strip()]
|
||||
for line in reversed(lines):
|
||||
if line.startswith("[ok") or line.startswith("ok:"):
|
||||
continue # a passing cell is not this row's cause
|
||||
if any(k in line for k in keys):
|
||||
return line[:160]
|
||||
# Nothing verdict-shaped: prefer stderr's last line, because that is where a
|
||||
# traceback puts the reason, and stdout's last line is often a passing cell
|
||||
# the file printed before the thing that killed it.
|
||||
for text in (se, so):
|
||||
lines = [l.strip() for l in (text or "").splitlines() if l.strip()]
|
||||
if lines:
|
||||
return lines[-1][:160]
|
||||
return "" if rc == 0 else "exited rc %d with no stdout/stderr" % rc
|
||||
|
||||
|
||||
HEADER = """\
|
||||
# Q87 verifier ledger -- one row per committed verify_*.py in wizard-work/forge,
|
||||
# with the exit code it actually returned at the stated rev.
|
||||
# row: <invocation>\t<path>\t<verdict>\t<rc>\t<seconds>\t<note>\t<blob>\t<host>
|
||||
# verdicts: PASS rc0 / FAIL rc1 / REFUSED rc2 / OTHER_RC >2 / TIMEOUT / ERROR /
|
||||
# MISSING. A non-PASS row must carry a note: the cause, in the file's words.
|
||||
# invocation is what runs the file: `gate` when this gate is the only runner,
|
||||
# `also-run` when another committed line invokes it (the census evidence is
|
||||
# `--census`'s job, not a column here).
|
||||
# blob is the git blob id the rc was measured against; --check recomputes it and
|
||||
# a mismatch is DRIFT, so editing a verifier is a request for --emit.
|
||||
# host is the absolute host root the blob reads (AST, non-docstring strings), or
|
||||
# empty. A PASS carrying one prints as PASS* and is counted separately.
|
||||
# seconds is wall clock under that file's window (VERIFIER_WINDOWS.tsv); a
|
||||
# TIMEOUT note names the window it got.
|
||||
"""
|
||||
|
||||
|
||||
def read_ledger(path):
|
||||
rows, meta = [], {}
|
||||
p = Path(path)
|
||||
if not p.exists():
|
||||
return rows, meta, "no ledger at %s (run --emit)" % p
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
if line.startswith("#"):
|
||||
m = re.match(r"#\s*(rev|emitted_at):\s*(\S+)", line)
|
||||
if m:
|
||||
meta[m.group(1)] = m.group(2)
|
||||
continue
|
||||
parts = line.split("\t")
|
||||
if len(parts) < 6:
|
||||
return rows, meta, "ledger row has %d column(s): %r" % (len(parts), line)
|
||||
while len(parts) < 8:
|
||||
parts.append("")
|
||||
rows.append({"invocation": parts[0], "path": parts[1], "verdict": parts[2],
|
||||
"rc": parts[3], "seconds": parts[4], "note": parts[5],
|
||||
"blob": parts[6], "host": parts[7]})
|
||||
return rows, meta, ""
|
||||
|
||||
|
||||
def write_ledger(path, rows, repo, rev):
|
||||
rc, out, _ = git(repo, "rev-parse", "--short", rev)
|
||||
sha = out.strip() or rev
|
||||
body = [HEADER.rstrip("\n"),
|
||||
"# rev: %s" % sha,
|
||||
"# emitted_at: %sZ" % time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime())]
|
||||
for r in sorted(rows, key=lambda x: x["path"]):
|
||||
body.append("\t".join([r["invocation"], r["path"], r["verdict"],
|
||||
"" if r["rc"] is None else str(r["rc"]),
|
||||
"%.2f" % float(r["seconds"]), r["note"],
|
||||
r["blob"], r["host"]]))
|
||||
Path(path).write_text("\n".join(body) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def rc_consistent(row):
|
||||
"""Wording a verdict has to match. Returns a reason string or ''."""
|
||||
v, rc = row["verdict"], row["rc"]
|
||||
if v not in VERDICTS:
|
||||
return "verdict %r is not one of %s" % (v, "/".join(VERDICTS))
|
||||
if v in (TIMEOUT, ERROR, MISSING):
|
||||
return "" if rc == "" else "%s carries rc %s" % (v, rc)
|
||||
if rc == "" or not re.fullmatch(r"-?\d+", rc):
|
||||
return "%s carries rc %r" % (v, rc)
|
||||
n = int(rc)
|
||||
if v == PASS and n != 0:
|
||||
return "PASS with rc %d" % n
|
||||
if v == FAIL and n != 1:
|
||||
return "FAIL with rc %d" % n
|
||||
if v == REFUSED and n != 2:
|
||||
return "REFUSED with rc %d" % n
|
||||
if v == OTHER_RC and n <= 2:
|
||||
return "OTHER_RC with rc %d" % n
|
||||
return ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- invocation
|
||||
DOC_SUFFIXES = (".md", ".txt", ".log", ".tsv", ".json", ".pyc")
|
||||
RUN_TOKENS = ("python3", "python ", "sys.executable", "subprocess", "os.system",
|
||||
"pytest", "bash ")
|
||||
|
||||
|
||||
def invocation_census(repo, rev, paths, exclude_rel):
|
||||
"""{path: (class, evidence)} -- who runs this file, from committed content.
|
||||
|
||||
A mention is not an invocation. A reference counts as a RUN only when it sits
|
||||
in an executable file (.py/.sh/CHECK*, not a document) on a line that also
|
||||
names an interpreter or a spawner; every other reference is prose. That is
|
||||
the distinction reply 21 asked for, and it is a rule with a test: --selftest
|
||||
holds one prose mention and one `python3 x.py` line and requires the
|
||||
classifier to separate them.
|
||||
"""
|
||||
rc, out, _ = git(repo, "ls-tree", "-r", "--name-only", rev)
|
||||
if rc != 0:
|
||||
raise SystemExit("git ls-tree failed at %s" % rev)
|
||||
files = [f.strip() for f in out.splitlines() if f.strip()]
|
||||
texts = {}
|
||||
for f in files:
|
||||
base = os.path.basename(f)
|
||||
if base.endswith(DOC_SUFFIXES):
|
||||
continue
|
||||
rc, blob, _ = git(repo, "show", "%s:%s" % (rev, f))
|
||||
if rc == 0:
|
||||
texts[f] = blob
|
||||
res = {}
|
||||
for path in paths:
|
||||
name = os.path.basename(path)
|
||||
runs = []
|
||||
named = 0
|
||||
for f, blob in texts.items():
|
||||
if f == path or f == exclude_rel:
|
||||
continue
|
||||
if name not in blob:
|
||||
continue
|
||||
for i, line in enumerate(blob.splitlines(), 1):
|
||||
if name not in line:
|
||||
continue
|
||||
if line.lstrip().startswith(("#", "//", "*", "--")):
|
||||
named += 1
|
||||
continue
|
||||
if any(t in line for t in RUN_TOKENS):
|
||||
runs.append("%s:%d" % (f, i))
|
||||
else:
|
||||
named += 1
|
||||
if runs:
|
||||
res[path] = ("also-run", ", ".join(runs[:2]))
|
||||
elif named:
|
||||
res[path] = ("gate", "%d prose ref(s), no invocation" % named)
|
||||
else:
|
||||
res[path] = ("gate", "named by nothing else at all")
|
||||
return res
|
||||
|
||||
|
||||
def check(repo, rev, ledger_path, strict, quiet=False):
|
||||
try:
|
||||
pop = population(repo, rev)
|
||||
except SystemExit as exc:
|
||||
print("verifier_gate: %s" % exc)
|
||||
return 2
|
||||
if not pop:
|
||||
print("verifier_gate: population is empty at %s -- could not grade" % rev)
|
||||
return 2
|
||||
rows, meta, err = read_ledger(ledger_path)
|
||||
if err:
|
||||
print("verifier_gate: %s" % err)
|
||||
return 2
|
||||
faults, notes = [], []
|
||||
seen = {}
|
||||
for r in rows:
|
||||
seen[r["path"]] = seen.get(r["path"], 0) + 1
|
||||
why = rc_consistent(r)
|
||||
if why:
|
||||
faults.append("INCONSISTENT %s: %s" % (r["path"], why))
|
||||
if r["verdict"] != PASS and not r["note"].strip():
|
||||
faults.append("CAUSELESS %s: %s row with no reason in the note"
|
||||
% (r["path"], r["verdict"]))
|
||||
if not r["blob"]:
|
||||
faults.append("UNBOUND %s: no blob column -- the row does not say "
|
||||
"which content its rc describes (re-emit)" % r["path"])
|
||||
for p, n in sorted(seen.items()):
|
||||
if n > 1:
|
||||
faults.append("DUPLICATE %s: %d rows" % (p, n))
|
||||
for r in rows:
|
||||
if r["path"] not in pop:
|
||||
continue
|
||||
now = blob_of(repo, rev, r["path"])
|
||||
if r["blob"] and now and now != r["blob"]:
|
||||
faults.append("DRIFT %s: rc measured against blob %s, tree at %s has %s"
|
||||
% (r["path"], r["blob"][:10], rev, now[:10]))
|
||||
# A dirty worktree is not a commit, so it cannot be DRIFT -- but a verifier
|
||||
# whose working bytes are not the blob its rc describes is exactly the case
|
||||
# where the ledger's verdict is about content you are not running. Named,
|
||||
# not fatal: the fix is a commit (then DRIFT/UNGRADED decide) or --emit.
|
||||
for r in rows:
|
||||
if r["path"] not in pop or not r["blob"]:
|
||||
continue
|
||||
rc_w, out_w, _ = git(repo, "hash-object", "--", r["path"])
|
||||
if rc_w == 0 and out_w.strip() and out_w.strip() != r["blob"]:
|
||||
notes.append("UNCOMMITTED %s: worktree bytes are not the blob its rc "
|
||||
"was measured against -- commit it, then re-emit or "
|
||||
"expect DRIFT" % r["path"])
|
||||
ungraded = sorted(set(pop) - set(seen))
|
||||
for p in ungraded:
|
||||
faults.append("UNGRADED %s: committed at %s, no ledger row" % (p, rev))
|
||||
stale = sorted(p for p in set(seen) - set(pop)
|
||||
if not (Path(repo) / p).exists())
|
||||
for p in stale:
|
||||
faults.append("STALE %s: ledger row, file not in the tree" % p)
|
||||
outside = sorted(p for p in set(seen) - set(pop) if (Path(repo) / p).exists())
|
||||
for p in outside:
|
||||
notes.append("OUTSIDE %s: ledger row, not a committed verifier at %s "
|
||||
"(untracked beside the lane or renamed)" % (p, rev))
|
||||
counts = {}
|
||||
for r in rows:
|
||||
counts[r["verdict"]] = counts.get(r["verdict"], 0) + 1
|
||||
pinned = [r for r in rows if r["verdict"] == PASS and r["host"]]
|
||||
reds = [r for r in rows if r["verdict"] != PASS]
|
||||
clean = len(rows) - len(reds) - len(pinned)
|
||||
print("verifier_gate -- lane %s at %s, %d committed verifier(s), %d ledger row(s)"
|
||||
% (LANE, rev, len(pop), len(rows)))
|
||||
if meta.get("rev"):
|
||||
print(" ledger rev : %s emitted %s"
|
||||
% (meta["rev"], meta.get("emitted_at", "?")))
|
||||
print(" rows : %s" % ", ".join("%s %d" % (v, counts[v])
|
||||
for v in VERDICTS if counts.get(v)))
|
||||
print(" graded-clean : %d of %d (PASS on a host-independent blob)"
|
||||
% (clean, len(pop)))
|
||||
if pinned:
|
||||
print(" host-pinned : %d PASS row(s) read as PASS* -- they read an "
|
||||
"absolute host root" % len(pinned))
|
||||
for r in pinned:
|
||||
print(" PASS* %s %s" % (r["path"], r["host"]))
|
||||
if reds:
|
||||
print(" known-red : %d of %d row(s) -- every one named, every one "
|
||||
"with its cause" % (len(reds), len(rows)))
|
||||
for r in sorted(reds, key=lambda x: x["path"]):
|
||||
print(" %-8s rc %-3s %s %s" % (r["verdict"], r["rc"] or "-",
|
||||
r["path"], r["note"][:100]))
|
||||
for n in notes:
|
||||
print(" note: " + n)
|
||||
if strict:
|
||||
for r in reds:
|
||||
faults.append("%s %s rc %s %s" % (r["verdict"], r["path"], r["rc"] or "-",
|
||||
r["note"][:80]))
|
||||
for f in faults:
|
||||
print(" " + f)
|
||||
print("verifier_gate: %d fault(s)%s" % (len(faults),
|
||||
" (strict: every non-PASS row counts)"
|
||||
if strict else ""))
|
||||
return 1 if faults else 0
|
||||
|
||||
|
||||
def emit(repo, rev, ledger_path, timeout, windows, only=None, merge=False):
|
||||
pop = population(repo, rev)
|
||||
if only:
|
||||
pop = [p for p in pop if only in p]
|
||||
if not pop:
|
||||
print("verifier_gate: nothing to emit")
|
||||
return 2
|
||||
keep = []
|
||||
if merge and Path(ledger_path).exists():
|
||||
have, _, err = read_ledger(ledger_path)
|
||||
if err:
|
||||
print("verifier_gate: %s" % err)
|
||||
return 2
|
||||
keep = [r for r in have if r["path"] not in set(pop)]
|
||||
rc, before, _ = git(repo, "status", "--porcelain")
|
||||
t0 = time.time()
|
||||
rows = list(keep)
|
||||
for p in pop:
|
||||
r = run_one(repo, p, timeout, windows)
|
||||
r["blob"] = blob_of(repo, rev, p)
|
||||
r["host"] = host_pin(repo, rev, p)
|
||||
r["invocation"] = "gate"
|
||||
rows.append(r)
|
||||
print(" %-8s %6.2fs %s" % (r["verdict"], r["seconds"], p), flush=True)
|
||||
census = {}
|
||||
try:
|
||||
census = invocation_census(repo, rev, [r["path"] for r in rows],
|
||||
os.path.relpath(ledger_path, repo))
|
||||
except SystemExit as exc:
|
||||
print("verifier_gate: census skipped (%s)" % exc)
|
||||
for r in rows:
|
||||
cls, ev = census.get(r["path"], ("gate", ""))
|
||||
r["invocation"] = "also-run" if cls == "also-run" else "gate"
|
||||
write_ledger(ledger_path, rows, repo, rev)
|
||||
rc2, after, _ = git(repo, "status", "--porcelain")
|
||||
led_rel = os.path.relpath(str(ledger_path), str(repo))
|
||||
dirt = sorted(d for d in set(after.splitlines()) - set(before.splitlines())
|
||||
if led_rel not in d)
|
||||
counts = {}
|
||||
for r in rows:
|
||||
counts[r["verdict"]] = counts.get(r["verdict"], 0) + 1
|
||||
print("verifier_gate: %d row(s) in %.1fs -> %s"
|
||||
% (len(rows), time.time() - t0, ledger_path))
|
||||
print(" " + ", ".join("%s %d" % (v, counts[v]) for v in VERDICTS
|
||||
if counts.get(v)))
|
||||
if dirt:
|
||||
print(" WARNING: the run changed the worktree (these verifiers write "
|
||||
"into it):")
|
||||
for d in dirt:
|
||||
print(" " + d)
|
||||
return 0
|
||||
|
||||
|
||||
def census_report(repo, rev, ledger_path):
|
||||
pop = population(repo, rev)
|
||||
res = invocation_census(repo, rev, pop, os.path.relpath(str(ledger_path), str(repo)))
|
||||
run = sorted(p for p, (c, _) in res.items() if c == "also-run")
|
||||
print("verifier_census -- %d committed verifier(s) in %s at %s"
|
||||
% (len(pop), LANE, rev))
|
||||
print(" invoked by another committed line : %d" % len(run))
|
||||
for p in run:
|
||||
print(" %s %s" % (p, res[p][1]))
|
||||
print(" invoked by this gate only : %d" % (len(pop) - len(run)))
|
||||
for p in pop:
|
||||
if p in run:
|
||||
continue
|
||||
print(" %s %s" % (p, res[p][1]))
|
||||
return 0
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- selftest
|
||||
def selftest():
|
||||
"""Cells for the rules that can silently pass. Every cell builds a throwaway
|
||||
git repo, so the gate's logic is graded on a tree it did not ship with."""
|
||||
cells = []
|
||||
|
||||
def cell(name, fn):
|
||||
try:
|
||||
ok, detail = fn()
|
||||
except Exception as exc: # noqa: BLE001 - a cell that raises is a cell that failed
|
||||
ok, detail = False, "%s: %s" % (type(exc).__name__, exc)
|
||||
cells.append((name, ok, detail))
|
||||
|
||||
def mk(tmp, files, commit=True):
|
||||
repo = Path(tmp)
|
||||
(repo / LANE).mkdir(parents=True, exist_ok=True)
|
||||
for rel, body in files.items():
|
||||
p = repo / rel
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(body, encoding="utf-8")
|
||||
git(repo, "init", "-q")
|
||||
git(repo, "config", "user.email", "t@example.invalid")
|
||||
git(repo, "config", "user.name", "t")
|
||||
if commit:
|
||||
git(repo, "add", "-A")
|
||||
git(repo, "commit", "-qm", "cell")
|
||||
return repo
|
||||
|
||||
V = LANE + "/verify_alpha.py"
|
||||
|
||||
def c1():
|
||||
bad = {"path": V, "verdict": PASS, "rc": "1", "note": "", "blob": "x",
|
||||
"host": "", "seconds": "0", "invocation": "gate"}
|
||||
good = dict(bad, verdict=FAIL)
|
||||
a, b = rc_consistent(bad), rc_consistent(good)
|
||||
return (bool(a) and not b, "PASS with rc 1")
|
||||
|
||||
def c2():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: "print(0)\n"})
|
||||
n1 = len(population(repo, "HEAD"))
|
||||
(repo / LANE / "verify_beta.py").write_text("print(0)\n")
|
||||
git(repo, "add", "-A")
|
||||
git(repo, "commit", "-qm", "two")
|
||||
n2 = len(population(repo, "HEAD"))
|
||||
return (n1 == 1 and n2 == 2, "population follows the rev")
|
||||
|
||||
def c3():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: "print(0)\n"})
|
||||
led = repo / LANE / "L.tsv"
|
||||
write_ledger(led, [{"path": V, "verdict": PASS, "rc": 0, "seconds": 0.0,
|
||||
"note": "", "blob": blob_of(repo, "HEAD", V),
|
||||
"host": "", "invocation": "gate"}], repo, "HEAD")
|
||||
r0 = check(repo, "HEAD", led, False)
|
||||
(repo / V).write_text("print(1)\n")
|
||||
git(repo, "add", "-A")
|
||||
git(repo, "commit", "-qm", "edit")
|
||||
r1 = check(repo, "HEAD", led, False)
|
||||
return (r0 == 0 and r1 == 1, "a drifted blob is a fault")
|
||||
|
||||
def c4():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: "print(0)\n"})
|
||||
led = repo / LANE / "L.tsv"
|
||||
write_ledger(led, [], repo, "HEAD")
|
||||
return (check(repo, "HEAD", led, False) == 1, "a committed verifier with no row faults")
|
||||
|
||||
def c5():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: "print(0)\n"})
|
||||
led = repo / LANE / "L.tsv"
|
||||
write_ledger(led, [{"path": LANE + "/verify_gone.py", "verdict": PASS,
|
||||
"rc": 0, "seconds": 0.0, "note": "", "blob": "abc",
|
||||
"host": "", "invocation": "gate"}], repo, "HEAD")
|
||||
return (check(repo, "HEAD", led, False) == 1, "a row for a deleted file faults")
|
||||
|
||||
def c6():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
body = ('"""Reads /root/troupe-standup in a docstring."""\n'
|
||||
'P = "/root/troupe-standup/inbox"\n')
|
||||
repo = mk(tmp, {V: body})
|
||||
hit = host_pin(repo, "HEAD", V)
|
||||
repo2 = mk(tmp + "/b", {V: '"""Reads /root/troupe-standup."""\n'})
|
||||
miss = host_pin(repo2, "HEAD", V)
|
||||
return (hit != "" and miss == "", "host pin is AST, not prose")
|
||||
|
||||
def c7():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: "print(0)\n",
|
||||
"CHECK": "python3 %s\n" % V,
|
||||
"notes/x.md": "see %s\n" % V})
|
||||
res = invocation_census(repo, "HEAD", [V], "")
|
||||
return (res[V][0] == "also-run", "prose is not an invocation")
|
||||
|
||||
def c8():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: "import sys\nsys.exit(1)\n",
|
||||
LANE + "/verify_ok.py": "print(0)\n"})
|
||||
led = repo / LANE / "L.tsv"
|
||||
rc = emit(repo, "HEAD", led, 30.0, {}, only=None)
|
||||
rows, _, err = read_ledger(led)
|
||||
by = {r["path"]: r for r in rows}
|
||||
ok = (rc == 0 and not err and by[V]["verdict"] == FAIL
|
||||
and by[V]["note"] != "" and by[LANE + "/verify_ok.py"]["verdict"] == PASS)
|
||||
return (ok, "emit records rc and a cause")
|
||||
|
||||
def c9():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: "print(0)\n"})
|
||||
led = repo / LANE / "L.tsv"
|
||||
write_ledger(led, [{"path": V, "verdict": FAIL, "rc": 1, "seconds": 0.0,
|
||||
"note": "", "blob": blob_of(repo, "HEAD", V),
|
||||
"host": "", "invocation": "gate"}], repo, "HEAD")
|
||||
return (check(repo, "HEAD", led, False) == 1, "a red with no cause is a fault")
|
||||
|
||||
def c10():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: "print(0)\n"})
|
||||
led = repo / LANE / "L.tsv"
|
||||
blob = blob_of(repo, "HEAD", V)
|
||||
write_ledger(led, [{"path": V, "verdict": FAIL, "rc": 1, "seconds": 0.0,
|
||||
"note": "known red", "blob": blob, "host": "",
|
||||
"invocation": "gate"}], repo, "HEAD")
|
||||
loose = check(repo, "HEAD", led, False)
|
||||
strict = check(repo, "HEAD", led, True)
|
||||
return (loose == 0 and strict == 1, "strict promotes a red row")
|
||||
|
||||
def c11():
|
||||
import contextlib
|
||||
import io
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: "print(0)\n"})
|
||||
led = repo / LANE / "L.tsv"
|
||||
write_ledger(led, [{"path": V, "verdict": PASS, "rc": 0, "seconds": 0.0,
|
||||
"note": "", "blob": blob_of(repo, "HEAD", V),
|
||||
"host": "", "invocation": "gate"}], repo, "HEAD")
|
||||
(repo / V).write_text("print(0) # uncommitted edit\n")
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
rc = check(repo, "HEAD", led, False)
|
||||
out = buf.getvalue()
|
||||
return (rc == 0 and "UNCOMMITTED" in out,
|
||||
"a dirty worktree is named, not fatal")
|
||||
|
||||
def c12():
|
||||
body = ("import sys\n"
|
||||
"sys.stderr.write('warning: bad escape \\\\u\\n')\n"
|
||||
"print('VERDICT: MISMATCH -- behavior NOT preserved')\n"
|
||||
"sys.exit(1)\n")
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: body})
|
||||
led = repo / LANE / "L.tsv"
|
||||
emit(repo, "HEAD", led, 30.0, {}, only=None)
|
||||
rows, _, err = read_ledger(led)
|
||||
by = {r["path"]: r for r in rows}
|
||||
note = by[V]["note"] if V in by else ""
|
||||
return (not err and "VERDICT: MISMATCH" in note,
|
||||
"the note is the file's verdict line: %r" % note[:60])
|
||||
|
||||
def c13():
|
||||
body = ("import os, shutil, tempfile\n"
|
||||
"sandbox = tempfile.mkdtemp(prefix='x_', dir=os.environ['TMPDIR'])\n"
|
||||
"shutil.rmtree(os.path.dirname(sandbox), ignore_errors=True)\n"
|
||||
"print('cleaned up my parent')\n")
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
repo = mk(tmp, {V: body})
|
||||
led = repo / LANE / "L.tsv"
|
||||
survive = tempfile.mkdtemp(prefix="survive_")
|
||||
emit(repo, "HEAD", led, 30.0, {}, only=None)
|
||||
rows, _, err = read_ledger(led)
|
||||
by = {r["path"]: r for r in rows}
|
||||
alive = os.path.isdir(survive)
|
||||
shutil.rmtree(survive, ignore_errors=True)
|
||||
return (not err and by[V]["verdict"] == PASS and alive,
|
||||
"a verifier that deletes its TMPDIR leaves the caller's alone")
|
||||
|
||||
# the cell fixture needs a caller-level temp dir to watch
|
||||
for name, fn in (("c1 verdict/rc pairing", c1),
|
||||
("c2 population from the rev", c2),
|
||||
("c3 blob drift", c3),
|
||||
("c4 ungraded file", c4),
|
||||
("c5 stale row", c5),
|
||||
("c6 host-pin AST", c6),
|
||||
("c7 named-vs-run", c7),
|
||||
("c8 emit records", c8),
|
||||
("c9 causeless red", c9),
|
||||
("c10 strict", c10),
|
||||
("c11 uncommitted worktree", c11),
|
||||
("c12 note is the verdict line", c12),
|
||||
("c13 sandboxed TMPDIR", c13)):
|
||||
cell(name, fn)
|
||||
|
||||
bad = [c for c in cells if not c[1]]
|
||||
for name, ok, detail in cells:
|
||||
print(" [%s] %s -- %s" % ("ok " if ok else "FAIL", name, detail))
|
||||
print("verifier_gate selftest: %d cells, %d passed, %d failed"
|
||||
% (len(cells), len(cells) - len(bad), len(bad)))
|
||||
return 0 if not bad else 1
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(add_help=True)
|
||||
ap.add_argument("--repo", default=str(ROOT))
|
||||
ap.add_argument("--rev", default="HEAD")
|
||||
ap.add_argument("--ledger", default=str(LEDGER))
|
||||
ap.add_argument("--windows", default=str(WINDOWS))
|
||||
ap.add_argument("--timeout", type=float, default=90.0)
|
||||
ap.add_argument("--check", action="store_true")
|
||||
ap.add_argument("--emit", action="store_true")
|
||||
ap.add_argument("--census", action="store_true")
|
||||
ap.add_argument("--strict", action="store_true")
|
||||
ap.add_argument("--only")
|
||||
ap.add_argument("--merge", action="store_true",
|
||||
help="with --emit: keep the rows this run did not re-measure")
|
||||
ap.add_argument("--selftest", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.selftest:
|
||||
return selftest()
|
||||
repo = Path(args.repo).resolve()
|
||||
if not (repo / ".git").exists():
|
||||
print("verifier_gate: %s is not a git worktree -- could not grade" % repo)
|
||||
return 2
|
||||
ledger = Path(args.ledger)
|
||||
if not ledger.is_absolute():
|
||||
ledger = repo / ledger
|
||||
if args.emit:
|
||||
try:
|
||||
win = read_windows(args.windows)
|
||||
except SystemExit as exc:
|
||||
print("verifier_gate: %s" % exc)
|
||||
return 2
|
||||
return emit(repo, args.rev, ledger, args.timeout, win, only=args.only,
|
||||
merge=args.merge)
|
||||
if args.census:
|
||||
return census_report(repo, args.rev, ledger)
|
||||
return check(repo, args.rev, ledger, args.strict)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user