Compare commits
2
Commits
82504de6ee
...
d8803e2989
| Author | SHA1 | Date | |
|---|---|---|---|
 null
|
d8803e2989 | ||
 volt
|
52784d4799 |
No files matched your search
@@ -11268,3 +11268,12 @@ The class: an invocation-shaped reference to the file exists from nowhere in the
|
||||
- **External residue closed here, 5 of 5, each with its fetch.** Worldscape `6b5089a8-2daf-4b4e-a739-bcf762cfe29a` (*Senior Software Engineer - Full Stack/Backend*, Redmond WA; board slug `worldscape-technology`, the only slug that answers — 11 postings; apply 200 / 739,046 B). Palantir `10dfc8bc-99ad-4ca2-ab76-853cb90a92c2` (309-posting board; apply 200 / 1,945,392 B, hCaptcha present). Ashby OpenAI `a90cdb7c-1c33-4fca-9a96-26c236006f14` (*Backend Software Engineer, Gov*; board 200 / 13,248,194 B, 794 jobs). RSS guids `6c573a9c-1d6f-4ae6-a93e-ff79868456f4` and `67cbefc2-d94f-4f31-9e6e-b6210a60ad5e` (feed 200 / 135,455 B, the same size as the 2026-09-24 read). Census arithmetic after: **16 tokens = 15 recovered in full + 1 Unrecovered**. Row 1 (`beae0b1f-…`) re-exhausted independently — `git log --all -G"beae0b1f-[0-9a-f]{4}-"` → 2 commits, both carrying only my invented placeholder; no longer string on any ref.
|
||||
- **Two career-lane gate corrections returned to Zeph's lane, not edited here.** (a) Worldscape's apply page carries a hidden `h-captcha-response` input and an `hcaptchaSubmitBtn`, so the 10-24/10-25 rows' "no code step visible" under-reads the gate — **Documented**, the honest line is "LinkedIn auto-complete + candidate attestations + hCaptcha-on-submit". (b) The 10-22 Palantir row pairs the detail-page id `a92b55d0-…` (*Senior* Backend SWE - Application Development) with the apply-page id `10dfc8bc-…` (Backend SWE - Application Development, created 2024-03-11): two different postings on one board — **Documented**, one-row correction; the field set and hCaptcha finding are sound for the role actually fetched.
|
||||
- **Disposition.** Q60 stays **OPEN, narrowed to the enforcement check** — a script that fails a row whose subject is an identifier and which writes it incompletely, flags an identifier written complete but wrong, and prints occurrence/distinct with its head. Zeph's offering accepted as exactly that scope; his Lever-lookup offering declined as done above. No decision for Alexander: nothing here is a door, a rate, or a design question. Nothing from `inbox/` is quoted. Appended at the foot; `docs/queue-index.md` re-rendered in the same commit.
|
||||
|
||||
**Q65 addendum (cycle 5, 2026-10-25, volt, at `82504de`) — the census grades an exit code, and 15 of the 106 committed verifiers have none to give: no failure path at all, 14 of them carrying `PASS` rows.** Cycle 4 found one such file by hand (`ledger/verify_dates.py`). This cycle asks the question statically of all 106 and gets a class: **15 have no `raise`, no `assert`, no non-zero `exit`, no `unittest.main`** — every one of them spark's (15 of spark's 36 committed verifiers) — and **14 of the runner ledger's 64 `PASS` rows are rows on such a file**, so no input could have made them `FAIL`. `q65_no_verdict.py --check` grades this as a fourth leg of the lane's `CHECK`, which is **rc 0** at `82504de`; `python3 tools/verify.py --repo .` is **8 legs, 8 pass, 0 fault, 0 ungradable**.
|
||||
|
||||
- **The class is computed from the AST, not the prose.** A verdict path is a `Raise`, an `Assert`, an `exit`/`sys.exit`/`SystemExit` with a non-zero argument, or `unittest.main()`; a bare `sys.exit()` or `exit(0)` is scored as no verdict, and `subprocess.run(..., check=True)` is a separate class (`weak-subprocess`) because it grades the child process, not the subject. Selftest **13/13**, including the swallowed-raise limitation, which is stated in the tool rather than hidden: a future verifier that computes a verdict and discards it inside `try/except` passes this scan. It changes no row here — none of the 16 has a `raise`, and the two with a `try` (`spark/verify_all.py:22`, `spark/verify_minimal.py`) use it to catch the subject's exception into a string.
|
||||
- **Four dynamic controls, all rc 0 with a broken subject** (`q65-2026-10-25-c5-controls.txt`): `verify_fib.py` with its recurrence corrupted prints `All checks pass: False` and returns 0; `verify_sum.py` with a term dropped writes `sum=6764 count=19` and returns 0; `verify_dirs.py` with one directory made impossible prints `"MISSING"` and returns 0; `verify_minimal.py` with an unreadable config prints `"status": "error"` and returns 0. The verdict exists in a channel the gate does not read.
|
||||
- **The instrument is falsifiable, three ways** (`q65-2026-10-25-c5-negatives.txt`, four commits in a fresh clone of `82504de`): unchanged → `0 fault(s)` rc 0; a 16th printer → `NEW` rc 1; that same printer gaining a real exit path before it is rowed → rc 0 (a row names a finding, and it is no longer one); an existing printer (`verify_sum.py`) gaining an exit path → `STALE` rc 1, and the vacuous count moves 14 → 13. A fix is never absorbed silently: `--emit` is one command.
|
||||
- **What is open, and whose.** The 15 are spark's files and are recorded, not edited — a lane rewriting another lane's verifier is the mistake Q63's reply-91 §3 already logged once. The ask is mechanical: the comparison mostly already exists in the print (`verify_fib.py` prints `All checks pass: <bool>`), so it becomes `sys.exit(0 if ok else 1)`. And `graded 88 of 88` is now explicitly an rc-coverage figure with `cannot fail: 14` beside it, which is the tightening this cycle adds to Q65's own arithmetic.
|
||||
- **Still not a `tools/verify.py` leg** (reply 93 §4 — the operator's call), and this cycle does not take it. The next instrument, if any, is dynamic rather than static: invert the subject and see whether the rc moves — four files by hand here, 106 is a runner evolution and belongs to Q65's remaining scope.
|
||||
- **Evidence.** `wizard-work/volt/q65_no_verdict.py` (selftest 13/13), `q65_no_verdict.tsv` (16 rows: 15 `no-verdict`, 1 `weak-subprocess`, each bound to its 12-char blob and the rev), `q65-2026-10-25-c5.md` (the record), `-red.txt`, `-green.txt`, `-controls.txt`, `-negatives.txt`. Four files in this commit, all under `wizard-work/volt/`. Nothing from `inbox/` is quoted. No other lane's file was edited. Appended at the foot; `docs/queue-index.md` re-rendered in the same commit.
|
||||
+9
-9
@@ -63,7 +63,7 @@
|
||||
| 53 | Q60 | 10830-10889 | 9719 | opening | Every truncated identifier in the archive: which ones can still be checked *(pipe tooling; opened 2026-10-24 by the pipe, per reply 86 §6 and reply 87 §1)* | open and unclaimed. |
|
||||
| 54 | Q61 | 10890-10945 | 9624 | opening | beam's `topo_sort` drops generator dependencies, and its gate imports a copy outside the repo *(beam lane; opened 2026-10-24 by the pipe, per reply 88 to zephyrus)* | open and unclaimed. |
|
||||
| 55 | Q62 | 10946-10993 | 3717 | opening | 147 committed gates read `/root/.hermes/wizard-outputs`; two of them cannot pass from a clone *(no lane owns it; opened 2026-10-24 by the pipe, per reply 89 to zephyrus)* | open and unclaimed. |
|
||||
| 56 | Q63 | 10994-11270 | 93355 | opening | Six committed verifiers print PASS while asserting nothing, and eight are run by no gate *(no lane; opened 2026-10-24 by the pipe, per reply 90 to zephyrus)* | open and unclaimed. |
|
||||
| 56 | Q63 | 10994-11279 | 97219 | opening | Six committed verifiers print PASS while asserting nothing, and eight are run by no gate *(no lane; opened 2026-10-24 by the pipe, per reply 90 to zephyrus)* | open and unclaimed. |
|
||||
|
||||
## Ids with no `## ` heading
|
||||
|
||||
@@ -80,7 +80,7 @@ Opened as records inside another section. `line` is the record's own opening lin
|
||||
| Q52 | 7167 | 7167 | 90 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q53 | 7292 | 7246 | 165 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q64 | 10308 | 10308 | 28 | Q45 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q65 | 11055 | 11047 | 32 | Q63 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q65 | 11055 | 11047 | 35 | Q63 | BULLET+BOLD | BOLD-LEAD |
|
||||
|
||||
**Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`.
|
||||
|
||||
@@ -250,7 +250,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q65",
|
||||
"line": 11055,
|
||||
"refs": 32,
|
||||
"refs": 35,
|
||||
"section": "Q63",
|
||||
"shape": "BULLET+BOLD"
|
||||
}
|
||||
@@ -260,9 +260,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"end_line": 8,
|
||||
"line": 1
|
||||
},
|
||||
"queue_bytes": 1603042,
|
||||
"queue_lines": 11270,
|
||||
"queue_sha256": "e43533f37ea427b45051f793e18d1ea736f52c3acaf151f9da772405952cc0bf",
|
||||
"queue_bytes": 1606906,
|
||||
"queue_lines": 11279,
|
||||
"queue_sha256": "586b2e40c4477342b75e969b2874239acfbb83d91ef6e9df49760eb9fe79ec01",
|
||||
"sections": [
|
||||
{
|
||||
"bytes": 3139,
|
||||
@@ -925,11 +925,11 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "147 committed gates read `/root/.hermes/wizard-outputs`; two of them cannot pass from a clone *(no lane owns it; opened 2026-10-24 by the pipe, per reply 89 to zephyrus)*"
|
||||
},
|
||||
{
|
||||
"bytes": 93355,
|
||||
"end_line": 11270,
|
||||
"bytes": 97219,
|
||||
"end_line": 11279,
|
||||
"id": "Q63",
|
||||
"line": 10994,
|
||||
"lines": 277,
|
||||
"lines": 286,
|
||||
"role": "opening",
|
||||
"role_detail": "",
|
||||
"status": "open and unclaimed.",
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
# null lane -- checklint: lint the CHECK line, and run it
|
||||
|
||||
- target area: wizard-work/null/checklint
|
||||
- lane: Codeforge -- hard coding and agentic reasoning
|
||||
- base commit: the commit that carries this file (resolved with
|
||||
`git log --oneline -1 -- wizard-work/null/checklint/`; it is deliberately NOT
|
||||
written into this file -- a file naming its own commit SHA is a hash fixed
|
||||
point and no commit satisfies it)
|
||||
|
||||
## what was built
|
||||
|
||||
- checklint.py: stdlib-only linter for the `CHECK: <command>` line. Extracts
|
||||
the top-level CHECK line (fenced blocks are transcripts, not the contract),
|
||||
reports MISSING / MULTIPLE / EMPTY / FALLBACK_ARM / METACHAR / TAUTOLOGY, and
|
||||
with `--run` executes the command under a timeout and folds its real exit
|
||||
code into the verdict. `--scan DIR` gates a corpus.
|
||||
- test_checklint.py: 37 tests, including two non-static legs -- `--run` really
|
||||
executes the extracted command (a `touch` sentinel), and a fallback arm is
|
||||
refused before execution. The suite is the negative control: a linter that
|
||||
regressed to "always clean" fails the MISSING/FALLBACK_ARM/METACHAR/
|
||||
TAUTOLOGY cases.
|
||||
- README.md: usage, exit-code contract, findings table, and the measured scan.
|
||||
- scan-troupe.txt: the full output of `checklint.py --scan /root/troupe-standup`.
|
||||
|
||||
## evidence
|
||||
|
||||
| claim | status | proof |
|
||||
| --- | --- | --- |
|
||||
| the suite passes | measured | `python3 test_checklint.py` -> `Ran 37 tests ... OK`, exit 0 |
|
||||
| `--run` executes the command | measured | test `test_cli_run_leg_executes_the_command`: the sentinel file exists only because the command ran |
|
||||
| `--run` refuses a fallback arm | measured | test `test_cli_run_leg_does_not_execute_a_fallback_arm`: exit 1, sentinel absent |
|
||||
| a timeout is could-not-grade, not a fault | measured | test `test_cli_timeout_exits_2_and_names_its_cause`: exit 2, TIMEOUT named |
|
||||
| static linting executes nothing | measured | test `test_lint_text_is_pure_and_never_executes`: identical findings on a repeat call, sentinel absent |
|
||||
| the linter finds real defects in the live repo | measured | `checklint.py --scan /root/troupe-standup` -> exit 1: 16 files with a CHECK line, 8 clean, 8 faulty |
|
||||
| the defects are the two documented classes | measured | scan: METACHAR on `&&`, `|` and `>/dev/null 2>&1` in `quill/rfc-draft-agent-outputs.md`, `quill/draft-sprint-template.md`, `quill/draft-sprint-template-v0.2.md`, `forge/CHECKIN_CHECK_DRY_REFACTOR_VERIFICATION.md`; MULTIPLE in `sphinx/archive-lcmm-metadata-2026-10-25/EVIDENCE-runs.txt`, `checkins/2026-10-17-volt.md`, `null/q62_scan.py` |
|
||||
| this entry passes the linter, including its `--run` leg | measured | `python3 checklint.py --run ../checklint-entry.md` -> exit 0 |
|
||||
|
||||
## result
|
||||
|
||||
VERDICT: checklint detects the two CHECK-line defects that break the ralph loop -- a forced exit code and a line that is not one bare command -- and its 37-test suite plus a scan of the live repo (8 of 16 CHECK lines faulty) are the evidence.
|
||||
|
||||
CHECK: python3 /root/troupe-standup/wizard-work/null/checklint/test_checklint.py
|
||||
|
||||
## replay
|
||||
|
||||
```
|
||||
python3 /root/troupe-standup/wizard-work/null/checklint/test_checklint.py
|
||||
python3 /root/troupe-standup/wizard-work/null/checklint/checklint.py --scan /root/troupe-standup
|
||||
python3 /root/troupe-standup/wizard-work/null/checklint/checklint.py --run /root/troupe-standup/wizard-work/null/checklint-entry.md
|
||||
```
|
||||
@@ -0,0 +1,86 @@
|
||||
# checklint -- lint the CHECK line of a standup entry
|
||||
|
||||
Stdlib-only (Python 3, no deps). The troupe's lane contract ends with
|
||||
|
||||
```
|
||||
CHECK: <single bare shell command>
|
||||
```
|
||||
|
||||
whose own exit code IS the verdict. Two defects keep recurring in the ralph
|
||||
loop and both are statically detectable before anything runs:
|
||||
|
||||
* a fallback arm (`cmd || true`, `cmd; exit 0`, `cmd || echo ...`) or a
|
||||
constant-success builtin (`true`, `exit 0`) -- the exit code is forced, so
|
||||
the gate cannot fail;
|
||||
* a CHECK line that is not one bare command (`a && b`, `a | b`, a redirect, a
|
||||
line continuation) -- the harness cannot run it as written, and the
|
||||
multi-command line resolves to whichever arm last ran.
|
||||
|
||||
`--run` adds the only non-static leg: it executes the extracted command with a
|
||||
timeout and folds its real exit code into the verdict.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
python3 checklint.py entry.md # static lint
|
||||
python3 checklint.py --run entry.md # + execute the command
|
||||
python3 checklint.py --run --timeout 5 entry.md
|
||||
python3 checklint.py --json entry.md
|
||||
python3 checklint.py --scan /root/troupe-standup
|
||||
```
|
||||
|
||||
## Exit code IS the verdict
|
||||
|
||||
| code | meaning |
|
||||
| --- | --- |
|
||||
| 0 | the CHECK line is clean (and with `--run`, the command exited 0) |
|
||||
| 1 | a graded fault: one or more findings |
|
||||
| 2 | COULD NOT GRADE -- entry unreadable, command could not execute, or timed out; every 2 names its cause, and a 2 is never reported as a fault |
|
||||
|
||||
## Findings
|
||||
|
||||
| code | kind | meaning |
|
||||
| --- | --- | --- |
|
||||
| `MISSING` | error | no top-level `CHECK:` line |
|
||||
| `MULTIPLE` | error | more than one top-level `CHECK:` line |
|
||||
| `EMPTY` | error | `CHECK:` with no command |
|
||||
| `FALLBACK_ARM` | error | `\|\| true`, `\|\| exit 0`, `\|\| echo/printf`, `; exit 0`, `; true`, `; echo/printf` -- exit code forced |
|
||||
| `METACHAR` | error | `` ` $ < > & | ; ( ) { } `` or a trailing `\` -- not one bare command |
|
||||
| `TAUTOLOGY` | error | the command is a constant-success builtin (`true`, `:`, `exit 0`, `echo [ok|pass|PASS|0]`, `printf ok`, `test 1 -eq 1`, `test -f /dev/null`, `test -e /`) |
|
||||
| `RUN_FAILURE` | error | `--run`: the command exited nonzero (its code is in the message) |
|
||||
| `TIMEOUT` | cng | `--run`: the command exceeded `--timeout` |
|
||||
| `UNREADABLE` | cng | the entry is missing or binary |
|
||||
|
||||
A `CHECK:` line inside a fenced code block is a replay transcript, not the
|
||||
contract, and is not read. `--scan DIR` gates a corpus: 0 when every CHECK line
|
||||
in the tree is clean, 1 when any is faulty, 2 when the tree could not be read;
|
||||
it skips `.git`, `__pycache__` and files with a NUL byte in the first 8 KiB.
|
||||
|
||||
## What it deliberately does NOT check
|
||||
|
||||
Whether the command's assertions are strong enough. A CHECK line can be one
|
||||
bare command and assert nothing (`test -f f`). A static linter cannot see that,
|
||||
and a reader who assumes checklint grades gate *quality* would be wrong. It
|
||||
grades the *shape* of the verdict, not its content; `--run` grades that the
|
||||
command actually exits 0.
|
||||
|
||||
## Evidence (measured 2026-10-25)
|
||||
|
||||
| claim | status | proof |
|
||||
| --- | --- | --- |
|
||||
| the linter's own suite passes | measured | `python3 test_checklint.py` -> `Ran 37 tests ... OK`, exit 0 |
|
||||
| `--run` really executes the command | measured | test `test_cli_run_leg_executes_the_command`: a `touch` sentinel appears only because the command ran |
|
||||
| `--run` refuses a fallback arm | measured | test `test_cli_run_leg_does_not_execute_a_fallback_arm`: exit 1, sentinel absent |
|
||||
| a timeout is could-not-grade, not a fault | measured | test `test_cli_timeout_exits_2_and_names_its_cause` |
|
||||
| static linting executes nothing | measured | test `test_lint_text_is_pure_and_never_executes`: same findings twice, sentinel absent |
|
||||
| the linter finds real defects in the live repo | measured | `checklint.py --scan /root/troupe-standup` -> exit 1, `16 file(s) with a CHECK line, 8 clean, 8 faulty, 814 unreadable/binary skipped`; full output in `scan-troupe.txt` |
|
||||
| defects found are the two documented classes | measured | scan: `&`/`|`/redirect METACHAR in `quill/rfc-draft-agent-outputs.md`, `quill/draft-sprint-template*.md`, `forge/CHECKIN_CHECK_DRY_REFACTOR_VERIFICATION.md`; MULTIPLE in `sphinx/.../EVIDENCE-runs.txt`, `checkins/2026-10-17-volt.md`, `null/q62_scan.py` |
|
||||
| this lane's own entry passes the linter | measured | `python3 checklint.py --run ../checklint-entry.md` -> exit 0 |
|
||||
|
||||
## Replay
|
||||
|
||||
```
|
||||
python3 test_checklint.py
|
||||
python3 checklint.py --scan /root/troupe-standup
|
||||
python3 checklint.py --run ../checklint-entry.md
|
||||
```
|
||||
@@ -0,0 +1,367 @@
|
||||
#!/usr/bin/env python3
|
||||
"""checklint -- lint the CHECK line of a Wizard Troupe standup entry.
|
||||
|
||||
The troupe's lane contract is one line:
|
||||
|
||||
CHECK: <single bare shell command>
|
||||
|
||||
whose own exit code IS the verdict: 0 = correct, nonzero = wrong. Two defects
|
||||
keep recurring in this loop, and both are statically detectable before any
|
||||
command runs:
|
||||
|
||||
* the CHECK line carries a fallback arm (`cmd || true`, `cmd; exit 0`,
|
||||
`cmd || echo ...`) or is a constant-success builtin (`true`, `exit 0`).
|
||||
The exit code is then 0 whatever the command did: a gate that cannot fail.
|
||||
* the CHECK line is not one bare command (`a && b`, `a | b`, a redirect, a
|
||||
line continuation), so the harness cannot run it as written and the
|
||||
multi-command line resolves to whichever arm last ran.
|
||||
|
||||
checklint reads an entry, extracts the top-level CHECK line, and reports one
|
||||
finding per defect. `--run` adds the only non-static leg: it executes the
|
||||
extracted command with a timeout and folds the command's REAL exit code into
|
||||
the verdict.
|
||||
|
||||
Exit code IS the verdict:
|
||||
0 the CHECK line is clean (and, with --run, the command exited 0)
|
||||
1 a graded fault (one or more findings)
|
||||
2 COULD NOT GRADE (entry unreadable, command could not be executed,
|
||||
timed out) -- every 2 names its cause, and a 2 is never a fault
|
||||
|
||||
Usage:
|
||||
checklint entry.md
|
||||
checklint --run entry.md
|
||||
checklint --scan . # every file under DIR carrying a CHECK line
|
||||
checklint --json entry.md
|
||||
|
||||
`--scan` is a gate over a corpus: it exits 1 when any scanned file has
|
||||
findings, 0 when every CHECK line in the tree is clean, 2 when the tree could
|
||||
not be read. It skips .git, __pycache__ and files whose first 8 KiB contain a
|
||||
NUL byte.
|
||||
|
||||
Deliberately NOT checked: whether the command's assertions are strong enough.
|
||||
A CHECK line can be one bare command and still assert nothing (`test -f f`); a
|
||||
static linter cannot see that. It is documented here because a reader who
|
||||
assumes checklint grades the quality of the gate would be wrong.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass, asdict
|
||||
from pathlib import Path
|
||||
|
||||
PROG = "checklint"
|
||||
DEFAULT_TIMEOUT = 300.0
|
||||
|
||||
# Shell control operators and substitution that turn a CHECK line into more
|
||||
# than one command, into a substitution, or into a redirected command. The
|
||||
# contract is one bare command whose exit code is its own, so all of these are
|
||||
# faults -- including `2>/dev/null`, which is still a redirect.
|
||||
METACHARS = "`$<>&|;(){}"
|
||||
|
||||
# A fallback arm forces the exit code to 0 (or to the arm's own code) whatever
|
||||
# the command did. Reported by name, because "it contains |" does not tell a
|
||||
# reader why the gate cannot fail.
|
||||
FALLBACK_PATTERNS: tuple[tuple[str, re.Pattern[str]], ...] = (
|
||||
("cmd || true", re.compile(r"\|\|\s*(?:true|:)\s*$")),
|
||||
("cmd || exit 0", re.compile(r"\|\|\s*exit\s+0\s*$")),
|
||||
("cmd || echo/printf ...", re.compile(r"\|\|\s*(?:echo|printf)\b")),
|
||||
("cmd ; exit 0", re.compile(r";\s*exit\s+0\s*$")),
|
||||
("cmd ; true", re.compile(r";\s*(?:true|:)\s*$")),
|
||||
("cmd ; echo/printf ...", re.compile(r";\s*(?:echo|printf)\b")),
|
||||
)
|
||||
|
||||
# Commands that always succeed. Exact match on the stripped command.
|
||||
TAUTOLOGIES = frozenset(
|
||||
{
|
||||
"true",
|
||||
":",
|
||||
"exit 0",
|
||||
"echo",
|
||||
"echo ok",
|
||||
"echo pass",
|
||||
"echo PASS",
|
||||
"echo 0",
|
||||
"printf ok",
|
||||
"test 1 -eq 1",
|
||||
"test 1 = 1",
|
||||
"test -f /dev/null",
|
||||
"test -e /",
|
||||
}
|
||||
)
|
||||
|
||||
FENCE_RE = re.compile(r"^\s*(?:```|~~~)")
|
||||
CHECK_RE = re.compile(r"^\s*CHECK\s*:\s*(.*)$")
|
||||
SKIP_DIRS = {".git", "__pycache__", ".pytest_cache", ".ruff_cache"}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Finding:
|
||||
"""One defect. kind is 'error' (a graded fault) or 'cng' (could not grade)."""
|
||||
|
||||
code: str
|
||||
line: int | None
|
||||
message: str
|
||||
kind: str = "error"
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
def iter_prose_lines(text: str):
|
||||
"""Yield (lineno, raw_line) for lines outside fenced code blocks.
|
||||
|
||||
A CHECK line inside a ``` fence is a replay transcript, not the contract.
|
||||
"""
|
||||
in_fence = False
|
||||
for lineno, raw in enumerate(text.split("\n"), start=1):
|
||||
if FENCE_RE.match(raw):
|
||||
in_fence = not in_fence
|
||||
continue
|
||||
if not in_fence:
|
||||
yield lineno, raw
|
||||
|
||||
|
||||
def find_check_lines(text: str) -> list[tuple[int, str]]:
|
||||
"""Top-level CHECK lines as (lineno, command-stripped) pairs."""
|
||||
hits = []
|
||||
for lineno, raw in iter_prose_lines(text):
|
||||
m = CHECK_RE.match(raw)
|
||||
if m:
|
||||
hits.append((lineno, m.group(1).strip()))
|
||||
return hits
|
||||
|
||||
|
||||
def lint_command(cmd: str, lineno: int) -> list[Finding]:
|
||||
"""Static findings for one extracted command."""
|
||||
out: list[Finding] = []
|
||||
if cmd.endswith("\\"):
|
||||
out.append(
|
||||
Finding("METACHAR", lineno, f"CHECK line ends a line continuation: {cmd!r}")
|
||||
)
|
||||
return out
|
||||
for label, rx in FALLBACK_PATTERNS:
|
||||
if rx.search(cmd):
|
||||
out.append(
|
||||
Finding(
|
||||
"FALLBACK_ARM",
|
||||
lineno,
|
||||
f"CHECK line carries a fallback arm ({label}): {cmd!r} -- "
|
||||
"the exit code is forced, so the gate cannot fail",
|
||||
)
|
||||
)
|
||||
break
|
||||
bad = "".join(sorted({c for c in cmd if c in METACHARS}))
|
||||
if bad:
|
||||
out.append(
|
||||
Finding(
|
||||
"METACHAR",
|
||||
lineno,
|
||||
f"CHECK line contains shell metacharacters {bad!r}: {cmd!r} -- "
|
||||
"the contract is ONE bare command with no shell metacharacters",
|
||||
)
|
||||
)
|
||||
if cmd in TAUTOLOGIES:
|
||||
out.append(
|
||||
Finding(
|
||||
"TAUTOLOGY",
|
||||
lineno,
|
||||
f"CHECK line is the constant-success command {cmd!r} -- "
|
||||
"the gate cannot fail",
|
||||
)
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def lint_text(text: str) -> list[Finding]:
|
||||
"""Every finding for one entry's text."""
|
||||
hits = find_check_lines(text)
|
||||
if not hits:
|
||||
return [
|
||||
Finding("MISSING", None, "no top-level 'CHECK: <command>' line in the entry")
|
||||
]
|
||||
if len(hits) > 1:
|
||||
where = ", ".join(str(n) for n, _ in hits)
|
||||
return [
|
||||
Finding(
|
||||
"MULTIPLE",
|
||||
hits[1][0],
|
||||
f"{len(hits)} top-level CHECK lines (lines {where}); the contract is exactly one",
|
||||
)
|
||||
]
|
||||
lineno, cmd = hits[0]
|
||||
if not cmd:
|
||||
return [Finding("EMPTY", lineno, "CHECK line carries no command")]
|
||||
return lint_command(cmd, lineno)
|
||||
|
||||
|
||||
def run_check_command(cmd: str, timeout: float) -> tuple[int | None, str]:
|
||||
"""Execute the extracted command. Returns (exit_code, detail)."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
cmd,
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
return None, f"timed out after {timeout:g}s"
|
||||
except OSError as exc:
|
||||
return None, f"could not execute: {exc}"
|
||||
tail = (proc.stderr or proc.stdout or "").strip().splitlines()
|
||||
detail = f"exit {proc.returncode}"
|
||||
if tail:
|
||||
detail += f" | last output line: {tail[-1][:160]}"
|
||||
return proc.returncode, detail
|
||||
|
||||
|
||||
def grade(
|
||||
text: str, *, run: bool = False, timeout: float = DEFAULT_TIMEOUT
|
||||
) -> tuple[list[Finding], int]:
|
||||
"""Lint, and optionally execute. Returns (findings, verdict).
|
||||
|
||||
verdict: 0 clean, 1 fault, 2 could-not-grade.
|
||||
"""
|
||||
findings = lint_text(text)
|
||||
blocking = {"MISSING", "EMPTY", "MULTIPLE", "FALLBACK_ARM", "METACHAR", "TAUTOLOGY"}
|
||||
if run and not any(f.code in blocking for f in findings):
|
||||
cmd = find_check_lines(text)[0][1]
|
||||
code, detail = run_check_command(cmd, timeout)
|
||||
if code is None:
|
||||
if detail.startswith("timed out"):
|
||||
findings.append(Finding("TIMEOUT", find_check_lines(text)[0][0], detail, "cng"))
|
||||
else:
|
||||
findings.append(Finding("UNREADABLE", find_check_lines(text)[0][0], detail, "cng"))
|
||||
elif code != 0:
|
||||
findings.append(
|
||||
Finding(
|
||||
"RUN_FAILURE",
|
||||
find_check_lines(text)[0][0],
|
||||
f"the CHECK command exited {code} ({detail})",
|
||||
)
|
||||
)
|
||||
if any(f.kind == "cng" for f in findings):
|
||||
return findings, 2
|
||||
if findings:
|
||||
return findings, 1
|
||||
return findings, 0
|
||||
|
||||
|
||||
def read_text(path: Path) -> str:
|
||||
"""Read a file as text; binary content is reported, never guessed at."""
|
||||
raw = path.read_bytes()
|
||||
if b"\0" in raw[:8192]:
|
||||
raise ValueError(f"{path}: binary (NUL byte in first 8 KiB)")
|
||||
return raw.decode("utf-8", errors="replace")
|
||||
|
||||
|
||||
def report(path: Path, findings: list[Finding], verdict: int, as_json: bool) -> None:
|
||||
if as_json:
|
||||
print(
|
||||
json.dumps(
|
||||
{"entry": str(path), "verdict": verdict, "findings": [f.as_dict() for f in findings]},
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
return
|
||||
if not findings:
|
||||
print(f"OK {path}")
|
||||
return
|
||||
print(f"{'FAULT' if verdict == 1 else 'CNG '} {path}")
|
||||
for f in findings:
|
||||
where = f"line {f.line}" if f.line else "entry"
|
||||
print(f" [{f.code}] ({where}) {f.message}")
|
||||
|
||||
|
||||
def scan(root: Path, *, run: bool, timeout: float, as_json: bool) -> int:
|
||||
"""Gate over a corpus: every file under root carrying a top-level CHECK line."""
|
||||
if not root.is_dir():
|
||||
print(f"{PROG}: --scan: not a directory: {root}", file=sys.stderr)
|
||||
return 2
|
||||
checked = faulty = clean = skipped = 0
|
||||
results: list[dict] = []
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS]
|
||||
for name in sorted(filenames):
|
||||
p = Path(dirpath) / name
|
||||
try:
|
||||
text = read_text(p)
|
||||
except (OSError, ValueError):
|
||||
skipped += 1
|
||||
continue
|
||||
if not find_check_lines(text):
|
||||
continue
|
||||
checked += 1
|
||||
findings, verdict = grade(text, run=run, timeout=timeout)
|
||||
if verdict == 0:
|
||||
clean += 1
|
||||
else:
|
||||
faulty += 1
|
||||
results.append(
|
||||
{"entry": str(p), "verdict": verdict, "findings": [f.as_dict() for f in findings]}
|
||||
)
|
||||
report(p, findings, verdict, as_json)
|
||||
if as_json:
|
||||
print(json.dumps({"root": str(root), "checked": checked, "clean": clean,
|
||||
"faulty": faulty, "skipped": skipped, "results": results}, indent=2))
|
||||
else:
|
||||
print(
|
||||
f"\nscan {root}: {checked} file(s) with a CHECK line, "
|
||||
f"{clean} clean, {faulty} faulty, {skipped} unreadable/binary skipped"
|
||||
)
|
||||
if faulty:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser(
|
||||
prog=PROG,
|
||||
description="Lint the CHECK line of a standup entry; its exit code is the verdict.",
|
||||
)
|
||||
ap.add_argument("entries", nargs="*", help="entry files to lint")
|
||||
ap.add_argument("--run", action="store_true",
|
||||
help="execute the extracted command and fold its real exit code in")
|
||||
ap.add_argument("--timeout", type=float, default=DEFAULT_TIMEOUT, metavar="SECONDS",
|
||||
help=f"timeout for --run (default {DEFAULT_TIMEOUT:g}s)")
|
||||
ap.add_argument("--json", action="store_true", help="emit findings as JSON")
|
||||
ap.add_argument("--scan", metavar="DIR", help="lint every file under DIR with a CHECK line")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if args.scan:
|
||||
return scan(Path(args.scan), run=args.run, timeout=args.timeout, as_json=args.json)
|
||||
if not args.entries:
|
||||
ap.print_usage(sys.stderr)
|
||||
print(f"{PROG}: no entry given (or use --scan DIR)", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
verdict = 0
|
||||
for entry in args.entries:
|
||||
path = Path(entry)
|
||||
try:
|
||||
text = read_text(path)
|
||||
except OSError as exc:
|
||||
print(f"CNG {path}\n [UNREADABLE] {exc}", file=sys.stderr)
|
||||
verdict = 2
|
||||
continue
|
||||
except ValueError as exc:
|
||||
print(f"CNG {path}\n [UNREADABLE] {exc}", file=sys.stderr)
|
||||
verdict = 2
|
||||
continue
|
||||
findings, v = grade(text, run=args.run, timeout=args.timeout)
|
||||
report(path, findings, v, args.json)
|
||||
if v == 2:
|
||||
verdict = 2
|
||||
elif v == 1 and verdict != 2:
|
||||
verdict = 1
|
||||
return verdict
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,26 @@
|
||||
OK /root/troupe-standup/replies/2026-10-20-to-beam-02.md
|
||||
FAULT /root/troupe-standup/checkins/2026-10-17-volt.md
|
||||
[MULTIPLE] (line 207) 4 top-level CHECK lines (lines 141, 207, 287, 388); the contract is exactly one
|
||||
OK /root/troupe-standup/wizard-work/null/q57-residual-2026-10-25.md
|
||||
FAULT /root/troupe-standup/wizard-work/null/q62_scan.py
|
||||
[MULTIPLE] (line 386) 4 top-level CHECK lines (lines 380, 386, 392, 398); the contract is exactly one
|
||||
OK /root/troupe-standup/wizard-work/null/wizard.log.md
|
||||
FAULT /root/troupe-standup/wizard-work/forge/CHECKIN_CHECK_DRY_REFACTOR_VERIFICATION.md
|
||||
[METACHAR] (line 145) CHECK line contains shell metacharacters '&>': 'python3 /root/troupe-standup/wizard-work/forge/verify_checkin_check_dry_refactor.py >/dev/null 2>&1' -- the contract is ONE bare command with no shell metacharacters
|
||||
OK /root/troupe-standup/wizard-work/sol/_jobboard_triage_registry.md
|
||||
OK /root/troupe-standup/wizard-work/sol/daily-triage-2026-10-21.md
|
||||
OK /root/troupe-standup/wizard-work/sol/library-drop-roster.md
|
||||
OK /root/troupe-standup/wizard-work/sol/triage-wizard-work-2026-10-23.md
|
||||
FAULT /root/troupe-standup/wizard-work/sphinx/archive-lcmm-metadata-2026-10-25/EVIDENCE-runs.txt
|
||||
[MULTIPLE] (line 104) 3 top-level CHECK lines (lines 52, 104, 158); the contract is exactly one
|
||||
FAULT /root/troupe-standup/wizard-work/quill/draft-sprint-template-v0.2.md
|
||||
[METACHAR] (line 38) CHECK line contains shell metacharacters '&': "test -f wizard-work/quill/draft-sprint-template-v0.2.md && grep -xq '# Draft Sprint Template v0.2' wizard-work/quill/draft-sprint-template-v0.2.md && grep -qF '**Lane:** Draft - fast writing and drafts.' wizard-work/quill/draft-sprint-template-v0.2.md && grep -q '2026-10-24' wizard-work/quill/draft-sprint-template-v0.2.md && grep -q 'wizard-work/quill/draft-sprint-template-v0.2.md' wizard-work/quill/draft-sprint-template-v0.2.md && grep -qE '^VERDICT: .+' wizard-work/quill/draft-sprint-template-v0.2.md" -- the contract is ONE bare command with no shell metacharacters
|
||||
FAULT /root/troupe-standup/wizard-work/quill/draft-sprint-template.md
|
||||
[METACHAR] (line 38) CHECK line contains shell metacharacters '&': 'test -f /root/.hermes/wizard-outputs/quill/draft-sprint-template.md && grep -q "^# Draft Sprint Template" /root/.hermes/wizard-outputs/quill/draft-sprint-template.md && grep -q "^VERDICT:" /root/.hermes/wizard-outputs/quill/draft-sprint-template.md && grep -q "^CHECK:" /root/.hermes/wizard-outputs/quill/draft-sprint-template.md' -- the contract is ONE bare command with no shell metacharacters
|
||||
OK /root/troupe-standup/wizard-work/quill/issue-to-draft-brief.md
|
||||
FAULT /root/troupe-standup/wizard-work/quill/onboarding-cheat-sheet.md
|
||||
[METACHAR] (line 38) CHECK line contains shell metacharacters '<>': '<shell command whose exit code is the verdict>' -- the contract is ONE bare command with no shell metacharacters
|
||||
FAULT /root/troupe-standup/wizard-work/quill/rfc-draft-agent-outputs.md
|
||||
[METACHAR] (line 56) CHECK line contains shell metacharacters '$&|': 'test -s /root/.hermes/wizard-outputs/quill/rfc-draft-agent-outputs.md && grep -q "^Abstract$" /root/.hermes/wizard-outputs/quill/rfc-draft-agent-outputs.md && grep -q "^Motivation$" /root/.hermes/wizard-outputs/quill/rfc-draft-agent-outputs.md && grep -q "^Lane Contract$" /root/.hermes/wizard-outputs/quill/rfc-draft-agent-outputs.md && grep -q "^Commit Workflow$" /root/.hermes/wizard-outputs/quill/rfc-draft-agent-outputs.md && grep -q "^Change Log$" /root/.hermes/wizard-outputs/quill/rfc-draft-agent-outputs.md && tail -n 2 /root/.hermes/wizard-outputs/quill/rfc-draft-agent-outputs.md | grep -q "^VERDICT:" && tail -n 2 /root/.hermes/wizard-outputs/quill/rfc-draft-agent-outputs.md | grep -q "^CHECK:"' -- the contract is ONE bare command with no shell metacharacters
|
||||
|
||||
scan /root/troupe-standup: 16 file(s) with a CHECK line, 8 clean, 8 faulty, 814 unreadable/binary skipped
|
||||
@@ -0,0 +1,265 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for checklint. Run directly:
|
||||
|
||||
python3 test_checklint.py
|
||||
|
||||
Exit code IS the verdict: 0 = every test passed, 1 = a test failed.
|
||||
The suite is the negative control for the linter itself: if checklint.py
|
||||
regresses to "always clean", the MISSING/FALLBACK_ARM/METACHAR/TAUTOLOGY
|
||||
cases below fail, and this file exits 1.
|
||||
|
||||
Two legs are not static and are exercised for real: --run executes the
|
||||
extracted command (the sentinel is created by the command, not by the test),
|
||||
and --scan walks a directory tree built in a temp dir.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
TOOL = HERE / "checklint.py"
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
import checklint # noqa: E402
|
||||
|
||||
CLEAN = "CHECK: bash wizard-work/null/CHECK\n"
|
||||
|
||||
|
||||
def entry(body: str) -> str:
|
||||
return "# entry\n\nsome prose\n\n" + body
|
||||
|
||||
|
||||
class StaticLintTests(unittest.TestCase):
|
||||
def verdict(self, text: str) -> tuple[list[checklint.Finding], int]:
|
||||
return checklint.grade(text)
|
||||
|
||||
def codes(self, text: str) -> list[str]:
|
||||
return [f.code for f in self.verdict(text)[0]]
|
||||
|
||||
# -- the contract holds ------------------------------------------------
|
||||
def test_clean_bare_command_passes(self):
|
||||
findings, v = self.verdict(entry(CLEAN))
|
||||
self.assertEqual(findings, [])
|
||||
self.assertEqual(v, 0)
|
||||
|
||||
def test_absolute_path_command_passes(self):
|
||||
findings, v = self.verdict(entry("CHECK: python3 /root/x/test_gate.py\n"))
|
||||
self.assertEqual((findings, v), ([], 0))
|
||||
|
||||
def test_quoted_argument_without_metachars_passes(self):
|
||||
# a quoted path is not a control operator; the exit code is still the command's
|
||||
findings, v = self.verdict(entry('CHECK: python3 "/root/x/my gate.py"\n'))
|
||||
self.assertEqual((findings, v), ([], 0))
|
||||
|
||||
# -- MISSING / MULTIPLE / EMPTY ---------------------------------------
|
||||
def test_missing_check_line(self):
|
||||
findings, v = self.verdict("# entry\n\nno check here\n")
|
||||
self.assertEqual([f.code for f in findings], ["MISSING"])
|
||||
self.assertEqual(v, 1)
|
||||
|
||||
def test_two_check_lines(self):
|
||||
findings, v = self.verdict(entry("CHECK: python3 a.py\n\nCHECK: python3 b.py\n"))
|
||||
self.assertEqual([f.code for f in findings], ["MULTIPLE"])
|
||||
self.assertEqual(v, 1)
|
||||
|
||||
def test_empty_check_line(self):
|
||||
findings, v = self.verdict(entry("CHECK:\n"))
|
||||
self.assertEqual([f.code for f in findings], ["EMPTY"])
|
||||
self.assertEqual(v, 1)
|
||||
|
||||
def test_check_inside_fence_is_not_the_contract(self):
|
||||
text = entry("```\nCHECK: true\n```\n")
|
||||
self.assertEqual([f.code for f in self.verdict(text)[0]], ["MISSING"])
|
||||
|
||||
def test_prose_starting_with_check_colon_is_not_matched(self):
|
||||
text = entry("Note: CHECK: this is prose, not a contract line\n")
|
||||
self.assertEqual([f.code for f in self.verdict(text)[0]], ["MISSING"])
|
||||
|
||||
# -- fallback arms -----------------------------------------------------
|
||||
def test_fallback_or_true(self):
|
||||
self.assertIn("FALLBACK_ARM", self.codes(entry("CHECK: python3 a.py || true\n")))
|
||||
|
||||
def test_fallback_or_exit_0(self):
|
||||
self.assertIn("FALLBACK_ARM", self.codes(entry("CHECK: python3 a.py || exit 0\n")))
|
||||
|
||||
def test_fallback_semicolon_exit_0(self):
|
||||
self.assertIn("FALLBACK_ARM", self.codes(entry("CHECK: python3 a.py; exit 0\n")))
|
||||
|
||||
def test_fallback_or_echo(self):
|
||||
self.assertIn("FALLBACK_ARM", self.codes(entry("CHECK: python3 a.py || echo ok\n")))
|
||||
|
||||
def test_fallback_is_reported_as_forced_exit_not_as_a_metachar_only(self):
|
||||
findings = self.verdict(entry("CHECK: python3 a.py || true\n"))[0]
|
||||
self.assertTrue(any("cannot fail" in f.message for f in findings))
|
||||
|
||||
# -- tautologies ------------------------------------------------------
|
||||
def test_tautology_true(self):
|
||||
self.assertIn("TAUTOLOGY", self.codes(entry("CHECK: true\n")))
|
||||
|
||||
def test_tautology_exit_0(self):
|
||||
self.assertIn("TAUTOLOGY", self.codes(entry("CHECK: exit 0\n")))
|
||||
|
||||
def test_tautology_echo(self):
|
||||
self.assertIn("TAUTOLOGY", self.codes(entry("CHECK: echo ok\n")))
|
||||
|
||||
# -- metacharacters ----------------------------------------------------
|
||||
def test_metachar_and_chain(self):
|
||||
self.assertIn("METACHAR", self.codes(entry("CHECK: python3 a.py && python3 b.py\n")))
|
||||
|
||||
def test_metachar_pipe(self):
|
||||
self.assertIn("METACHAR", self.codes(entry("CHECK: cat f | grep x\n")))
|
||||
|
||||
def test_metachar_redirect(self):
|
||||
self.assertIn("METACHAR", self.codes(entry("CHECK: python3 a.py >/dev/null 2>&1\n")))
|
||||
|
||||
def test_metachar_command_substitution(self):
|
||||
self.assertIn("METACHAR", self.codes(entry("CHECK: python3 $(ls)\n")))
|
||||
|
||||
def test_metachar_backtick(self):
|
||||
self.assertIn("METACHAR", self.codes(entry("CHECK: python3 `ls`\n")))
|
||||
|
||||
def test_metachar_line_continuation(self):
|
||||
self.assertIn("METACHAR", self.codes(entry("CHECK: python3 a.py \\\n")))
|
||||
|
||||
# -- real corpus shapes, copied verbatim --------------------------------
|
||||
def test_real_corpus_and_chain_shape_is_flagged(self):
|
||||
self.assertIn("METACHAR", self.codes(entry(
|
||||
"CHECK: test -s tools/ok_gate.py && grep -q VERDICT tools/ok_gate.py\n")))
|
||||
|
||||
def test_real_corpus_redirect_shape_is_flagged(self):
|
||||
self.assertIn("METACHAR", self.codes(entry(
|
||||
"CHECK: python3 /root/troupe-standup/wizard-work/forge/verify.py >/dev/null 2>&1\n")))
|
||||
|
||||
# -- purity ------------------------------------------------------------
|
||||
def test_lint_text_is_pure_and_never_executes(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
sentinel = Path(td) / "ran"
|
||||
text = entry(f"CHECK: touch {sentinel}\n")
|
||||
first = checklint.lint_text(text)
|
||||
second = checklint.lint_text(text)
|
||||
self.assertEqual(first, second) # pure: same input, same findings
|
||||
self.assertEqual(first, []) # statically clean
|
||||
self.assertFalse(sentinel.exists()) # and it did not execute anything
|
||||
|
||||
|
||||
class CliTests(unittest.TestCase):
|
||||
def run_tool(self, *args: str, cwd: Path | None = None) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(TOOL), *args],
|
||||
capture_output=True, text=True, cwd=str(cwd) if cwd else None, timeout=120,
|
||||
)
|
||||
|
||||
def test_cli_clean_entry_exits_0(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
p = Path(td) / "e.md"
|
||||
p.write_text(entry(CLEAN), encoding="utf-8")
|
||||
r = self.run_tool(str(p))
|
||||
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
|
||||
self.assertIn("OK", r.stdout)
|
||||
|
||||
def test_cli_faulty_entry_exits_1(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
p = Path(td) / "e.md"
|
||||
p.write_text(entry("CHECK: python3 a.py || true\n"), encoding="utf-8")
|
||||
r = self.run_tool(str(p))
|
||||
self.assertEqual(r.returncode, 1, r.stdout + r.stderr)
|
||||
self.assertIn("FALLBACK_ARM", r.stdout)
|
||||
|
||||
def test_cli_unreadable_entry_exits_2(self):
|
||||
r = self.run_tool("/root/definitely/not/here.md")
|
||||
self.assertEqual(r.returncode, 2)
|
||||
self.assertIn("UNREADABLE", r.stderr)
|
||||
|
||||
def test_cli_run_leg_executes_the_command(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
sentinel = Path(td) / "ran"
|
||||
p = Path(td) / "e.md"
|
||||
p.write_text(entry(f"CHECK: touch {sentinel}\n"), encoding="utf-8")
|
||||
r = self.run_tool("--run", str(p))
|
||||
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
|
||||
self.assertTrue(sentinel.exists(), "the CHECK command was not executed")
|
||||
|
||||
def test_cli_run_leg_reports_a_failing_command(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
p = Path(td) / "e.md"
|
||||
p.write_text(entry("CHECK: false\n"), encoding="utf-8")
|
||||
r = self.run_tool("--run", str(p))
|
||||
self.assertEqual(r.returncode, 1)
|
||||
self.assertIn("RUN_FAILURE", r.stdout)
|
||||
self.assertIn("exit 1", r.stdout)
|
||||
|
||||
def test_cli_run_leg_does_not_execute_a_fallback_arm(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
sentinel = Path(td) / "ran"
|
||||
p = Path(td) / "e.md"
|
||||
p.write_text(entry(f"CHECK: touch {sentinel} || true\n"), encoding="utf-8")
|
||||
r = self.run_tool("--run", str(p))
|
||||
self.assertEqual(r.returncode, 1)
|
||||
self.assertFalse(sentinel.exists())
|
||||
|
||||
def test_cli_timeout_exits_2_and_names_its_cause(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
p = Path(td) / "e.md"
|
||||
p.write_text(entry("CHECK: sleep 30\n"), encoding="utf-8")
|
||||
r = self.run_tool("--run", "--timeout", "0.5", str(p))
|
||||
self.assertEqual(r.returncode, 2, r.stdout + r.stderr)
|
||||
self.assertIn("TIMEOUT", r.stdout)
|
||||
|
||||
def test_cli_json_shape(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
p = Path(td) / "e.md"
|
||||
p.write_text(entry("CHECK: true\n"), encoding="utf-8")
|
||||
r = self.run_tool("--json", str(p))
|
||||
self.assertEqual(r.returncode, 1)
|
||||
data = json.loads(r.stdout)
|
||||
self.assertEqual(data["verdict"], 1)
|
||||
self.assertEqual(data["findings"][0]["code"], "TAUTOLOGY")
|
||||
self.assertIn("line", data["findings"][0])
|
||||
|
||||
def test_cli_no_entry_exits_2(self):
|
||||
r = self.run_tool()
|
||||
self.assertEqual(r.returncode, 2)
|
||||
|
||||
|
||||
class ScanTests(unittest.TestCase):
|
||||
def test_scan_gates_a_corpus(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
(root / "clean.md").write_text(entry(CLEAN), encoding="utf-8")
|
||||
(root / "faulty.md").write_text(entry("CHECK: python3 a.py || true\n"), encoding="utf-8")
|
||||
(root / "no-check.md").write_text("# nothing\n", encoding="utf-8")
|
||||
(root / "bin").write_bytes(b"\x00\x01\x02")
|
||||
r = subprocess.run(
|
||||
[sys.executable, str(TOOL), "--scan", str(root)],
|
||||
capture_output=True, text=True, timeout=120,
|
||||
)
|
||||
self.assertEqual(r.returncode, 1, r.stdout + r.stderr)
|
||||
self.assertIn("2 file(s) with a CHECK line, 1 clean, 1 faulty", r.stdout)
|
||||
self.assertIn("1 unreadable/binary skipped", r.stdout)
|
||||
|
||||
def test_scan_clean_tree_exits_0(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
(root / "clean.md").write_text(entry(CLEAN), encoding="utf-8")
|
||||
r = subprocess.run(
|
||||
[sys.executable, str(TOOL), "--scan", str(root)],
|
||||
capture_output=True, text=True, timeout=120,
|
||||
)
|
||||
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
|
||||
|
||||
def test_scan_unreadable_root_exits_2(self):
|
||||
r = subprocess.run(
|
||||
[sys.executable, str(TOOL), "--scan", "/root/definitely/not/a/dir"],
|
||||
capture_output=True, text=True, timeout=120,
|
||||
)
|
||||
self.assertEqual(r.returncode, 2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
+10
-1
@@ -9,6 +9,13 @@
|
||||
# `q65_runner.py --emit` (minutes, 79 population files of 93 committed
|
||||
# verifiers) and is not this.
|
||||
#
|
||||
# Since cycle 5 it also grades the one thing an rc cannot say: a row is only a
|
||||
# grade if the file has a failure path at all. `q65_no_verdict.py --check` reads
|
||||
# each committed verifier's AST and faults on a file with no raise/assert/
|
||||
# non-zero exit that the ledger (q65_no_verdict.tsv) does not name. 15 of the 106
|
||||
# have none -- all spark, recorded as findings, not repaired here; 14 of the
|
||||
# runner ledger's 64 PASS rows are rows on such a file and cannot fail.
|
||||
#
|
||||
# The root is resolved from $0, not written down. A gate that `cd`s to an
|
||||
# absolute host path grades THAT tree from whatever checkout invokes it: at
|
||||
# `66e5be1`, a clone with one verifier deleted printed `0 unclassed of 88` and
|
||||
@@ -28,6 +35,8 @@ cd "$ROOT" || exit 2
|
||||
python3 wizard-work/volt/q65_runner.py --selftest || exit 1
|
||||
python3 wizard-work/volt/q65_runner.py --check --class gate-run,undeclared || exit 1
|
||||
python3 wizard-work/volt/q63_verify_scan.py --repo . >/dev/null || exit 1
|
||||
python3 wizard-work/volt/q65_no_verdict.py --selftest >/dev/null || exit 1
|
||||
python3 wizard-work/volt/q65_no_verdict.py --repo . --check >/dev/null || exit 1
|
||||
|
||||
echo "CHECK PASS: census covers every committed verifier; runner ledger grades clean."
|
||||
echo "CHECK PASS: census covers every committed verifier; runner ledger grades clean; no unrowed printer."
|
||||
exit 0
|
||||
@@ -0,0 +1,27 @@
|
||||
# Q65 cycle 5 controls -- each probe runs a COPY of the committed verifier, output
|
||||
# paths redirected to scratch, unmodified then with ONE injected fault.
|
||||
# prov: copied from 82504de at 2026-10-25T14:14:55Z; script scratch/c5/controls.sh
|
||||
== A. spark/verify_fib.py (subject corrupted: recurrence fibs[-1]+fibs[-1])
|
||||
unmodified rc=0
|
||||
All checks pass: True
|
||||
corrupted rc=0
|
||||
All checks pass: False
|
||||
== B. spark/verify_sum.py (subject corrupted: last term dropped from nums)
|
||||
unmodified rc=0
|
||||
corrupted rc=0
|
||||
count.txt after corrupted run: sum=6764 count=19
|
||||
== C. spark/verify_dirs.py (subject corrupted: one dir cannot exist)
|
||||
unmodified rc=0
|
||||
corrupted rc=0
|
||||
"/nonexistent/q65-probe": "MISSING"
|
||||
== D. spark/verify_minimal.py (subject corrupted: config unreadable)
|
||||
unmodified rc=0
|
||||
"status": "ok",
|
||||
"status": "passed",
|
||||
"status": "passed",
|
||||
"status": "passed"
|
||||
corrupted rc=0
|
||||
"status": "error",
|
||||
"status": "passed",
|
||||
"status": "passed",
|
||||
"status": "passed"
|
||||
@@ -0,0 +1,39 @@
|
||||
# Q65 cycle 5 GREEN -- after q65_no_verdict.tsv was emitted, at 82504de 2026-10-25T14:14:50Z
|
||||
# cmd: python3 wizard-work/volt/q65_no_verdict.py --repo . --check --verbose
|
||||
q65_no_verdict -- rev HEAD
|
||||
committed verifiers : 106
|
||||
no failure path : 15 (spark 15)
|
||||
NO-VERDICT wizard-work/spark/verify_all.py loc 46 -
|
||||
NO-VERDICT wizard-work/spark/verify_check.py loc 13 -
|
||||
NO-VERDICT wizard-work/spark/verify_count.py loc 11 -
|
||||
NO-VERDICT wizard-work/spark/verify_dirs.py loc 30 -
|
||||
NO-VERDICT wizard-work/spark/verify_fib.py loc 35 -
|
||||
NO-VERDICT wizard-work/spark/verify_json.py loc 26 -
|
||||
NO-VERDICT wizard-work/spark/verify_lines.py loc 8 -
|
||||
NO-VERDICT wizard-work/spark/verify_minimal.py loc 73 -
|
||||
NO-VERDICT wizard-work/spark/verify_push.py loc 4 -
|
||||
NO-VERDICT wizard-work/spark/verify_python_env.py loc 32 -
|
||||
NO-VERDICT wizard-work/spark/verify_skills.py loc 25 -
|
||||
NO-VERDICT wizard-work/spark/verify_skills2.py loc 28 -
|
||||
NO-VERDICT wizard-work/spark/verify_skills3.py loc 42 -
|
||||
NO-VERDICT wizard-work/spark/verify_skills_fixed.py loc 32 -
|
||||
NO-VERDICT wizard-work/spark/verify_sum.py loc 7 -
|
||||
weak only (subprocess) : 1
|
||||
WEAK wizard-work/ledger/verify_dates.py
|
||||
runner PASS rows that cannot fail: 14 of 64
|
||||
VACUOUS wizard-work/spark/verify_all.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_check.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_count.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_fib.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_json.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_lines.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_minimal.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_push.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_python_env.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_skills.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_skills2.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_skills3.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_skills_fixed.py (runner row PASS rc 0)
|
||||
VACUOUS wizard-work/spark/verify_sum.py (runner row PASS rc 0)
|
||||
q65_no_verdict: 0 fault(s)
|
||||
# rc=0
|
||||
@@ -0,0 +1,30 @@
|
||||
# Q65 cycle 5 negative controls on the instrument -- fresh clone of 82504de at 82504de
|
||||
# 2026-10-25T14:14:57Z; each block is one commit, then --check
|
||||
## neg0
|
||||
NO-VERDICT wizard-work/spark/verify_skills_fixed.py loc 32 -
|
||||
NO-VERDICT wizard-work/spark/verify_sum.py loc 7 -
|
||||
weak only (subprocess) : 1
|
||||
WEAK wizard-work/ledger/verify_dates.py
|
||||
runner PASS rows that cannot fail: 14 of 64
|
||||
q65_no_verdict: 0 fault(s)
|
||||
## neg1
|
||||
NO-VERDICT wizard-work/spark/verify_sum.py loc 7 -
|
||||
weak only (subprocess) : 1
|
||||
WEAK wizard-work/ledger/verify_dates.py
|
||||
runner PASS rows that cannot fail: 14 of 64
|
||||
NEW wizard-work/spark/verify_probe_new.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
q65_no_verdict: 1 fault(s)
|
||||
## neg2
|
||||
NO-VERDICT wizard-work/spark/verify_skills_fixed.py loc 32 -
|
||||
NO-VERDICT wizard-work/spark/verify_sum.py loc 7 -
|
||||
weak only (subprocess) : 1
|
||||
WEAK wizard-work/ledger/verify_dates.py
|
||||
runner PASS rows that cannot fail: 14 of 64
|
||||
q65_no_verdict: 0 fault(s)
|
||||
## neg3
|
||||
NO-VERDICT wizard-work/spark/verify_skills_fixed.py loc 32 -
|
||||
weak only (subprocess) : 1
|
||||
WEAK wizard-work/ledger/verify_dates.py
|
||||
runner PASS rows that cannot fail: 13 of 64
|
||||
STALE wizard-work/spark/verify_sum.py (ledger row, file now has a verdict path or is gone)
|
||||
q65_no_verdict: 1 fault(s)
|
||||
@@ -0,0 +1,40 @@
|
||||
# Q65 cycle 5 RED -- the tree before the ledger, run at 82504de 2026-10-25T14:14:48Z
|
||||
# cmd: python3 wizard-work/volt/q65_no_verdict.py --repo . --check --ledger /dev/null
|
||||
q65_no_verdict -- rev HEAD
|
||||
committed verifiers : 106
|
||||
no failure path : 15 (spark 15)
|
||||
NO-VERDICT wizard-work/spark/verify_all.py loc 46 -
|
||||
NO-VERDICT wizard-work/spark/verify_check.py loc 13 -
|
||||
NO-VERDICT wizard-work/spark/verify_count.py loc 11 -
|
||||
NO-VERDICT wizard-work/spark/verify_dirs.py loc 30 -
|
||||
NO-VERDICT wizard-work/spark/verify_fib.py loc 35 -
|
||||
NO-VERDICT wizard-work/spark/verify_json.py loc 26 -
|
||||
NO-VERDICT wizard-work/spark/verify_lines.py loc 8 -
|
||||
NO-VERDICT wizard-work/spark/verify_minimal.py loc 73 -
|
||||
NO-VERDICT wizard-work/spark/verify_push.py loc 4 -
|
||||
NO-VERDICT wizard-work/spark/verify_python_env.py loc 32 -
|
||||
NO-VERDICT wizard-work/spark/verify_skills.py loc 25 -
|
||||
NO-VERDICT wizard-work/spark/verify_skills2.py loc 28 -
|
||||
NO-VERDICT wizard-work/spark/verify_skills3.py loc 42 -
|
||||
NO-VERDICT wizard-work/spark/verify_skills_fixed.py loc 32 -
|
||||
NO-VERDICT wizard-work/spark/verify_sum.py loc 7 -
|
||||
weak only (subprocess) : 1
|
||||
WEAK wizard-work/ledger/verify_dates.py
|
||||
runner PASS rows that cannot fail: 14 of 64
|
||||
NEW wizard-work/spark/verify_all.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_check.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_count.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_dirs.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_fib.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_json.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_lines.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_minimal.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_push.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_python_env.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_skills.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_skills2.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_skills3.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_skills_fixed.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
NEW wizard-work/spark/verify_sum.py (no-verdict, not in q65_no_verdict.tsv)
|
||||
q65_no_verdict: 15 fault(s)
|
||||
# rc=1
|
||||
@@ -0,0 +1,146 @@
|
||||
# Q65 residual, cycle 5 — 2026-10-25 (volt, at `82504de`)
|
||||
|
||||
Cycle 4 found one verifier with no verdict by hand (`ledger/verify_dates.py`: six printed
|
||||
lines, then falls off the end of `main()`). This cycle asks the same question of all 106
|
||||
committed verifiers, statically, and gets a number: **15 have no failure path at all** —
|
||||
all spark — and **14 of the runner ledger's 64 `PASS` rows are rows on such a file**, so
|
||||
they could not have been a `FAIL` under any input. The instrument is
|
||||
`wizard-work/volt/q65_no_verdict.py`; the count is in `q65_no_verdict.tsv` (16 rows: 15
|
||||
`no-verdict`, 1 `weak-subprocess`); the lane's `CHECK` grades it as a fourth leg and is
|
||||
**rc 0** at `82504de`. `tools/verify.py` is **8 legs, 8 pass**.
|
||||
|
||||
Nothing in another lane's files was edited: the 15 are recorded as findings and handed to
|
||||
the lane that owns them.
|
||||
|
||||
## 1. Red, reproduced — not relayed
|
||||
|
||||
`q65-2026-10-25-c5-red.txt`, verbatim, with the ledger deliberately empty:
|
||||
|
||||
$ python3 wizard-work/volt/q65_no_verdict.py --repo . --check --ledger /dev/null
|
||||
committed verifiers : 106
|
||||
no failure path : 15 (spark 15)
|
||||
NEW wizard-work/spark/verify_all.py … (15 lines, one per file)
|
||||
q65_no_verdict: 15 fault(s) rc 1
|
||||
|
||||
That is the tree as it stands, not a constructed fixture: every one of the 15 was
|
||||
committed by spark and none of them can return anything but 0.
|
||||
|
||||
## 2. What counts as a verdict path, and what that misses
|
||||
|
||||
The question is asked of the **AST**, so a docstring or a comment that says "verify" or
|
||||
"assert" is invisible to it — which is the defect being measured, since prose about
|
||||
failing is not a failure path. A file has a verdict path if its code contains any of:
|
||||
|
||||
raise <anything> assert <expr>
|
||||
sys.exit(<nonzero>) raise SystemExit(<nonzero>)
|
||||
unittest.main()
|
||||
|
||||
`subprocess.run(..., check=True)` / `check_call` / `check_output` / `pytest.main()` are a
|
||||
separate class, **`weak-subprocess`**, and not a verdict: a `check=True` grades the child
|
||||
process, not the subject. `sys.exit()` and `exit(0)` are scored as **no** verdict — a bare
|
||||
`sys.exit()` is rc 0 — and the selftest holds that case.
|
||||
|
||||
**What this cannot see**, stated so the figure is not read as stronger than it is: a
|
||||
`raise` inside a swallowed `try/except` counts as a verdict path (the selftest fixes that
|
||||
behaviour at `assert + swallowed`), and a file that never terminates is the runner's
|
||||
question, not this one's — its row is `TIMEOUT`. Both limitations are stated in the
|
||||
tool's docstring. Neither changes a row here: **no file among the 16 contains a `raise`
|
||||
at all** (`grep -l raise` over the ledger's paths → none), and the two that contain a
|
||||
`try` (`verify_all.py:22`, `verify_minimal.py`, four blocks) use it to *catch* the
|
||||
subject's exception and store it as a string — which is the defect, recorded, not a
|
||||
blind spot in the scan.
|
||||
|
||||
## 3. The 15, and the 14 rows that cannot fail
|
||||
|
||||
All 15 are in spark's 36 committed verifiers (42%), and 14 of them carry `PASS rc 0` rows
|
||||
in `q65_runner_baseline.tsv` — so Q65's headline figure `graded 88 of 88` counts 14 rows
|
||||
that grade nothing. `verify_dirs.py`, the fifteenth, is not in the runner ledger because a
|
||||
committed line invokes it (`wizard-work/spark/run_verify.sh:2`, cycle 4's `wired-invoke`
|
||||
class), which is the worse case: an invoked printer is a gate that is constantly green.
|
||||
|
||||
The sample, read rather than inferred — each line is the file's whole body of "checking":
|
||||
|
||||
verify_sum.py sums a literal list and writes sum=/count= to a file (7 loc)
|
||||
verify_push.py prints a sha256 of one file (4 loc)
|
||||
verify_fib.py prints "All checks pass: <bool>"; no exit path (35 loc)
|
||||
verify_dirs.py prints {"<dir>": "EXISTS"|"MISSING"}; no exit path (30 loc)
|
||||
verify_minimal.py records "status": "error" into a JSON report, returns (73 loc)
|
||||
|
||||
`q65_no_verdict.tsv` carries all 16 rows with `class`, `path`, `loc`, `blob` (12 chars of
|
||||
the committed blob, so a row is bound to the bytes it was written against) and the token
|
||||
counts that produced the class.
|
||||
|
||||
## 4. Four dynamic controls: the rc does not move when the subject breaks
|
||||
|
||||
Copies of the committed files, output paths redirected to scratch, run unmodified and then
|
||||
with exactly one injected fault (`q65-2026-10-25-c5-controls.txt`):
|
||||
|
||||
| probe | unmodified | corrupted subject | what changed |
|
||||
|---|---|---|---|
|
||||
| `spark/verify_fib.py` recurrence `fibs[-1]+fibs[-2]` → `+fibs[-1]` | rc **0**, `All checks pass: True` | rc **0**, `All checks pass: False` | the print, not the status |
|
||||
| `spark/verify_sum.py` last term dropped from `nums` | rc **0**, writes `sum=10945` | rc **0**, writes `sum=6764 count=19` | the artifact, not the status |
|
||||
| `spark/verify_dirs.py` one dir → `/nonexistent/q65-probe` | rc **0** | rc **0**, prints `"/nonexistent/q65-probe": "MISSING"` | the JSON, not the status |
|
||||
| `spark/verify_minimal.py` config path → `/nonexistent/q65-probe.yaml` | rc **0**, `"status": "ok"` | rc **0**, `"status": "error"` | the JSON, not the status |
|
||||
|
||||
In every case the file *knows* the subject is broken and says so where nobody grades it.
|
||||
That is the class in one sentence: **the verdict exists, in a channel the gate does not
|
||||
read.**
|
||||
|
||||
## 5. Negative controls on the instrument — it can fail
|
||||
|
||||
Four commits in a fresh clone of `82504de` (`q65-2026-10-25-c5-negatives.txt`):
|
||||
|
||||
commit 1 nothing changed -> 0 fault(s) rc 0
|
||||
commit 2 a 16th printer, verify_probe_new.py -> NEW … verify_probe_new.py rc 1
|
||||
commit 3 that printer gains sys.exit(...) -> 0 fault(s) rc 0
|
||||
commit 4 an existing printer (verify_sum.py)
|
||||
gains a real exit path -> STALE … verify_sum.py rc 1
|
||||
vacuous 14 -> 13
|
||||
|
||||
Commit 3 is the intended asymmetry and it is worth naming: a file that is `NEW` because it
|
||||
has no ledger row and *then* gains a verdict path before anyone rows it is clean — the
|
||||
ledger names findings, and it is not a finding. Commit 4 is the row that has to be
|
||||
re-emitted, so a fix is never silently absorbed: `--emit` is one command.
|
||||
|
||||
## 6. Numbers, reconciled rather than restated
|
||||
|
||||
committed verify_*.py 106 (spark 36 forge 29 ledger 18 sphinx 14 sol 6 null 2 beam 1)
|
||||
no failure path 15 all spark, all listed in q65_no_verdict.tsv
|
||||
weak-subprocess only 1 wizard-work/ledger/verify_dates.py
|
||||
runner ledger PASS rows 64 of 88
|
||||
… on a no-verdict file 14 vacuous: cannot be FAIL under any input
|
||||
q65_no_verdict --check 0 fault(s) rc 0
|
||||
bash wizard-work/volt/CHECK 4 legs incl. this one; CHECK PASS rc 0
|
||||
python3 tools/verify.py --repo . 8 legs, 8 pass, 0 fault, 0 ungradable rc 0
|
||||
|
||||
`verify_dates.py` scores `weak-subprocess` and not `no-verdict` for a reason worth
|
||||
recording: its single `check=True` is on a `git log` call, so the *call* has a failure
|
||||
path and the file's comparison loop does not. Cycle 4's finding — impossible dates pass —
|
||||
stands unchanged, and the class here agrees with it rather than replacing it.
|
||||
|
||||
## 7. What is open, and whose
|
||||
|
||||
- **15 spark verifiers have no verdict path.** Owner: spark. The ask is one line each: a
|
||||
comparison that already exists in the print (`verify_fib.py` prints
|
||||
`All checks pass: False`) becomes `sys.exit(0 if ok else 1)`. Not repaired here — the
|
||||
files are spark's, and a lane rewriting another lane's verifier is the mistake Q63's
|
||||
reply-91 §3 already recorded once.
|
||||
- **The 14 vacuous `PASS` rows are Q65's own bookkeeping**, and this is the tightening:
|
||||
`graded 88 of 88` is an rc-coverage figure and always was. The record now carries
|
||||
`cannot fail: 14` beside it, so the number is not read as semantic coverage.
|
||||
- **The class is static.** A future verifier that computes a verdict and swallows it in
|
||||
`try/except` passes this instrument. The next instrument, if any, is the dynamic one:
|
||||
invert the subject and see whether rc moves — cycle 4 did that by hand for one file, this
|
||||
cycle for four. Doing it for 106 is a runner feature (a mutation axis), not a ledger
|
||||
column, and it belongs to Q65's remaining scope rather than to this cycle.
|
||||
- **Still not a `tools/verify.py` leg.** Reply 93 §4: promoting the census to a shared leg
|
||||
is the operator's call. This cycle adds a leg to the lane's own `CHECK` only.
|
||||
|
||||
## Evidence
|
||||
|
||||
`q65_no_verdict.py` (instrument, selftest 13/13), `q65_no_verdict.tsv` (16 rows, header
|
||||
carries the rev and the committed-verifier count), `q65-2026-10-25-c5-red.txt`,
|
||||
`-green.txt`, `-controls.txt`, `-negatives.txt` (each with a provenance line naming the
|
||||
command and the rev). `CHECK` diff: two added lines and the pass message. `git diff
|
||||
--numstat` for this cycle touches four files, all under `wizard-work/volt/`. Nothing from
|
||||
`inbox/` is quoted. No other lane's file was edited.
|
||||
@@ -0,0 +1,326 @@
|
||||
#!/usr/bin/env python3
|
||||
"""q65_no_verdict.py -- does a committed verifier have a failure path at all?
|
||||
|
||||
Q65's census grades each committed `wizard-work/**/verify_*.py` by running it and
|
||||
recording its rc. A verifier whose source has no failure path returns rc 0 for
|
||||
every input, so its row is a constant: it can never be a FAIL, and `graded 88 of
|
||||
88` counts it as graded. Q65 cycle 4 found one such file by hand
|
||||
(`ledger/verify_dates.py`: prints six lines and falls off the end of main());
|
||||
this instrument is the static question asked of all of them.
|
||||
|
||||
WHAT COUNTS AS A VERDICT PATH (ast, so docstrings and comments are invisible --
|
||||
which is the defect being measured: prose about failing is not a failure path):
|
||||
|
||||
raise <anything> a Raise node
|
||||
assert <expr> an Assert node
|
||||
sys.exit(<nonzero>) / exit(<nonzero>) / raise SystemExit(<nonzero>)
|
||||
unittest.main() / pytest.main() the runner exits (pytest.main: see WEAK)
|
||||
subprocess.run(..., check=True) raises on a bad rc -- WEAK, see below
|
||||
|
||||
WEAK-SUBPROCESS is its own class and not a verdict: a `check=True` grades the
|
||||
child process, not the subject. `ledger/verify_dates.py` carries exactly one
|
||||
(its `git log` call) and scores WEAK here while its comparison loop prints and
|
||||
returns -- so WEAK is printed per-file for a human to read and is never a fault
|
||||
on its own.
|
||||
|
||||
WHAT THIS CANNOT SEE, stated so the numbers are not read as stronger than they
|
||||
are: a `raise` inside a swallowed `try/except` counts as a verdict path (false
|
||||
negative for the printer class); a comparison whose result is printed and
|
||||
`sys.exit(0 if ok else 1)`-ed counts correctly; a file that never runs to
|
||||
completion is the runner's question (its row is TIMEOUT, not mine).
|
||||
|
||||
Exit codes are the tools/ contract's: 0 clean, 1 graded a fault, 2 could not grade.
|
||||
--check ledger vs tree at --rev; NEW no-verdict file or a STALE ledger row = rc 1
|
||||
--emit rewrite the ledger from the tree
|
||||
--json the full classification, for re-derivation
|
||||
"""
|
||||
import argparse
|
||||
import ast
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import warnings
|
||||
|
||||
# several committed verifiers carry a "\u" in a docstring; that is a SyntaxWarning
|
||||
# on parse and not this instrument's business.
|
||||
warnings.filterwarnings("ignore", category=SyntaxWarning)
|
||||
|
||||
DEFAULT_LEDGER = os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||||
"q65_no_verdict.tsv")
|
||||
RUNNER_LEDGER = os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||||
"q65_runner_baseline.tsv")
|
||||
EXIT_NAMES = {"exit", "_exit", "quit"}
|
||||
WEAK_KW = {"check_call", "check_output"}
|
||||
|
||||
|
||||
def git(repo, *args):
|
||||
p = subprocess.run(["git", "-C", repo, *args], capture_output=True, text=True)
|
||||
if p.returncode != 0:
|
||||
raise RuntimeError((p.stderr or p.stdout).strip())
|
||||
return p.stdout
|
||||
|
||||
|
||||
def _nonzero(arg):
|
||||
"""Is this exit() argument a non-zero code? Unknown expressions count as
|
||||
non-zero: an exit(<expr>) is a deliberate status, and the defect class is the
|
||||
file with no status at all."""
|
||||
if arg is None:
|
||||
return False
|
||||
if isinstance(arg, ast.Constant):
|
||||
if arg.value is None or arg.value == 0 or arg.value is False:
|
||||
return False
|
||||
return True
|
||||
return True
|
||||
|
||||
|
||||
def classify(src):
|
||||
"""Return (klass, tokens) for one verifier's source text."""
|
||||
t = {"raise": 0, "assert": 0, "exit": 0, "runner_main": 0, "weak": 0,
|
||||
"exit_zero": 0}
|
||||
try:
|
||||
tree = ast.parse(src)
|
||||
except SyntaxError as e:
|
||||
return "UNPARSEABLE", dict(t, error=str(e))
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Raise):
|
||||
t["raise"] += 1
|
||||
elif isinstance(node, ast.Assert):
|
||||
t["assert"] += 1
|
||||
elif isinstance(node, ast.Call):
|
||||
f = node.func
|
||||
name = (f.attr if isinstance(f, ast.Attribute)
|
||||
else f.id if isinstance(f, ast.Name) else "")
|
||||
if name in EXIT_NAMES:
|
||||
arg = node.args[0] if node.args else None
|
||||
if _nonzero(arg):
|
||||
t["exit"] += 1
|
||||
else:
|
||||
t["exit_zero"] += 1 # sys.exit() / exit(0) is rc 0
|
||||
elif name in WEAK_KW:
|
||||
t["weak"] += 1
|
||||
elif name == "run":
|
||||
for kw in node.keywords:
|
||||
if kw.arg == "check" and isinstance(kw.value, ast.Constant) \
|
||||
and kw.value.value:
|
||||
t["weak"] += 1
|
||||
elif name == "main" and isinstance(f, ast.Attribute) and \
|
||||
isinstance(f.value, ast.Name) and f.value.id == "unittest":
|
||||
t["runner_main"] += 1
|
||||
elif name == "main" and isinstance(f, ast.Attribute) and \
|
||||
isinstance(f.value, ast.Name) and f.value.id == "pytest":
|
||||
t["weak"] += 1 # pytest.main returns an int, does not exit
|
||||
strong = t["raise"] + t["assert"] + t["exit"] + t["runner_main"]
|
||||
if strong and not t["weak"]:
|
||||
return "has-verdict", t
|
||||
if strong:
|
||||
return "has-verdict+weak", t
|
||||
if t["weak"]:
|
||||
return "weak-subprocess", t
|
||||
return "no-verdict", t
|
||||
|
||||
|
||||
def verifiers(repo, rev):
|
||||
out = git(repo, "ls-tree", "-r", "--name-only", rev).split()
|
||||
return sorted(p for p in out
|
||||
if p.startswith("wizard-work/") and "/verify_" in p
|
||||
and p.endswith(".py"))
|
||||
|
||||
|
||||
def census(repo, rev):
|
||||
rows = []
|
||||
for p in verifiers(repo, rev):
|
||||
src = git(repo, "show", f"{rev}:{p}")
|
||||
klass, t = classify(src)
|
||||
rows.append({"path": p, "class": klass, "tokens": t,
|
||||
"loc": src.count("\n") + 1,
|
||||
"blob": git(repo, "rev-parse", f"{rev}:{p}").strip()[:12]})
|
||||
return rows
|
||||
|
||||
|
||||
def read_ledger(path):
|
||||
led = {}
|
||||
if not os.path.isfile(path):
|
||||
return led
|
||||
for ln in open(path, encoding="utf-8"):
|
||||
if not ln.strip() or ln.lstrip().startswith("#"):
|
||||
continue
|
||||
parts = ln.rstrip("\n").split("\t")
|
||||
if len(parts) >= 2:
|
||||
led[parts[1]] = parts
|
||||
return led
|
||||
|
||||
|
||||
def read_runner(path):
|
||||
"""path -> verdict, for rows whose rc agrees with the verdict."""
|
||||
rows = {}
|
||||
if not os.path.isfile(path):
|
||||
return rows
|
||||
for ln in open(path, encoding="utf-8"):
|
||||
if not ln.strip() or ln.lstrip().startswith("#"):
|
||||
continue
|
||||
parts = ln.rstrip("\n").split("\t")
|
||||
if len(parts) >= 4:
|
||||
rows[parts[1]] = (parts[2], parts[3])
|
||||
return rows
|
||||
|
||||
|
||||
def emit(rows, path, rev):
|
||||
bad = [r for r in rows if r["class"] in ("no-verdict", "weak-subprocess",
|
||||
"UNPARSEABLE")]
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
f.write("# Q65 no-verdict census -- one row per committed verifier with no\n")
|
||||
f.write("# failure path of its own (or only a subprocess check). Written by\n")
|
||||
f.write("# q65_no_verdict.py --emit; a row states the finding, it is not an\n")
|
||||
f.write("# exemption: --check faults on a no-verdict file this file does not\n")
|
||||
f.write("# name, and on a row whose file has since gained a verdict path.\n")
|
||||
f.write(f"# rev {rev} committed verifiers {len(rows)} rows {len(bad)}\n")
|
||||
f.write("# class\tpath\tloc\tblob\ttokens\tevidence\n")
|
||||
for r in bad:
|
||||
tok = ",".join(f"{k}={v}" for k, v in sorted(r["tokens"].items()) if v)
|
||||
f.write("\t".join([r["class"], r["path"], str(r["loc"]), r["blob"],
|
||||
tok or "no-failure-token", ""]) + "\n")
|
||||
return bad
|
||||
|
||||
|
||||
def check(rows, led, runner):
|
||||
nov = [r for r in rows if r["class"] == "no-verdict"]
|
||||
weak = [r for r in rows if r["class"] == "weak-subprocess"]
|
||||
known = {p: v for p, v in led.items() if v[0] == "no-verdict"}
|
||||
new = [r for r in nov if r["path"] not in known]
|
||||
stale = [p for p in known if p not in {r["path"] for r in nov}]
|
||||
# a PASS row on a file with no verdict path is a constant, not a grade
|
||||
vacuous = [r for r in nov
|
||||
if runner.get(r["path"], ("", ""))[0] in ("PASS",)]
|
||||
return nov, weak, new, stale, vacuous
|
||||
|
||||
|
||||
def report(repo, rev, led, runner, verbose=False):
|
||||
rows = census(repo, rev)
|
||||
nov, weak, new, stale, vacuous = check(rows, led, runner)
|
||||
lanes = {}
|
||||
for r in nov:
|
||||
lane = r["path"].split("/")[1]
|
||||
lanes[lane] = lanes.get(lane, 0) + 1
|
||||
print(f"q65_no_verdict -- rev {rev}")
|
||||
print(" committed verifiers : %d" % len(rows))
|
||||
print(" no failure path : %d (%s)"
|
||||
% (len(nov), " ".join(f"{k} {v}" for k, v in sorted(lanes.items(),
|
||||
key=lambda kv: -kv[1]))
|
||||
or "none"))
|
||||
for r in nov:
|
||||
tok = ",".join(f"{k}={v}" for k, v in sorted(r["tokens"].items()) if v) or "-"
|
||||
print(f" NO-VERDICT {r['path']:64s} loc {r['loc']:3d} {tok}")
|
||||
print(" weak only (subprocess) : %d" % len(weak))
|
||||
for r in weak:
|
||||
print(f" WEAK {r['path']}")
|
||||
print(" runner PASS rows that cannot fail: %d of %d"
|
||||
% (len(vacuous), sum(1 for v in runner.values() if v[0] == "PASS")))
|
||||
if verbose:
|
||||
for r in vacuous:
|
||||
print(f" VACUOUS {r['path']} "
|
||||
f"(runner row {runner[r['path']][0]} rc {runner[r['path']][1]})")
|
||||
for r in new:
|
||||
print(f" NEW {r['path']} (no-verdict, not in {os.path.basename(DEFAULT_LEDGER)})")
|
||||
for p in stale:
|
||||
print(f" STALE {p} (ledger row, file now has a verdict path or is gone)")
|
||||
faults = len(new) + len(stale)
|
||||
print("q65_no_verdict: %d fault(s)" % faults)
|
||||
return faults
|
||||
|
||||
|
||||
FIXTURES = [
|
||||
("printer", "print('all good')\nopen('/tmp/x','w').write('ok')\n",
|
||||
"no-verdict"),
|
||||
("bare exit", "import sys\nprint('fine')\nsys.exit()\n", "no-verdict"),
|
||||
("exit 0", "import sys\nsys.exit(0)\n", "no-verdict"),
|
||||
("assert only", "assert 1 + 1 == 2\n", "has-verdict"),
|
||||
("sys.exit(nonzero)", "import sys\nsys.exit(1)\n", "has-verdict"),
|
||||
("raise SystemExit", "raise SystemExit(2)\n", "has-verdict"),
|
||||
("subprocess only", "import subprocess\nsubprocess.run(['true'], check=True)\n",
|
||||
"weak-subprocess"),
|
||||
("unittest.main", "import unittest\nclass T(unittest.TestCase):\n"
|
||||
" def test_a(self):\n self.assertTrue(True)\n"
|
||||
"if __name__ == '__main__':\n unittest.main()\n",
|
||||
"has-verdict"),
|
||||
("assert + swallowed", "try:\n assert False\nexcept Exception:\n pass\n",
|
||||
"has-verdict"),
|
||||
]
|
||||
|
||||
|
||||
def selftest():
|
||||
bad = 0
|
||||
for name, src, want in FIXTURES:
|
||||
got, t = classify(src)
|
||||
ok = got == want
|
||||
bad += 0 if ok else 1
|
||||
print(f" {'ok ' if ok else 'FAIL'} {name:22s} -> {got} (want {want})")
|
||||
# check(): a NEW no-verdict file faults, a STALE row faults, a match is clean
|
||||
rows = [{"path": "wizard-work/spark/verify_a.py", "class": "no-verdict",
|
||||
"tokens": {}, "loc": 1, "blob": "x"},
|
||||
{"path": "wizard-work/spark/verify_b.py", "class": "has-verdict",
|
||||
"tokens": {}, "loc": 1, "blob": "y"}]
|
||||
led = {"wizard-work/spark/verify_a.py": ["no-verdict", "wizard-work/spark/verify_a.py"]}
|
||||
nov, weak, new, stale, vac = check(rows, led, {})
|
||||
ok = (len(nov) == 1 and len(new) == 0 and len(stale) == 0)
|
||||
print(f" {'ok ' if ok else 'FAIL'} ledger match -> new {len(new)} stale {len(stale)}")
|
||||
bad += 0 if ok else 1
|
||||
rows[0]["class"] = "has-verdict"
|
||||
nov, weak, new, stale, vac = check(rows, led, {})
|
||||
ok = len(stale) == 1
|
||||
print(f" {'ok ' if ok else 'FAIL'} fixed file is STALE -> stale {len(stale)}")
|
||||
bad += 0 if ok else 1
|
||||
rows[0]["class"] = "no-verdict"
|
||||
nov, weak, new, stale, vac = check(rows, {}, {})
|
||||
ok = len(new) == 1
|
||||
print(f" {'ok ' if ok else 'FAIL'} unrowed file is NEW -> new {len(new)}")
|
||||
bad += 0 if ok else 1
|
||||
vac_rows = check(rows, led, {"wizard-work/spark/verify_a.py": ("PASS", "0")})
|
||||
ok = len(vac_rows[4]) == 1
|
||||
print(f" {'ok ' if ok else 'FAIL'} PASS row is vacuous -> vacuous {len(vac_rows[4])}")
|
||||
bad += 0 if ok else 1
|
||||
print(f"q65_no_verdict selftest: {'PASS' if not bad else 'FAIL'} "
|
||||
f"({len(FIXTURES) + 4 - bad}/{len(FIXTURES) + 4})")
|
||||
return bad
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--repo", default=".")
|
||||
ap.add_argument("--rev", default="HEAD")
|
||||
ap.add_argument("--ledger", default=DEFAULT_LEDGER)
|
||||
ap.add_argument("--runner-ledger", default=RUNNER_LEDGER)
|
||||
ap.add_argument("--check", action="store_true")
|
||||
ap.add_argument("--emit", action="store_true")
|
||||
ap.add_argument("--json")
|
||||
ap.add_argument("--verbose", action="store_true")
|
||||
ap.add_argument("--selftest", action="store_true")
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.selftest:
|
||||
return 0 if selftest() == 0 else 1
|
||||
try:
|
||||
git(a.repo, "rev-parse", a.rev)
|
||||
except (RuntimeError, FileNotFoundError) as e:
|
||||
print(f"q65_no_verdict: cannot grade -- {e}", file=sys.stderr)
|
||||
return 2
|
||||
try:
|
||||
if a.emit:
|
||||
rows = census(a.repo, a.rev)
|
||||
bad = emit(rows, a.ledger, a.rev)
|
||||
print(f"q65_no_verdict: {len(bad)} row(s) -> {a.ledger} "
|
||||
f"({len(rows)} committed verifiers at {a.rev})")
|
||||
return 0
|
||||
faults = report(a.repo, a.rev, read_ledger(a.ledger),
|
||||
read_runner(a.runner_ledger), a.verbose)
|
||||
if a.json:
|
||||
with open(a.json, "w", encoding="utf-8") as f:
|
||||
json.dump(census(a.repo, a.rev), f, indent=1)
|
||||
return 0 if faults == 0 else 1
|
||||
except (RuntimeError, FileNotFoundError) as e:
|
||||
print(f"q65_no_verdict: cannot grade -- {e}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,23 @@
|
||||
# Q65 no-verdict census -- one row per committed verifier with no
|
||||
# failure path of its own (or only a subprocess check). Written by
|
||||
# q65_no_verdict.py --emit; a row states the finding, it is not an
|
||||
# exemption: --check faults on a no-verdict file this file does not
|
||||
# name, and on a row whose file has since gained a verdict path.
|
||||
# rev HEAD committed verifiers 106 rows 16
|
||||
# class path loc blob tokens evidence
|
||||
weak-subprocess wizard-work/ledger/verify_dates.py 38 dd3f643441c9 weak=1
|
||||
no-verdict wizard-work/spark/verify_all.py 46 41029ee9976f no-failure-token
|
||||
no-verdict wizard-work/spark/verify_check.py 13 3b7ef86aa2f7 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_count.py 11 640a65519f81 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_dirs.py 30 d2846e746204 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_fib.py 35 163468d71cd7 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_json.py 26 71b343f0b1f3 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_lines.py 8 5be4cf7b5f1e no-failure-token
|
||||
no-verdict wizard-work/spark/verify_minimal.py 73 2b1b18fc7ed7 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_push.py 4 707bdcb80646 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_python_env.py 32 17c1d5f6ab0b no-failure-token
|
||||
no-verdict wizard-work/spark/verify_skills.py 25 be69f202a823 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_skills2.py 28 9dd92b966a6e no-failure-token
|
||||
no-verdict wizard-work/spark/verify_skills3.py 42 364a78ffbe33 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_skills_fixed.py 32 4bc11949dc38 no-failure-token
|
||||
no-verdict wizard-work/spark/verify_sum.py 7 8faaade76bc3 no-failure-token
|
||||
|
Reference in new issue
Block a user