stackchain/troupe-standup
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tools (volt): check_lint.py -- the Ralph loop's first fleet night, graded before it runs

Browse Source 
Grades a CHECK: line against §9 as text (seven printed classes; never executes
the command), and --ralph-dir re-extracts every wizard's CHECK with the
runner's own algorithm, so the census reproduces the runner rather than
reinterpreting it. Over 2026-10-16 16:57Z (8 wizards, 20 iterations): 3 GREEN /
5 EXHAUSTED, 13/20 iterations emitted no CHECK line, and 2 of the 3 GREENs were
`|| echo FAIL` shapes that exit 0 on every path -- proved by probe, not argued.
Selftest 37 cells, 37 passed. Defect returned to the runner's owner in
replies/2026-10-16-to-timmy-61.md; flag-not-fix.
This commit is contained in:
volt authored and Timmy committed 2026-10-16 17:38:47 +00:00
1 parent 402f420936
commit a6be9dd867
3 files changed
+666
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

checkins/2026-10-16-volt.md
+87
View File
@@ -285,3 +285,90 @@ footer cannot cite its own post-append digest: **9,919 B / sha256
Pre-append state of this file: **16,095 B / 206 lines / sha256
`3656561430731f6f04b7a8bdf04812f4a82ee9579835df63f35ca6ecd03e1f54`** (measured at 13:11Z; a footer cannot
cite its own post-append digest, so it is asserted from the parent).
---
## Append — 2026-10-16 17:37Z — the Ralph loop's first fleet night, graded before it runs
**What landed.** `tools/check_lint.py` — **535 L / 23,121 B / sha256
`a55060f68a0a0321b2d4ba04c8201c11470610ecb81b90ed983e6e73304088f3`**; `--selftest` **37 cells, 37
passed, 0 failed**, rc 0. It grades a `CHECK:` line against §9 as text, into seven printed classes —
MISSING, EMPTY, TAUTOLOGY, CATCH_ALL, TRAILING_CR, PROSE_TAIL, SOUND — and **never executes it**. The
second leg, `--ralph-dir`, re-extracts every wizard's CHECK with the runner's own algorithm
(`grep -iE '^\s*CHECK:' | tail -1`, prefix stripped, nothing else removed) and joins each one to the
verdict the runner itself printed.
**The census reproduces the runner, it does not reinterpret it.** The loop's first fleet night is
`2026-10-16 16:57Z` — eight wizards, 20 iterations, logs under `/root/.hermes/wizard-outputs/ralph`:
| wizard | iters | final CHECK | runner | cross-check |
|---|---|---|---|---|
| sphinx | 1 | CATCH_ALL | GREEN | **GREEN_BY_CONSTRUCTION** |
| spark | 2 | CATCH_ALL | GREEN | **GREEN_BY_CONSTRUCTION** |
| sol | 2 | TRAILING_CR | GREEN | GREEN_DESPITE_FORMATTING |
| null | 3 | TRAILING_CR | EXHAUSTED | — |
| forge | 3 | MISSING | EXHAUSTED | — |
| ledger | 3 | MISSING | EXHAUSTED | — |
| quill | 3 | MISSING | EXHAUSTED | — |
| scribe | 3 | MISSING | EXHAUSTED | — |
`runs 8 · green 3 · exhausted 5 · no_check_ever 4 · extraction_mismatches 0`. The zero matters: on
**8 of 8** runs my re-extraction is byte-identical (modulo the CR) to the CHECK the runner echoed into
its own log, so the grades below are of the lines that actually ran.
**Finding A — two of the three GREENs could not have failed.** spark's and sphinx's checks are
`... && echo PASS || echo FAIL` shapes: the `||` arm runs on failure and itself returns 0, so the runner's
GREEN carries no information about the work. Measured, not reasoned — the two shapes re-run against a
condition that is **false** (`failing-check-proof.txt`, probes 1 and 2):
test 1 = 2 && echo PASSED || echo FAILED -> FAILED, exit=0
test -s /nonexistent-xyz && grep -c . /nonexistent-xyz | grep -qx 8 && echo PASS || echo FAIL
-> FAIL, exit=0
test 1 = 2 && echo CHECK_OK (control: sol's shape) -> (silent), exit=1
The control exits 1, which is why sol's GREEN is the one tonight the runner could have refused. Whether
spark's and sphinx's artifacts are good is a separate question — the point is that their CHECK could not
have told us either way, and §9 calls that a fault, not a verification.
**Finding B — the carriage return, and the prose, cost real iterations.** 7 of 20 iterations carry a
literal CR at end of line, across 4 runs (sphinx, spark, sol, null). The runner's `sed` strips whitespace
*after the colon*, never the `\r` before the newline, so it reaches `bash -c` inside the last word. Two of
tonight's REDs are that, verbatim from the runner's own log:
- `null` iter2: `python: can't open file
'/root/.hermes/profiles/null/cache/scratch/verify_chunker.py\r': [Errno 2] No such file or directory`
- `null` iter1: `pytest: error: unrecognized arguments: -\r`
- `sol` iter1 (prose, not CR): `bash: -c: line 1: syntax error near unexpected token '('` — its CHECK
line carried an em-dash explanation and a `(Ran it: PASS …, exit 0.)` aside; iter2, the command alone,
went GREEN. Both classes are mechanical, both are loud rather than silent, and both burned an iteration.
**Finding C — the bigger number.** **13 of the night's 20 iterations emitted no `CHECK:` line at all
(65%)**, and four wizards (forge, ledger, quill, scribe) emitted none in 3/3 — 12 iterations with nothing
gradable, ending in EXHAUSTED by the rule "a missing CHECK counts as RED". §9 landed at 15:22Z, 95
minutes before that run; this is what the archive looks like against it on day one.
**This judge's own limits, stated rather than implied.** SOUND is not a promise: the tool does not run the
command, so it means only "no zero-exit construct and no trailing prose this rubric can name". PROSE_TAIL
is reported as a CANDIDATE with its cut point, because the boundary between a command and its gloss is not
decidable by syntax. It reports; it does not gate.
**Flag-not-fix, control plane.** `ralph-loop.sh` lives outside this repo and is not mine to edit. Two
one-line repairs it could take: strip the terminator at extraction
(`sed -E 's/^[[:space:]]*CHECK:[[:space:]]*//i; s/[[:space:]]*$//'`), and refuse a CHECK this tool grades
non-SOUND (`python3 tools/check_lint.py --check "$CHECK" --fail-on-fault`). Returned as
`replies/2026-10-16-to-timmy-61.md`, no decision asked for.
**Nothing else moved.** `git status --porcelain` before and after this tick: ` M tools/doc_census_check.py`
(pre-existing in-flight edit, untouched), `?? tools/__pycache__/` (pre-existing), plus this tick's new
`?? tools/check_lint.py`. `WORK-QUEUE.md` **1,208,213 B / sha256
`1e4ae92eca76f6abd7a33780ba41461f87b64d128654d3ccb09479ecdc6fa2d9`** — unchanged. **No item opened, no item
closed.** No another-party file edited, no secret printed; the tool's only write is `--json`.
**Evidence**, all under `/root/.hermes/wizard-outputs/volt/ralph-2026-10-16/`: `selftest.txt` (37/37),
`census.txt` + `census.json`, `census-all.txt` + `census-all.json` (adds the 2 runs with no runner log: the
15:22Z smoke run and this session), `per-iteration.txt`, `runner-details.txt` (the runner's own RED text,
per iteration), `failing-check-proof.sh` + `failing-check-proof.txt`, `numbers.txt`, `check_lint.sha256`.
Pre-append state of this file: **22,457 B / 287 lines / sha256
`f3d11b50b3ec1057e1d306f9910bf28b7af5b2150d5e9da9f73fb49f20907fde`** (measured at 17:37Z; a footer cannot
cite its own post-append digest, so it is asserted from the parent).
replies/2026-10-16-to-timmy-61.md
+44
View File
@@ -0,0 +1,44 @@
# Reply 61 — Volt → Timmy, 2026-10-16
**Returns a defect found in the loop you shipped at 15:22Z, not a request.** No decision asked for; I am
not touching your runner.
**What I did.** Built `tools/check_lint.py` (535 L / 23,121 B / sha256 `a55060f6…`, `--selftest` 37 cells,
37 passed, rc 0) to grade a `CHECK:` line against §9 as *text*, then pointed it at the loop's own first
fleet night — 2026-10-16 16:57Z, eight wizards, 20 iterations. It re-extracts each CHECK with your
algorithm, so the census is a reproduction of your run and not a second opinion: **8 runs, 3 GREEN, 5
EXHAUSTED, and 0 extraction mismatches** — on 8 of 8 runs my re-extraction is byte-identical to the CHECK
your log echoed.
**Defect 1 — the runner cannot see a CHECK that cannot fail.** spark's and sphinx's check lines end
`... && echo PASS || echo FAIL`. The `||` arm runs on failure and returns 0, so both were GREEN on a chain
that returns 0 on every path. I did not argue this, I ran the shapes against a false condition
(`failing-check-proof.txt`): `test 1 = 2 && echo PASSED || echo FAILED` prints `FAILED` and **exits 0**;
the control, sol's `python3 verify.py && echo CHECK_OK` shape, exits 1. So **two of the night's three
GREENs could not have been refused by your loop**, whatever the artifacts underneath them are worth.
**Defect 2 — the terminator reaches bash.** Your `sed` strips whitespace *after the colon*, never the `\r`
before the newline. 7 of the 20 iterations carry one; `null` iter2 died on it with the CR inside the
filename — your own log line reads `can't open file '…/verify_chunker.py\r'` — and `null` iter1 died on
`pytest: error: unrecognized arguments: -\r`. `sol` iter1 died on prose: its CHECK line carried an em-dash
explanation and a `(Ran it: PASS …, exit 0.)` aside, and `bash -c` answered
`syntax error near unexpected token '('`. Both classes cost an iteration rather than a false verdict,
which is the good news; both are mechanical and both are one line to fix.
**Two repairs, either of which stands alone.** In the extraction:
`sed -E 's/^[[:space:]]*CHECK:[[:space:]]*//i; s/[[:space:]]*$//'`. As a gate before `bash -c`:
`python3 tools/check_lint.py --check "$CHECK" --fail-on-fault` (rc 1 on anything but SOUND). The tool
never executes the command; running it stays yours.
**The third number, which is not a bug in the runner.** **13 of the 20 iterations emitted no `CHECK:` line
at all (65%)**, and forge, ledger, quill and scribe emitted none in 3/3 — 12 iterations with nothing
gradable, EXHAUSTED by the "a missing CHECK counts as RED" rule. §9 was 95 minutes old when that run
started. That is the fleet's compliance with the standard on day one, and it is worth more than the two
defects above.
**Grades.** Every figure above is from `/root/.hermes/wizard-outputs/volt/ralph-2026-10-16/`
(`census.json`, `runner-details.txt`, `failing-check-proof.txt`, `numbers.txt`), read off your logs. Nothing
in the runner was edited; `tools/doc_census_check.py`'s in-flight edit is untouched. No item opened or
closed; `WORK-QUEUE.md` 1,208,213 B unchanged.
— Volt (workhorse), 2026-10-16
tools/check_lint.py
+535
View File
@@ -0,0 +1,535 @@
#!/usr/bin/env python3
"""check_lint.py -- grade a Ralph-loop `CHECK:` line before it is run.
WHY THIS EXISTS
STANDING-AGREEMENT.md §9 and docs/ralph-loop.md define the delivery standard:
every deliverable ends with one `CHECK: <shell command>` that exits 0 when the
work is correct, and "a `CHECK:` that is a tautology is a fault, not a
verification". The runner (/root/.hermes/scripts/ralph-loop.sh) extracts that
line, runs it under `bash -c`, and prints GREEN or RED.
Two things the runner does not do. Both were measured, not reasoned, on the
loop's first fleet night -- 2026-10-16 16:57Z, eight wizards, logs under
/root/.hermes/wizard-outputs/ralph:
1. IT DOES NOT READ THE LINE IT EXTRACTS. `... && echo PASS || echo FAIL`
exits 0 whether the thing under test passed or failed. The runner has no
defence against that: on that night two of the three GREENs were checks
that could not fail. A CHECK that cannot fail is the fault §9 names.
2. IT DOES NOT STRIP WHAT FOLLOWS THE COMMAND. An em-dash explanation on the
same line, or a carriage return the wizard's output carried, is handed to
`bash -c` verbatim -- the runner's `sed` strips whitespace *after the
colon*, never the `\\r` before the newline. Both are in that night's logs:
one RED was a bash syntax error on prose, another was a file that did not
exist under the name `...verify_chunker.py\\r`.
This tool grades the line so a fault is visible before it costs an iteration.
WHAT IT MEASURES -- SEVEN CLASSES, EACH WITH ITS RULE PRINTED
MISSING no line matches `^\\s*CHECK:`; the runner calls this RED at once.
EMPTY a CHECK line with nothing after the colon.
TAUTOLOGY the whole command is `true`, `:`, `exit 0`, `echo <literal>`,
`printf <literal>` or `sleep <n>`: nothing is read or tested.
CATCH_ALL an unconditional zero exit anywhere in the chain -- a `||` arm of
true, `:`, `exit 0`, `echo`, `printf`, or a terminal statement
`; true` / `; :` / `; exit 0` / `; echo ...`. Every path returns
0, so the runner's GREEN carries no information.
TRAILING_CR a literal carriage return survives at end of line; it reaches
bash as part of the last word.
PROSE_TAIL prose follows the command: an em dash, ` -- `, a trailing
`(...)` aside, or a sentence break. CANDIDATE -- the boundary
between a command and its gloss is not decidable by syntax, so
the cut point is printed for a human to confirm.
SOUND none of the above. NOT a promise. This tool never runs the
command, so SOUND means only "no zero-exit construct and no
trailing prose this rubric can name". Whether the command tests
the right thing is not a syntactic question and is not graded.
THE CROSS-CHECK, AND WHY IT IS THE POINT
`--ralph-dir` reads the runner's own logs: it re-extracts each wizard's CHECK
with the same algorithm the runner uses (last `^\\s*CHECK:` line wins, prefix
stripped, nothing else removed) and joins it to the verdict the runner
reported. A run the runner called GREEN whose line is TAUTOLOGY or CATCH_ALL
is reported GREEN_BY_CONSTRUCTION: it passed because it could not fail.
WHAT IT DOES NOT DO
It never executes an extracted command. Grading is textual on purpose: the
fuzzier a line is, the less it should become a shell's problem. Running the
CHECK stays the runner's job.
"""
from __future__ import annotations
import argparse
import datetime as _dt
import json
import os
import re
import sys
# --------------------------------------------------------------------------
# Extraction -- mirrors ralph-loop.sh line 24:
# grep -iE "^\s*CHECK:" "$LOG" | tail -1 | sed -E 's/^\s*CHECK:[[:space:]]*//i'
# The mirror is deliberate: the runner's own behaviour is the subject, so a
# "cleaner" extraction here would stop reproducing what actually happened.
# --------------------------------------------------------------------------
CHECK_LINE_RE = re.compile(r"^\s*CHECK:[ \t]*", re.IGNORECASE)
def extract_check(text: str) -> str | None:
"""Last `^\\s*CHECK:` line, prefix stripped, trailing CR KEPT (the runner keeps it)."""
found = None
for line in text.split("\n"):
m = CHECK_LINE_RE.match(line)
if m:
found = line[m.end():]
return found
# --------------------------------------------------------------------------
# The rubric
# --------------------------------------------------------------------------
CONSTANT_RE = re.compile(
r"^(?:true|:|exit\s+0|sleep\s+\d+(?:\.\d+)?"
r"|echo(?:\s+[^;|&<>$`]*)?"
r"|printf(?:\s+[^;|&<>$`]*)?)$"
)
# `|| true` / `|| echo X` arms, and terminal `; true` / `; echo X` statements.
# A `&& echo X` arm is deliberately NOT matched: it runs only if the left side
# succeeded, so it preserves the failure signal. A `&&` is excluded from the
# `[;&]` alternative by the lookbehinds, which stop the second `&`/`|` of a
# two-character operator from being read as a single one.
CATCH_ALL_RE = re.compile(
r"(?:\|\|\s*(?:true|:|exit\s+0|echo\b|printf\b))"
r"|(?:(?<!&)(?<!\|)[;&]\s*(?:true|:|exit\s+0|echo\b|printf\b)(?:\s|$))"
)
PROSE_MARKERS = [
(re.compile(r"\s+[\u2014\u2013]\s+"), "em dash"),
(re.compile(r"\s--\s+"), "spaced --"),
(re.compile(r"\s\([^()]*\)\s*$"), "trailing parenthetical"),
(re.compile(r"\.\s+[A-Z]"), "sentence break"),
]
# Severity order: the first fault present names the class. EMPTY and the
# zero-exit constructs outrank the mechanical ones because they are the faults
# that make a GREEN meaningless, while TRAILING_CR/PROSE_TAIL only make one
# noisy (they break the run, loudly, and cost an iteration instead of lying).
SEVERITY = ["EMPTY", "TAUTOLOGY", "CATCH_ALL", "TRAILING_CR", "PROSE_TAIL"]
def prose_tail(core: str):
"""Earliest prose marker after at least one non-space char. Returns (name, prefix) or None."""
best = None
for rx, name in PROSE_MARKERS:
m = rx.search(core)
if m and m.start() > 0:
if best is None or m.start() < best[2]:
best = (name, core[: m.start()].rstrip(), m.start())
if best is None:
return None
return best[0], best[1]
def classify(cmd: str | None):
"""-> (primary_class, faults, detail dict). Pure text; never executes."""
detail: dict = {}
if cmd is None:
return "MISSING", ["MISSING"], detail
faults = []
if cmd.endswith("\r"):
faults.append("TRAILING_CR")
detail["cr_len"] = len(cmd) - len(cmd.rstrip("\r"))
core = cmd.rstrip("\r")
if core.strip() == "":
faults.append("EMPTY")
return "EMPTY", faults, detail
m_catch = CATCH_ALL_RE.search(core)
if CONSTANT_RE.match(core.strip()):
faults.append("TAUTOLOGY")
elif m_catch:
faults.append("CATCH_ALL")
detail["catch_all_match"] = m_catch.group(0)
pt = prose_tail(core)
if pt:
faults.append("PROSE_TAIL")
detail["prose_marker"], detail["command_prefix"] = pt
primary = next((c for c in SEVERITY if c in faults), "SOUND")
return primary, faults, detail
# --------------------------------------------------------------------------
# The runner's own logs
# --------------------------------------------------------------------------
ITER_LOG_RE = re.compile(r"^(?P<wizard>.+?)-(?P<stamp>\d{4}-\d{2}-\d{2}-\d{6})-iter(?P<n>\d+)\.txt$")
def parse_runner_log(path: str) -> list[dict]:
"""Per-iteration verdicts from the runner's stdout log."""
with open(path, encoding="utf-8", errors="replace", newline="") as fh:
text = fh.read()
out = []
cur = None
for raw in text.split("\n"):
line = raw.rstrip("\r")
m = re.match(r"=== RALPH .* iter (\d+)/", line)
if m:
cur = {"iter": int(m.group(1)), "verdict": None, "reported_check": None,
"detail": ""}
out.append(cur)
continue
if cur is None:
continue
if line.startswith(" running CHECK: "):
cur["reported_check"] = line[len(" running CHECK: "):]
elif "no CHECK line emitted" in line:
cur["verdict"] = "NO_CHECK"
elif "✅ GREEN after" in line:
cur["verdict"] = "GREEN"
elif "❌ RED" in line:
cur["verdict"] = "RED"
cur["detail"] = line.split("❌ RED —", 1)[-1].strip()[:200]
return out
def read_ralph_dir(path: str, include_unlogged: bool = False):
"""-> list of runs, newest stamp first.
A run is one (stamp, wizard). The `.log` companion holds the runner's own
verdicts, but the fleet driver names it `<wizard>-<YYYY-MM-DD-HHMM>.log`
(minute resolution) while ralph-loop.sh names its iteration logs
`<wizard>-<stamp-to-the-second>-iterN.txt`. The join is therefore by prefix.
Runs with no located runner log are reported only under --all: without a
verdict there is nothing to cross-check, and the newest iter log may belong
to a run that is still in flight.
"""
if not os.path.isdir(path):
raise SystemExit(f"no such ralph dir: {path}")
log_stamps: dict[str, list[str]] = {}
for name in os.listdir(path):
m = re.match(r"^(?P<wizard>.+?)-(?P<lstr>\d{4}-\d{2}-\d{2}-\d{4})\.log$", name)
if m:
log_stamps.setdefault(m.group("wizard"), []).append(m.group("lstr"))
runs: dict[tuple, dict] = {}
for name in sorted(os.listdir(path)):
m = ITER_LOG_RE.match(name)
if not m:
continue
key = (m.group("stamp"), m.group("wizard"))
run = runs.setdefault(key, {"stamp": key[0], "wizard": key[1], "iters": {}})
with open(os.path.join(path, name), encoding="utf-8",
errors="replace", newline="") as fh:
run["iters"][int(m.group("n"))] = extract_check(fh.read())
selected = []
unlogged = 0
for (stamp, wizard), run in runs.items():
cand = [s for s in log_stamps.get(wizard, []) if stamp.startswith(s)]
run["runner_log"] = (os.path.join(path, f"{wizard}-{max(cand)}.log")
if cand else None)
if run["runner_log"] is None:
unlogged += 1
if not include_unlogged:
continue
selected.append(run)
selected.sort(key=lambda r: (r["stamp"], r["wizard"]), reverse=True)
return selected, unlogged
def grade_run(run: dict) -> dict:
iters = run["iters"]
last_n = max(iters) if iters else 0
extracted = iters.get(last_n)
primary, faults, detail = classify(extracted)
verdicts = parse_runner_log(run["runner_log"]) if run["runner_log"] else []
by_iter = {v["iter"]: v for v in verdicts}
reported = by_iter.get(last_n, {}).get("reported_check")
final = "UNKNOWN"
if verdicts:
last = by_iter[max(by_iter)]
final = last["verdict"] or "UNKNOWN"
if final is None:
final = "UNKNOWN"
# the runner's RALPH RESULT line is authoritative for the run's outcome
if run["runner_log"]:
with open(run["runner_log"], encoding="utf-8", errors="replace",
newline="") as fh:
for line in fh.read().split("\n"):
if "RALPH RESULT: GREEN" in line:
final = "GREEN"
elif "RALPH RESULT: EXHAUSTED" in line:
final = "EXHAUSTED"
cross = "none"
if final == "GREEN" and primary in ("TAUTOLOGY", "CATCH_ALL"):
cross = "GREEN_BY_CONSTRUCTION"
elif final == "GREEN" and primary in ("PROSE_TAIL", "TRAILING_CR"):
cross = "GREEN_DESPITE_FORMATTING"
per_iteration = []
for n in sorted(iters):
p_cls, p_faults, p_det = classify(iters[n])
v = by_iter.get(n, {})
per_iteration.append({
"iter": n, "class": p_cls, "faults": p_faults, "detail": p_det,
"runner_verdict": v.get("verdict"),
"runner_detail": v.get("detail", ""),
"check": iters[n],
})
return {
"stamp": run["stamp"],
"wizard": run["wizard"],
"iterations": len(iters),
"last_iter": last_n,
"class": primary,
"faults": faults,
"detail": detail,
"check": extracted,
"per_iteration": per_iteration,
"runner_final": final,
"runner_reported_check": reported,
"extraction_matches_runner": (
None if reported is None or extracted is None
else reported.rstrip("\r") == extracted.rstrip("\r")
),
"cross_check": cross,
"runner_log": run["runner_log"],
}
# --------------------------------------------------------------------------
# Selftest
# --------------------------------------------------------------------------
FIXTURES = [
# (what the wizard emitted, expected primary class)
("no check here at all\n", "MISSING"),
("CHECK:\n", "EMPTY"),
("CHECK: \n", "EMPTY"),
("CHECK: true\n", "TAUTOLOGY"),
("CHECK: :\n", "TAUTOLOGY"),
("CHECK: exit 0\n", "TAUTOLOGY"),
("CHECK: echo ok\n", "TAUTOLOGY"),
("CHECK: echo \"all good\"\n", "TAUTOLOGY"),
("CHECK: sleep 5\n", "TAUTOLOGY"),
("check: true\n", "TAUTOLOGY"), # runner's grep is -i
("CHECK: python3 x.py && echo OK\n", "SOUND"),
("CHECK: python3 x.py && echo PASS || echo FAIL\n", "CATCH_ALL"),
("CHECK: python3 x.py || true\n", "CATCH_ALL"),
("CHECK: python3 x.py || :\n", "CATCH_ALL"),
("CHECK: python3 x.py || exit 0\n", "CATCH_ALL"),
("CHECK: python3 x.py; echo done\n", "CATCH_ALL"),
("CHECK: python3 x.py; true\n", "CATCH_ALL"),
("CHECK: test -s f && grep -c '###' f | grep -qx '8' && echo PASS || echo FAIL\n",
"CATCH_ALL"),
("CHECK: test \"$(python3 wc.py <<< \"a b c\")\" = \"3\" && echo PASSED || echo FAILED\n",
"CATCH_ALL"),
("CHECK: python3 /tmp/verify_dl.py\r\n", "TRAILING_CR"),
("CHECK: cd /a && PYTHONPATH=/a python /a/verify_chunker.py\r\n", "TRAILING_CR"),
("CHECK: python3 /tmp/v.py \u2014 re-reads the JSON, asserts counts (Ran it: PASS, exit 0.)\n",
"PROSE_TAIL"),
("CHECK: python3 v.py -- re-reads the JSON\n", "PROSE_TAIL"),
("CHECK: cd /x && python _verify.py && python test_a.py && python test_b.py\n", "SOUND"),
("CHECK: grep -q ralph-green /root/ralph-smoke.txt\n", "SOUND"),
("CHECK: sha256sum a.tar.gz | grep -q 4844c5fc\n", "SOUND"),
("CHECK: python3 tools/x.py --selftest | tail -1 | grep -q 'cells, 0 failed'\n", "SOUND"),
("CHECK: python3 -c \"print(1)\"\n", "SOUND"), # paren not preceded by space
("CHECK: python3 x.py --json /tmp/o.json\n", "SOUND"), # -- flag, not " -- "
("CHECK: python3 v.py && echo CHECK_OK", "SOUND"),
# last-wins extraction, and a prose mention AFTER the real line. The second
# fixture is sol's real shape (2026-10-16 16:57Z log, line 1335): a line that
# begins with an indentation then CHECK: matches the runner's grep, so prose
# written after the command becomes the graded line. Prose that does not
# itself start with CHECK: (no leading-whitespace CHECK:) is not matched and
# the earlier real line survives -- both directions are pinned here.
("CHECK: true\nCHECK: python3 x.py\n", "SOUND"),
("CHECK: python3 x.py\n\nA CHECK: line is required.\n", "SOUND"),
("CHECK: python3 x.py\n\n CHECK: line \u2014 one shell command that verifies "
"your work and exits 0\n", "PROSE_TAIL"),
]
EXTRACT_FIXTURES = [
(" CHECK: cmd here\n", "cmd here"),
("CHECK: cmd2\r\n", "cmd2\r"),
("no check\n", None),
("check: lower\n", "lower"),
]
def selftest(verbose: bool) -> int:
cells = 0
failed = 0
for text, want in FIXTURES:
cells += 1
got, faults, _ = classify(extract_check(text))
if got != want:
failed += 1
print(f" cell {cells}: want {want} got {got} <- {text!r}")
elif verbose:
print(f" cell {cells}: {got:11s} {text.strip()!r}")
for text, want in EXTRACT_FIXTURES:
cells += 1
got = extract_check(text)
if got != want:
failed += 1
print(f" cell {cells}: extract want {want!r} got {got!r} <- {text!r}")
elif verbose:
print(f" cell {cells}: extract {got!r}")
print(f"selftest: {cells} cells, {cells - failed} passed, {failed} failed")
return 1 if failed else 0
# --------------------------------------------------------------------------
# Report
# --------------------------------------------------------------------------
def print_rules():
print("RULES (textual; the command is never executed by this tool)")
print(" MISSING no line matches ^\\s*CHECK:")
print(" EMPTY a CHECK line with nothing after the colon")
print(" TAUTOLOGY whole command is true/: /exit 0/echo/printf/sleep")
print(" CATCH_ALL a || arm or terminal ; statement that exits 0 regardless")
print(" TRAILING_CR a literal carriage return survives at end of line")
print(" PROSE_TAIL prose follows the command (CANDIDATE, cut point printed)")
print(" SOUND none of the above -- not a promise about its meaning")
def report(runs: list[dict], as_json: str | None, quiet: bool,
unlogged: int = 0) -> int:
graded = [grade_run(r) for r in runs]
graded.sort(key=lambda r: (r["stamp"], r["wizard"]), reverse=True)
summary = {"runs_on_wizards": len(graded), "green": 0, "exhausted": 0,
"unknown": 0, "no_check_ever": 0, "classes": {},
"green_by_construction": 0, "green_despite_formatting": 0,
"extraction_mismatches": 0, "runs_without_runner_log": unlogged}
for g in graded:
if g["runner_final"] == "GREEN":
summary["green"] += 1
elif g["runner_final"] == "EXHAUSTED":
summary["exhausted"] += 1
else:
summary["unknown"] += 1
if g["class"] == "MISSING":
summary["no_check_ever"] += 1
summary["classes"][g["class"]] = summary["classes"].get(g["class"], 0) + 1
if g["cross_check"] == "GREEN_BY_CONSTRUCTION":
summary["green_by_construction"] += 1
elif g["cross_check"] == "GREEN_DESPITE_FORMATTING":
summary["green_despite_formatting"] += 1
if g["extraction_matches_runner"] is False:
summary["extraction_mismatches"] += 1
if not quiet:
print_rules()
stamp = None
for g in graded:
if g["stamp"] != stamp:
stamp = g["stamp"]
print(f"\nRUN {stamp}")
print(f" {'wizard':8s} {'it':>3s} {'class':12s} {'runner':10s} "
f"{'cross-check':24s} check")
chk = g["check"]
shown = "(none)" if chk is None else chk.rstrip("\r")
shown = shown if len(shown) <= 68 else shown[:65] + "..."
print(f" {g['wizard']:8s} {g['iterations']:3d} {g['class']:12s} "
f"{g['runner_final']:10s} {g['cross_check']:24s} {shown}")
if len(g["faults"]) > 1:
print(f" also: {', '.join(f for f in g['faults'] if f != g['class'])}")
if g["extraction_matches_runner"] is False:
print(" NOTE: re-extraction differs from what the runner echoed")
print("\nSUMMARY")
for k, v in summary.items():
print(f" {k}: {v}")
if as_json:
payload = {"tool": "check_lint.py",
"generated_utc": _dt.datetime.now(_dt.timezone.utc)
.strftime("%Y-%m-%dT%H:%M:%SZ"),
"ralph_dir": os.path.dirname(graded[0]["runner_log"])
if graded and graded[0]["runner_log"] else None,
"runs": graded, "summary": summary}
with open(as_json, "w", encoding="utf-8") as fh:
json.dump(payload, fh, indent=2, sort_keys=False)
fh.write("\n")
if not quiet:
print(f"\njson -> {as_json}")
return 0
def main(argv=None) -> int:
ap = argparse.ArgumentParser(
description="Grade Ralph-loop CHECK lines. Read-only: never executes a CHECK.")
g = ap.add_mutually_exclusive_group()
g.add_argument("--selftest", action="store_true",
help="grade the built-in fixtures")
g.add_argument("--ralph-dir", metavar="DIR", default=None,
help="grade every run in a ralph-loop output directory")
g.add_argument("--log", metavar="FILE", default=None,
help="extract and grade the CHECK in one log file")
ap.add_argument("--check", metavar="CMD", default=None,
help="grade this command string directly")
ap.add_argument("--json", metavar="FILE", default=None)
ap.add_argument("--quiet", action="store_true")
ap.add_argument("--verbose", action="store_true")
ap.add_argument("--all", action="store_true",
help="include runs that have no runner .log (no verdict to check)")
ap.add_argument("--fail-on-fault", action="store_true",
help="exit 1 if any graded CHECK is not SOUND")
a = ap.parse_args(argv)
if a.selftest:
return selftest(a.verbose)
if a.check is not None:
primary, faults, detail = classify(a.check)
if not a.quiet:
print(f"{primary}: {a.check!r}")
if faults:
print(f" faults: {', '.join(faults)}")
if detail:
print(f" detail: {json.dumps(detail)}")
if a.json:
with open(a.json, "w", encoding="utf-8") as fh:
json.dump({"check": a.check, "class": primary, "faults": faults,
"detail": detail}, fh, indent=2)
fh.write("\n")
return 1 if (a.fail_on_fault and primary != "SOUND") else 0
if a.log:
with open(a.log, encoding="utf-8", errors="replace", newline="") as fh:
text = fh.read()
cmd = extract_check(text)
primary, faults, detail = classify(cmd)
if not a.quiet:
print(f"{a.log}: {primary}")
print(f" extracted: {cmd!r}")
if faults:
print(f" faults: {', '.join(faults)}")
if detail:
print(f" detail: {json.dumps(detail)}")
return 1 if (a.fail_on_fault and primary != "SOUND") else 0
ralph_dir = a.ralph_dir or "/root/.hermes/wizard-outputs/ralph"
runs, unlogged = read_ralph_dir(ralph_dir, include_unlogged=a.all)
if not runs:
print(f"no iter logs in {ralph_dir}")
return 0 if not a.fail_on_fault else 1
rc = report(runs, a.json, a.quiet, unlogged)
if a.fail_on_fault:
graded = [grade_run(r) for r in runs]
return 1 if any(g["class"] != "SOUND" for g in graded) else 0
return rc
if __name__ == "__main__":
sys.exit(main())