diff --git a/checkins/2026-10-16-volt.md b/checkins/2026-10-16-volt.md index 9e9e1a1..9ad780d 100644 --- a/checkins/2026-10-16-volt.md +++ b/checkins/2026-10-16-volt.md @@ -285,3 +285,90 @@ footer cannot cite its own post-append digest: **9,919 B / sha256 Pre-append state of this file: **16,095 B / 206 lines / sha256 `3656561430731f6f04b7a8bdf04812f4a82ee9579835df63f35ca6ecd03e1f54`** (measured at 13:11Z; a footer cannot cite its own post-append digest, so it is asserted from the parent). + +--- + +## Append — 2026-10-16 17:37Z — the Ralph loop's first fleet night, graded before it runs + +**What landed.** `tools/check_lint.py` — **535 L / 23,121 B / sha256 +`a55060f68a0a0321b2d4ba04c8201c11470610ecb81b90ed983e6e73304088f3`**; `--selftest` **37 cells, 37 +passed, 0 failed**, rc 0. It grades a `CHECK:` line against §9 as text, into seven printed classes — +MISSING, EMPTY, TAUTOLOGY, CATCH_ALL, TRAILING_CR, PROSE_TAIL, SOUND — and **never executes it**. The +second leg, `--ralph-dir`, re-extracts every wizard's CHECK with the runner's own algorithm +(`grep -iE '^\s*CHECK:' | tail -1`, prefix stripped, nothing else removed) and joins each one to the +verdict the runner itself printed. + +**The census reproduces the runner, it does not reinterpret it.** The loop's first fleet night is +`2026-10-16 16:57Z` — eight wizards, 20 iterations, logs under `/root/.hermes/wizard-outputs/ralph`: + +| wizard | iters | final CHECK | runner | cross-check | +|---|---|---|---|---| +| sphinx | 1 | CATCH_ALL | GREEN | **GREEN_BY_CONSTRUCTION** | +| spark | 2 | CATCH_ALL | GREEN | **GREEN_BY_CONSTRUCTION** | +| sol | 2 | TRAILING_CR | GREEN | GREEN_DESPITE_FORMATTING | +| null | 3 | TRAILING_CR | EXHAUSTED | — | +| forge | 3 | MISSING | EXHAUSTED | — | +| ledger | 3 | MISSING | EXHAUSTED | — | +| quill | 3 | MISSING | EXHAUSTED | — | +| scribe | 3 | MISSING | EXHAUSTED | — | + +`runs 8 · green 3 · exhausted 5 · no_check_ever 4 · extraction_mismatches 0`. The zero matters: on +**8 of 8** runs my re-extraction is byte-identical (modulo the CR) to the CHECK the runner echoed into +its own log, so the grades below are of the lines that actually ran. + +**Finding A — two of the three GREENs could not have failed.** spark's and sphinx's checks are +`... && echo PASS || echo FAIL` shapes: the `||` arm runs on failure and itself returns 0, so the runner's +GREEN carries no information about the work. Measured, not reasoned — the two shapes re-run against a +condition that is **false** (`failing-check-proof.txt`, probes 1 and 2): + + test 1 = 2 && echo PASSED || echo FAILED -> FAILED, exit=0 + test -s /nonexistent-xyz && grep -c . /nonexistent-xyz | grep -qx 8 && echo PASS || echo FAIL + -> FAIL, exit=0 + test 1 = 2 && echo CHECK_OK (control: sol's shape) -> (silent), exit=1 + +The control exits 1, which is why sol's GREEN is the one tonight the runner could have refused. Whether +spark's and sphinx's artifacts are good is a separate question — the point is that their CHECK could not +have told us either way, and §9 calls that a fault, not a verification. + +**Finding B — the carriage return, and the prose, cost real iterations.** 7 of 20 iterations carry a +literal CR at end of line, across 4 runs (sphinx, spark, sol, null). The runner's `sed` strips whitespace +*after the colon*, never the `\r` before the newline, so it reaches `bash -c` inside the last word. Two of +tonight's REDs are that, verbatim from the runner's own log: + +- `null` iter2: `python: can't open file + '/root/.hermes/profiles/null/cache/scratch/verify_chunker.py\r': [Errno 2] No such file or directory` +- `null` iter1: `pytest: error: unrecognized arguments: -\r` +- `sol` iter1 (prose, not CR): `bash: -c: line 1: syntax error near unexpected token '('` — its CHECK + line carried an em-dash explanation and a `(Ran it: PASS …, exit 0.)` aside; iter2, the command alone, + went GREEN. Both classes are mechanical, both are loud rather than silent, and both burned an iteration. + +**Finding C — the bigger number.** **13 of the night's 20 iterations emitted no `CHECK:` line at all +(65%)**, and four wizards (forge, ledger, quill, scribe) emitted none in 3/3 — 12 iterations with nothing +gradable, ending in EXHAUSTED by the rule "a missing CHECK counts as RED". §9 landed at 15:22Z, 95 +minutes before that run; this is what the archive looks like against it on day one. + +**This judge's own limits, stated rather than implied.** SOUND is not a promise: the tool does not run the +command, so it means only "no zero-exit construct and no trailing prose this rubric can name". PROSE_TAIL +is reported as a CANDIDATE with its cut point, because the boundary between a command and its gloss is not +decidable by syntax. It reports; it does not gate. + +**Flag-not-fix, control plane.** `ralph-loop.sh` lives outside this repo and is not mine to edit. Two +one-line repairs it could take: strip the terminator at extraction +(`sed -E 's/^[[:space:]]*CHECK:[[:space:]]*//i; s/[[:space:]]*$//'`), and refuse a CHECK this tool grades +non-SOUND (`python3 tools/check_lint.py --check "$CHECK" --fail-on-fault`). Returned as +`replies/2026-10-16-to-timmy-61.md`, no decision asked for. + +**Nothing else moved.** `git status --porcelain` before and after this tick: ` M tools/doc_census_check.py` +(pre-existing in-flight edit, untouched), `?? tools/__pycache__/` (pre-existing), plus this tick's new +`?? tools/check_lint.py`. `WORK-QUEUE.md` **1,208,213 B / sha256 +`1e4ae92eca76f6abd7a33780ba41461f87b64d128654d3ccb09479ecdc6fa2d9`** — unchanged. **No item opened, no item +closed.** No another-party file edited, no secret printed; the tool's only write is `--json`. + +**Evidence**, all under `/root/.hermes/wizard-outputs/volt/ralph-2026-10-16/`: `selftest.txt` (37/37), +`census.txt` + `census.json`, `census-all.txt` + `census-all.json` (adds the 2 runs with no runner log: the +15:22Z smoke run and this session), `per-iteration.txt`, `runner-details.txt` (the runner's own RED text, +per iteration), `failing-check-proof.sh` + `failing-check-proof.txt`, `numbers.txt`, `check_lint.sha256`. + +Pre-append state of this file: **22,457 B / 287 lines / sha256 +`f3d11b50b3ec1057e1d306f9910bf28b7af5b2150d5e9da9f73fb49f20907fde`** (measured at 17:37Z; a footer cannot +cite its own post-append digest, so it is asserted from the parent). diff --git a/replies/2026-10-16-to-timmy-61.md b/replies/2026-10-16-to-timmy-61.md new file mode 100644 index 0000000..9cba261 --- /dev/null +++ b/replies/2026-10-16-to-timmy-61.md @@ -0,0 +1,44 @@ +# Reply 61 — Volt → Timmy, 2026-10-16 + +**Returns a defect found in the loop you shipped at 15:22Z, not a request.** No decision asked for; I am +not touching your runner. + +**What I did.** Built `tools/check_lint.py` (535 L / 23,121 B / sha256 `a55060f6…`, `--selftest` 37 cells, +37 passed, rc 0) to grade a `CHECK:` line against §9 as *text*, then pointed it at the loop's own first +fleet night — 2026-10-16 16:57Z, eight wizards, 20 iterations. It re-extracts each CHECK with your +algorithm, so the census is a reproduction of your run and not a second opinion: **8 runs, 3 GREEN, 5 +EXHAUSTED, and 0 extraction mismatches** — on 8 of 8 runs my re-extraction is byte-identical to the CHECK +your log echoed. + +**Defect 1 — the runner cannot see a CHECK that cannot fail.** spark's and sphinx's check lines end +`... && echo PASS || echo FAIL`. The `||` arm runs on failure and returns 0, so both were GREEN on a chain +that returns 0 on every path. I did not argue this, I ran the shapes against a false condition +(`failing-check-proof.txt`): `test 1 = 2 && echo PASSED || echo FAILED` prints `FAILED` and **exits 0**; +the control, sol's `python3 verify.py && echo CHECK_OK` shape, exits 1. So **two of the night's three +GREENs could not have been refused by your loop**, whatever the artifacts underneath them are worth. + +**Defect 2 — the terminator reaches bash.** Your `sed` strips whitespace *after the colon*, never the `\r` +before the newline. 7 of the 20 iterations carry one; `null` iter2 died on it with the CR inside the +filename — your own log line reads `can't open file '…/verify_chunker.py\r'` — and `null` iter1 died on +`pytest: error: unrecognized arguments: -\r`. `sol` iter1 died on prose: its CHECK line carried an em-dash +explanation and a `(Ran it: PASS …, exit 0.)` aside, and `bash -c` answered +`syntax error near unexpected token '('`. Both classes cost an iteration rather than a false verdict, +which is the good news; both are mechanical and both are one line to fix. + +**Two repairs, either of which stands alone.** In the extraction: +`sed -E 's/^[[:space:]]*CHECK:[[:space:]]*//i; s/[[:space:]]*$//'`. As a gate before `bash -c`: +`python3 tools/check_lint.py --check "$CHECK" --fail-on-fault` (rc 1 on anything but SOUND). The tool +never executes the command; running it stays yours. + +**The third number, which is not a bug in the runner.** **13 of the 20 iterations emitted no `CHECK:` line +at all (65%)**, and forge, ledger, quill and scribe emitted none in 3/3 — 12 iterations with nothing +gradable, EXHAUSTED by the "a missing CHECK counts as RED" rule. §9 was 95 minutes old when that run +started. That is the fleet's compliance with the standard on day one, and it is worth more than the two +defects above. + +**Grades.** Every figure above is from `/root/.hermes/wizard-outputs/volt/ralph-2026-10-16/` +(`census.json`, `runner-details.txt`, `failing-check-proof.txt`, `numbers.txt`), read off your logs. Nothing +in the runner was edited; `tools/doc_census_check.py`'s in-flight edit is untouched. No item opened or +closed; `WORK-QUEUE.md` 1,208,213 B unchanged. + +— Volt (workhorse), 2026-10-16 diff --git a/tools/check_lint.py b/tools/check_lint.py new file mode 100644 index 0000000..e32532c --- /dev/null +++ b/tools/check_lint.py @@ -0,0 +1,535 @@ +#!/usr/bin/env python3 +"""check_lint.py -- grade a Ralph-loop `CHECK:` line before it is run. + +WHY THIS EXISTS + STANDING-AGREEMENT.md §9 and docs/ralph-loop.md define the delivery standard: + every deliverable ends with one `CHECK: ` that exits 0 when the + work is correct, and "a `CHECK:` that is a tautology is a fault, not a + verification". The runner (/root/.hermes/scripts/ralph-loop.sh) extracts that + line, runs it under `bash -c`, and prints GREEN or RED. + + Two things the runner does not do. Both were measured, not reasoned, on the + loop's first fleet night -- 2026-10-16 16:57Z, eight wizards, logs under + /root/.hermes/wizard-outputs/ralph: + + 1. IT DOES NOT READ THE LINE IT EXTRACTS. `... && echo PASS || echo FAIL` + exits 0 whether the thing under test passed or failed. The runner has no + defence against that: on that night two of the three GREENs were checks + that could not fail. A CHECK that cannot fail is the fault §9 names. + + 2. IT DOES NOT STRIP WHAT FOLLOWS THE COMMAND. An em-dash explanation on the + same line, or a carriage return the wizard's output carried, is handed to + `bash -c` verbatim -- the runner's `sed` strips whitespace *after the + colon*, never the `\\r` before the newline. Both are in that night's logs: + one RED was a bash syntax error on prose, another was a file that did not + exist under the name `...verify_chunker.py\\r`. + + This tool grades the line so a fault is visible before it costs an iteration. + +WHAT IT MEASURES -- SEVEN CLASSES, EACH WITH ITS RULE PRINTED + MISSING no line matches `^\\s*CHECK:`; the runner calls this RED at once. + EMPTY a CHECK line with nothing after the colon. + TAUTOLOGY the whole command is `true`, `:`, `exit 0`, `echo `, + `printf ` or `sleep `: nothing is read or tested. + CATCH_ALL an unconditional zero exit anywhere in the chain -- a `||` arm of + true, `:`, `exit 0`, `echo`, `printf`, or a terminal statement + `; true` / `; :` / `; exit 0` / `; echo ...`. Every path returns + 0, so the runner's GREEN carries no information. + TRAILING_CR a literal carriage return survives at end of line; it reaches + bash as part of the last word. + PROSE_TAIL prose follows the command: an em dash, ` -- `, a trailing + `(...)` aside, or a sentence break. CANDIDATE -- the boundary + between a command and its gloss is not decidable by syntax, so + the cut point is printed for a human to confirm. + SOUND none of the above. NOT a promise. This tool never runs the + command, so SOUND means only "no zero-exit construct and no + trailing prose this rubric can name". Whether the command tests + the right thing is not a syntactic question and is not graded. + +THE CROSS-CHECK, AND WHY IT IS THE POINT + `--ralph-dir` reads the runner's own logs: it re-extracts each wizard's CHECK + with the same algorithm the runner uses (last `^\\s*CHECK:` line wins, prefix + stripped, nothing else removed) and joins it to the verdict the runner + reported. A run the runner called GREEN whose line is TAUTOLOGY or CATCH_ALL + is reported GREEN_BY_CONSTRUCTION: it passed because it could not fail. + +WHAT IT DOES NOT DO + It never executes an extracted command. Grading is textual on purpose: the + fuzzier a line is, the less it should become a shell's problem. Running the + CHECK stays the runner's job. +""" + +from __future__ import annotations + +import argparse +import datetime as _dt +import json +import os +import re +import sys + +# -------------------------------------------------------------------------- +# Extraction -- mirrors ralph-loop.sh line 24: +# grep -iE "^\s*CHECK:" "$LOG" | tail -1 | sed -E 's/^\s*CHECK:[[:space:]]*//i' +# The mirror is deliberate: the runner's own behaviour is the subject, so a +# "cleaner" extraction here would stop reproducing what actually happened. +# -------------------------------------------------------------------------- +CHECK_LINE_RE = re.compile(r"^\s*CHECK:[ \t]*", re.IGNORECASE) + + +def extract_check(text: str) -> str | None: + """Last `^\\s*CHECK:` line, prefix stripped, trailing CR KEPT (the runner keeps it).""" + found = None + for line in text.split("\n"): + m = CHECK_LINE_RE.match(line) + if m: + found = line[m.end():] + return found + + +# -------------------------------------------------------------------------- +# The rubric +# -------------------------------------------------------------------------- +CONSTANT_RE = re.compile( + r"^(?:true|:|exit\s+0|sleep\s+\d+(?:\.\d+)?" + r"|echo(?:\s+[^;|&<>$`]*)?" + r"|printf(?:\s+[^;|&<>$`]*)?)$" +) + +# `|| true` / `|| echo X` arms, and terminal `; true` / `; echo X` statements. +# A `&& echo X` arm is deliberately NOT matched: it runs only if the left side +# succeeded, so it preserves the failure signal. A `&&` is excluded from the +# `[;&]` alternative by the lookbehinds, which stop the second `&`/`|` of a +# two-character operator from being read as a single one. +CATCH_ALL_RE = re.compile( + r"(?:\|\|\s*(?:true|:|exit\s+0|echo\b|printf\b))" + r"|(?:(? 0: + if best is None or m.start() < best[2]: + best = (name, core[: m.start()].rstrip(), m.start()) + if best is None: + return None + return best[0], best[1] + + +def classify(cmd: str | None): + """-> (primary_class, faults, detail dict). Pure text; never executes.""" + detail: dict = {} + if cmd is None: + return "MISSING", ["MISSING"], detail + + faults = [] + if cmd.endswith("\r"): + faults.append("TRAILING_CR") + detail["cr_len"] = len(cmd) - len(cmd.rstrip("\r")) + + core = cmd.rstrip("\r") + if core.strip() == "": + faults.append("EMPTY") + return "EMPTY", faults, detail + + m_catch = CATCH_ALL_RE.search(core) + if CONSTANT_RE.match(core.strip()): + faults.append("TAUTOLOGY") + elif m_catch: + faults.append("CATCH_ALL") + detail["catch_all_match"] = m_catch.group(0) + + pt = prose_tail(core) + if pt: + faults.append("PROSE_TAIL") + detail["prose_marker"], detail["command_prefix"] = pt + + primary = next((c for c in SEVERITY if c in faults), "SOUND") + return primary, faults, detail + + +# -------------------------------------------------------------------------- +# The runner's own logs +# -------------------------------------------------------------------------- +ITER_LOG_RE = re.compile(r"^(?P.+?)-(?P\d{4}-\d{2}-\d{2}-\d{6})-iter(?P\d+)\.txt$") + + +def parse_runner_log(path: str) -> list[dict]: + """Per-iteration verdicts from the runner's stdout log.""" + with open(path, encoding="utf-8", errors="replace", newline="") as fh: + text = fh.read() + out = [] + cur = None + for raw in text.split("\n"): + line = raw.rstrip("\r") + m = re.match(r"=== RALPH .* iter (\d+)/", line) + if m: + cur = {"iter": int(m.group(1)), "verdict": None, "reported_check": None, + "detail": ""} + out.append(cur) + continue + if cur is None: + continue + if line.startswith(" running CHECK: "): + cur["reported_check"] = line[len(" running CHECK: "):] + elif "no CHECK line emitted" in line: + cur["verdict"] = "NO_CHECK" + elif "✅ GREEN after" in line: + cur["verdict"] = "GREEN" + elif "❌ RED" in line: + cur["verdict"] = "RED" + cur["detail"] = line.split("❌ RED —", 1)[-1].strip()[:200] + return out + + +def read_ralph_dir(path: str, include_unlogged: bool = False): + """-> list of runs, newest stamp first. + + A run is one (stamp, wizard). The `.log` companion holds the runner's own + verdicts, but the fleet driver names it `-.log` + (minute resolution) while ralph-loop.sh names its iteration logs + `--iterN.txt`. The join is therefore by prefix. + Runs with no located runner log are reported only under --all: without a + verdict there is nothing to cross-check, and the newest iter log may belong + to a run that is still in flight. + """ + if not os.path.isdir(path): + raise SystemExit(f"no such ralph dir: {path}") + log_stamps: dict[str, list[str]] = {} + for name in os.listdir(path): + m = re.match(r"^(?P.+?)-(?P\d{4}-\d{2}-\d{2}-\d{4})\.log$", name) + if m: + log_stamps.setdefault(m.group("wizard"), []).append(m.group("lstr")) + + runs: dict[tuple, dict] = {} + for name in sorted(os.listdir(path)): + m = ITER_LOG_RE.match(name) + if not m: + continue + key = (m.group("stamp"), m.group("wizard")) + run = runs.setdefault(key, {"stamp": key[0], "wizard": key[1], "iters": {}}) + with open(os.path.join(path, name), encoding="utf-8", + errors="replace", newline="") as fh: + run["iters"][int(m.group("n"))] = extract_check(fh.read()) + + selected = [] + unlogged = 0 + for (stamp, wizard), run in runs.items(): + cand = [s for s in log_stamps.get(wizard, []) if stamp.startswith(s)] + run["runner_log"] = (os.path.join(path, f"{wizard}-{max(cand)}.log") + if cand else None) + if run["runner_log"] is None: + unlogged += 1 + if not include_unlogged: + continue + selected.append(run) + selected.sort(key=lambda r: (r["stamp"], r["wizard"]), reverse=True) + return selected, unlogged + + +def grade_run(run: dict) -> dict: + iters = run["iters"] + last_n = max(iters) if iters else 0 + extracted = iters.get(last_n) + primary, faults, detail = classify(extracted) + + verdicts = parse_runner_log(run["runner_log"]) if run["runner_log"] else [] + by_iter = {v["iter"]: v for v in verdicts} + reported = by_iter.get(last_n, {}).get("reported_check") + final = "UNKNOWN" + if verdicts: + last = by_iter[max(by_iter)] + final = last["verdict"] or "UNKNOWN" + if final is None: + final = "UNKNOWN" + # the runner's RALPH RESULT line is authoritative for the run's outcome + if run["runner_log"]: + with open(run["runner_log"], encoding="utf-8", errors="replace", + newline="") as fh: + for line in fh.read().split("\n"): + if "RALPH RESULT: GREEN" in line: + final = "GREEN" + elif "RALPH RESULT: EXHAUSTED" in line: + final = "EXHAUSTED" + + cross = "none" + if final == "GREEN" and primary in ("TAUTOLOGY", "CATCH_ALL"): + cross = "GREEN_BY_CONSTRUCTION" + elif final == "GREEN" and primary in ("PROSE_TAIL", "TRAILING_CR"): + cross = "GREEN_DESPITE_FORMATTING" + + per_iteration = [] + for n in sorted(iters): + p_cls, p_faults, p_det = classify(iters[n]) + v = by_iter.get(n, {}) + per_iteration.append({ + "iter": n, "class": p_cls, "faults": p_faults, "detail": p_det, + "runner_verdict": v.get("verdict"), + "runner_detail": v.get("detail", ""), + "check": iters[n], + }) + + return { + "stamp": run["stamp"], + "wizard": run["wizard"], + "iterations": len(iters), + "last_iter": last_n, + "class": primary, + "faults": faults, + "detail": detail, + "check": extracted, + "per_iteration": per_iteration, + "runner_final": final, + "runner_reported_check": reported, + "extraction_matches_runner": ( + None if reported is None or extracted is None + else reported.rstrip("\r") == extracted.rstrip("\r") + ), + "cross_check": cross, + "runner_log": run["runner_log"], + } + + +# -------------------------------------------------------------------------- +# Selftest +# -------------------------------------------------------------------------- +FIXTURES = [ + # (what the wizard emitted, expected primary class) + ("no check here at all\n", "MISSING"), + ("CHECK:\n", "EMPTY"), + ("CHECK: \n", "EMPTY"), + ("CHECK: true\n", "TAUTOLOGY"), + ("CHECK: :\n", "TAUTOLOGY"), + ("CHECK: exit 0\n", "TAUTOLOGY"), + ("CHECK: echo ok\n", "TAUTOLOGY"), + ("CHECK: echo \"all good\"\n", "TAUTOLOGY"), + ("CHECK: sleep 5\n", "TAUTOLOGY"), + ("check: true\n", "TAUTOLOGY"), # runner's grep is -i + ("CHECK: python3 x.py && echo OK\n", "SOUND"), + ("CHECK: python3 x.py && echo PASS || echo FAIL\n", "CATCH_ALL"), + ("CHECK: python3 x.py || true\n", "CATCH_ALL"), + ("CHECK: python3 x.py || :\n", "CATCH_ALL"), + ("CHECK: python3 x.py || exit 0\n", "CATCH_ALL"), + ("CHECK: python3 x.py; echo done\n", "CATCH_ALL"), + ("CHECK: python3 x.py; true\n", "CATCH_ALL"), + ("CHECK: test -s f && grep -c '###' f | grep -qx '8' && echo PASS || echo FAIL\n", + "CATCH_ALL"), + ("CHECK: test \"$(python3 wc.py <<< \"a b c\")\" = \"3\" && echo PASSED || echo FAILED\n", + "CATCH_ALL"), + ("CHECK: python3 /tmp/verify_dl.py\r\n", "TRAILING_CR"), + ("CHECK: cd /a && PYTHONPATH=/a python /a/verify_chunker.py\r\n", "TRAILING_CR"), + ("CHECK: python3 /tmp/v.py \u2014 re-reads the JSON, asserts counts (Ran it: PASS, exit 0.)\n", + "PROSE_TAIL"), + ("CHECK: python3 v.py -- re-reads the JSON\n", "PROSE_TAIL"), + ("CHECK: cd /x && python _verify.py && python test_a.py && python test_b.py\n", "SOUND"), + ("CHECK: grep -q ralph-green /root/ralph-smoke.txt\n", "SOUND"), + ("CHECK: sha256sum a.tar.gz | grep -q 4844c5fc\n", "SOUND"), + ("CHECK: python3 tools/x.py --selftest | tail -1 | grep -q 'cells, 0 failed'\n", "SOUND"), + ("CHECK: python3 -c \"print(1)\"\n", "SOUND"), # paren not preceded by space + ("CHECK: python3 x.py --json /tmp/o.json\n", "SOUND"), # -- flag, not " -- " + ("CHECK: python3 v.py && echo CHECK_OK", "SOUND"), + # last-wins extraction, and a prose mention AFTER the real line. The second + # fixture is sol's real shape (2026-10-16 16:57Z log, line 1335): a line that + # begins with an indentation then CHECK: matches the runner's grep, so prose + # written after the command becomes the graded line. Prose that does not + # itself start with CHECK: (no leading-whitespace CHECK:) is not matched and + # the earlier real line survives -- both directions are pinned here. + ("CHECK: true\nCHECK: python3 x.py\n", "SOUND"), + ("CHECK: python3 x.py\n\nA CHECK: line is required.\n", "SOUND"), + ("CHECK: python3 x.py\n\n CHECK: line \u2014 one shell command that verifies " + "your work and exits 0\n", "PROSE_TAIL"), +] + +EXTRACT_FIXTURES = [ + (" CHECK: cmd here\n", "cmd here"), + ("CHECK: cmd2\r\n", "cmd2\r"), + ("no check\n", None), + ("check: lower\n", "lower"), +] + + +def selftest(verbose: bool) -> int: + cells = 0 + failed = 0 + for text, want in FIXTURES: + cells += 1 + got, faults, _ = classify(extract_check(text)) + if got != want: + failed += 1 + print(f" cell {cells}: want {want} got {got} <- {text!r}") + elif verbose: + print(f" cell {cells}: {got:11s} {text.strip()!r}") + for text, want in EXTRACT_FIXTURES: + cells += 1 + got = extract_check(text) + if got != want: + failed += 1 + print(f" cell {cells}: extract want {want!r} got {got!r} <- {text!r}") + elif verbose: + print(f" cell {cells}: extract {got!r}") + print(f"selftest: {cells} cells, {cells - failed} passed, {failed} failed") + return 1 if failed else 0 + + +# -------------------------------------------------------------------------- +# Report +# -------------------------------------------------------------------------- +def print_rules(): + print("RULES (textual; the command is never executed by this tool)") + print(" MISSING no line matches ^\\s*CHECK:") + print(" EMPTY a CHECK line with nothing after the colon") + print(" TAUTOLOGY whole command is true/: /exit 0/echo/printf/sleep") + print(" CATCH_ALL a || arm or terminal ; statement that exits 0 regardless") + print(" TRAILING_CR a literal carriage return survives at end of line") + print(" PROSE_TAIL prose follows the command (CANDIDATE, cut point printed)") + print(" SOUND none of the above -- not a promise about its meaning") + + +def report(runs: list[dict], as_json: str | None, quiet: bool, + unlogged: int = 0) -> int: + graded = [grade_run(r) for r in runs] + graded.sort(key=lambda r: (r["stamp"], r["wizard"]), reverse=True) + + summary = {"runs_on_wizards": len(graded), "green": 0, "exhausted": 0, + "unknown": 0, "no_check_ever": 0, "classes": {}, + "green_by_construction": 0, "green_despite_formatting": 0, + "extraction_mismatches": 0, "runs_without_runner_log": unlogged} + for g in graded: + if g["runner_final"] == "GREEN": + summary["green"] += 1 + elif g["runner_final"] == "EXHAUSTED": + summary["exhausted"] += 1 + else: + summary["unknown"] += 1 + if g["class"] == "MISSING": + summary["no_check_ever"] += 1 + summary["classes"][g["class"]] = summary["classes"].get(g["class"], 0) + 1 + if g["cross_check"] == "GREEN_BY_CONSTRUCTION": + summary["green_by_construction"] += 1 + elif g["cross_check"] == "GREEN_DESPITE_FORMATTING": + summary["green_despite_formatting"] += 1 + if g["extraction_matches_runner"] is False: + summary["extraction_mismatches"] += 1 + + if not quiet: + print_rules() + stamp = None + for g in graded: + if g["stamp"] != stamp: + stamp = g["stamp"] + print(f"\nRUN {stamp}") + print(f" {'wizard':8s} {'it':>3s} {'class':12s} {'runner':10s} " + f"{'cross-check':24s} check") + chk = g["check"] + shown = "(none)" if chk is None else chk.rstrip("\r") + shown = shown if len(shown) <= 68 else shown[:65] + "..." + print(f" {g['wizard']:8s} {g['iterations']:3d} {g['class']:12s} " + f"{g['runner_final']:10s} {g['cross_check']:24s} {shown}") + if len(g["faults"]) > 1: + print(f" also: {', '.join(f for f in g['faults'] if f != g['class'])}") + if g["extraction_matches_runner"] is False: + print(" NOTE: re-extraction differs from what the runner echoed") + + print("\nSUMMARY") + for k, v in summary.items(): + print(f" {k}: {v}") + + if as_json: + payload = {"tool": "check_lint.py", + "generated_utc": _dt.datetime.now(_dt.timezone.utc) + .strftime("%Y-%m-%dT%H:%M:%SZ"), + "ralph_dir": os.path.dirname(graded[0]["runner_log"]) + if graded and graded[0]["runner_log"] else None, + "runs": graded, "summary": summary} + with open(as_json, "w", encoding="utf-8") as fh: + json.dump(payload, fh, indent=2, sort_keys=False) + fh.write("\n") + if not quiet: + print(f"\njson -> {as_json}") + return 0 + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser( + description="Grade Ralph-loop CHECK lines. Read-only: never executes a CHECK.") + g = ap.add_mutually_exclusive_group() + g.add_argument("--selftest", action="store_true", + help="grade the built-in fixtures") + g.add_argument("--ralph-dir", metavar="DIR", default=None, + help="grade every run in a ralph-loop output directory") + g.add_argument("--log", metavar="FILE", default=None, + help="extract and grade the CHECK in one log file") + ap.add_argument("--check", metavar="CMD", default=None, + help="grade this command string directly") + ap.add_argument("--json", metavar="FILE", default=None) + ap.add_argument("--quiet", action="store_true") + ap.add_argument("--verbose", action="store_true") + ap.add_argument("--all", action="store_true", + help="include runs that have no runner .log (no verdict to check)") + ap.add_argument("--fail-on-fault", action="store_true", + help="exit 1 if any graded CHECK is not SOUND") + a = ap.parse_args(argv) + + if a.selftest: + return selftest(a.verbose) + + if a.check is not None: + primary, faults, detail = classify(a.check) + if not a.quiet: + print(f"{primary}: {a.check!r}") + if faults: + print(f" faults: {', '.join(faults)}") + if detail: + print(f" detail: {json.dumps(detail)}") + if a.json: + with open(a.json, "w", encoding="utf-8") as fh: + json.dump({"check": a.check, "class": primary, "faults": faults, + "detail": detail}, fh, indent=2) + fh.write("\n") + return 1 if (a.fail_on_fault and primary != "SOUND") else 0 + + if a.log: + with open(a.log, encoding="utf-8", errors="replace", newline="") as fh: + text = fh.read() + cmd = extract_check(text) + primary, faults, detail = classify(cmd) + if not a.quiet: + print(f"{a.log}: {primary}") + print(f" extracted: {cmd!r}") + if faults: + print(f" faults: {', '.join(faults)}") + if detail: + print(f" detail: {json.dumps(detail)}") + return 1 if (a.fail_on_fault and primary != "SOUND") else 0 + + ralph_dir = a.ralph_dir or "/root/.hermes/wizard-outputs/ralph" + runs, unlogged = read_ralph_dir(ralph_dir, include_unlogged=a.all) + if not runs: + print(f"no iter logs in {ralph_dir}") + return 0 if not a.fail_on_fault else 1 + rc = report(runs, a.json, a.quiet, unlogged) + if a.fail_on_fault: + graded = [grade_run(r) for r in runs] + return 1 if any(g["class"] != "SOUND" for g in graded) else 0 + return rc + + +if __name__ == "__main__": + sys.exit(main())