sol: CHECK-2026-09-28 grades the tree it ships in, not the host (Q106)
This commit is contained in:
ledger committed
1 parent
a9bc1f7080
commit
062e2789c3
4 files changed
+326
-59
No files matched your search
@@ -13430,3 +13430,17 @@ m1’s nine: `test_no_pinned_root_paths_in_either_file`, `test_snippet_bare_inte
|
||||
**Not claimed:** that `hermes_tools` is unimportable inside a real Hermes session — it is unimportable from a plain `python3` on this host, measured, and neither verifier was run from inside the agent runtime; that `HERMES_HOME` always names a profile home — that is this install’s shape (`/root/.hermes/profiles/volt`), and the derivation falls back to the parent when the home does not sit under `profiles`, but a root literally named `profiles` would be mis-derived (unmeasured); that skills/plugins should be required — they were advisory before and stay advisory; that `wizard-work/spark/CHECK` now collects the new cells — it does not, it still grades artifacts by absolute path and runs no `test_*.py`, and that blindness is the Q109/Q120 family this item does not close; that either old file had no other defect — only the readings in (1) were measured; that Q107, Q120 or Q125 are reopened — different artifacts, untouched; that the stub-module method proves anything about the real package beyond importability and attribute presence — it proves the check *consults* the package, which is what the item asked. Scratch trees for (5): `/root/.hermes/profiles/volt/cache/scratch/q109-2620292/{control,m1,m2,m3,m4}`, one `.out` beside each. Files edited for this item: `wizard-work/spark/snippet.py`, `wizard-work/spark/verify_hermes_setup.sh`, `wizard-work/spark/test_env_verifiers.py` (new), `wizard-work/spark/CHECK-env-verifiers` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Nothing from `inbox/` is quoted and no other lane’s artifact was edited. **Report:** `/root/.hermes/wizard-outputs/volt/2026-10-29-q109.md`.
|
||||
|
||||
**Q117 — DONE (zeph, 2026-10-29; zeph lane by authorship; measured at `ebba640`, this record and `docs/queue-index.md` land in the commit above it).** The `KNOWN_PROBE` table now sits beside `KNOWN_FIXTURE` in `wizard-work/zeph/q60_enforce.py`, the probe value built by concatenation so no literal for it exists in the source (`grep -c "beae0b1f-9f2c"` over the file returns 0); the recovery leg skips probe values in both branches (general longer-string look and the exhausted-stem full-value look), so a stem whose only in-tree longer string is a probe value reads `UNRECOVERED-JUSTIFIED` with the value still in the tree. Cell 9 (Q96 exhausted-stem flip) is reworked to a non-probe synthetic value under the stem, because the probe-C value no longer counts as a recovery by design; new cell 11 `probe-value-not-a-recovery` owns the probe case: a tree holding the truncated stem plus the probe value reads `UNRECOVERED-JUSTIFIED` with the gate green, and with the table cleared the SAME tree flips to `RECOVERED-IN-TREE` at its site, table restored afterwards, so the cell is not vacuous. Evidence, all this host: `--selftest` 11 cells passed, 0 failed, rc 0 (10 at the Q121 landing); standalone tree both readings quoted in the check-in (with table `UNRECOVERED-JUSTIFIED`, without table `RECOVERED-IN-TREE beae0b1f-9f2c-4d1e-8a77-3c5b6e9d0a11 at n.md`); full tree at `ebba640` with these edits uncommitted: `OCC=132 DISTINCT=24 FILES=40`, `VERDICT=PASS`, rc 0, all three `beae0b1f` stems `UNRECOVERED-JUSTIFIED`, zero FAIL lines; `wizard-work/zeph/test_q60_absence.py` 5 tests OK after its count pin moved 10 to 11 (docstring line and assertion only, the pin's purpose unchanged); `tools/check_lint.py --check` SOUND on the module. Numbering note: Q117 was filed asking for "a new cell (10)", but Q121 landed first and took that number, so the probe cell is cell 11 and the done-means are unchanged. **Not claimed:** that the Q99 record was edited (it stands as dated fact; the table declares the provenance instead); that any other stem is affected (only `beae0b1f` carries an invented full value); that the narrowed Q96 stem rule is wrong (it is not -- this is provenance, one layer above it). Close call Timmy's.
|
||||
|
||||
**Q106 — DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest is doctored reads `VERDICT FALSE`, rc 1, over the clone's cells, where it used to read `VERDICT TRUE`, rc 0, over the host's.** The filed defect reproduces at `a9bc1f7` (`wizard-work/sol/CHECK-2026-09-28`, md5 `ff9a28a2103184ffb96d650503572a36`, `ROOT = "/root/troupe-standup"` at `:14`): a `--depth 5` clone reads green on its own manifest, and with one item flipped `archive_fix` → `app_pr` (aggregate rewritten to `{6,4,2,0}` so the only defect is provenance) the committed CHECK still prints `VERDICT TRUE (categories={'archive_fix': 7, 'app_pr': 3, 'general': 2, 'other': 0} table=12)` rc **0** — the host worktree's cells, from a clone that carries neither of those numbers.
|
||||
|
||||
**(1) The repair** (`wizard-work/sol/CHECK-2026-09-28`, 130 lines, md5 `c67de7923026db0dedc3c0fce6f31dc1`, `+104/−52`): `HERE = os.path.dirname(os.path.realpath(__file__))` and `DEFAULT_ROOT = os.path.dirname(os.path.dirname(HERE))`, so the subject is the tree the file sits in; `--root DIR` grades another tree on request; every run prints its provenance first (`CHECK-2026-09-28: subject root=<abs tree graded>`), so the reading states which tree it came from. Exit codes follow the repo tool contract — 0 verdict TRUE, 1 verdict FALSE, 2 could not grade — and the two `open()` calls that used to raise tracebacks now print `CANNOT GRADE -- <label> not readable at <path>` and exit 2. **Readings with the repair**, all in this clone unless named: green in the tree and byte-identical verdict line to the old file (`VERDICT TRUE (categories={'archive_fix': 7, 'app_pr': 3, 'general': 2, 'other': 0} table=12)`, rc 0); green from `cwd=/` (rc 0); green in the host worktree; `--root <poisoned copy>` → `VERDICT FALSE`, `categories {'archive_fix': 6, 'app_pr': 4, 'general': 2, 'other': 0} != expected {'archive_fix': 7, …}`, rc **1**; `--bogus` → argparse usage line, rc **2**; the manifest deleted from a copy → `CANNOT GRADE -- manifest not readable at <path>`, rc **2**, no traceback.
|
||||
|
||||
**(2) The cell that keeps it from regressing** — `wizard-work/sol/test_check_provenance.py` (new, 174 lines, md5 `96f61842030b87a3e49f59c435bce965`, 8 `TestCase` cells). It runs the CHECK as a subprocess and imports nothing from it, and it builds its own substrate under `tempfile` (a minimal tree: `inbox/seed-2026-09-26.json`, the manifest, the CHECK), so no cell can agree with the code by sharing its constants. Cells: tree-ships-green (and names the tree it lives in); poisoned-copy-is-red (rc 1, prints the copy's `6/4`, names the flipped item); clean-copy-green-from-a-foreign-cwd; `--root` grades the tree it names; no-host-pin (no `/root/` byte, no `ROOT = "` literal); usage-error-is-2; missing-manifest-is-2; and the falsification cell — the pin reinstated over the same poisoned tree reads `VERDICT TRUE` rc **0** while the repaired file beside it reads `VERDICT FALSE` rc **1**, so the poisoned-copy cell fails on the exact old defect rather than passing by construction.
|
||||
|
||||
**(3) Falsification, both directions.** Green: `python3 wizard-work/sol/test_check_provenance.py` → `Ran 8 tests … OK`, rc 0, from the repo root, from `wizard-work/sol/`, and from `/`. Red: the pre-repair blob (`git show HEAD:…`, md5 `ff9a28a2…`) put back beside the same cell → `Ran 8 tests … FAILED (failures=8)`, rc 1, in a `git archive` of the same rev — so the cells measure the defect, not the new file's prose.
|
||||
|
||||
**(4) Gate.** `python3 tools/verify.py --repo . --fast` in this tree, read before the edits and again over the landing tree (queue append + index render in place): `8 legs, 7 pass, 0 fault, 0 ungradable, 1 skipped (selftest-sweep: --fast)`, `gate: OK`. `python3 tools/check_lint.py --check` on this record's CHECK line: `SOUND`.
|
||||
|
||||
CHECK: python3 wizard-work/sol/test_check_provenance.py
|
||||
|
||||
**Not claimed:** that `wizard-work/sol/Triage-2026-09-28-classify.py` is repaired — it still pins `ROOT = "/root/troupe-standup"` at `:14` and mirrors to `/root/.hermes/wizard-outputs/sol`; it is the generator, it writes, and the CHECK's re-derivation is deliberately a separate implementation, so the generator's pin cannot launder this verdict — grading the generator is a separate item, not this one; that the CHECK's verdict semantics changed — the same six comparisons and the same rc-1 path are kept, and the only behavioural change beyond provenance is that a seed of the wrong shape is now `CANNOT GRADE` rc 2 instead of an `AssertionError` traceback; that the Q105/Q107 tail-line class is addressed — this CHECK's summary carries both denominators (`categories=… table=…`) and was already the honest shape; that the other sol gates (`CHECK`, `CHECK-triage-2026-10-29`) were graded — they were not run; that the host worktree was edited — it was not, and the two files land from a clean clone of `a9bc1f7`. Files edited for this item: `wizard-work/sol/CHECK-2026-09-28`, `wizard-work/sol/test_check_provenance.py` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Nothing from `inbox/` is quoted and no other lane's artifact was edited. Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q106.md`.
|
||||
+44
-17
@@ -4,7 +4,7 @@
|
||||
|
||||
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
|
||||
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 91 record blocks in all, 55 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 29 times.
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 92 record blocks in all, 56 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 29 times.
|
||||
|
||||
| # | id | kind | lines | bytes | role | title | first `Status:` |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -195,7 +195,8 @@
|
||||
| 185 | Q120 | record | 13400-13401 | 3580 | cont: DONE | **Q120 — DONE (volt, 2026-10-29; this record and `docs/queue-index.md` land in the commit above it; spark lane by authorship, unclaimed and taken).** The health check now fails. Before: `python3 … | -- |
|
||||
| 186 | Q121 | record | 13402-13403 | 5321 | cont: DONE | **Q121 — DONE (volt, 2026-10-29; zeph lane by authorship, unclaimed and taken; this record and `docs/queue-index.md` land in the commit above the head `5623170` the item was measured against). The … | -- |
|
||||
| 187 | Q109 | record | 13404-13431 | 7753 | cont: DONE | **Q109 — DONE (volt, 2026-10-29; spark lane by authorship, unclaimed and taken). The tool check measures the import now, and the shell script grades the tree it is handed. Both named readings … | -- |
|
||||
| 188 | Q117 | record | 13432-13432 | 2222 | cont: DONE | **Q117 — DONE (zeph, 2026-10-29; zeph lane by authorship; measured at `ebba640`, this record and `docs/queue-index.md` land in the commit above it).** The `KNOWN_PROBE` table now sits beside … | -- |
|
||||
| 188 | Q117 | record | 13432-13433 | 2223 | cont: DONE | **Q117 — DONE (zeph, 2026-10-29; zeph lane by authorship; measured at `ebba640`, this record and `docs/queue-index.md` land in the commit above it).** The `KNOWN_PROBE` table now sits beside … | -- |
|
||||
| 189 | Q106 | record | 13434-13446 | 5421 | cont: DONE | **Q106 — DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest … | -- |
|
||||
|
||||
## Ids with no `## ` heading
|
||||
|
||||
@@ -219,9 +220,9 @@ Opened as records inside another section. `line` is the record's own opening lin
|
||||
| Q102 | 12985 | 12985 | 9 | Q100 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q103 | 13010 | 13010 | 18 | Q102 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q104 | 13103 | 13103 | 18 | Q101 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q105 | 13135 | 13135 | 15 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q106 | 13171 | 13171 | 12 | Q105 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q107 | 13189 | 13189 | 16 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q105 | 13135 | 13135 | 16 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q106 | 13171 | 13171 | 13 | Q105 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q107 | 13189 | 13189 | 17 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q108 | 13267 | 13263 | 3 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q109 | 13270 | 13270 | 11 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q110 | 13272 | 13272 | 3 | Q109 | BULLET+BOLD | BOLD-LEAD |
|
||||
@@ -1421,17 +1422,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q109 \u2014 DONE (volt, 2026-10-29; spark lane by authorship, unclaimed and taken). The tool check measures the import now, and the shell script grades the tree it is handed. Both named readings \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 2222,
|
||||
"end_line": 13432,
|
||||
"bytes": 2223,
|
||||
"end_line": 13433,
|
||||
"id": "Q117",
|
||||
"kind": "record",
|
||||
"line": 13432,
|
||||
"lines": 1,
|
||||
"lines": 2,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q117 \u2014 DONE (zeph, 2026-10-29; zeph lane by authorship; measured at `ebba640`, this record and `docs/queue-index.md` land in the commit above it).** The `KNOWN_PROBE` table now sits beside \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 5421,
|
||||
"end_line": 13446,
|
||||
"id": "Q106",
|
||||
"kind": "record",
|
||||
"line": 13434,
|
||||
"lines": 13,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q106 \u2014 DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest \u2026"
|
||||
}
|
||||
],
|
||||
"ids_multi_section": {
|
||||
@@ -4001,17 +4015,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q109 \u2014 DONE (volt, 2026-10-29; spark lane by authorship, unclaimed and taken). The tool check measures the import now, and the shell script grades the tree it is handed. Both named readings \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 2222,
|
||||
"end_line": 13432,
|
||||
"bytes": 2223,
|
||||
"end_line": 13433,
|
||||
"id": "Q117",
|
||||
"kind": "record",
|
||||
"line": 13432,
|
||||
"lines": 1,
|
||||
"lines": 2,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q117 \u2014 DONE (zeph, 2026-10-29; zeph lane by authorship; measured at `ebba640`, this record and `docs/queue-index.md` land in the commit above it).** The `KNOWN_PROBE` table now sits beside \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 5421,
|
||||
"end_line": 13446,
|
||||
"id": "Q106",
|
||||
"kind": "record",
|
||||
"line": 13434,
|
||||
"lines": 13,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q106 \u2014 DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest \u2026"
|
||||
}
|
||||
],
|
||||
"nested": [
|
||||
@@ -4164,7 +4191,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q105",
|
||||
"line": 13135,
|
||||
"refs": 15,
|
||||
"refs": 16,
|
||||
"section": "Q104",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4173,7 +4200,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q106",
|
||||
"line": 13171,
|
||||
"refs": 12,
|
||||
"refs": 13,
|
||||
"section": "Q105",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4182,7 +4209,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q107",
|
||||
"line": 13189,
|
||||
"refs": 16,
|
||||
"refs": 17,
|
||||
"section": "Q106",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4345,9 +4372,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"end_line": 8,
|
||||
"line": 1
|
||||
},
|
||||
"queue_bytes": 2095631,
|
||||
"queue_lines": 13432,
|
||||
"queue_sha256": "cc261e2bd90f1c26f67d31204a6c50643ffe3606eaba2b60a9213c535be972e3",
|
||||
"queue_bytes": 2101053,
|
||||
"queue_lines": 13446,
|
||||
"queue_sha256": "0a700d5d26e320ac137967c200edcfe29947ae73b68a64dcabf83c101e809b62",
|
||||
"sections": [
|
||||
{
|
||||
"bytes": 3139,
|
||||
|
||||
@@ -4,21 +4,32 @@
|
||||
Exit 0 iff the committed manifest (Triage-2026-09-28-classify.json) matches a
|
||||
fresh, independent re-derivation of the classification straight from the seed's
|
||||
own fields (summary prefix), with counts and orderings identical. A tautology
|
||||
or a mismatch both exit nonzero. Run from any directory:
|
||||
python3 CHECK-2026-09-28
|
||||
(The repo's ralph-loop convention invokes the committed CHECK and reads its rc.)
|
||||
or a mismatch both exit nonzero.
|
||||
|
||||
The subject is resolved from this file's own location -- the tree the CHECK
|
||||
ships in -- never from a pinned absolute path, so a clone whose own manifest is
|
||||
doctored is graded as doctored. Q106 measured the old form, which pinned the
|
||||
host worktree's absolute path, printing VERDICT TRUE rc 0 over a poisoned clone.
|
||||
|
||||
python3 wizard-work/sol/CHECK-2026-09-28 # grade this tree
|
||||
python3 wizard-work/sol/CHECK-2026-09-28 --root DIR # grade another tree
|
||||
|
||||
Exit codes follow the repo's tool contract: 0 verdict TRUE, 1 verdict FALSE,
|
||||
2 could not grade (usage error, or an input the tree does not carry).
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
ROOT = "/root/troupe-standup"
|
||||
SEED = f"{ROOT}/inbox/seed-2026-09-26.json"
|
||||
MANIFEST = f"{ROOT}/wizard-work/sol/Triage-2026-09-28-classify.json"
|
||||
HERE = os.path.dirname(os.path.realpath(__file__))
|
||||
DEFAULT_ROOT = os.path.dirname(os.path.dirname(HERE))
|
||||
|
||||
# Independent re-implementation of the classifier (deliberately separate
|
||||
# expression shape from the builder that produced the manifest).
|
||||
CATS = ["archive_fix", "app_pr", "general", "other"]
|
||||
|
||||
|
||||
def expected_category(summary):
|
||||
s = (summary or "").strip()
|
||||
if s.startswith("Archive fix:"):
|
||||
@@ -29,50 +40,91 @@ def expected_category(summary):
|
||||
return "general"
|
||||
return "other"
|
||||
|
||||
seed = json.load(open(SEED))
|
||||
assert set(seed.keys()) == {"cutover", "decided", "exported_at", "pending", "source"}
|
||||
|
||||
expected_items = []
|
||||
for it in sorted(seed["pending"], key=lambda x: int(x["id"])):
|
||||
expected_items.append({
|
||||
"id": int(it["id"]),
|
||||
"category": expected_category(it["summary"]),
|
||||
"status": "table",
|
||||
})
|
||||
def read_json(path):
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
expected_counts = {"archive_fix": 0, "app_pr": 0, "general": 0, "other": 0}
|
||||
for it in expected_items:
|
||||
expected_counts[it["category"]] += 1
|
||||
|
||||
manifest = json.load(open(MANIFEST))
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(
|
||||
prog="CHECK-2026-09-28",
|
||||
description="Grade Triage-2026-09-28-classify.json against the seed it "
|
||||
"claims to derive from.",
|
||||
)
|
||||
ap.add_argument("--root", default=DEFAULT_ROOT,
|
||||
help="tree to grade (default: the tree this CHECK lives in)")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
errors = []
|
||||
root = os.path.abspath(args.root)
|
||||
seed_path = os.path.join(root, "inbox", "seed-2026-09-26.json")
|
||||
manifest_path = os.path.join(root, "wizard-work", "sol",
|
||||
"Triage-2026-09-28-classify.json")
|
||||
|
||||
# Structural keys that any genuine manifest carries (not a flake).
|
||||
for key in ("artifact", "seed", "generated_at", "lane", "summary", "decision_counts", "categories", "items"):
|
||||
if key not in manifest:
|
||||
errors.append(f"manifest missing key {key!r}")
|
||||
print(f"CHECK-2026-09-28: subject root={root}")
|
||||
|
||||
if [i["id"] for i in manifest.get("items", [])] != [i["id"] for i in expected_items]:
|
||||
errors.append("item id order differs from seed")
|
||||
for label, path in (("seed", seed_path), ("manifest", manifest_path)):
|
||||
if not os.path.isfile(path):
|
||||
print(f"CHECK-2026-09-28: CANNOT GRADE -- {label} not readable at {path}")
|
||||
return 2
|
||||
|
||||
if manifest.get("decision_counts") != {"approve": 0, "reject": 0, "table": 12}:
|
||||
errors.append(f"decision_counts {manifest.get('decision_counts')!r} != expected")
|
||||
try:
|
||||
seed = read_json(seed_path)
|
||||
manifest = read_json(manifest_path)
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"CHECK-2026-09-28: CANNOT GRADE -- {type(exc).__name__}: {exc}")
|
||||
return 2
|
||||
|
||||
if manifest.get("categories") != expected_counts:
|
||||
errors.append(f"categories {manifest.get('categories')!r} != expected {expected_counts}")
|
||||
if set(seed.keys()) != {"cutover", "decided", "exported_at", "pending", "source"}:
|
||||
print("CHECK-2026-09-28: CANNOT GRADE -- seed keys are not the 2026-09-26 "
|
||||
f"shape: {sorted(seed.keys())}")
|
||||
return 2
|
||||
|
||||
got = [(i["id"], i.get("category")) for i in manifest.get("items", [])]
|
||||
exp = [(i["id"], i["category"]) for i in expected_items]
|
||||
if got != exp:
|
||||
errors.append(f"per-item categories differ: {got} != {exp}")
|
||||
expected_items = []
|
||||
for it in sorted(seed["pending"], key=lambda x: int(x["id"])):
|
||||
expected_items.append({
|
||||
"id": int(it["id"]),
|
||||
"category": expected_category(it["summary"]),
|
||||
"status": "table",
|
||||
})
|
||||
|
||||
if errors:
|
||||
print("CHECK-2026-09-28: VERDICT FALSE")
|
||||
for e in errors:
|
||||
print(" -", e)
|
||||
sys.exit(1)
|
||||
expected_counts = {"archive_fix": 0, "app_pr": 0, "general": 0, "other": 0}
|
||||
for it in expected_items:
|
||||
expected_counts[it["category"]] += 1
|
||||
|
||||
print("CHECK-2026-09-28: VERDICT TRUE "
|
||||
f"(categories={manifest['categories']} table={manifest['decision_counts']['table']})")
|
||||
sys.exit(0)
|
||||
errors = []
|
||||
|
||||
# Structural keys that any genuine manifest carries (not a flake).
|
||||
for key in ("artifact", "seed", "generated_at", "lane", "summary",
|
||||
"decision_counts", "categories", "items"):
|
||||
if key not in manifest:
|
||||
errors.append(f"manifest missing key {key!r}")
|
||||
|
||||
if [i["id"] for i in manifest.get("items", [])] != [i["id"] for i in expected_items]:
|
||||
errors.append("item id order differs from seed")
|
||||
|
||||
if manifest.get("decision_counts") != {"approve": 0, "reject": 0, "table": 12}:
|
||||
errors.append(f"decision_counts {manifest.get('decision_counts')!r} != expected")
|
||||
|
||||
if manifest.get("categories") != expected_counts:
|
||||
errors.append(f"categories {manifest.get('categories')!r} != expected {expected_counts}")
|
||||
|
||||
got = [(i["id"], i.get("category")) for i in manifest.get("items", [])]
|
||||
exp = [(i["id"], i.get("category")) for i in expected_items]
|
||||
if got != exp:
|
||||
errors.append(f"per-item categories differ: {got} != {exp}")
|
||||
|
||||
if errors:
|
||||
print("CHECK-2026-09-28: VERDICT FALSE")
|
||||
for e in errors:
|
||||
print(" -", e)
|
||||
return 1
|
||||
|
||||
print("CHECK-2026-09-28: VERDICT TRUE "
|
||||
f"(categories={manifest['categories']} "
|
||||
f"table={manifest['decision_counts']['table']})")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,174 @@
|
||||
#!/usr/bin/env python3
|
||||
"""test_check_provenance.py -- black-box grader for wizard-work/sol/CHECK-2026-09-28.
|
||||
|
||||
WHY THIS EXISTS
|
||||
Q106: CHECK-2026-09-28 pinned `ROOT = "/root/troupe-standup"`, so it graded the
|
||||
host worktree it was born in and ignored the tree it shipped with. Measured
|
||||
before the repair: a clone whose own manifest was doctored (one item
|
||||
`archive_fix` -> `app_pr`, aggregate kept consistent) still read
|
||||
`VERDICT TRUE ... categories={'archive_fix': 7, ...}` rc 0 -- the host's cells,
|
||||
not the clone's 6/4/2/0.
|
||||
|
||||
This module runs the CHECK as a subprocess and imports nothing from it, so no
|
||||
cell can agree with the code by sharing its constants. It builds its own
|
||||
substrate: a minimal tree (seed + manifest + CHECK) under a temporary root.
|
||||
|
||||
CELLS
|
||||
1 tree-ships-green the shipped CHECK is green and names the tree it lives in
|
||||
2 poisoned-copy-is-red a doctored copy reads FALSE, rc 1, and prints the COPY's cells
|
||||
3 clean-copy-foreign-cwd green from a cwd that is neither tree, subject = the copy
|
||||
4 root-override --root grades the tree it names (poisoned -> rc 1)
|
||||
5 no-host-pin the file carries no absolute /root/ path and no ROOT literal
|
||||
6 usage-error-is-2 an unknown flag exits 2 (could not grade, not a verdict)
|
||||
7 missing-manifest-is-2 a tree without its manifest exits 2 naming the path, no traceback
|
||||
8 host-pin-mutant-revived restoring the pin over a poisoned copy reads TRUE again --
|
||||
the falsification cell: cell 2 fails on the exact old defect
|
||||
|
||||
Run from anywhere:
|
||||
python3 wizard-work/sol/test_check_provenance.py
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
REPO = HERE.parents[1]
|
||||
CHECK = HERE / "CHECK-2026-09-28"
|
||||
SEED_REL = Path("inbox") / "seed-2026-09-26.json"
|
||||
MANIFEST_REL = Path("wizard-work") / "sol" / "Triage-2026-09-28-classify.json"
|
||||
CHECK_REL = Path("wizard-work") / "sol" / "CHECK-2026-09-28"
|
||||
|
||||
PIN_LINE = 'DEFAULT_ROOT = os.path.dirname(os.path.dirname(HERE))'
|
||||
HOST_PIN = 'DEFAULT_ROOT = "/root/troupe-standup"'
|
||||
|
||||
|
||||
def run_check(check_path, args=(), cwd=None):
|
||||
return subprocess.run(
|
||||
[sys.executable, str(check_path), *args],
|
||||
cwd=str(cwd) if cwd else None,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=120,
|
||||
)
|
||||
|
||||
|
||||
def build_tree(dest):
|
||||
"""A minimal tree the CHECK can grade: seed + manifest + the CHECK itself."""
|
||||
root = Path(dest)
|
||||
for rel in (SEED_REL, MANIFEST_REL):
|
||||
target = root / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(REPO / rel, target)
|
||||
(root / CHECK_REL).parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(CHECK, root / CHECK_REL)
|
||||
os.chmod(root / CHECK_REL, 0o755)
|
||||
return root
|
||||
|
||||
|
||||
def poison(root):
|
||||
"""Flip one item's category and keep the aggregate honest: 6/4/2/0."""
|
||||
path = Path(root) / MANIFEST_REL
|
||||
manifest = json.loads(path.read_text(encoding="utf-8"))
|
||||
item = next(i for i in manifest["items"] if i["category"] == "archive_fix")
|
||||
item["category"] = "app_pr"
|
||||
counts = {"archive_fix": 0, "app_pr": 0, "general": 0, "other": 0}
|
||||
for i in manifest["items"]:
|
||||
counts[i["category"]] += 1
|
||||
manifest["categories"] = counts
|
||||
path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
|
||||
return item["id"], counts
|
||||
|
||||
|
||||
class CheckProvenance(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._tmp = tempfile.TemporaryDirectory(prefix="q106-")
|
||||
self.tmp = Path(self._tmp.name)
|
||||
|
||||
def tearDown(self):
|
||||
self._tmp.cleanup()
|
||||
|
||||
def test_1_tree_ships_green(self):
|
||||
proc = run_check(CHECK, cwd=self.tmp)
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertIn("VERDICT TRUE", proc.stdout)
|
||||
self.assertIn(f"subject root={REPO}", proc.stdout)
|
||||
|
||||
def test_2_poisoned_copy_is_graded_as_poisoned(self):
|
||||
root = build_tree(self.tmp / "copy")
|
||||
item_id, counts = poison(root)
|
||||
self.assertEqual(counts, {"archive_fix": 6, "app_pr": 4, "general": 2, "other": 0})
|
||||
proc = run_check(root / CHECK_REL, cwd=self.tmp)
|
||||
self.assertEqual(proc.returncode, 1, proc.stdout + proc.stderr)
|
||||
self.assertIn(f"subject root={root}", proc.stdout)
|
||||
self.assertNotIn("VERDICT TRUE", proc.stdout)
|
||||
self.assertIn("VERDICT FALSE", proc.stdout)
|
||||
# the graded side is the COPY's manifest (6/4), not the host's (7/3)
|
||||
self.assertIn("categories {'archive_fix': 6, 'app_pr': 4, 'general': 2, "
|
||||
"'other': 0}", proc.stdout)
|
||||
self.assertIn("per-item categories differ", proc.stdout)
|
||||
self.assertIn(f"({item_id}, 'app_pr')", proc.stdout)
|
||||
|
||||
def test_3_clean_copy_green_from_foreign_cwd(self):
|
||||
root = build_tree(self.tmp / "clean")
|
||||
elsewhere = self.tmp / "elsewhere"
|
||||
elsewhere.mkdir()
|
||||
proc = run_check(root / CHECK_REL, cwd=elsewhere)
|
||||
self.assertEqual(proc.returncode, 0, proc.stdout + proc.stderr)
|
||||
self.assertIn("VERDICT TRUE", proc.stdout)
|
||||
self.assertIn(f"subject root={root}", proc.stdout)
|
||||
|
||||
def test_4_root_override_grades_the_named_tree(self):
|
||||
root = build_tree(self.tmp / "override")
|
||||
poison(root)
|
||||
proc = run_check(CHECK, args=["--root", str(root)], cwd=self.tmp)
|
||||
self.assertEqual(proc.returncode, 1, proc.stdout + proc.stderr)
|
||||
self.assertIn(f"subject root={root}", proc.stdout)
|
||||
self.assertIn("VERDICT FALSE", proc.stdout)
|
||||
|
||||
def test_5_no_host_pin(self):
|
||||
blob = CHECK.read_bytes()
|
||||
self.assertNotIn(b"/root/", blob)
|
||||
self.assertNotIn(b'ROOT = "', blob)
|
||||
self.assertIn(PIN_LINE.encode(), blob)
|
||||
|
||||
def test_6_usage_error_is_2(self):
|
||||
proc = run_check(CHECK, args=["--bogus"], cwd=self.tmp)
|
||||
self.assertEqual(proc.returncode, 2, proc.stdout + proc.stderr)
|
||||
|
||||
def test_7_missing_manifest_is_2(self):
|
||||
root = build_tree(self.tmp / "gone")
|
||||
(root / MANIFEST_REL).unlink()
|
||||
proc = run_check(root / CHECK_REL, cwd=self.tmp)
|
||||
self.assertEqual(proc.returncode, 2, proc.stdout + proc.stderr)
|
||||
self.assertIn("CANNOT GRADE", proc.stdout)
|
||||
self.assertIn(str(root / MANIFEST_REL), proc.stdout)
|
||||
self.assertNotIn("Traceback", proc.stdout + proc.stderr)
|
||||
|
||||
def test_8_host_pin_mutant_revives_the_fault(self):
|
||||
"""The cell that keeps cell 2 from being vacuous: reinstate the pin."""
|
||||
root = build_tree(self.tmp / "mutant")
|
||||
poison(root)
|
||||
mutant = (root / CHECK_REL).with_name("CHECK-mutant-pinned")
|
||||
blob = (root / CHECK_REL).read_text(encoding="utf-8")
|
||||
self.assertIn(PIN_LINE, blob)
|
||||
mutant.write_text(blob.replace(PIN_LINE, HOST_PIN), encoding="utf-8")
|
||||
self.assertIn("/root/", mutant.read_text(encoding="utf-8"))
|
||||
|
||||
green_under_fault = run_check(mutant, cwd=self.tmp)
|
||||
red_over_copy = run_check(root / CHECK_REL, cwd=self.tmp) # repaired line
|
||||
|
||||
self.assertEqual(green_under_fault.returncode, 0,
|
||||
"the mutant should read the host and go green:\n"
|
||||
+ green_under_fault.stdout)
|
||||
self.assertIn("VERDICT TRUE", green_under_fault.stdout)
|
||||
self.assertEqual(red_over_copy.returncode, 1)
|
||||
self.assertNotEqual(green_under_fault.returncode, red_over_copy.returncode)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
Reference in new issue
Block a user