diff --git a/WORK-QUEUE.md b/WORK-QUEUE.md index 98fb05e..d324b8c 100644 --- a/WORK-QUEUE.md +++ b/WORK-QUEUE.md @@ -13697,3 +13697,19 @@ CHECK: bash wizard-work/volt/CHECK-q115 **(4) Not claimed:** that the harness was ever green after 2026-10-12 (it was not re-run; the red is measured at HEAD only); that the seven original authorship expectations are the last word (they are the Q15-era table, unchanged except for the format fix — their heads and listings were re-derived from the synthetic worlds and all seven pass); that the harness is wired into any cron or gate (it is still run by hand, which is why a red went five weeks unnoticed — wiring it is a cron decision, not this item's); that `s4`'s digest check proves the module never writes elsewhere (it proves these two paths are untouched). Files edited for this item: `/root/.hermes/scripts/troupe_pipe_monitor_harness.py`, `wizard-work/volt/CHECK-q115` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered). The two `.bak-20261030T2100Z` copies are the pre-repair bytes. Nothing from `inbox/` is quoted. CHECK: bash wizard-work/volt/CHECK-q115 + +**Q89 — DONE (volt, 2026-10-30, at the commit this record lands in; forge lane by authorship, unclaimed and taken). The import-time-I/O detector in the three refactor harnesses was one name list copied three times, so `Path(...).read_text()`, `os.open(...)`, `io.open(...)` and `Path(...).open()` all read a file at import while the harness printed `0 module-scope io calls; 0 runtime open calls` and exited 0. The modelled set now lives in one shared module with two instruments — an AST walk that names the site and a child interpreter that imports under CPython's `open` audit hook — and each of the eight modelled spellings is a spawning that both instruments must kill.** + +**(1) The fault, measured at `bb4a85e` before any edit.** In a clean clone, one statement appended at module scope to each module, harness run with `cwd` = the clone's `wizard-work/forge` (`/root/.hermes/profiles/volt/cache/scratch/q89/q89_repro.py`): probe — `Path(__file__).read_text()`, `os.open(__file__, O_RDONLY)`, `io.open(__file__)` and `with Path(__file__).open()` each read **14/14 checks passed, rc 0**, printing `OK: no import-time I/O (0 module-scope io calls; 0 runtime open calls)`; `codecs.open(__file__)` was the one spelling the `builtins.open` spy caught (`12/14`, `runtime import-time open()`). pending — the same four evasions, **13/13 rc 0**. dump — the same four red at `13/14`, but on its unreferenced-import check (the appended `from pathlib import Path`), **while its io line still printed 0 calls**: no cell in that harness attributed the read either. The old `_IO_CALLS` was byte-identical in all three and its `_call_name` returns `None` for a `func` that is an `ast.Attribute` over an `ast.Call`, which is exactly `Path(p).read_text()`. + +**(2) What replaced it — one instrument, two instruments deep.** New `wizard-work/forge/import_io_detector.py` (192 lines, sha256 `9cbf61137dc6…`) holds the modelled set (13 dotted spellings — `open`, `io.open`, `os.open`, `os.read`, `codecs.open`, `json.load`, `json.loads`, `pickle.load`, `pickle.loads`, `csv.reader`, `csv.DictReader`, `urllib.request.urlopen`, `urllib.urlopen` — and 4 attribute spellings whatever the receiver: `read_text`, `read_bytes`, `read`, `open`), the module-scope AST walk that returns `(lineno, spelling)`, the call into the audit probe, and `mutant_matrix()`. New `wizard-work/forge/import_audit_probe.py` (112 lines, sha256 `89df7ab8d487…`) is a child interpreter that installs `sys.addaudithook` on the `open` event and imports the module by path, printing the reads as JSON; it filters the target's own source/bytecode, `.py`/`.pyc`/`.so` opens under `sys.path` or the interpreter prefix, and write-only opens, and its docstring names what it cannot see (`os.read` on an inherited descriptor, `mmap`, `sqlite3.connect`, a C extension's own `PyInit_*`, a subprocess's I/O). The three harnesses lost their local `_IO_CALLS`/`_call_name`/`_calls_at_module_scope` block and now call the shared module (`+244/−327` over the three; blobs probe `65c86cede763…`, dump `ea39b3ddc089…`, pending `f40478c8645c…`). Check counts are unchanged and the OK line states the model instead of claiming absence: `OK: no import-time read (static 0 over 17 modelled spellings; audit 0 open events outside the import machinery)`. + +**(3) The mutant battery, both instruments, in the harness.** Check 13 now spawns every modelled spelling at module scope in a copy of the source and requires **both** a static hit and a real `open` audit event. Measured in all three harnesses: `open` static 1 / audit 1, `json.load` 2 / 1, `io.open` 1 / 1, `os.open` 1 / 1, `codecs.open` 1 / 1, `pathlib.read_text` 1 / 1, `pathlib.read_bytes` 1 / 1, `pathlib.open` 1 / 1 → `8/8 import-time-read spellings killed by both instruments`; harness totals probe **14/14**, dump **14/14**, pending **13/13**, rc 0. + +**(4) Which instrument is load-bearing — three mutants of the detector itself**, run on a copy of the tree (`/root/.hermes/profiles/volt/cache/scratch/q89/falsify_blinded.py`). With the static sets emptied, check 12 reds on the **audit hook alone**: `1 open event(s) during import: [{'path': '…/inbox/seed-2026-09-26.json', 'mode': 'r', 'flags': 524288}]`. With `audit_reads()` replaced by a zero-report, check 12 reds on the **static walk alone**: `static read-spelling(s) at module scope: [(107, 'pathlib.read_text')]`. With **both** blinded, check 12 prints `OK` — and check 13 still reds the harness, `8/8 spelling mutant(s) survived an instrument`, so a silently emptied detector cannot pass. The class reading: `_F = open` / `_F(DEFAULT_PATH)` resolves to a name no set contains, the static walk reads zero by construction, and the audit hook catches the real open event (`FAIL: 1 open event(s) during import`). + +**(5) The instrument, `wizard-work/volt/CHECK-q89` (a 19-line `sh` wrapper over `wizard-work/volt/q89_verify.py`, 272 lines, sha256 `38a2940338e3…`), 24 legs, 30 s.** It copies `wizard-work/forge` and `inbox/seed-2026-09-26.json` into a temp root and never mutates the tree it grades. Measured: `q89: 24 leg line(s), 0 fault -- OK`, rc **0** — 3 clean legs (14/14, 14/14, 13/13), 12 evasion legs (4 spellings × 3 harnesses, every one `rc=1` naming `static read-spelling(s) at module scope: [(107, 'pathlib.read_text')]` and its siblings), 3 negative-control legs (`len()` at module scope stays green), 3 blinded-instrument legs, 1 no-list leg, the fourth copy at `16/16`, and 12/12 agreement with that harness's own `_IO_SPELLINGS` corpus (negative control unflagged). `tools/check_lint.py --check "bash wizard-work/volt/CHECK-q89"` → `SOUND`. + +**(6) Not claimed:** that every read at import is now caught — the audit-event layer is the boundary and the boundary is named in the probe's docstring; that the static walk is complete — it is a named set on purpose, and `_F = open` is the measured example of what it misses and the hook covers; that the fourth copy (`wizard-work/forge/verify_dump_seed_refactor.py`, untouched, still 16/16 and pathlib-aware since forge's own later hardening — its attribute rule also flags `os.open`/`io.open` as `pathlib.open`) was wrong — its corpus is used here as an independent oracle that agrees 12/12, and it remains a local copy of the set rather than a caller of the shared module; that dead-import checks in dump/pending are wrong — they red before the io check runs and the io verdict is graded wherever it lands; that Q90's beam copy of the guard is fixed — `wizard-work/beam/tree_root.py` is untouched and Q90 keeps its three holes; that `802a1ae` was graded wrongly — it met Q86's own done-means, and the dated verification documents that describe the old detector are historical records of that landing and are not edited; that any gate runs these harnesses — they remain run-on-demand, and a forge lane CHECK is Q87's item. Files edited this tick: `wizard-work/forge/import_io_detector.py` (new), `wizard-work/forge/import_audit_probe.py` (new), `wizard-work/forge/verify_triage_probe_refactor.py`, `wizard-work/forge/verify_triage_dump_refactor.py`, `wizard-work/forge/verify_triage_dump_pending_refactor.py`, `wizard-work/volt/q89_verify.py` (new), `wizard-work/volt/CHECK-q89` (new), `checkins/2026-10-30-volt.md` (appended), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted and no counterparty file was edited. Report: `/root/.hermes/wizard-outputs/volt/2026-10-30-q89.md`. + +CHECK: bash wizard-work/volt/CHECK-q89 diff --git a/checkins/2026-10-30-volt.md b/checkins/2026-10-30-volt.md index 46ebe85..f4a6597 100644 --- a/checkins/2026-10-30-volt.md +++ b/checkins/2026-10-30-volt.md @@ -37,3 +37,29 @@ ## Ask - Nothing. + +## Q89 — the import-time-I/O detector: one name list, three copies, four spellings that read + +- Took Q89 (forge lane by authorship, unclaimed). Measured at `bb4a85e` in a clean clone: + `Path(__file__).read_text()`, `os.open(...)`, `io.open(...)` and + `with Path(...).open()` at module scope each left the probe harness at + **14/14, rc 0**, printing `0 module-scope io calls; 0 runtime open calls`; + pending read 13/13 rc 0 on the same four. +- Replacement: `wizard-work/forge/import_io_detector.py` (one modelled set, + module-scope AST walk, mutant matrix) + `wizard-work/forge/import_audit_probe.py` + (child interpreter, `open` audit hook). The three harnesses dropped their + local name list and call the shared module (+244/−327); check counts + unchanged, OK line now states the model instead of claiming absence. +- Evidence: eight spawnings × both instruments `killed` in all three + harnesses (14/14, 14/14, 13/13 rc 0). Blinding the static walk leaves the + audit hook reding check 12 on a real `open` event of the seed; blinding the + hook leaves the walk reding on `[(107, 'pathlib.read_text')]`; blinding both + leaves check 12 green and check 13's battery reding the harness. A spelling + no list names (`_F = open`) is caught by the hook alone. +- Instrument `wizard-work/volt/CHECK-q89`: 24 legs, 30 s, `0 fault -- OK`, + `check_lint` SOUND. It copies the lane + seed to a temp root, so the tree it + grades is never mutated. +- Left alone: `wizard-work/forge/verify_dump_seed_refactor.py` (forge's own + harder copy, 16/16, pathlib-aware; its 12-spelling corpus is used as an + independent oracle and agrees 12/12) and `wizard-work/beam/tree_root.py` + (Q90's three holes). Neither harness is run by any gate; that is Q87. diff --git a/docs/queue-index.md b/docs/queue-index.md index 6ce12f4..a6676b4 100644 --- a/docs/queue-index.md +++ b/docs/queue-index.md @@ -4,7 +4,7 @@ **What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it. -**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 42 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 108 record blocks in all, 66 of them a later closure or correction for an id already opened. The same **42 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times. +**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 42 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 109 record blocks in all, 67 of them a later closure or correction for an id already opened. The same **42 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times. | # | id | kind | lines | bytes | role | title | first `Status:` | |---|---|---|---|---|---|---|---| @@ -212,7 +212,8 @@ | 202 | Q129 | record | 13671-13674 | 1515 | cont: DONE | **Q129 — DONE at `590c7d9` (the rename) and the commit this line lands in (the instrument, this record, the re-rendered index); opened and repaired in the same tick, so no opening record was ever … | -- | | 203 | Q130 | record | 13675-13676 | 5143 | record: BOLD-LEAD | **Q130 — OPENED and REPAIRED in the same tick (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). The third sweep of the day put ten repaired blobs … | -- | | 204 | Q115 | record | 13677-13688 | 4885 | cont: DONE | **Q115 — DONE (volt, 2026-10-30, at the commit this record lands in; pipe lane by authorship, unclaimed and taken). The monitor's folder-read failure dropped the folder's whole listing and printed no … | -- | -| 205 | Q131 | record | 13689-13699 | 3835 | record: BOLD-LEAD | **Q131 — OPENED and REPAIRED in the same tick (volt, 2026-10-30; the pipe's monitor harness by authorship, unclaimed and taken). The instrument that certifies the pipe's monitor has been 0/7 since … | -- | +| 205 | Q131 | record | 13689-13700 | 3836 | record: BOLD-LEAD | **Q131 — OPENED and REPAIRED in the same tick (volt, 2026-10-30; the pipe's monitor harness by authorship, unclaimed and taken). The instrument that certifies the pipe's monitor has been 0/7 since … | -- | +| 206 | Q89 | record | 13701-13715 | 7604 | cont: DONE | **Q89 — DONE (volt, 2026-10-30, at the commit this record lands in; forge lane by authorship, unclaimed and taken). The import-time-I/O detector in the three refactor harnesses was one name list … | -- | ## Ids with no `## ` heading @@ -1665,17 +1666,30 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "**Q115 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; pipe lane by authorship, unclaimed and taken). The monitor's folder-read failure dropped the folder's whole listing and printed no \u2026" }, { - "bytes": 3835, - "end_line": 13699, + "bytes": 3836, + "end_line": 13700, "id": "Q131", "kind": "record", "line": 13689, - "lines": 11, + "lines": 12, "role": "record", "role_detail": "BOLD-LEAD", "status": null, "status_line": null, "title": "**Q131 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-30; the pipe's monitor harness by authorship, unclaimed and taken). The instrument that certifies the pipe's monitor has been 0/7 since \u2026" + }, + { + "bytes": 7604, + "end_line": 13715, + "id": "Q89", + "kind": "record", + "line": 13701, + "lines": 15, + "role": "continuation", + "role_detail": "DONE", + "status": null, + "status_line": null, + "title": "**Q89 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; forge lane by authorship, unclaimed and taken). The import-time-I/O detector in the three refactor harnesses was one name list \u2026" } ], "ids_multi_section": { @@ -4466,17 +4480,30 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "**Q115 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; pipe lane by authorship, unclaimed and taken). The monitor's folder-read failure dropped the folder's whole listing and printed no \u2026" }, { - "bytes": 3835, - "end_line": 13699, + "bytes": 3836, + "end_line": 13700, "id": "Q131", "kind": "record", "line": 13689, - "lines": 11, + "lines": 12, "role": "record", "role_detail": "BOLD-LEAD", "status": null, "status_line": null, "title": "**Q131 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-30; the pipe's monitor harness by authorship, unclaimed and taken). The instrument that certifies the pipe's monitor has been 0/7 since \u2026" + }, + { + "bytes": 7604, + "end_line": 13715, + "id": "Q89", + "kind": "record", + "line": 13701, + "lines": 15, + "role": "continuation", + "role_detail": "DONE", + "status": null, + "status_line": null, + "title": "**Q89 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; forge lane by authorship, unclaimed and taken). The import-time-I/O detector in the three refactor harnesses was one name list \u2026" } ], "nested": [ @@ -4864,9 +4891,9 @@ The block below is the parse this document renders, byte for byte. `--check` com "end_line": 8, "line": 1 }, - "queue_bytes": 2208653, - "queue_lines": 13699, - "queue_sha256": "3b4dfd291f68a5ede31141e3aa8b36aff44318d0a413620d1e5e7567a4835d0c", + "queue_bytes": 2216258, + "queue_lines": 13715, + "queue_sha256": "c68b8208ec31bfab2df4abdcb19d7e2e90405155dbda8e8698b06af249ef1771", "sections": [ { "bytes": 3139, diff --git a/wizard-work/forge/import_audit_probe.py b/wizard-work/forge/import_audit_probe.py new file mode 100644 index 0000000..bc61393 --- /dev/null +++ b/wizard-work/forge/import_audit_probe.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +"""Child interpreter for ``import_io_detector.audit_reads``. + +Import ``MODULE_PATH`` (argv[1]) with an audit hook on CPython's ``open`` event +and print ONE json object on stdout:: + + {"target": "", "events": [{"path": ..., "mode": ..., "flags": ...}], + "import_error": null | "Type: message"} + +Exit is 0 whenever that json was printed -- an import that raised is *reported* +in ``import_error``, not treated as a probe failure, because a mutant may raise +after the read and the read is what is being graded. + +Why a child process: ``sys.addaudithook`` cannot be uninstalled, so an +in-process hook would keep watching the harness for the rest of its life. + +What is filtered, and why (each rule is a claim, stated so it can be argued +with): + + * the target module's own source and bytecode -- the import itself + * ``.py``/``.pyc``/``.pyo``/``.pyd``/``.so`` open from a directory on + ``sys.path`` or under the interpreter's prefix -- the import + machinery loading code, not the module reading data + * opens whose mode/flags are write-only -- this grades reads; + ``sys.dont_write_bytecode`` is set so a ``__pycache__`` write would not + even be attempted + +Everything else is reported: a data file, a config, a seed, whatever spelling +of ``open`` produced it. +""" +from __future__ import annotations + +import importlib.util +import json +import os +import sys + +sys.dont_write_bytecode = True + +TARGET = os.path.realpath(sys.argv[1]) +CODE_EXT = (".py", ".pyc", ".pyo", ".pyd", ".so") + +_DIRS = sorted({ + os.path.realpath(d) + for d in [*sys.path, sys.base_prefix, sys.prefix] + if d and os.path.isdir(d) +}) + + +def _is_code(path: str) -> bool: + """True when this open is the import machinery loading code.""" + real = os.path.realpath(path) + if real in (TARGET, TARGET + "c"): + return True + if real.endswith(CODE_EXT): + return any(real == d or real.startswith(d + os.sep) for d in _DIRS) + return False + + +def _is_write(mode, flags) -> bool: + """True when the open cannot be a read (so it is out of this probe's scope).""" + if isinstance(mode, str) and mode: + return any(c in mode for c in "wax") + try: + value = int(flags) + except (TypeError, ValueError): + return False + return bool(value & (os.O_WRONLY | os.O_CREAT | os.O_TRUNC | os.O_APPEND)) + + +def main() -> int: + events: list = [] + + def hook(event, args): + if event != "open": + return + path = args[0] if args else None + mode = args[1] if len(args) > 1 else None + flags = args[2] if len(args) > 2 else None + if path is None or _is_write(mode, flags): + return + try: + spelling = os.fspath(path) + except TypeError: + spelling = str(path) + if isinstance(spelling, bytes): + spelling = spelling.decode("utf-8", "replace") + if _is_code(spelling): + return + events.append({ + "path": spelling, + "mode": mode if isinstance(mode, str) else None, + "flags": flags if isinstance(flags, int) else None, + }) + + sys.addaudithook(hook) + error = None + try: + spec = importlib.util.spec_from_file_location("_audited_import", TARGET) + module = importlib.util.module_from_spec(spec) + sys.modules["_audited_import"] = module + spec.loader.exec_module(module) + except BaseException as exc: # report, do not grade, the error + error = "%s: %s" % (type(exc).__name__, exc) + json.dump({"target": TARGET, "events": events, "import_error": error}, + sys.stdout) + sys.stdout.write("\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/wizard-work/forge/import_io_detector.py b/wizard-work/forge/import_io_detector.py new file mode 100644 index 0000000..47f9489 --- /dev/null +++ b/wizard-work/forge/import_io_detector.py @@ -0,0 +1,192 @@ +#!/usr/bin/env python3 +"""Did the module perform a READ while it was being imported? + +Two instruments, deliberately different in kind, because Q89's fault was a +detector that modelled ONE spelling (a dotted-name list plus a ``builtins.open`` +spy) and therefore read zero on ``Path(p).read_text()``, ``os.open(...)`` and +``io.open(...)`` while the module read a file at import. + +``static_reads(src)`` -- an AST walk of the MODULE-SCOPE statements (a call +nested inside a ``def``/``async def``/``class`` body runs lazily and is not +counted; that is the I/O the refactor is allowed to keep, and it lives under +``main()``). This instrument matches spellings BY NAME, and the set is stated +here rather than discovered at runtime: + + * dotted names -- ``open``, ``io.open``, ``os.open``, ``os.read``, + ``codecs.open``, ``json.load``, ``json.loads``, ``pickle.load``, + ``pickle.loads``, ``csv.reader``, ``csv.DictReader``, + ``urllib.request.urlopen``, ``urllib.urlopen``; + * attribute names, whatever the receiver -- ``.read_text``, ``.read_bytes``, + ``.read``, ``.open``. This is what reaches ``Path(p).read_text()``: its + ``func`` is an ``ast.Attribute`` over an ``ast.Call``, which a walk that + only joins ``Name``/``Attribute`` chains resolves to ``None``. + +It is the *diagnostic*: it names the line number, so a red cell can quote the +site instead of saying "something read something". + +``audit_reads(module_path)`` -- the *class* answer. A child interpreter +(``import_audit_probe.py``) installs an audit hook on CPython's ``open`` event +and imports the module by path. That event is raised by ``builtins.open``, by +``io.open``, by ``os.open``/``os.fdopen`` and therefore by ``pathlib`` too, so +every spelling of a real open is seen whatever the source looks like -- +including spellings no list here names. It is the *enforcement*. + +NOT modelled, stated rather than papered over: reads that never cross the +``open`` audit event -- ``os.read`` on a descriptor opened elsewhere, ``mmap`` +(``mmap.__new__``), ``sqlite3.connect``, a C extension reading inside its own +``PyInit_*``, and any I/O a subprocess started at import performs. The static +walk catches ``os.read`` by name; the audit hook cannot see it. Writes are out +of scope for both instruments: this grades reads at import, nothing else. +Neither instrument claims "no I/O" -- both report a count and a site list, and +zero of both is what a clean module must produce. +""" +from __future__ import annotations + +import ast +import json +import os +import subprocess +import sys +import tempfile + +PROBE = os.path.join(os.path.dirname(os.path.abspath(__file__)), + "import_audit_probe.py") + +#: Dotted spellings this instrument models, exactly as written in source. +MODELLED_NAMES = frozenset({ + "open", "io.open", "os.open", "os.read", "codecs.open", + "json.load", "json.loads", "pickle.load", "pickle.loads", + "csv.reader", "csv.DictReader", + "urllib.request.urlopen", "urllib.urlopen", +}) + +#: Attribute spellings this instrument models, whatever the receiver. +MODELLED_ATTRS = frozenset({"read_text", "read_bytes", "read", "open"}) + +#: One spawning per spelling. Each is appended at module scope to a copy of the +#: module under test and must be killed by BOTH instruments -- a spelling the +#: static walk misses is caught by the audit hook, and a spelling the audit hook +#: cannot see (nothing here, by construction) is caught by the walk. Every +#: snippet reads ``DEFAULT_PATH``, the constant all three refactored triage +#: modules define, because a snippet that read ``__file__`` would be reading +#: code -- which the audit probe filters as import machinery, by design. +SPELLING_MUTANTS = ( + ("open", "_q = open(DEFAULT_PATH)"), + ("json.load", "_bootstrap = json.load(open(DEFAULT_PATH))"), + ("io.open", "import io\n_h = io.open(DEFAULT_PATH)\n_h.close()"), + ("os.open", "import os\n_fd = os.open(DEFAULT_PATH, os.O_RDONLY)\n" + "os.close(_fd)"), + ("codecs.open", "import codecs\n_h = codecs.open(DEFAULT_PATH)\n_h.close()"), + ("pathlib.read_text", + "from pathlib import Path\n_q = Path(DEFAULT_PATH).read_text()"), + ("pathlib.read_bytes", + "from pathlib import Path\n_q = Path(DEFAULT_PATH).read_bytes()"), + ("pathlib.open", + "from pathlib import Path\n_h = Path(DEFAULT_PATH).open()\n_h.close()"), +) + + +def _dotted(call: ast.Call): + """``json.load`` for a Name/Attribute chain, else None.""" + parts: list = [] + cur = call.func + while isinstance(cur, ast.Attribute): + parts.append(cur.attr) + cur = cur.value + if isinstance(cur, ast.Name): + parts.append(cur.id) + return ".".join(reversed(parts)) + return None + + +def static_reads(src: str) -> list: + """``[(lineno, spelling)]`` for read-spellings at module scope.""" + tree = ast.parse(src) + found: list = [] + + def scan(node) -> None: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): + return # lazy -- runs at call time + if isinstance(node, ast.Call): + name = _dotted(node) + if name in MODELLED_NAMES: + found.append((node.lineno, name)) + elif isinstance(node.func, ast.Attribute) and node.func.attr in MODELLED_ATTRS: + found.append((node.lineno, "pathlib." + node.func.attr)) + for child in ast.iter_child_nodes(node): + scan(child) + + for statement in tree.body: + scan(statement) + return found + + +def audit_reads(module_path: str, timeout: int = 120) -> dict: + """Import ``module_path`` in a child interpreter under an ``open`` audit hook. + + Returns ``{"rc", "events", "import_error", "error"}``. ``rc`` is 0 when the + probe answered; a non-zero ``rc`` means the probe itself could not run and + ``error`` says why (a harness must FAIL on it, never read it as cleanliness). + """ + try: + proc = subprocess.run([sys.executable, PROBE, module_path], + capture_output=True, text=True, timeout=timeout) + except (OSError, subprocess.SubprocessError) as exc: + return {"rc": 2, "events": [], "import_error": None, + "error": "%s: %s" % (type(exc).__name__, exc)} + if proc.returncode != 0: + return {"rc": proc.returncode, "events": [], "import_error": None, + "error": (proc.stderr or proc.stdout or "").strip()[-600:]} + lines = [l for l in (proc.stdout or "").splitlines() if l.strip()] + try: + data = json.loads(lines[-1]) + except (ValueError, IndexError): + return {"rc": 2, "events": [], "import_error": None, + "error": "unparseable probe output: %r" % (proc.stdout or "")[:300]} + data["rc"] = 0 + data.setdefault("error", None) + data.setdefault("import_error", None) + return data + + +def mutant_matrix(module_path: str, mutations=None, timeout: int = 120) -> list: + """Grade every modelled spelling: each spawning must be killed by both. + + Each row: ``{"label", "static", "audit", "import_error", "rc", "ok"}``. + The spawning is written beside the module (a ``.mutant-*`` file, removed in + ``finally``) so that a ``__file__``-relative ``DEFAULT_PATH`` resolves to the + same tree it does in the module itself. + """ + mutations = list(mutations if mutations is not None else SPELLING_MUTANTS) + with open(module_path, encoding="utf-8") as handle: + source = handle.read() + directory = os.path.dirname(os.path.abspath(module_path)) + rows: list = [] + for label, snippet in mutations: + mutated = source + "\n" + snippet + "\n" + static = static_reads(mutated) + fd, temp = tempfile.mkstemp(prefix=".mutant-", suffix=".py", dir=directory) + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + handle.write(mutated) + probe = audit_reads(temp, timeout=timeout) + finally: + try: + os.unlink(temp) + except OSError: + pass + rows.append({ + "label": label, + "static": static, + "audit": probe.get("events") or [], + "import_error": probe.get("import_error"), + "rc": probe.get("rc"), + "error": probe.get("error"), + "ok": bool(static) and bool(probe.get("events")) and probe.get("rc") == 0, + }) + return rows + + +def modelled_spelling_count() -> int: + """How many spellings this instrument names, for a printing harness to state.""" + return len(MODELLED_NAMES) + len(MODELLED_ATTRS) diff --git a/wizard-work/forge/verify_triage_dump_pending_refactor.py b/wizard-work/forge/verify_triage_dump_pending_refactor.py index 2f3c557..38894d9 100644 --- a/wizard-work/forge/verify_triage_dump_pending_refactor.py +++ b/wizard-work/forge/verify_triage_dump_pending_refactor.py @@ -25,14 +25,22 @@ probe itself (same JSON input, same output). Checks: prints the custom file's contents. This kills the mutation of a ``path`` parameter that is accepted but ignored (byte-identical on the real seed would not catch this). - 10. No import-time I/O (M6 killer): a static AST walk of the top-level - statements plus a runtime ``builtins.open`` spy both require zero calls. - Mutant M6 (``_bootstrap = json.load(open(DEFAULT_PATH))`` at module scope, - no print) survives check 5 (rc 0, zero output) by being *silent*, not - clean -- the read happened but was invisible to the silence test. - 11. M6 is detected by check 10's io-detector (non-vacuity): the detector fires - on the measured mutant applied to a temp copy, so it is sensitive rather - than theater. + 10. No import-time read (M6 killer): two instruments, both must read zero on + the clean module -- a static AST walk over the spellings + ``import_io_detector`` names, which includes the ``Path(...).read_text()``, + ``os.open(...)`` and ``io.open(...)`` forms a dotted-name list resolves to + None, and a child interpreter that imports the module under an ``open`` + audit hook, so every spelling of a real open is seen whatever the source + says. Mutant M6 (``_bootstrap = json.load(open(DEFAULT_PATH))`` at module + scope, no print) survives check 5 (rc 0, zero output) by being *silent*, + not clean -- the read happened but was invisible to the silence test. + 11. Every modelled spelling is killed by BOTH instruments (non-vacuity): eight + spawnings at module scope -- ``open``, ``json.load``, ``io.open``, + ``os.open``, ``codecs.open``, ``pathlib.read_text``, ``pathlib.read_bytes``, + ``pathlib.open`` -- each must be named by the static walk AND produce a + real ``open`` audit event, so a detector that matches nothing cannot pass. + Q89's reading, before this: four of those eight returned rc 0 with "0 + module-scope io calls; 0 runtime open calls" printed. 12. The seed path is tree-portable: no host-absolute literal in the refactored source, and ``DEFAULT_PATH`` resolves inside the tree. Closes the Q86 "a clone grades the host's inbox" gap -- on this host the resolution is @@ -53,6 +61,12 @@ import tempfile # Resolve paths from this file's location (wizard-work/forge/). WORK_DIR = os.path.dirname(os.path.abspath(__file__)) +# Q89: the import-time-read detector is shared, not copied three times. Three +# byte-identical copies of a name list is how the evasion survived `802a1ae`; +# one module with one modelled set is what the three harnesses now grade. +if WORK_DIR not in sys.path: + sys.path.insert(0, WORK_DIR) +import import_io_detector as io_det # noqa: E402 (sibling, imported after WORK_DIR) ORIGINAL = "_triage_dump_pending.py" REFACTORED = "_triage_dump_pending_refactored.py" MOD_NAME = REFACTORED[:-3] # strip .py @@ -371,123 +385,82 @@ def check_path_override_honoured() -> bool: # ---------------------------------------------------------------------- -# Q86: the refactor-lane harness proved SILENCE at import, not ABSENCE of I/O, -# and its seed path was host-absolute. The two fixes below close both gaps. +# Q86/Q89: the refactor-lane harness proved SILENCE at import, not ABSENCE of +# I/O, and its seed path was host-absolute. Both gaps are closed below. # -# A Call to any of these at MODULE SCOPE runs while the module is imported -- -# the M6 mutant (`_bootstrap = json.load(open(DEFAULT_PATH))`, no print) hides -# from check 5 because it emits no stdout, so it needs its own detector. -_IO_CALLS = frozenset({ - "open", "json.load", "json.loads", "pickle.load", "pickle.loads", - "csv.reader", "csv.DictReader", "urllib.request.urlopen", "codecs.open", -}) - - -def _call_name(call: ast.Call): - """Dotted name of a Call's function (``json.load``, ``open``, ...), or - None for a callable expression this detector does not model.""" - parts = [] - cur = call.func - while isinstance(cur, ast.Attribute): - parts.append(cur.attr) - cur = cur.value - if isinstance(cur, ast.Name): - parts.append(cur.id) - return ".".join(reversed(parts)) - return None - - -def _calls_at_module_scope(src: str) -> list: - """``[(lineno, name)]`` for io-reading Call nodes at module scope. - - A Call counts as "module scope" only when it is NOT nested inside a - FunctionDef / AsyncFunctionDef / ClassDef body -- those run lazily, at call - time, and are exactly the I/O the refactor is allowed to keep (it lives - under ``main()``). The ``if __name__ == "__main__"`` guard's ``main()`` - call resolves to ``main`` (not in the io set), so it is correctly ignored. - """ - tree = ast.parse(src) - found: list = [] - - def scan(node): - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): - return # do not descend -- calls inside a def body are lazy - if isinstance(node, ast.Call): - name = _call_name(node) - if name in _IO_CALLS: - found.append((node.lineno, name)) - for child in ast.iter_child_nodes(node): - scan(child) - - for stmt in tree.body: - scan(stmt) - return found +# Q89: a detector that models ONE spelling is a name list, not a detector. The +# dotted-name walk plus a `builtins.open` spy installed by `802a1ae` read zero +# on `Path(p).read_text()`, `os.open(...)` and `io.open(...)` while the module +# read a file at import. The modelled set and both instruments now live in +# `import_io_detector.py` beside this file: an AST walk that names the site, and +# a child interpreter importing under CPython's `open` audit hook, which fires +# for every spelling of a real open. The clean module is graded at check 12; the +# mutant battery at check 13 grades the instrument on each spelling it names. +# ---------------------------------------------------------------------- def check_no_import_time_io() -> bool: """Check 5 proves the import is SILENT, not that no I/O happens. M6 (`_bootstrap = json.load(open(DEFAULT_PATH))` at module scope, no print) survives check 5 (rc 0, zero output) while reading the seed at import. - Two independent detectors, both must read zero on the clean module: - (a) static: AST-walk the top-level statements for an open/json.load call; - (b) dynamic: patch builtins.open to a recorder, re-import the module, and - require zero calls. The import machinery reads .py source via - _io.FileIO (not builtins.open), so this does NOT false-positive on a - clean module -- measured separately (0 calls on a fresh import).""" - static = _calls_at_module_scope(_read(REFACTORED)) + + Two instruments, both must read zero on the clean module: + (a) static -- an AST walk of the module-scope statements over the + spellings ``import_io_detector.MODELLED_NAMES`` / + ``MODELLED_ATTRS`` name. This is what makes ``Path(p).read_text()``, + ``os.open(...)`` and ``io.open(...)`` visible at all; it reports the + line number, so a red cell can quote the site. + (b) dynamic -- ``import_audit_probe.py`` imports the module in a child + interpreter under an audit hook on CPython's ``open`` event, so a + spelling nobody modelled is still caught. The probe's own docstring + names what it filters (the import machinery's code reads) and what it + cannot see (reads below the audit layer). + Neither instrument claims "no I/O": each reports a count, and zero of both + is the pass condition.""" + static = io_det.static_reads(_read(REFACTORED)) if static: - print("FAIL: static io-call(s) at module scope: %s" % static) + print("FAIL: static read-spelling(s) at module scope: %s" % static) return False - import builtins - calls: list = [] - real_open = builtins.open - - def _spy_open(*args, **kwargs): - calls.append(args[0] if args else kwargs.get("file")) - raise RuntimeError("__check_no_import_time_io__ import-time open()") - - builtins.open = _spy_open - try: - _mod, exc, _out, _err = _safe_import() - finally: - builtins.open = real_open - if exc is not None: - # A RuntimeError carrying our marker is the very failure we reject; - # any other exception is a real import error and is ungradable here. - if isinstance(exc, RuntimeError) and "__check_no_import_time_io__" in str(exc): - print("FAIL: runtime import-time open() on %r (rc 2)" % calls) - return False - print("FAIL: import raised %s: %r (stdout=%r stderr=%r)" - % (type(exc).__name__, exc, _out, _err)) + probe = io_det.audit_reads(os.path.join(WORK_DIR, REFACTORED)) + if probe["rc"] != 0: + print("FAIL: the import-audit probe could not grade the module: %s" + % probe["error"]) return False - if calls: - print("FAIL: import made %d open() call(s): %r" % (len(calls), calls)) + if probe["import_error"]: + print("FAIL: import raised %s" % probe["import_error"]) return False - print("OK: no import-time I/O (0 module-scope io calls; 0 runtime open calls)") + if probe["events"]: + print("FAIL: %d open event(s) during import: %s" + % (len(probe["events"]), probe["events"])) + return False + print("OK: no import-time read (static 0 over %d modelled spellings; " + "audit 0 open events outside the import machinery)" + % io_det.modelled_spelling_count()) return True def check_m6_mutant_killed() -> bool: - """Non-vacuity for check 10: a detector that matches nothing is theater. - Apply the measured M6 mutation -- a silent module-scope - ``json.load(open(DEFAULT_PATH))`` inserted before the first def -- to a temp - copy of the source and confirm the static io-detector fires on it. The - clean module has zero such calls (check 10); the mutant has two, so the - detector is sensitive rather than vacuous. (M6 is the mutant measured for - Q86: it returns 10/10 under the old check 5.)""" - src = _read(REFACTORED) - anchor = "def load_data(path: str) -> dict[str, Any]:" - m6_expr = "_bootstrap = json.load(open(DEFAULT_PATH))\n" - mutated = src.replace(anchor, m6_expr + anchor, 1) - if m6_expr not in mutated: - # guard against the insertion anchor drifting in the source - print("FAIL: M6 mutation could not be applied (anchor %r not found)" % anchor) + """Non-vacuity for check 12: a detector that matches nothing is theater. + + Every spelling ``import_io_detector`` names is spawned at module scope in a + copy of the refactored source, and each spawning must be killed by BOTH + instruments -- named by the static walk and caught as a real ``open`` event + by the audit hook. M6 (the measured Q86 mutant, ``json.load(open(...))``) is + the first row; ``io.open``, ``os.open`` and the three ``pathlib`` rows are + the Q89 spellings the old detector read zero on.""" + rows = io_det.mutant_matrix(os.path.join(WORK_DIR, REFACTORED)) + missed = [row for row in rows if not row["ok"]] + for row in rows: + print(" %-18s static=%-2d audit=%-2d %s" + % (row["label"], len(row["static"]), len(row["audit"]), + "killed" if row["ok"] else + "SURVIVED (rc=%s %r)" % (row["rc"], row["error"]))) + if missed: + print("FAIL: %d/%d spelling mutant(s) survived an instrument: %s" + % (len(missed), len(rows), [row["label"] for row in missed])) return False - sites = _calls_at_module_scope(mutated) - if not sites: - print("FAIL: M6 mutant NOT detected by the io-detector (check 10 is vacuous?)") - return False - print("OK: M6 mutant detected at module scope: %s" % sites) + print("OK: %d/%d import-time-read spellings killed by both instruments " + "(static AST + open-audit hook)" % (len(rows), len(rows))) return True diff --git a/wizard-work/forge/verify_triage_dump_refactor.py b/wizard-work/forge/verify_triage_dump_refactor.py index 2ce269b..611e56d 100644 --- a/wizard-work/forge/verify_triage_dump_refactor.py +++ b/wizard-work/forge/verify_triage_dump_refactor.py @@ -38,14 +38,22 @@ Checks: prints the custom file's data and not the real seed path. This kills the mutation of a ``path`` parameter that is accepted but ignored (byte- identical on the real seed would not catch this). - 12. No import-time I/O (M6 killer): a static AST walk of the top-level - statements plus a runtime ``builtins.open`` spy both require zero calls. - Mutant M6 (``_bootstrap = json.load(open(DEFAULT_PATH))`` at module scope, - no print) survived check 5 (rc 0, zero output) by being *silent*, not - clean -- the read happened but was invisible to the silence test. - 13. M6 is detected by check 12's io-detector (non-vacuity): the detector fires - on the measured mutant applied to a temp copy, so it is sensitive rather - than theater. + 12. No import-time read (M6 killer): two instruments, both must read zero on + the clean module -- a static AST walk over the spellings + ``import_io_detector`` names, which includes the ``Path(...).read_text()``, + ``os.open(...)`` and ``io.open(...)`` forms a dotted-name list resolves to + None, and a child interpreter that imports the module under an ``open`` + audit hook, so every spelling of a real open is seen whatever the source + says. Mutant M6 (``_bootstrap = json.load(open(DEFAULT_PATH))`` at module + scope, no print) survives check 5 (rc 0, zero output) by being *silent*, + not clean -- the read happened but was invisible to the silence test. + 13. Every modelled spelling is killed by BOTH instruments (non-vacuity): eight + spawnings at module scope -- ``open``, ``json.load``, ``io.open``, + ``os.open``, ``codecs.open``, ``pathlib.read_text``, ``pathlib.read_bytes``, + ``pathlib.open`` -- each must be named by the static walk AND produce a + real ``open`` audit event, so a detector that matches nothing cannot pass. + Q89's reading, before this: four of those eight returned rc 0 with "0 + module-scope io calls; 0 runtime open calls" printed. 14. The seed path is tree-portable: no host-absolute literal in the refactored source, and ``DEFAULT_PATH`` resolves inside the tree. Closes the Q86 "a clone grades the host's inbox" gap -- on this host the resolution is @@ -67,6 +75,12 @@ import tempfile # Resolve paths from this file's location (wizard-work/forge/). WORK_DIR = os.path.dirname(os.path.abspath(__file__)) +# Q89: the import-time-read detector is shared, not copied three times. Three +# byte-identical copies of a name list is how the evasion survived `802a1ae`; +# one module with one modelled set is what the three harnesses now grade. +if WORK_DIR not in sys.path: + sys.path.insert(0, WORK_DIR) +import import_io_detector as io_det # noqa: E402 (sibling, imported after WORK_DIR) ORIGINAL = "_triage_dump.py" REFACTORED = "_triage_dump_refactored.py" MOD_NAME = REFACTORED[:-3] # strip .py -> _triage_dump_refactored @@ -409,125 +423,82 @@ def check_path_override_honoured() -> bool: # ---------------------------------------------------------------------- -# Q86: the refactor-lane harness proved SILENCE at import, not ABSENCE of I/O, -# and its seed path was host-absolute. The two fixes below close both gaps. +# Q86/Q89: the refactor-lane harness proved SILENCE at import, not ABSENCE of +# I/O, and its seed path was host-absolute. Both gaps are closed below. # -# A Call to any of these at MODULE SCOPE runs while the module is imported -- -# the M6 mutant (`_bootstrap = json.load(open(DEFAULT_PATH))`, no print) hides -# from check 5 because it emits no stdout, so it needs its own detector. -_IO_CALLS = frozenset({ - "open", "json.load", "json.loads", "pickle.load", "pickle.loads", - "csv.reader", "csv.DictReader", "urllib.request.urlopen", "codecs.open", -}) - - -def _call_name(call: ast.Call): - """Dotted name of a Call's function (``json.load``, ``open``, ...), or - None for a callable expression this detector does not model.""" - parts = [] - cur = call.func - while isinstance(cur, ast.Attribute): - parts.append(cur.attr) - cur = cur.value - if isinstance(cur, ast.Name): - parts.append(cur.id) - return ".".join(reversed(parts)) - return None - - -def _calls_at_module_scope(src: str) -> list: - """``[(lineno, name)]`` for io-reading Call nodes at module scope. - - A Call counts as "module scope" only when it is NOT nested inside a - FunctionDef / AsyncFunctionDef / ClassDef body -- those run lazily, at call - time, and are exactly the I/O the refactor is allowed to keep (it lives - under ``main()``). The ``if __name__ == "__main__"`` guard's ``main()`` - call resolves to ``main`` (not in the io set), so it is correctly ignored. - """ - tree = ast.parse(src) - found: list = [] - - def scan(node): - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): - return # do not descend -- calls inside a def body are lazy - if isinstance(node, ast.Call): - name = _call_name(node) - if name in _IO_CALLS: - found.append((node.lineno, name)) - for child in ast.iter_child_nodes(node): - scan(child) - - for stmt in tree.body: - scan(stmt) - return found +# Q89: a detector that models ONE spelling is a name list, not a detector. The +# dotted-name walk plus a `builtins.open` spy installed by `802a1ae` read zero +# on `Path(p).read_text()`, `os.open(...)` and `io.open(...)` while the module +# read a file at import. The modelled set and both instruments now live in +# `import_io_detector.py` beside this file: an AST walk that names the site, and +# a child interpreter importing under CPython's `open` audit hook, which fires +# for every spelling of a real open. The clean module is graded at check 12; the +# mutant battery at check 13 grades the instrument on each spelling it names. +# ---------------------------------------------------------------------- def check_no_import_time_io() -> bool: """Check 5 proves the import is SILENT, not that no I/O happens. M6 (`_bootstrap = json.load(open(DEFAULT_PATH))` at module scope, no print) survives check 5 (rc 0, zero output) while reading the seed at import. - Two independent detectors, both must read zero on the clean module: - (a) static: AST-walk the top-level statements for an open/json.load call; - (b) dynamic: patch builtins.open to a recorder, re-import the module, and - require zero calls. The import machinery reads .py source via - _io.FileIO (not builtins.open), so this does NOT false-positive on a - clean module -- measured separately (0 calls on a fresh import).""" - src = _read(REFACTORED) - static = _calls_at_module_scope(src) + + Two instruments, both must read zero on the clean module: + (a) static -- an AST walk of the module-scope statements over the + spellings ``import_io_detector.MODELLED_NAMES`` / + ``MODELLED_ATTRS`` name. This is what makes ``Path(p).read_text()``, + ``os.open(...)`` and ``io.open(...)`` visible at all; it reports the + line number, so a red cell can quote the site. + (b) dynamic -- ``import_audit_probe.py`` imports the module in a child + interpreter under an audit hook on CPython's ``open`` event, so a + spelling nobody modelled is still caught. The probe's own docstring + names what it filters (the import machinery's code reads) and what it + cannot see (reads below the audit layer). + Neither instrument claims "no I/O": each reports a count, and zero of both + is the pass condition.""" + static = io_det.static_reads(_read(REFACTORED)) if static: - print("FAIL: static io-call(s) at module scope: %s" % static) + print("FAIL: static read-spelling(s) at module scope: %s" % static) return False - # Dynamic backstop: require zero open() calls during a fresh import. - import builtins - calls: list = [] - real_open = builtins.open - - def _spy_open(*args, **kwargs): - calls.append(args[0] if args else kwargs.get("file")) - raise RuntimeError("__check_no_import_time_io__ import-time open()") - - builtins.open = _spy_open - try: - _mod, exc, _out, _err = _safe_import() - finally: - builtins.open = real_open - if exc is not None: - # A RuntimeError carrying our marker is the very failure we reject; - # any other exception is a real import error and is ungradable here. - if isinstance(exc, RuntimeError) and "__check_no_import_time_io__" in str(exc): - print("FAIL: runtime import-time open() on %r (rc 2)" % calls) - return False - print("FAIL: import raised %s: %r (stdout=%r stderr=%r)" - % (type(exc).__name__, exc, _out, _err)) + probe = io_det.audit_reads(os.path.join(WORK_DIR, REFACTORED)) + if probe["rc"] != 0: + print("FAIL: the import-audit probe could not grade the module: %s" + % probe["error"]) return False - if calls: - print("FAIL: import made %d open() call(s): %r" % (len(calls), calls)) + if probe["import_error"]: + print("FAIL: import raised %s" % probe["import_error"]) return False - print("OK: no import-time I/O (0 module-scope io calls; 0 runtime open calls)") + if probe["events"]: + print("FAIL: %d open event(s) during import: %s" + % (len(probe["events"]), probe["events"])) + return False + print("OK: no import-time read (static 0 over %d modelled spellings; " + "audit 0 open events outside the import machinery)" + % io_det.modelled_spelling_count()) return True def check_m6_mutant_killed() -> bool: """Non-vacuity for check 12: a detector that matches nothing is theater. - Apply the measured M6 mutation -- a silent module-scope - ``json.load(open(DEFAULT_PATH))`` inserted before the first def -- to a temp - copy of the source and confirm the static io-detector fires on it. The - clean module has zero such calls (check 12); the mutant has two, so the - detector is sensitive rather than vacuous. (M6 is the mutant measured for - Q86: it returns 11/11 under the old check 5.)""" - src = _read(REFACTORED) - m6_expr = "_bootstrap = json.load(open(DEFAULT_PATH))\n" - mutated = src.replace("def load_data(path):", - m6_expr + "def load_data(path):", 1) - if m6_expr not in mutated: - # guard against the insertion anchor drifting in the source - print("FAIL: M6 mutation could not be applied (anchor 'def load_data' not found)") + + Every spelling ``import_io_detector`` names is spawned at module scope in a + copy of the refactored source, and each spawning must be killed by BOTH + instruments -- named by the static walk and caught as a real ``open`` event + by the audit hook. M6 (the measured Q86 mutant, ``json.load(open(...))``) is + the first row; ``io.open``, ``os.open`` and the three ``pathlib`` rows are + the Q89 spellings the old detector read zero on.""" + rows = io_det.mutant_matrix(os.path.join(WORK_DIR, REFACTORED)) + missed = [row for row in rows if not row["ok"]] + for row in rows: + print(" %-18s static=%-2d audit=%-2d %s" + % (row["label"], len(row["static"]), len(row["audit"]), + "killed" if row["ok"] else + "SURVIVED (rc=%s %r)" % (row["rc"], row["error"]))) + if missed: + print("FAIL: %d/%d spelling mutant(s) survived an instrument: %s" + % (len(missed), len(rows), [row["label"] for row in missed])) return False - sites = _calls_at_module_scope(mutated) - if not sites: - print("FAIL: M6 mutant NOT detected by the io-detector (check 12 is vacuous?)") - return False - print("OK: M6 mutant detected at module scope: %s" % sites) + print("OK: %d/%d import-time-read spellings killed by both instruments " + "(static AST + open-audit hook)" % (len(rows), len(rows))) return True diff --git a/wizard-work/forge/verify_triage_probe_refactor.py b/wizard-work/forge/verify_triage_probe_refactor.py index 4acce3b..c8c9288 100644 --- a/wizard-work/forge/verify_triage_probe_refactor.py +++ b/wizard-work/forge/verify_triage_probe_refactor.py @@ -36,14 +36,22 @@ Checks: prints the custom file's contents. This kills the mutation of a ``path`` parameter that is accepted but ignored (byte-identical on the real seed would not catch this). - 12. No import-time I/O (M6 killer): a static AST walk of the top-level - statements plus a runtime ``builtins.open`` spy both require zero calls. - Mutant M6 (``_bootstrap = json.load(open(DEFAULT_PATH))`` at module scope, - no print) survives check 5 (rc 0, zero output) by being *silent*, not - clean -- the read happened but was invisible to the silence test. - 13. M6 is detected by check 12's io-detector (non-vacuity): the detector fires - on the measured mutant applied to a temp copy, so it is sensitive rather - than theater. + 12. No import-time read (M6 killer): two instruments, both must read zero on + the clean module -- a static AST walk over the spellings + ``import_io_detector`` names, which includes the ``Path(...).read_text()``, + ``os.open(...)`` and ``io.open(...)`` forms a dotted-name list resolves to + None, and a child interpreter that imports the module under an ``open`` + audit hook, so every spelling of a real open is seen whatever the source + says. Mutant M6 (``_bootstrap = json.load(open(DEFAULT_PATH))`` at module + scope, no print) survives check 5 (rc 0, zero output) by being *silent*, + not clean -- the read happened but was invisible to the silence test. + 13. Every modelled spelling is killed by BOTH instruments (non-vacuity): eight + spawnings at module scope -- ``open``, ``json.load``, ``io.open``, + ``os.open``, ``codecs.open``, ``pathlib.read_text``, ``pathlib.read_bytes``, + ``pathlib.open`` -- each must be named by the static walk AND produce a + real ``open`` audit event, so a detector that matches nothing cannot pass. + Q89's reading, before this: four of those eight returned rc 0 with "0 + module-scope io calls; 0 runtime open calls" printed. 14. The seed path is tree-portable: no host-absolute literal in the refactored source, and ``DEFAULT_PATH`` resolves inside the tree. Closes the Q86 "a clone grades the host's inbox" gap -- on this host the resolution is @@ -65,6 +73,12 @@ import tempfile # Resolve paths from this file's location (wizard-work/forge/). WORK_DIR = os.path.dirname(os.path.abspath(__file__)) +# Q89: the import-time-read detector is shared, not copied three times. Three +# byte-identical copies of a name list is how the evasion survived `802a1ae`; +# one module with one modelled set is what the three harnesses now grade. +if WORK_DIR not in sys.path: + sys.path.insert(0, WORK_DIR) +import import_io_detector as io_det # noqa: E402 (sibling, imported after WORK_DIR) ORIGINAL = "_triage_probe.py" REFACTORED = "_triage_probe_refactored.py" MOD_NAME = REFACTORED[:-3] # strip .py -> _triage_probe_refactored @@ -395,123 +409,82 @@ def check_path_override_honoured() -> bool: # ---------------------------------------------------------------------- -# Q86: the refactor-lane harness proved SILENCE at import, not ABSENCE of I/O, -# and its seed path was host-absolute. The two fixes below close both gaps. +# Q86/Q89: the refactor-lane harness proved SILENCE at import, not ABSENCE of +# I/O, and its seed path was host-absolute. Both gaps are closed below. # -# A Call to any of these at MODULE SCOPE runs while the module is imported -- -# the M6 mutant (`_bootstrap = json.load(open(DEFAULT_PATH))`, no print) hides -# from check 5 because it emits no stdout, so it needs its own detector. -_IO_CALLS = frozenset({ - "open", "json.load", "json.loads", "pickle.load", "pickle.loads", - "csv.reader", "csv.DictReader", "urllib.request.urlopen", "codecs.open", -}) - - -def _call_name(call: ast.Call): - """Dotted name of a Call's function (``json.load``, ``open``, ...), or - None for a callable expression this detector does not model.""" - parts = [] - cur = call.func - while isinstance(cur, ast.Attribute): - parts.append(cur.attr) - cur = cur.value - if isinstance(cur, ast.Name): - parts.append(cur.id) - return ".".join(reversed(parts)) - return None - - -def _calls_at_module_scope(src: str) -> list: - """``[(lineno, name)]`` for io-reading Call nodes at module scope. - - A Call counts as "module scope" only when it is NOT nested inside a - FunctionDef / AsyncFunctionDef / ClassDef body -- those run lazily, at call - time, and are exactly the I/O the refactor is allowed to keep (it lives - under ``main()``). The ``if __name__ == "__main__"`` guard's ``main()`` - call resolves to ``main`` (not in the io set), so it is correctly ignored. - """ - tree = ast.parse(src) - found: list = [] - - def scan(node): - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): - return # do not descend -- calls inside a def body are lazy - if isinstance(node, ast.Call): - name = _call_name(node) - if name in _IO_CALLS: - found.append((node.lineno, name)) - for child in ast.iter_child_nodes(node): - scan(child) - - for stmt in tree.body: - scan(stmt) - return found +# Q89: a detector that models ONE spelling is a name list, not a detector. The +# dotted-name walk plus a `builtins.open` spy installed by `802a1ae` read zero +# on `Path(p).read_text()`, `os.open(...)` and `io.open(...)` while the module +# read a file at import. The modelled set and both instruments now live in +# `import_io_detector.py` beside this file: an AST walk that names the site, and +# a child interpreter importing under CPython's `open` audit hook, which fires +# for every spelling of a real open. The clean module is graded at check 12; the +# mutant battery at check 13 grades the instrument on each spelling it names. +# ---------------------------------------------------------------------- def check_no_import_time_io() -> bool: """Check 5 proves the import is SILENT, not that no I/O happens. M6 (`_bootstrap = json.load(open(DEFAULT_PATH))` at module scope, no print) survives check 5 (rc 0, zero output) while reading the seed at import. - Two independent detectors, both must read zero on the clean module: - (a) static: AST-walk the top-level statements for an open/json.load call; - (b) dynamic: patch builtins.open to a recorder, re-import the module, and - require zero calls. The import machinery reads .py source via - _io.FileIO (not builtins.open), so this does NOT false-positive on a - clean module -- measured separately (0 calls on a fresh import).""" - static = _calls_at_module_scope(_read(REFACTORED)) + + Two instruments, both must read zero on the clean module: + (a) static -- an AST walk of the module-scope statements over the + spellings ``import_io_detector.MODELLED_NAMES`` / + ``MODELLED_ATTRS`` name. This is what makes ``Path(p).read_text()``, + ``os.open(...)`` and ``io.open(...)`` visible at all; it reports the + line number, so a red cell can quote the site. + (b) dynamic -- ``import_audit_probe.py`` imports the module in a child + interpreter under an audit hook on CPython's ``open`` event, so a + spelling nobody modelled is still caught. The probe's own docstring + names what it filters (the import machinery's code reads) and what it + cannot see (reads below the audit layer). + Neither instrument claims "no I/O": each reports a count, and zero of both + is the pass condition.""" + static = io_det.static_reads(_read(REFACTORED)) if static: - print("FAIL: static io-call(s) at module scope: %s" % static) + print("FAIL: static read-spelling(s) at module scope: %s" % static) return False - import builtins - calls: list = [] - real_open = builtins.open - - def _spy_open(*args, **kwargs): - calls.append(args[0] if args else kwargs.get("file")) - raise RuntimeError("__check_no_import_time_io__ import-time open()") - - builtins.open = _spy_open - try: - _mod, exc, _out, _err = _safe_import() - finally: - builtins.open = real_open - if exc is not None: - # A RuntimeError carrying our marker is the very failure we reject; - # any other exception is a real import error and is ungradable here. - if isinstance(exc, RuntimeError) and "__check_no_import_time_io__" in str(exc): - print("FAIL: runtime import-time open() on %r (rc 2)" % calls) - return False - print("FAIL: import raised %s: %r (stdout=%r stderr=%r)" - % (type(exc).__name__, exc, _out, _err)) + probe = io_det.audit_reads(os.path.join(WORK_DIR, REFACTORED)) + if probe["rc"] != 0: + print("FAIL: the import-audit probe could not grade the module: %s" + % probe["error"]) return False - if calls: - print("FAIL: import made %d open() call(s): %r" % (len(calls), calls)) + if probe["import_error"]: + print("FAIL: import raised %s" % probe["import_error"]) return False - print("OK: no import-time I/O (0 module-scope io calls; 0 runtime open calls)") + if probe["events"]: + print("FAIL: %d open event(s) during import: %s" + % (len(probe["events"]), probe["events"])) + return False + print("OK: no import-time read (static 0 over %d modelled spellings; " + "audit 0 open events outside the import machinery)" + % io_det.modelled_spelling_count()) return True def check_m6_mutant_killed() -> bool: """Non-vacuity for check 12: a detector that matches nothing is theater. - Apply the measured M6 mutation -- a silent module-scope - ``json.load(open(DEFAULT_PATH))`` inserted before the first def -- to a temp - copy of the source and confirm the static io-detector fires on it. The - clean module has zero such calls (check 12); the mutant has two, so the - detector is sensitive rather than vacuous. (M6 is the mutant measured for - Q86: it returns 11/11 under the old check 5.)""" - src = _read(REFACTORED) - anchor = "def load_data(path):" - m6_expr = "_bootstrap = json.load(open(DEFAULT_PATH))\n" - mutated = src.replace(anchor, m6_expr + anchor, 1) - if m6_expr not in mutated: - # guard against the insertion anchor drifting in the source - print("FAIL: M6 mutation could not be applied (anchor %r not found)" % anchor) + + Every spelling ``import_io_detector`` names is spawned at module scope in a + copy of the refactored source, and each spawning must be killed by BOTH + instruments -- named by the static walk and caught as a real ``open`` event + by the audit hook. M6 (the measured Q86 mutant, ``json.load(open(...))``) is + the first row; ``io.open``, ``os.open`` and the three ``pathlib`` rows are + the Q89 spellings the old detector read zero on.""" + rows = io_det.mutant_matrix(os.path.join(WORK_DIR, REFACTORED)) + missed = [row for row in rows if not row["ok"]] + for row in rows: + print(" %-18s static=%-2d audit=%-2d %s" + % (row["label"], len(row["static"]), len(row["audit"]), + "killed" if row["ok"] else + "SURVIVED (rc=%s %r)" % (row["rc"], row["error"]))) + if missed: + print("FAIL: %d/%d spelling mutant(s) survived an instrument: %s" + % (len(missed), len(rows), [row["label"] for row in missed])) return False - sites = _calls_at_module_scope(mutated) - if not sites: - print("FAIL: M6 mutant NOT detected by the io-detector (check 12 is vacuous?)") - return False - print("OK: M6 mutant detected at module scope: %s" % sites) + print("OK: %d/%d import-time-read spellings killed by both instruments " + "(static AST + open-audit hook)" % (len(rows), len(rows))) return True diff --git a/wizard-work/volt/CHECK-q89 b/wizard-work/volt/CHECK-q89 new file mode 100755 index 0000000..184343e --- /dev/null +++ b/wizard-work/volt/CHECK-q89 @@ -0,0 +1,19 @@ +#!/bin/sh +# Q89 CHECK -- the import-time-read detector, graded on the spellings it names +# and on one no list can name. +# +# rc 0 = every leg as expected; 1 = a graded fault; 2 = could not grade. +# The work is done by wizard-work/volt/q89_verify.py beside this file; it copies +# the forge lane and the seed it reads into a temp root, so the tree this +# instrument ships in is never mutated. Run it from any checkout. +set -u +REPO="$(cd "$(dirname "$0")/../.." && pwd)" || exit 2 +if ! cd "$REPO"; then + echo "q89: could not grade -- cannot cd to $REPO" + exit 2 +fi +if ! command -v python3 >/dev/null 2>&1; then + echo "q89: could not grade -- no python3 on PATH" + exit 2 +fi +exec python3 wizard-work/volt/q89_verify.py diff --git a/wizard-work/volt/q89_verify.py b/wizard-work/volt/q89_verify.py new file mode 100644 index 0000000..e686ca6 --- /dev/null +++ b/wizard-work/volt/q89_verify.py @@ -0,0 +1,272 @@ +#!/usr/bin/env python3 +"""Q89: grade the import-time-read detector on every spelling it names, and on +one spelling no list can name. + +What it grades, in legs: + + 1. clean the three refactor harnesses are green on the tree as committed + and print their own totals (14/14, 14/14, 13/13 today). + 2. evasion the four Q89 spellings -- ``Path(...).read_text()``, + ``os.open(...)``, ``io.open(...)``, ``Path(...).open()`` -- are + appended at module scope to each harness's module in turn and the + harness must go red (rc 1) naming the site. Before the repair all + four read rc 0 with "0 module-scope io calls; 0 runtime open + calls" printed. + 3. control a module-scope ``len(DEFAULT_PATH)`` is not a read: it must stay + green, so the detector is not a constant-red. + 4. blinding the two instruments are disabled in turn. With the static walk + blinded the audit hook must be the one that reds check 12; with + the audit hook blinded the static walk must. With both blinded + check 12 prints OK and the mutant battery (check 13) must still + red the harness -- a detector cannot be emptied quietly. + 5. no-list ``_F = open`` / ``_F(DEFAULT_PATH)`` resolves to a name no set + contains; the audit hook must catch the real open event anyway. + This is the difference between a longer list and a closed class. + 6. fourth the fourth copy of this detector (``verify_dump_seed_refactor.py``, + forge's own later hardening) still passes 16/16 and the shared + walk agrees with its 12-spelling corpus, so this module is a + superset rather than a third convention. + +Everything runs in a temporary copy of ``wizard-work/forge`` beside a copy of +``inbox/seed-2026-09-26.json``; the tree the instrument ships in is never +mutated, which is what lets it grade in one checkout and be run in another. + +rc 0 = every leg as expected; 1 = a graded fault; 2 = could not grade. +""" +from __future__ import annotations + +import importlib.util +import json +import os +import shutil +import subprocess +import sys +import tempfile + +ROOT = os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(__file__)))) +SEED = os.path.join("inbox", "seed-2026-09-26.json") +SEED_SHA_PIN = None # the seed is copied, not pinned: its bytes are not graded + +HARNESSES = ( + ("probe", "verify_triage_probe_refactor.py", "_triage_probe_refactored.py"), + ("dump", "verify_triage_dump_refactor.py", "_triage_dump_refactored.py"), + ("pending", "verify_triage_dump_pending_refactor.py", + "_triage_dump_pending_refactored.py"), +) +FOURTH = ("verify_dump_seed_refactor.py", "_dump_seed_refactored.py") + +EVASIONS = { + "pathlib.read_text": + "\nfrom pathlib import Path as _P\n_evade = _P(DEFAULT_PATH).read_text()\n", + "os.open": + "\nimport os as _o\n_fd = _o.open(DEFAULT_PATH, _o.O_RDONLY)\n_o.close(_fd)\n", + "io.open": + "\nimport io as _io\n_h = _io.open(DEFAULT_PATH)\n_h.close()\n", + "Path.open": + "\nfrom pathlib import Path as _P\nwith _P(DEFAULT_PATH).open() as _fh:\n" + " _evade = _fh.read()\n", +} +CONTROL = "\n_neg = len(DEFAULT_PATH)\n" +NO_LIST = "_F = open\n_h = _F(DEFAULT_PATH)\n_h.close()\n" +BLIND_STATIC = "\nMODELLED_NAMES = frozenset()\nMODELLED_ATTRS = frozenset()\n" +BLIND_AUDIT = ('\n\ndef audit_reads(module_path, timeout=120):\n' + ' return {"rc": 0, "events": [], "import_error": None,\n' + ' "error": None}\n') + + +class Report: + def __init__(self) -> None: + self.faults = 0 + self.legs = 0 + + def ok(self, leg: str, note: str) -> None: + self.legs += 1 + print("ok %-22s %s" % (leg, note)) + + def fail(self, leg: str, note: str) -> None: + self.legs += 1 + self.faults += 1 + print("FAIL %-20s %s" % (leg, note)) + + +def run(forge: str, harness: str): + proc = subprocess.run(["python3", harness], cwd=forge, capture_output=True, + text=True, timeout=300) + out = proc.stdout or "" + total = next((l.strip() for l in out.splitlines() + if l.startswith("=== Verification")), "") + first_fail = next((l.strip() for l in out.splitlines() + if l.strip().startswith("FAIL")), "") + # The io detector's own verdict, wherever it lands: a harness whose earlier + # checks red first (the dump lane's dead-import check shadows it) must still + # be graded on whether the io detector fired, not on check order. + io_fail = next((l.strip() for l in out.splitlines() + if l.strip().startswith("FAIL") + and ("static read-spelling" in l or "open event(s)" in l)), "") + return proc.returncode, total, first_fail, io_fail, out + + +def load(name: str, path: str): + spec = importlib.util.spec_from_file_location(name, path) + if spec is None or spec.loader is None: + raise SystemExit("cannot load %s" % path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def build_copy(base: str) -> str: + """Copy the forge lane and the seed the modules read; return the copy's forge.""" + work = os.path.join(base, "tree") + forge = os.path.join(work, "wizard-work", "forge") + os.makedirs(os.path.dirname(forge)) + shutil.copytree(os.path.join(ROOT, "wizard-work", "forge"), forge, + ignore=shutil.ignore_patterns("__pycache__", ".pytest_cache")) + os.makedirs(os.path.join(work, "inbox")) + shutil.copy2(os.path.join(ROOT, SEED), os.path.join(work, SEED)) + return forge + + +def with_suffix(forge: str, name: str, suffix: str, fn): + """Run ``fn`` with ``suffix`` appended to one module, then restore the bytes.""" + path = os.path.join(forge, name) + with open(path, encoding="utf-8") as handle: + original = handle.read() + try: + with open(path, "w", encoding="utf-8") as handle: + handle.write(original + suffix) + return fn() + finally: + with open(path, "w", encoding="utf-8") as handle: + handle.write(original) + with open(path, encoding="utf-8") as handle: + if handle.read() != original: + raise SystemExit(2) + + +def main() -> int: + if not os.path.isdir(os.path.join(ROOT, "wizard-work", "forge")): + print("FAIL tree: %s is not this instrument's repo" % ROOT) + return 2 + report = Report() + base = tempfile.mkdtemp(prefix="q89-") + try: + forge = build_copy(base) + + # (1) clean + totals = {} + for label, harness, _module in HARNESSES: + rc, total, fail, _io_fail, _out = run(forge, harness) + totals[label] = total + if rc == 0 and total: + report.ok("clean " + label, total.replace("=== Verification: ", "")) + else: + report.fail("clean " + label, "rc=%d %s %s" % (rc, total, fail)) + + # (2) evasion: every spelling must red every harness + for label, harness, module in HARNESSES: + for spelling, body in EVASIONS.items(): + rc, _total, fail, io_fail, _out = with_suffix( + forge, module, body, lambda: run(forge, harness)) + if rc == 1 and io_fail: + report.ok("%s %s" % (label, spelling), "rc=1, %s" % io_fail[:60]) + else: + report.fail("%s %s" % (label, spelling), + "rc=%d, io detector said %r (first FAIL %r)" + % (rc, io_fail[:50], fail[:50])) + + # (3) the negative control is not a read + for label, harness, module in HARNESSES: + rc, _total, fail, _io_fail, _out = with_suffix( + forge, module, CONTROL, lambda: run(forge, harness)) + if rc == 0: + report.ok("control " + label, "len() at module scope stays green") + else: + report.fail("control " + label, "rc=%d %r" % (rc, fail[:80])) + + # (4) each instrument is load-bearing, and the battery is the tripwire + probe_module = HARNESSES[0][2] + probe_harness = HARNESSES[0][1] + + def _blinded(static_patch: str, audit_patch: str): + detector = os.path.join(forge, "import_io_detector.py") + with open(detector, encoding="utf-8") as handle: + saved = handle.read() + + def go(): + with open(detector, "w", encoding="utf-8") as handle: + handle.write(saved + static_patch + audit_patch) + return run(forge, probe_harness) + + try: + return with_suffix(forge, probe_module, EVASIONS["pathlib.read_text"], go) + finally: + with open(detector, "w", encoding="utf-8") as handle: + handle.write(saved) + + rc, _t, fail, io_fail, out = _blinded(BLIND_STATIC, "") + if rc == 1 and io_fail and "open event(s) during import" in io_fail: + report.ok("blinded static", "audit hook alone reds check 12") + else: + report.fail("blinded static", "rc=%d %r" % (rc, io_fail[:90])) + + rc, _t, fail, io_fail, out = _blinded("", BLIND_AUDIT) + if rc == 1 and io_fail and "static read-spelling" in io_fail: + report.ok("blinded audit", "static walk alone reds check 12") + else: + report.fail("blinded audit", "rc=%d %r" % (rc, io_fail[:90])) + + rc, _t, fail, io_fail, out = _blinded(BLIND_STATIC, BLIND_AUDIT) + check12_clean = ("OK: no import-time read" in out + and "modelled spellings" in out) + battery_red = "spelling mutant(s) survived" in out + if rc == 1 and check12_clean and battery_red: + report.ok("blinded both", "check 12 green, check 13 battery reds it") + else: + report.fail("blinded both", + "rc=%d check12_clean=%s battery_red=%s" + % (rc, check12_clean, battery_red)) + + # (5) a spelling no list names + rc, _t, fail, io_fail, out = with_suffix( + forge, probe_module, NO_LIST, lambda: run(forge, probe_harness)) + if rc == 1 and io_fail and "open event(s) during import" in io_fail: + report.ok("no-list spelling", "_F = open caught by the audit hook") + else: + report.fail("no-list spelling", "rc=%d %r" % (rc, io_fail[:100])) + + # (6) the fourth copy still passes, and the shared walk is a superset + rc, total, fail, _io_fail, _out = run(forge, FOURTH[0]) + if rc == 0 and "16/16" in total: + report.ok("fourth copy", total.replace("=== Verification: ", "")) + else: + report.fail("fourth copy", "rc=%d %s %s" % (rc, total, fail)) + detector = load("q89_detector", + os.path.join(forge, "import_io_detector.py")) + fourth = load("q89_fourth", os.path.join(forge, FOURTH[0])) + with open(os.path.join(forge, fourth.REFACTORED), encoding="utf-8") as handle: + source = handle.read() + missed = [label for label, snippet in fourth._IO_SPELLINGS + if not detector.static_reads(source + "\n" + snippet + "\n")] + if not missed: + report.ok("corpus agreement", + "%d/%d of the fourth's spellings flagged" + % (len(fourth._IO_SPELLINGS), len(fourth._IO_SPELLINGS))) + else: + report.fail("corpus agreement", "shared walk missed %s" % missed) + except (OSError, subprocess.SubprocessError, SystemExit) as exc: + print("FAIL instrument: %s: %s" % (type(exc).__name__, exc)) + return 2 + finally: + shutil.rmtree(base, ignore_errors=True) + + if report.faults: + print("q89: FAULT -- %d leg line(s) red of %d graded" % (report.faults, report.legs)) + return 1 + print("q89: %d leg line(s), 0 fault -- OK" % report.legs) + return 0 + + +if __name__ == "__main__": + sys.exit(main())