Merlin
8cb941fc26
The subject's source is now read the way CPython reads it: bytes through tokenize.detect_encoding (BOM stripped, PEP 263 cookie honoured), with a dated exit-2 refusal for a file that cannot be read that way, at Q46's guard point — after Q35's precondition, before the admission probe, before any leg, before any sandbox. All four subject-source reads route through the one reader; one stderr NOTE when the encoding is not plain utf-8; the Q46 parse guard, the Q45 redirect, every check() and the 169 count untouched. Deliverable recoveries/helm-read-path-guard.md (12,821 B), scratch tree /root/crucible/q47/, evidence out/Q47-NEGATIVES.json (21 rows), out/Q47-VERDICT.json, out/Q47-GREEN.json, out/DIFF-vs-q46.txt. Measured, not asserted: latin-1 cookie subject rc 0 / 169 passed / ALL GATES PASS under the shipped gate (Q39-Q45 crash with UnicodeDecodeError); BOM'd collector rc 0 / 169 passed (Q43/Q45 SyntaxError at the sandbox copy, Q44 AttributeError, sandboxes left); genuinely undecodable subject rc 2 with the dated Q47 REFUSAL and no sandbox. Q39 runs the BOM subject green with 164 passed — the read-path class is the text-compiling gates (Q43-Q46), not subject handling generally. Green pair 5x rc 0, 169/0, 13,312 B stdout / 425 B stderr, live and held delta 0 on all 26 invocations. Adoption pair moves to collector 4e45021d + gate 76f2292b. Nothing deployed, no cp into /root/.helm/bin/, decisions/ and inbox/ untouched, no another-party file edited, no secret. Run 13 in the check-in; queue block appended.
162 lines
13 KiB
Markdown
162 lines
13 KiB
Markdown
# Q47 — the read path: a subject the interpreter can run must neither crash the gate nor be falsely refused
|
||
|
||
Merlin, 2026-10-13. Helm lane. Scratch tree `/root/crucible/q47/`; nothing deployed, no `cp` into
|
||
`/root/.helm/bin/`, `decisions/` and `inbox/` untouched, no another-party file edited, no secret.
|
||
Item opened by the pipe on reply 30's C1 (2026-10-13 01:58Z), measured there on the shipped Q46 gate.
|
||
|
||
**The defect.** Two directions, one cause. (a) A subject with `# -*- coding: latin-1 -*-` and one
|
||
latin-1 byte — bytes-compile OK, `python3 <fixture>` rc 0, so the **interpreter runs it** — crashed
|
||
the gate at `COLLECTOR.read_text()`: UTF-8 decode raises `UnicodeDecodeError`, rc 1, a traceback, no
|
||
dated refusal. (b) The real collector with a UTF-8 BOM — bytes-compile OK, because CPython strips a
|
||
BOM when it reads a source file — was **falsely refused** by the Q46 guard: the decoded text still
|
||
carries U+FEFF and `compile()` calls it an invalid non-printable character. The cause in both cases:
|
||
the gate reads a decoded `str`, the interpreter compiles the file's **bytes**.
|
||
|
||
## 1. The pair
|
||
|
||
Collector **byte-identical to Q46's** (`4e45021d80395d3f…`, 720 lines / 29,954 B) — measured by
|
||
sha256, not asserted. Every change is in the gate.
|
||
|
||
| | Q46 | Q47 |
|
||
|---|---|---|
|
||
| sha256 | `ae69d5dd735b6f3cfe82b3682b02acf4557b2939d104b9f15b81cffaeef016d4` | `76f2292b9a6cad08505956787cf05e65e3edaf7e0f23f73f4aed2e59a62866c1` |
|
||
| bytes | 110,509 | 115,182 |
|
||
| lines | 2,032 | 2,124 |
|
||
|
||
`out/DIFF-vs-q46.txt` (fresh `diff -u`): **130 lines = 2 file headers + 4 hunk headers + 96 added +
|
||
4 removed + 24 context**, and 2,032 + 96 − 4 = **2,124** exactly. The decomposition convention is
|
||
the lane rule: added = `^+` lines minus the `+++` header, removed = `^-` minus `---`.
|
||
|
||
## 2. What changed — one reader, four read sites, one refusal
|
||
|
||
1. **`_q47_subject_text(path)`** reads the subject the way CPython reads a source file: the file's
|
||
**bytes** through `tokenize.detect_encoding` (BOM first, then the PEP 263 cookie, then utf-8),
|
||
decoding with `utf-8-sig` when a BOM is present so the BOM is stripped exactly as the
|
||
interpreter strips it. A non-file path returns `("", "utf-8")`, the shape Q35's marker check
|
||
expects. An unreadable cookie or bytes that do not decode raise `_Q47ReadError`.
|
||
2. The **dated exit-2 refusal** sits at the same point as Q46's guard — after Q35's source
|
||
precondition, before the admission probe, before any leg, before any sandbox is created — with
|
||
the live snapshot read before and after and the delta printed. A file that cannot be read the
|
||
CPython way cannot be run by the interpreter either, so refusal is the honest shape.
|
||
3. **All four subject-source reads are routed through the one reader**: the Q35 precondition's
|
||
`_q35_src` (line 210), the admission's `_q39_src_text` (was line 361), the G8 source read (was
|
||
1704), and the G8 source-guard check (was 1971). No `read_text()` on the subject survives.
|
||
4. One **`Q47 NOTE` on stderr, never stdout, and only when the subject's encoding is not plain
|
||
utf-8** — so a green pair's stdout shape is unchanged and the positive side of a read-path fix
|
||
is visible rather than inferred (the Q38 precedent).
|
||
5. The **Q46 parse guard, the Q45 redirect, every `check()` and the suite count are untouched**: the
|
||
guard compiles the same `_q35_src` it always did — it now compiles what the interpreter would
|
||
compile. Green stays **169 passed / 0 failed**; the guard registers no check.
|
||
|
||
## 3. The fixtures, and the interpreter's own verdict (measured, not assumed)
|
||
|
||
`make_fixtures.py` builds the three from the **real collector** (digest asserted against
|
||
`4e45021d…`), each construction asserting its own counts:
|
||
|
||
| fixture | construction | `python3 <fixture> --skip-tests` (the ground truth) |
|
||
|---|---|---|
|
||
| `collector-latin1.py` (30,028 B / `6ce8bf90292d83f1…`) | cookie on line 1, one `0xe9` in a trailing comment | **rc 2** — the destination rule refuses; the file **runs** |
|
||
| `collector-bom.py` (29,957 B / `d14906c868e24997…`) | UTF-8 BOM prepended, no cookie | **rc 2** — the destination rule refuses; the file **runs** |
|
||
| `collector-undecodable.py` (30,006 B / `47fc3580fdfe8308…`) | one `0xe9` in a trailing comment, **no** cookie | **rc 1** — `SyntaxError: Non-UTF-8 code starting with '\xe9'` |
|
||
|
||
## 4. The discriminator — fifteen invocations, five gates
|
||
|
||
`run_negatives_q47.py` → `out/Q47-NEGATIVES.json` (57,247 B, 21 rows, `generated_at
|
||
2026-10-13T03:46:55.002706+00:00`); verdict in `out/Q47-VERDICT.json` (recomputed from the rows by
|
||
`recompute_verdict.py` — see §7 defect 1). Legacy gates digest-verified against the Q46 record
|
||
(`9b09f328…`, `c22601ec…`, `18580c10…`, `2bbca218…`).
|
||
|
||
| fixture | Q39 | Q43 | Q44 | Q45 | **Q47 (shipped)** |
|
||
|---|---|---|---|---|---|
|
||
| `collector-latin1.py` | rc 1 — `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 30026` (0.16 s, no sandbox) | rc 1 — same (0.13 s) | rc 1 — same (0.15 s) | rc 1 — same (0.14 s) | **rc 0, 169 passed / 0 failed, `VERDICT: ALL GATES PASS`, 13,312 B stdout, 12.44 s**, `Q47 NOTE … (iso-8859-1)`, no leftover |
|
||
| `collector-bom.py` | **rc 0, 164 passed, ALL GATES PASS, 11,398 B, 10.12 s** — Q39 runs the BOM subject **green** (see §6) | rc 1 — `SyntaxError: invalid non-printable character U+FEFF` at the sandbox copy's import (`\ufeff"""The Helm — P0 collector.`), 575 B, **`legacy/q43/helm-legs-_sja2yko` left** (0.61 s) | rc 1 — `AttributeError: 'NoneType' object has no attribute 'body'` (`_q39_tree.body` on a BOM'd parse), 0 B, **`legacy/q44/helm-legs-sl1v6yx5` left** (0.22 s) | rc 1 — `SyntaxError: … U+FEFF` at the sandbox copy, 575 B, **`legacy/q45/helm-legs-8qmmr11u` left** (0.66 s) | **rc 0, 169 passed / 0 failed, ALL GATES PASS, 13,312 B, 12.22 s**, `Q47 NOTE … (utf-8-sig)`, no leftover |
|
||
| `collector-undecodable.py` | rc 1 — `UnicodeDecodeError … position 30004` (0.14 s) | rc 1 — same (0.14 s) | rc 1 — same (0.19 s) | rc 1 — same (0.14 s) | **rc 2, stdout 0 B (`e3b0c44298fc1c149…`), dated `Q47 REFUSAL: … cannot be read the way CPython reads a source file - 'utf-8' codec can't decode byte 0xe9 in position 30004: invalid continuation byte - refused at 2026-10-13T03:47:32.685519+00:00, before the admission probe, before any leg, and before any sandbox is created.`** — no traceback, no leftover (0.13 s) |
|
||
|
||
The item's four done-criteria, as measured: the latin-1 subject **proceeds past the guard and runs
|
||
the whole suite green**; the BOM subject is **not refused** and runs green; a genuinely undecodable
|
||
subject is a **dated exit-2 refusal**; the four pre-Q46 gates' rcs are measured on both fixtures
|
||
above (rc 1 crashes — frozen controls, not patched); **green still 169 passed / 0 failed**; **no
|
||
sandbox left**. `live_delta` = 0 and `live_held_delta` = 0 on **all 21 rows**.
|
||
|
||
## 5. Regression controls, all still refused
|
||
|
||
- `q47-q46reg-noparse`: the Q46 class re-measured under the Q47 gate — **rc 2**, the dated
|
||
`Q46 REFUSAL` unchanged (`invalid syntax (noparse-with-markers.py, line 13) (line 13, offset 12)`).
|
||
The read-path fix did not loosen the parse guard.
|
||
- `q47-neg-n1…n5`: all five Q39 negative fixtures **rc 2**, live delta 0 — the admission machinery
|
||
is untouched. (n5's refusal line is the Q39 NOTE plus the leg's own
|
||
`refusing to write the live snapshot … without --deploy`, rc 2, 575 B.)
|
||
|
||
## 6. The green pair, and the stdout identity
|
||
|
||
`run_green_q47.py` → `out/Q47-GREEN.json` (2,725 B), logs `out/q47-green-{1..5}.{out,err}`: five
|
||
consecutive **rc 0**, **169 passed / 0 failed**, `VERDICT: ALL GATES PASS`, **13,312 B stdout /
|
||
425 B stderr** each, 12.03–12.58 s, live delta 0, held delta 0, no leftover. Green stdout digests:
|
||
`21920260cade855a…` / `580021754ed55d3d…` / `49025d65cc3082e5…` / `c8155f076d214d24…` /
|
||
`1d2c7d69a8531d39…`.
|
||
|
||
`out/q47-green-1.out` vs `out/../q46/out/q46-green-1.out`: **184 lines each, exactly 4 differ**, and
|
||
all four are the randomized names — line 1 (`/tmp/tmpkhag_ezo/` vs `/tmp/tmpl46pgy2a/`), line 10
|
||
(the `q31-g8-deploy` temp dir) and lines 183–184 (the two sandbox-path lines, which name the tree).
|
||
So the Q47 gate's green output is **byte-identical to Q46's but for the random names** — the read
|
||
path changed nothing the suite prints.
|
||
|
||
**The Q39-bom row is a finding, not noise.** The oldest gate runs the BOM subject **green with 164
|
||
passed** while Q43/Q44/Q45 crash on it. The mechanism is visible in its own report:
|
||
`parse line -: the subject does not parse: invalid non-printable character U+FEFF` — Q39 records the
|
||
parse failure and continues, because **Q39 never compiles the subject's text**; it only imports the
|
||
file (where CPython strips the BOM) and runs the probe copy as a subprocess (whose SyntaxError the
|
||
Q46-era crash detector cannot see). The read-path class is therefore a property of the gates that
|
||
compile the subject's **text** — Q43/Q45's sandbox-copy import, Q44's `ast` — not of subject
|
||
handling generally. The BOM subject is runnable by the interpreter and green on the oldest gate; it
|
||
is Q43–Q46's text compilation that made it dangerous.
|
||
|
||
## 7. Limits and defects — mine first, logged not deleted
|
||
|
||
1. **Harness verdict bug (mine).** `row_for()` was passed `READPATH_FIXTURES` (which carry `.py`)
|
||
while the discriminator loop names rows with the suffix stripped, so the verdict's generator
|
||
raised `StopIteration` → `RuntimeError` **after** the rows JSON was written. The rows are all on
|
||
disk; the verdict was recomputed from them by `recompute_verdict.py`, and `run_negatives_q47.py`
|
||
is patched (the strip) so the file on disk is right. No gate invocation was re-run.
|
||
2. **Vacuous fixture-dest census (mine).** The negative fixtures' destination literals still name
|
||
the **q46** tree — `cp -a` inherited fixtures already retargeted by q46's `setup_fixtures.py` —
|
||
so this run's `fixture_dest_census()` (which watches `q47/fixtures`) was empty on every row.
|
||
Evidence no cross-tree write happened: `ls /root/crucible/q46/fixtures` shows **no**
|
||
`n1-dest…n5-dest`/`dep-dest` directories, and each negative row's stderr shows the leg refusing
|
||
its own destination. The right fix was to re-run `setup_fixtures.py` retargeted into
|
||
`q47/fixtures`.
|
||
3. **The copies are UTF-8 by construction (recorded limit).** The probe copy and the legs' sandbox
|
||
copy prepend their own header lines, which push the subject's cookie past PEP 263's line-2
|
||
limit, so a cookie subject's non-ASCII bytes in **executable strings** would be re-encoded as
|
||
UTF-8 in the copies. The fixture therefore carries its single latin-1 byte in a comment only. A
|
||
copy faithful in the subject's own encoding would need the cookie hoisted — not built.
|
||
4. **The guard is static** (carried from Q46): runtime-generated code, or a source produced by an
|
||
import hook, is out of every static guard's reach.
|
||
5. `tokenize.detect_encoding` reads the first two lines only — **faithful**, not a limit: CPython
|
||
does not honour a cookie past line 2 either.
|
||
6. Byte counts, hashes and timings are dated readings (2026-10-13 ~03:30–04:00Z).
|
||
|
||
## 8. Live state, and the §4 consequence
|
||
|
||
Censused on (bytes, sha256, mtime_ns, ctime_ns) before and after **all 26 invocations** (21 harness
|
||
rows + 5 green runs), delta 0 on every field: snapshot `/root/.helm/data/snapshot.json` **14,039 B /
|
||
`aa842b0fd88037f83115b4a54d617f27666f4de06ef84f9a64c8fd0a5f04656d`** / mtime_ns
|
||
`1791479579475780303` / ctime_ns `1791789490146606356`; held pin
|
||
`/root/.helm/data/held/stackchain-home.html.sha256` **120 B / `5b6d70716153c3f4…`** / mtime_ns
|
||
`1791478687099553071` / ctime_ns `1791846636217687791` (that ctime is the Q45 restore's trace, as
|
||
recorded there). Deployed bin untouched: `helm-selftest.py` **33,647 B / `7ed1bca8f9da624f…`**,
|
||
`helm-collect.py` `ee0b36a82f23fcf3…`, `helm-render.py` `5c67473d89283c3d…`, `helm-validate.py`
|
||
`12570786ba703787…`, `summarize.py` `7386999867789694…`. Final tree: **0 sandboxes**.
|
||
|
||
**The adoption pair is now collector `4e45021d…` + gate `76f2292b…`** (superseding `ae69d5dd…`).
|
||
Q40's finding is unchanged and is why the `cp` matters: the deployed gate `7ed1bca8…` reads
|
||
`HELM_COLLECTOR` **zero** times, so it has no subject-selection path and none of Q31–Q47's
|
||
protection is in production until the pair lands. The `cp` stays Alexander's.
|
||
|
||
## 9. Q7, measured
|
||
|
||
After this item the Helm lane's executable set is empty again: the adoption `cp`, Q9 (checksum
|
||
handover), Q13/Q19 and Q42 are human- or counterparty-gated. Merlin's Q45 §9 follow-on — resolve
|
||
every path the subject declares **at runtime**, not only the literals it declares statically —
|
||
stays open and unclaimed beside the adoption. **No new item opened.**
|