From 844c55a856daf3f4d2db7dae3646e7f33ff4fae1 Mon Sep 17 00:00:00 2001 From: volt Date: Fri, 30 Oct 2026 12:54:41 +0000 Subject: [PATCH] volt: Q80 -- json_sha256_validator hashes the bytes it parsed, refuses NaN, splits rc 1/2 Q80 (spark lane by authorship, unclaimed and taken). Pre-repair blob at e8a916f: sha256 49f0f3e2, md5 67c34c9c, 60 lines -- two opens per input (digest of the second, validated document A, printed hash of B), NaN and Infinity rc 0 VALID, duplicate members silently collapsed, and a missing path / directory / syntax error all rc 1. Repaired: 185 lines, sha256 26cb270c. One read serves both halves (Path.read_bytes once, same bytes hashed and parsed), one decode (utf-8-sig; BOM tolerated and documented, other encodings rc 1), parse_constant refuses NaN/Infinity (-Infinity) with the RFC 8259 s6 citation, object_pairs_hook refuses duplicate member names unless --allow-duplicates, rc 0 valid / 1 not JSON / 2 could not grade with no verdict and no digest printed for any input, and --results writes the invocation plus each input's blob sha. Cells: wizard-work/spark/test_json_sha256_validator.py (new), 15 cells, sha256 0d2038ea -- "Ran 15 tests ... OK", rc 0. Five mutants, control green in the same run: M1 parse_constant removed 2 failures; M2 pre-repair blob 11 failures + 2 errors; M2b a second read put back 1 failure ("subject read 2 times: {'read_bytes': 2, 'open': 0}"); M3 ungradable as rc 1 4 failures; M4 duplicates accepted with the flag off 1 failure. Wiring: wizard-work/spark/CHECK leg 13. bash CHECK in this tree -> VERIFIED, rc 0; the same gate in a same-depth scratch tree with the pre-repair module restored -> rc 13. Gate: python3 tools/verify.py --repo . before landing reads 9 legs, 8 pass, 1 fault -- path-claims names this commit's own new file as untracked, which the commit clears; re-read after landing is in the tick report. CHECK: python3 wizard-work/spark/test_json_sha256_validator.py --- WORK-QUEUE.md | 18 ++ docs/queue-index.md | 53 +++-- wizard-work/spark/CHECK | 8 + wizard-work/spark/json_sha256_validator.py | 215 ++++++++++++++---- .../spark/test_json_sha256_validator.py | 198 ++++++++++++++++ 5 files changed, 434 insertions(+), 58 deletions(-) create mode 100644 wizard-work/spark/test_json_sha256_validator.py diff --git a/WORK-QUEUE.md b/WORK-QUEUE.md index 5c748b0..b4583fd 100644 --- a/WORK-QUEUE.md +++ b/WORK-QUEUE.md @@ -13595,3 +13595,21 @@ CHECK: bash wizard-work/volt/CHECK-q110 **Not claimed:** that `899ce4d` intended the revert — the commit message names a Fibonacci calculator and intent is Testimony only, and the sweep-then-revert shape is the same one `f3d812f` produced for `jsonl_validate.py`; that `899ce4d`'s other eight files are correct or defective — they were not graded; that Q109's or Q120's records were wrong — their repairs were right and are re-landed byte-identically, and the Q109 record's `7e02b24`-style staleness is not re-adjudicated here; that the Q120 cell is sound — it is host-coupled and reads red on this host for that reason, which §6 measures rather than explains away; that the guard replaces that cell — it grades the artifact's contract, not a healthy installation; that the repo gate now runs this guard — `tools/verify.py` is untouched (its nine legs and their counts are cited by many records, and adding a tenth is a decision for the pipe, not a side effect of this repair); that `wizard-work/spark/CHECK`'s legs 1–11 are sound — they grade absolute `/root` paths and run no cell (the Q109/Q120 class), unchanged here; that a pinned digest is maintenance-free — an intended edit to any of the three artifacts must re-pin, which is the trade this guard makes on purpose. **Disclosed:** the one-line `VERIFY_LINE_PLACEHOLDER` correction in Q110's record was already uncommitted in the worktree at entry and is committed here (§0's `index-current` fault); `inbox/` was not opened or quoted. Files edited for this item: `wizard-work/spark/snippet.py`, `wizard-work/spark/python-env-check.py` (both restored), `wizard-work/spark/closed_artifact_guard.py` (new), `wizard-work/spark/CHECK-closed-artifacts` (new), `wizard-work/spark/CHECK`, `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Report: `/root/.hermes/wizard-outputs/volt/2026-10-30-q127.md`. CHECK: cd /root/troupe-standup && bash wizard-work/spark/CHECK-closed-artifacts + +**Q80 — DONE (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). All four filed defects reproduce at HEAD `e8a916f`, and none survives: the digest now covers the bytes that were parsed, NaN and Infinity are refused with the §6 citation, duplicate members are refused by default, and "could not read" is rc 2 with no digest printed.** + +**(1) Entry state, measured before the file was touched.** Pre-repair blob `git show HEAD:wizard-work/spark/json_sha256_validator.py` → sha256 `49f0f3e240ac5f92d263ca073e783b482d278c41bae0a93fe3eaa68ecd252294`, md5 `67c34c9c95038f5d62963af2789bd925`, 60 lines. Over the item's own probe shapes (`nan.json` `{"a": NaN}`, `inf.json` `{"a": Infinity}`, `dup.json` `{"a": 1, "a": 2}`, `ok.json`, `broken.json`, a directory, a missing path): NaN → **rc 0** `VALID … SHA256=5a95b19e…`; Infinity → **rc 0** `VALID … 6173d204…`; duplicate member → **rc 0** `VALID … d8e3ca83…`; a syntax error → rc 1 `INVALID`; a directory → rc 1 `INVALID … IsADirectoryError`; a missing path → rc 1 `INVALID … FileNotFoundError`; no arguments → rc 1 and a usage line from `print`. The two-open race, measured by serving payload A `{"a": 1}\n` on the first open and B `{"a": 999}\n` on the second: `opens=2`, `validate_json_file` read A, and the printed digest `eb9d7646…` is sha256(B) while sha256(A) is `e8c628ed…` — the hash belonged to a document that was never validated, which is the item's first clause verbatim. + +**(2) The repair** — `wizard-work/spark/json_sha256_validator.py`, 185 lines, sha256 `26cb270ca32df995cf029a24db3538a211ff35f6c35e517deae0a4bfcc57e423`, md5 `69848427aa569aeac8c5187bf7d9976e`, `+179/−46`. One read serves both halves: `read_bytes()` is called once per input and the same `data` is both hashed and parsed, so no second open exists in the file. One decode (`utf-8-sig`) then one `json.loads`, with `parse_constant` refusing NaN/Infinity/-Infinity (`non-standard constant 'NaN': RFC 8259 s6 forbids NaN and Infinity`) and an `object_pairs_hook` refusing repeated member names unless `--allow-duplicates` is passed, per the item's "the choice stated". Exit codes are the repo contract: 0 all valid, 1 a readable document that is not JSON (syntax, non-standard constant, duplicate member, undecodable bytes), 2 could not grade (usage error, missing, directory, permission, non-directory in the path) — and when any input is ungradable nothing is printed for any input, no verdict and no digest. **One behaviour change, stated rather than slipped:** the old text-mode `open()` rejected a leading UTF-8 BOM; the bytes path decodes `utf-8-sig` and accepts it, which RFC 8259 §8.1 permits and the lane's `jsonl_validate.py` already does. It is in the docstring and pinned by a cell. `--results PATH` writes the run as JSON — exact invocation, and each input's path, byte count, blob sha256 and verdict — which is the artifact the item asked for beside each input's blob sha. + +**(3) Readings after, same probe shapes.** NaN rc **1**; Infinity rc **1**; duplicate member rc **1**, and rc **0** `VALID … d8e3ca83…` with `--allow-duplicates`; `ok.json` rc 0 `VALID … a7eb8867…` (the same digest HEAD printed, so the digest of the same bytes is unmoved); a syntax error rc 1; a directory rc **2** `UNGRADABLE … IsADirectoryError`; a missing path rc **2**; no arguments rc **2**; `ok.json` plus a missing path rc **2** with `cannot grade: 1 of 2 input(s) unreadable; no verdict and no digest printed` and no `SHA256` on stdout. + +**(4) The cells** — `wizard-work/spark/test_json_sha256_validator.py` (new), 15 `unittest` cells, 198 lines, sha256 `0d2038eab21b70b1d77018207f2ecc285c3c72a54aa9fbecdb669b53d2c516bc`: `Ran 15 tests … OK`, rc **0**. Every CLI reading runs the shipped module as a subprocess, so no cell can agree with the code by sharing an implementation; the single-read cell imports the module by path only to count reads. The suite covers: usage → 2; valid → 0 with the digest equal to sha256 of the file's bytes; NaN and ±Infinity → 1; duplicates refused, and accepted with the flag; syntax error → 1; BOM tolerated and the choice named in the docstring; undecodable bytes → 1; missing / directory / non-directory-in-path → 2; ungradable suppressing every verdict and digest; the read count; and `--results` carrying the invocation and each blob's sha. + +**(5) Mutants, control green in the same run, every mutant red.** Battery in `q80/mutants.py`, each mutant a module written beside a copy of the cell file so the cell resolves it through its own `__file__`: control rc 0 `Ran 15 … OK`; **M1** `parse_constant` removed → rc 1 `failures=2` (`test_nan_is_invalid_rc_1`, `test_infinity_and_negative_infinity_are_invalid`); **M2** the pre-repair blob restored → rc 1 `failures=11, errors=2`; **M2b** the repaired module with only a second read put back → rc 1 `failures=1`, `test_one_read_serves_both_the_parse_and_the_digest` dying on `AssertionError: 2 != 1 : subject read 2 times: {'read_bytes': 2, 'open': 0}` (the kill is the property, not a missing symbol); **M3** ungradable folded into rc 1 → rc 1 `failures=4`, the whole `Ungradable` class; **M4** duplicates accepted with the flag off → rc 1 `failures=1`, `test_allow_duplicates_accepts_them`. + +**(6) Wiring and the lane's number.** The file was named by no gate before this; `wizard-work/spark/CHECK` gains **leg 13**, which runs the cell file quietly and exits 13 with one line if it fails. Measured both ways: in the worktree `bash wizard-work/spark/CHECK` → `VERIFIED`, rc **0**; in a same-depth scratch tree with the pre-repair module restored → `CHECK: json_sha256_validator.py cells failed (Q80) -- run test_json_sha256_validator.py`, rc **13**. The lane's own count, `python3 -m unittest discover -p 'test_*.py'` in `wizard-work/spark/`: **57 tests, `FAILED (failures=24, errors=9)`** before and **72 tests, `FAILED (failures=24, errors=9)`** after — the count moved by exactly the 15 new cells and the pre-existing red in other files is neither fixed nor claimed here. + +**Not claimed:** that any other spark file shares this defect (only this file, its cells and one gate leg were touched); that `spark/CHECK`'s legs 1–11 are sound (they grade absolute `/root` paths by existence — Q109's class, untouched); that `VALIDATOR_TEST_RESULTS.txt` was regenerated (it was not; its `validator_run.json` digest does reproduce, `sha256sum wizard-work/spark/validator_run.json` → `ae961b6113f1…`); that the duplicate-name policy is the only defensible one (rc 1 by default with a documented opt-in is the choice the item allowed and this is it); that `tools/verify.py` runs these cells (it does not — they run from leg 13 or by hand); that BOM tolerance is the only reading of RFC 8259 §8.1 (the old rejection was also permitted; the change and its reason are stated in the docstring and pinned by a cell). Report: `/root/.hermes/wizard-outputs/volt/2026-10-30-q80.md`. Files edited for this item: `wizard-work/spark/json_sha256_validator.py` (repaired), `wizard-work/spark/test_json_sha256_validator.py` (new), `wizard-work/spark/CHECK` (leg 13), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Nothing from `inbox/` is quoted. + +CHECK: python3 wizard-work/spark/test_json_sha256_validator.py diff --git a/docs/queue-index.md b/docs/queue-index.md index db2e16d..eda1d04 100644 --- a/docs/queue-index.md +++ b/docs/queue-index.md @@ -4,7 +4,7 @@ **What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it. -**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 38 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 100 record blocks in all, 62 of them a later closure or correction for an id already opened. The same **38 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times. +**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 38 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 101 record blocks in all, 63 of them a later closure or correction for an id already opened. The same **38 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times. | # | id | kind | lines | bytes | role | title | first `Status:` | |---|---|---|---|---|---|---|---| @@ -204,7 +204,8 @@ | 194 | Q113 | record | 13525-13544 | 12052 | cont: DONE | **Q113 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The roster gate could not fail, graded its own hardcoded copy of the CSV, never … | -- | | 195 | Q126 | record | 13545-13556 | 6952 | record: BOLD-LEAD | **Q126 — DONE (volt, 2026-10-30; both faults were unowned tooling debt and both are closed by measurement: the nine host-bound test files now bind the module beside them, and the path-claims ledger … | -- | | 196 | Q110 | record | 13557-13576 | 8439 | cont: DONE | **Q110 — DONE (volt, 2026-10-30, at the commit this record lands in; quill lane by authorship, unclaimed and taken). `wizard-work/quill/QUICKSTART.md` no longer prints a command nobody ran: every … | -- | -| 197 | Q127 | record | 13577-13597 | 9706 | record: BOLD-LEAD | **Q127 — OPENED and REPAIRED in the same tick (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). A ten-file sweep put Q109's and Q120's pre-repair … | -- | +| 197 | Q127 | record | 13577-13598 | 9707 | record: BOLD-LEAD | **Q127 — OPENED and REPAIRED in the same tick (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). A ten-file sweep put Q109's and Q120's pre-repair … | -- | +| 198 | Q80 | record | 13599-13615 | 7450 | cont: DONE | **Q80 — DONE (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). All four filed defects reproduce at HEAD `e8a916f`, and none survives: the digest … | -- | ## Ids with no `## ` heading @@ -232,7 +233,7 @@ Opened as records inside another section. `line` is the record's own opening lin | Q106 | 13171 | 13171 | 14 | Q105 | BULLET+BOLD | BOLD-LEAD | | Q107 | 13189 | 13189 | 18 | Q106 | BULLET+BOLD | BOLD-LEAD | | Q108 | 13267 | 13263 | 3 | Q106 | BULLET+BOLD | BOLD-LEAD | -| Q109 | 13270 | 13270 | 22 | Q104 | BULLET+BOLD | BOLD-LEAD | +| Q109 | 13270 | 13270 | 23 | Q104 | BULLET+BOLD | BOLD-LEAD | | Q110 | 13272 | 13272 | 6 | Q109 | BULLET+BOLD | BOLD-LEAD | | Q111 | 13274 | 13274 | 7 | Q110 | BULLET+BOLD | BOLD-LEAD | | Q112 | 13276 | 13276 | 6 | Q111 | BULLET+BOLD | BOLD-LEAD | @@ -1549,17 +1550,30 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "**Q110 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; quill lane by authorship, unclaimed and taken). `wizard-work/quill/QUICKSTART.md` no longer prints a command nobody ran: every \u2026" }, { - "bytes": 9706, - "end_line": 13597, + "bytes": 9707, + "end_line": 13598, "id": "Q127", "kind": "record", "line": 13577, - "lines": 21, + "lines": 22, "role": "record", "role_detail": "BOLD-LEAD", "status": null, "status_line": null, "title": "**Q127 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). A ten-file sweep put Q109's and Q120's pre-repair \u2026" + }, + { + "bytes": 7450, + "end_line": 13615, + "id": "Q80", + "kind": "record", + "line": 13599, + "lines": 17, + "role": "continuation", + "role_detail": "DONE", + "status": null, + "status_line": null, + "title": "**Q80 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). All four filed defects reproduce at HEAD `e8a916f`, and none survives: the digest \u2026" } ], "ids_multi_section": { @@ -4246,17 +4260,30 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "**Q110 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; quill lane by authorship, unclaimed and taken). `wizard-work/quill/QUICKSTART.md` no longer prints a command nobody ran: every \u2026" }, { - "bytes": 9706, - "end_line": 13597, + "bytes": 9707, + "end_line": 13598, "id": "Q127", "kind": "record", "line": 13577, - "lines": 21, + "lines": 22, "role": "record", "role_detail": "BOLD-LEAD", "status": null, "status_line": null, "title": "**Q127 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). A ten-file sweep put Q109's and Q120's pre-repair \u2026" + }, + { + "bytes": 7450, + "end_line": 13615, + "id": "Q80", + "kind": "record", + "line": 13599, + "lines": 17, + "role": "continuation", + "role_detail": "DONE", + "status": null, + "status_line": null, + "title": "**Q80 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). All four filed defects reproduce at HEAD `e8a916f`, and none survives: the digest \u2026" } ], "nested": [ @@ -4445,7 +4472,7 @@ The block below is the parse this document renders, byte for byte. `--check` com "form": "BOLD-LEAD", "id": "Q109", "line": 13270, - "refs": 22, + "refs": 23, "section": "Q104", "shape": "BULLET+BOLD" }, @@ -4608,9 +4635,9 @@ The block below is the parse this document renders, byte for byte. `--check` com "end_line": 8, "line": 1 }, - "queue_bytes": 2167300, - "queue_lines": 13597, - "queue_sha256": "3ed7ff7e05b14f66be05cf3a3e266f43934c3de2135b2b5c30dd970b762473f0", + "queue_bytes": 2174751, + "queue_lines": 13615, + "queue_sha256": "8dcbb0dc01717d8841ef41738f0b297ba674148eddf76109227ddeddb6c7e166", "sections": [ { "bytes": 3139, diff --git a/wizard-work/spark/CHECK b/wizard-work/spark/CHECK index 86db41d..8517a15 100755 --- a/wizard-work/spark/CHECK +++ b/wizard-work/spark/CHECK @@ -30,6 +30,14 @@ python3 --version 2>&1 | grep -q "Python" || exit 10 # notice, because it ran none of the lane's cells. It runs one now. python3 "$(dirname "$(readlink -f "$0")")/closed_artifact_guard.py" >/dev/null || exit 12 +# Q80: json_sha256_validator.py's cells. Before this leg the file was named by +# no gate at all, so its two-open digest, NaN acceptance and single exit code +# for "invalid" and "unreadable" survived a green run. +if ! python3 "$(dirname "$(readlink -f "$0")")/test_json_sha256_validator.py" >/dev/null 2>&1; then + echo "CHECK: json_sha256_validator.py cells failed (Q80) -- run test_json_sha256_validator.py" >&2 + exit 13 +fi + # Check git commit pushed git -C /root/troupe-standup rev-parse ed959f0 >/dev/null 2>&1 || exit 11 diff --git a/wizard-work/spark/json_sha256_validator.py b/wizard-work/spark/json_sha256_validator.py index 76101ee..68b0eb3 100644 --- a/wizard-work/spark/json_sha256_validator.py +++ b/wizard-work/spark/json_sha256_validator.py @@ -1,60 +1,185 @@ #!/usr/bin/env python3 -"""Simple JSON validator that generates SHA256 checksums for verification.""" +"""Validate JSON documents and print the SHA-256 of the exact bytes parsed. -import json -import sys +One read serves both halves of the verdict: the bytes that are hashed are the +bytes that were parsed, so a file rewritten between the parse and the digest +cannot be described as "valid" while a different document's hash is printed +(Q80 measured the two-open form: two reads, and the printed digest belonged to +the second one). + +Contract -- the repo's tools/ exit codes: + + 0 every input is a strict RFC 8259 JSON text; one ``VALID SHA256=`` + line per input. + 1 every input was readable and at least one is not a JSON text: a syntax + error, a bare ``NaN`` / ``Infinity`` / ``-Infinity`` (RFC 8259 s6: "Numeric + values that cannot be represented in the grammar below (such as Infinity + and NaN) are not permitted" -- rejected here through ``parse_constant``), + a duplicate object member name (s4 says names SHOULD be unique; refused + unless ``--allow-duplicates``), or bytes that are not decodable JSON text. + 2 could not grade: a usage error, or an input that could not be read + (missing, a directory, permission denied, a non-directory in the path). + Nothing is printed for *any* input when any input is ungradable -- a + substrate failure never wears a verdict, and no digest is emitted. + +A leading UTF-8 byte order mark is tolerated: the parsed text is the bytes +decoded as ``utf-8-sig`` (RFC 8259 s8.1 lets a parser ignore a BOM, and the +lane's ``jsonl_validate.py`` already reads ``utf-8-sig``). Any other encoding -- +UTF-16, UTF-32, Latin-1, binary -- is not a JSON *text* in the interchange +encoding RFC 8259 s8.1 requires, and grades rc 1 as undecodable. Both are stated +so neither is filed as a bug. ``--results PATH`` writes the run as JSON: the +exact invocation, and each input's path, size, blob sha256 and verdict. +""" +import argparse import hashlib +import json +import os +import sys +from datetime import datetime, timezone from pathlib import Path +RC_OK = 0 +RC_INVALID = 1 +RC_UNGRADABLE = 2 -def validate_json_file(filepath: str) -> tuple[bool, str]: - """Validate a JSON file and return (is_valid, error_message).""" +VALID = "valid" +INVALID = "invalid" +UNGRADABLE = "ungradable" + + +class _NonStandardConstant(ValueError): + """NaN / Infinity / -Infinity: not JSON (RFC 8259 s6).""" + + +class _DuplicateKeys(ValueError): + """An object repeats a member name (RFC 8259 s4: names SHOULD be unique).""" + + +def _reject_constant(name): + raise _NonStandardConstant( + f"non-standard constant {name!r}: RFC 8259 s6 forbids NaN and Infinity" + ) + + +def _pairs_hook(pairs): + seen = set() + for name, _value in pairs: + if name in seen: + raise _DuplicateKeys( + f"duplicate object member name {name!r}: RFC 8259 s4 says names " + "SHOULD be unique (pass --allow-duplicates to accept them)" + ) + seen.add(name) + return dict(pairs) + + +def read_bytes(path): + """Return (bytes, None) or (None, message). One read, no second open.""" try: - with open(filepath, 'r') as f: - json.load(f) - return True, "" - except json.JSONDecodeError as e: - return False, f"Invalid JSON: {e}" - except FileNotFoundError: - return False, f"File not found: {filepath}" - except Exception as e: - return False, f"Error: {e}" + return Path(path).read_bytes(), None + except OSError as exc: + detail = exc.strerror or str(exc) or type(exc).__name__ + return None, f"{type(exc).__name__}: {detail}" -def compute_sha256(filepath: str) -> str: - """Compute SHA256 hash of a file.""" - sha256_hash = hashlib.sha256() - with open(filepath, 'rb') as f: - for byte_block in iter(lambda: f.read(4096), b""): - sha256_hash.update(byte_block) - return sha256_hash.hexdigest() +def parse_bytes(data, allow_duplicates=False): + """Return (parsed, None) or (None, message). Parses the bytes it is given. + + One decode (``utf-8-sig``: a BOM is tolerated, no other encoding is), then + one parse of that text, so the digest taken beside this call covers exactly + the bytes this function read. + """ + try: + text = data.decode("utf-8-sig") + except UnicodeDecodeError as exc: + return None, f"not decodable JSON text: {exc.reason} at byte {exc.start}" + try: + if allow_duplicates: + return json.loads(text, parse_constant=_reject_constant), None + return json.loads(text, parse_constant=_reject_constant, + object_pairs_hook=_pairs_hook), None + except ValueError as exc: + return None, str(exc) -def main(): - if len(sys.argv) < 2: - print("Usage: json_sha256_validator.py [file2.json ...]") - sys.exit(1) - - results = [] - all_valid = True - - for filepath in sys.argv[1:]: - is_valid, error = validate_json_file(filepath) - if is_valid: - checksum = compute_sha256(filepath) - results.append(f"VALID {filepath} SHA256={checksum}") - else: - results.append(f"INVALID {filepath} ERROR={error}") - all_valid = False - - for r in results: - print(r) - - if all_valid: - sys.exit(0) +def grade(path, allow_duplicates=False): + """Grade one input. Returns a record; the digest covers the parsed bytes.""" + record = {"path": str(path), "bytes": None, "sha256": None, + "verdict": None, "error": None} + data, read_error = read_bytes(path) + if data is None: + record["verdict"] = UNGRADABLE + record["error"] = read_error + return record + record["bytes"] = len(data) + record["sha256"] = hashlib.sha256(data).hexdigest() + _, parse_error = parse_bytes(data, allow_duplicates=allow_duplicates) + if parse_error is None: + record["verdict"] = VALID else: - sys.exit(1) + record["verdict"] = INVALID + record["error"] = parse_error + return record + + +def _write_results(path, argv, records): + payload = { + "tool": Path(__file__).name, + "invocation": [sys.executable, str(Path(__file__).resolve())] + list(argv), + "argv": list(argv), + "cwd": os.getcwd(), + "generated_at_utc": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "inputs": records, + } + try: + Path(path).write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", + encoding="utf-8") + except OSError as exc: + return f"{type(exc).__name__}: {exc.strerror or exc}" + return None + + +def main(argv=None): + ap = argparse.ArgumentParser( + prog="json_sha256_validator.py", + description="Validate JSON and print the SHA-256 of the parsed bytes.", + ) + ap.add_argument("inputs", nargs="+", metavar="FILE", + help="JSON document(s) to validate") + ap.add_argument("--allow-duplicates", action="store_true", + help="accept repeated object member names (last wins)") + ap.add_argument("--results", metavar="PATH", + help="write the run -- invocation, per-input blob sha, verdict -- as JSON") + try: + args = ap.parse_args(argv) + except SystemExit as exc: + return RC_UNGRADABLE if exc.code not in (0, None) else RC_OK + + records = [grade(p, allow_duplicates=args.allow_duplicates) for p in args.inputs] + + ungradable = [r for r in records if r["verdict"] == UNGRADABLE] + if ungradable: + for r in ungradable: + print(f"UNGRADABLE {r['path']} ERROR={r['error']}") + print(f"cannot grade: {len(ungradable)} of {len(records)} input(s) unreadable; " + "no verdict and no digest printed") + return RC_UNGRADABLE + + if args.results: + error = _write_results(args.results, argv if argv is not None else sys.argv[1:], + records) + if error is not None: + print(f"UNGRADABLE --results ERROR={error}") + return RC_UNGRADABLE + + for r in records: + if r["verdict"] == VALID: + print(f"VALID {r['path']} SHA256={r['sha256']}") + else: + print(f"INVALID {r['path']} ERROR={r['error']}") + + return RC_OK if all(r["verdict"] == VALID for r in records) else RC_INVALID if __name__ == "__main__": - main() \ No newline at end of file + sys.exit(main()) diff --git a/wizard-work/spark/test_json_sha256_validator.py b/wizard-work/spark/test_json_sha256_validator.py new file mode 100644 index 0000000..64dc28e --- /dev/null +++ b/wizard-work/spark/test_json_sha256_validator.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +"""Cells for wizard-work/spark/json_sha256_validator.py (Q80). + +Black box: every CLI reading runs the shipped module as a subprocess, so the +cells cannot agree with the code by sharing an implementation. The one +in-process cell imports the module by path to count how many times the subject +is read -- the property Q80 asks for, which a subprocess cannot observe. + +Run: python3 wizard-work/spark/test_json_sha256_validator.py +Exit 0 iff every cell passes. +""" +import hashlib +import importlib.util +import io +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +TOOL = Path(__file__).resolve().with_name("json_sha256_validator.py") + + +def run(*args, cwd=None): + proc = subprocess.run([sys.executable, str(TOOL), *args], + capture_output=True, text=True, cwd=cwd) + return proc.returncode, proc.stdout, proc.stderr + + +class Base(unittest.TestCase): + def setUp(self): + self._tmp = tempfile.TemporaryDirectory() + self.dir = Path(self._tmp.name) + self.ok = self.write("ok.json", b'{"a": [1, 2, 3], "b": "x"}\n') + self.nan = self.write("nan.json", b'{"a": NaN}\n') + self.inf = self.write("inf.json", b'{"a": Infinity}\n') + self.neginf = self.write("neginf.json", b'{"a": -Infinity}\n') + self.dup = self.write("dup.json", b'{"a": 1, "a": 2}\n') + self.broken = self.write("broken.json", b'{"a": 1,\n') + self.bom = self.write("bom.json", b'\xef\xbb\xbf{"a": 1}\n') + + def tearDown(self): + self._tmp.cleanup() + + def write(self, name, data): + path = self.dir / name + path.write_bytes(data) + return path + + def digest(self, path): + return hashlib.sha256(Path(path).read_bytes()).hexdigest() + + +class Contract(Base): + def test_usage_error_is_rc_2(self): + rc, out, err = run() + self.assertEqual(rc, 2, out + err) + + def test_valid_document_rc_0_and_digest_covers_the_file(self): + rc, out, err = run(str(self.ok)) + self.assertEqual(rc, 0, out + err) + self.assertIn("VALID", out) + self.assertIn(f"SHA256={self.digest(self.ok)}", out) + + def test_nan_is_invalid_rc_1(self): + rc, out, err = run(str(self.nan)) + self.assertEqual(rc, 1, out + err) + self.assertIn("INVALID", out) + self.assertIn("NaN", out) + self.assertNotIn("SHA256", out) + + def test_infinity_and_negative_infinity_are_invalid(self): + for path, name in ((self.inf, "Infinity"), (self.neginf, "Infinity")): + rc, out, err = run(str(path)) + self.assertEqual(rc, 1, f"{path.name}: {out}{err}") + self.assertIn("INVALID", out) + self.assertIn(name, out) + + def test_duplicate_member_names_are_refused_by_default(self): + rc, out, err = run(str(self.dup)) + self.assertEqual(rc, 1, out + err) + self.assertIn("duplicate object member name", out) + self.assertNotIn("SHA256", out) + + def test_allow_duplicates_accepts_them(self): + rc, out, err = run("--allow-duplicates", str(self.dup)) + self.assertEqual(rc, 0, out + err) + self.assertIn(f"SHA256={self.digest(self.dup)}", out) + + def test_syntax_error_is_rc_1(self): + rc, out, err = run(str(self.broken)) + self.assertEqual(rc, 1, out + err) + self.assertIn("INVALID", out) + + def test_bom_is_tolerated_and_the_choice_is_documented(self): + rc, out, err = run(str(self.bom)) + self.assertEqual(rc, 0, out + err) + self.assertIn(f"SHA256={self.digest(self.bom)}", out) + self.assertIn("byte order mark", TOOL.read_text(encoding="utf-8").lower()) + + def test_undecodable_bytes_are_invalid_rc_1(self): + bad = self.write("bad.bin", b'\xff\xfe\x00\x01') + rc, out, err = run(str(bad)) + self.assertEqual(rc, 1, out + err) + self.assertIn("not decodable JSON text", out) + + +class Ungradable(Base): + def test_missing_input_is_rc_2(self): + rc, out, err = run(str(self.dir / "nope.json")) + self.assertEqual(rc, 2, out + err) + self.assertIn("UNGRADABLE", out) + self.assertNotIn("VALID", out) + self.assertNotIn("SHA256", out) + + def test_directory_input_is_rc_2(self): + rc, out, err = run(str(self.dir)) + self.assertEqual(rc, 2, out + err) + self.assertIn("UNGRADABLE", out) + + def test_non_directory_in_path_is_rc_2(self): + rc, out, err = run(str(self.ok / "child.json")) + self.assertEqual(rc, 2, out + err) + self.assertIn("UNGRADABLE", out) + + def test_ungradable_suppresses_every_verdict_and_digest(self): + rc, out, err = run(str(self.ok), str(self.dir / "nope.json")) + self.assertEqual(rc, 2, out + err) + self.assertNotIn("VALID", out) + self.assertNotIn("INVALID", out) + self.assertNotIn("SHA256", out) + self.assertIn("no verdict and no digest printed", out) + + +class SingleRead(Base): + def test_one_read_serves_both_the_parse_and_the_digest(self): + """Serve payload A first and B after; the digest must be A's, read once.""" + target = self.dir / "race.json" + target.write_bytes(b'{"a": 1}\n') + a = b'{"a": 1}\n' + b = b'{"a": 999}\n' + counts = {"read_bytes": 0, "open": 0} + + spec = importlib.util.spec_from_file_location("json_sha256_validator", TOOL) + if spec is None or spec.loader is None: + self.fail(f"cannot load the module under test at {TOOL}") + loader = spec.loader + mod = importlib.util.module_from_spec(spec) + loader.exec_module(mod) + + real_read_bytes = Path.read_bytes + import builtins + real_builtin_open = builtins.open + + def fake_read_bytes(self): + counts["read_bytes"] += 1 + return a if counts["read_bytes"] == 1 else b + + def fake_open(path, mode="r", *args, **kwargs): + if str(path) == str(target): + counts["open"] += 1 + if "b" in mode: + return io.BytesIO(b) + return io.StringIO(b.decode()) + return real_builtin_open(path, mode, *args, **kwargs) + + Path.read_bytes = fake_read_bytes + builtins.open = fake_open + try: + record = mod.grade(str(target)) + finally: + Path.read_bytes = real_read_bytes + builtins.open = real_builtin_open + + reads = counts["read_bytes"] + counts["open"] + self.assertEqual(record["verdict"], "valid", record) + self.assertEqual(reads, 1, f"subject read {reads} times: {counts}") + self.assertEqual(record["sha256"], hashlib.sha256(a).hexdigest()) + + +class Results(Base): + def test_results_file_carries_the_invocation_and_each_blob_sha(self): + out_path = self.dir / "run.json" + rc, out, err = run("--results", str(out_path), str(self.ok), str(self.dup)) + self.assertIn(rc, (0, 1), out + err) + payload = json.loads(out_path.read_text(encoding="utf-8")) + self.assertEqual(payload["tool"], TOOL.name) + self.assertIn(str(TOOL.resolve()), payload["invocation"]) + self.assertIn(str(self.ok), payload["argv"]) + by_path = {item["path"]: item for item in payload["inputs"]} + self.assertEqual(by_path[str(self.ok)]["sha256"], self.digest(self.ok)) + self.assertEqual(by_path[str(self.ok)]["verdict"], "valid") + self.assertEqual(by_path[str(self.dup)]["verdict"], "invalid") + + +if __name__ == "__main__": + unittest.main(verbosity=2)