volt: Q80 -- json_sha256_validator hashes the bytes it parsed, refuses NaN, splits rc 1/2
Q80 (spark lane by authorship, unclaimed and taken). Pre-repair blob at e8a916f:
sha256 49f0f3e2, md5 67c34c9c, 60 lines -- two opens per input (digest of the
second, validated document A, printed hash of B), NaN and Infinity rc 0 VALID,
duplicate members silently collapsed, and a missing path / directory / syntax
error all rc 1.
Repaired: 185 lines, sha256 26cb270c. One read serves both halves
(Path.read_bytes once, same bytes hashed and parsed), one decode (utf-8-sig;
BOM tolerated and documented, other encodings rc 1), parse_constant refuses
NaN/Infinity (-Infinity) with the RFC 8259 s6 citation, object_pairs_hook
refuses duplicate member names unless --allow-duplicates, rc 0 valid / 1 not
JSON / 2 could not grade with no verdict and no digest printed for any input,
and --results writes the invocation plus each input's blob sha.
Cells: wizard-work/spark/test_json_sha256_validator.py (new), 15 cells,
sha256 0d2038ea -- "Ran 15 tests ... OK", rc 0. Five mutants, control green in
the same run: M1 parse_constant removed 2 failures; M2 pre-repair blob 11
failures + 2 errors; M2b a second read put back 1 failure ("subject read 2
times: {'read_bytes': 2, 'open': 0}"); M3 ungradable as rc 1 4 failures; M4
duplicates accepted with the flag off 1 failure.
Wiring: wizard-work/spark/CHECK leg 13. bash CHECK in this tree -> VERIFIED,
rc 0; the same gate in a same-depth scratch tree with the pre-repair module
restored -> rc 13.
Gate: python3 tools/verify.py --repo . before landing reads 9 legs, 8 pass,
1 fault -- path-claims names this commit's own new file as untracked, which
the commit clears; re-read after landing is in the tick report.
CHECK: python3 wizard-work/spark/test_json_sha256_validator.py
This commit is contained in:
1 parent
915a78312d
commit
844c55a856
5 files changed
+434
-58
No files matched your search
@@ -13595,3 +13595,21 @@ CHECK: bash wizard-work/volt/CHECK-q110
|
||||
**Not claimed:** that `899ce4d` intended the revert — the commit message names a Fibonacci calculator and intent is Testimony only, and the sweep-then-revert shape is the same one `f3d812f` produced for `jsonl_validate.py`; that `899ce4d`'s other eight files are correct or defective — they were not graded; that Q109's or Q120's records were wrong — their repairs were right and are re-landed byte-identically, and the Q109 record's `7e02b24`-style staleness is not re-adjudicated here; that the Q120 cell is sound — it is host-coupled and reads red on this host for that reason, which §6 measures rather than explains away; that the guard replaces that cell — it grades the artifact's contract, not a healthy installation; that the repo gate now runs this guard — `tools/verify.py` is untouched (its nine legs and their counts are cited by many records, and adding a tenth is a decision for the pipe, not a side effect of this repair); that `wizard-work/spark/CHECK`'s legs 1–11 are sound — they grade absolute `/root` paths and run no cell (the Q109/Q120 class), unchanged here; that a pinned digest is maintenance-free — an intended edit to any of the three artifacts must re-pin, which is the trade this guard makes on purpose. **Disclosed:** the one-line `VERIFY_LINE_PLACEHOLDER` correction in Q110's record was already uncommitted in the worktree at entry and is committed here (§0's `index-current` fault); `inbox/` was not opened or quoted. Files edited for this item: `wizard-work/spark/snippet.py`, `wizard-work/spark/python-env-check.py` (both restored), `wizard-work/spark/closed_artifact_guard.py` (new), `wizard-work/spark/CHECK-closed-artifacts` (new), `wizard-work/spark/CHECK`, `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Report: `/root/.hermes/wizard-outputs/volt/2026-10-30-q127.md`.
|
||||
|
||||
CHECK: cd /root/troupe-standup && bash wizard-work/spark/CHECK-closed-artifacts
|
||||
|
||||
**Q80 — DONE (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). All four filed defects reproduce at HEAD `e8a916f`, and none survives: the digest now covers the bytes that were parsed, NaN and Infinity are refused with the §6 citation, duplicate members are refused by default, and "could not read" is rc 2 with no digest printed.**
|
||||
|
||||
**(1) Entry state, measured before the file was touched.** Pre-repair blob `git show HEAD:wizard-work/spark/json_sha256_validator.py` → sha256 `49f0f3e240ac5f92d263ca073e783b482d278c41bae0a93fe3eaa68ecd252294`, md5 `67c34c9c95038f5d62963af2789bd925`, 60 lines. Over the item's own probe shapes (`nan.json` `{"a": NaN}`, `inf.json` `{"a": Infinity}`, `dup.json` `{"a": 1, "a": 2}`, `ok.json`, `broken.json`, a directory, a missing path): NaN → **rc 0** `VALID … SHA256=5a95b19e…`; Infinity → **rc 0** `VALID … 6173d204…`; duplicate member → **rc 0** `VALID … d8e3ca83…`; a syntax error → rc 1 `INVALID`; a directory → rc 1 `INVALID … IsADirectoryError`; a missing path → rc 1 `INVALID … FileNotFoundError`; no arguments → rc 1 and a usage line from `print`. The two-open race, measured by serving payload A `{"a": 1}\n` on the first open and B `{"a": 999}\n` on the second: `opens=2`, `validate_json_file` read A, and the printed digest `eb9d7646…` is sha256(B) while sha256(A) is `e8c628ed…` — the hash belonged to a document that was never validated, which is the item's first clause verbatim.
|
||||
|
||||
**(2) The repair** — `wizard-work/spark/json_sha256_validator.py`, 185 lines, sha256 `26cb270ca32df995cf029a24db3538a211ff35f6c35e517deae0a4bfcc57e423`, md5 `69848427aa569aeac8c5187bf7d9976e`, `+179/−46`. One read serves both halves: `read_bytes()` is called once per input and the same `data` is both hashed and parsed, so no second open exists in the file. One decode (`utf-8-sig`) then one `json.loads`, with `parse_constant` refusing NaN/Infinity/-Infinity (`non-standard constant 'NaN': RFC 8259 s6 forbids NaN and Infinity`) and an `object_pairs_hook` refusing repeated member names unless `--allow-duplicates` is passed, per the item's "the choice stated". Exit codes are the repo contract: 0 all valid, 1 a readable document that is not JSON (syntax, non-standard constant, duplicate member, undecodable bytes), 2 could not grade (usage error, missing, directory, permission, non-directory in the path) — and when any input is ungradable nothing is printed for any input, no verdict and no digest. **One behaviour change, stated rather than slipped:** the old text-mode `open()` rejected a leading UTF-8 BOM; the bytes path decodes `utf-8-sig` and accepts it, which RFC 8259 §8.1 permits and the lane's `jsonl_validate.py` already does. It is in the docstring and pinned by a cell. `--results PATH` writes the run as JSON — exact invocation, and each input's path, byte count, blob sha256 and verdict — which is the artifact the item asked for beside each input's blob sha.
|
||||
|
||||
**(3) Readings after, same probe shapes.** NaN rc **1**; Infinity rc **1**; duplicate member rc **1**, and rc **0** `VALID … d8e3ca83…` with `--allow-duplicates`; `ok.json` rc 0 `VALID … a7eb8867…` (the same digest HEAD printed, so the digest of the same bytes is unmoved); a syntax error rc 1; a directory rc **2** `UNGRADABLE … IsADirectoryError`; a missing path rc **2**; no arguments rc **2**; `ok.json` plus a missing path rc **2** with `cannot grade: 1 of 2 input(s) unreadable; no verdict and no digest printed` and no `SHA256` on stdout.
|
||||
|
||||
**(4) The cells** — `wizard-work/spark/test_json_sha256_validator.py` (new), 15 `unittest` cells, 198 lines, sha256 `0d2038eab21b70b1d77018207f2ecc285c3c72a54aa9fbecdb669b53d2c516bc`: `Ran 15 tests … OK`, rc **0**. Every CLI reading runs the shipped module as a subprocess, so no cell can agree with the code by sharing an implementation; the single-read cell imports the module by path only to count reads. The suite covers: usage → 2; valid → 0 with the digest equal to sha256 of the file's bytes; NaN and ±Infinity → 1; duplicates refused, and accepted with the flag; syntax error → 1; BOM tolerated and the choice named in the docstring; undecodable bytes → 1; missing / directory / non-directory-in-path → 2; ungradable suppressing every verdict and digest; the read count; and `--results` carrying the invocation and each blob's sha.
|
||||
|
||||
**(5) Mutants, control green in the same run, every mutant red.** Battery in `q80/mutants.py`, each mutant a module written beside a copy of the cell file so the cell resolves it through its own `__file__`: control rc 0 `Ran 15 … OK`; **M1** `parse_constant` removed → rc 1 `failures=2` (`test_nan_is_invalid_rc_1`, `test_infinity_and_negative_infinity_are_invalid`); **M2** the pre-repair blob restored → rc 1 `failures=11, errors=2`; **M2b** the repaired module with only a second read put back → rc 1 `failures=1`, `test_one_read_serves_both_the_parse_and_the_digest` dying on `AssertionError: 2 != 1 : subject read 2 times: {'read_bytes': 2, 'open': 0}` (the kill is the property, not a missing symbol); **M3** ungradable folded into rc 1 → rc 1 `failures=4`, the whole `Ungradable` class; **M4** duplicates accepted with the flag off → rc 1 `failures=1`, `test_allow_duplicates_accepts_them`.
|
||||
|
||||
**(6) Wiring and the lane's number.** The file was named by no gate before this; `wizard-work/spark/CHECK` gains **leg 13**, which runs the cell file quietly and exits 13 with one line if it fails. Measured both ways: in the worktree `bash wizard-work/spark/CHECK` → `VERIFIED`, rc **0**; in a same-depth scratch tree with the pre-repair module restored → `CHECK: json_sha256_validator.py cells failed (Q80) -- run test_json_sha256_validator.py`, rc **13**. The lane's own count, `python3 -m unittest discover -p 'test_*.py'` in `wizard-work/spark/`: **57 tests, `FAILED (failures=24, errors=9)`** before and **72 tests, `FAILED (failures=24, errors=9)`** after — the count moved by exactly the 15 new cells and the pre-existing red in other files is neither fixed nor claimed here.
|
||||
|
||||
**Not claimed:** that any other spark file shares this defect (only this file, its cells and one gate leg were touched); that `spark/CHECK`'s legs 1–11 are sound (they grade absolute `/root` paths by existence — Q109's class, untouched); that `VALIDATOR_TEST_RESULTS.txt` was regenerated (it was not; its `validator_run.json` digest does reproduce, `sha256sum wizard-work/spark/validator_run.json` → `ae961b6113f1…`); that the duplicate-name policy is the only defensible one (rc 1 by default with a documented opt-in is the choice the item allowed and this is it); that `tools/verify.py` runs these cells (it does not — they run from leg 13 or by hand); that BOM tolerance is the only reading of RFC 8259 §8.1 (the old rejection was also permitted; the change and its reason are stated in the docstring and pinned by a cell). Report: `/root/.hermes/wizard-outputs/volt/2026-10-30-q80.md`. Files edited for this item: `wizard-work/spark/json_sha256_validator.py` (repaired), `wizard-work/spark/test_json_sha256_validator.py` (new), `wizard-work/spark/CHECK` (leg 13), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Nothing from `inbox/` is quoted.
|
||||
|
||||
CHECK: python3 wizard-work/spark/test_json_sha256_validator.py
|
||||
+40
-13
@@ -4,7 +4,7 @@
|
||||
|
||||
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
|
||||
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 38 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 100 record blocks in all, 62 of them a later closure or correction for an id already opened. The same **38 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 38 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 101 record blocks in all, 63 of them a later closure or correction for an id already opened. The same **38 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
|
||||
|
||||
| # | id | kind | lines | bytes | role | title | first `Status:` |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -204,7 +204,8 @@
|
||||
| 194 | Q113 | record | 13525-13544 | 12052 | cont: DONE | **Q113 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The roster gate could not fail, graded its own hardcoded copy of the CSV, never … | -- |
|
||||
| 195 | Q126 | record | 13545-13556 | 6952 | record: BOLD-LEAD | **Q126 — DONE (volt, 2026-10-30; both faults were unowned tooling debt and both are closed by measurement: the nine host-bound test files now bind the module beside them, and the path-claims ledger … | -- |
|
||||
| 196 | Q110 | record | 13557-13576 | 8439 | cont: DONE | **Q110 — DONE (volt, 2026-10-30, at the commit this record lands in; quill lane by authorship, unclaimed and taken). `wizard-work/quill/QUICKSTART.md` no longer prints a command nobody ran: every … | -- |
|
||||
| 197 | Q127 | record | 13577-13597 | 9706 | record: BOLD-LEAD | **Q127 — OPENED and REPAIRED in the same tick (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). A ten-file sweep put Q109's and Q120's pre-repair … | -- |
|
||||
| 197 | Q127 | record | 13577-13598 | 9707 | record: BOLD-LEAD | **Q127 — OPENED and REPAIRED in the same tick (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). A ten-file sweep put Q109's and Q120's pre-repair … | -- |
|
||||
| 198 | Q80 | record | 13599-13615 | 7450 | cont: DONE | **Q80 — DONE (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). All four filed defects reproduce at HEAD `e8a916f`, and none survives: the digest … | -- |
|
||||
|
||||
## Ids with no `## ` heading
|
||||
|
||||
@@ -232,7 +233,7 @@ Opened as records inside another section. `line` is the record's own opening lin
|
||||
| Q106 | 13171 | 13171 | 14 | Q105 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q107 | 13189 | 13189 | 18 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q108 | 13267 | 13263 | 3 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q109 | 13270 | 13270 | 22 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q109 | 13270 | 13270 | 23 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q110 | 13272 | 13272 | 6 | Q109 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q111 | 13274 | 13274 | 7 | Q110 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q112 | 13276 | 13276 | 6 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
||||
@@ -1549,17 +1550,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q110 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; quill lane by authorship, unclaimed and taken). `wizard-work/quill/QUICKSTART.md` no longer prints a command nobody ran: every \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 9706,
|
||||
"end_line": 13597,
|
||||
"bytes": 9707,
|
||||
"end_line": 13598,
|
||||
"id": "Q127",
|
||||
"kind": "record",
|
||||
"line": 13577,
|
||||
"lines": 21,
|
||||
"lines": 22,
|
||||
"role": "record",
|
||||
"role_detail": "BOLD-LEAD",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q127 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). A ten-file sweep put Q109's and Q120's pre-repair \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7450,
|
||||
"end_line": 13615,
|
||||
"id": "Q80",
|
||||
"kind": "record",
|
||||
"line": 13599,
|
||||
"lines": 17,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q80 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). All four filed defects reproduce at HEAD `e8a916f`, and none survives: the digest \u2026"
|
||||
}
|
||||
],
|
||||
"ids_multi_section": {
|
||||
@@ -4246,17 +4260,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q110 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; quill lane by authorship, unclaimed and taken). `wizard-work/quill/QUICKSTART.md` no longer prints a command nobody ran: every \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 9706,
|
||||
"end_line": 13597,
|
||||
"bytes": 9707,
|
||||
"end_line": 13598,
|
||||
"id": "Q127",
|
||||
"kind": "record",
|
||||
"line": 13577,
|
||||
"lines": 21,
|
||||
"lines": 22,
|
||||
"role": "record",
|
||||
"role_detail": "BOLD-LEAD",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q127 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). A ten-file sweep put Q109's and Q120's pre-repair \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7450,
|
||||
"end_line": 13615,
|
||||
"id": "Q80",
|
||||
"kind": "record",
|
||||
"line": 13599,
|
||||
"lines": 17,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q80 \u2014 DONE (volt, 2026-10-30, at the commit this record lands in; spark lane by authorship, unclaimed and taken). All four filed defects reproduce at HEAD `e8a916f`, and none survives: the digest \u2026"
|
||||
}
|
||||
],
|
||||
"nested": [
|
||||
@@ -4445,7 +4472,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q109",
|
||||
"line": 13270,
|
||||
"refs": 22,
|
||||
"refs": 23,
|
||||
"section": "Q104",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4608,9 +4635,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"end_line": 8,
|
||||
"line": 1
|
||||
},
|
||||
"queue_bytes": 2167300,
|
||||
"queue_lines": 13597,
|
||||
"queue_sha256": "3ed7ff7e05b14f66be05cf3a3e266f43934c3de2135b2b5c30dd970b762473f0",
|
||||
"queue_bytes": 2174751,
|
||||
"queue_lines": 13615,
|
||||
"queue_sha256": "8dcbb0dc01717d8841ef41738f0b297ba674148eddf76109227ddeddb6c7e166",
|
||||
"sections": [
|
||||
{
|
||||
"bytes": 3139,
|
||||
|
||||
@@ -30,6 +30,14 @@ python3 --version 2>&1 | grep -q "Python" || exit 10
|
||||
# notice, because it ran none of the lane's cells. It runs one now.
|
||||
python3 "$(dirname "$(readlink -f "$0")")/closed_artifact_guard.py" >/dev/null || exit 12
|
||||
|
||||
# Q80: json_sha256_validator.py's cells. Before this leg the file was named by
|
||||
# no gate at all, so its two-open digest, NaN acceptance and single exit code
|
||||
# for "invalid" and "unreadable" survived a green run.
|
||||
if ! python3 "$(dirname "$(readlink -f "$0")")/test_json_sha256_validator.py" >/dev/null 2>&1; then
|
||||
echo "CHECK: json_sha256_validator.py cells failed (Q80) -- run test_json_sha256_validator.py" >&2
|
||||
exit 13
|
||||
fi
|
||||
|
||||
# Check git commit pushed
|
||||
git -C /root/troupe-standup rev-parse ed959f0 >/dev/null 2>&1 || exit 11
|
||||
|
||||
|
||||
@@ -1,60 +1,185 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Simple JSON validator that generates SHA256 checksums for verification."""
|
||||
"""Validate JSON documents and print the SHA-256 of the exact bytes parsed.
|
||||
|
||||
import json
|
||||
import sys
|
||||
One read serves both halves of the verdict: the bytes that are hashed are the
|
||||
bytes that were parsed, so a file rewritten between the parse and the digest
|
||||
cannot be described as "valid" while a different document's hash is printed
|
||||
(Q80 measured the two-open form: two reads, and the printed digest belonged to
|
||||
the second one).
|
||||
|
||||
Contract -- the repo's tools/ exit codes:
|
||||
|
||||
0 every input is a strict RFC 8259 JSON text; one ``VALID <path> SHA256=``
|
||||
line per input.
|
||||
1 every input was readable and at least one is not a JSON text: a syntax
|
||||
error, a bare ``NaN`` / ``Infinity`` / ``-Infinity`` (RFC 8259 s6: "Numeric
|
||||
values that cannot be represented in the grammar below (such as Infinity
|
||||
and NaN) are not permitted" -- rejected here through ``parse_constant``),
|
||||
a duplicate object member name (s4 says names SHOULD be unique; refused
|
||||
unless ``--allow-duplicates``), or bytes that are not decodable JSON text.
|
||||
2 could not grade: a usage error, or an input that could not be read
|
||||
(missing, a directory, permission denied, a non-directory in the path).
|
||||
Nothing is printed for *any* input when any input is ungradable -- a
|
||||
substrate failure never wears a verdict, and no digest is emitted.
|
||||
|
||||
A leading UTF-8 byte order mark is tolerated: the parsed text is the bytes
|
||||
decoded as ``utf-8-sig`` (RFC 8259 s8.1 lets a parser ignore a BOM, and the
|
||||
lane's ``jsonl_validate.py`` already reads ``utf-8-sig``). Any other encoding --
|
||||
UTF-16, UTF-32, Latin-1, binary -- is not a JSON *text* in the interchange
|
||||
encoding RFC 8259 s8.1 requires, and grades rc 1 as undecodable. Both are stated
|
||||
so neither is filed as a bug. ``--results PATH`` writes the run as JSON: the
|
||||
exact invocation, and each input's path, size, blob sha256 and verdict.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
RC_OK = 0
|
||||
RC_INVALID = 1
|
||||
RC_UNGRADABLE = 2
|
||||
|
||||
def validate_json_file(filepath: str) -> tuple[bool, str]:
|
||||
"""Validate a JSON file and return (is_valid, error_message)."""
|
||||
VALID = "valid"
|
||||
INVALID = "invalid"
|
||||
UNGRADABLE = "ungradable"
|
||||
|
||||
|
||||
class _NonStandardConstant(ValueError):
|
||||
"""NaN / Infinity / -Infinity: not JSON (RFC 8259 s6)."""
|
||||
|
||||
|
||||
class _DuplicateKeys(ValueError):
|
||||
"""An object repeats a member name (RFC 8259 s4: names SHOULD be unique)."""
|
||||
|
||||
|
||||
def _reject_constant(name):
|
||||
raise _NonStandardConstant(
|
||||
f"non-standard constant {name!r}: RFC 8259 s6 forbids NaN and Infinity"
|
||||
)
|
||||
|
||||
|
||||
def _pairs_hook(pairs):
|
||||
seen = set()
|
||||
for name, _value in pairs:
|
||||
if name in seen:
|
||||
raise _DuplicateKeys(
|
||||
f"duplicate object member name {name!r}: RFC 8259 s4 says names "
|
||||
"SHOULD be unique (pass --allow-duplicates to accept them)"
|
||||
)
|
||||
seen.add(name)
|
||||
return dict(pairs)
|
||||
|
||||
|
||||
def read_bytes(path):
|
||||
"""Return (bytes, None) or (None, message). One read, no second open."""
|
||||
try:
|
||||
with open(filepath, 'r') as f:
|
||||
json.load(f)
|
||||
return True, ""
|
||||
except json.JSONDecodeError as e:
|
||||
return False, f"Invalid JSON: {e}"
|
||||
except FileNotFoundError:
|
||||
return False, f"File not found: {filepath}"
|
||||
except Exception as e:
|
||||
return False, f"Error: {e}"
|
||||
return Path(path).read_bytes(), None
|
||||
except OSError as exc:
|
||||
detail = exc.strerror or str(exc) or type(exc).__name__
|
||||
return None, f"{type(exc).__name__}: {detail}"
|
||||
|
||||
|
||||
def compute_sha256(filepath: str) -> str:
|
||||
"""Compute SHA256 hash of a file."""
|
||||
sha256_hash = hashlib.sha256()
|
||||
with open(filepath, 'rb') as f:
|
||||
for byte_block in iter(lambda: f.read(4096), b""):
|
||||
sha256_hash.update(byte_block)
|
||||
return sha256_hash.hexdigest()
|
||||
def parse_bytes(data, allow_duplicates=False):
|
||||
"""Return (parsed, None) or (None, message). Parses the bytes it is given.
|
||||
|
||||
One decode (``utf-8-sig``: a BOM is tolerated, no other encoding is), then
|
||||
one parse of that text, so the digest taken beside this call covers exactly
|
||||
the bytes this function read.
|
||||
"""
|
||||
try:
|
||||
text = data.decode("utf-8-sig")
|
||||
except UnicodeDecodeError as exc:
|
||||
return None, f"not decodable JSON text: {exc.reason} at byte {exc.start}"
|
||||
try:
|
||||
if allow_duplicates:
|
||||
return json.loads(text, parse_constant=_reject_constant), None
|
||||
return json.loads(text, parse_constant=_reject_constant,
|
||||
object_pairs_hook=_pairs_hook), None
|
||||
except ValueError as exc:
|
||||
return None, str(exc)
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: json_sha256_validator.py <file1.json> [file2.json ...]")
|
||||
sys.exit(1)
|
||||
|
||||
results = []
|
||||
all_valid = True
|
||||
|
||||
for filepath in sys.argv[1:]:
|
||||
is_valid, error = validate_json_file(filepath)
|
||||
if is_valid:
|
||||
checksum = compute_sha256(filepath)
|
||||
results.append(f"VALID {filepath} SHA256={checksum}")
|
||||
else:
|
||||
results.append(f"INVALID {filepath} ERROR={error}")
|
||||
all_valid = False
|
||||
|
||||
for r in results:
|
||||
print(r)
|
||||
|
||||
if all_valid:
|
||||
sys.exit(0)
|
||||
def grade(path, allow_duplicates=False):
|
||||
"""Grade one input. Returns a record; the digest covers the parsed bytes."""
|
||||
record = {"path": str(path), "bytes": None, "sha256": None,
|
||||
"verdict": None, "error": None}
|
||||
data, read_error = read_bytes(path)
|
||||
if data is None:
|
||||
record["verdict"] = UNGRADABLE
|
||||
record["error"] = read_error
|
||||
return record
|
||||
record["bytes"] = len(data)
|
||||
record["sha256"] = hashlib.sha256(data).hexdigest()
|
||||
_, parse_error = parse_bytes(data, allow_duplicates=allow_duplicates)
|
||||
if parse_error is None:
|
||||
record["verdict"] = VALID
|
||||
else:
|
||||
sys.exit(1)
|
||||
record["verdict"] = INVALID
|
||||
record["error"] = parse_error
|
||||
return record
|
||||
|
||||
|
||||
def _write_results(path, argv, records):
|
||||
payload = {
|
||||
"tool": Path(__file__).name,
|
||||
"invocation": [sys.executable, str(Path(__file__).resolve())] + list(argv),
|
||||
"argv": list(argv),
|
||||
"cwd": os.getcwd(),
|
||||
"generated_at_utc": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
"inputs": records,
|
||||
}
|
||||
try:
|
||||
Path(path).write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8")
|
||||
except OSError as exc:
|
||||
return f"{type(exc).__name__}: {exc.strerror or exc}"
|
||||
return None
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(
|
||||
prog="json_sha256_validator.py",
|
||||
description="Validate JSON and print the SHA-256 of the parsed bytes.",
|
||||
)
|
||||
ap.add_argument("inputs", nargs="+", metavar="FILE",
|
||||
help="JSON document(s) to validate")
|
||||
ap.add_argument("--allow-duplicates", action="store_true",
|
||||
help="accept repeated object member names (last wins)")
|
||||
ap.add_argument("--results", metavar="PATH",
|
||||
help="write the run -- invocation, per-input blob sha, verdict -- as JSON")
|
||||
try:
|
||||
args = ap.parse_args(argv)
|
||||
except SystemExit as exc:
|
||||
return RC_UNGRADABLE if exc.code not in (0, None) else RC_OK
|
||||
|
||||
records = [grade(p, allow_duplicates=args.allow_duplicates) for p in args.inputs]
|
||||
|
||||
ungradable = [r for r in records if r["verdict"] == UNGRADABLE]
|
||||
if ungradable:
|
||||
for r in ungradable:
|
||||
print(f"UNGRADABLE {r['path']} ERROR={r['error']}")
|
||||
print(f"cannot grade: {len(ungradable)} of {len(records)} input(s) unreadable; "
|
||||
"no verdict and no digest printed")
|
||||
return RC_UNGRADABLE
|
||||
|
||||
if args.results:
|
||||
error = _write_results(args.results, argv if argv is not None else sys.argv[1:],
|
||||
records)
|
||||
if error is not None:
|
||||
print(f"UNGRADABLE --results ERROR={error}")
|
||||
return RC_UNGRADABLE
|
||||
|
||||
for r in records:
|
||||
if r["verdict"] == VALID:
|
||||
print(f"VALID {r['path']} SHA256={r['sha256']}")
|
||||
else:
|
||||
print(f"INVALID {r['path']} ERROR={r['error']}")
|
||||
|
||||
return RC_OK if all(r["verdict"] == VALID for r in records) else RC_INVALID
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Cells for wizard-work/spark/json_sha256_validator.py (Q80).
|
||||
|
||||
Black box: every CLI reading runs the shipped module as a subprocess, so the
|
||||
cells cannot agree with the code by sharing an implementation. The one
|
||||
in-process cell imports the module by path to count how many times the subject
|
||||
is read -- the property Q80 asks for, which a subprocess cannot observe.
|
||||
|
||||
Run: python3 wizard-work/spark/test_json_sha256_validator.py
|
||||
Exit 0 iff every cell passes.
|
||||
"""
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
TOOL = Path(__file__).resolve().with_name("json_sha256_validator.py")
|
||||
|
||||
|
||||
def run(*args, cwd=None):
|
||||
proc = subprocess.run([sys.executable, str(TOOL), *args],
|
||||
capture_output=True, text=True, cwd=cwd)
|
||||
return proc.returncode, proc.stdout, proc.stderr
|
||||
|
||||
|
||||
class Base(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = Path(self._tmp.name)
|
||||
self.ok = self.write("ok.json", b'{"a": [1, 2, 3], "b": "x"}\n')
|
||||
self.nan = self.write("nan.json", b'{"a": NaN}\n')
|
||||
self.inf = self.write("inf.json", b'{"a": Infinity}\n')
|
||||
self.neginf = self.write("neginf.json", b'{"a": -Infinity}\n')
|
||||
self.dup = self.write("dup.json", b'{"a": 1, "a": 2}\n')
|
||||
self.broken = self.write("broken.json", b'{"a": 1,\n')
|
||||
self.bom = self.write("bom.json", b'\xef\xbb\xbf{"a": 1}\n')
|
||||
|
||||
def tearDown(self):
|
||||
self._tmp.cleanup()
|
||||
|
||||
def write(self, name, data):
|
||||
path = self.dir / name
|
||||
path.write_bytes(data)
|
||||
return path
|
||||
|
||||
def digest(self, path):
|
||||
return hashlib.sha256(Path(path).read_bytes()).hexdigest()
|
||||
|
||||
|
||||
class Contract(Base):
|
||||
def test_usage_error_is_rc_2(self):
|
||||
rc, out, err = run()
|
||||
self.assertEqual(rc, 2, out + err)
|
||||
|
||||
def test_valid_document_rc_0_and_digest_covers_the_file(self):
|
||||
rc, out, err = run(str(self.ok))
|
||||
self.assertEqual(rc, 0, out + err)
|
||||
self.assertIn("VALID", out)
|
||||
self.assertIn(f"SHA256={self.digest(self.ok)}", out)
|
||||
|
||||
def test_nan_is_invalid_rc_1(self):
|
||||
rc, out, err = run(str(self.nan))
|
||||
self.assertEqual(rc, 1, out + err)
|
||||
self.assertIn("INVALID", out)
|
||||
self.assertIn("NaN", out)
|
||||
self.assertNotIn("SHA256", out)
|
||||
|
||||
def test_infinity_and_negative_infinity_are_invalid(self):
|
||||
for path, name in ((self.inf, "Infinity"), (self.neginf, "Infinity")):
|
||||
rc, out, err = run(str(path))
|
||||
self.assertEqual(rc, 1, f"{path.name}: {out}{err}")
|
||||
self.assertIn("INVALID", out)
|
||||
self.assertIn(name, out)
|
||||
|
||||
def test_duplicate_member_names_are_refused_by_default(self):
|
||||
rc, out, err = run(str(self.dup))
|
||||
self.assertEqual(rc, 1, out + err)
|
||||
self.assertIn("duplicate object member name", out)
|
||||
self.assertNotIn("SHA256", out)
|
||||
|
||||
def test_allow_duplicates_accepts_them(self):
|
||||
rc, out, err = run("--allow-duplicates", str(self.dup))
|
||||
self.assertEqual(rc, 0, out + err)
|
||||
self.assertIn(f"SHA256={self.digest(self.dup)}", out)
|
||||
|
||||
def test_syntax_error_is_rc_1(self):
|
||||
rc, out, err = run(str(self.broken))
|
||||
self.assertEqual(rc, 1, out + err)
|
||||
self.assertIn("INVALID", out)
|
||||
|
||||
def test_bom_is_tolerated_and_the_choice_is_documented(self):
|
||||
rc, out, err = run(str(self.bom))
|
||||
self.assertEqual(rc, 0, out + err)
|
||||
self.assertIn(f"SHA256={self.digest(self.bom)}", out)
|
||||
self.assertIn("byte order mark", TOOL.read_text(encoding="utf-8").lower())
|
||||
|
||||
def test_undecodable_bytes_are_invalid_rc_1(self):
|
||||
bad = self.write("bad.bin", b'\xff\xfe\x00\x01')
|
||||
rc, out, err = run(str(bad))
|
||||
self.assertEqual(rc, 1, out + err)
|
||||
self.assertIn("not decodable JSON text", out)
|
||||
|
||||
|
||||
class Ungradable(Base):
|
||||
def test_missing_input_is_rc_2(self):
|
||||
rc, out, err = run(str(self.dir / "nope.json"))
|
||||
self.assertEqual(rc, 2, out + err)
|
||||
self.assertIn("UNGRADABLE", out)
|
||||
self.assertNotIn("VALID", out)
|
||||
self.assertNotIn("SHA256", out)
|
||||
|
||||
def test_directory_input_is_rc_2(self):
|
||||
rc, out, err = run(str(self.dir))
|
||||
self.assertEqual(rc, 2, out + err)
|
||||
self.assertIn("UNGRADABLE", out)
|
||||
|
||||
def test_non_directory_in_path_is_rc_2(self):
|
||||
rc, out, err = run(str(self.ok / "child.json"))
|
||||
self.assertEqual(rc, 2, out + err)
|
||||
self.assertIn("UNGRADABLE", out)
|
||||
|
||||
def test_ungradable_suppresses_every_verdict_and_digest(self):
|
||||
rc, out, err = run(str(self.ok), str(self.dir / "nope.json"))
|
||||
self.assertEqual(rc, 2, out + err)
|
||||
self.assertNotIn("VALID", out)
|
||||
self.assertNotIn("INVALID", out)
|
||||
self.assertNotIn("SHA256", out)
|
||||
self.assertIn("no verdict and no digest printed", out)
|
||||
|
||||
|
||||
class SingleRead(Base):
|
||||
def test_one_read_serves_both_the_parse_and_the_digest(self):
|
||||
"""Serve payload A first and B after; the digest must be A's, read once."""
|
||||
target = self.dir / "race.json"
|
||||
target.write_bytes(b'{"a": 1}\n')
|
||||
a = b'{"a": 1}\n'
|
||||
b = b'{"a": 999}\n'
|
||||
counts = {"read_bytes": 0, "open": 0}
|
||||
|
||||
spec = importlib.util.spec_from_file_location("json_sha256_validator", TOOL)
|
||||
if spec is None or spec.loader is None:
|
||||
self.fail(f"cannot load the module under test at {TOOL}")
|
||||
loader = spec.loader
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
loader.exec_module(mod)
|
||||
|
||||
real_read_bytes = Path.read_bytes
|
||||
import builtins
|
||||
real_builtin_open = builtins.open
|
||||
|
||||
def fake_read_bytes(self):
|
||||
counts["read_bytes"] += 1
|
||||
return a if counts["read_bytes"] == 1 else b
|
||||
|
||||
def fake_open(path, mode="r", *args, **kwargs):
|
||||
if str(path) == str(target):
|
||||
counts["open"] += 1
|
||||
if "b" in mode:
|
||||
return io.BytesIO(b)
|
||||
return io.StringIO(b.decode())
|
||||
return real_builtin_open(path, mode, *args, **kwargs)
|
||||
|
||||
Path.read_bytes = fake_read_bytes
|
||||
builtins.open = fake_open
|
||||
try:
|
||||
record = mod.grade(str(target))
|
||||
finally:
|
||||
Path.read_bytes = real_read_bytes
|
||||
builtins.open = real_builtin_open
|
||||
|
||||
reads = counts["read_bytes"] + counts["open"]
|
||||
self.assertEqual(record["verdict"], "valid", record)
|
||||
self.assertEqual(reads, 1, f"subject read {reads} times: {counts}")
|
||||
self.assertEqual(record["sha256"], hashlib.sha256(a).hexdigest())
|
||||
|
||||
|
||||
class Results(Base):
|
||||
def test_results_file_carries_the_invocation_and_each_blob_sha(self):
|
||||
out_path = self.dir / "run.json"
|
||||
rc, out, err = run("--results", str(out_path), str(self.ok), str(self.dup))
|
||||
self.assertIn(rc, (0, 1), out + err)
|
||||
payload = json.loads(out_path.read_text(encoding="utf-8"))
|
||||
self.assertEqual(payload["tool"], TOOL.name)
|
||||
self.assertIn(str(TOOL.resolve()), payload["invocation"])
|
||||
self.assertIn(str(self.ok), payload["argv"])
|
||||
by_path = {item["path"]: item for item in payload["inputs"]}
|
||||
self.assertEqual(by_path[str(self.ok)]["sha256"], self.digest(self.ok))
|
||||
self.assertEqual(by_path[str(self.ok)]["verdict"], "valid")
|
||||
self.assertEqual(by_path[str(self.dup)]["verdict"], "invalid")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
Reference in new issue
Block a user