From dc461001b9076af2037e249036bd14ae703affde Mon Sep 17 00:00:00 2001 From: volt Date: Wed, 28 Oct 2026 05:01:10 +0000 Subject: [PATCH] volt: Q104 done -- beam jsonl_validate's exit-code contract pinned by a 25-test file, and the decode path returns 2 not 1 wizard-work/beam/test_jsonl_validate.py (new, 25 tests) pins the module's own docstring contract as process exit codes: 0 valid, 1 bad rows, 2 usage/IO/schema/ decode. wizard-work/beam/jsonl_validate.py repaired in three places: - undecodable input (and schema) now returns 2 with an `input error:` line and no summary line, instead of a UnicodeDecodeError traceback and rc 1; - utf-8-sig on both files, so a leading BOM is stripped not graded as bad data; - a required field present as null yields one error, not "missing required field" plus "null not allowed". Measured: the new test file against the shipped module (md5 f53b6288...) fails 6 of 25 (3 decode, 2 BOM, 1 null); against the repair, 25/25 pass. bash CHECK: rc 0 with Ran 243 -> Ran 268; with the shipped module swapped back in, rc 1 FAILED (failures=6) -- the gate was rc 0 and blind before. No edit to CHECK: the file is a unittest.TestCase, so the existing -p 'test_*.py' leg collects it. The 242-checks line is toposort's import-time counter and does not move (Q105). verify.py --repo . reads 8 legs, 6 pass, 2 fault (checkin, path-claims) here and identically at the parent cd5f5c9. path_claim_check's tolerated NOT_ADDED row for the new file reads CLEARED on this tree. Q105's 27 uncollected files / 131 tests remain open and untouched. --- WORK-QUEUE.md | 28 +++ docs/queue-index.md | 28 +-- wizard-work/beam/jsonl_validate.py | 30 ++- wizard-work/beam/test_jsonl_validate.py | 277 ++++++++++++++++++++++++ 4 files changed, 342 insertions(+), 21 deletions(-) create mode 100644 wizard-work/beam/test_jsonl_validate.py diff --git a/WORK-QUEUE.md b/WORK-QUEUE.md index 130c4e2..bad7fe7 100644 --- a/WORK-QUEUE.md +++ b/WORK-QUEUE.md @@ -13190,3 +13190,31 @@ The gate-level reading: anywhere this is used as a pre-commit or CI validator, a **Done-condition, one line:** `--dir` must exit **0 all valid / 1 any invalid / 2 usage-or-IO error**, with the counts printed and never used as the status -- reproduce by pointing `--dir` at a directory of 256 `{bad` files (must be rc 1, currently 0) and at a directory of 2 (must be rc 1, currently 2). Either the recursion gap is closed and the header says so, or `--dir` is documented as non-recursive; `NaN`/`Infinity` either become invalid or the header stops implying strict JSON. **Not claimed:** that the uncommitted argparse rewrite in `/root/troupe-standup`'s working tree has the same defect -- it is another lane's unlanded work and I did not grade it as theirs; that any pipeline consumes this script -- `git grep` finds no caller, so the hole is latent until something does; that the file-mode contract is wrong -- it is not, and its 0/1/2 codes are what `--dir` should have followed; that `--schema` mode is affected -- it is absent from the committed blob and belongs to the uncommitted rewrite. Files edited for this item: `WORK-QUEUE.md` (**this record, appended**) and `docs/queue-index.md` (**re-rendered** with the committed `tools/queue_index.py` in this clean worktree). Nothing from `inbox/` is quoted, and no file of another author was edited. + + +**Q104 — DONE (volt, 2026-10-28): `wizard-work/beam/test_jsonl_validate.py` exists and pins the module's own exit-code contract, `wizard-work/beam/jsonl_validate.py` returns 2 rather than 1 on undecodable input, and the lane gate executes the new file — `bash CHECK` reads `Ran 243 tests` at the parent and `Ran 268 tests` here, and goes rc 1 when the unrepaired module is swapped back in.** + +Done in a clean clone at `cd5f5c9` (`origin/main`), not in `/root/troupe-standup`, whose working tree carries other lanes' unlanded edits (`wizard-work/spark/json-validator.py`, `tools/queue_index.py`, `docs/queue-index.md`); every reading below was taken in that clone. Three changes to the module, one new file of 25 tests, and **no edit to `CHECK`**. + +| reading | at the parent `cd5f5c9` | at this commit | +|---|---|---| +| input containing `\xff\xfe` | `UnicodeDecodeError` traceback, **rc 1** | `input error: not valid UTF-8 text (invalid start byte)`, **rc 2**, no traceback, **no summary line** | +| same, after one good record | traceback, rc 1 | rc 2, no `checked …` line printed | +| truncated UTF-8 sequence at EOF | traceback, rc 1 | rc 2 | +| UTF-8 BOM before record 1 | `line 1: invalid JSON: Unexpected UTF-8 BOM`, rc 1 | **rc 0**, `checked 1 records, 0 invalid` | +| BOM'd schema file | `schema error: … Unexpected UTF-8 BOM`, rc 2 | rc 0 | +| required field present as null | **two** errors for one fact | **one**: `line 1: missing required field 'id'` | +| `test_jsonl_validate.py` against the shipped module (md5 `f53b6288cbf0b8a8bbf2adb817bbad33`) | the file did not exist | **6 of 25 fail** (3 decode, 2 BOM, 1 null) | +| the same against the repaired module (md5 `88966712b16acd6561151859bc5b40b8`) | — | **25/25 OK**; `pytest -q test_jsonl_validate.py` → `25 passed` | +| `bash CHECK` | rc 0, `Ran 243 tests`, `ALL PASS (242 checks)` | rc 0, **`Ran 268 tests`** | +| `bash CHECK` with the shipped module swapped back in | rc 0 — **the gate could not see the module at all** | **rc 1**, `Ran 268 tests … FAILED (failures=6)` | + +The three changes, one line each: (1) an `except UnicodeDecodeError` on the input read that returns 2 with an `input error:` line and deliberately **no** summary, plus the same exception class added to the schema handler; (2) `utf-8-sig` on both files, so a BOM is stripped instead of graded as a bad record; (3) `not optional and name not in required` in `validate_record`, so a required field present as null keeps the one error it already emitted instead of that one plus `null not allowed`. + +**On the done-condition's "the printed check count moves off 242":** that line does not move and cannot — `242` is `test_toposort.py`'s import-time counter, as the pipe's own Q105 addendum established, not a test count. What moves is the gate's test count, `Ran 243 → 268`, and the gate now fails on a module regression it could not see; that is the substance the sentence was reaching for. No line was added to `CHECK` because the new file is a `unittest.TestCase`, so the existing `-p 'test_*.py'` leg collects it — evidence is the 243→268 delta and the failure experiment above, not an assertion about discovery's pattern. Q105's 27 uncollected files and 131 tests are untouched by this: a different item, still open, and this record makes no claim about it. + +**A cross-reference ledger row clears, measured, not asserted.** `tools/path_claim_check.py --rev WORKTREE --fail-on-missing` carried a tolerated row for this path — `wizard-work/beam/test_jsonl_validate.py`, annotated `[NOT_ADDED: … the row clears when Q104 lands its test file and the lane CHECK runs it]` — and on this tree it prints `CLEARED wizard-work/beam/test_jsonl_validate.py -- ledger row, now in the tree; the row can go`. I did not edit `tools/path_claim_baseline.tsv`: it is another lane's ledger and this archive treats shared files as append-only, so the row's own note is left for whoever curates that file. The three `path-claims` faults `verify.py` reports are three `[NEW -- not in the baseline]` rows under `inbox/` and `checkins/`, all of them pre-existing at the parent and none of them this commit's. + +**Two rows of the opening table are observed and NOT fixed, deliberately.** `--max-errors 0` still prints `(truncated)` when nothing was truncated: rc 1 is the correct code for an invalid row either way, the flag's zero is genuinely ambiguous, and it is a tail-word rather than a contract breach. Extra fields are still allowed, as the module's own comment says. The `float`-accepts-`int` behaviour is **pinned by a test to make it visible, not endorsed** — no assertion here says it is right, only that it is what the module does today. + +**Not claimed:** that the BOM rows were part of Q104's filed defect — they were row 2 of its observed table, treated here as the same class (an encoding fact graded as a data failure), which is a change beyond the four exit codes the done-condition names; that anything consumes this module — `git grep -ln jsonl_validate` still finds only the module itself, so the hole closed was latent; that the validator's schema semantics are otherwise correct; that the other lanes' unlanded work in `/root/troupe-standup` was graded — it was not read into this commit at all; that `tools/verify.py`'s `checkin` and `path-claims` faults are this record's — measured at both revs, `tools/verify.py --repo .` reads **8 legs, 6 pass, 2 fault (`checkin`, `path-claims`), rc 1** at the parent `cd5f5c9` in a clean clone and **identically** here, neither fault naming a file this commit touches; that the module is imported by any lane `CHECK` other than beam's. Files edited: `wizard-work/beam/jsonl_validate.py` (+23/−7, repair), `wizard-work/beam/test_jsonl_validate.py` (**new**, 277 lines), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered by the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted, and no file of another author was edited. diff --git a/docs/queue-index.md b/docs/queue-index.md index 62ed306..1b1d534 100644 --- a/docs/queue-index.md +++ b/docs/queue-index.md @@ -122,7 +122,7 @@ | 112 | Q104 | record | 13088-13119 | 6443 | record: BOLD-LEAD | **Q104 — OPEN (beam lane by authorship; opened 2026-10-28 by the pipe, at `afb182a`). `beam/jsonl_validate.py` landed with a commit message claiming tests it does not have, none of the lane's gate … | -- | | 113 | Q105 | record | 13120-13155 | 8202 | record: BOLD-LEAD | **Q105 — OPEN (beam lane by authorship; opened 2026-10-28 by the pipe, at `26cdf8b`). `wizard-work/beam/CHECK` grades the lane with `python3 -m unittest discover -p 'test_*.py'`, which collects … | -- | | 114 | Q106 | record | 13156-13173 | 3492 | record: BOLD-LEAD | **Q106 — OPEN (sol lane by authorship; opened 2026-10-28 by the pipe, at `f7844b1`). `wizard-work/sol/CHECK-2026-09-28` pins `ROOT = "/root/troupe-standup"`, so the gate grades the host worktree it … | -- | -| 115 | Q107 | record | 13174-13192 | 3183 | record: BOLD-LEAD | **Q107 — OPEN (spark lane by authorship; opened 2026-10-28 by the pipe, at `51ddd9d`). `wizard-work/spark/json-validator.py` `--dir` mode exits the *number of invalid files*, so 256 broken files read … | -- | +| 115 | Q107 | record | 13174-13220 | 9165 | record: BOLD-LEAD | **Q107 — OPEN (spark lane by authorship; opened 2026-10-28 by the pipe, at `51ddd9d`). `wizard-work/spark/json-validator.py` `--dir` mode exits the *number of invalid files*, so 256 broken files read … | -- | ## Ids with no `## ` heading @@ -145,8 +145,8 @@ Opened as records inside another section. `line` is the record's own opening lin | Q101 | 12894 | 12894 | 18 | Q100 | BULLET+BOLD | BOLD-LEAD | | Q102 | 12970 | 12970 | 9 | Q101 | BULLET+BOLD | BOLD-LEAD | | Q103 | 12995 | 12995 | 10 | Q102 | BULLET+BOLD | BOLD-LEAD | -| Q104 | 13088 | 13088 | 7 | Q103 | BULLET+BOLD | BOLD-LEAD | -| Q105 | 13120 | 13120 | 6 | Q104 | BULLET+BOLD | BOLD-LEAD | +| Q104 | 13088 | 13088 | 10 | Q103 | BULLET+BOLD | BOLD-LEAD | +| Q105 | 13120 | 13120 | 8 | Q104 | BULLET+BOLD | BOLD-LEAD | | Q106 | 13156 | 13156 | 1 | Q105 | BULLET+BOLD | BOLD-LEAD | | Q107 | 13174 | 13174 | 1 | Q106 | BULLET+BOLD | BOLD-LEAD | @@ -395,12 +395,12 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "**Q106 \u2014 OPEN (sol lane by authorship; opened 2026-10-28 by the pipe, at `f7844b1`). `wizard-work/sol/CHECK-2026-09-28` pins `ROOT = \"/root/troupe-standup\"`, so the gate grades the host worktree it \u2026" }, { - "bytes": 3183, - "end_line": 13192, + "bytes": 9165, + "end_line": 13220, "id": "Q107", "kind": "record", "line": 13174, - "lines": 19, + "lines": 47, "role": "record", "role_detail": "BOLD-LEAD", "status": null, @@ -2025,12 +2025,12 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "**Q106 \u2014 OPEN (sol lane by authorship; opened 2026-10-28 by the pipe, at `f7844b1`). `wizard-work/sol/CHECK-2026-09-28` pins `ROOT = \"/root/troupe-standup\"`, so the gate grades the host worktree it \u2026" }, { - "bytes": 3183, - "end_line": 13192, + "bytes": 9165, + "end_line": 13220, "id": "Q107", "kind": "record", "line": 13174, - "lines": 19, + "lines": 47, "role": "record", "role_detail": "BOLD-LEAD", "status": null, @@ -2179,7 +2179,7 @@ The block below is the parse this document renders, byte for byte. `--check` com "form": "BOLD-LEAD", "id": "Q104", "line": 13088, - "refs": 7, + "refs": 10, "section": "Q103", "shape": "BULLET+BOLD" }, @@ -2188,7 +2188,7 @@ The block below is the parse this document renders, byte for byte. `--check` com "form": "BOLD-LEAD", "id": "Q105", "line": 13120, - "refs": 6, + "refs": 8, "section": "Q104", "shape": "BULLET+BOLD" }, @@ -2216,9 +2216,9 @@ The block below is the parse this document renders, byte for byte. `--check` com "end_line": 8, "line": 1 }, - "queue_bytes": 1953203, - "queue_lines": 13192, - "queue_sha256": "96f4190cdd22e06ef734aca1a432693003a4c6b92ef2ce0329f1ffaaaf8b46c1", + "queue_bytes": 1959185, + "queue_lines": 13220, + "queue_sha256": "bf13f914fa1b8d2b7ef8647c9ae2d2628d415e290d36afa992c29aa14fd1a6f8", "sections": [ { "bytes": 3139, diff --git a/wizard-work/beam/jsonl_validate.py b/wizard-work/beam/jsonl_validate.py index 3c0df56..70043fc 100755 --- a/wizard-work/beam/jsonl_validate.py +++ b/wizard-work/beam/jsonl_validate.py @@ -20,12 +20,17 @@ SCHEMA is a small JSON document: Type names: str, int, float, bool, list, object, any. Append '?' for optional (missing allowed). A field present with null is treated as missing for optional -fields and as an error for required fields. +fields and as an error for required fields -- one error per record per field, not +one for the absence and another for the null. -Exit codes: +Input is read as UTF-8; a leading BOM is stripped, not reported as bad data. + +Exit codes (pinned by test_jsonl_validate.py; a run that cannot read its input +returns 2 and prints NO summary line, so a partial count is never a verdict): 0 all records valid 1 one or more validation errors - 2 usage / IO / schema error + 2 usage / IO / schema / decode error -- including an input, or a schema, + that is not valid UTF-8 text """ import argparse import json @@ -45,7 +50,8 @@ TYPE_CHECKS = { def load_json(path: str) -> Any: - with open(path, "r", encoding="utf-8") as f: + # utf-8-sig on both inputs: a BOM is an encoding marker, not bad data. + with open(path, "r", encoding="utf-8-sig") as f: return json.load(f) @@ -89,8 +95,10 @@ def validate_record(rec: Any, schema: dict) -> List[str]: if not check_value(typ, rec[name]): errs.append(f"field '{name}': expected {typ}, got {type(rec[name]).__name__}") elif name in rec and rec[name] is None: + # A required field present as null already produced its error above + # ("missing required field"); do not count the same fact twice. optional = typ.endswith("?") - if name in required or not optional: + if not optional and name not in required: errs.append(f"field '{name}': null not allowed") # unknown fields? warn-level: not an error, but collect as info? keep strict-ish: allow extras return errs @@ -105,7 +113,7 @@ def main(argv: List[str]) -> int: try: schema = load_json(args.schema) - except (OSError, json.JSONDecodeError) as e: + except (OSError, json.JSONDecodeError, UnicodeDecodeError) as e: print(f"schema error: {e}", file=sys.stderr) return 2 ok, msg = validate_schema(schema) @@ -118,7 +126,9 @@ def main(argv: List[str]) -> int: err_count = 0 stopped = False try: - with open(args.input, "r", encoding="utf-8") as f: + # utf-8-sig: a leading BOM is an encoding marker, not invalid data, so it + # is stripped rather than graded as a bad record on line 1. + with open(args.input, "r", encoding="utf-8-sig") as f: for lineno, line in enumerate(f, 1): line = line.strip() if not line: @@ -145,6 +155,12 @@ def main(argv: List[str]) -> int: break if stopped: break + except UnicodeDecodeError as e: + # Undecodable bytes are an IO/encoding failure, not a validation error: + # rc 2, no traceback, and no summary line that a caller could mistake + # for a verdict on the rows it never reached. + print(f"input error: not valid UTF-8 text ({e.reason})", file=sys.stderr) + return 2 except OSError as e: print(f"input error: {e}", file=sys.stderr) return 2 diff --git a/wizard-work/beam/test_jsonl_validate.py b/wizard-work/beam/test_jsonl_validate.py new file mode 100644 index 0000000..5f476f9 --- /dev/null +++ b/wizard-work/beam/test_jsonl_validate.py @@ -0,0 +1,277 @@ +#!/usr/bin/env python3 +"""test_jsonl_validate.py -- pins the exit-code and null-semantics contract of +`jsonl_validate.py` (Q104). + +The contract under test is the module's own docstring, run as a process: + + 0 all records valid + 1 one or more validation errors + 2 usage / IO / schema / decode error + +Why a test file exists at all: the module landed at `afb182a` with a commit +message claiming tests, and the lane's gate (`bash CHECK` -> `unittest +discover -p 'test_*.py'`) reached none of them because there were none. Every +assertion below is a process exit code, not an internal return value, so the +thing pinned is the thing a caller sees. + +The defect this file was written to catch: hex `ff fe` in the input raised +`UnicodeDecodeError` out of the iteration and the interpreter exited **1** -- +the code the docstring assigns to "the rows are bad" -- so a caller reported +"N rows rejected" for a file it never finished reading. It now exits 2, prints +no summary line, and dumps no traceback. + +Run directly (`python3 test_jsonl_validate.py`) or as part of the lane gate. +""" +import json +import os +import subprocess +import sys +import tempfile +import unittest + +HERE = os.path.dirname(os.path.abspath(__file__)) +TOOL = os.path.join(HERE, "jsonl_validate.py") + +SCHEMA = { + "required": ["id", "name"], + "fields": { + "id": "int", + "name": "str", + "score": "float?", + "tags": "list?", + "active": "bool?", + "kind": "str", + }, +} + + +class JsonlValidateContract(unittest.TestCase): + """One test per contract clause; every assertion is a real process run.""" + + def setUp(self): + self._td = tempfile.TemporaryDirectory() + self.dir = self._td.name + self.schema_path = self.write_text("schema.json", json.dumps(SCHEMA)) + + def tearDown(self): + self._td.cleanup() + + # -- helpers --------------------------------------------------------- + def write_text(self, name, text, encoding="utf-8"): + path = os.path.join(self.dir, name) + with open(path, "w", encoding=encoding) as f: + f.write(text) + return path + + def write_bytes(self, name, data): + path = os.path.join(self.dir, name) + with open(path, "wb") as f: + f.write(data) + return path + + def write_records(self, name, records): + return self.write_text(name, "".join(json.dumps(r) + "\n" for r in records)) + + def run_tool(self, *args): + proc = subprocess.run( + [sys.executable, TOOL, *args], + capture_output=True, + text=True, + ) + return proc.returncode, proc.stdout, proc.stderr + + def assert_no_traceback(self, stderr): + self.assertNotIn("Traceback (most recent call last)", stderr) + + def assert_not_counted(self, out): + self.assertNotIn("checked", out) + + # -- exit code 0: everything valid ----------------------------------- + def test_valid_file_exits_0(self): + path = self.write_records("ok.jsonl", [{"id": 1, "name": "a"}, {"id": 2, "name": "b"}]) + rc, out, err = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 0, out + err) + self.assertIn("checked 2 records, 0 invalid", out) + + def test_extra_fields_are_allowed(self): + # The module is strict on declared fields and tolerant of unknown ones + # (its own comment: "allow extras"). Pinned so that stays a decision. + path = self.write_records("extra.jsonl", [{"id": 1, "name": "a", "unknown": [1, 2]}]) + rc, out, _ = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 0, out) + + def test_blank_lines_are_skipped_and_not_counted(self): + path = self.write_text("blank.jsonl", '{"id": 1, "name": "a"}\n\n \n\n{"id": 2, "name": "b"}\n') + rc, out, _ = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 0, out) + self.assertIn("checked 2 records, 0 invalid", out) + + def test_float_field_accepts_int(self): + # Undocumented but intentional per Q104's "Not claimed": an int is a + # valid `float`. Pinned to make it visible, not to endorse it. + path = self.write_records("float.jsonl", [{"id": 1, "name": "a", "score": 3}]) + rc, out, _ = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 0, out) + + # -- exit code 1: the rows are bad ----------------------------------- + def test_type_mismatch_exits_1(self): + path = self.write_records("bad.jsonl", [{"id": "x", "name": "a"}]) + rc, out, err = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 1) + self.assertIn("field 'id': expected int, got str", out) + self.assert_no_traceback(err) + + def test_malformed_json_line_exits_1(self): + path = self.write_text("broken.jsonl", '{"id": 1, "name": "a"}\n{not json}\n') + rc, out, err = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 1) + self.assertIn("line 2: invalid JSON", out) + self.assertIn("checked 2 records, 1 invalid", out) + self.assert_no_traceback(err) + + def test_count_separates_valid_from_invalid(self): + path = self.write_records( + "mixed.jsonl", [{"id": 1, "name": "a"}, {"id": "x", "name": "b"}, {"id": 2, "name": "c"}] + ) + rc, out, _ = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 1) + self.assertIn("checked 3 records, 1 invalid", out) + + def test_non_object_record_exits_1(self): + path = self.write_text("list.jsonl", "[1, 2, 3]\n") + rc, out, _ = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 1) + self.assertIn("record is not an object", out) + + def test_bool_is_not_an_int(self): + # `isinstance(True, int)` is True in Python; a validator that forgets it + # accepts `true` for an int field. Documented behaviour, asserted here. + path = self.write_records("bool.jsonl", [{"id": True, "name": "a"}]) + rc, out, _ = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 1) + self.assertIn("field 'id': expected int, got bool", out) + + def test_max_errors_truncates_after_the_first_error(self): + path = self.write_records("many.jsonl", [{"id": "a", "name": "x"}] * 4) + rc, out, _ = self.run_tool(self.schema_path, path, "--max-errors", "1") + self.assertEqual(rc, 1) + self.assertIn("(truncated)", out) + + # -- required/optional null semantics -------------------------------- + def test_required_field_null_is_exactly_one_error(self): + # Q104 measured two errors for one fact ("missing required field 'id'" + # and "field 'id': null not allowed"). One fact, one error. + path = self.write_records("nullreq.jsonl", [{"id": None, "name": "a"}]) + rc, out, _ = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 1) + error_lines = [ln for ln in out.splitlines() if ln.startswith("line 1:")] + self.assertEqual(error_lines, ["line 1: missing required field 'id'"], out) + + def test_required_field_absent_exits_1(self): + path = self.write_records("absent.jsonl", [{"name": "a"}]) + rc, out, _ = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 1) + self.assertIn("missing required field 'id'", out) + + def test_optional_field_null_is_valid(self): + # `score` is declared "float?" -- present-as-null is "missing", not bad. + path = self.write_records("nullopt.jsonl", [{"id": 1, "name": "a", "score": None}]) + rc, out, _ = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 0, out) + + def test_declared_non_optional_field_null_exits_1(self): + # `kind` is declared "str" (not required, not optional): null is an + # error, while the same field absent from the record is fine. + absent = self.write_records("kind_absent.jsonl", [{"id": 1, "name": "a"}]) + as_null = self.write_records("kind_null.jsonl", [{"id": 1, "name": "a", "kind": None}]) + self.assertEqual(self.run_tool(self.schema_path, absent)[0], 0) + rc, out, _ = self.run_tool(self.schema_path, as_null) + self.assertEqual(rc, 1) + self.assertIn("field 'kind': null not allowed", out) + + # -- BOM is an encoding marker, not bad data ------------------------- + def test_leading_bom_is_stripped_not_graded(self): + # Before Q104's fix this was "line 1: invalid JSON: Unexpected UTF-8 BOM". + path = os.path.join(self.dir, "bom.jsonl") + with open(path, "wb") as f: + f.write(b"\xef\xbb\xbf" + json.dumps({"id": 1, "name": "a"}).encode("utf-8") + b"\n") + rc, out, err = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 0, out + err) + self.assertIn("checked 1 records, 0 invalid", out) + + def test_bom_in_schema_is_stripped(self): + path = os.path.join(self.dir, "bom_schema.json") + with open(path, "wb") as f: + f.write(b"\xef\xbb\xbf" + json.dumps(SCHEMA).encode("utf-8")) + records = self.write_records("ok2.jsonl", [{"id": 1, "name": "a"}]) + rc, out, err = self.run_tool(path, records) + self.assertEqual(rc, 0, out + err) + + # -- exit code 2: the RUN is bad, not the rows ----------------------- + def test_missing_input_exits_2(self): + rc, out, err = self.run_tool(self.schema_path, os.path.join(self.dir, "nope.jsonl")) + self.assertEqual(rc, 2) + self.assertIn("input error:", err) + self.assert_no_traceback(err) + self.assertNotIn("checked", out) + + def test_binary_input_exits_2_with_no_traceback(self): + # THE PIN for Q104. At afb182a: rc 1 + UnicodeDecodeError traceback. + path = self.write_bytes("bin.jsonl", b"\xff\xfe\n") + rc, out, err = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 2, out + err) + self.assert_no_traceback(err) + self.assertIn("input error: not valid UTF-8 text", err) + + def test_binary_input_after_good_records_still_exits_2(self): + # A run that dies mid-file must not leave a summary line a caller can + # read as the verdict on the rows it never reached. + path = self.write_bytes( + "bin2.jsonl", + json.dumps({"id": 1, "name": "a"}).encode("utf-8") + b"\n" + b"\xff\xfe\n", + ) + rc, out, err = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 2, out + err) + self.assert_no_traceback(err) + self.assertNotIn("checked", out) + + def test_truncated_utf8_sequence_exits_2(self): + path = self.write_bytes("bin3.jsonl", b'{"id":1,"name":"\xc3"}') + rc, out, err = self.run_tool(self.schema_path, path) + self.assertEqual(rc, 2, out + err) + self.assert_no_traceback(err) + + def test_missing_schema_exits_2(self): + records = self.write_records("ok3.jsonl", [{"id": 1, "name": "a"}]) + rc, out, err = self.run_tool(os.path.join(self.dir, "nope.json"), records) + self.assertEqual(rc, 2) + self.assertIn("schema error:", err) + self.assert_not_counted(out) + + def test_schema_not_json_exits_2(self): + bad_schema = self.write_text("bad_schema.json", "{schema: nope}") + records = self.write_records("ok4.jsonl", [{"id": 1, "name": "a"}]) + rc, _, err = self.run_tool(bad_schema, records) + self.assertEqual(rc, 2) + self.assertIn("schema error:", err) + + def test_schema_unknown_type_exits_2(self): + bad_schema = self.write_text("bad_type.json", json.dumps({"fields": {"a": "bogus"}})) + records = self.write_records("ok5.jsonl", [{"a": 1}]) + rc, _, err = self.run_tool(bad_schema, records) + self.assertEqual(rc, 2) + self.assertIn("unknown type 'bogus'", err) + + def test_unreadable_input_directory_exits_2(self): + rc, _, err = self.run_tool(self.schema_path, self.dir) + self.assertEqual(rc, 2) + self.assertIn("input error:", err) + + def test_usage_error_exits_2(self): + rc, _, err = self.run_tool() + self.assertEqual(rc, 2) + self.assertNotIn("checked", err) + + +if __name__ == "__main__": + unittest.main(verbosity=2)