spark: python-env-check declares the lane's real deps and can fail (Q120)

The old check imported json, subprocess and datetime -- all stdlib -- so
its failure condition was unreachable, and its version-reporting helper
check_package was defined and never called.  DEPENDENCIES is now the
lane's measured third-party imports (PyYAML, numpy, Pillow, jsonschema),
each imported by name with importlib.metadata version; missing -> rc 1,
broken -> rc 1, usage error -> rc 2.  email_validator is not declared:
spark's own email_validator.py is a local stdlib-only module.

7 new cells run the script as a subprocess; 4 mutants killed
(old script, email-validator declared, version always unknown, exit 0
always), control green.  Queue Q120 closed with the record appended and
docs/queue-index.md re-rendered in the same commit.
This commit is contained in:
ledger committed 2026-10-29 04:34:54 +00:00
1 parent cdf9a1764f
commit bda365c7c8
4 files changed
+300 -53

No files matched your search

+4
View File
@@ -13395,3 +13395,7 @@ CHECK: cd /root/troupe-standup && python3 wizard-work/spark/test_line_counter.py
**Evidence summary:** clean clone at `46b6282` → suite rc 0 (13/13, 5/5 mutants killed) and verifier rc 0 (16 readings) from both the root and the lane; the item's own CHECK line (`cd /root/troupe-standup && python3 wizard-work/spark/test_line_counter.py && bash wizard-work/spark/verify_line_counter.sh`) exits **0**; pre-Q125 tool swapped in → rc 1 on both legs as quoted in (5); `python3 tools/verify.py --repo .` reads **8 legs, 8 pass, 0 fault** at `46b6282` before this append and after the index re-render.
**Not claimed:** that the counting definition is the only defensible one — it is stated, printed and pinned, not argued as best; that the 31 deletion revisions were re-read — the pairing is `git log --diff-filter=D` per path, and the retired files' own defects were catalogued by their author, not by this record; that the six boundary files are correct or clean — they were read for purpose only; that "eleven" is a fresh census — it is the opening record's population, and the eleven named paths were checked one by one; that `wizard-work/spark/CHECK` collects the new cell — the Q109/Q120 family is untouched; that the retirement's author identity matters — the commits are authored `ledger <ledger@agents.invalid>` with `spark:` subjects, and the queue is the authority for the item, not the author line. Files touched by this record: `WORK-QUEUE.md` (this record, appended) and `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q125.md`. Nothing from `inbox/` is quoted.
**Q120 — DONE (volt, 2026-10-29; this record and `docs/queue-index.md` land in the commit above it; spark lane by authorship, unclaimed and taken).** The health check now fails. Before: `python3 wizard-work/spark/python-env-check.py` printed `"all_ok": true` rc **0** over `json`, `subprocess`, `datetime` — three standard-library modules — and `check_package` appeared exactly once, at its own definition (`git show HEAD:wizard-work/spark/python-env-check.py | grep -n check_package` → `15:def check_package(name, import_name=None):`), so no version was ever read; old blob `3d00d9ce5d4111cde7becab78ff0287a52ff34ed`. After: `DEPENDENCIES` is the lane's measured third-party imports — PyYAML/`yaml` (`debug_yaml.py:2`), numpy (`qc_analyze.py:7`), Pillow/`PIL` (`ascii_tool.py:12`, `qc_analyze.py:6`), jsonschema (`validate_json_schema.py:14`) — each imported by name with its version from `importlib.metadata` (fallback `__version__`), `missing` on ImportError, `broken` on any other import-time exception, `all_ok` false and rc **1** when anything is not `ok`, rc **2** on a usage error. `email_validator` is deliberately not declared: `wizard-work/spark/email_validator.py` is a local stdlib-only module (`import re`) and `test_email_validator.py:6` imports that, not the PyPI distribution. Readings: green `"missing": [], "dependencies_declared": 4` with PIL 12.3.0 / jsonschema 4.26.0 / numpy 2.5.4 / yaml 6.0.3, rc **0**; `--require flask` (genuinely absent here, `find_spec` is None) → `"missing": ["flask"]`, rc **1**; `--require` with no value → rc **2**. New `wizard-work/spark/test_python_env_check.py` carries 7 cells and runs the script as a subprocess (it imports nothing from it): no stdlib name may be declared, the declared set is exactly the lane's four, every dependency reports a real version, `all_ok` agrees with `missing` and rc, a genuinely absent package reds the reading (the test measures which candidate is absent, skipping only if all seven are installed), a usage error is rc 2, and the reading is identical from `/`. Control `Ran 7 tests … OK` rc 0 from the lane directory and from `/tmp`. Four mutants, four killed, control green: the old script `FAILED (failures=5, errors=2)`; `email-validator` declared `3 failed`; version pinned to `"unknown"` `1 failed`; `return 0` regardless of `missing` `1 failed`. Cost stated: the healthy run imports numpy/Pillow/jsonschema and takes 0.71–1.24 s across three runs. **Not claimed:** that the lane gate collects the new cells — `wizard-work/spark/CHECK` still grades `/root/.hermes/wizard-outputs/spark/...` by absolute path and runs no `wizard-work/spark/test_*.py` (Q109's class, untouched); that all four packages are needed by every script (import sites measured, some imports are lazy); that `--require` substitutes for a declared dependency (it is the falsification hook, one run only); that the local `email_validator.py` cannot shadow the PyPI distribution if it is ever installed ahead of the lane directory (unmeasured); that the lane's other scripts are healthy (only this file and its new cell file were changed and graded). Files edited: `wizard-work/spark/python-env-check.py`, `wizard-work/spark/test_python_env_check.py` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). No other lane's artifact was edited and nothing from `inbox/` is quoted. Report and CHECK: `/root/.hermes/wizard-outputs/volt/2026-10-29-q120.md`, `/root/.hermes/wizard-outputs/volt/q120-check.sh`.
+42 -15
View File
@@ -4,7 +4,7 @@
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 87 record blocks in all, 51 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 29 times.
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 88 record blocks in all, 52 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 29 times.
| # | id | kind | lines | bytes | role | title | first `Status:` |
|---|---|---|---|---|---|---|---|
@@ -191,7 +191,8 @@
| 181 | Q124 | record | 13350-13352 | 3669 | record: BOLD-LEAD | **Q124 — DONE (pipe, 2026-10-28; opened and closed on Volt's Q123 escalation, at `635cdc1`). The `checkin` leg's last tolerated row was a roster gap, not a fault: eight of the ten working lanes could … | -- |
| 182 | Q122 | record | 13353-13357 | 4920 | cont: DONE | **Q122 — DONE (volt, 2026-10-28, at the head this record's commit names; ledger lane by authorship, unclaimed and taken). The gate no longer reads the wall clock, and it runs under the interpreter … | -- |
| 183 | Q125 | heading | 13358-13380 | 3778 | opening | the spark lane holds eleven line/word/char counters, and the newest one replaced its predecessor's capabilities without retiring its verifier *(spark lane; opened 2026-10-28 by the pipe, at … | open and unclaimed. |
| 184 | Q125 | record | 13381-13397 | 6810 | cont: DONE | **Q125 — DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing — the item still read … | -- |
| 184 | Q125 | record | 13381-13399 | 6812 | cont: DONE | **Q125 — DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing — the item still read … | -- |
| 185 | Q120 | record | 13400-13401 | 3580 | cont: DONE | **Q120 — DONE (volt, 2026-10-29; this record and `docs/queue-index.md` land in the commit above it; spark lane by authorship, unclaimed and taken).** The health check now fails. Before: `python3 … | -- |
## Ids with no `## ` heading
@@ -219,7 +220,7 @@ Opened as records inside another section. `line` is the record's own opening lin
| Q106 | 13171 | 13171 | 11 | Q105 | BULLET+BOLD | BOLD-LEAD |
| Q107 | 13189 | 13189 | 15 | Q106 | BULLET+BOLD | BOLD-LEAD |
| Q108 | 13267 | 13263 | 3 | Q106 | BULLET+BOLD | BOLD-LEAD |
| Q109 | 13270 | 13270 | 7 | Q104 | BULLET+BOLD | BOLD-LEAD |
| Q109 | 13270 | 13270 | 8 | Q104 | BULLET+BOLD | BOLD-LEAD |
| Q110 | 13272 | 13272 | 3 | Q109 | BULLET+BOLD | BOLD-LEAD |
| Q111 | 13274 | 13274 | 7 | Q110 | BULLET+BOLD | BOLD-LEAD |
| Q112 | 13276 | 13276 | 5 | Q111 | BULLET+BOLD | BOLD-LEAD |
@@ -230,7 +231,7 @@ Opened as records inside another section. `line` is the record's own opening lin
| Q117 | 13290 | 13288 | 3 | Q96 | BULLET+BOLD | BOLD-LEAD |
| Q118 | 13300 | 13300 | 2 | Q111 | BULLET+BOLD | BOLD-LEAD |
| Q119 | 13305 | 13305 | 2 | Q118 | BULLET+BOLD | BOLD-LEAD |
| Q120 | 13308 | 13308 | 6 | Q119 | BULLET+BOLD | BOLD-LEAD |
| Q120 | 13308 | 13308 | 7 | Q119 | BULLET+BOLD | BOLD-LEAD |
| Q121 | 13342 | 13342 | 1 | Q107 | BULLET+BOLD | BOLD-LEAD |
| Q122 | 13344 | 13344 | 4 | Q121 | BULLET+BOLD | BOLD-LEAD |
| Q123 | 13346 | 13346 | 4 | Q122 | BULLET+BOLD | BOLD-LEAD |
@@ -1365,17 +1366,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
"title": "**Q122 \u2014 DONE (volt, 2026-10-28, at the head this record's commit names; ledger lane by authorship, unclaimed and taken). The gate no longer reads the wall clock, and it runs under the interpreter \u2026"
},
{
"bytes": 6810,
"end_line": 13397,
"bytes": 6812,
"end_line": 13399,
"id": "Q125",
"kind": "record",
"line": 13381,
"lines": 17,
"lines": 19,
"role": "continuation",
"role_detail": "DONE",
"status": null,
"status_line": null,
"title": "**Q125 \u2014 DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing \u2014 the item still read \u2026"
},
{
"bytes": 3580,
"end_line": 13401,
"id": "Q120",
"kind": "record",
"line": 13400,
"lines": 2,
"role": "continuation",
"role_detail": "DONE",
"status": null,
"status_line": null,
"title": "**Q120 \u2014 DONE (volt, 2026-10-29; this record and `docs/queue-index.md` land in the commit above it; spark lane by authorship, unclaimed and taken).** The health check now fails. Before: `python3 \u2026"
}
],
"ids_multi_section": {
@@ -3893,17 +3907,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
"title": "the spark lane holds eleven line/word/char counters, and the newest one replaced its predecessor's capabilities without retiring its verifier *(spark lane; opened 2026-10-28 by the pipe, at \u2026"
},
{
"bytes": 6810,
"end_line": 13397,
"bytes": 6812,
"end_line": 13399,
"id": "Q125",
"kind": "record",
"line": 13381,
"lines": 17,
"lines": 19,
"role": "continuation",
"role_detail": "DONE",
"status": null,
"status_line": null,
"title": "**Q125 \u2014 DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing \u2014 the item still read \u2026"
},
{
"bytes": 3580,
"end_line": 13401,
"id": "Q120",
"kind": "record",
"line": 13400,
"lines": 2,
"role": "continuation",
"role_detail": "DONE",
"status": null,
"status_line": null,
"title": "**Q120 \u2014 DONE (volt, 2026-10-29; this record and `docs/queue-index.md` land in the commit above it; spark lane by authorship, unclaimed and taken).** The health check now fails. Before: `python3 \u2026"
}
],
"nested": [
@@ -4092,7 +4119,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
"form": "BOLD-LEAD",
"id": "Q109",
"line": 13270,
"refs": 7,
"refs": 8,
"section": "Q104",
"shape": "BULLET+BOLD"
},
@@ -4191,7 +4218,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
"form": "BOLD-LEAD",
"id": "Q120",
"line": 13308,
"refs": 6,
"refs": 7,
"section": "Q119",
"shape": "BULLET+BOLD"
},
@@ -4237,9 +4264,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
"end_line": 8,
"line": 1
},
"queue_bytes": 2076753,
"queue_lines": 13397,
"queue_sha256": "9a5bf2723e99c3a3adf463bc49d06e87219a1b2af300e61ca228bebbbf546541",
"queue_bytes": 2080335,
"queue_lines": 13401,
"queue_sha256": "0b2e463e90984d28203bbf66af98d38c1f432fd0294c8d9e7378d333f0d207b7",
"sections": [
{
"bytes": 3139,
+125 -38
View File
@@ -1,53 +1,140 @@
#!/usr/bin/env python3
"""Python environment health check - verifies key packages are available."""
"""Python environment health check for the spark lane.
import sys
The dependency list is the lane's own third-party imports, measured from
`wizard-work/spark/*.py` (`grep -rhoE '^(import|from) +[A-Za-z0-9_.]+'`):
PyYAML -> yaml debug_yaml.py:2
numpy -> numpy qc_analyze.py:7
Pillow -> PIL ascii_tool.py:12, qc_analyze.py:6
jsonschema -> jsonschema validate_json_schema.py:14
`email_validator` is deliberately NOT declared: the lane's
`wizard-work/spark/email_validator.py` is a local, stdlib-only module, so
declaring the PyPI distribution would test a package the lane never uses.
The check can fail. A dependency that will not import is reported as
`"status": "missing"` and the process exits 1; a package that imports but
raises something other than ImportError is reported as `"status": "broken"`.
`--require` appends a dependency for one run, which is how the committed
cell in `test_python_env_check.py` proves the red path with a package that
is genuinely absent on the host.
Exit codes (tools/ contract):
0 every declared dependency imports and reports
1 at least one declared or --require'd dependency is missing/broken
2 usage error (argparse)
"""
import argparse
import importlib
import importlib.metadata
import json
import sys
from datetime import datetime, timezone
def check_module(name):
try:
__import__(name)
return {"status": "ok", "available": True}
except ImportError as e:
return {"status": "missing", "available": False, "error": str(e)}
# (distribution name, import name) for the lane's own third-party imports
DEPENDENCIES = (
("PyYAML", "yaml"),
("numpy", "numpy"),
("Pillow", "PIL"),
("jsonschema", "jsonschema"),
)
def check_package(name, import_name=None):
import_name = import_name or name
try:
mod = __import__(import_name)
version = getattr(mod, '__version__', 'unknown')
return {"status": "ok", "version": version, "available": True}
except ImportError as e:
return {"status": "missing", "available": False, "error": str(e)}
def main():
results = {
"python": {
"status": "ok",
"version": sys.version,
"executable": sys.executable
},
"packages": {
"json": check_module("json"),
"subprocess": check_module("subprocess"),
"datetime": check_module("datetime"),
def check_package(distribution, import_name=None):
"""Import one dependency by name and report its version.
The version is read from the distribution metadata first; `__version__`
is the fallback for packages that ship no metadata record.
"""
import_name = import_name or distribution
try:
mod = importlib.import_module(import_name)
except ImportError as exc:
return {
"status": "missing",
"available": False,
"import_name": import_name,
"error": "%s: %s" % (type(exc).__name__, exc),
}
except Exception as exc: # a package that raises on import is not healthy
return {
"status": "broken",
"available": False,
"import_name": import_name,
"error": "%s: %s" % (type(exc).__name__, exc),
}
try:
version = importlib.metadata.version(distribution)
except importlib.metadata.PackageNotFoundError:
version = getattr(mod, "__version__", "unknown")
return {
"status": "ok",
"available": True,
"import_name": import_name,
"version": version,
}
all_ok = all(
p.get("status") == "ok" or p.get("available", False)
for p in results["packages"].values()
def parse_requirement(text):
"""`NAME` or `DISTRIBUTION:IMPORT` -> (distribution, import name)."""
if ":" in text:
distribution, import_name = text.split(":", 1)
if not distribution or not import_name:
raise argparse.ArgumentTypeError(
"--require needs NAME or DISTRIBUTION:IMPORT, got %r" % text
)
return distribution, import_name
return text, text
def build_parser():
parser = argparse.ArgumentParser(
prog="python-env-check.py",
description="Verify the spark lane's third-party dependencies import, "
"and report their versions.",
)
parser.add_argument(
"--require",
metavar="NAME[:IMPORT]",
action="append",
default=[],
type=parse_requirement,
help="also require this package for this run (repeatable); "
"useful for proving the red path",
)
return parser
def main(argv=None):
args = build_parser().parse_args(argv)
declared = list(DEPENDENCIES) + list(args.require)
packages = {}
for distribution, import_name in declared:
result = check_package(distribution, import_name)
result["distribution"] = distribution
packages[import_name] = result
missing = sorted(
name for name, result in packages.items() if result["status"] != "ok"
)
output = {
"all_ok": all_ok,
"all_ok": not missing,
"timestamp": datetime.now(timezone.utc).isoformat(),
"components": results
"dependencies_declared": len(declared),
"missing": missing,
"components": {
"python": {
"status": "ok",
"version": sys.version,
"executable": sys.executable,
},
"packages": packages,
},
}
print(json.dumps(output, indent=2))
sys.exit(0 if all_ok else 1)
print(json.dumps(output, indent=2, sort_keys=True))
return 0 if not missing else 1
if __name__ == "__main__":
main()
sys.exit(main())
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env python3
"""Cells for wizard-work/spark/python-env-check.py (queue Q120).
The script under test is run as a subprocess, exactly as an operator runs
it; nothing here imports it. The red cell picks its missing package by
measuring the host first, so it is a real absence rather than a name the
test hopes is absent.
Run: python3 -m unittest discover -p 'test_python_env_check.py' -v
(from wizard-work/spark, or by path from anywhere)
"""
import importlib.util
import json
import os
import subprocess
import sys
import unittest
HERE = os.path.dirname(os.path.abspath(__file__))
SCRIPT = os.path.join(HERE, "python-env-check.py")
# Real, widely-installed packages the spark lane does not depend on. The red
# cell requires whichever of these the host genuinely lacks.
RED_CANDIDATES = ("flask", "django", "pandas", "scipy", "tensorflow", "torch", "black")
def run_script(*args, cwd=None):
proc = subprocess.run(
[sys.executable, SCRIPT] + list(args),
cwd=cwd or HERE,
capture_output=True,
text=True,
)
return proc
def absent_package():
"""Return the first candidate module that cannot be found on this host."""
for name in RED_CANDIDATES:
try:
found = importlib.util.find_spec(name)
except (ImportError, ValueError):
found = None
if found is None:
return name
return None
class EnvCheckContract(unittest.TestCase):
def setUp(self):
self.proc = run_script()
self.assertEqual(
self.proc.returncode in (0, 1),
True,
"expected rc 0 or 1, got %r: %s" % (self.proc.returncode, self.proc.stderr),
)
self.report = json.loads(self.proc.stdout)
def test_declared_list_is_not_the_standard_library(self):
# The defect Q120 filed: the only names checked were json/subprocess/
# datetime, so no healthy interpreter could ever fail this script.
packages = self.report["components"]["packages"]
self.assertTrue(packages, "no packages reported at all")
stdlib = set(getattr(sys, "stdlib_module_names", ()))
for import_name in packages:
self.assertNotIn(
import_name,
stdlib,
"%r is a standard-library module, not a lane dependency" % import_name,
)
def test_declared_list_is_the_lane_dependencies(self):
packages = self.report["components"]["packages"]
self.assertEqual(
set(packages), {"yaml", "numpy", "PIL", "jsonschema"}
)
# email_validator is the lane's own stdlib-only module, not the PyPI
# distribution; declaring it would test something the lane never uses.
self.assertNotIn("email_validator", packages)
def test_every_declared_dependency_reports_a_version(self):
packages = self.report["components"]["packages"]
for import_name, result in packages.items():
self.assertEqual(result["status"], "ok", "%s: %r" % (import_name, result))
self.assertIn("version", result)
self.assertIn("distribution", result)
self.assertFalse(
result["version"] in ("", "unknown"),
"%s reported no version: %r" % (import_name, result),
)
def test_all_ok_agrees_with_the_missing_list(self):
self.assertEqual(self.report["all_ok"], self.report["missing"] == [])
self.assertEqual(self.proc.returncode, 0 if self.report["all_ok"] else 1)
def test_genuinely_absent_package_turns_the_reading_red(self):
name = absent_package()
if name is None:
self.skipTest(
"every candidate %r is installed here; no real package to "
"prove the red path with" % (RED_CANDIDATES,)
)
proc = run_script("--require", name)
self.assertEqual(proc.returncode, 1, proc.stdout + proc.stderr)
report = json.loads(proc.stdout)
self.assertFalse(report["all_ok"])
self.assertEqual(report["missing"], [name])
result = report["components"]["packages"][name]
self.assertEqual(result["status"], "missing")
self.assertIn(name, result["error"])
# The healthy declared dependencies still report alongside it.
self.assertEqual(report["dependencies_declared"], 5)
self.assertEqual(
report["components"]["packages"]["yaml"]["status"], "ok"
)
def test_usage_error_is_rc_two(self):
proc = run_script("--require")
self.assertEqual(proc.returncode, 2, proc.stderr)
def test_same_reading_from_a_different_cwd(self):
other = run_script(cwd="/")
self.assertEqual(other.returncode, self.proc.returncode)
self.assertEqual(json.loads(other.stdout)["missing"], self.report["missing"])
if __name__ == "__main__":
unittest.main(verbosity=2)