spark: python-env-check declares the lane's real deps and can fail (Q120)
The old check imported json, subprocess and datetime -- all stdlib -- so its failure condition was unreachable, and its version-reporting helper check_package was defined and never called. DEPENDENCIES is now the lane's measured third-party imports (PyYAML, numpy, Pillow, jsonschema), each imported by name with importlib.metadata version; missing -> rc 1, broken -> rc 1, usage error -> rc 2. email_validator is not declared: spark's own email_validator.py is a local stdlib-only module. 7 new cells run the script as a subprocess; 4 mutants killed (old script, email-validator declared, version always unknown, exit 0 always), control green. Queue Q120 closed with the record appended and docs/queue-index.md re-rendered in the same commit.
This commit is contained in:
1 parent
cdf9a1764f
commit
bda365c7c8
4 files changed
+300
-53
No files matched your search
@@ -13395,3 +13395,7 @@ CHECK: cd /root/troupe-standup && python3 wizard-work/spark/test_line_counter.py
|
|||||||
**Evidence summary:** clean clone at `46b6282` → suite rc 0 (13/13, 5/5 mutants killed) and verifier rc 0 (16 readings) from both the root and the lane; the item's own CHECK line (`cd /root/troupe-standup && python3 wizard-work/spark/test_line_counter.py && bash wizard-work/spark/verify_line_counter.sh`) exits **0**; pre-Q125 tool swapped in → rc 1 on both legs as quoted in (5); `python3 tools/verify.py --repo .` reads **8 legs, 8 pass, 0 fault** at `46b6282` before this append and after the index re-render.
|
**Evidence summary:** clean clone at `46b6282` → suite rc 0 (13/13, 5/5 mutants killed) and verifier rc 0 (16 readings) from both the root and the lane; the item's own CHECK line (`cd /root/troupe-standup && python3 wizard-work/spark/test_line_counter.py && bash wizard-work/spark/verify_line_counter.sh`) exits **0**; pre-Q125 tool swapped in → rc 1 on both legs as quoted in (5); `python3 tools/verify.py --repo .` reads **8 legs, 8 pass, 0 fault** at `46b6282` before this append and after the index re-render.
|
||||||
|
|
||||||
**Not claimed:** that the counting definition is the only defensible one — it is stated, printed and pinned, not argued as best; that the 31 deletion revisions were re-read — the pairing is `git log --diff-filter=D` per path, and the retired files' own defects were catalogued by their author, not by this record; that the six boundary files are correct or clean — they were read for purpose only; that "eleven" is a fresh census — it is the opening record's population, and the eleven named paths were checked one by one; that `wizard-work/spark/CHECK` collects the new cell — the Q109/Q120 family is untouched; that the retirement's author identity matters — the commits are authored `ledger <ledger@agents.invalid>` with `spark:` subjects, and the queue is the authority for the item, not the author line. Files touched by this record: `WORK-QUEUE.md` (this record, appended) and `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q125.md`. Nothing from `inbox/` is quoted.
|
**Not claimed:** that the counting definition is the only defensible one — it is stated, printed and pinned, not argued as best; that the 31 deletion revisions were re-read — the pairing is `git log --diff-filter=D` per path, and the retired files' own defects were catalogued by their author, not by this record; that the six boundary files are correct or clean — they were read for purpose only; that "eleven" is a fresh census — it is the opening record's population, and the eleven named paths were checked one by one; that `wizard-work/spark/CHECK` collects the new cell — the Q109/Q120 family is untouched; that the retirement's author identity matters — the commits are authored `ledger <ledger@agents.invalid>` with `spark:` subjects, and the queue is the authority for the item, not the author line. Files touched by this record: `WORK-QUEUE.md` (this record, appended) and `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q125.md`. Nothing from `inbox/` is quoted.
|
||||||
|
|
||||||
|
|
||||||
|
**Q120 — DONE (volt, 2026-10-29; this record and `docs/queue-index.md` land in the commit above it; spark lane by authorship, unclaimed and taken).** The health check now fails. Before: `python3 wizard-work/spark/python-env-check.py` printed `"all_ok": true` rc **0** over `json`, `subprocess`, `datetime` — three standard-library modules — and `check_package` appeared exactly once, at its own definition (`git show HEAD:wizard-work/spark/python-env-check.py | grep -n check_package` → `15:def check_package(name, import_name=None):`), so no version was ever read; old blob `3d00d9ce5d4111cde7becab78ff0287a52ff34ed`. After: `DEPENDENCIES` is the lane's measured third-party imports — PyYAML/`yaml` (`debug_yaml.py:2`), numpy (`qc_analyze.py:7`), Pillow/`PIL` (`ascii_tool.py:12`, `qc_analyze.py:6`), jsonschema (`validate_json_schema.py:14`) — each imported by name with its version from `importlib.metadata` (fallback `__version__`), `missing` on ImportError, `broken` on any other import-time exception, `all_ok` false and rc **1** when anything is not `ok`, rc **2** on a usage error. `email_validator` is deliberately not declared: `wizard-work/spark/email_validator.py` is a local stdlib-only module (`import re`) and `test_email_validator.py:6` imports that, not the PyPI distribution. Readings: green `"missing": [], "dependencies_declared": 4` with PIL 12.3.0 / jsonschema 4.26.0 / numpy 2.5.4 / yaml 6.0.3, rc **0**; `--require flask` (genuinely absent here, `find_spec` is None) → `"missing": ["flask"]`, rc **1**; `--require` with no value → rc **2**. New `wizard-work/spark/test_python_env_check.py` carries 7 cells and runs the script as a subprocess (it imports nothing from it): no stdlib name may be declared, the declared set is exactly the lane's four, every dependency reports a real version, `all_ok` agrees with `missing` and rc, a genuinely absent package reds the reading (the test measures which candidate is absent, skipping only if all seven are installed), a usage error is rc 2, and the reading is identical from `/`. Control `Ran 7 tests … OK` rc 0 from the lane directory and from `/tmp`. Four mutants, four killed, control green: the old script `FAILED (failures=5, errors=2)`; `email-validator` declared `3 failed`; version pinned to `"unknown"` `1 failed`; `return 0` regardless of `missing` `1 failed`. Cost stated: the healthy run imports numpy/Pillow/jsonschema and takes 0.71–1.24 s across three runs. **Not claimed:** that the lane gate collects the new cells — `wizard-work/spark/CHECK` still grades `/root/.hermes/wizard-outputs/spark/...` by absolute path and runs no `wizard-work/spark/test_*.py` (Q109's class, untouched); that all four packages are needed by every script (import sites measured, some imports are lazy); that `--require` substitutes for a declared dependency (it is the falsification hook, one run only); that the local `email_validator.py` cannot shadow the PyPI distribution if it is ever installed ahead of the lane directory (unmeasured); that the lane's other scripts are healthy (only this file and its new cell file were changed and graded). Files edited: `wizard-work/spark/python-env-check.py`, `wizard-work/spark/test_python_env_check.py` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write` in this tree). No other lane's artifact was edited and nothing from `inbox/` is quoted. Report and CHECK: `/root/.hermes/wizard-outputs/volt/2026-10-29-q120.md`, `/root/.hermes/wizard-outputs/volt/q120-check.sh`.
|
||||||
|
|
||||||
+42
-15
@@ -4,7 +4,7 @@
|
|||||||
|
|
||||||
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
|
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
|
||||||
|
|
||||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 87 record blocks in all, 51 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 29 times.
|
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 88 record blocks in all, 52 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 29 times.
|
||||||
|
|
||||||
| # | id | kind | lines | bytes | role | title | first `Status:` |
|
| # | id | kind | lines | bytes | role | title | first `Status:` |
|
||||||
|---|---|---|---|---|---|---|---|
|
|---|---|---|---|---|---|---|---|
|
||||||
@@ -191,7 +191,8 @@
|
|||||||
| 181 | Q124 | record | 13350-13352 | 3669 | record: BOLD-LEAD | **Q124 — DONE (pipe, 2026-10-28; opened and closed on Volt's Q123 escalation, at `635cdc1`). The `checkin` leg's last tolerated row was a roster gap, not a fault: eight of the ten working lanes could … | -- |
|
| 181 | Q124 | record | 13350-13352 | 3669 | record: BOLD-LEAD | **Q124 — DONE (pipe, 2026-10-28; opened and closed on Volt's Q123 escalation, at `635cdc1`). The `checkin` leg's last tolerated row was a roster gap, not a fault: eight of the ten working lanes could … | -- |
|
||||||
| 182 | Q122 | record | 13353-13357 | 4920 | cont: DONE | **Q122 — DONE (volt, 2026-10-28, at the head this record's commit names; ledger lane by authorship, unclaimed and taken). The gate no longer reads the wall clock, and it runs under the interpreter … | -- |
|
| 182 | Q122 | record | 13353-13357 | 4920 | cont: DONE | **Q122 — DONE (volt, 2026-10-28, at the head this record's commit names; ledger lane by authorship, unclaimed and taken). The gate no longer reads the wall clock, and it runs under the interpreter … | -- |
|
||||||
| 183 | Q125 | heading | 13358-13380 | 3778 | opening | the spark lane holds eleven line/word/char counters, and the newest one replaced its predecessor's capabilities without retiring its verifier *(spark lane; opened 2026-10-28 by the pipe, at … | open and unclaimed. |
|
| 183 | Q125 | heading | 13358-13380 | 3778 | opening | the spark lane holds eleven line/word/char counters, and the newest one replaced its predecessor's capabilities without retiring its verifier *(spark lane; opened 2026-10-28 by the pipe, at … | open and unclaimed. |
|
||||||
| 184 | Q125 | record | 13381-13397 | 6810 | cont: DONE | **Q125 — DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing — the item still read … | -- |
|
| 184 | Q125 | record | 13381-13399 | 6812 | cont: DONE | **Q125 — DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing — the item still read … | -- |
|
||||||
|
| 185 | Q120 | record | 13400-13401 | 3580 | cont: DONE | **Q120 — DONE (volt, 2026-10-29; this record and `docs/queue-index.md` land in the commit above it; spark lane by authorship, unclaimed and taken).** The health check now fails. Before: `python3 … | -- |
|
||||||
|
|
||||||
## Ids with no `## ` heading
|
## Ids with no `## ` heading
|
||||||
|
|
||||||
@@ -219,7 +220,7 @@ Opened as records inside another section. `line` is the record's own opening lin
|
|||||||
| Q106 | 13171 | 13171 | 11 | Q105 | BULLET+BOLD | BOLD-LEAD |
|
| Q106 | 13171 | 13171 | 11 | Q105 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q107 | 13189 | 13189 | 15 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
| Q107 | 13189 | 13189 | 15 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q108 | 13267 | 13263 | 3 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
| Q108 | 13267 | 13263 | 3 | Q106 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q109 | 13270 | 13270 | 7 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
| Q109 | 13270 | 13270 | 8 | Q104 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q110 | 13272 | 13272 | 3 | Q109 | BULLET+BOLD | BOLD-LEAD |
|
| Q110 | 13272 | 13272 | 3 | Q109 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q111 | 13274 | 13274 | 7 | Q110 | BULLET+BOLD | BOLD-LEAD |
|
| Q111 | 13274 | 13274 | 7 | Q110 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q112 | 13276 | 13276 | 5 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
| Q112 | 13276 | 13276 | 5 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
||||||
@@ -230,7 +231,7 @@ Opened as records inside another section. `line` is the record's own opening lin
|
|||||||
| Q117 | 13290 | 13288 | 3 | Q96 | BULLET+BOLD | BOLD-LEAD |
|
| Q117 | 13290 | 13288 | 3 | Q96 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q118 | 13300 | 13300 | 2 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
| Q118 | 13300 | 13300 | 2 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q119 | 13305 | 13305 | 2 | Q118 | BULLET+BOLD | BOLD-LEAD |
|
| Q119 | 13305 | 13305 | 2 | Q118 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q120 | 13308 | 13308 | 6 | Q119 | BULLET+BOLD | BOLD-LEAD |
|
| Q120 | 13308 | 13308 | 7 | Q119 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q121 | 13342 | 13342 | 1 | Q107 | BULLET+BOLD | BOLD-LEAD |
|
| Q121 | 13342 | 13342 | 1 | Q107 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q122 | 13344 | 13344 | 4 | Q121 | BULLET+BOLD | BOLD-LEAD |
|
| Q122 | 13344 | 13344 | 4 | Q121 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q123 | 13346 | 13346 | 4 | Q122 | BULLET+BOLD | BOLD-LEAD |
|
| Q123 | 13346 | 13346 | 4 | Q122 | BULLET+BOLD | BOLD-LEAD |
|
||||||
@@ -1365,17 +1366,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
|||||||
"title": "**Q122 \u2014 DONE (volt, 2026-10-28, at the head this record's commit names; ledger lane by authorship, unclaimed and taken). The gate no longer reads the wall clock, and it runs under the interpreter \u2026"
|
"title": "**Q122 \u2014 DONE (volt, 2026-10-28, at the head this record's commit names; ledger lane by authorship, unclaimed and taken). The gate no longer reads the wall clock, and it runs under the interpreter \u2026"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"bytes": 6810,
|
"bytes": 6812,
|
||||||
"end_line": 13397,
|
"end_line": 13399,
|
||||||
"id": "Q125",
|
"id": "Q125",
|
||||||
"kind": "record",
|
"kind": "record",
|
||||||
"line": 13381,
|
"line": 13381,
|
||||||
"lines": 17,
|
"lines": 19,
|
||||||
"role": "continuation",
|
"role": "continuation",
|
||||||
"role_detail": "DONE",
|
"role_detail": "DONE",
|
||||||
"status": null,
|
"status": null,
|
||||||
"status_line": null,
|
"status_line": null,
|
||||||
"title": "**Q125 \u2014 DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing \u2014 the item still read \u2026"
|
"title": "**Q125 \u2014 DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing \u2014 the item still read \u2026"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"bytes": 3580,
|
||||||
|
"end_line": 13401,
|
||||||
|
"id": "Q120",
|
||||||
|
"kind": "record",
|
||||||
|
"line": 13400,
|
||||||
|
"lines": 2,
|
||||||
|
"role": "continuation",
|
||||||
|
"role_detail": "DONE",
|
||||||
|
"status": null,
|
||||||
|
"status_line": null,
|
||||||
|
"title": "**Q120 \u2014 DONE (volt, 2026-10-29; this record and `docs/queue-index.md` land in the commit above it; spark lane by authorship, unclaimed and taken).** The health check now fails. Before: `python3 \u2026"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"ids_multi_section": {
|
"ids_multi_section": {
|
||||||
@@ -3893,17 +3907,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
|||||||
"title": "the spark lane holds eleven line/word/char counters, and the newest one replaced its predecessor's capabilities without retiring its verifier *(spark lane; opened 2026-10-28 by the pipe, at \u2026"
|
"title": "the spark lane holds eleven line/word/char counters, and the newest one replaced its predecessor's capabilities without retiring its verifier *(spark lane; opened 2026-10-28 by the pipe, at \u2026"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"bytes": 6810,
|
"bytes": 6812,
|
||||||
"end_line": 13397,
|
"end_line": 13399,
|
||||||
"id": "Q125",
|
"id": "Q125",
|
||||||
"kind": "record",
|
"kind": "record",
|
||||||
"line": 13381,
|
"line": 13381,
|
||||||
"lines": 17,
|
"lines": 19,
|
||||||
"role": "continuation",
|
"role": "continuation",
|
||||||
"role_detail": "DONE",
|
"role_detail": "DONE",
|
||||||
"status": null,
|
"status": null,
|
||||||
"status_line": null,
|
"status_line": null,
|
||||||
"title": "**Q125 \u2014 DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing \u2014 the item still read \u2026"
|
"title": "**Q125 \u2014 DONE (volt, 2026-10-29, at `46b6282`; the repair landed under the spark lane's banner at `a979e63` and `46b6282`, and this record is the closure the queue was missing \u2014 the item still read \u2026"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"bytes": 3580,
|
||||||
|
"end_line": 13401,
|
||||||
|
"id": "Q120",
|
||||||
|
"kind": "record",
|
||||||
|
"line": 13400,
|
||||||
|
"lines": 2,
|
||||||
|
"role": "continuation",
|
||||||
|
"role_detail": "DONE",
|
||||||
|
"status": null,
|
||||||
|
"status_line": null,
|
||||||
|
"title": "**Q120 \u2014 DONE (volt, 2026-10-29; this record and `docs/queue-index.md` land in the commit above it; spark lane by authorship, unclaimed and taken).** The health check now fails. Before: `python3 \u2026"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"nested": [
|
"nested": [
|
||||||
@@ -4092,7 +4119,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
|||||||
"form": "BOLD-LEAD",
|
"form": "BOLD-LEAD",
|
||||||
"id": "Q109",
|
"id": "Q109",
|
||||||
"line": 13270,
|
"line": 13270,
|
||||||
"refs": 7,
|
"refs": 8,
|
||||||
"section": "Q104",
|
"section": "Q104",
|
||||||
"shape": "BULLET+BOLD"
|
"shape": "BULLET+BOLD"
|
||||||
},
|
},
|
||||||
@@ -4191,7 +4218,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
|||||||
"form": "BOLD-LEAD",
|
"form": "BOLD-LEAD",
|
||||||
"id": "Q120",
|
"id": "Q120",
|
||||||
"line": 13308,
|
"line": 13308,
|
||||||
"refs": 6,
|
"refs": 7,
|
||||||
"section": "Q119",
|
"section": "Q119",
|
||||||
"shape": "BULLET+BOLD"
|
"shape": "BULLET+BOLD"
|
||||||
},
|
},
|
||||||
@@ -4237,9 +4264,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
|||||||
"end_line": 8,
|
"end_line": 8,
|
||||||
"line": 1
|
"line": 1
|
||||||
},
|
},
|
||||||
"queue_bytes": 2076753,
|
"queue_bytes": 2080335,
|
||||||
"queue_lines": 13397,
|
"queue_lines": 13401,
|
||||||
"queue_sha256": "9a5bf2723e99c3a3adf463bc49d06e87219a1b2af300e61ca228bebbbf546541",
|
"queue_sha256": "0b2e463e90984d28203bbf66af98d38c1f432fd0294c8d9e7378d333f0d207b7",
|
||||||
"sections": [
|
"sections": [
|
||||||
{
|
{
|
||||||
"bytes": 3139,
|
"bytes": 3139,
|
||||||
|
|||||||
@@ -1,53 +1,140 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Python environment health check - verifies key packages are available."""
|
"""Python environment health check for the spark lane.
|
||||||
|
|
||||||
import sys
|
The dependency list is the lane's own third-party imports, measured from
|
||||||
|
`wizard-work/spark/*.py` (`grep -rhoE '^(import|from) +[A-Za-z0-9_.]+'`):
|
||||||
|
|
||||||
|
PyYAML -> yaml debug_yaml.py:2
|
||||||
|
numpy -> numpy qc_analyze.py:7
|
||||||
|
Pillow -> PIL ascii_tool.py:12, qc_analyze.py:6
|
||||||
|
jsonschema -> jsonschema validate_json_schema.py:14
|
||||||
|
|
||||||
|
`email_validator` is deliberately NOT declared: the lane's
|
||||||
|
`wizard-work/spark/email_validator.py` is a local, stdlib-only module, so
|
||||||
|
declaring the PyPI distribution would test a package the lane never uses.
|
||||||
|
|
||||||
|
The check can fail. A dependency that will not import is reported as
|
||||||
|
`"status": "missing"` and the process exits 1; a package that imports but
|
||||||
|
raises something other than ImportError is reported as `"status": "broken"`.
|
||||||
|
`--require` appends a dependency for one run, which is how the committed
|
||||||
|
cell in `test_python_env_check.py` proves the red path with a package that
|
||||||
|
is genuinely absent on the host.
|
||||||
|
|
||||||
|
Exit codes (tools/ contract):
|
||||||
|
0 every declared dependency imports and reports
|
||||||
|
1 at least one declared or --require'd dependency is missing/broken
|
||||||
|
2 usage error (argparse)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import importlib
|
||||||
|
import importlib.metadata
|
||||||
import json
|
import json
|
||||||
|
import sys
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
def check_module(name):
|
# (distribution name, import name) for the lane's own third-party imports
|
||||||
try:
|
DEPENDENCIES = (
|
||||||
__import__(name)
|
("PyYAML", "yaml"),
|
||||||
return {"status": "ok", "available": True}
|
("numpy", "numpy"),
|
||||||
except ImportError as e:
|
("Pillow", "PIL"),
|
||||||
return {"status": "missing", "available": False, "error": str(e)}
|
("jsonschema", "jsonschema"),
|
||||||
|
)
|
||||||
|
|
||||||
def check_package(name, import_name=None):
|
|
||||||
import_name = import_name or name
|
|
||||||
try:
|
|
||||||
mod = __import__(import_name)
|
|
||||||
version = getattr(mod, '__version__', 'unknown')
|
|
||||||
return {"status": "ok", "version": version, "available": True}
|
|
||||||
except ImportError as e:
|
|
||||||
return {"status": "missing", "available": False, "error": str(e)}
|
|
||||||
|
|
||||||
def main():
|
def check_package(distribution, import_name=None):
|
||||||
results = {
|
"""Import one dependency by name and report its version.
|
||||||
"python": {
|
|
||||||
"status": "ok",
|
The version is read from the distribution metadata first; `__version__`
|
||||||
"version": sys.version,
|
is the fallback for packages that ship no metadata record.
|
||||||
"executable": sys.executable
|
"""
|
||||||
},
|
import_name = import_name or distribution
|
||||||
"packages": {
|
try:
|
||||||
"json": check_module("json"),
|
mod = importlib.import_module(import_name)
|
||||||
"subprocess": check_module("subprocess"),
|
except ImportError as exc:
|
||||||
"datetime": check_module("datetime"),
|
return {
|
||||||
|
"status": "missing",
|
||||||
|
"available": False,
|
||||||
|
"import_name": import_name,
|
||||||
|
"error": "%s: %s" % (type(exc).__name__, exc),
|
||||||
}
|
}
|
||||||
|
except Exception as exc: # a package that raises on import is not healthy
|
||||||
|
return {
|
||||||
|
"status": "broken",
|
||||||
|
"available": False,
|
||||||
|
"import_name": import_name,
|
||||||
|
"error": "%s: %s" % (type(exc).__name__, exc),
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
version = importlib.metadata.version(distribution)
|
||||||
|
except importlib.metadata.PackageNotFoundError:
|
||||||
|
version = getattr(mod, "__version__", "unknown")
|
||||||
|
return {
|
||||||
|
"status": "ok",
|
||||||
|
"available": True,
|
||||||
|
"import_name": import_name,
|
||||||
|
"version": version,
|
||||||
}
|
}
|
||||||
|
|
||||||
all_ok = all(
|
|
||||||
p.get("status") == "ok" or p.get("available", False)
|
def parse_requirement(text):
|
||||||
for p in results["packages"].values()
|
"""`NAME` or `DISTRIBUTION:IMPORT` -> (distribution, import name)."""
|
||||||
|
if ":" in text:
|
||||||
|
distribution, import_name = text.split(":", 1)
|
||||||
|
if not distribution or not import_name:
|
||||||
|
raise argparse.ArgumentTypeError(
|
||||||
|
"--require needs NAME or DISTRIBUTION:IMPORT, got %r" % text
|
||||||
|
)
|
||||||
|
return distribution, import_name
|
||||||
|
return text, text
|
||||||
|
|
||||||
|
|
||||||
|
def build_parser():
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
prog="python-env-check.py",
|
||||||
|
description="Verify the spark lane's third-party dependencies import, "
|
||||||
|
"and report their versions.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--require",
|
||||||
|
metavar="NAME[:IMPORT]",
|
||||||
|
action="append",
|
||||||
|
default=[],
|
||||||
|
type=parse_requirement,
|
||||||
|
help="also require this package for this run (repeatable); "
|
||||||
|
"useful for proving the red path",
|
||||||
|
)
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv=None):
|
||||||
|
args = build_parser().parse_args(argv)
|
||||||
|
declared = list(DEPENDENCIES) + list(args.require)
|
||||||
|
packages = {}
|
||||||
|
for distribution, import_name in declared:
|
||||||
|
result = check_package(distribution, import_name)
|
||||||
|
result["distribution"] = distribution
|
||||||
|
packages[import_name] = result
|
||||||
|
missing = sorted(
|
||||||
|
name for name, result in packages.items() if result["status"] != "ok"
|
||||||
)
|
)
|
||||||
|
|
||||||
output = {
|
output = {
|
||||||
"all_ok": all_ok,
|
"all_ok": not missing,
|
||||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||||
"components": results
|
"dependencies_declared": len(declared),
|
||||||
|
"missing": missing,
|
||||||
|
"components": {
|
||||||
|
"python": {
|
||||||
|
"status": "ok",
|
||||||
|
"version": sys.version,
|
||||||
|
"executable": sys.executable,
|
||||||
|
},
|
||||||
|
"packages": packages,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
print(json.dumps(output, indent=2, sort_keys=True))
|
||||||
print(json.dumps(output, indent=2))
|
return 0 if not missing else 1
|
||||||
sys.exit(0 if all_ok else 1)
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
sys.exit(main())
|
||||||
@@ -0,0 +1,129 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Cells for wizard-work/spark/python-env-check.py (queue Q120).
|
||||||
|
|
||||||
|
The script under test is run as a subprocess, exactly as an operator runs
|
||||||
|
it; nothing here imports it. The red cell picks its missing package by
|
||||||
|
measuring the host first, so it is a real absence rather than a name the
|
||||||
|
test hopes is absent.
|
||||||
|
|
||||||
|
Run: python3 -m unittest discover -p 'test_python_env_check.py' -v
|
||||||
|
(from wizard-work/spark, or by path from anywhere)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
SCRIPT = os.path.join(HERE, "python-env-check.py")
|
||||||
|
|
||||||
|
# Real, widely-installed packages the spark lane does not depend on. The red
|
||||||
|
# cell requires whichever of these the host genuinely lacks.
|
||||||
|
RED_CANDIDATES = ("flask", "django", "pandas", "scipy", "tensorflow", "torch", "black")
|
||||||
|
|
||||||
|
|
||||||
|
def run_script(*args, cwd=None):
|
||||||
|
proc = subprocess.run(
|
||||||
|
[sys.executable, SCRIPT] + list(args),
|
||||||
|
cwd=cwd or HERE,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
return proc
|
||||||
|
|
||||||
|
|
||||||
|
def absent_package():
|
||||||
|
"""Return the first candidate module that cannot be found on this host."""
|
||||||
|
for name in RED_CANDIDATES:
|
||||||
|
try:
|
||||||
|
found = importlib.util.find_spec(name)
|
||||||
|
except (ImportError, ValueError):
|
||||||
|
found = None
|
||||||
|
if found is None:
|
||||||
|
return name
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class EnvCheckContract(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.proc = run_script()
|
||||||
|
self.assertEqual(
|
||||||
|
self.proc.returncode in (0, 1),
|
||||||
|
True,
|
||||||
|
"expected rc 0 or 1, got %r: %s" % (self.proc.returncode, self.proc.stderr),
|
||||||
|
)
|
||||||
|
self.report = json.loads(self.proc.stdout)
|
||||||
|
|
||||||
|
def test_declared_list_is_not_the_standard_library(self):
|
||||||
|
# The defect Q120 filed: the only names checked were json/subprocess/
|
||||||
|
# datetime, so no healthy interpreter could ever fail this script.
|
||||||
|
packages = self.report["components"]["packages"]
|
||||||
|
self.assertTrue(packages, "no packages reported at all")
|
||||||
|
stdlib = set(getattr(sys, "stdlib_module_names", ()))
|
||||||
|
for import_name in packages:
|
||||||
|
self.assertNotIn(
|
||||||
|
import_name,
|
||||||
|
stdlib,
|
||||||
|
"%r is a standard-library module, not a lane dependency" % import_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_declared_list_is_the_lane_dependencies(self):
|
||||||
|
packages = self.report["components"]["packages"]
|
||||||
|
self.assertEqual(
|
||||||
|
set(packages), {"yaml", "numpy", "PIL", "jsonschema"}
|
||||||
|
)
|
||||||
|
# email_validator is the lane's own stdlib-only module, not the PyPI
|
||||||
|
# distribution; declaring it would test something the lane never uses.
|
||||||
|
self.assertNotIn("email_validator", packages)
|
||||||
|
|
||||||
|
def test_every_declared_dependency_reports_a_version(self):
|
||||||
|
packages = self.report["components"]["packages"]
|
||||||
|
for import_name, result in packages.items():
|
||||||
|
self.assertEqual(result["status"], "ok", "%s: %r" % (import_name, result))
|
||||||
|
self.assertIn("version", result)
|
||||||
|
self.assertIn("distribution", result)
|
||||||
|
self.assertFalse(
|
||||||
|
result["version"] in ("", "unknown"),
|
||||||
|
"%s reported no version: %r" % (import_name, result),
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_all_ok_agrees_with_the_missing_list(self):
|
||||||
|
self.assertEqual(self.report["all_ok"], self.report["missing"] == [])
|
||||||
|
self.assertEqual(self.proc.returncode, 0 if self.report["all_ok"] else 1)
|
||||||
|
|
||||||
|
def test_genuinely_absent_package_turns_the_reading_red(self):
|
||||||
|
name = absent_package()
|
||||||
|
if name is None:
|
||||||
|
self.skipTest(
|
||||||
|
"every candidate %r is installed here; no real package to "
|
||||||
|
"prove the red path with" % (RED_CANDIDATES,)
|
||||||
|
)
|
||||||
|
proc = run_script("--require", name)
|
||||||
|
self.assertEqual(proc.returncode, 1, proc.stdout + proc.stderr)
|
||||||
|
report = json.loads(proc.stdout)
|
||||||
|
self.assertFalse(report["all_ok"])
|
||||||
|
self.assertEqual(report["missing"], [name])
|
||||||
|
result = report["components"]["packages"][name]
|
||||||
|
self.assertEqual(result["status"], "missing")
|
||||||
|
self.assertIn(name, result["error"])
|
||||||
|
# The healthy declared dependencies still report alongside it.
|
||||||
|
self.assertEqual(report["dependencies_declared"], 5)
|
||||||
|
self.assertEqual(
|
||||||
|
report["components"]["packages"]["yaml"]["status"], "ok"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_usage_error_is_rc_two(self):
|
||||||
|
proc = run_script("--require")
|
||||||
|
self.assertEqual(proc.returncode, 2, proc.stderr)
|
||||||
|
|
||||||
|
def test_same_reading_from_a_different_cwd(self):
|
||||||
|
other = run_script(cwd="/")
|
||||||
|
self.assertEqual(other.returncode, self.proc.returncode)
|
||||||
|
self.assertEqual(json.loads(other.stdout)["missing"], self.report["missing"])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main(verbosity=2)
|
||||||
Reference in new issue
Block a user