volt: Q161 -- jobs-table entry point is a bash wrapper over a grader that can say 'cannot grade'

wizard-work/ledger/CHECK-2026-11-02-jobs-table landed at 2e57e6b as a Python program with a python3 shebang, so `bash <path>` -- the invocation the q143 sweep, verify.py's doc-command leg and any reader use -- read rc 2 on shell syntax before grading a cell. Its derivation also read three source JSONs from the literal path /root, so with those absent it raised FileNotFoundError and exited 1: could-not-grade printed as a fault.

The grader moved to grade_jobs_table.py; the path keeps a bash wrapper. SRC_ROOT is JOBS_SRC_ROOT (default /root); a source mode with sources absent now exits 2 and says CANNOT GRADE; a new --offline mode grades the committed artifact and states that re-derivation was not attempted. CHECK-q161 (6 legs) grades the wrapper, the pre-repair blob at 2e57e6b under bash (rc 2), the hostless copy (rc 2), a tampered copy (rc 1), --offline, and the census row.
This commit is contained in:
volt committed 2026-11-03 05:52:58 +00:00
1 parent 1bd59257fe
commit 82878043cb
7 files changed
+501 -208

No files matched your search

+105
View File
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
# CHECK-q161 -- the jobs-table entry point ran under an interpreter nobody
# invokes, and read a host root as if it were the repository (volt, 2026-11-03).
#
# The defect, measured at 2e57e6b: `wizard-work/ledger/CHECK-2026-11-02-jobs-table`
# was a Python program with a `python3` shebang, so the invocation every caller in
# this tree uses -- `bash <path>`, per the q143 sweep and tools/verify.py's
# doc-command leg -- read rc 2 on shell syntax errors before grading a cell. And
# its derivation read three source JSONs from the literal path `/root`, so with
# those absent it raised FileNotFoundError and exited 1: "could not grade" printed
# as a fault. The grader moved beside it (grade_jobs_table.py) and the path keeps
# a bash wrapper; the source root is overridable with JOBS_SRC_ROOT and an absent
# source now exits 2.
#
# Six legs:
# entry bash <path> at the tree reads 0 and prints VERDICT: PASS
# prefix the pre-repair blob from 2e57e6b, run under bash, reads 2 -- the
# cell that only works if the regression is real, not remembered
# hostless a copy of the grader with JOBS_SRC_ROOT at an empty directory
# reads 2 and says CANNOT GRADE (before: traceback, rc 1)
# tamper the same copy with one byte appended to jobs.csv reads 1
# offline `--offline` at the tree reads 0 and states re-derivation was
# not attempted, so the weaker reading is never silent
# sweep the q143 path census grades this entry point at 0
#
# Exit codes: 0 all legs passed, 1 a fault, 2 could not grade.
set -u
set -o pipefail
cd "$(dirname "${BASH_SOURCE[0]}")" || exit 2
ROOT="$(pwd)"
LEDGER="$ROOT/wizard-work/ledger"
ENTRY="$LEDGER/CHECK-2026-11-02-jobs-table"
GRADER="$LEDGER/grade_jobs_table.py"
SWEEP="$ROOT/wizard-work/volt/q143-check-sweep/check_sweep.py"
PY="${PYTHON:-python3}"
WORK="$(mktemp -d "${TMPDIR:-/tmp}/check-q161.XXXXXX")" || exit 2
trap 'rm -rf "$WORK"' EXIT
for f in "$ENTRY" "$GRADER" "$LEDGER/manifest.json" "$LEDGER/jobs.csv" "$SWEEP"; do
if [ ! -f "$f" ]; then
printf 'CANNOT GRADE: missing %s\n' "$f"
exit 2
fi
done
# The prefix leg grades a REVISION, so a .git-less export cannot answer it.
# That is rc 2 -- "could not grade" -- not a fault, per the repo's tools/ contract.
if ! git -C "$ROOT" rev-parse --show-toplevel >/dev/null 2>&1; then
printf 'CANNOT GRADE: %s is not a git work tree\n' "$ROOT"
exit 2
fi
passed=0
failed=0
row() { printf ' %-11s %s\n' "$1" "$2"; }
verdict() { if [ "$2" -eq 0 ]; then passed=$((passed + 1)); else failed=$((failed + 1)); fi; row "$1" "$3"; }
# --- leg 1: the entry point as every caller invokes it ----------------------
out="$(bash "$ENTRY" 2>&1)"; erc=$?
last="$(printf '%s' "$out" | tail -1)"
verdict entry "$([ "$erc" -eq 0 ] && printf '%s' "$last" | grep -q '^VERDICT: PASS' && echo 0 || echo 1)" \
"rc=$erc ${last:-<no output>}"
# --- leg 2: the pre-repair blob, same invocation ----------------------------
git -C "$ROOT" show 2e57e6b:wizard-work/ledger/CHECK-2026-11-02-jobs-table \
> "$WORK/pre_check" 2>/dev/null || { printf 'CANNOT GRADE: 2e57e6b blob unreadable\n'; exit 2; }
out="$(bash "$WORK/pre_check" 2>&1)"; prc=$?
last="$(printf '%s' "$out" | grep -E 'syntax error|command not found' | tail -1)"
verdict prefix "$([ "$prc" -eq 2 ] && echo 0 || echo 1)" \
"rc=$prc (want 2) ${last:-<no shell error line>}"
# --- legs 3 and 4: a copy, graded where the sources are absent --------------
cp "$GRADER" "$LEDGER/manifest.json" "$LEDGER/jobs.csv" "$WORK/" \
|| { printf 'CANNOT GRADE: artifact copy failed\n'; exit 2; }
mkdir -p "$WORK/empty" || exit 2
out="$(cd "$WORK" && JOBS_SRC_ROOT="$WORK/empty" "$PY" grade_jobs_table.py 2>&1)"; hrc=$?
last="$(printf '%s' "$out" | tail -1)"
verdict hostless "$([ "$hrc" -eq 2 ] && printf '%s' "$last" | grep -q '^CANNOT GRADE' && echo 0 || echo 1)" \
"rc=$hrc (want 2) ${last:-<no output>}"
printf 'x' >> "$WORK/jobs.csv" || { printf 'CANNOT GRADE: tamper failed\n'; exit 2; }
out="$(cd "$WORK" && "$PY" grade_jobs_table.py --offline 2>&1)"; trc=$?
last="$(printf '%s' "$out" | grep -E '^(VERDICT| - )' | tail -1)"
verdict tamper "$([ "$trc" -eq 1 ] && echo 0 || echo 1)" \
"rc=$trc (want 1) ${last:-<no verdict line>}"
# --- leg 5: the weaker mode must say what it did not measure ----------------
out="$(bash "$ENTRY" --offline 2>&1)"; orc=$?
last="$(printf '%s' "$out" | grep 're_derivation' | tail -1)"
verdict offline "$([ "$orc" -eq 0 ] && printf '%s' "$last" | grep -q 'NOT ATTEMPTED' && echo 0 || echo 1)" \
"rc=$orc ${last:-<no re_derivation line>}"
# --- leg 6: the census of entry points grades this path ---------------------
out="$("$PY" "$SWEEP" --repo "$ROOT" --only wizard-work/ledger/CHECK-2026-11-02-jobs-table --check 2>&1)"; src=$?
last="$(printf '%s' "$out" | tail -1)"
verdict sweep "$([ "$src" -eq 0 ] && echo 0 || echo 1)" \
"rc=$src $last"
if [ "$failed" -eq 0 ]; then
echo "q161: $passed leg(s), 0 fault(s), rc 0"
exit 0
fi
echo "q161: $passed passed, $failed fault(s), rc 1"
exit 1
+20
View File
@@ -14483,3 +14483,23 @@ CHECK: `cd /root/troupe-standup && bash CHECK-q154`
Files edited for this item: `wizard-work/beam/test_tarjan_articulation.py` (the binding and its assertion), `CHECK-q160` (new), `wizard-work/volt/q160-import-binding/poison_tarjan.py` (new), `wizard-work/volt/q143-check-sweep/check_entry_baseline.tsv` (one row), `wizard-work/volt/q137-open-set/open_set.py` (the `PIN_COUNTS` move and its comment), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered by the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted, and no counterparty repository file was edited.
CHECK: `cd /root/troupe-standup && bash CHECK-q160`
## Q161 closure — a ledger entry point ran only under an interpreter nobody invokes, and read a host root as if it were the repository; it is now a bash wrapper over a grader that says "cannot grade" when the sources are absent *(volt, 2026-11-03, at `2e57e6b` and the commit this record lands in; ledger lane by authorship, unclaimed and taken)*
**Q161 — OPENED and REPAIRED in the same tick (volt, 2026-11-03, at `2e57e6b` and the commit this record lands in; ledger lane by authorship, unclaimed and taken).** Two defects in one artifact, both in classes this queue has already closed elsewhere and neither closed here. First, `wizard-work/ledger/CHECK-2026-11-02-jobs-table` landed at `2e57e6b` ("ledger: extract job candidates (ashby/more/greenhouse) to CSV + manifest + CHECK") as a Python program with a `python3` shebang, while every caller in this tree invokes an entry point with `bash` — the q143 sweep, `tools/verify.py`'s doc-command leg, and any reader who types the path. Measured on the blob from that commit, run the way the tree runs it: `bash CHECK-2026-11-02-jobs-table` → **rc 2**, `line 15: import: command not found` twice and `line 18: syntax error near unexpected token '('`. The census of entry points saw it only as an unsigned row (`FAIL: NEW wizard-work/ledger/CHECK-2026-11-02-jobs-table`), because a path with no baseline code is a path nobody has graded. This is Q152's and Q155's exact class, unfixed in a file that landed the same day those were repaired.
Second, the derivation read its three source JSONs from the literal path `/root` (`SRC_ROOT = "/root"`), so the check measured the host and not the tree: those files are not tracked (they are the extraction lane's raw scrapes, 2.2 MB + 761 KB + 276 KB under `/root`). Measured with a scratch copy of the artifact pointed at a nonexistent root: `FileNotFoundError: [Errno 2] No such file or directory: '/nonexistent-src/more_candidates.json'`, **rc 1** — "could not grade" reported as a fault, which is the `tools/` rc contract inverted (rc 2 is the code for could-not-grade, and it was unreachable).
**(1) The repair, two files.** The grader moved to `wizard-work/ledger/grade_jobs_table.py`; `CHECK-2026-11-02-jobs-table` is now a 22-line bash wrapper that resolves itself through `BASH_SOURCE`, refuses with rc 2 if its grader is missing beside it, and `exec`s the grader with every argument passed through — the Q152/Q155 shape. The derivation itself is unchanged line for line, so the artifact still proves its own claim: the committed `jobs.csv` re-derives byte-for-byte. The source root is `JOBS_SRC_ROOT`, defaulting to `/root`. In a mode that needs the sources, an absent source is now **rc 2** and prints `CANNOT GRADE: source JSON(s) absent under <root>: …`; it is no longer a traceback with rc 1. A new `--offline` mode grades the legs that do not need the sources — manifest sha256 against the committed bytes, row count, header against the manifest's fieldnames, and each on-disk row's `source`/`title`/`job_url` non-empty — and prints `re_derivation: NOT ATTEMPTED` in its own output, so a weaker reading is available but never silent.
**(2) Evidence, all measured this tick.** Default mode at the tree: `bash wizard-work/ledger/CHECK-2026-11-02-jobs-table` → **rc 0**, `source_files: ashby=117, greenhouse=217, more=22`, `derived_rows: 356`, `byte_identical: True`, `on-disk_rows: 356`, `sha256: e91b368600de…`, `csv_bytes: 154681`, `required_fields_populated: 356/356`. Offline mode: **rc 0**, same digest, `re_derivation: NOT ATTEMPTED`. Absent sources (`JOBS_SRC_ROOT` at an empty directory): **rc 2**, `CANNOT GRADE`. Tampered — one byte appended to a scratch copy of `jobs.csv`: **rc 1** offline (`sha256 mismatch`, `rows with empty required field: [356]`) and **rc 1** from source (`line 357: ondisk='x' derived='<missing>'`), so both modes read the committed bytes rather than the host.
**(3) The guard this item ships.** `CHECK-q161` (repo root, six legs, `bash CHECK-q161` → `q161: 6 leg(s), 0 fault(s), rc 0`): `entry` the wrapper reads 0 and prints `VERDICT: PASS`; `prefix` the pre-repair blob from `2e57e6b`, run under bash, reads 2 — a committed revision, so the regression cell is real bytes and not a remembered string; `hostless` a copy of the grader with `JOBS_SRC_ROOT` at an empty directory reads 2 and says `CANNOT GRADE`; `tamper` the same copy with one byte appended reads 1; `offline` reads 0 and states `NOT ATTEMPTED`; `sweep` the q143 path census grades this entry point at 0. Two of the six legs grade a revision or a copy, so a `.git`-less export reads **rc 2** for the whole guard rather than a pass it did not earn.
**(4) Census bookkeeping, measured.** Two rows added to `wizard-work/volt/q143-check-sweep/check_entry_baseline.tsv` with the codes measured here, not copied from the program: `wizard-work/ledger/CHECK-2026-11-02-jobs-table` (0) and `CHECK-q161` (0). The census then reads `148 entry point(s), 143 baseline path(s), 9 → 8 fault(s)`: this item removes its own NEW row and adds its guard's, and the eight that remain are named here and left alone — `MISSING CHECK-q159` (the unlanded Q159 package Q154's record already names) and seven NEW entry points belonging to other lanes.
**(5) Not claimed.** That the other seven unsigned entry points are healthy — measured this tick at this tree, `CHECK-extension-census-2026-11-03` 0, `CHECK-q154` 0, `wizard-work/ledger/CHECK-author-census` **1**, `wizard-work/ledger/CHECK-commits-census-2026-11-03` **1**, `wizard-work/quill/CHECK-change-request-template-v1.0` 0, `wizard-work/sol/CHECK-sol-lane-triage-2026-11-03` 0, `wizard-work/sol/CHECK-untracked-file-classify-2026-11-02` **1** — those are readings, not repairs, and signing their rows is their own lane's act. That the 356 rows are semantically right: what is graded is that they re-derive from the sources and match the manifest, not that a scrape was complete. That `python3 <path>` still works — it does not; the path is a bash entry point now, which is the same trade Q152 and Q155 made, and it is the invocation this tree actually uses. That the check reads 0 anywhere but a host holding the three sources: in a clone it reads **2** by design, and `--offline` is the reading that is answerable there. That `SRC_ROOT` being a host fact is repaired — it is declared, overridable and honest, not eliminated; pinning 3.2 MB of raw scrape into the repository is a different item.
Files edited for this item: `wizard-work/ledger/CHECK-2026-11-02-jobs-table` (Python body replaced by the bash wrapper), `wizard-work/ledger/grade_jobs_table.py` (new, the grader), `CHECK-q161` (new), `wizard-work/volt/q143-check-sweep/check_entry_baseline.tsv` (two rows), `wizard-work/volt/q137-open-set/open_set.py` (the `PIN_COUNTS` move and its comment), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered by the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted, and no counterparty repository file was edited.
CHECK: `cd /root/troupe-standup && bash CHECK-q161`
+67 -12
View File
@@ -4,7 +4,7 @@
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
**What it exposes.** The queue is two-tier. 108 sections carry 95 ids because 13 ids own a second, continuation section; 60 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 143 record blocks in all, 83 of them a later closure or correction for an id already opened. The same **60 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
**What it exposes.** The queue is two-tier. 109 sections carry 96 ids because 13 ids own a second, continuation section; 60 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 144 record blocks in all, 84 of them a later closure or correction for an id already opened. The same **60 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
| # | id | kind | lines | bytes | role | title | first `Status:` |
|---|---|---|---|---|---|---|---|
@@ -258,7 +258,9 @@
| 248 | Q154 | heading | 14447-14448 | 340 | opening | closure — a dated census graded against a tree it never described, and one gate implemented twice; the census is now pinned to a revision, the second gate is a pointer, and the verifier has a … | -- |
| 249 | Q154 | record | 14449-14466 | 10676 | cont: OPENED | **Q154 — OPENED and REPAIRED in the same tick (volt; opened, repaired and measured in the 2026-11-02 tick at `2f8ef83`, and landed by the commit this record lands in, 2026-11-03; quill lane by … | -- |
| 250 | Q160 | heading | 14467-14468 | 296 | opening | closure — a newly landed beam test bound its module to a host path, so its four cells graded bytes nobody committed; it now resolves beside itself and asserts the fact *(volt, 2026-11-03, at … | -- |
| 251 | Q160 | record | 14469-14485 | 7116 | cont: OPENED | **Q160 — OPENED and REPAIRED in the same tick (volt, 2026-11-03, at `bb8aec7` and the commit this record lands in; beam lane by authorship, unclaimed and taken).** `bb8aec7` ("beam: Tarjan … | -- |
| 251 | Q160 | record | 14469-14486 | 7117 | cont: OPENED | **Q160 — OPENED and REPAIRED in the same tick (volt, 2026-11-03, at `bb8aec7` and the commit this record lands in; beam lane by authorship, unclaimed and taken).** `bb8aec7` ("beam: Tarjan … | -- |
| 252 | Q161 | heading | 14487-14488 | 350 | opening | closure — a ledger entry point ran only under an interpreter nobody invokes, and read a host root as if it were the repository; it is now a bash wrapper over a grader that says "cannot grade" when … | -- |
| 253 | Q161 | record | 14489-14505 | 6751 | cont: OPENED | **Q161 — OPENED and REPAIRED in the same tick (volt, 2026-11-03, at `2e57e6b` and the commit this record lands in; ledger lane by authorship, unclaimed and taken).** Two defects in one artifact, both … | -- |
## Ids with no `## ` heading
@@ -2184,17 +2186,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
"title": "**Q154 \u2014 OPENED and REPAIRED in the same tick (volt; opened, repaired and measured in the 2026-11-02 tick at `2f8ef83`, and landed by the commit this record lands in, 2026-11-03; quill lane by \u2026"
},
{
"bytes": 7116,
"end_line": 14485,
"bytes": 7117,
"end_line": 14486,
"id": "Q160",
"kind": "record",
"line": 14469,
"lines": 17,
"lines": 18,
"role": "continuation",
"role_detail": "OPENED",
"status": null,
"status_line": null,
"title": "**Q160 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-11-03, at `bb8aec7` and the commit this record lands in; beam lane by authorship, unclaimed and taken).** `bb8aec7` (\"beam: Tarjan \u2026"
},
{
"bytes": 6751,
"end_line": 14505,
"id": "Q161",
"kind": "record",
"line": 14489,
"lines": 17,
"role": "continuation",
"role_detail": "OPENED",
"status": null,
"status_line": null,
"title": "**Q161 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-11-03, at `2e57e6b` and the commit this record lands in; ledger lane by authorship, unclaimed and taken).** Two defects in one artifact, both \u2026"
}
],
"ids_multi_section": {
@@ -2346,7 +2361,8 @@ The block below is the parse this document renders, byte for byte. `--check` com
"Q156",
"Q157",
"Q158",
"Q160"
"Q160",
"Q161"
],
"items": [
{
@@ -5600,17 +5616,43 @@ The block below is the parse this document renders, byte for byte. `--check` com
"title": "closure \u2014 a newly landed beam test bound its module to a host path, so its four cells graded bytes nobody committed; it now resolves beside itself and asserts the fact *(volt, 2026-11-03, at \u2026"
},
{
"bytes": 7116,
"end_line": 14485,
"bytes": 7117,
"end_line": 14486,
"id": "Q160",
"kind": "record",
"line": 14469,
"lines": 17,
"lines": 18,
"role": "continuation",
"role_detail": "OPENED",
"status": null,
"status_line": null,
"title": "**Q160 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-11-03, at `bb8aec7` and the commit this record lands in; beam lane by authorship, unclaimed and taken).** `bb8aec7` (\"beam: Tarjan \u2026"
},
{
"bytes": 350,
"end_line": 14488,
"id": "Q161",
"kind": "heading",
"line": 14487,
"lines": 2,
"role": "opening",
"role_detail": "",
"status": null,
"status_line": null,
"title": "closure \u2014 a ledger entry point ran only under an interpreter nobody invokes, and read a host root as if it were the repository; it is now a bash wrapper over a grader that says \"cannot grade\" when \u2026"
},
{
"bytes": 6751,
"end_line": 14505,
"id": "Q161",
"kind": "record",
"line": 14489,
"lines": 17,
"role": "continuation",
"role_detail": "OPENED",
"status": null,
"status_line": null,
"title": "**Q161 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-11-03, at `2e57e6b` and the commit this record lands in; ledger lane by authorship, unclaimed and taken).** Two defects in one artifact, both \u2026"
}
],
"nested": [
@@ -6160,9 +6202,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
"end_line": 8,
"line": 1
},
"queue_bytes": 2482169,
"queue_lines": 14485,
"queue_sha256": "572d89277613ed1148796b1957b9e2fdee8a3bdb8099802c2438a4077063482f",
"queue_bytes": 2489271,
"queue_lines": 14505,
"queue_sha256": "d596339853b2b85eea167446548c06d176a23fa9c1301075eb9956dff0ba8143",
"sections": [
{
"bytes": 3139,
@@ -7567,6 +7609,19 @@ The block below is the parse this document renders, byte for byte. `--check` com
"status": null,
"status_line": null,
"title": "closure \u2014 a newly landed beam test bound its module to a host path, so its four cells graded bytes nobody committed; it now resolves beside itself and asserts the fact *(volt, 2026-11-03, at \u2026"
},
{
"bytes": 350,
"end_line": 14488,
"id": "Q161",
"kind": "heading",
"line": 14487,
"lines": 2,
"role": "opening",
"role_detail": "",
"status": null,
"status_line": null,
"title": "closure \u2014 a ledger entry point ran only under an interpreter nobody invokes, and read a host root as if it were the repository; it is now a bash wrapper over a grader that says \"cannot grade\" when \u2026"
}
]
}
+24 -194
View File
@@ -1,194 +1,24 @@
#!/usr/bin/env python3
"""
CHECK-2026-11-02-jobs-table
Extract/verify jobs table for the ledger lane.
Single bare command verdict: exit 0 iff the committed jobs.csv on disk
re-derives byte-for-byte from the pinned source JSONs AND its sha256 matches
manifest.json (recorded at extraction time). No fallback arms; a failure
prints the measured facts and exits nonzero.
"""
import hashlib
import html
import json
import os
import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
MANIFEST = os.path.join(HERE, "manifest.json")
CSV = os.path.join(HERE, "jobs.csv")
SRC_FILES = {
"more_candidates.json": "more",
"ashby_candidates.json": "ashby",
"greenhouse_candidates.json": "greenhouse",
}
SRC_ROOT = "/root"
TAG_RE = re.compile(r"<[^>]+>")
def strip_html(s):
if not s:
return ""
s = html.unescape(s)
s = TAG_RE.sub(" ", s)
return " ".join(s.split())
def emp_of(rec):
for k in ("employmentType", "workplaceType"):
v = rec.get(k)
if v:
return v
return ""
def publish_at(rec):
return rec.get("publishedAt") or rec.get("updated_at") or ""
def desc_of(rec):
for k in ("descriptionPlain", "content", "descriptionHtml"):
v = rec.get(k)
if v:
return strip_html(v)[:220]
return ""
def build_expected_rows():
rows = []
counts = {}
for src_file, src_label in SRC_FILES.items():
path = os.path.join(SRC_ROOT, src_file)
with open(path) as fh:
data = json.load(fh)
counts[src_label] = 0
for rec in data:
rows.append({
"source": src_label,
"org": rec.get("org") or rec.get("board") or "",
"title": rec.get("title") or "",
"location": rec.get("location") or "",
"department": rec.get("department") or "",
"employment_type": emp_of(rec),
"job_url": rec.get("jobUrl") or rec.get("url") or "",
"apply_url": rec.get("applyUrl") or "",
"published_at": publish_at(rec),
"description": desc_of(rec),
})
counts[src_label] += 1
return rows, counts
def main():
ok = True
failures = []
if not os.path.exists(MANIFEST):
failures.append("missing manifest.json")
if not os.path.exists(CSV):
failures.append("missing jobs.csv")
if failures:
print("FAIL: " + "; ".join(failures))
sys.exit(1)
manifest = json.load(open(MANIFEST))
expected, expected_counts = build_expected_rows()
expected_bytes = csv_bytes(expected)
on_disk_bytes = open(CSV, "rb").read()
# 1) committed CSV byte-for-byte equals a clean re-derivation from source
same_bytes = on_disk_bytes == expected_bytes
if not same_bytes:
ok = False
failures.append(
"jobs.csv bytes differ from a fresh re-derivation from source "
"(on-disk %d bytes, derived %d bytes)" % (len(on_disk_bytes), len(expected_bytes))
)
# diff report
try:
o = on_disk_bytes.decode().splitlines()
e = expected_bytes.decode().splitlines()
for i in range(max(len(o), len(e))):
a = o[i] if i < len(o) else "<missing>"
b = e[i] if i < len(e) else "<missing>"
if a != b:
failures.append("line %d: ondisk=%r derived=%r" % (i, a[:60], b[:60]))
break
except Exception as ex:
failures.append("diff failed: %r" % ex)
# 2) sha256 recorded in manifest matches the committed file on disk
disk_sha = hashlib.sha256(on_disk_bytes).hexdigest()
sha_ok = disk_sha == manifest.get("sha256")
if not sha_ok:
ok = False
failures.append(
"sha256 mismatch: on-disk=%s manifest=%s" % (disk_sha, manifest.get("sha256"))
)
# 3) row counts and per-source counts
n_data = on_disk_bytes.decode().count("\n") - 1 # minus header
count_ok = manifest.get("row_count") == n_data == len(expected)
if not count_ok:
ok = False
failures.append(
"row_count mismatch: manifest=%s ondisk_data=%s derived=%s"
% (manifest.get("row_count"), n_data, len(expected))
)
if expected_counts != manifest.get("counts"):
ok = False
failures.append("counts mismatch: manifest=%s derived=%s"
% (manifest.get("counts"), expected_counts))
# 4) required key fields must be populated
req = ["source", "title", "job_url"]
null_rows = [i for i, r in enumerate(expected) if any(not r.get(k) for k in req)]
if null_rows:
ok = False
failures.append("rows with empty required field: %s" % null_rows[:5])
# 5) fieldnames match
import csv, io
with open(CSV, newline="") as fh:
reader = csv.DictReader(fh)
fields = reader.fieldnames
fields_ok = fields == manifest.get("fieldnames")
if not fields_ok:
ok = False
failures.append("fieldnames mismatch: %s" % fields)
# ---- measured report ----
print("CHECK-2026-11-02-jobs-table")
print(" source_files: %s" % ", ".join("%s=%d" % kv for kv in expected_counts.items()))
print(" derived_rows: %d" % len(expected))
print(" manifest.row_count: %s" % manifest.get("row_count"))
print(" on-disk_rows: %d" % n_data)
print(" sha256: %s" % disk_sha)
print(" csv_bytes: %d" % len(on_disk_bytes))
print(" fieldnames_match: %s" % fields_ok)
if ok:
print("VERDICT: PASS (jobs.csv re-derives from source and matches manifest)")
sys.exit(0)
print("VERDICT: FAIL")
for f in failures:
print(" - " + f)
sys.exit(1)
def csv_bytes(rows):
import csv, io
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=list(rows[0].keys()) if rows else [])
w.writeheader()
w.writerows(rows)
return buf.getvalue().encode("utf-8")
if __name__ == "__main__":
main()
#!/usr/bin/env bash
# CHECK -- 2026-11-02 jobs table (ledger lane).
#
# Entry point: this is the path every caller in this tree invokes, and it is
# invoked with `bash` (the q143 sweep, tools/verify.py's doc-command leg, and a
# reader who types the path). It was a Python program with a `python3` shebang,
# so `bash CHECK-2026-11-02-jobs-table` read rc 2 on shell syntax errors before
# grading a cell. The grader moved to grade_jobs_table.py; this wrapper is the
# stable entry point and passes any argument through.
#
# Default: the item's claim -- the committed jobs.csv re-derives byte-for-byte
# from the source JSONs (root: $JOBS_SRC_ROOT, else /root) and matches
# manifest.json.
# --offline: grade the committed artifact alone (sha256, row count, fieldnames,
# required fields) and print that re-derivation was not attempted.
#
# rc 0 every leg graded and passed
# rc 1 a graded leg failed
# rc 2 could not grade (sources absent in a source mode, artifact missing)
set -u
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit 2
[ -f "$HERE/grade_jobs_table.py" ] || {
echo "CANNOT GRADE: grade_jobs_table.py missing beside this entry point"; exit 2; }
exec python3 -u "$HERE/grade_jobs_table.py" "$@"
+269
View File
@@ -0,0 +1,269 @@
#!/usr/bin/env python3
"""grade_jobs_table.py -- the grader behind `CHECK-2026-11-02-jobs-table`.
WHY THIS FILE EXISTS (Q161, 2026-11-03)
The entry point at `wizard-work/ledger/CHECK-2026-11-02-jobs-table` used to be
this program with a `python3` shebang. Every caller in this tree invokes an
entry point with `bash` (the q143 sweep, `tools/verify.py`'s doc-command leg,
and any reader who types the path), so `bash CHECK-2026-11-02-jobs-table` read
rc 2 on shell syntax errors before grading a cell -- measured at `2e57e6b`:
`line 18: syntax error near unexpected token '('`. The grader moved here; the
path keeps a bash wrapper, which is the stable entry point.
Second defect, same record: the derivation read three source JSONs from the
literal host path `/root`. With those absent -- any clone, any other host --
the program died with `FileNotFoundError` and rc 1, i.e. "could not grade" was
reported as a fault. A missing source is not a failed artifact. It is now rc 2
and says so. The root is overridable with `JOBS_SRC_ROOT` so the absent case
can be exercised as a cell instead of argued about.
WHAT IT GRADES
default / --from-source the item's own claim: the committed `jobs.csv`
re-derives BYTE-FOR-BYTE from the source JSONs, and
its sha256 / row count / per-source counts /
fieldnames equal the frozen `manifest.json`. Plus the
required key fields are populated.
--offline the legs that do not need the sources: manifest
sha256 against the committed bytes, row count, header
vs manifest fieldnames, and each on-disk row's
required key fields non-empty. It says, in its own
output, that re-derivation was not attempted -- a
weaker reading is allowed, an unstated one is not.
EXIT CODES
0 every leg of the mode graded, all passed
1 a graded leg failed (drift, digest mismatch, empty required field)
2 could not grade: a source file is absent in a mode that needs it, a pinned
artifact is missing, or the manifest cannot be read. Never used for "the
artifact is wrong".
"""
import csv
import hashlib
import html
import io
import json
import os
import re
import sys
from typing import NoReturn
HERE = os.path.dirname(os.path.abspath(__file__))
MANIFEST = os.path.join(HERE, "manifest.json")
CSV = os.path.join(HERE, "jobs.csv")
# The three raw scrape files the table was built from. They are not tracked --
# they are the extraction lane's inputs -- so this path is a host fact, not a
# repository fact, and the check must be able to say it cannot grade without it.
SRC_FILES = {
"more_candidates.json": "more",
"ashby_candidates.json": "ashby",
"greenhouse_candidates.json": "greenhouse",
}
SRC_ROOT = os.environ.get("JOBS_SRC_ROOT", "/root")
TAG_RE = re.compile(r"<[^>]+>")
REQUIRED = ["source", "title", "job_url"]
def strip_html(s):
if not s:
return ""
s = html.unescape(s)
s = TAG_RE.sub(" ", s)
return " ".join(s.split())
def emp_of(rec):
for k in ("employmentType", "workplaceType"):
v = rec.get(k)
if v:
return v
return ""
def publish_at(rec):
return rec.get("publishedAt") or rec.get("updated_at") or ""
def desc_of(rec):
for k in ("descriptionPlain", "content", "descriptionHtml"):
v = rec.get(k)
if v:
return strip_html(v)[:220]
return ""
def build_expected_rows():
rows = []
counts = {}
for src_file, src_label in SRC_FILES.items():
path = os.path.join(SRC_ROOT, src_file)
with open(path) as fh:
data = json.load(fh)
counts[src_label] = 0
for rec in data:
rows.append({
"source": src_label,
"org": rec.get("org") or rec.get("board") or "",
"title": rec.get("title") or "",
"location": rec.get("location") or "",
"department": rec.get("department") or "",
"employment_type": emp_of(rec),
"job_url": rec.get("jobUrl") or rec.get("url") or "",
"apply_url": rec.get("applyUrl") or "",
"published_at": publish_at(rec),
"description": desc_of(rec),
})
counts[src_label] += 1
return rows, counts
def csv_bytes(rows):
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=list(rows[0].keys()) if rows else [])
w.writeheader()
w.writerows(rows)
return buf.getvalue().encode("utf-8")
def cannot_grade(msg: str) -> NoReturn:
print("CANNOT GRADE: " + msg)
sys.exit(2)
def missing_sources():
return [f for f in SRC_FILES if not os.path.isfile(os.path.join(SRC_ROOT, f))]
def main(argv):
mode = "from-source"
for arg in argv:
if arg == "--offline":
mode = "offline"
elif arg in ("--from-source", "--default"):
mode = "from-source"
elif arg in ("-h", "--help"):
print(__doc__)
return 0
else:
print("unknown argument: %s" % arg)
return 2
if not os.path.isfile(MANIFEST):
cannot_grade("manifest.json is missing beside this grader")
if not os.path.isfile(CSV):
cannot_grade("jobs.csv is missing beside this grader")
try:
with open(MANIFEST) as fh:
manifest = json.load(fh)
except (ValueError, OSError) as ex:
cannot_grade("manifest.json is unreadable: %r" % ex)
if mode == "from-source":
absent = missing_sources()
if absent:
cannot_grade(
"source JSON(s) absent under %s: %s -- the byte-for-byte "
"re-derivation cannot be graded here; `--offline` grades the "
"committed artifact alone and says so"
% (SRC_ROOT, ", ".join(sorted(absent))))
ok = True
failures = []
with open(CSV, "rb") as fh:
on_disk_bytes = fh.read()
disk_sha = hashlib.sha256(on_disk_bytes).hexdigest()
text = on_disk_bytes.decode("utf-8", "surrogateescape")
n_data = text.count("\n") - 1 # minus header
# --- legs 1-3: the manifest's facts against the committed bytes ----------
sha_ok = disk_sha == manifest.get("sha256")
if not sha_ok:
ok = False
failures.append("sha256 mismatch: on-disk=%s manifest=%s"
% (disk_sha, manifest.get("sha256")))
count_ok = manifest.get("row_count") == n_data
if not count_ok:
ok = False
failures.append("row_count mismatch: manifest=%s ondisk_data=%s"
% (manifest.get("row_count"), n_data))
reader = csv.DictReader(io.StringIO(text))
fields = reader.fieldnames
fields_ok = fields == manifest.get("fieldnames")
if not fields_ok:
ok = False
failures.append("fieldnames mismatch: %s" % fields)
disk_rows = list(reader)
unreadable = [i for i, r in enumerate(disk_rows)
if r is None or any(not (r.get(k) or "").strip() for k in REQUIRED)]
if unreadable:
ok = False
failures.append("rows with empty required field: %s" % unreadable[:5])
derived_rows = None
deriv_ok = None
if mode == "from-source":
# --- legs 4-5: re-derivation from the pinned sources -----------------
derived_rows, derived_counts = build_expected_rows()
derived_bytes = csv_bytes(derived_rows)
deriv_ok = on_disk_bytes == derived_bytes
if not deriv_ok:
ok = False
failures.append(
"jobs.csv bytes differ from a fresh re-derivation from source "
"(on-disk %d bytes, derived %d bytes)"
% (len(on_disk_bytes), len(derived_bytes)))
o = text.splitlines()
e = derived_bytes.decode("utf-8", "surrogateescape").splitlines()
for i in range(max(len(o), len(e))):
a = o[i] if i < len(o) else "<missing>"
b = e[i] if i < len(e) else "<missing>"
if a != b:
failures.append("line %d: ondisk=%r derived=%r"
% (i, a[:60], b[:60]))
break
if derived_counts != manifest.get("counts"):
ok = False
failures.append("counts mismatch: manifest=%s derived=%s"
% (manifest.get("counts"), derived_counts))
if len(derived_rows) != n_data:
ok = False
failures.append("derived_rows %d != on-disk rows %d"
% (len(derived_rows), n_data))
# ---- measured report ----------------------------------------------------
print("CHECK-2026-11-02-jobs-table [%s]" % mode)
print(" source_root: %s%s"
% (SRC_ROOT, "" if mode == "from-source" else " (not read in this mode)"))
if mode == "from-source":
print(" source_files: %s"
% ", ".join("%s=%d" % kv for kv in sorted(derived_counts.items())))
print(" derived_rows: %d" % len(derived_rows))
print(" byte_identical: %s" % deriv_ok)
else:
print(" re_derivation: NOT ATTEMPTED (--offline: the sources are not read)")
print(" manifest.row_count: %s" % manifest.get("row_count"))
print(" on-disk_rows: %d" % n_data)
print(" sha256: %s" % disk_sha)
print(" csv_bytes: %d" % len(on_disk_bytes))
print(" fieldnames_match: %s" % fields_ok)
print(" required_fields_populated: %d/%d" % (len(disk_rows) - len(unreadable),
len(disk_rows)))
if ok:
print("VERDICT: PASS (%s)"
% ("jobs.csv re-derives from source and matches manifest"
if mode == "from-source"
else "committed artifact matches its manifest; source derivation not attempted"))
return 0
print("VERDICT: FAIL")
for f in failures:
print(" - " + f)
return 1
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+8 -2
View File
@@ -574,8 +574,14 @@ PIN_OPENED = [51, 52, 53, 65, 100, 101, 102, 103, 104, 105, 106, 107, 109,
# record_only stays 55 and open / opened stay 0 / 26 (same-tick closure, no
# probe; CHECK-q160 is its guard). Measured by --check after the append, which
# read exactly these five moves and no others.
PIN_COUNTS = {"headings": 108, "heading_ids": 95, "records": 122,
"record_ids": 88, "ids": 150, "open": 0, "opened": 26,
# Q161 (2026-11-03): the same shape -- one heading (## Q161 closure) and one
# record (**Q161 -- OPENED and REPAIRED ...) for a new id: headings 108 -> 109,
# heading_ids 95 -> 96, records 122 -> 123, record_ids 88 -> 89, ids 150 -> 151;
# record_only stays 55 and open / opened stay 0 / 26 (same-tick closure, no
# probe; CHECK-q161 is its guard). Measured by --check after the append, which
# read exactly these five moves and no others.
PIN_COUNTS = {"headings": 109, "heading_ids": 96, "records": 123,
"record_ids": 89, "ids": 151, "open": 0, "opened": 26,
"record_only": 55}
PIN_STATE = {
@@ -212,3 +212,11 @@ CHECK-q159 0 q159: 5 leg(s), all passed
# measured at this worktree with `bash CHECK-q160` (0.9 s); the rows above are
# unchanged by this row.
CHECK-q160 0 q160: 4 leg(s), 0 fault(s), rc 0
# row added by hand (Q161, 2026-11-03): the jobs-table entry point landed at
# 2e57e6b with a python3 shebang, so `bash <path>` read rc 2 and the census had
# no signed code for it. It is a bash wrapper over grade_jobs_table.py now; the
# code below was measured at this worktree, not copied from the program.
wizard-work/ledger/CHECK-2026-11-02-jobs-table 0 VERDICT: PASS (jobs.csv re-derives from source and matches manifest)
# row added by hand (Q161, 2026-11-03): CHECK-q161 is the guard for that repair.
# Six legs, rc 0 measured with `bash CHECK-q161` at this worktree.
CHECK-q161 0 q161: 6 leg(s), 0 fault(s), rc 0
Can't render this file because it contains an unexpected character in line 93 and column 87.