sweep: ECHOES_USAGE -- a refusal that echoes the invocation is not an input refusal (Q97)
This commit is contained in:
1 parent
9a5367be0b
commit
58895d9513
6 files changed
+380
-16
No files matched your search
@@ -13492,3 +13492,19 @@ CHECK: bash wizard-work/sol/verify.sh
|
||||
**Not claimed:** that record-form ids are table rows — they are enumerated and counted but get no row, because a bold-lead record has no heading title and no block bounds, and the row unit is unchanged; that the two scans are an oracle — extractor and CHECK implement the same stated rule in two places, so their agreement is a transcription check, not an independent derivation; that `_debug.py` (which still pins `/root/troupe-standup`) or the pre-v3 `manifest.json` beside the artifact were repaired or removed — both were read, neither touched, neither graded by this CHECK; that `extracted_at_utc` is now provenance — it is a measured stamp graded for shape only; that `ab7767d`'s rewrite was intended for this directory — its message names jsonplaceholder users and says nothing about the census, and intent is Testimony only; that Q122's v2 directory was re-graded — different artifact, not read this tick beyond the receipt it records; that the row semantics (title trimming, sentence cut) were re-adjudicated — they were not, only the reader, the record-form leg and the tail lines moved; that `wizard-work/ledger/2026-10-28-workqueue-census/README.md` existed before this tick — the extractor writes it and it was absent from the tree, so it is new here rather than edited; that the TSV changed — it was regenerated and is **byte-identical** (md5 `29eb6a255b225198942ef6e8e52b0207`), so it is not in the diff, and the only TSV change is the extractor's ability to state coverage. Files edited for this item: `wizard-work/ledger/2026-10-28-workqueue-census/check_workqueue_census.py`, `wizard-work/ledger/2026-10-28-workqueue-census/CHECK`, `wizard-work/ledger/2026-10-28-workqueue-census/extract_workqueue_census.py`, `wizard-work/ledger/2026-10-28-workqueue-census/workqueue-census.md`, `wizard-work/ledger/2026-10-28-workqueue-census/extract_manifest.json`, `wizard-work/ledger/2026-10-28-workqueue-census/README.md` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted. Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q112.md`.
|
||||
|
||||
CHECK: python3 wizard-work/ledger/2026-10-28-workqueue-census/CHECK
|
||||
|
||||
**Q97 — DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now `ECHOES_USAGE`, fatal on both input axes, so a renamed flag can no longer turn the malformed or read axis green — and the item's own mutant is corrected, because `--repo` → `--repo-dir` is invisible to `argparse`.**
|
||||
|
||||
**(0) The mutant the item files does not reach the parser, measured.** The record's reproduction renames `ap.add_argument('--repo', …)` to `--repo-dir` with `dest='repo'`. `argparse` abbreviates an unambiguous long option, so that rename parses: with a two-line stub carrying only `--repo-dir`, `prog --repo /tmp/nope` reads `parsed repo= /tmp/nope`, **rc 0**, while `prog --tree /tmp/nope` reads `usage: … error: unrecognized arguments: --tree /tmp/nope`, **rc 2**. So the filed mutation leaves the tool reading its input normally, and *any* correct implementation — the old one included — reads it clean for a reason the item does not name. The shape that reaches the parser is a rename that is not a prefix; `--tree` is used in the cells and in the gate below.
|
||||
|
||||
**(1) The defect, reproduced against HEAD before any edit.** `wizard-work/volt/q97_check.py`-independent falsification harness (`/root/.hermes/profiles/volt/cache/scratch/q97b/falsify_q97.py`, 2.3 KB) loads `git show HEAD:tools/selftest_sweep.py` and the repaired worktree file side by side and hands **both** the same mutant copy. Pre-repair: `verdicts=OK`, `axis rc=0 (read 0 fault, malformed 0 fault)` — the tool never read anything and both axes certified it. Repaired, same mutant, same call: `verdicts=ECHOES_USAGE`, `axis rc=1 (read 1 fault, malformed 1 fault)`, named `ECHOES_USAGE citation_hash_class.py/missing` and `…/not-a-repo`. One measurement, two module versions, defect and fix on the same input.
|
||||
|
||||
**(2) The repair** (`tools/selftest_sweep.py`, `+136/−5`): `echoes_invocation(text)` returns True when the first non-empty line leads with `usage:` or the message carries one of `ARGV_ECHO_MARKERS` (`unrecognized arguments`, `error: argument`, `the following arguments are required`, `invalid choice:`, `ambiguous option:`, `no such option`); `ECHOES_USAGE` joins `FATAL_READ`; `judge_read` and `judge_malformed` both return it with the first line of the message in `detail`, so the summary names the tool and the probe. The test is deliberately negative — a refusal is red for what it *is*, not for what a table remembers it saying (Q93's defect) — and deliberately not "must name the probe's path", because two tools in this directory refuse correctly with a boundary line that names no path. A new docstring section states the rule and the abbreviation measurement. Five cells (c45–c49) join the sweep's own selftest: judge on the malformed axis, judge on the read axis, the negative control (a refusal naming its input stays `OK` on both), an end-to-end cell whose stub carries a **real** `argparse` with the flag genuinely renamed, and a summary cell that pins `malformed-input: 1 probes, 0 ok, 1 fault (ECHOES_USAGE t.py/not-a-repo)` with the run red. `tools/selftest_baseline.tsv` re-pinned `selftest_sweep.py 44 → 49 cells`.
|
||||
|
||||
**(3) Readings on this tree.** `python3 tools/selftest_sweep.py --selftest` → `selftest: 49 cells, 49 passed, 0 failed`, rc **0**. Full sweep: `read-refusal: 28 probes, 28 ok, 0 fault`, `malformed-input: 27 probes, 27 ok, 0 fault`, `pin: 23 pinned, 23 agree, 0 drift`, `verdict: rc 0 -- clean: tools 0 fatal, read-refusal 0 fault, malformed-input 0 fault, pin 0 drift`, rc **0** — so the new rule reds nothing that refuses correctly across all **55** live probes. Gate: `python3 wizard-work/volt/q97_check.py` → four legs PASS, `Q97 GATE: rc 0`, rc **0**, from the repo root and (as `bash wizard-work/volt/CHECK-q97`) from `cwd=/`. Negative control inside the gate: the unmutated `citation_hash_class.py` reads 2 probes, **0** `ECHOES_USAGE`, 0 fault.
|
||||
|
||||
**(4) Gate.** `python3 tools/verify.py` (all eight legs, sweep included) is run over the landing tree — queue append and index render in place — before this record is committed; `python3 tools/check_lint.py --check` on this record's CHECK line reads `SOUND`.
|
||||
|
||||
**Not claimed:** that any shipped tool is currently broken this way — all 23 are green on the clean head, and this is a hole in the instrument, not a live defect; that the prose pins should come back — they were wrong twice in one week and the contract grading fixed both; that CLI drift is now caught anywhere else in the repo — nothing in the gate invokes a tool with the flags its own documentation pins, and the `pin` axis still grades counts, not invocations; that `report_read`'s read/malformed split is a new schema — it is unchanged, and `NO_PROBE` rows still land there by probe name; that the pre-repair module is otherwise sound — only its refusal grading was graded here; that the filed record's other figures were re-measured — the `9878f12`/`f400640` tables were read, not re-run, and the one figure this record corrects is the mutant's reachability. Files edited for this item: `tools/selftest_sweep.py`, `tools/selftest_baseline.tsv`, `wizard-work/volt/q97_check.py` (new), `wizard-work/volt/CHECK-q97` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted and no other lane's artifact was edited. Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q97.md`.
|
||||
|
||||
CHECK: cd /root/troupe-standup && bash wizard-work/volt/CHECK-q97
|
||||
+38
-11
@@ -4,7 +4,7 @@
|
||||
|
||||
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
|
||||
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 95 record blocks in all, 59 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 96 record blocks in all, 60 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
|
||||
|
||||
| # | id | kind | lines | bytes | role | title | first `Status:` |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -199,7 +199,8 @@
|
||||
| 189 | Q106 | record | 13434-13447 | 5422 | cont: DONE | **Q106 — DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest … | -- |
|
||||
| 190 | Q102 | record | 13448-13461 | 4987 | cont: DONE | **Q102 — DONE (volt, 2026-10-29, at `062e278`; beam lane by authorship, unclaimed and taken). `majority()` now has three outcomes on three channels, and a non-reflexive element that fills the list is … | -- |
|
||||
| 191 | Q119 | record | 13462-13479 | 7662 | cont: DONE | **Q119 — DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five … | -- |
|
||||
| 192 | Q112 | record | 13480-13494 | 7899 | cont: DONE | **Q112 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not … | -- |
|
||||
| 192 | Q112 | record | 13480-13495 | 7900 | cont: DONE | **Q112 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not … | -- |
|
||||
| 193 | Q97 | record | 13496-13510 | 5649 | cont: DONE | **Q97 — DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now … | -- |
|
||||
|
||||
## Ids with no `## ` heading
|
||||
|
||||
@@ -1477,17 +1478,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q119 \u2014 DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7899,
|
||||
"end_line": 13494,
|
||||
"bytes": 7900,
|
||||
"end_line": 13495,
|
||||
"id": "Q112",
|
||||
"kind": "record",
|
||||
"line": 13480,
|
||||
"lines": 15,
|
||||
"lines": 16,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q112 \u2014 DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 5649,
|
||||
"end_line": 13510,
|
||||
"id": "Q97",
|
||||
"kind": "record",
|
||||
"line": 13496,
|
||||
"lines": 15,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q97 \u2014 DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now \u2026"
|
||||
}
|
||||
],
|
||||
"ids_multi_section": {
|
||||
@@ -4109,17 +4123,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q119 \u2014 DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7899,
|
||||
"end_line": 13494,
|
||||
"bytes": 7900,
|
||||
"end_line": 13495,
|
||||
"id": "Q112",
|
||||
"kind": "record",
|
||||
"line": 13480,
|
||||
"lines": 15,
|
||||
"lines": 16,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q112 \u2014 DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 5649,
|
||||
"end_line": 13510,
|
||||
"id": "Q97",
|
||||
"kind": "record",
|
||||
"line": 13496,
|
||||
"lines": 15,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q97 \u2014 DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now \u2026"
|
||||
}
|
||||
],
|
||||
"nested": [
|
||||
@@ -4453,9 +4480,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"end_line": 8,
|
||||
"line": 1
|
||||
},
|
||||
"queue_bytes": 2121602,
|
||||
"queue_lines": 13494,
|
||||
"queue_sha256": "3c472c477f6f040f23fb23582afda7836a42d08401090a8987c02c1d461e520d",
|
||||
"queue_bytes": 2127252,
|
||||
"queue_lines": 13510,
|
||||
"queue_sha256": "385f1aeab3df366641f1798080fd7efbd71384e7bedf02a735474a3032606cae",
|
||||
"sections": [
|
||||
{
|
||||
"bytes": 3139,
|
||||
|
||||
@@ -27,7 +27,7 @@ queue_append_audit.py 25 cells
|
||||
queue_claim_check.py 24 fixtures
|
||||
queue_index.py 80 cells
|
||||
reply_check.py 49 cells
|
||||
selftest_sweep.py 44 cells
|
||||
selftest_sweep.py 49 cells
|
||||
served_drift_check.py 25 cells
|
||||
tool_map.py 20 cells
|
||||
verify.py 59 cells
|
||||
|
||||
|
+135
-4
@@ -105,6 +105,33 @@ WHAT A REFUSAL IS GRADED ON, FIXED 2026-10-27 (Q93)
|
||||
table's own comment: the contract keeps "refused, legibly, at the documented
|
||||
code" and stops keeping "refused for the reason the table remembers".
|
||||
|
||||
WHAT A REFUSAL MUST NOT BE, ADDED 2026-10-29 (Q97)
|
||||
The contract above has one blind spot and it is not theoretical: Python's
|
||||
`argparse` writes `usage: ...` and `prog: error: unrecognized arguments: ...`
|
||||
to stderr and exits 2, which satisfies "rc 2, no traceback, a message"
|
||||
exactly. Every probe invokes its tool with flags, so a tool whose flag is
|
||||
renamed -- or removed -- never reads the input at all, and an axis that
|
||||
exists to grade malformed-input handling reads clean. Measured at 9878f12:
|
||||
renaming `--repo` to `--repo-dir` in `citation_hash_class.py` alone left
|
||||
`malformed-input: 26 probes, 26 ok` and `verdict: rc 0 -- clean`.
|
||||
A refusal is now ALSO required to be about the input: a message that leads
|
||||
with `usage:` or carries an argparse failure shape (`unrecognized
|
||||
arguments`, `error: argument`, `the following arguments are required`,
|
||||
`invalid choice:`, `ambiguous option:`, `no such option`) is ECHOES_USAGE --
|
||||
fatal on both input axes, the read axis included, because that axis has had
|
||||
the same hole since it was written (it pinned no prose at all). The check is
|
||||
negative on purpose: requiring the message to name the probe's path would
|
||||
redden tools that refuse correctly with a boundary line, which is the Q93
|
||||
defect coming back the other way.
|
||||
One measurement the filed record does not carry: its mutant renames `--repo`
|
||||
to `--repo-dir` with the same dest, and that rename is INVISIBLE to
|
||||
argparse -- a long option may be abbreviated to any unambiguous prefix, so
|
||||
`--repo` still parses as `--repo-dir` and the tool reads its input
|
||||
normally (measured: same refusal line, rc 2, flag and renamed flag alike).
|
||||
The filed mutant therefore stays green for a reason other than the one the
|
||||
item names. Cell c48 and the verification below use a rename that is not a
|
||||
prefix (`--tree`), which is the shape that actually reaches the parser.
|
||||
|
||||
ONE VERDICT LINE, ADDED 2026-10-27 (Q93)
|
||||
The leg printed `sweep: 22 tools, 22 OK, 0 fatal` and returned **rc 1**: the
|
||||
sweep line is about the TOOLS, the exit code is about the RUN, and nothing
|
||||
@@ -163,7 +190,7 @@ USAGE:
|
||||
python3 selftest_sweep.py --repo /root/troupe-standup
|
||||
python3 selftest_sweep.py --repo /root/troupe-standup --write-baseline tools/selftest_baseline.tsv
|
||||
python3 selftest_sweep.py --tools-dir /tmp/fake --json /tmp/sweep.json
|
||||
python3 selftest_sweep.py --selftest # 44 cells, parser, sweep, pin and both input axes
|
||||
python3 selftest_sweep.py --selftest # 49 cells, parser, sweep, pin and both input axes
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -238,9 +265,23 @@ READ_AXIS_UNIT = REFUSAL_RC
|
||||
# green gate. An ungraded tool may not be reported clean; the tables are the
|
||||
# gate's only cover, so a gap in them has to fail like a fault in a tool.
|
||||
FATAL_READ = ("TRACEBACK", "RC_NOT_2", "SILENT", "NO_TOOL", "RC_WRONG",
|
||||
"UNEXPLAINED", "NO_PROBE")
|
||||
"UNEXPLAINED", "NO_PROBE", "ECHOES_USAGE")
|
||||
NO_PROBE = "NO_PROBE" # fatal: a tool with no probe defined
|
||||
|
||||
# Q97 (2026-10-29): the shapes argparse prints when it cannot parse the
|
||||
# INVOCATION. A refusal carrying one of these is the tool telling the operator
|
||||
# the probe used a flag it does not have -- the input was never read, so the
|
||||
# probe graded nothing. Checked on both input axes; the usage-line lead is
|
||||
# checked separately by `echoes_invocation` because argparse always prints one.
|
||||
ARGV_ECHO_MARKERS = (
|
||||
"unrecognized arguments",
|
||||
"error: argument",
|
||||
"the following arguments are required",
|
||||
"invalid choice:",
|
||||
"ambiguous option:",
|
||||
"no such option",
|
||||
)
|
||||
|
||||
# {nope} = a path that does not exist; {bin} = a file that is not UTF-8;
|
||||
# {bindir} = a directory holding one non-UTF-8 *.md (added 2026-10-18 with
|
||||
# checkin_check.py, whose input is a directory of UTF-8 files -- a {bin} file
|
||||
@@ -737,6 +778,26 @@ def _first_line(text: str) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def echoes_invocation(text):
|
||||
"""True when a rc-2 message is argparse failing to PARSE THE INVOCATION
|
||||
rather than the tool refusing the input (Q97, 2026-10-29).
|
||||
|
||||
Two shapes, both measured from real argparse output rather than invented:
|
||||
the message's first non-empty line leads with `usage:`, or it carries one
|
||||
of ARGV_ECHO_MARKERS. The test is negative -- a refusal is red for what it
|
||||
IS, not for what a table remembers it saying (that was Q93's defect) --
|
||||
and it is deliberately not "must name the probe's path", because two tools
|
||||
in this directory refuse correctly with a boundary line that names no
|
||||
path."""
|
||||
low = (text or "").lower()
|
||||
if any(m in low for m in ARGV_ECHO_MARKERS):
|
||||
return True
|
||||
for line in (text or "").splitlines():
|
||||
if line.strip():
|
||||
return line.strip().lower().startswith("usage:")
|
||||
return False
|
||||
|
||||
|
||||
def judge_read(rc, text):
|
||||
"""One probe's verdict. Three classes, in this order on purpose: a
|
||||
traceback is a defect whatever its exit code, and a silent rc 2 is a
|
||||
@@ -748,6 +809,10 @@ def judge_read(rc, text):
|
||||
"detail": _first_line(text) or f"rc {rc}, no output"}
|
||||
if not (text or "").strip():
|
||||
return {"verdict": "SILENT", "rc": rc, "detail": "rc 2 with no message"}
|
||||
if echoes_invocation(text):
|
||||
return {"verdict": "ECHOES_USAGE", "rc": rc,
|
||||
"detail": "argparse failure, not an input refusal: "
|
||||
+ _first_line(text)}
|
||||
return {"verdict": "OK", "rc": rc, "detail": _first_line(text)}
|
||||
|
||||
|
||||
@@ -826,7 +891,9 @@ def judge_malformed(rc, out, err, expect_rc, need):
|
||||
|
||||
A REFUSAL (expect_rc == 2) is graded on the contract, not on prose (Q93):
|
||||
the documented code, no traceback, one non-empty message on either stream.
|
||||
Which stream is not part of the contract -- measured 2026-10-27,
|
||||
Since Q97 (2026-10-29) the message must also NOT be argparse failing to
|
||||
parse the invocation -- that is ECHOES_USAGE, fatal, because the input was
|
||||
never read. Which stream is not part of the contract -- measured 2026-10-27,
|
||||
`tool_map.py` writes its refusal to stdout and `citation_hash_class.py` to
|
||||
stderr, both at rc 2, both correct. `need` is only consulted where the tool
|
||||
GRADED the malformed input (rc 0/1): there the word is its verdict."""
|
||||
@@ -840,6 +907,11 @@ def judge_malformed(rc, out, err, expect_rc, need):
|
||||
if not text.strip():
|
||||
return {"verdict": "SILENT", "rc": rc, "detail": "rc %s with no message" % rc}
|
||||
if expect_rc == REFUSAL_RC:
|
||||
if echoes_invocation(text):
|
||||
# Q97: rc 2 from the argument parser is not a refusal of the input.
|
||||
return {"verdict": "ECHOES_USAGE", "rc": rc, "contract": "refusal",
|
||||
"detail": "argparse failure, not an input refusal: "
|
||||
+ _first_line(text)}
|
||||
# a refusal: the code and a readable line are the whole contract
|
||||
return {"verdict": "OK", "rc": rc, "contract": "refusal",
|
||||
"detail": "refusal rc 2: " + _first_line(text)}
|
||||
@@ -953,8 +1025,10 @@ def run_verdict(tool_fatal, read_fatal, malformed_fatal, pin_rc, pin_fatal):
|
||||
|
||||
def selftest():
|
||||
bad = []
|
||||
ran = []
|
||||
|
||||
def cell(name, ok, detail=""):
|
||||
ran.append(name)
|
||||
print(f"{'PASS' if ok else 'FAIL'} {name}"
|
||||
+ (f" [{detail}]" if (detail and not ok) else ""))
|
||||
if not ok:
|
||||
@@ -1228,7 +1302,64 @@ def selftest():
|
||||
and "pin ungradable" in bad_line,
|
||||
f"{ok_line} | {red_line} | {bad_line}")
|
||||
|
||||
print(f"selftest: 44 cells, {44 - len(bad)} passed, {len(bad)} failed")
|
||||
# c45-c49: Q97 -- a refusal that is argparse failing to parse the
|
||||
# INVOCATION is not a refusal of the input. c45/c46 are the judge on both
|
||||
# axes; c47 is the control (a refusal that names its input stays OK); c48
|
||||
# is end to end through run_read_probe against a stub whose REAL argparse
|
||||
# renames the flag the probe passes -- the filed mutant, shrunk to one
|
||||
# cell; c49 is the summary line naming the fault.
|
||||
jm = judge_malformed(
|
||||
2, "", "usage: t.py [-h] --repo-dir REPO_DIT\nt.py: error: "
|
||||
"unrecognized arguments: --repo /tmp/x\n", 2, None)
|
||||
cell("c45 argparse usage at rc 2 on the malformed axis -> ECHOES_USAGE, fatal",
|
||||
jm["verdict"] == "ECHOES_USAGE" and axis_faults([jm]) == [jm],
|
||||
jm["verdict"])
|
||||
jr = judge_read(2, "usage: t.py [-h] --queue QUEUE\nt.py: error: the "
|
||||
"following arguments are required: --queue\n")
|
||||
cell("c46 argparse usage at rc 2 on the read axis -> ECHOES_USAGE, fatal",
|
||||
jr["verdict"] == "ECHOES_USAGE" and axis_faults([jr]) == [jr],
|
||||
jr["verdict"])
|
||||
jm = judge_malformed(2, "", "cannot read /tmp/nope: [Errno 2] No such file "
|
||||
"or directory\n", 2, None)
|
||||
jr = judge_read(2, "cannot read /tmp/nope: [Errno 2] No such file or directory")
|
||||
cell("c47 a refusal that names its input stays OK on both axes",
|
||||
jm["verdict"] == "OK" and jr["verdict"] == "OK",
|
||||
f"{jm['verdict']}/{jr['verdict']}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
stub = os.path.join(td, "renamed_flag.py")
|
||||
with open(stub, "w", encoding="utf-8") as fh:
|
||||
fh.write("import argparse\n"
|
||||
"ap = argparse.ArgumentParser()\n"
|
||||
# deliberately NOT a prefix of --repo: argparse accepts an
|
||||
# unambiguous abbreviation, so --repo -> --repo-dir still
|
||||
# parses and reads the input (measured, Q97).
|
||||
"ap.add_argument('--tree', dest='repo')\n"
|
||||
"a = ap.parse_args()\n"
|
||||
"print('cannot read ' + a.repo)\n"
|
||||
"raise SystemExit(2)\n")
|
||||
rc_m, out_m, err_m = run_read_probe(stub, ["--repo", "{nope}"], 30)
|
||||
jm_bad = judge_malformed(rc_m, out_m, err_m, 2, None)
|
||||
rc_c, out_c, err_c = run_read_probe(stub, ["--tree", "{nope}"], 30)
|
||||
jm_ok = judge_malformed(rc_c, out_c, err_c, 2, None)
|
||||
cell("c48 end to end: a renamed flag reads ECHOES_USAGE, its real flag OK",
|
||||
rc_m == 2 and jm_bad["verdict"] == "ECHOES_USAGE"
|
||||
and rc_c == 2 and jm_ok["verdict"] == "OK",
|
||||
f"{rc_m}:{jm_bad['verdict']} / {rc_c}:{jm_ok['verdict']}")
|
||||
|
||||
axis_echo = [{"tool": "t.py", "probe": "not-a-repo", "verdict": "ECHOES_USAGE",
|
||||
"rc": 2, "expect_rc": 2, "detail": "argparse failure"}]
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
erc, ehard, esoft = report_read(axis_echo, quiet=True)
|
||||
line_e = buf.getvalue()
|
||||
cell("c49 the summary names ECHOES_USAGE with its probe, and the run is red",
|
||||
erc == 1 and (ehard, esoft) == (0, 1)
|
||||
and "malformed-input: 1 probes, 0 ok, 1 fault "
|
||||
"(ECHOES_USAGE t.py/not-a-repo)" in line_e,
|
||||
line_e.strip().splitlines()[-1] if line_e.strip() else "no output")
|
||||
|
||||
print(f"selftest: {len(ran)} cells, {len(ran) - len(bad)} passed, {len(bad)} failed")
|
||||
return 1 if bad else 0
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# CHECK-q97 -- the Q97 gate, rooted at the tree this file sits in.
|
||||
#
|
||||
# Grades tools/selftest_sweep.py's Q97 rule: a refusal at rc 2 that is really
|
||||
# argparse failing to parse the INVOCATION (usage: lead, unrecognized
|
||||
# arguments, ...) is ECHOES_USAGE -- fatal on both input axes -- so a renamed
|
||||
# flag can no longer turn the malformed/read axes green. Exit codes follow the
|
||||
# repo contract: 0 clean, 1 graded fault, 2 could not grade.
|
||||
#
|
||||
# bash wizard-work/volt/CHECK-q97
|
||||
set -u
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/../.." || exit 2
|
||||
exec python3 wizard-work/volt/q97_check.py
|
||||
@@ -0,0 +1,177 @@
|
||||
#!/usr/bin/env python3
|
||||
"""q97_check.py -- the Q97 gate: a refusal must be about the INPUT.
|
||||
|
||||
The defect this grades was filed against `tools/selftest_sweep.py`: a probe
|
||||
that is refused at rc 2 with `usage: ... / error: unrecognized arguments: ...`
|
||||
is an argparse failure to parse the INVOCATION, not a refusal of the input.
|
||||
The tool never read anything, and the malformed-input axis -- the one place CLI
|
||||
drift could not hide -- read clean.
|
||||
|
||||
Four legs, all measured on this tree:
|
||||
|
||||
1. the sweep's own selftest is green and its printed cell count equals the
|
||||
pinned row in `tools/selftest_baseline.tsv` (a new cell without a re-pin
|
||||
is a PIN_DRIFT, so the gate cannot pass on a cell nobody counted);
|
||||
2. the live axes on this checkout are clean -- 0 faults on read-refusal and
|
||||
malformed-input -- so the new rule does not redden a tool that refuses
|
||||
correctly (28 + 27 probes at the landing head);
|
||||
3. MUTANT: `tools/citation_hash_class.py` copied to a scratch file with its
|
||||
`--repo` flag renamed to `--tree` (a rename that is NOT a prefix of the
|
||||
old one: argparse abbreviates long options, so `--repo` -> `--repo-dir`
|
||||
still parses and this check refuses to pretend otherwise). Both probes
|
||||
that invoke it with `--repo` must read ECHOES_USAGE, and the axis must be
|
||||
red, rc 1;
|
||||
4. CONTROL: the same copy unmutated reads no ECHOES_USAGE at all and every
|
||||
probe is OK -- so leg 3 is the mutation, not a file that was already red.
|
||||
|
||||
Exit codes follow the repo contract: 0 clean, 1 graded fault, 2 could not
|
||||
grade (no sweep module, no tool copy, no python3).
|
||||
|
||||
Run: python3 wizard-work/volt/q97_check.py
|
||||
"""
|
||||
import importlib.util
|
||||
import io
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from contextlib import redirect_stdout
|
||||
|
||||
HERE = os.path.dirname(os.path.realpath(__file__))
|
||||
ROOT = os.path.dirname(os.path.dirname(HERE))
|
||||
TOOLS = os.path.join(ROOT, "tools")
|
||||
SWEEP = os.path.join(TOOLS, "selftest_sweep.py")
|
||||
BASELINE = os.path.join(TOOLS, "selftest_baseline.tsv")
|
||||
SUBJECT = os.path.join(TOOLS, "citation_hash_class.py")
|
||||
|
||||
failures = []
|
||||
|
||||
|
||||
def note(ok, line):
|
||||
print(("PASS " if ok else "FAIL ") + line)
|
||||
if not ok:
|
||||
failures.append(line)
|
||||
|
||||
|
||||
def leg_selftest():
|
||||
"""The sweep's own cells, and the pin that counts them."""
|
||||
proc = subprocess.run([sys.executable, SWEEP, "--selftest"],
|
||||
capture_output=True, text=True, timeout=300)
|
||||
out = (proc.stdout or "").splitlines()
|
||||
cells = [l for l in out if l.startswith("selftest:")]
|
||||
printed = cells[-1] if cells else "no count line"
|
||||
note(proc.returncode == 0 and bool(cells),
|
||||
"leg 1a sweep --selftest rc %d: %s"
|
||||
% (proc.returncode, printed))
|
||||
pinned = None
|
||||
with open(BASELINE, encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
parts = line.split("\t")
|
||||
if len(parts) == 3 and parts[0] == "selftest_sweep.py":
|
||||
pinned = int(parts[1])
|
||||
if pinned is None:
|
||||
note(False, "leg 1b no pinned row for selftest_sweep.py -- cannot grade")
|
||||
return
|
||||
seen = None
|
||||
if cells:
|
||||
try:
|
||||
seen = int(cells[-1].split()[1].replace(",", ""))
|
||||
except (IndexError, ValueError):
|
||||
seen = None
|
||||
note(seen == pinned,
|
||||
"leg 1b printed %s cells, pinned %s" % (seen, pinned))
|
||||
|
||||
|
||||
_SWEEP = []
|
||||
|
||||
|
||||
def sweep_module():
|
||||
"""`tools/selftest_sweep.py`, loaded by path so this gate grades the tree
|
||||
it sits in and not a copy on sys.path."""
|
||||
if not _SWEEP:
|
||||
spec = importlib.util.spec_from_file_location("q97_selftest_sweep", SWEEP)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError("cannot load %s" % SWEEP)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
_SWEEP.append(mod)
|
||||
return _SWEEP[0]
|
||||
|
||||
|
||||
def axis_for(path):
|
||||
"""The rows for one `citation_hash_class.py` copy, through the real probe
|
||||
path. Rows for other tools are dropped: with a one-row table every other
|
||||
probe is NO_TOOL, which is noise about the table, not about the copy."""
|
||||
sweep = sweep_module()
|
||||
rows = sweep.read_axis([("citation_hash_class.py", path)], 60)
|
||||
return sweep, [r for r in rows if r["tool"] == "citation_hash_class.py"]
|
||||
|
||||
|
||||
def leg_mutant_and_control():
|
||||
sweep, control = axis_for(SUBJECT)
|
||||
echo_c = [r for r in control if r["verdict"] == "ECHOES_USAGE"]
|
||||
bad_c = [r for r in control if r["verdict"] in sweep.FATAL_READ]
|
||||
note(not echo_c and not bad_c,
|
||||
"leg 4 control: clean copy reads %d probe(s), 0 ECHOES_USAGE, "
|
||||
"0 fault" % len(control))
|
||||
|
||||
td = tempfile.mkdtemp(prefix="q97-")
|
||||
try:
|
||||
mutant = os.path.join(td, "citation_hash_class.py")
|
||||
shutil.copyfile(SUBJECT, mutant)
|
||||
with open(mutant, encoding="utf-8") as fh:
|
||||
src = fh.read()
|
||||
old = "ap.add_argument('--repo', default=DEFAULT_REPO)"
|
||||
new = "ap.add_argument('--tree', dest='repo', default=DEFAULT_REPO)"
|
||||
if src.count(old) != 1:
|
||||
note(False, "leg 3 cannot build the mutant: %r appears %d time(s)"
|
||||
% (old, src.count(old)))
|
||||
return
|
||||
with open(mutant, "w", encoding="utf-8") as fh:
|
||||
fh.write(src.replace(old, new, 1))
|
||||
sweep2, rows = axis_for(mutant)
|
||||
echo_m = [r for r in rows if r["verdict"] == "ECHOES_USAGE"]
|
||||
named = sorted((r["probe"] for r in echo_m))
|
||||
note(named == ["missing", "not-a-repo"],
|
||||
"leg 3 mutant (--repo -> --tree): ECHOES_USAGE on %s" % (named,))
|
||||
buf = io.StringIO()
|
||||
with redirect_stdout(buf):
|
||||
rc, hard, soft = sweep2.report_read(rows, quiet=True)
|
||||
line = buf.getvalue().strip().splitlines()
|
||||
summary = line[-1] if line else "no summary"
|
||||
note(rc == 1 and (hard, soft) == (1, 1)
|
||||
and "ECHOES_USAGE citation_hash_class.py/not-a-repo" in buf.getvalue(),
|
||||
"leg 3b mutant axis %s" % summary)
|
||||
finally:
|
||||
shutil.rmtree(td, ignore_errors=True)
|
||||
|
||||
|
||||
def leg_live_axes():
|
||||
sweep = sweep_module()
|
||||
live, _source, _untracked = sweep.discover(ROOT, None)
|
||||
axis = sweep.read_axis(live, 60)
|
||||
bad = [r for r in axis if r["verdict"] in sweep.FATAL_READ]
|
||||
echo = [r for r in axis if r["verdict"] == "ECHOES_USAGE"]
|
||||
note(not bad and not echo,
|
||||
"leg 2 live axes: %d probe(s), 0 fault, 0 ECHOES_USAGE" % len(axis))
|
||||
|
||||
|
||||
def main():
|
||||
if not (os.path.isfile(SWEEP) and os.path.isfile(SUBJECT)):
|
||||
print("cannot grade: tools/selftest_sweep.py or citation_hash_class.py "
|
||||
"missing under %s" % ROOT, file=sys.stderr)
|
||||
return 2
|
||||
leg_selftest()
|
||||
leg_live_axes()
|
||||
leg_mutant_and_control()
|
||||
if failures:
|
||||
print("Q97 GATE: rc 1 -- %d leg(s) failed" % len(failures))
|
||||
return 1
|
||||
print("Q97 GATE: rc 0 -- a refusal that echoes the invocation is fatal on "
|
||||
"both input axes, and a mutating flag rename is caught.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user