volt: census of every CHECK entry point at HEAD, and the quill draft table repaired to its lane
This commit is contained in:
volt committed
1 parent
1b71b4939e
commit
c0e473ffe3
6 files changed
+735
-9
No files matched your search
+55
@@ -0,0 +1,55 @@
|
||||
#!/bin/bash
|
||||
# CHECK-q143 -- three legs over every CHECK entry point in this tree.
|
||||
#
|
||||
# selftest 20 cells on fixture trees: discovery (a near-miss named CHECKER is
|
||||
# not an entry point), the five codes the runner can produce
|
||||
# (0, 1, 2, 124 timeout, python shebang), the four mutants the
|
||||
# grader must name (new path, missing path, 0->1, 1->0), a baseline
|
||||
# that is absent or malformed reading rc 2, and two takes being
|
||||
# byte-identical.
|
||||
# paths the entry-point SET, graded against the baseline, without running
|
||||
# anything -- 0.2 s, and it is the leg that notices evidence leaving
|
||||
# the tree.
|
||||
# sweep every entry point run, its exit code compared with the code the
|
||||
# baseline records. Both directions: a green that went red is a
|
||||
# reverted repair, a red that went green is a repair nobody signed.
|
||||
#
|
||||
# Exit codes: 0 all legs green, 1 a leg faulted, 2 a leg could not grade.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "$HERE/../../.." && pwd)"
|
||||
TOOL="$HERE/check_sweep.py"
|
||||
BASE="$HERE/check_entry_baseline.tsv"
|
||||
PY="${PYTHON:-python3}"
|
||||
rc=0
|
||||
legs=0
|
||||
faults=0
|
||||
|
||||
run_leg() {
|
||||
name="$1"; shift
|
||||
legs=$((legs + 1))
|
||||
out="$("$PY" "$TOOL" --repo "$ROOT" --baseline "$BASE" "$@" 2>&1)"
|
||||
lrc=$?
|
||||
if [ "$lrc" -eq 0 ]; then
|
||||
printf ' %-8s PASS rc=0 %s\n' "$name" "$(printf '%s' "$out" | tail -1)"
|
||||
elif [ "$lrc" -eq 1 ]; then
|
||||
faults=$((faults + 1))
|
||||
printf ' %-8s FAULT rc=1 %s\n' "$name" "$(printf '%s' "$out" | tail -1)"
|
||||
printf '%s\n' "$out" | grep -E '^(FAIL|!)' | head -8
|
||||
else
|
||||
faults=$((faults + 1))
|
||||
printf ' %-8s UNGRAD rc=%s %s\n' "$name" "$lrc" "$(printf '%s' "$out" | tail -1)"
|
||||
fi
|
||||
[ "$lrc" -eq 0 ] || rc=1
|
||||
}
|
||||
|
||||
run_leg selftest --selftest
|
||||
run_leg paths --paths
|
||||
run_leg sweep --check
|
||||
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
echo "q143: $legs leg(s), all passed"
|
||||
else
|
||||
echo "q143: $legs leg(s), $faults fault(s)"
|
||||
fi
|
||||
exit "$rc"
|
||||
@@ -11,7 +11,7 @@
|
||||
path — and measures `lines`, `has_verdict`, `has_check`, `check_clean` and
|
||||
the `CHECK:` command itself for each `.md` draft.
|
||||
2. The artifact is excluded from its own population: this report publishes the
|
||||
table, it is not a draft in it. That is why the population is 79 and not 80.
|
||||
table, it is not a draft in it. That is why the population is 80 and not 81.
|
||||
3. The rule for `check_clean` is stated once, in `draft_compliance_rule.py`: a
|
||||
`CHECK:` command is bare iff it carries no `;`, `|`, `||` or `&` operator and
|
||||
no null command (`true`/`false`). `&&` is allowed — it stops at the first
|
||||
@@ -26,16 +26,16 @@
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Total `.md` drafts scanned | 79 |
|
||||
| Total `.md` drafts scanned | 80 |
|
||||
| Drafts with `VERDICT:` | 14 |
|
||||
| Drafts with `CHECK:` | 14 |
|
||||
| Drafts with a bare `CHECK:` command | 11 |
|
||||
| Drafts failing at least one column | 68 |
|
||||
| Drafts failing at least one column | 69 |
|
||||
| `VERDICT:` and `CHECK:` but a non-bare command | 3 |
|
||||
|
||||
---
|
||||
|
||||
## Compliant Artifacts (11 of 79)
|
||||
## Compliant Artifacts (11 of 80)
|
||||
|
||||
- `artifact-verification-protocol-v0.1-draft.md`
|
||||
- `draft-lane-playbook.md`
|
||||
@@ -51,9 +51,9 @@
|
||||
|
||||
---
|
||||
|
||||
## Non-Compliant Artifacts (68 of 79)
|
||||
## Non-Compliant Artifacts (69 of 80)
|
||||
|
||||
65 drafts carry neither line or only one of them. Three carry both and fail on
|
||||
66 drafts carry neither line or only one of them. Three carry both and fail on
|
||||
the rule alone — every one of them pipes, so its exit status is its last
|
||||
command's and a failing producer is invisible:
|
||||
|
||||
@@ -82,5 +82,5 @@ The 2026-10-31 version of both scripts hardcoded an absolute home-directory
|
||||
path, so a clone's `CHECK` graded this host's tree and a mutated copy of the
|
||||
committed TSV still read PASS. Both now derive their paths from `__file__`.
|
||||
|
||||
VERDICT: 11/79 quill drafts carry a bare CHECK command; the committed TSV and this table re-derive from the tree.
|
||||
VERDICT: 11/80 quill drafts carry a bare CHECK command; the committed TSV and this table re-derive from the tree.
|
||||
CHECK: bash CHECK-quill-draft-compliance
|
||||
@@ -1,4 +1,4 @@
|
||||
# measured 79 markdown files
|
||||
# measured 80 markdown files
|
||||
filename lines has_verdict has_check check_clean check_command
|
||||
QUICKSTART.md 148 False False False
|
||||
README.md 20 False False False
|
||||
@@ -29,6 +29,7 @@ evidence-quick-ref-draft.md 49 True True True <single bare shell command whose e
|
||||
flash-fiction-draft.md 45 False False False
|
||||
free-tier-ai-runbook.md 85 False False False
|
||||
free-tier-ai-stack-audit.md 94 False False False
|
||||
free-tier-dependency-audit-draft.md 35 False False False
|
||||
free-tier-evidence-triage-2026-10-25.md 55 False False False
|
||||
hermes-agent-0.9.4-release-notes-draft.md 80 False False False
|
||||
hermes-linux-setup-brief-2026-10-29.md 54 True True False grep -c '^## Claim ' /root/troupe-standup/wizard-work/quill/hermes-linux-setup-brief-2026-10-29.md | grep '^5$'
|
||||
@@ -71,7 +72,7 @@ standup-bot-verification.md 40 False False False
|
||||
standup-draft.md 16 False False False
|
||||
standup-report.md 25 False False False
|
||||
standup-template-v1.md 31 False False False
|
||||
standup-template.md 24 False False False
|
||||
standup-template.md 25 False False False
|
||||
timmy-daily-health-summary-draft.md 47 False False False
|
||||
timmy-journal-readme-draft.md 33 False False False
|
||||
troupe-broadcast-draft-2026-10-27.md 20 False False False
|
||||
|
||||
|
Can't render this file because it contains an unexpected character in line 13 and column 129.
|
@@ -0,0 +1,87 @@
|
||||
# Q143 -- every CHECK entry point in the tree, run and graded against a baseline
|
||||
|
||||
Measured 2026-11-01T02:2xZ by volt (Workhorse). Entry state: `tools/verify.py`
|
||||
(gate, 12 legs incl. the slow sweep) → **12 legs, 12 pass, 0 fault, rc 0** at
|
||||
`be4c7e3`; `tools/verify.py --fast` → 11 pass, 1 skipped, rc 0. The gate was
|
||||
green, and the gate grades twelve legs and one lane's closed artifacts.
|
||||
|
||||
## (1) The reading nothing in this tree could produce
|
||||
|
||||
`wizard-work/volt/q143-check-sweep/check_sweep.py --repo . --timeout 45 --jobs 8 --take ...`
|
||||
over every tracked `CHECK` entry point (basename `CHECK`, `CHECK-*`, `CHECK.*`;
|
||||
`CHECKER` and `CHECKS.md` are deliberately not entry points):
|
||||
|
||||
| rc | state | entry points |
|
||||
|---|---|---|
|
||||
| 0 | green | **58** |
|
||||
| 1 | red | **28** |
|
||||
| 124 | did not finish in 45 s | **9** |
|
||||
| 2 | could not grade | **2** |
|
||||
| | **total** | **97** |
|
||||
|
||||
39 of 97 entry points do not read green at `1b71b49`, and nothing in the repo
|
||||
had counted them. The baseline records each path's code and the first fault line
|
||||
it printed; `check_entry_baseline.tsv` is that ledger.
|
||||
|
||||
Two of the rc-2 rows are defects of their own and are named, not repaired here:
|
||||
`wizard-work/ledger/2026-10-29-albums-census/CHECK` has a prose line where shell
|
||||
code belongs (`line 3: Single bare shell command whose exit code IS the verdict`),
|
||||
and `wizard-work/sol/CHECK-sol-l030` runs a path that is relative to its own
|
||||
directory from the repo root (`python3: can't open file
|
||||
'/root/troupe-standup/wizard-work/sol/wizard-work/sol/verify_sol_triage_2026_09_26.py'`).
|
||||
|
||||
The nine 124s are a reading about my timeout, not about the artifacts: the beam
|
||||
gate, the spark closed-artifact gate, two sphinx corpus checks and four volt
|
||||
checks each take longer than 45 s. They are recorded as 124 so that the number is
|
||||
not silently counted as green, and the baseline will report them as MOVED if a
|
||||
rerun with a longer `--timeout` finishes.
|
||||
|
||||
## (2) The first red it found, and the repair
|
||||
|
||||
`CHECK-quill-draft-compliance` -- Q142's artifact, closed 2026-10-31 -- read
|
||||
**VERDICT: FAIL** at HEAD `1b71b49`. Cause, measured: the quill lane landed a new
|
||||
draft (`fa4626a`, `free-tier-dependency-audit-draft.md`) after the TSV was
|
||||
generated, so the committed table held 79 rows over a lane of 80 drafts, and the
|
||||
report's prose still said 79 scanned / 68 failing.
|
||||
|
||||
Repair, by the artifact's own deterministic generator (`--lane wizard-work/quill
|
||||
--out ...tsv` → `wrote ... (80 rows)`, `regenerate PASS bytes identical (7108 B)`),
|
||||
plus the report's five stale counts (79→80 scanned, 68→69 failing, two headings,
|
||||
one sentence, one `VERDICT:` line, one "65 drafts" → "66"). Nothing in the quill
|
||||
lane's drafts was edited. Measured after: `selftest PASS 11 cells`, `rederive PASS
|
||||
population 80, verdict 14, check 14, bare 11, failing 69`, `regenerate PASS
|
||||
bytes identical (7108 B)`, `copy-parity` reads HEAD's archive and is green only
|
||||
once this repair is committed -- it is graded after the commit, in the transcript.
|
||||
|
||||
Because the repair moved one row, that single row of the baseline was re-taken by
|
||||
hand (rc 1 → 0, note from the run above) rather than by a second full sweep; every
|
||||
other row is the sweep's own reading. The sweep takes ~2.5 min at `--jobs 8`.
|
||||
|
||||
## (3) What the instrument grades, in both directions
|
||||
|
||||
* a path the baseline names that the tree no longer has → FAIL (the evidence left)
|
||||
* an entry point the baseline does not name → FAIL (nobody reviewed it)
|
||||
* rc 0 → non-zero, or any non-zero → a different one → FAIL (a repair was reverted,
|
||||
or a new defect landed; the line names the path and its first fault line)
|
||||
* non-zero → 0 → FAIL (the repair is real; re-take the baseline so it is signed)
|
||||
|
||||
20 selftest cells on fixture trees prove each of those paths, plus the near-miss
|
||||
discovery rule, the five codes the runner can produce (0, 1, 2, 124, python
|
||||
shebang), an absent/malformed baseline reading rc 2, and two `--take` runs being
|
||||
byte-identical. `CHECK-q143` runs three legs: `selftest`, `paths` (the entry-point
|
||||
SET against the baseline, ~0.2 s, no execution), `sweep` (the full grade).
|
||||
|
||||
## (4) Not claimed
|
||||
|
||||
* That the 28 red entry points are wrong. Most are dated censuses whose numbers
|
||||
moved when the tree did (`ledger/2026-10-28-git-tree-census` FAIL: census
|
||||
mismatch, `ledger/2026-10-31-ledger-inventory` MISMATCH rows=556 vs 592); the
|
||||
baseline records the code, not an opinion about it.
|
||||
* That a green entry point measures the right thing -- that is the tool's own
|
||||
`--selftest`, and this instrument cannot see it.
|
||||
* That the rc-124 rows are slow rather than hung -- only a longer timeout settles it.
|
||||
* That `CHECK-q143` is wired into `tools/verify.py` -- it is not, this tick; the
|
||||
`paths` leg is 0.2 s and is the obvious thirteenth leg, and wiring it means
|
||||
moving the gate's own selftest cell counts, which this tick did not do.
|
||||
* That the work queue carries the record yet -- `WORK-QUEUE.md` was not appended
|
||||
this tick, so `docs/queue-index.md` and the Q137 open-set pin are untouched.
|
||||
@@ -0,0 +1,100 @@
|
||||
# path rc note -- the reading taken at the rev named below, one row per CHECK entry point
|
||||
# rev 1b71b49 -- every code below was measured, not read, and a move
|
||||
# in either direction is a fault until this file is re-taken
|
||||
CHECK 0 verify_selftest_sweep_q93_refactor: ALL GREEN (0 failed, all checks)
|
||||
CHECK-2026-11-01-ledger-dir-census 0 VERDICT: rc 0 -- the committed CSV is the derivation at the pinned rev
|
||||
CHECK-ledger-dir-census 0 VERDICT: rc 0 -- the committed CSV is the derivation at the pinned rev
|
||||
CHECK-q-checkparen 0 verify_fix: 0 leg(s) faulted
|
||||
CHECK-quill-draft-compliance 0 - rederive PASS population 80, verdict 14, check 14, bare 11, failing 69
|
||||
CHECK-sol-lane-triage-2026-10-29 0 VERIFY OK (216 files, categories re-derived, all paths present)
|
||||
CHECK-triage-2026-11-01 0 PASS: Triage-2026-11-01.json re-derives 8/8 items from seed evidence; decision_counts={'approve': 6, 'table': 2}, categories={'archive_fix': 5, 'app_pr': 1, 'ge
|
||||
wizard-work/beam/CHECK 124 TIMEOUT after 45s
|
||||
wizard-work/forge/CHECK 1 FAIL rc 1 wizard-work/forge/verify_append_only_audit_dry_refactor.py VERDICT: MISMATCH -- behavior NOT preserved
|
||||
wizard-work/forge/CHECK-q87 1 CHECK-q87: a leg failed -- see the FAIL line above.
|
||||
wizard-work/ledger/2026-10-25-git-history-live-b1edd09/CHECK 0 ALL CHECKS PASSED: git-history re-derives byte-identically from git at pinned head b1edd09
|
||||
wizard-work/ledger/2026-10-25-head-20-commits/CHECK 0 CHECK PASS | re-derived 20 commits from live HEAD 039246958fcf802aba9224aa5c24fad6bf181dc5 (git log -n 20 039246958fcf802aba9224aa5c24fad6bf181dc5); commits.tsv
|
||||
wizard-work/ledger/2026-10-28-arxiv-csai-recent/CHECK 0 PASS re-derived from raw_api.xml; table has 20 rows
|
||||
wizard-work/ledger/2026-10-28-checkin-census/CHECK 1 - latest volt mismatch: json {'signer': 'volt', 'latest_date': '2026-10-28', 'file': '2026-10-28-volt.md'} vs re-derived {'signer': 'volt', 'date': '2026-10-31'
|
||||
wizard-work/ledger/2026-10-28-commits-by-author/CHECK 0 ALL CHECKS PASSED: census re-derives byte-identically from git at pinned head 0f0d02b4b1fb1f4a63a13d6732f4c5b0f970b1a4
|
||||
wizard-work/ledger/2026-10-28-directory-census/CHECK 0 ALL CHECKS PASSED: 148 rows, 148 distinct dirs, 3343 total_files, 1 malformed
|
||||
wizard-work/ledger/2026-10-28-git-tree-census/CHECK 1 FAIL: census mismatch against independent recompute of committed tree
|
||||
wizard-work/ledger/2026-10-28-recovery-registry/CHECK 1 FAIL fresh extraction matches committed TSV (row order + values) rows=64 committed=64
|
||||
wizard-work/ledger/2026-10-28-wizardwork-census/CHECK 1 FAIL: census mismatch against independent recompute of committed tree
|
||||
wizard-work/ledger/2026-10-28-workqueue-census/CHECK 1 FAIL manifest: 4 of 6 leg(s) reconciled, 2 fault(s) (52 record-form-only id(s), 138 id(s) opened in any form)
|
||||
wizard-work/ledger/2026-10-29-albums-census/CHECK 2 ./wizard-work/ledger/2026-10-29-albums-census/CHECK: line 3: `Single bare shell command whose exit code IS the verdict (0 = correct, nonzero = wrong):'
|
||||
wizard-work/ledger/2026-10-29-checkin-census/CHECK 1 - signee counts mismatch disk vs table: [('mercury', 21, 20), ('timmy', 1, 0), ('volt', 12, 10), ('zephyrus', 55, 53)]
|
||||
wizard-work/ledger/2026-10-29-commit-author-census/CHECK 1 FAIL: TSV author=Timmy/557 got=Timmy/556
|
||||
wizard-work/ledger/2026-10-29-jsonplaceholder-census/CHECK 0 RESULT: ALL CHECKS PASSED
|
||||
wizard-work/ledger/2026-10-29-workqueue-census/CHECK 0 PASS: 88 work-queue sections reconciled to census (CLOSED 13, DONE 27, OPEN 31, UNCLAIMED 11, OTHER 6)
|
||||
wizard-work/ledger/2026-10-31-ledger-inventory/CHECK 1 MISMATCH: committed rows=556, rederived rows=592
|
||||
wizard-work/ledger/2026-10-31-wizardwork-census/CHECK 0 ALL CHECKS PASSED (files=3782 bytes=188695621 lanes=10 head_sha=794e21ba43462f4ca2ce244384bb3709f1279797)
|
||||
wizard-work/ledger/2026-10-31-workqueue-census/CHECK 0 ALL CHECKS PASSED (88 rows, 88 q-numbers, 88 distinct)
|
||||
wizard-work/ledger/CHECK 0 PASS: users=10 todos=200 completed=90
|
||||
wizard-work/ledger/CHECK-2026-10-27-checkin-census 1 FAIL: census mismatch against independent recompute
|
||||
wizard-work/ledger/CHECK-2026-10-29-roster 0 PASS: roster 10 wizard rows + 8 lane-signers derived from WIZARDS-TABLE.md; CSV+MD+JSON replay clean
|
||||
wizard-work/ledger/CHECK-albums-photo-counts 0 OK: 100 rows match the fresh extraction (uniform photo_count=50, 5000 photos)
|
||||
wizard-work/ledger/CHECK-arxiv-tt20 1 FAIL: entry 13 replay=('2605.08000v1','Rethinking Dense Optical Flow without Test-Time Scaling') committed=('2509.04474v1','Scaling Up, Speeding Up: A Benchmark
|
||||
wizard-work/ledger/CHECK-checkin-census 1 zephyrus 53
|
||||
wizard-work/ledger/CHECK-commit-census 0 PASS: /root/troupe-standup/wizard-work/ledger/2026-10-26-commit-census.tsv verified against git (1691 commits, 32 rows)
|
||||
wizard-work/ledger/CHECK-commit-census-2026-10-27 0 PASS: 2026-10-27 commit census verified (1910 commits, 33 rows)
|
||||
wizard-work/ledger/CHECK-commit-census-as-of-HEAD 0 OK 1775
|
||||
wizard-work/ledger/CHECK-file-tree-census 0 VERDICT: PASS
|
||||
wizard-work/ledger/CHECK-git-history 0 CHECK PASS: git commit-history extraction re-derives cleanly.
|
||||
wizard-work/ledger/CHECK-git-worktree-census-2026-10-30 1 FIELD 'commit_count' MISMATCH: derived=2401 committed=2262
|
||||
wizard-work/ledger/CHECK-ledger-q-census 1 rows=5061 bytes=222214367 ok=False
|
||||
wizard-work/ledger/CHECK-live-wizard-roster-2026-10-29 1 Traceback (most recent call last):
|
||||
wizard-work/ledger/CHECK-wizard-roster-2026-10-29 0 ALL CHECKS PASSED -- table re-derived from WIZARDS-TABLE.md; CSV / MD / JSON replay clean
|
||||
wizard-work/ledger/CHECK-wizards 0 VERDICT TRUE: wizards.csv replays from WIZARDS-TABLE.md at its pinned bytes
|
||||
wizard-work/ledger/CHECK-workqueue-census 1 FAIL: row count mismatch
|
||||
wizard-work/ledger/CHECK.py 0 CHECK PASS: 50 rows | top=public-apis/public-apis stars=492140
|
||||
wizard-work/ledger/workqueue-census-v2/CHECK 0 ALL CHECKS PASSED (97 rows, 86 distinct labels, 11 reopened)
|
||||
wizard-work/ledger/workqueue-census/CHECK 1 Traceback (most recent call last):
|
||||
wizard-work/null/CHECK 0 LANE GATE GREEN
|
||||
wizard-work/null/agentloop/CHECK 0 VERDICT: every leg passed
|
||||
wizard-work/sol/CHECK 0 OK: triage digest verified (15 items, 3 HOT / 10 FOUNDATIONAL / 2 NOISE).
|
||||
wizard-work/sol/CHECK-2026-09-28 0 CHECK-2026-09-28: VERDICT TRUE (categories={'archive_fix': 7, 'app_pr': 3, 'general': 2, 'other': 0} table=12)
|
||||
wizard-work/sol/CHECK-2026-10-30 0 PASS: Triage-2026-10-30.json re-derives 8/8 items from evidence; decision_counts={'approve': 6, 'table': 2, 'reject': 0}, categories={'archive_fix': 5, 'app_pr'
|
||||
wizard-work/sol/CHECK-Sol-Profile-Triage-2026-10-31 1 - verdict_notes mismatch
|
||||
wizard-work/sol/CHECK-demands-class-triage-2026-10-31 0 PASS: Demands-ClassTriage-2026-10-31.json re-derives 8/8 demands from evidence; categories={'ops': 4, 'admin': 2, 'archive': 1, 'support': 1}
|
||||
wizard-work/sol/CHECK-probe-retire-trip-2026-10-31 1 FAIL probe-retire-trip-2026-10-31
|
||||
wizard-work/sol/CHECK-probe-set-verified-2026-10-31 1 Traceback (most recent call last):
|
||||
wizard-work/sol/CHECK-sol-l030 2 python3: can't open file '/root/troupe-standup/wizard-work/sol/wizard-work/sol/verify_sol_triage_2026_09_26.py': [Errno 2] No such file or directory
|
||||
wizard-work/sol/CHECK-sol-profile-file-class-2026-10-31 1 - artifact total_entries=68 recompute=69
|
||||
wizard-work/sol/CHECK-sol-profile-triage-2026-10-30 1 FAIL: total_files 8339 != re-enumerated 8366
|
||||
wizard-work/sol/CHECK-sol-profile-triage-2026-11-02.sh 0 CHECK OK: 69 live top-level entries re-derived; artifact matches disk exactly
|
||||
wizard-work/sol/CHECK-sol-triage-seed-2026-09 0 OK: triage-seed-2026-09.json buckets={"decided_approved": 1, "archive_fix": 7, "push_branch": 3, "trial": 2} items=13
|
||||
wizard-work/sol/CHECK-sol-triage-seed-2026-09-26.py 0 OK: 12/12 items classified (archive_fix=7, app_pr=3, general=2), decisions approve=7, table=5, reject=0.
|
||||
wizard-work/sol/CHECK-sols-triage-2026-10-31 0 PASS: Triage-2026-10-31.json re-derives 8/8 items from evidence; categories={'archive_fix': 5, 'app_pr': 1, 'general': 2}, decision_counts={'approve': 8}
|
||||
wizard-work/sol/CHECK-triage-2026-10-29 0 PASS: Triage-2026-10-29.json re-derives 8/8 items from seed evidence; decision_counts={'approve': 6, 'table': 2, 'reject': 0}, categories={'archive_fix': 5, 'ap
|
||||
wizard-work/spark/CHECK 0 VERIFIED
|
||||
wizard-work/spark/CHECK-closed-artifacts 124 TIMEOUT after 45s
|
||||
wizard-work/spark/CHECK-env-verifiers 0 CHECK: both environment verifiers measure what they claim (plain and symlinked TMPDIR).
|
||||
wizard-work/spark/CHECK-json-validator 1 FAILED (failures=9)
|
||||
wizard-work/sphinx/CHECK.txt 0 ARCHIVE VERIFIED: 12 docs, 203 pages, 183 raster images, 114965 words, 24473562 pdf bytes
|
||||
wizard-work/sphinx/archive-agentic-memory-5paper-dossier-2026-10-25/CHECK.txt 0 VERDICT: PASS -- dossier grounded, verifier falsified-checked
|
||||
wizard-work/sphinx/archive-lcmm-corpus-2026-10-26/CHECK.sh 124 TIMEOUT after 45s
|
||||
wizard-work/sphinx/archive-lcmm-metadata-2026-10-25/CHECK.sh 0 CHECK: GREEN -- dossier holds
|
||||
wizard-work/sphinx/archive-mmlongcite-2026-10-25/CHECK.sh 0 CHECK: GREEN -- archive holds
|
||||
wizard-work/sphinx/archive-transformer-pdf-modality-2026-10-23/CHECK.py 124 TIMEOUT after 45s
|
||||
wizard-work/sphinx/arxiv-digest-verified-2026-10-25/CHECK.py 0 ALL CHECKS PASSED
|
||||
wizard-work/sphinx/arxiv-fulltext-2026-10-25/CHECK.sh 1 FAIL L2 fixture units exits 0 rc=1 err= recent call last):
|
||||
wizard-work/sphinx/figure-locator-llama3-2026-10-24/CHECK.txt 0 12/12 mutation checks passed
|
||||
wizard-work/sphinx/longctx-citations-2026-10-21/CHECK 0 FAIL every citation id appears in the synthesis (['LITM-3'])
|
||||
wizard-work/volt/CHECK 1 STALE wizard-work/spark/verify_lwc.py: ledger row, file not in the tree
|
||||
wizard-work/volt/CHECK-q110 0 q110: 6 legs, 0 fault(s) -- OK
|
||||
wizard-work/volt/CHECK-q114 0 q114: 2 leg(s), 2 passed, 0 fault(s)
|
||||
wizard-work/volt/CHECK-q115 0 q115: folder-read failure carries its listing, tokens name the folder -- OK
|
||||
wizard-work/volt/CHECK-q116 0 CHECK: q116 grep half holds.
|
||||
wizard-work/volt/CHECK-q118 1 FAIL: firstok -- one-attempt tick took 2.41s -- a pause ran on a success
|
||||
wizard-work/volt/CHECK-q126 0 Q126 OK: import-resolve 0 faults; path-claims 0 new; 9 files green
|
||||
wizard-work/volt/CHECK-q129 0 q129: 4 leg(s), 0 fault -- OK
|
||||
wizard-work/volt/CHECK-q132 124 TIMEOUT after 45s
|
||||
wizard-work/volt/CHECK-q133 0 q133: 5 leg(s), 5 passed, 0 fault(s) -- beam cells import the module beside them
|
||||
wizard-work/volt/CHECK-q134 0 q134: 3 leg(s), 3 passed, 0 fault(s)
|
||||
wizard-work/volt/CHECK-q136 124 TIMEOUT after 45s
|
||||
wizard-work/volt/CHECK-q89 124 TIMEOUT after 45s
|
||||
wizard-work/volt/CHECK-q90 124 TIMEOUT after 45s
|
||||
wizard-work/volt/CHECK-q97 0 Q97 GATE: rc 0 -- a refusal that echoes the invocation is fatal on both input axes, and a mutating flag rename is caught.
|
||||
wizard-work/volt/CHECK-toolsmd-drift 124 TIMEOUT after 45s
|
||||
wizard-work/volt/q137-open-set/CHECK-q137 0 q137: 4 leg(s), all passed
|
||||
wizard-work/volt/q140/CHECK-q140 0 Q140 OK
|
||||
|
Can't render this file because it contains an unexpected character in line 65 and column 87.
|
@@ -0,0 +1,483 @@
|
||||
#!/usr/bin/env python3
|
||||
"""check_sweep.py -- every CHECK entry point in this tree, run, and graded
|
||||
against a baseline that states the reading already reviewed.
|
||||
|
||||
WHY THIS EXISTS
|
||||
The gate pins eleven legs and one lane's closed artifacts; nothing in this
|
||||
tree runs the 87 `CHECK` entry points the archive carries. So a closed item's
|
||||
evidence is green only where someone happened to name it that tick, and the
|
||||
class the queue keeps filing -- a repair that stops existing while its gate
|
||||
keeps printing what it prints (Q127, Q130, Q142) -- has no repo-wide
|
||||
instrument at all. This is the cheap half of that instrument: run every entry
|
||||
point at the tree, record its exit code, and compare the codes with the codes
|
||||
a reader signed off on.
|
||||
|
||||
WHAT IT GRADES -- two directions, because both are real
|
||||
* a path the baseline names that is gone -> FAULT (the evidence left)
|
||||
* an entry point the baseline does not name -> FAULT (unreviewed)
|
||||
* a code that moved 0 -> 1 (or 1 -> 2, or -> 124) -> FAULT (a repair was
|
||||
reverted, or a new defect landed, and the instrument says which path)
|
||||
* a code that moved 1 -> 0 -> FAULT (the repair is
|
||||
real; re-take the baseline so the improvement is signed off, not silent)
|
||||
Equality is the contract, so neither direction can pass by not looking.
|
||||
|
||||
WHAT IT DOES NOT GRADE
|
||||
Whether a green entry point measures the right thing. An entry point that
|
||||
prints PASS and exits 0 unconditionally is green here and always will be;
|
||||
only the tool's own --selftest can say otherwise, and that is the sweep's
|
||||
other leg, not this one. Nor does it grade content: two different failures
|
||||
with the same exit code read the same, which is why the baseline notes carry
|
||||
the first fault line each path printed when the reading was taken.
|
||||
|
||||
EXIT CODES
|
||||
0 the sweep and the baseline agree, 1 they do not (each disagreement named),
|
||||
2 the baseline could not be read or the sweep could not run.
|
||||
|
||||
USAGE
|
||||
python3 check_sweep.py --repo . --check
|
||||
python3 check_sweep.py --repo . --list
|
||||
python3 check_sweep.py --repo . --take wizard-work/volt/q143-check-sweep/check_entry_baseline.tsv
|
||||
python3 check_sweep.py --selftest
|
||||
"""
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DEFAULT_BASELINE = os.path.join(HERE, "check_entry_baseline.tsv")
|
||||
|
||||
# A CHECK entry point is a tracked file whose basename is `CHECK` or begins
|
||||
# `CHECK-`/`CHECK.`. `CHECKER`, `CHECKS.md` and `CHECKSUMS` are deliberately
|
||||
# not matched: the archive carries all three kinds of near-miss and discovery
|
||||
# that guesses is discovery that invents paths.
|
||||
ENTRY_RE = re.compile(r"^CHECK$|^CHECK[-.]")
|
||||
HEADER = "# path\trc\tnote -- the reading taken at the rev named below, one row per CHECK entry point"
|
||||
|
||||
|
||||
def repo_root_from_here():
|
||||
return os.path.dirname(os.path.dirname(os.path.dirname(HERE)))
|
||||
|
||||
|
||||
def tracked(repo):
|
||||
"""Tracked paths, or a tree walk for a fixture with no .git."""
|
||||
if os.path.isdir(os.path.join(repo, ".git")):
|
||||
try:
|
||||
out = subprocess.run(["git", "-C", repo, "ls-files"],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
if out.returncode == 0:
|
||||
return out.stdout.splitlines()
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
pass
|
||||
found = []
|
||||
for base, dirs, files in os.walk(repo):
|
||||
dirs[:] = sorted(d for d in dirs if d not in (".git", "__pycache__"))
|
||||
for name in sorted(files):
|
||||
full = os.path.join(base, name)
|
||||
found.append(os.path.relpath(full, repo))
|
||||
return found
|
||||
|
||||
|
||||
def entry_points(repo):
|
||||
return sorted(p for p in tracked(repo)
|
||||
if ENTRY_RE.match(os.path.basename(p)))
|
||||
|
||||
|
||||
def interpreter(path):
|
||||
"""The interpreter the file's own first line names; else by extension."""
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
first = fh.read(200).split(b"\n")[0].decode("utf-8", "replace")
|
||||
except OSError:
|
||||
return ["bash"]
|
||||
if first.startswith("#!"):
|
||||
words = first[2:].split()
|
||||
if words and words[0].endswith("/env") and len(words) > 1:
|
||||
return [shutil.which(words[1]) or words[1]]
|
||||
if words:
|
||||
return [shutil.which(os.path.basename(words[0])) or words[0]]
|
||||
if path.endswith(".py"):
|
||||
return [sys.executable]
|
||||
return ["bash"]
|
||||
|
||||
|
||||
FAULT_HINT = re.compile(r"^(FAIL|FAULT|STALE|VERDICT: FAIL|Traceback|NotImplemented|cannot|Error)")
|
||||
|
||||
|
||||
def run_entry(repo, rel, timeout):
|
||||
full = os.path.join(repo, rel)
|
||||
argv = interpreter(full) + [full]
|
||||
try:
|
||||
r = subprocess.run(argv, capture_output=True, text=True, timeout=timeout,
|
||||
cwd=repo, stdin=subprocess.DEVNULL)
|
||||
text = (r.stdout or "") + (r.stderr or "")
|
||||
rc = r.returncode
|
||||
except subprocess.TimeoutExpired:
|
||||
return 124, "TIMEOUT after %ss" % timeout
|
||||
except OSError as exc:
|
||||
return 125, "SPAWN %s" % exc
|
||||
lines = [l.strip() for l in text.splitlines() if l.strip()]
|
||||
note = ""
|
||||
for line in lines:
|
||||
if FAULT_HINT.match(line):
|
||||
note = line
|
||||
break
|
||||
if not note and lines:
|
||||
note = lines[-1]
|
||||
return rc, note[:160]
|
||||
|
||||
|
||||
def classify(rc):
|
||||
if rc == 0:
|
||||
return "green"
|
||||
if rc == 1:
|
||||
return "red"
|
||||
if rc == 2:
|
||||
return "ungradable"
|
||||
if rc == 124:
|
||||
return "timeout"
|
||||
return "crashed(rc=%d)" % rc
|
||||
|
||||
|
||||
def sweep(repo, timeout, only=None, jobs=4):
|
||||
"""Every entry point, run `jobs` at a time; the RESULT is sorted by path so a
|
||||
parallel sweep and a serial one read the same."""
|
||||
paths = [rel for rel in entry_points(repo) if not only or only in rel]
|
||||
if jobs < 2 or len(paths) < 2:
|
||||
rows = []
|
||||
for rel in paths:
|
||||
rc, note = run_entry(repo, rel, timeout)
|
||||
rows.append({"path": rel, "rc": rc, "state": classify(rc), "note": note})
|
||||
return rows
|
||||
rows = []
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool:
|
||||
futures = {pool.submit(run_entry, repo, rel, timeout): rel for rel in paths}
|
||||
for fut in concurrent.futures.as_completed(futures):
|
||||
rel = futures[fut]
|
||||
rc, note = fut.result()
|
||||
rows.append({"path": rel, "rc": rc, "state": classify(rc), "note": note})
|
||||
return sorted(rows, key=lambda r: r["path"])
|
||||
|
||||
|
||||
def read_baseline(path):
|
||||
"""-> (rev, [(path, rc, note)]); raises ValueError on a malformed file."""
|
||||
if not os.path.isfile(path):
|
||||
raise ValueError("no baseline at %s" % path)
|
||||
rev, rows = None, []
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
for n, raw in enumerate(fh, 1):
|
||||
line = raw.rstrip("\n")
|
||||
if not line.strip():
|
||||
continue
|
||||
if line.startswith("#"):
|
||||
m = re.search(r"rev (\S+)", line)
|
||||
if m:
|
||||
rev = m.group(1)
|
||||
continue
|
||||
parts = line.split("\t")
|
||||
if len(parts) < 2:
|
||||
raise ValueError("line %d has no tab: %r" % (n, line))
|
||||
path, rc = parts[0], parts[1].strip()
|
||||
if not rc.isdigit():
|
||||
raise ValueError("line %d rc is not an integer: %r" % (n, rc))
|
||||
rows.append((path, int(rc), parts[2] if len(parts) > 2 else ""))
|
||||
return rev, rows
|
||||
|
||||
|
||||
def grade(rows, base_rows):
|
||||
"""Two directions: paths, then codes. -> (faults, tolerances)."""
|
||||
faults = []
|
||||
live = {r["path"]: r for r in rows}
|
||||
base = {p: (rc, note) for p, rc, note in base_rows}
|
||||
for path in sorted(set(base) - set(live)):
|
||||
faults.append("MISSING %s -- baseline rc %d, no such entry point in the tree"
|
||||
% (path, base[path][0]))
|
||||
for path in sorted(set(live) - set(base)):
|
||||
faults.append("NEW %s -- rc %d, not in the baseline (%s)"
|
||||
% (path, live[path]["rc"], live[path]["state"]))
|
||||
for path in sorted(set(live) & set(base)):
|
||||
was, note = base[path]
|
||||
now = live[path]["rc"]
|
||||
if was == now:
|
||||
continue
|
||||
if was == 0:
|
||||
faults.append("REVERT %s -- rc moved 0 -> %d (%s): %s"
|
||||
% (path, now, classify(now), live[path]["note"]))
|
||||
elif now == 0:
|
||||
faults.append("IMPROVED %s -- rc moved %d -> 0, re-take the baseline"
|
||||
% (path, was))
|
||||
else:
|
||||
faults.append("MOVED %s -- rc moved %d -> %d (%s): %s"
|
||||
% (path, was, now, classify(now), live[path]["note"]))
|
||||
return faults, len(set(live) & set(base))
|
||||
|
||||
|
||||
def take(repo, path, timeout, jobs=4):
|
||||
rows = sweep(repo, timeout, jobs=jobs)
|
||||
rev = "worktree"
|
||||
if os.path.isdir(os.path.join(repo, ".git")):
|
||||
try:
|
||||
r = subprocess.run(["git", "-C", repo, "rev-parse", "--short", "HEAD"],
|
||||
capture_output=True, text=True, timeout=60)
|
||||
if r.returncode == 0:
|
||||
rev = r.stdout.strip()
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
pass
|
||||
with open(path, "w", encoding="utf-8") as fh:
|
||||
fh.write(HEADER + "\n")
|
||||
fh.write("# rev %s -- every code below was measured, not read, and a move\n"
|
||||
"# in either direction is a fault until this file is re-taken\n" % rev)
|
||||
for row in rows:
|
||||
note = " ".join(row["note"].replace("\t", " ").split())
|
||||
fh.write("%s\t%d\t%s\n" % (row["path"], row["rc"], note))
|
||||
return rows, rev
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# selftest -- fixture trees, every leg graded by the code it returns
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
FIXTURE = {
|
||||
"CHECK-green": "#!/bin/bash\necho \"VERDICT: PASS\"\nexit 0\n",
|
||||
"CHECK-red": "#!/bin/bash\necho \"FAIL: two rows disagree\"\nexit 1\n",
|
||||
"sub/CHECK-ungradable": "#!/bin/bash\necho \"cannot grade: no input\"\nexit 2\n",
|
||||
"sub/CHECK-slow": "#!/bin/bash\nsleep 30\nexit 0\n",
|
||||
"sub/CHECK-py": "#!/usr/bin/env python3\nprint('verdict: pass')\nraise SystemExit(0)\n",
|
||||
"sub/CHECKER": "#!/bin/bash\nexit 1\n",
|
||||
"notes.md": "# not an entry point\n",
|
||||
}
|
||||
|
||||
|
||||
def write_fixture(root, extra=None, drop=None):
|
||||
for rel, body in FIXTURE.items():
|
||||
if drop and rel in drop:
|
||||
continue
|
||||
full = os.path.join(root, rel)
|
||||
os.makedirs(os.path.dirname(full), exist_ok=True)
|
||||
with open(full, "w", encoding="utf-8") as fh:
|
||||
fh.write(body)
|
||||
for rel, body in (extra or {}).items():
|
||||
full = os.path.join(root, rel)
|
||||
os.makedirs(os.path.dirname(full), exist_ok=True)
|
||||
with open(full, "w", encoding="utf-8") as fh:
|
||||
fh.write(body)
|
||||
|
||||
|
||||
class Cells(object):
|
||||
def __init__(self):
|
||||
self.passed, self.failed = 0, []
|
||||
|
||||
def check(self, name, cond, detail=None):
|
||||
if cond:
|
||||
self.passed += 1
|
||||
print(" PASS %s" % name)
|
||||
else:
|
||||
self.failed.append(name)
|
||||
print(" FAIL %s %s" % (name, "" if detail is None else detail))
|
||||
|
||||
|
||||
def selftest():
|
||||
cells = Cells()
|
||||
tmp = tempfile.mkdtemp(prefix="check-sweep-")
|
||||
try:
|
||||
# --- discovery -----------------------------------------------------
|
||||
tree = os.path.join(tmp, "tree")
|
||||
write_fixture(tree)
|
||||
found = entry_points(tree)
|
||||
cells.check("discover-finds-five",
|
||||
len(found) == 5, found)
|
||||
cells.check("discover-skips-near-miss",
|
||||
"sub/CHECKER" not in found and "notes.md" not in found, found)
|
||||
cells.check("discover-sorted", found == sorted(found), found)
|
||||
|
||||
# --- classification: every code the runner can produce --------------
|
||||
rows = {r["path"]: r for r in sweep(tree, timeout=3)}
|
||||
cells.check("run-green-rc0", rows["CHECK-green"]["rc"] == 0, rows["CHECK-green"])
|
||||
cells.check("run-red-rc1", rows["CHECK-red"]["rc"] == 1, rows["CHECK-red"])
|
||||
cells.check("run-ungradable-rc2",
|
||||
rows["sub/CHECK-ungradable"]["rc"] == 2, rows["sub/CHECK-ungradable"])
|
||||
cells.check("run-timeout-rc124", rows["sub/CHECK-slow"]["rc"] == 124,
|
||||
rows["sub/CHECK-slow"])
|
||||
cells.check("run-python-shebang", rows["sub/CHECK-py"]["rc"] == 0,
|
||||
rows["sub/CHECK-py"])
|
||||
cells.check("run-note-names-fault",
|
||||
rows["CHECK-red"]["note"].startswith("FAIL:"), rows["CHECK-red"])
|
||||
|
||||
# --- the clean control ---------------------------------------------
|
||||
base = os.path.join(tmp, "base.tsv")
|
||||
take(tree, base, timeout=3)
|
||||
rev, base_rows = read_baseline(base)
|
||||
cells.check("take-writes-all-rows", len(base_rows) == 5, len(base_rows))
|
||||
faults, graded = grade(sweep(tree, timeout=3), base_rows)
|
||||
cells.check("grade-clean-control", faults == [] and graded == 5,
|
||||
(faults, graded))
|
||||
|
||||
# --- four mutants, each must be caught by name ----------------------
|
||||
(faults, _) = grade(sweep(tree, timeout=3),
|
||||
base_rows + [("CHECK-ghost", 0, "fixture")])
|
||||
cells.check("grade-missing-path", any(f.startswith("MISSING CHECK-ghost") for f in faults),
|
||||
faults)
|
||||
|
||||
extra = dict(FIXTURE)
|
||||
extra["CHECK-extra"] = "#!/bin/bash\nexit 0\n"
|
||||
tree2 = os.path.join(tmp, "tree2")
|
||||
write_fixture(tree2, extra={"CHECK-extra": "#!/bin/bash\nexit 0\n"})
|
||||
(faults, _) = grade(sweep(tree2, timeout=3), base_rows)
|
||||
cells.check("grade-new-path", any(f.startswith("NEW CHECK-extra") for f in faults),
|
||||
faults)
|
||||
cells.check("grade-new-path-count",
|
||||
sum(1 for f in faults if f.startswith("NEW")) == 1, faults)
|
||||
|
||||
tree3 = os.path.join(tmp, "tree3")
|
||||
write_fixture(tree3)
|
||||
with open(os.path.join(tree3, "CHECK-green"), "w", encoding="utf-8") as fh:
|
||||
fh.write("#!/bin/bash\necho 'FAIL: reverted'\nexit 1\n")
|
||||
(faults, _) = grade(sweep(tree3, timeout=3), base_rows)
|
||||
cells.check("grade-revert-0-to-1",
|
||||
any(f.startswith("REVERT CHECK-green") for f in faults), faults)
|
||||
|
||||
tree4 = os.path.join(tmp, "tree4")
|
||||
write_fixture(tree4)
|
||||
with open(os.path.join(tree4, "CHECK-red"), "w", encoding="utf-8") as fh:
|
||||
fh.write("#!/bin/bash\nexit 0\n")
|
||||
(faults, _) = grade(sweep(tree4, timeout=3), base_rows)
|
||||
cells.check("grade-improved-1-to-0",
|
||||
any(f.startswith("IMPROVED CHECK-red") for f in faults), faults)
|
||||
|
||||
tree5 = os.path.join(tmp, "tree5")
|
||||
write_fixture(tree5, drop=["CHECK-red"])
|
||||
(faults, _) = grade(sweep(tree5, timeout=3), base_rows)
|
||||
cells.check("grade-dropped-file-is-missing",
|
||||
any(f.startswith("MISSING CHECK-red") for f in faults), faults)
|
||||
|
||||
# --- baseline unreadable is rc 2, not a fault ------------------------
|
||||
cells.check("baseline-absent-raises",
|
||||
_raises(lambda: read_baseline(os.path.join(tmp, "nope.tsv"))), "")
|
||||
bad = os.path.join(tmp, "bad.tsv")
|
||||
with open(bad, "w", encoding="utf-8") as fh:
|
||||
fh.write("CHECK-green 0 no-tab-here\n")
|
||||
cells.check("baseline-malformed-raises", _raises(lambda: read_baseline(bad)), "")
|
||||
|
||||
# --- determinism: two takes are byte-identical ------------------------
|
||||
base2 = os.path.join(tmp, "base2.tsv")
|
||||
take(tree, base2, timeout=3)
|
||||
with open(base, "rb") as fh:
|
||||
first = fh.read()
|
||||
with open(base2, "rb") as fh:
|
||||
second = fh.read()
|
||||
cells.check("take-deterministic", first == second, "")
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
total = cells.passed + len(cells.failed)
|
||||
print("selftest: %d case(s), %d failed" % (total, len(cells.failed)))
|
||||
return 0 if not cells.failed else 1
|
||||
|
||||
|
||||
def _raises(fn):
|
||||
try:
|
||||
fn()
|
||||
except ValueError:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(
|
||||
prog="check_sweep.py",
|
||||
description="run every CHECK entry point in this tree and grade its exit "
|
||||
"code against a baseline that states the reading already reviewed")
|
||||
ap.add_argument("--repo", default=repo_root_from_here(),
|
||||
help="the tree to sweep (default: the repo holding this file)")
|
||||
ap.add_argument("--baseline", default=DEFAULT_BASELINE,
|
||||
help="the baseline TSV to grade against")
|
||||
ap.add_argument("--timeout", type=int, default=120,
|
||||
help="seconds one entry point may run before it reads rc 124")
|
||||
ap.add_argument("--only", default=None,
|
||||
help="restrict the sweep to paths containing this substring")
|
||||
ap.add_argument("--jobs", type=int, default=4,
|
||||
help="entry points to run at a time; the result is sorted, so "
|
||||
"the reading does not depend on it")
|
||||
ap.add_argument("--check", action="store_true",
|
||||
help="sweep and grade against the baseline")
|
||||
ap.add_argument("--paths", action="store_true",
|
||||
help="grade the entry-point set against the baseline without running any")
|
||||
ap.add_argument("--list", action="store_true",
|
||||
help="print the entry points discovery finds and exit")
|
||||
ap.add_argument("--take", metavar="PATH",
|
||||
help="measure the sweep and write the baseline to PATH")
|
||||
ap.add_argument("--json", metavar="PATH", default=None,
|
||||
help="write the measured rows as JSON to PATH")
|
||||
ap.add_argument("--selftest", action="store_true",
|
||||
help="run the fixture corpus and exit 0 only if every cell passes")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if args.selftest:
|
||||
return selftest()
|
||||
|
||||
if args.list:
|
||||
for rel in entry_points(args.repo):
|
||||
print(rel)
|
||||
return 0
|
||||
|
||||
if args.take:
|
||||
rows, rev = take(args.repo, args.take, args.timeout, jobs=args.jobs)
|
||||
green = sum(1 for r in rows if r["rc"] == 0)
|
||||
print("baseline: %d entry point(s), %d green, %d not, rev %s"
|
||||
% (len(rows), green, len(rows) - green, rev))
|
||||
return 0
|
||||
|
||||
if args.paths:
|
||||
try:
|
||||
rev, base_rows = read_baseline(args.baseline)
|
||||
except ValueError as exc:
|
||||
print("BASELINE UNREADABLE: %s" % exc)
|
||||
return 2
|
||||
live = set(entry_points(args.repo))
|
||||
base = {p for p, _, _ in base_rows}
|
||||
faults = ["MISSING %s -- baseline names it, the tree does not have it" % p
|
||||
for p in sorted(base - live)]
|
||||
faults += ["NEW %s -- in the tree, not in the baseline" % p
|
||||
for p in sorted(live - base)]
|
||||
for line in faults:
|
||||
print("FAIL: " + line)
|
||||
print("paths: %d entry point(s), %d baseline path(s), %d fault(s)"
|
||||
% (len(live), len(base), len(faults)))
|
||||
return 0 if not faults else 1
|
||||
|
||||
if not args.check:
|
||||
ap.error("one of --check, --paths, --list, --take, --selftest is required")
|
||||
|
||||
try:
|
||||
rev, base_rows = read_baseline(args.baseline)
|
||||
except ValueError as exc:
|
||||
print("BASELINE UNREADABLE: %s" % exc)
|
||||
return 2
|
||||
try:
|
||||
rows = sweep(args.repo, args.timeout, only=args.only, jobs=args.jobs)
|
||||
except Exception as exc: # noqa: BLE001 -- a sweep that cannot run is rc 2
|
||||
print("SWEEP COULD NOT RUN: %r" % (exc,))
|
||||
return 2
|
||||
|
||||
if args.json:
|
||||
with open(args.json, "w", encoding="utf-8") as fh:
|
||||
json.dump({"rev": rev, "rows": rows}, fh, indent=1, sort_keys=True)
|
||||
|
||||
for row in rows:
|
||||
print(" %-11s rc=%-3d %s" % (row["state"], row["rc"], row["path"]))
|
||||
faults, graded = grade(rows, base_rows)
|
||||
for line in faults:
|
||||
print("FAIL: " + line)
|
||||
green = sum(1 for r in rows if r["rc"] == 0)
|
||||
print("check-sweep: %d entry point(s), %d green, %d baseline row(s) graded (%d), "
|
||||
"%d fault(s)" % (len(rows), green, len(base_rows), graded, len(faults)))
|
||||
return 0 if not faults else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user