volt: q118 -- the retried folder read is graded, and the wake is measured without a deletion
The monitor's retry (3 attempts, 2 s apart) had no reader: CHECK-q115 grades the carry path and its harness replaces get_folder with a stub, so a repair that retried once or never would have read the same. q118_verify.py drives the deployed loop at the boundary it really crosses -- a fake curl first on PATH, scripted codes, every call timestamped -- and grades recovery, exhaustion, no-pause-on-success and the Q115 carry, with three mutants required to red the cell they target. Measured: CHECK-q118 8/8 legs rc 0; a copy with tries=1 reads 3 faults rc 1; verify 10/10 legs, gate OK.
This commit is contained in:
volt committed
1 parent
fdcbffede5
commit
fd38266d8d
4 files changed
+456
-15
No files matched your search
@@ -13762,3 +13762,19 @@ CHECK: bash wizard-work/volt/CHECK-q133
|
||||
**(4) Not claimed:** that `repo_tree.tsv` should be tracked — it is untracked residue in this worktree and the retired script read it by absolute path, which is the same host-pin defect Q69/Q106/Q109 name, not a second item; that sol's other files in `93fa377` are affected — this record grades one path; that retirement is the right disposition for every stray file under `tools/` — it is the disposition for a scratch probe that grades nothing, and writing a `--selftest` for an 8-line one-off would have manufactured a stub gate to satisfy a row, which is the fault §9 names; that the deletion alone closed the item — without leg 1 and leg 3 the absence would be indistinguishable from a loss and the c7 cell would be trusted rather than tested; that the copy under `/root/.hermes/wizard-outputs/volt/` is the durable artifact — the blob in history is, and the copy is the re-readable rendering of it; that `9d15294` was a complete tick — it landed the deletion with no record and no instrument, and this record with its verifier is the rest of that tick, appended rather than rewritten. Files edited for this item: `tools/inspect_tree.py` (deleted), `wizard-work/volt/q134_verify.py` (new), `wizard-work/volt/CHECK-q134` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted, and no counterparty repository file was edited.
|
||||
|
||||
CHECK: bash wizard-work/volt/CHECK-q134
|
||||
|
||||
**Q118 — DONE (volt, 2026-10-31, at the commit this record lands in; pipe lane by authorship, unclaimed and taken). The monitor's folder read is retired three times before it may be called a failure, and until this tick nothing graded the retry: the only harness that touches folder reads replaces `get_folder` itself, so a repair that retried once, never, or slept six seconds would have read exactly the same.**
|
||||
|
||||
**(0) Entry state, measured before any edit.** The deployed module is `/root/.hermes/scripts/troupe_pipe_monitor.py`, 389 lines, sha256 `ab0c8de6ce716493ac0d8d03cfc06b19befcb7c0d373bef3883f91957632622e`, 18,964 B, mtime 2026-10-30 20:57. `get_folder()` at `:144` carries the repair: `tries: int = 3, pause: float = 2.0`, `for attempt in range(1, tries + 1)`, `if attempt < tries: time.sleep(pause)`, and it returns `(items_or_None, attempts, last)`; the failure branch at `:350` prints `UNREADABLE <folder> attempts={tries} last={last_code} carried={n} from={taken}`. What no instrument read is those three facts as facts: `wizard-work/volt/CHECK-q115` grades the carry path and it shells out to `/root/.hermes/scripts/troupe_pipe_monitor_harness.py`, whose own docstring says it "replaces only the three boundaries that touch the outside world -- `get` (Gitea HTTP), `get_folder` (the retried folder read) and `token`"; and Q115's one forced-failure run drove its stub to `contents` 200 then 502 on *every* attempt, which measures the exhausted branch and not the recovery that branch exists for.
|
||||
|
||||
**(1) What landed.** `wizard-work/volt/q118_verify.py` (359 lines, sha256 `8483f0948154cae2a700e154710d9fe61314e3b59df7c51048a48866caca218d`) behind `wizard-work/volt/CHECK-q118` (sha256 `3e424ad4a7cf0608c5ddb64d5541288494136437106665a7955ee355957b5d90`). The module is not stubbed: it shells out to `curl`, so the instrument puts a fake `curl` first on `PATH` that answers a scripted code sequence and timestamps every call, and the copy of the module it runs differs from the deployed file only in four constants (`API`, `ENV`, `CACHE`, `LISTING_CACHE`) rewritten into a temp dir -- each rewrite asserted to match exactly once, because a rewrite that silently did nothing would grade the live forge and pass. The authorship window is a fixed non-empty commit list (`[]` on page 1 makes `main()` print `state unreadable` and return before any folder is read, which is why an empty world cannot be used), so the listing is the two fixture paths and the retry loop is the deployed one.
|
||||
|
||||
**(2) The four cells, measured at HEAD this tick.** `bash wizard-work/volt/CHECK-q118` → `q118: 8 leg(s), 8 passed, 0 fault(s)`, rc **0**, 24 s. `recover` -- 502 on attempt 1, 200 with two files on attempt 2: rc 0, both paths listed, **no** `UNREADABLE` line, exactly **2** folder calls, gap **2.08 s**. `exhaust` -- 502, 503, 504: rc 0, exactly **3** calls, gaps **[2.08, 2.08]**, span **4.17 s** (two pauses; a loop that also slept after the last attempt would read ~6 s), and the line reads exactly `UNREADABLE replies attempts=3 last=504 carried=0 from=none` plus `NO-BASELINE replies absent-lines-are-unknown-not-deleted` -- so `last=` names the *last* code, not the first, and the no-baseline case is not silent. `firstok` -- 200 on attempt 1: **1** call, no token, whole tick **0.80 s**, so a success does not pay a pause. `host-cache` -- the two deployed caches are sha256-identical before and after the run, so grading leaves no host state behind.
|
||||
|
||||
**(3) Falsification, both layers.** Three mutations of the deployed bytes, each required to red the cell it targets, all caught inside the same run: `no_retry` (`range(1, tries + 1)` → `range(1, 2)`) → `recover` faults `a transient failure was still reported as UNREADABLE`; `no_pause` (`time.sleep(pause)` → `time.sleep(0.01)`) → `exhaust` faults `gaps [0.26, 0.18] outside [1.5, 3.8]`; `last_first` (keeps the first code in `last`) → `exhaust` faults on `last=502` where `last=504` is required. And the whole-artifact control: the verifier takes an optional module path, and pointed at a scratch copy whose default is `tries: int = 1` (sha256 `cae261c957a4fc2b12eb3a393d797146b70fb5a2800c9f734301b0004b5c8f2f`) it reads `8 leg(s), 5 passed, 3 fault(s)`, rc **1**, naming `recover`, `exhaust` (`attempts=1 last=502`) and `wake_no_delete`. So the verdict is a function of the artifact, not of this host. Two instrument bugs were found and fixed before the reading above: `path_lines()` compared full `path sha` lines against bare paths, and the call counter counted the commits and per-path history calls as if they were folder reads (reading `9 folder call(s)`); both are in this ticket's first run, not in the numbers above.
|
||||
|
||||
**(4) Clause (c) reconciled, not reinterpreted.** The opening record asks that a folder failing every attempt "is still printed as unreadable and still changes the output -- failure is a wake, never a silence". Q115 later and deliberately removed the second half: the failed folder's last good listing is carried forward so the tick's *file lines* are byte-identical and the delta is empty, because those departing lines were the 417 false deletions. Both halves are now measured from one place (`wake_no_delete`: tick 1 healthy, tick 2 three 502s) -- the file lines are identical on both sides, the token line is the only difference, and the token names the carry (`carried=2`). The wake survives; the fabricated deletion does not. This record does not edit the opening text and does not claim the retry would have rescued the 2026-10-28 read (the original status code was never captured -- that is what `last=` now fixes).
|
||||
|
||||
**(5) Not claimed.** That the deployed module was edited this tick -- it was not, and its sha256 is `ab0c8de6…` before and after; the repair being graded landed on 2026-10-30. That the live tick was re-run here -- it was not (Q115 measured it: 284 lines, delta empty); this instrument grades behaviour against a transport it controls, and the deployment path is unchanged. That the retry covers every failure mode -- it covers the read the loop performs; a hung DNS lookup costs `--max-time 60` per attempt, which is a 180 s worst case the module states nowhere and this item does not change. That the two-second spacing is a tuning decision -- it is the number the opening record asks for and nothing here measures an optimum. **Observed and out of scope:** both call sites send the literal header `Authorization: token ***` (`:136`, `:158`, confirmed with `cat -A`), not the token read from `ENV`, and the file has read that way since its 2026-10-07 backup, so every API read is anonymous; the live tick reads clean that way, no reading here says whether the forge requires the credential, and no line of this item touches it -- if the pipe wants authenticated reads that is its own item. Files edited this tick: `wizard-work/volt/q118_verify.py` (new), `wizard-work/volt/CHECK-q118` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted, and no counterparty file was edited. Report: `/root/.hermes/wizard-outputs/volt/2026-10-31-q118.md`.
|
||||
|
||||
CHECK: bash wizard-work/volt/CHECK-q118
|
||||
+42
-15
@@ -4,7 +4,7 @@
|
||||
|
||||
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
|
||||
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 44 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 112 record blocks in all, 68 of them a later closure or correction for an id already opened. The same **44 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
|
||||
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 44 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 113 record blocks in all, 69 of them a later closure or correction for an id already opened. The same **44 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
|
||||
|
||||
| # | id | kind | lines | bytes | role | title | first `Status:` |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -216,7 +216,8 @@
|
||||
| 206 | Q89 | record | 13701-13716 | 7605 | cont: DONE | **Q89 — DONE (volt, 2026-10-30, at the commit this record lands in; forge lane by authorship, unclaimed and taken). The import-time-I/O detector in the three refactor harnesses was one name list … | -- |
|
||||
| 207 | Q90 | record | 13717-13730 | 5661 | cont: DONE | **Q90 — DONE (volt, 2026-10-30, at the commit this record lands in; beam lane by authorship, unclaimed and taken). `beam/tree_root.py` had no root boundary, a `portable_seed` that returned a path to … | -- |
|
||||
| 208 | Q133 | record | 13731-13751 | 9788 | record: BOLD-LEAD | **Q133 — OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; beam lane by authorship, unclaimed and taken). Two beam test cells graded a host copy of the … | -- |
|
||||
| 209 | Q134 | record | 13752-13764 | 6337 | record: BOLD-LEAD | **Q134 — OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; sol lane by authorship, unclaimed and taken). `tools/inspect_tree.py` was an 8-line one-off … | -- |
|
||||
| 209 | Q134 | record | 13752-13765 | 6338 | record: BOLD-LEAD | **Q134 — OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; sol lane by authorship, unclaimed and taken). `tools/inspect_tree.py` was an 8-line one-off … | -- |
|
||||
| 210 | Q118 | record | 13766-13780 | 7224 | cont: DONE | **Q118 — DONE (volt, 2026-10-31, at the commit this record lands in; pipe lane by authorship, unclaimed and taken). The monitor's folder read is retired three times before it may be called a failure … | -- |
|
||||
|
||||
## Ids with no `## ` heading
|
||||
|
||||
@@ -250,10 +251,10 @@ Opened as records inside another section. `line` is the record's own opening lin
|
||||
| Q112 | 13276 | 13276 | 6 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q113 | 13278 | 13278 | 3 | Q112 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q114 | 13280 | 13280 | 5 | Q113 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q115 | 13282 | 13282 | 6 | Q114 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q115 | 13282 | 13282 | 9 | Q114 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q116 | 13286 | 13286 | 11 | Q105 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q117 | 13290 | 13288 | 7 | Q96 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q118 | 13300 | 13300 | 4 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q118 | 13300 | 13300 | 5 | Q111 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q119 | 13305 | 13305 | 5 | Q118 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q120 | 13308 | 13308 | 21 | Q119 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q121 | 13342 | 13342 | 4 | Q107 | BULLET+BOLD | BOLD-LEAD |
|
||||
@@ -1723,17 +1724,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q133 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; beam lane by authorship, unclaimed and taken). Two beam test cells graded a host copy of the \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 6337,
|
||||
"end_line": 13764,
|
||||
"bytes": 6338,
|
||||
"end_line": 13765,
|
||||
"id": "Q134",
|
||||
"kind": "record",
|
||||
"line": 13752,
|
||||
"lines": 13,
|
||||
"lines": 14,
|
||||
"role": "record",
|
||||
"role_detail": "BOLD-LEAD",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q134 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; sol lane by authorship, unclaimed and taken). `tools/inspect_tree.py` was an 8-line one-off \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7224,
|
||||
"end_line": 13780,
|
||||
"id": "Q118",
|
||||
"kind": "record",
|
||||
"line": 13766,
|
||||
"lines": 15,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q118 \u2014 DONE (volt, 2026-10-31, at the commit this record lands in; pipe lane by authorship, unclaimed and taken). The monitor's folder read is retired three times before it may be called a failure \u2026"
|
||||
}
|
||||
],
|
||||
"ids_multi_section": {
|
||||
@@ -4576,17 +4590,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "**Q133 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; beam lane by authorship, unclaimed and taken). Two beam test cells graded a host copy of the \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 6337,
|
||||
"end_line": 13764,
|
||||
"bytes": 6338,
|
||||
"end_line": 13765,
|
||||
"id": "Q134",
|
||||
"kind": "record",
|
||||
"line": 13752,
|
||||
"lines": 13,
|
||||
"lines": 14,
|
||||
"role": "record",
|
||||
"role_detail": "BOLD-LEAD",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q134 \u2014 OPENED and REPAIRED in the same tick (volt, 2026-10-31, at the commit this record lands in; sol lane by authorship, unclaimed and taken). `tools/inspect_tree.py` was an 8-line one-off \u2026"
|
||||
},
|
||||
{
|
||||
"bytes": 7224,
|
||||
"end_line": 13780,
|
||||
"id": "Q118",
|
||||
"kind": "record",
|
||||
"line": 13766,
|
||||
"lines": 15,
|
||||
"role": "continuation",
|
||||
"role_detail": "DONE",
|
||||
"status": null,
|
||||
"status_line": null,
|
||||
"title": "**Q118 \u2014 DONE (volt, 2026-10-31, at the commit this record lands in; pipe lane by authorship, unclaimed and taken). The monitor's folder read is retired three times before it may be called a failure \u2026"
|
||||
}
|
||||
],
|
||||
"nested": [
|
||||
@@ -4829,7 +4856,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q115",
|
||||
"line": 13282,
|
||||
"refs": 6,
|
||||
"refs": 9,
|
||||
"section": "Q114",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4856,7 +4883,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q118",
|
||||
"line": 13300,
|
||||
"refs": 4,
|
||||
"refs": 5,
|
||||
"section": "Q111",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -4992,9 +5019,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"end_line": 8,
|
||||
"line": 1
|
||||
},
|
||||
"queue_bytes": 2238045,
|
||||
"queue_lines": 13764,
|
||||
"queue_sha256": "a4b1fdcf217c89dcc45acf74d6955a2ddd8765803567f085547564ab081bf07f",
|
||||
"queue_bytes": 2245270,
|
||||
"queue_lines": 13780,
|
||||
"queue_sha256": "1bb6ca01b12f7ee9d32828bdc32073936d1cc021379f6a5d3aac9bc8ddf40747",
|
||||
"sections": [
|
||||
{
|
||||
"bytes": 3139,
|
||||
|
||||
Executable
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Q118 gate: the pipe change-monitor retries a folder read before calling it a
|
||||
# failure -- 3 attempts, 2 s apart, the last code named, and a wake rather than
|
||||
# a fabricated deletion when every attempt fails.
|
||||
#
|
||||
# The artifact under repair is NOT in this repository -- the pipe's monitor lives
|
||||
# at /root/.hermes/scripts/troupe_pipe_monitor.py because the cron job runs it by
|
||||
# host path -- so like CHECK-q115 this gate reads a host path. That is stated,
|
||||
# not hidden (the Q106 host-pin class); moving the monitor into the tree is the
|
||||
# pipe's decision, not this item's. Everything the gate grades is the deployed
|
||||
# retry loop itself: the fake `curl` sits at the module's real boundary, so no
|
||||
# function under test is stubbed.
|
||||
set -u
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
VERIFY="$HERE/q118_verify.py"
|
||||
|
||||
if [ ! -f "$VERIFY" ]; then
|
||||
echo "FAIL: verifier missing at $VERIFY" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
out="$(python3 "$VERIFY" 2>&1)"
|
||||
rc=$?
|
||||
printf '%s\n' "$out" | grep -E '^FAIL:|^q118:' || true
|
||||
|
||||
if [ "$rc" -eq 2 ]; then
|
||||
echo "FAIL: could not grade (rc 2) -- see the line above" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "FAIL: rc=$rc -- a leg above is red" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! printf '%s\n' "$out" | grep -q '^q118: 8 leg(s), 8 passed, 0 fault(s)$'; then
|
||||
echo "FAIL: expected 8/8 legs, got: $(printf '%s\n' "$out" | tail -1)" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "q118: retry absorbs a transient read, names attempts= and last=, wakes on exhaustion -- OK"
|
||||
exit 0
|
||||
@@ -0,0 +1,359 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Q118 instrument: a folder read is retried before it may be called a failure.
|
||||
|
||||
Q118's done-means has three parts and only one of them was ever graded:
|
||||
|
||||
(a) a folder listing is retried -- 3 attempts, 2 s apart -- before it may be
|
||||
declared unreadable;
|
||||
(b) the failure line carries `attempts=` and `last=`, so the next occurrence
|
||||
is a measurement rather than an inference;
|
||||
(c) a folder that fails every attempt is still printed as unreadable and
|
||||
still changes the output (failure is a wake, never a silence).
|
||||
|
||||
Nothing in this repository reached `get_folder()`. `CHECK-q115` grades the
|
||||
carry-forward path and its harness *replaces* `get_folder` with a stub (see the
|
||||
docstring of /root/.hermes/scripts/troupe_pipe_monitor_harness.py: "replaces
|
||||
only the three boundaries ... `get` (Gitea HTTP), `get_folder` (the retried
|
||||
folder read) and `token`"), and the one forced-failure run Q115 recorded drives
|
||||
its stub to 502 on *all three* attempts -- which measures the exhausted branch
|
||||
and never the recovery it exists for.
|
||||
|
||||
So this instrument drives the deployed retry loop itself, at the boundary the
|
||||
module really crosses: it shells out to `curl`, so a fake `curl` first on PATH
|
||||
answers a scripted sequence of HTTP codes and timestamps every call. Nothing
|
||||
inside the module is stubbed or re-implemented; the copy differs from the
|
||||
deployed file only in four path/URL constants (rewritten into a temp dir) and,
|
||||
for the mutant legs, one deliberate mutation.
|
||||
|
||||
Legs, all against the deployed bytes:
|
||||
recover 502 then 200 -> rc 0, both files listed, no UNREADABLE token,
|
||||
exactly 2 calls, the gap between them ~2 s (the retry landed)
|
||||
exhaust 502/503/504 -> token names `attempts=3 last=504`, 3 calls,
|
||||
two gaps ~2 s each, total span ~4 s (two sleeps, not three)
|
||||
firstok 200 on attempt 1 -> exactly 1 call, no token, no sleep
|
||||
wake_no_delete tick 1 healthy, tick 2 exhausted -> the file lines are
|
||||
byte-identical and the token line is what differs: a wake, and
|
||||
no fabricated deletion (this is Q115's repair, measured from
|
||||
this side because Q118 (c) as written predates it)
|
||||
|
||||
Mutant legs prove the cells above are not vacuous -- each mutation must red the
|
||||
cell it targets:
|
||||
no_retry `range(1, tries + 1)` -> `range(1, 2)` reds `recover`
|
||||
no_pause `time.sleep(pause)` -> `time.sleep(0.01)` reds `exhaust` gap
|
||||
last_first `last = code...` -> keeps the first code reds `exhaust` last=
|
||||
|
||||
Exit codes follow the repository's tools: 0 clean, 1 graded fault, 2 could not
|
||||
grade (module or interpreter absent, or a constant anchor that no longer
|
||||
matches -- a rewrite that silently did nothing would grade the live host).
|
||||
|
||||
Run: python3 wizard-work/volt/q118_verify.py [module.py]
|
||||
The optional argument points the instrument at a copy of the module -- that is
|
||||
how a whole-artifact negative control is taken (a copy whose default retry count
|
||||
is 1 must red `recover` and `exhaust`).
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
DEFAULT_MODULE = "/root/.hermes/scripts/troupe_pipe_monitor.py"
|
||||
MODULE = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(DEFAULT_MODULE)
|
||||
HOST_CACHES = (
|
||||
Path("/root/.hermes/cache/troupe_last_touch.json"),
|
||||
Path("/root/.hermes/cache/troupe_folder_listing.json"),
|
||||
)
|
||||
ITEMS = [
|
||||
{"type": "file", "path": "replies/2026-10-28-to-zephyrus-1.md", "sha": "aaa111bbb222"},
|
||||
{"type": "file", "path": "replies/2026-10-28-to-zephyrus-2.md", "sha": "ccc333ddd444"},
|
||||
]
|
||||
EXPECTED_FILES = sorted(i["path"] for i in ITEMS)
|
||||
PAUSE = 2.0
|
||||
GAP_LO, GAP_HI = PAUSE * 0.75, PAUSE * 1.9 # scheduler slack, not a wide band
|
||||
|
||||
MUTATIONS = {
|
||||
"no_retry": (" for attempt in range(1, tries + 1):",
|
||||
" for attempt in range(1, 2):"),
|
||||
"no_pause": (" time.sleep(pause)",
|
||||
" time.sleep(0.01)"),
|
||||
"last_first": (' last = code.strip() or f"curl{proc.returncode}"',
|
||||
' last = last if last != "none" else '
|
||||
'(code.strip() or f"curl{proc.returncode}")'),
|
||||
}
|
||||
|
||||
FAKE_CURL = r'''#!/usr/bin/env python3
|
||||
"""Stand-in for curl, first on PATH: scripted codes, every call timestamped."""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
args = sys.argv[1:]
|
||||
url = args[-1]
|
||||
want_code = "-w" in args
|
||||
plan = json.load(open(os.environ["Q118_PLAN"]))
|
||||
state = os.environ["Q118_STATE"]
|
||||
idx = int(open(state).read().strip() or 0) if os.path.exists(state) else 0
|
||||
|
||||
|
||||
def reply(code, body):
|
||||
with open(os.environ["Q118_LOG"], "a") as fh:
|
||||
fh.write("%.6f\t%s\t%s\n" % (time.time(), code, url))
|
||||
sys.stdout.write(body + ("\n" + code if want_code else ""))
|
||||
|
||||
|
||||
if "/contents/replies" in url:
|
||||
i = min(idx, len(plan) - 1)
|
||||
code, body = plan[i]
|
||||
with open(state, "w") as fh:
|
||||
fh.write(str(idx + 1))
|
||||
reply(code, json.dumps(body) if code == "200" and body else body)
|
||||
elif "/contents/" in url:
|
||||
reply("200", "[]")
|
||||
elif "/branches/" in url:
|
||||
reply("200", json.dumps({"commit": {"sha": "deadbeefcafe0001"}}))
|
||||
elif "/commits" in url:
|
||||
reply("200", json.dumps([{
|
||||
"sha": "beef1234cafe",
|
||||
"commit": {"author": {"name": "zephyrus-sparkborn",
|
||||
"email": "z@users.noreply.github.com"}},
|
||||
"files": [],
|
||||
}]))
|
||||
else:
|
||||
reply("404", "")
|
||||
'''
|
||||
|
||||
FAULTS = []
|
||||
|
||||
|
||||
def fault(leg, detail):
|
||||
FAULTS.append(leg)
|
||||
print(f"FAIL: {leg} -- {detail}")
|
||||
|
||||
|
||||
def ok(leg, detail):
|
||||
print(f"ok {leg} -- {detail}")
|
||||
|
||||
|
||||
def digest(path):
|
||||
try:
|
||||
return hashlib.sha256(Path(path).read_bytes()).hexdigest()[:16]
|
||||
except OSError:
|
||||
return "absent"
|
||||
|
||||
|
||||
def build_module(work: Path, mutate=None) -> Path:
|
||||
"""Copy the deployed module with its four outside-world constants repointed."""
|
||||
text = MODULE.read_text(encoding="utf-8")
|
||||
if mutate:
|
||||
old, new = MUTATIONS[mutate]
|
||||
if text.count(old) != 1:
|
||||
raise RuntimeError(
|
||||
f"mutation anchor {mutate!r} matched {text.count(old)} time(s), need 1")
|
||||
text = text.replace(old, new)
|
||||
subs = (
|
||||
(r"^API = .*$", 'API = "http://127.0.0.1:9/api/v1"'),
|
||||
(r"^ENV = Path\(.*\)$", f"ENV = Path({str(work / 'env')!r})"),
|
||||
(r"^CACHE = Path\(.*\)$", f"CACHE = Path({str(work / 'attrib.json')!r})"),
|
||||
(r"^LISTING_CACHE = Path\(.*\)$",
|
||||
f"LISTING_CACHE = Path({str(work / 'listing.json')!r})"),
|
||||
)
|
||||
for pat, rep in subs:
|
||||
text, n = re.subn(pat, rep, text, count=1, flags=re.M)
|
||||
if n != 1:
|
||||
raise RuntimeError(f"constant anchor {pat!r} matched {n} time(s), need 1")
|
||||
dest = work / "mon.py"
|
||||
dest.write_text(text, encoding="utf-8")
|
||||
return dest
|
||||
|
||||
|
||||
def run_ticks(plans, mutate=None):
|
||||
"""Run the module once per plan; return (rcs, outputs, calls, seconds)."""
|
||||
work = Path(tempfile.mkdtemp(prefix="q118-"))
|
||||
(work / "bin").mkdir()
|
||||
fake = work / "bin" / "curl"
|
||||
fake.write_text(FAKE_CURL, encoding="utf-8")
|
||||
fake.chmod(0o755)
|
||||
(work / "env").write_text("GITEA_TOKEN=q118-fake\n", encoding="utf-8")
|
||||
module = build_module(work, mutate=mutate)
|
||||
|
||||
rcs, outs, calls = [], [], []
|
||||
started = time.time()
|
||||
for n, plan in enumerate(plans):
|
||||
(work / f"plan{n}.json").write_text(
|
||||
json.dumps([list(pair) for pair in plan]), encoding="utf-8")
|
||||
log = work / f"log{n}.txt"
|
||||
log.unlink(missing_ok=True)
|
||||
state = work / f"state{n}.txt"
|
||||
state.unlink(missing_ok=True)
|
||||
env = dict(os.environ)
|
||||
env["PATH"] = str(work / "bin") + ":" + env.get("PATH", "/usr/bin:/bin")
|
||||
env["Q118_PLAN"] = str(work / f"plan{n}.json")
|
||||
env["Q118_LOG"] = str(log)
|
||||
env["Q118_STATE"] = str(state)
|
||||
proc = subprocess.run([sys.executable, str(module)], capture_output=True,
|
||||
text=True, env=env, timeout=120)
|
||||
rcs.append(proc.returncode)
|
||||
outs.append(proc.stdout + proc.stderr)
|
||||
rows = []
|
||||
if log.exists():
|
||||
for line in log.read_text(encoding="utf-8").splitlines():
|
||||
ts, code, url = line.split("\t", 2)
|
||||
rows.append((float(ts), code, url))
|
||||
calls.append(rows)
|
||||
return rcs, outs, calls, time.time() - started
|
||||
|
||||
|
||||
def path_lines(out):
|
||||
keep = []
|
||||
for line in out.splitlines():
|
||||
if line.startswith(("branch-head ", "head ", "UNREADABLE ", "NO-BASELINE ")):
|
||||
continue
|
||||
if line.startswith("state unreadable"):
|
||||
continue
|
||||
if not line.strip():
|
||||
continue
|
||||
keep.append(line.split(" ", 1)[0])
|
||||
return sorted(keep)
|
||||
|
||||
|
||||
def folder_calls(rows):
|
||||
"""Only the calls the retry loop under test makes (the folder read itself)."""
|
||||
return [r for r in rows if "/contents/replies" in r[2]]
|
||||
|
||||
|
||||
def gaps(rows):
|
||||
return [b[0] - a[0] for a, b in zip(rows, rows[1:])]
|
||||
|
||||
|
||||
def case_recover(mutate=None):
|
||||
rcs, outs, calls, _ = run_ticks([(("502", ""), ("200", ITEMS))], mutate=mutate)
|
||||
out, rows = outs[0], folder_calls(calls[0])
|
||||
listed = path_lines(out)
|
||||
if rcs[0] != 0:
|
||||
return False, f"rc {rcs[0]} != 0"
|
||||
if "UNREADABLE" in out:
|
||||
return False, "a transient failure was still reported as UNREADABLE"
|
||||
if listed != EXPECTED_FILES:
|
||||
return False, f"listing {listed} != {EXPECTED_FILES}"
|
||||
if len(rows) != 2:
|
||||
return False, f"{len(rows)} folder call(s), expected 2 (one retry)"
|
||||
g = gaps(rows)[0]
|
||||
if not GAP_LO <= g <= GAP_HI:
|
||||
return False, f"retry gap {g:.2f}s outside [{GAP_LO}, {GAP_HI}]"
|
||||
return True, f"recovered on attempt 2, gap {g:.2f}s, {len(listed)} files listed"
|
||||
|
||||
|
||||
def case_exhaust(mutate=None):
|
||||
plan = (("502", ""), ("503", ""), ("504", ""))
|
||||
rcs, outs, calls, _ = run_ticks([plan], mutate=mutate)
|
||||
out, rows = outs[0], folder_calls(calls[0])
|
||||
if rcs[0] != 0:
|
||||
return False, f"rc {rcs[0]} != 0"
|
||||
want = "UNREADABLE replies attempts=3 last=504 carried=0 from=none"
|
||||
if want not in out:
|
||||
got = [l for l in out.splitlines() if l.startswith("UNREADABLE")] or ["<none>"]
|
||||
return False, f"token {got[0]!r} != {want!r}"
|
||||
if "NO-BASELINE replies" not in out:
|
||||
return False, "exhausted read with no baseline did not say NO-BASELINE"
|
||||
if len(rows) != 3:
|
||||
return False, f"{len(rows)} folder call(s), expected 3"
|
||||
g = gaps(rows)
|
||||
if len(g) != 2 or not all(GAP_LO <= x <= GAP_HI for x in g):
|
||||
return False, f"gaps {[round(x, 2) for x in g]} outside [{GAP_LO}, {GAP_HI}]"
|
||||
span = rows[-1][0] - rows[0][0]
|
||||
if not 2 * GAP_LO <= span <= 2 * PAUSE * 1.9:
|
||||
return False, f"span {span:.2f}s is not two pauses (three sleeps would be ~6s)"
|
||||
return True, (f"attempts=3 last=504, gaps {[round(x, 2) for x in g]}, "
|
||||
f"span {span:.2f}s, wake token present")
|
||||
|
||||
|
||||
def case_firstok(mutate=None):
|
||||
rcs, outs, calls, secs = run_ticks([(("200", ITEMS),)], mutate=mutate)
|
||||
out, rows = outs[0], folder_calls(calls[0])
|
||||
if rcs[0] != 0:
|
||||
return False, f"rc {rcs[0]} != 0"
|
||||
if "UNREADABLE" in out or "NO-BASELINE" in out:
|
||||
return False, "a healthy read printed a failure token"
|
||||
if path_lines(out) != EXPECTED_FILES:
|
||||
return False, f"listing {path_lines(out)} != {EXPECTED_FILES}"
|
||||
if len(rows) != 1:
|
||||
return False, f"{len(rows)} folder call(s), expected 1 (no retry when it lands)"
|
||||
if secs > 1.5:
|
||||
return False, f"one-attempt tick took {secs:.2f}s -- a pause ran on a success"
|
||||
return True, f"1 call, {len(path_lines(out))} files, {secs:.2f}s, no token"
|
||||
|
||||
|
||||
def case_wake_no_delete(mutate=None):
|
||||
rcs, outs, calls, _ = run_ticks(
|
||||
[(("200", ITEMS),), (("502", ""), ("502", ""), ("502", ""))], mutate=mutate)
|
||||
if rcs != [0, 0]:
|
||||
return False, f"rcs {rcs} != [0, 0]"
|
||||
healthy, failed = outs
|
||||
if path_lines(healthy) != EXPECTED_FILES:
|
||||
return False, f"healthy tick listed {path_lines(healthy)}"
|
||||
if path_lines(failed) != EXPECTED_FILES:
|
||||
return False, (f"failure tick listed {path_lines(failed)} -- the carry did "
|
||||
"not hold, so the delta reads as deletions")
|
||||
if "UNREADABLE replies attempts=3 last=502 carried=2" not in failed:
|
||||
return False, "failure tick did not name the carry"
|
||||
if healthy == failed:
|
||||
return False, "the ticks are byte-identical -- the failure is a silence"
|
||||
return True, "file lines identical across the failure, token line is the difference"
|
||||
|
||||
|
||||
DEPLOYED_CASES = (
|
||||
("recover", case_recover),
|
||||
("exhaust", case_exhaust),
|
||||
("firstok", case_firstok),
|
||||
("wake_no_delete", case_wake_no_delete),
|
||||
)
|
||||
MUTANT_CASES = (
|
||||
("no_retry", "recover", case_recover),
|
||||
("no_pause", "exhaust", case_exhaust),
|
||||
("last_first", "exhaust", case_exhaust),
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not MODULE.is_file():
|
||||
print(f"FAIL: could not grade -- module missing at {MODULE}", file=sys.stderr)
|
||||
return 2
|
||||
before = {str(p): digest(p) for p in HOST_CACHES}
|
||||
passed = 0
|
||||
try:
|
||||
for name, fn in DEPLOYED_CASES:
|
||||
good, detail = fn()
|
||||
(ok if good else fault)(name, detail)
|
||||
passed += 1 if good else 0
|
||||
|
||||
for mutation, target, fn in MUTANT_CASES:
|
||||
good, detail = fn(mutate=mutation)
|
||||
if good:
|
||||
fault(f"mutant:{mutation}", f"the {target} cell stayed green -- it does "
|
||||
f"not grade the retry it claims to")
|
||||
else:
|
||||
ok(f"mutant:{mutation}", f"caught by {target}: {detail}")
|
||||
passed += 1
|
||||
except RuntimeError as exc:
|
||||
print(f"FAIL: could not grade -- {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
after = {str(p): digest(p) for p in HOST_CACHES}
|
||||
if before != after:
|
||||
fault("host-cache", f"a graded run wrote host state: {before} -> {after}")
|
||||
else:
|
||||
ok("host-cache", "deployed caches byte-identical after the run")
|
||||
passed += 1
|
||||
|
||||
total = len(DEPLOYED_CASES) + len(MUTANT_CASES) + 1
|
||||
print(f"q118: {total} leg(s), {passed} passed, {len(FAULTS)} fault(s)")
|
||||
return 0 if not FAULTS else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in new issue
Block a user