tools (volt): doc_census_check -- the claim separator was a class, not an adjacency
The full census at 4ce1aed (the run my 2026-10-18 check-in owed) reported two
STALE claims against checkin_check.py:59 -- "read as UTF-8. Never 1, never a
traceback." -- where the 8 is the tail of UTF-8 and the 1 is an exit code.
Both are false defect claims.
Measured over all 12 committed docstrings: 140 claims read, 95 real. The rest
were welded hex (83acffe0 -> 83 acffe0, 11), ordinals/sentence ends (1.
INTERNAL, 8. Never, 5), compound adjectives (120-char, 5), list markers (0
ran, 4) and trailing commas (1, which -> claim 1 of unit which, 8).
Three rules: sep is one space or a closing delimiter + space; num is
[0-9]+(?:,[0-9]{3})*, so a trailing comma is punctuation; ordinal_at() refuses
a line-leading number followed by 2+ spaces. Six cells: 35 -> 41.
Same revision, same docstrings, after: 145 -> 105 verdict rows, STALE 2 -> 0,
NOT_PRINTED 87 -> 52, and the 26 MATCHES are the same 26 rows (lost: [],
gained: []) -- no adjudicated verification was dropped.
selftest 41/41 rc 0; selftest_sweep 12 tools 12 OK 0 fatal; read-refusal 17/17;
malformed-input 15/15.
Named, not fixed: doc_census_check returns 0 with STALE > 0 and has no --fail,
so a gate cannot fail on the class it exists to catch.
This commit is contained in:
volt committed
1 parent
4ce1aedb48
commit
ac0e211ad1
2 files changed
+165
-12
No files matched your search
@@ -234,3 +234,109 @@ only thing I need from Alexander). Nothing new.
|
||||
Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-18-malformed-axis/`
|
||||
(`decision.md`, `selftest.txt`, `sweep-head.txt`, `head-copies.txt`,
|
||||
`ab-461eede.txt`, `discrimination.txt`).
|
||||
|
||||
## Verdict, appended 04:55Z — the carried `doc_census_check` full run, and the two false defects it was reporting
|
||||
|
||||
Reply 20 carried one item on my side: the full census my check-in promised to
|
||||
append had not landed. It is run, at `4ce1aed`, and it found a defect in the
|
||||
instrument that was running it.
|
||||
|
||||
**The run.** 12 tracked tools, 145 verdict rows, 4m47s, `rc=0`:
|
||||
|
||||
--- verdict totals (unit: claims) ---
|
||||
AMBIGUOUS 11
|
||||
MATCHES 26
|
||||
NOT_PRINTED 87
|
||||
REV_UNCHECKED 8
|
||||
SELF_NOT_RUN 11
|
||||
STALE 2
|
||||
|
||||
**Both STALE rows are false, and both are in one sentence.** The single-tool
|
||||
view is the indictment:
|
||||
|
||||
$ python3 tools/doc_census_check.py --repo . --tools tools/checkin_check.py
|
||||
tools/checkin_check.py: 0 INTERNAL SPLIT, 2 STALE claim(s), 0 leg failure(s), 13 claims checked
|
||||
STALE tools/checkin_check.py:59 Never claimed 8 at HEAD -- printed 2 [selftest]
|
||||
STALE tools/checkin_check.py:59 never claimed 1 at HEAD -- printed 2 [selftest]
|
||||
|
||||
Line 59 is `read as UTF-8. Never 1, never a traceback.` — an exit-code sentence.
|
||||
The `8` is the tail of `UTF-8`, paired across the sentence's period with the
|
||||
word `Never`; the `1` is the code, paired with `never` across a comma. Neither
|
||||
is a count of a unit. The oracle then found some two-count elsewhere in the
|
||||
selftest output and graded both STALE: two defect claims against a docstring
|
||||
that is correct, which is the failure mode this archive grades other lanes for.
|
||||
|
||||
**The cause is the separator class, measured on the whole corpus, not reasoned
|
||||
from the two rows.** The same 12 committed docstrings at `HEAD` were read by the
|
||||
old parser and the new one (`vocab-before.txt`, `vocab-after.txt`, same code,
|
||||
`git show` corpus):
|
||||
|
||||
before: 140 claims after: 104 claims
|
||||
sep=' ' 95 sep=' ' 87
|
||||
sep='' 11 sep='(unit:value form)' 9
|
||||
sep='(unit:value form)' 9 sep='` ' 6
|
||||
sep='` ' 6 sep=' ' 1
|
||||
sep='-' 5 sep=') ' 1
|
||||
sep='. ' 5
|
||||
sep=' ' 4
|
||||
sep='; ' 2 sep=' " ' 1 sep='+ ' 1 sep=') ' 1
|
||||
|
||||
Only 95 of 140 were a count and its unit. The rest: a hex run welded to its
|
||||
letters (`83acffe0` -> 83 `acffe0`, 11), an ordinal or a sentence end (`1.
|
||||
INTERNAL`, `0. Usage`, `8. Never`, 5), a compound adjective (`120-char`,
|
||||
`1-based`, 5), an aligned list marker (`0 ran`, 4), punctuation spills (5).
|
||||
|
||||
**Fix — three rules, six cells.** `CLAIM.sep` is now one space, or a closing
|
||||
delimiter followed by one space. `num` is `[0-9]+(?:,[0-9]{3})*`, so a trailing
|
||||
comma is punctuation and not a thousands separator — the old class read
|
||||
`1, which` as claim 1 of unit `which` and `680, refs` as claim 680, which is 8
|
||||
of the 40 removed verdict rows on its own. `ordinal_at()` refuses a line-leading number
|
||||
followed by 2+ spaces, the exit-code block's `0 ran` / `1 ran` / `2 could`.
|
||||
Six cells pin them (welded, ordinal, list marker, sentence end, compound
|
||||
adjective, and a positive control that `471 UNRESOLVED` / `1,612 citations`
|
||||
still read): 35 -> 41.
|
||||
|
||||
**After, at the same revision and over the same docstrings:**
|
||||
|
||||
$ python3 tools/doc_census_check.py --repo . --rev 4ce1aed
|
||||
--- verdict totals (unit: claims) ---
|
||||
AMBIGUOUS 11
|
||||
MATCHES 26
|
||||
NOT_PRINTED 52
|
||||
REV_UNCHECKED 8
|
||||
SELF_NOT_RUN 8
|
||||
|
||||
145 -> 105 rows; STALE 2 -> 0; NOT_PRINTED 87 -> 52; SELF_NOT_RUN 11 -> 8
|
||||
|
||||
**The invariant that makes this a repair and not a blindfold:** the 26 MATCHES
|
||||
are the same 26 rows, by tool, unit, value and verdict —
|
||||
|
||||
$ python3 scratch/diff_census.py census-head.json census-head-tightened.json
|
||||
MATCHES identical: True | lost: [] | gained: []
|
||||
|
||||
so the tightening dropped no adjudicated verification; it dropped only rows that
|
||||
were never a count and a unit. `MATCHES` unchanged is the whole evidence claim.
|
||||
|
||||
**Gates, re-run after the change:** `--selftest` 41 cells, 41 passed, 0 failed,
|
||||
rc 0; `tools/selftest_sweep.py` 12 tools, 12 OK, 0 fatal (`doc_census_check.py
|
||||
41 cells` claimed and printed), read-refusal 17/17, malformed-input 15/15, rc 0.
|
||||
The sweep's advertised-count gate caught nothing because the usage line was
|
||||
updated in the same commit, which is the point of it.
|
||||
|
||||
**What the tightening does not reach, stated rather than implied.** Six claims
|
||||
behind a backticked token (`` `-07` suffixes ``), one aligned mid-line
|
||||
(`WORK-QUEUE.md:8003 cites`) and one `120) characters` are still read; none is
|
||||
in a unit any tool prints, so all are NOT_PRINTED and none can become a STALE.
|
||||
`NOT_PRINTED` at 52 remains the bulk of the verdict rows, which is this tool's
|
||||
stated gap (§"WHAT THIS DOES NOT CHECK"), unchanged in kind.
|
||||
|
||||
**A contract gap this run exposed, named not fixed.** `doc_census_check.py`
|
||||
exits 0 while reporting STALE > 0 (measured above: rc 0 with two STALE rows);
|
||||
it has no `--fail`, where `checkin_check.py` does. A gate cannot fail on the
|
||||
class this tool exists to catch without parsing its text. That is the next
|
||||
increment, not something to fold into this one silently.
|
||||
|
||||
Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-18-doc-census/`
|
||||
(`census-head.txt`/`.json` pre-fix, `census-head-tightened.txt`/`.json`
|
||||
post-fix, `diff.txt` row-by-row, `vocab-before.txt`/`vocab-after.txt`,
|
||||
`sweep-after.txt`).
|
||||
+59
-12
@@ -102,15 +102,16 @@ SCOPE, STATED RATHER THAN IMPLIED
|
||||
|
||||
WHAT THIS DOES NOT CHECK, and the measured size of the gap. A claim is only
|
||||
adjudicated when the unit it names is a unit the tool PRINTS. Over the three
|
||||
scanners at d355395 the run read ninety-two claims: twelve were verified
|
||||
against the run they point at, eight were in units the tool prints with two
|
||||
values (SECTION_SPLIT_BY_TOOL -- no oracle), and seventy-one were in units the
|
||||
tool never prints at all. So the oracle's reach is narrow and its precision is
|
||||
the reason: it reported exactly one bad claim on that corpus, and that one was
|
||||
the returned defect. A wider rule was tried first and falsified by its own
|
||||
first run -- num_agreement_check.py flagged a hundred and seventeen
|
||||
restatements across the archive, too coarse to be read -- so the narrow rule
|
||||
is the one that ships. Two claim forms are read (`<value> <unit>` and
|
||||
scanners at d355395 the run read ninety-two claims (with the pre-2026-10-18
|
||||
separator rule, which read punctuation and list markers as claims): twelve
|
||||
were verified against the run they point at, eight were in units the tool
|
||||
prints with two values (SECTION_SPLIT_BY_TOOL -- no oracle), and seventy-one
|
||||
were in units the tool never prints at all. So the oracle's reach is narrow
|
||||
and its precision is the reason: it reported exactly one bad claim on that
|
||||
corpus, and that one was the returned defect. A wider rule was tried first and
|
||||
falsified by its own first run -- num_agreement_check.py flagged a hundred and
|
||||
seventeen restatements across the archive, too coarse to be read -- so the
|
||||
narrow rule is the one that ships. Two claim forms are read (`<value> <unit>` and
|
||||
`<unit>: <value>`); a claim written in neither form is not seen, and that is a
|
||||
stated gap, not a silent one.
|
||||
|
||||
@@ -123,7 +124,7 @@ SCOPE, STATED RATHER THAN IMPLIED
|
||||
|
||||
USAGE
|
||||
python3 doc_census_check.py [--repo DIR] [--rev HEAD] [--tools NAME...]
|
||||
[--json OUT] [--no-run] [--selftest] # selftest: 35 cells, one per claim class
|
||||
[--json OUT] [--no-run] [--selftest] # selftest: 41 cells, one per claim class
|
||||
"""
|
||||
import argparse
|
||||
import ast
|
||||
@@ -142,11 +143,25 @@ DEFAULT_REPO = "/root/troupe-standup"
|
||||
# `<unit> [: =] <value>` -- "ACS = 72", "overlap = 20" (markdown bold allowed)
|
||||
# A bare number is not a claim. Deliberately not "any word": `471 cannot be
|
||||
# resolved` is prose, and a checker that flags prose is a checker nobody reads.
|
||||
#
|
||||
# THE SEPARATOR IS SPACE, OR A CLOSING DELIMITER AND A SPACE -- tightened
|
||||
# 2026-10-18 after the census was read off every tool at HEAD (140 claims) and
|
||||
# the separator vocabulary was printed: `''` (11 claims, a hex run read as
|
||||
# digits+word: `83acffe0` -> 83 `acffe0`), `'. '` (5, an ordinal or a sentence
|
||||
# end: `1. INTERNAL`, `615. Cite`), `'-'` (5, a compound adjective: `120-char`,
|
||||
# `1-based`), `' '` (4, an aligned list marker: `0 ran`), and the punctuation
|
||||
# spills `'; '` `' "'` `'+ '` `') '`. Only 95 of the 140 were a count and its
|
||||
# unit. The loose class fired two STALE verdicts in the full run at 4ce1aed
|
||||
# against `checkin_check.py:59` -- "Never 1, never a traceback", where the digit
|
||||
# is a code, not a census -- which is a false defect claim of exactly the kind
|
||||
# this archive grades lanes for. A hex run is welded to its letters and has no
|
||||
# separator; an ordinal's separator is a period; a compound adjective's is a
|
||||
# hyphen. None of the three is `<value> <unit>`.
|
||||
CLAIM = re.compile(
|
||||
r"(?<![0-9A-Za-z_.])(?P<num>[0-9][0-9,]*)(?P<sep>[^A-Za-z0-9\n]{0,2})"
|
||||
r"(?<![0-9A-Za-z_.])(?P<num>[0-9]+(?:,[0-9]{3})*)(?P<sep>\s{1,2}|[)\]}\"'`*]\s)"
|
||||
r"(?P<unit>[A-Za-z_][A-Za-z0-9_]*|[A-Z][A-Z_]{2,})\b")
|
||||
CLAIM_EQ = re.compile(
|
||||
r"(?P<unit>[A-Za-z_][A-Za-z0-9_]*)\s*[:=]\s*\**\s*(?P<num>[0-9][0-9,]*)")
|
||||
r"(?P<unit>[A-Za-z_][A-Za-z0-9_]*)\s*[:=]\s*\**\s*(?P<num>[0-9]+(?:,[0-9]{3})*)")
|
||||
REV_LONG = re.compile(r"\b[0-9a-f]{7,40}\b")
|
||||
|
||||
# A hex run introduced as a DIGEST is not a revision, and reading it as one costs
|
||||
@@ -321,6 +336,18 @@ def parse_printed(stdout):
|
||||
return {k: dict(v) for k, v in units.items()}, clone
|
||||
|
||||
|
||||
def ordinal_at(raw, pos):
|
||||
"""True when `pos` is the number that OPENS this line and 2+ spaces follow.
|
||||
|
||||
That is a list marker -- the exit-code block's `0 ran; nothing to report`,
|
||||
`1 ran, and graded ...`, `2 could not grade` -- and the word after it is
|
||||
prose, not a unit. The same shape mid-line (`WORK-QUEUE.md:8003 cites`) is
|
||||
a citation's line number, not a count, and is excluded by the same rule.
|
||||
"""
|
||||
m = re.match(r"[ \t]*([0-9][0-9,]*)([ \t]{2,})", raw)
|
||||
return bool(m) and m.start(1) == pos
|
||||
|
||||
|
||||
def claims_in(blk):
|
||||
"""[(line, value, unit, token, offset)] for one block.
|
||||
|
||||
@@ -336,6 +363,8 @@ def claims_in(blk):
|
||||
u = m.group("unit")
|
||||
if len(u) < 3 or u.lower() in STOPWORDS or re.fullmatch(r"[qQ]\d+", u):
|
||||
continue
|
||||
if ordinal_at(raw, m.start()):
|
||||
continue
|
||||
found[m.start()] = (int(m.group("num").replace(",", "")), u, m.group(0))
|
||||
for m in CLAIM_EQ.finditer(raw):
|
||||
u = m.group("unit")
|
||||
@@ -486,6 +515,24 @@ def _fixture_checks():
|
||||
[("UNRESOLVED", 471), ("UNRESOLVED", 441)])
|
||||
check("claims: `471 cannot be resolved` is not a claim",
|
||||
any(u in ("cannot", "resolved") for u, _ in got), False)
|
||||
# the tightened separator, one cell per form it now refuses, plus the
|
||||
# positive control -- a rule that refuses everything proves nothing
|
||||
def claims_of(text):
|
||||
return [(u, v) for _ln, v, u, _t, _p in claims_in([(1, text)])]
|
||||
|
||||
check("claims: a welded digit-and-letter run is not a claim",
|
||||
claims_of("sha 83acffe0 / rev 9188dd3 at HEAD"), [])
|
||||
check("claims: an ordinal is not a claim",
|
||||
claims_of("1. INTERNAL SPLIT -- no oracle, no run"), [])
|
||||
check("claims: a list marker is not a claim",
|
||||
claims_of("0 ran; nothing to report"), [])
|
||||
check("claims: a sentence end is not a claim",
|
||||
claims_of("read as UTF-8. Never 1, never a traceback"), [])
|
||||
check("claims: a compound adjective is not a claim",
|
||||
claims_of("a 120-char window and a 1-based index"), [])
|
||||
check("claims: the tightening keeps a count and its unit",
|
||||
claims_of("471 UNRESOLVED and 1,612 citations"),
|
||||
[("UNRESOLVED", 471), ("citations", 1612)])
|
||||
check("claims: the block's rev is read", "babdadc" in revs_seen, True)
|
||||
# the digest rule, one signal per cell, and the negative control between
|
||||
# them -- the text of each is the text of the defect, measured at HEAD
|
||||
|
||||
Reference in new issue
Block a user