Fuzz-probe DRY refactor verification

Work

Refactored _fuzz_probe.py — a 64-line adversarial fuzz probe that sweeps 383 chunker cases across five input shapes (tiny sentence groups, hard-split longs, mixed, no-after-space punctuation runs, whitespace variants) with a per-chunk token bounds check.

Problem (code smell)

The original defined a roundtrip() helper (lines 7-11) and never called it. The main loop re-implemented the same chunk_text -> reconstruct -> compare sequence inline, so the helper was dead code. The bounds check was also inline in the main loop with no abstraction. No if __name__ == "__main__" guard.

Refactor

Repurposed the dead roundtrip() into _check_roundtrip(text, mt, ot) and actually called it from the loop — single source of truth for the chunk/reconstruct/compare sequence.
Extracted the inline per-chunk bounds check into a pure generator _check_bounds(cs, mt) yielding (index, est, body) for offending chunks.
Replaced scattered module-global counters (fails, total, bounds_fail) with a single Stats dataclass.
Extracted case generation into _gen_cases().
Added if __name__ == "__main__" guard.

Evidence (measured)

Both runs executed against the live chunker at /root/.hermes/wizard-outputs/forge/chunker.py.

$ python3 /root/.hermes/profiles/forge/_fuzz_probe.py        (original, 64 lines)
total=383 roundtrip_fails=0 bounds_failures=0
RESULT: ALL PASS
EXIT=0

$ python3 _fuzz_probe_refactored.py                            (refactored, 155 lines)
total=383 roundtrip_fails=0 bounds_failures=0
RESULT: ALL PASS
EXIT=0

Byte-for-byte output equivalence (both captured into committed artifacts):

$ diff baseline_fuzz_output.txt refactored_fuzz_output.txt
$ # no output — files are identical

py_compile:

$ python3 -m py_compile _fuzz_probe_refactored.py
$ # no output — compiles clean

Negative control (probe is discriminating)

A green-only probe grades nothing. _fuzz_probe_negctrl.py monkey-patches chunker.chunk_text with a truncating mutant — for all-a inputs only, the last character of the final chunk is dropped — then runs the refactored probe against the mutant. The probe must detect the injection:

$ python3 _fuzz_probe_negctrl.py
total=383 roundtrip_fails=105 bounds_failures=0
RESULT: HAS FAILURES

105 of 383 cases fail — every "hard-mt{mt}-ot{ot}" case whose text is "a"*500, which is 1+2+...+14 = 105 (triangular sum over mt=2..15). The original probe catches the same 105 against the same mutant, so the refactor preserves detection case-for-case, not just the totals. The negctrl exits 0 = probe caught the bug; 1 = probe missed a real truncation (control failed).

Cross-check — all other chunker probes still pass with the same chunker:

$ python3 _edge_sweep_refactored.py
total=600 roundtrip_fails=0 bounds_fails=0
RESULT: ALL PASS
EXIT=0

$ python3 _test_roundtrip.py
=== Results: 34 passed, 0 failed ===
EXIT=0

$ python3 _fuzz_adversarial.py
trials=2400 fails=0
PASS: lossless round-trip across all adversarial shapes.
EXIT=0

Metrics

metric	original	refactored
lines	64	155 (incl. docstrings)
function defs	1 (dead)	4 (all used)
dead code	1 unused fn	0
total cases	383	383
roundtrip_fails	0	0
bounds_failures	0	0
exit code	0	0

Verdict

Refactoring is behavior-preserving and measurably equivalent: 383 cases, identical output, exit 0. The dead roundtrip() helper is now wired in, the bounds check is a pure generator, module-global counters are a Stats dataclass, and all 2400 adversarial trials across the wider probe suite still pass with no regressions. The negative control proves the probe is discriminating — it catches a truncating-chunker mutant on 105/383 cases.

3.9 KiB Raw Permalink Blame History