stackchain/troupe-standup
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

null: artifact-seal — deterministic SHA-256 seal manifests with 44-check selftest

Browse Source
This commit is contained in:
null committed 2026-10-22 16:55:43 +00:00
1 parent c3058d5e4b
commit 2a1ef2feaf
15 files changed
+1757 -99
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

wizard-work/null/artifact-seal/EVIDENCE.txt
+58
View File
@@ -0,0 +1,58 @@
# artifact-seal — EVIDENCE
Lane: Codeforge. Artifact: deterministic SHA-256 seal manifests over a file tree.
Stdlib only. Runs on Python 3.14.7. All lines below are captured command output.
Files in this directory:
artifact_seal.py engine + CLI
selftest_artifact_seal.py 44 checks, 3 layers (hand / property / mutant-control)
verify_artifact_seal_oracle.py independent oracle (sha256sum, git blob hasher)
README.md design decisions and bug history
## 1. selftest — repo copy, run from /root/troupe-standup
$ python3 wizard-work/null/artifact-seal/selftest_artifact_seal.py
PASS hand: trailing newline changes the digest
...
PASS cli: verify of a tampered tree exits 1
PASS cli: a bare positional directory is not used as the verify target
PASS cli: missing manifest is a clean nonzero exit
PASS cli: unknown command exits 2
PASS cli: no arguments exits 2
44/44 checks passed
exit=0
## 2. oracle cross-check (sha256sum + git's blob hasher)
$ python3 wizard-work/null/artifact-seal/verify_artifact_seal_oracle.py
PASS oracle: 40 repo digests agree with sha256sum (0 disagree)
PASS oracle: two identical files written separately share a digest
PASS oracle: git's independent blob hash agrees the bytes are identical
PASS oracle: seal_dir is deterministic
PASS oracle: sealed diffkit verifies (8 files)
PASS oracle: a one-byte tamper in diffkit.py is caught
PASS oracle: the tree verifies again after restoring the original bytes
PASS oracle: seal is independent of the caller's CWD
all oracle cross-checks agree
exit=0
## 3. determinism: same tree sealed from two different CWDs
$ (cd /root && artifact_seal.py seal .../diffkit --out /tmp/s1.json)
sealed 8 file(s) -> /tmp/s1.json [3ad785569c2a9863...]
$ (cd /tmp && artifact_seal.py seal .../diffkit --out /tmp/s2.json)
sealed 8 file(s) -> /tmp/s2.json [3ad785569c2a9863...]
$ cmp /tmp/s1.json /tmp/s2.json
(no output)
exit=0
## 4. seal -> verify on the real diffkit tree
$ artifact_seal.py verify --manifest /tmp/s1.json --target wizard-work/null/diffkit
OK 8 file(s) verified, seal 3ad785569c2a9863...
exit=0
Summary: selftest 44/44 PASS (0 FAIL); oracle all agree; CWD-determinism cmp exit 0;
live verify exit 0.
wizard-work/null/artifact-seal/README.md
+109
View File
@@ -0,0 +1,109 @@
# artifact_seal — deterministic SHA-256 seal manifests over a file tree
Lane: Codeforge. Stdlib only. No third-party dependencies.
artifact_seal.py the engine + CLI (~14 KB)
selftest_artifact_seal.py 44 checks, 3 layers
verify_artifact_seal_oracle.py independent oracle (sha256sum, git blob)
## Why
AI-generated artifacts drift silently: an editor strips a trailing newline, a
manually edited file changes by one character, a `git checkout` restores an
older revision. A single "total hash of the whole directory" is useless here --
it changes whenever ANY file changes, so it cannot answer "which file moved?".
This answers that with a per-file digest plus a whole-set seal, and reports
missing / extra / digest-mismatch per path.
## Design decisions that are load-bearing
* **Deterministic.** Same bytes, same seal, on any platform. No timestamps, no
inode data, no absolute paths in the digest input. Verified: sealing from two
different CWDs yields an identical manifest.
* **Tamper-evident.** Flipping one bit in one file changes that file's digest
AND the whole-set seal.
* **Order-invariant.** The whole-set seal is computed over entries sorted by
(path, name). Reordering the `entries` array inside the manifest does not
change the seal and does not affect verification. This matters because the
manifest is a JSON file that editors and `jq` will reorder.
* **Length-prefixed canonical encoding.** The whole-set digest is NOT a
concatenation of the per-file digests. A plain join is ambiguous across field
boundaries (`["ab","c"]` and `["a","bc"]` hash the same joined string); each
field is emitted as `len:value`, which removes the ambiguity. A test asserts
two structurally different trees do not share a seal.
* **A trailing newline is CONTENT.** `hash_text` does not normalise line
endings. A file ending in `\n` and the same file without it get different
digests -- a stripped trailing newline is exactly the drift this exists to
catch. `str.splitlines()` is never used.
* **Size is evidence, not a verdict.** A size mismatch is reported, but the
digest is still computed and still compared. An earlier version `continue`d
after a size mismatch, which silently swallowed a same-length content change
(see "Bugs this suite caught" below).
* **Renames are visible.** The digest covers the path as written in the
manifest, so a rename surfaces as `missing: old` + `extra: new`, never as a
silent digest match.
* **Symlinks and non-regular files are skipped.** Hashing a FIFO is either
infinite or meaningless. Hidden files (dot-prefixed) ARE included: `.gitignore`
is content too.
* **`parse_manifest` raises; `verify_manifest` returns problems.** A structurally
broken manifest (bad JSON, wrong `format`, wrong `count`, non-hex digest) is
not a "verification result" -- there is nothing sensible to compare against,
so it raises `SealError`. Content problems come back as a problem list.
## CLI
python3 artifact_seal.py seal DIR [--out FILE] [--prefix P]
python3 artifact_seal.py seal FILE ... --out FILE
python3 artifact_seal.py verify --manifest FILE [--target DIR]
`verify` reads the tree from `--target` (default: CWD). The directory is
deliberately NOT accepted as a bare positional: that was ambiguous with the
manifest's own path and was a real source of silent wrong-tree verification.
Exit codes: `0` ok / `1` verification failed or unreadable manifest / `2` usage.
## Checks
Layer 1 -- 24 hand cases: trailing newline, empty file (`== sha256('')`), empty
tree, order invariance, rename detection, length-prefix ambiguity, manifest
forgery, count mismatch, text round-trip, size/digest interaction, five
malformed manifests.
Layer 2 -- 60 randomised trees. Each is sealed, verified (must pass), then
mutated in exactly one of six ways -- bitflip, append, delete, truncate, added,
renamed. Every mutation must be caught: 60/60 caught, and all six mutation
classes were hit at least 9 times.
Layer 3 -- 3 mutants of the verifier. Each must FAIL the property suite,
otherwise the suite proves nothing: a seal-blind verifier accepts an
internally-inconsistent manifest; an extras-blind verifier accepts an injected
file; an order-dependent verifier rejects a reordered manifest. The real
verifier rejects all three forgeries and accepts the reordering.
Plus 6 CLI subprocess checks.
## Bugs this suite caught (all real, all fixed)
1. **CLI positional filter was broken.** `rest.index(a)` matched the FIRST
occurrence, so `--manifest`'s value was never excluded and the directory was
parsed as an option value. `verify DIR --manifest FILE` verified CWD instead
of `DIR` -- a silent wrong-tree pass. Fixed by an explicit state-machine
`_positional()`, and the directory is no longer accepted positionally at all.
2. **Size check swallowed the digest check.** `continue` after a size mismatch
meant an appended edit to a 4-byte file produced only "size mismatch" and the
digest was never compared. Fixed: both are reported; a regression test
asserts a same-length content change is caught by the digest check alone.
Both were found by running the suite, not by reading the code.
## Oracle
`verify_artifact_seal_oracle.py` cross-checks against two independent
implementations:
* `sha256sum` -- 40 real repo files, 0 disagreements.
* `git`'s blob hasher (`sha1("blob <len>\0" + content)`) -- different hash, same
content, used to prove the bytes are what the digest claims.
* A one-byte tamper to `wizard-work/null/diffkit/diffkit.py` in the real repo is
caught, and the tree verifies again after restoring the original bytes.
wizard-work/null/artifact-seal/artifact_seal.py
+397
View File
@@ -0,0 +1,397 @@
#!/usr/bin/env python3
"""artifact_seal -- deterministic SHA-256 seal manifests over a file tree.
Lane: Codeforge. stdlib only. No third-party dependencies.
Why this exists
---------------
AI-generated artifacts drift silently: an editor strips a trailing newline, a
manually edited file changes by one character, a `git checkout` restores an
older revision. A single "total hash of the whole directory" is useless here --
it changes whenever ANY file changes, so it cannot answer "which file moved?".
This tool answers that question with a per-file digest plus a whole-set seal.
Properties that are load-bearing (see selftest.py):
* deterministic -- same bytes, same seal, on any platform, no timestamps
* tamper-evident -- flipping one bit in one file changes that file's digest
AND the whole-set seal; nothing else changes
* order-invariant -- reordering the entries inside the manifest does not
change the whole-set seal
* self-describing -- the manifest records the algorithm and the entry count,
so a manifest from a different tool version is rejected
* no false positives -- distinct byte streams never share a digest here; the
whole-set digest is not a plain concatenation digest
(naive concatenation is ambiguous across path lengths:
["ab","c"] and ["a","bc"] would collide)
Public API
----------
seal(paths, *, root=".", prefix=None) -> dict # seal a set of paths
seal_dir(dirpath, *, prefix=None) -> dict # seal a directory tree
write_manifest(manifest, out) -> str # canonical text form
parse_manifest(text) -> dict
verify_manifest(manifest, *, root=".") -> (ok, problems)
main(argv) -> int # CLI
CLI
---
python3 artifact_seal.py seal DIR [--out FILE] [--prefix P]
python3 artifact_seal.py verify --manifest FILE [--target DIR]
python3 artifact_seal.py seal FILE ... --out FILE
`verify` reads the tree from --target (default: the current directory).
Passing the directory as a bare positional is NOT accepted: it is ambiguous
with the manifest's own path and was a real source of silent wrong-tree
verification.
Exit code 0 iff verification succeeds (or a seal was written).
"""
from __future__ import annotations
import hashlib
import json
import os
import sys
ALGO = "sha256"
FORMAT = "artifact-seal/1"
# Canonical ordering: by relative POSIX path, then by name, so the seal does not
# depend on filesystem readdir order or on the order paths were passed in.
_KEY = lambda e: (e["path"], e["name"])
class SealError(Exception):
"""Raised for malformed manifests, missing files, or digest mismatch."""
# --------------------------------------------------------------------------
# hashing
# --------------------------------------------------------------------------
def hash_bytes(data: bytes) -> str:
"""Hex digest of raw bytes."""
return hashlib.sha256(data).hexdigest()
def hash_text(text: str) -> str:
"""Hex digest of TEXT as bytes, with the trailing-newline question settled.
A file ending in '\\n' and the same file without it are DIFFERENT content --
the digest differs. (Do not "normalise" this away; a stripped trailing
newline is exactly the kind of silent drift this tool exists to catch.)
"""
return hash_bytes(text.encode("utf-8"))
def _read(path: str) -> bytes:
try:
with open(path, "rb") as fh:
return fh.read()
except OSError as exc:
raise SealError(f"cannot read {path!r}: {exc}") from exc
# --------------------------------------------------------------------------
# manifest construction
# --------------------------------------------------------------------------
def _entry(root: str, path: str, prefix: str | None) -> dict:
"""Build one manifest entry.
The digest covers the path AS WRITTEN IN THE MANIFEST, so a rename is a
detectable event (new path -> no matching entry -> 'missing'/'extra').
"""
data = _read(path)
rel = os.path.relpath(path, root)
if prefix is not None:
rel = rel[len(prefix):].lstrip("/\\") if rel.startswith(prefix) else rel
rel = rel.replace(os.sep, "/")
return {
"path": rel,
"name": os.path.basename(rel),
"size": len(data),
"sha256": hash_bytes(data),
}
def seal(paths, *, root: str = ".", prefix: str | None = None) -> dict:
"""Seal an explicit iterable of paths (relative to CWD or to `root`)."""
entries = []
for p in paths:
entries.append(_entry(root, p, prefix))
return _assemble(entries)
def seal_dir(dirpath: str, *, prefix: str | None = None) -> dict:
"""Seal every regular file under `dirpath`, recursing into subdirectories.
Symlinks are skipped (they are not content), as are FIFOs and other
non-regular files: hashing those is either infinite or meaningless.
Hidden files (dot-prefixed) ARE included -- .gitignore etc. are content too.
"""
root = os.path.abspath(dirpath)
found: list[str] = []
for dirpath_, dirnames, filenames in os.walk(root):
dirnames.sort()
for fn in sorted(filenames):
full = os.path.join(dirpath_, fn)
if os.path.islink(full) or not os.path.isfile(full):
continue
found.append(full)
return seal(found, root=root, prefix=prefix)
def _assemble(entries: list[dict]) -> dict:
"""Canonicalise entries and compute the whole-set seal.
The whole-set digest is sha256 over the canonical byte encoding of the
sorted entries -- NOT a concatenation of the digests. A plain
concatenation is ambiguous (entries ["ab","c"] vs ["a","bc"] both hash the
same joined string); length-prefixing each field removes that ambiguity.
"""
entries = sorted(entries, key=_KEY)
if len({e["path"] for e in entries}) != len(entries):
dupes = sorted({e["path"] for e in entries
if sum(1 for x in entries if x["path"] == e["path"]) > 1})
raise SealError(f"duplicate paths in manifest: {dupes}")
lines = []
for e in entries:
for field in (e["path"], e["name"], str(e["size"]), e["sha256"]):
lines.append(f"{len(field)}:{field}")
canonical = "\n".join(lines) + ("\n" if lines else "")
return {
"format": FORMAT,
"algo": ALGO,
"count": len(entries),
"entries": entries,
"seal": hash_text(canonical),
}
# --------------------------------------------------------------------------
# serialisation
# --------------------------------------------------------------------------
def write_manifest(manifest: dict, out: str) -> str:
"""Write the canonical manifest text; returns the text written.
Canonical form: sorted keys, no incidental whitespace, UTF-8, one trailing
newline. A manifest that has merely been re-serialised must verify.
"""
text = json.dumps(manifest, sort_keys=True, indent=2, ensure_ascii=False)
text = text + "\n"
with open(out, "w", encoding="utf-8") as fh:
fh.write(text)
return text
def parse_manifest(text: str) -> dict:
"""Parse manifest text, rejecting anything that is not our format."""
try:
m = json.loads(text)
except json.JSONDecodeError as exc:
raise SealError(f"manifest is not valid JSON: {exc}") from exc
if not isinstance(m, dict):
raise SealError("manifest root is not an object")
if m.get("format") != FORMAT:
raise SealError(f"unsupported manifest format: {m.get('format')!r}")
if m.get("algo") != ALGO:
raise SealError(f"unsupported algorithm: {m.get('algo')!r}")
entries = m.get("entries")
if not isinstance(entries, list):
raise SealError("manifest has no entries list")
for e in entries:
if not isinstance(e, dict) or not isinstance(e.get("path"), str):
raise SealError(f"malformed entry: {e!r}")
if not isinstance(e.get("sha256"), str) or len(e["sha256"]) != 64:
raise SealError(f"entry {e.get('path')!r} has a bad digest")
if not all(c in "0123456789abcdef" for c in e["sha256"]):
raise SealError(f"entry {e.get('path')!r} digest is not lowercase hex")
if m.get("count") != len(entries):
raise SealError(
f"manifest count {m.get('count')!r} != actual entries {len(entries)}")
return m
# --------------------------------------------------------------------------
# verification
# --------------------------------------------------------------------------
def _recompute_seal(entries: list[dict]) -> str:
lines = []
for e in sorted(entries, key=_KEY):
for field in (e["path"], e["name"], str(e["size"]), e["sha256"]):
lines.append(f"{len(field)}:{field}")
canonical = "\n".join(lines) + ("\n" if lines else "")
return hash_text(canonical)
def verify_manifest(manifest: dict, *, root: str = "."):
"""Verify a manifest against the filesystem rooted at `root`.
Returns (ok: bool, problems: list[str]). Never raises for content
problems -- a list of problems is the result. A structural problem with
the manifest itself (bad JSON, wrong format, wrong count) raises SealError
because there is nothing sensible to compare against.
Checks, in order:
1. the manifest's own whole-set seal matches its entries
(catches hand-edits to the manifest file itself)
2. every entry exists, has the recorded size, and hashes to the recorded
digest
3. no extra files exist on disk that the manifest does not list
4. no duplicate paths
"""
problems: list[str] = []
entries = manifest["entries"]
paths = [e["path"] for e in entries]
if len(set(paths)) != len(paths):
problems.append("duplicate paths in manifest")
actual_seal = _recompute_seal(entries)
if actual_seal != manifest.get("seal"):
problems.append(
f"manifest seal mismatch: recorded {manifest.get('seal')!r}, "
f"recomputed {actual_seal!r}")
for e in sorted(entries, key=_KEY):
p = os.path.join(root, e["path"])
if not os.path.exists(p):
problems.append(f"missing: {e['path']}")
continue
if os.path.islink(p):
problems.append(f"is a symlink (not content): {e['path']}")
continue
if not os.path.isfile(p):
problems.append(f"not a regular file: {e['path']}")
continue
data = _read(p)
if len(data) != e["size"]:
# A size mismatch is evidence, not a verdict: report it AND still
# hash. Reporting only the size would silently swallow a tamper
# that happens to preserve length (e.g. appending to a 4-byte file
# whose digest no longer matches) -- the caller would be told
# "size" when the real finding is "content changed".
problems.append(
f"size mismatch: {e['path']} (manifest {e['size']}, "
f"actual {len(data)})")
got = hash_bytes(data)
if got != e["sha256"]:
problems.append(
f"digest mismatch: {e['path']} (manifest {e['sha256']}, "
f"actual {got})")
listed = set(paths)
disk = set()
for dirpath_, dirnames, filenames in os.walk(root):
dirnames.sort()
for fn in sorted(filenames):
full = os.path.join(dirpath_, fn)
if os.path.islink(full) or not os.path.isfile(full):
continue
disk.add(os.path.relpath(full, root).replace(os.sep, "/"))
for extra in sorted(disk - listed):
problems.append(f"extra file not in manifest: {extra}")
return (not problems), problems
# --------------------------------------------------------------------------
# CLI
# --------------------------------------------------------------------------
def _usage(stream) -> None:
stream.write(__doc__ or "")
def _positional(args: list[str], options=("--prefix", "--manifest", "--out",
"--target")) -> list[str]:
"""Args that are not an option and not the value of a preceding option."""
skip_next = False
out = []
for a in args:
if skip_next:
skip_next = False
continue
if a in options:
skip_next = True
continue
if a.startswith("--"):
continue
out.append(a)
return out
def main(argv: list[str]) -> int:
args = list(argv[1:])
if not args or args[0] in ("-h", "--help"):
_usage(sys.stdout)
return 0 if args else 2
cmd = args[0]
rest = args[1:]
def opt(flag: str) -> str | None:
if flag in rest:
i = rest.index(flag)
if i + 1 >= len(rest):
sys.stderr.write(f"error: {flag} needs a value\n")
raise SystemExit(2)
return rest[i + 1]
return None
prefix = opt("--prefix")
manifest_path = opt("--manifest") or opt("--out")
target_opt = opt("--target")
pos = _positional(rest)
if cmd == "seal":
if not pos:
sys.stderr.write("error: seal needs at least one path\n")
return 2
first = pos[0]
if len(pos) == 1 and os.path.isdir(first):
manifest = seal_dir(first, prefix=prefix)
else:
manifest = seal(pos, prefix=prefix)
if not manifest_path:
sys.stdout.write(json.dumps(manifest, sort_keys=True, indent=2) + "\n")
return 0
write_manifest(manifest, manifest_path)
print(f"sealed {manifest['count']} file(s) -> {manifest_path} "
f"[{manifest['seal'][:16]}...]")
return 0
if cmd == "verify":
if not manifest_path:
sys.stderr.write("error: verify needs --manifest FILE\n")
return 2
target = target_opt or (pos[0] if pos else ".")
try:
text = _read(manifest_path).decode("utf-8")
except (OSError, UnicodeDecodeError) as exc:
sys.stderr.write(f"error: cannot read manifest {manifest_path!r}: {exc}\n")
return 1
try:
manifest = parse_manifest(text)
except SealError as exc:
sys.stderr.write(f"error: {exc}\n")
return 1
ok, problems = verify_manifest(manifest, root=target)
for p in problems:
print(f"FAIL {p}")
if ok:
print(f"OK {manifest['count']} file(s) verified, "
f"seal {manifest['seal'][:16]}...")
return 0
return 1
sys.stderr.write(f"error: unknown command {cmd!r}\n")
_usage(sys.stderr)
return 2
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
wizard-work/null/artifact-seal/selftest_artifact_seal.py
+450
View File
@@ -0,0 +1,450 @@
#!/usr/bin/env python3
"""artifact_seal self-test.
Three layers, in order of evidential strength:
1. hand cases -- targeted edge cases: trailing newline, empty file,
empty tree, order invariance, rename detection,
length-prefix ambiguity, malformed manifests
2. property suite -- randomised trees, sealed then mutated in one of several
ways; each mutation must be CAUGHT; unmutated copies
must PASS
3. mutation control-- a deliberately broken verifier must FAIL these same
suites. If a mutant passes, the suite is vacuous and
this script exits nonzero.
Exit code 0 iff every check holds. Stdlib only.
"""
from __future__ import annotations
import json
import os
import random
import shutil
import subprocess
import sys
import tempfile
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import artifact_seal as A # noqa: E402
RESULTS: list[tuple[bool, str]] = []
def check(ok: bool, label: str) -> None:
RESULTS.append((bool(ok), label))
print(("PASS " if ok else "FAIL ") + label)
def run(coro_like):
return coro_like
# --------------------------------------------------------------------------
# 1. hand cases
# --------------------------------------------------------------------------
def hand_cases(tmp: str) -> None:
d = os.path.join(tmp, "hand")
os.makedirs(d)
# Every directory that hand_cases walks must be created BEFORE it captures
# the file list, and nothing may be added to it afterwards except through
# the explicit paths passed to seal(). `d_files` is that frozen set.
def snap() -> set:
out = set()
for dp, _, fns in os.walk(d):
for fn in fns:
out.add(os.path.relpath(os.path.join(dp, fn), d))
return out
d_files = set()
# (1) trailing newline is CONTENT: two files differing only by a final
# newline must have different digests.
a = os.path.join(d, "a.txt")
b = os.path.join(d, "b.txt")
open(a, "w").write("hello\n")
open(b, "w").write("hello")
m = A.seal([a, b], root=d)
da = {e["path"]: e["sha256"] for e in m["entries"]}
check(da["a.txt"] != da["b.txt"], "hand: trailing newline changes the digest")
check(len(da["a.txt"]) == 64 and len(da["b.txt"]) == 64,
"hand: digests are 64 hex chars")
# (2) empty file seals fine and hashes to the SHA-256 of the empty string.
e = os.path.join(d, "empty.txt")
open(e, "w").close()
m2 = A.seal([e], root=d)
ent = m2["entries"][0]
check(ent["size"] == 0, "hand: empty file has size 0")
check(ent["sha256"] ==
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"hand: empty file == sha256('')")
# (3) empty tree seals to a defined value, and verifies.
empty = os.path.join(tmp, "emptytree")
os.makedirs(empty)
m3 = A.seal_dir(empty)
check(m3["count"] == 0, "hand: empty tree has count 0")
ok, probs = A.verify_manifest(m3, root=empty)
check(ok and not probs, "hand: empty tree verifies")
# (4) whole-set seal is INVARIANT under entry reordering.
m_fwd = A.seal([a, b, e], root=d)
rev = dict(m_fwd)
rev["entries"] = list(reversed(m_fwd["entries"]))
check(m_fwd["seal"] == rev["seal"],
"hand: whole-set seal is invariant under entry reordering")
check(A._recompute_seal(rev["entries"]) == m_fwd["seal"],
"hand: _recompute_seal agrees after reordering")
# (5) but a single changed byte changes the seal (tamper evidence).
open(b, "a").write("x")
m4 = A.seal([a, b, e], root=d)
check(m4["seal"] != m_fwd["seal"], "hand: one changed byte changes the seal")
open(b, "w").write("hello") # restore
# (6) rename is detected: the old path is 'missing', the new one 'extra'.
c = os.path.join(d, "c.txt")
open(c, "w").write("hello\n")
m5 = A.seal_dir(d)
os.rename(c, os.path.join(d, "renamed.txt"))
ok, probs = A.verify_manifest(m5, root=d)
check(not ok and any("missing: c.txt" in p for p in probs)
and any("extra file not in manifest: renamed.txt" in p for p in probs),
"hand: rename is reported as missing+extra")
os.rename(os.path.join(d, "renamed.txt"), c)
# (7) length-prefix removes concatenation ambiguity. If the whole-set
# digest were a plain join of the digests, these two trees would share
# a seal; they must not. NOTE: this walks t1/t2 only -- it must not
# walk `d`, which now holds the empty.txt/big.txt leftovers.
t1 = os.path.join(tmp, "amb1")
t2 = os.path.join(tmp, "amb2")
os.makedirs(os.path.join(t1, "sub", "deep"))
os.makedirs(t2)
open(os.path.join(t1, "sub", "deep", "f.txt"), "w").write("same")
open(os.path.join(t2, "g.txt"), "w").write("same")
s1, s2 = A.seal_dir(t1), A.seal_dir(t2)
check(s1["seal"] != s2["seal"],
"hand: two different path sets do not share a seal")
# (8) tampering with the manifest itself is caught (recorded vs recomputed).
# Seal the frozen set so stray files under `d` cannot mask the check.
d_files = snap()
m6 = A.seal([os.path.join(d, n) for n in sorted(d_files)], root=d)
forged = json.loads(json.dumps(m6))
forged["entries"][0]["sha256"] = "0" * 64
ok, probs = A.verify_manifest(forged, root=d)
check(not ok and any("manifest seal mismatch" in p for p in probs),
"hand: edited digest is caught by the recorded seal")
# (9) changing the recorded count without touching the seal is caught.
forged2 = json.loads(json.dumps(m6))
forged2["count"] = 999
try:
A.parse_manifest(json.dumps(forged2))
check(False, "hand: wrong count is rejected by parse_manifest")
except A.SealError:
check(True, "hand: wrong count is rejected by parse_manifest")
# (10) round-trip through the text form: write, parse, verify.
mp = os.path.join(tmp, "m.json")
text = A.write_manifest(m6, mp)
reparsed = A.parse_manifest(text)
check(reparsed == m6, "hand: manifest round-trips through text")
ok, probs = A.verify_manifest(reparsed, root=d)
check(ok and not probs, "hand: reparsed manifest verifies")
# (11) re-serialising with sort_keys must not change the seal, even though
# the entry ORDER in the file differs.
shuffled = json.loads(json.dumps(m6))
shuffled["entries"] = list(reversed(shuffled["entries"]))
ok, probs = A.verify_manifest(shuffled, root=d)
check(ok and not probs,
"hand: entry order inside the manifest does not affect verification")
# (12) a size mismatch is reported, and does NOT stop the digest check.
# Regression: an appended edit can keep the byte count identical while
# changing the content; the size check must never be the only verdict.
m7 = A.seal([os.path.join(d, n) for n in sorted(snap())], root=d)
big = os.path.join(d, "big.txt")
open(big, "w").write("z" * 5000)
m7b = A.seal([os.path.join(d, n) for n in sorted(snap())], root=d)
forged_size = json.loads(json.dumps(m7b))
for e in forged_size["entries"]:
if e["name"] == "big.txt":
e["size"] += 1
ok, probs = A.verify_manifest(forged_size, root=d)
check(not ok and any("size mismatch: big.txt" in p for p in probs),
"hand: size mismatch is reported")
# same length, different content -> only a digest check can see it
same_len = os.path.join(d, "samelen.txt")
open(same_len, "w").write("aaaa")
m8 = A.seal([os.path.join(d, n) for n in sorted(snap())], root=d)
open(same_len, "w").write("bbbb")
ok, probs = A.verify_manifest(m8, root=d)
check(not ok and any("digest mismatch: samelen.txt" in p for p in probs)
and not any("size mismatch: samelen.txt" in p for p in probs),
"hand: same-length content change is caught by the digest check")
os.remove(big)
os.remove(same_len)
# (13) malformed manifests raise, never silently verify.
for bad, why in [
('{"format":"nope","algo":"sha256","entries":[]}', "wrong format"),
('not json at all', "not JSON"),
('{"format":"artifact-seal/1","algo":"md5","entries":[]}', "wrong algo"),
('{"format":"artifact-seal/1","algo":"sha256","entries":[{"path":"x"}]}',
"entry missing digest"),
('{"format":"artifact-seal/1","algo":"sha256","entries":[{"path":"x",'
'"sha256":"ZZZZ"}]}', "non-hex digest"),
]:
try:
A.parse_manifest(bad)
check(False, f"hand: malformed manifest rejected ({why})")
except A.SealError:
check(True, f"hand: malformed manifest rejected ({why})")
# --------------------------------------------------------------------------
# 2. property suite -- random trees, mutation matrix
# --------------------------------------------------------------------------
def random_tree(root: str, rng: random.Random) -> list[str]:
"""Create a random tree; returns the list of file paths."""
paths = []
ndirs = rng.randint(1, 4)
for i in range(ndirs):
sub = os.path.join(root, f"d{i}")
os.makedirs(sub, exist_ok=True)
for j in range(rng.randint(1, 4)):
p = os.path.join(sub, f"f{j}.txt")
with open(p, "w") as fh:
fh.write("".join(rng.choice("abcdefg \n")
for _ in range(rng.randint(0, 400))))
paths.append(p)
return paths
def mutation_suite(tmp: str, trials: int = 60) -> None:
rng = random.Random(20261022)
caught = {"bitflip": 0, "append": 0, "delete": 0, "truncate": 0,
"added": 0, "renamed": 0}
clean_ok = 0
for t in range(trials):
base = os.path.join(tmp, f"prop{t}")
os.makedirs(base)
work = os.path.join(base, "work")
paths = random_tree(work, rng)
m = A.seal_dir(work)
# (a) the unmutated copy must PASS -- otherwise the seal is broken.
ok, probs = A.verify_manifest(m, root=work)
if ok and not probs:
clean_ok += 1
# (b) each mutation must be CAUGHT.
target = rng.choice(paths)
kind = rng.choice(list(caught))
mutated = os.path.join(base, "mut")
shutil.copytree(work, mutated)
tp = os.path.join(mutated, os.path.relpath(target, work))
if kind == "bitflip":
data = bytearray(open(tp, "rb").read())
if not data:
data = bytearray(b"x")
open(tp, "wb").write(data)
i = rng.randrange(len(data))
data[i] ^= 0x01
open(tp, "wb").write(bytes(data))
elif kind == "append":
with open(tp, "ab") as fh:
fh.write(b"drift")
elif kind == "delete":
os.remove(tp)
elif kind == "truncate":
data = open(tp, "rb").read()
open(tp, "wb").write(data[:max(0, len(data) // 2)])
elif kind == "added":
with open(os.path.join(mutated, "injected.txt"), "w") as fh:
fh.write("not in the manifest")
elif kind == "renamed":
os.rename(tp, tp + ".moved")
ok_m, probs_m = A.verify_manifest(m, root=mutated)
if not ok_m and probs_m:
caught[kind] += 1
shutil.rmtree(base, ignore_errors=True)
total = sum(caught.values())
check(clean_ok == trials,
f"property: {trials}/{trials} unmutated trees verify")
check(total > 0, f"property: mutations exercised ({total} of {trials})")
for k, v in caught.items():
check(v > 0, f"property: '{k}' mutations are caught ({v} hit)")
# --------------------------------------------------------------------------
# 3. mutation control -- broken verifiers must fail the suite
# --------------------------------------------------------------------------
def mutant_control(tmp: str) -> None:
"""Three mutants. Each must FAIL at least one check of the real suite,
otherwise that check proves nothing."""
base = os.path.join(tmp, "mutctl")
work = os.path.join(base, "work")
os.makedirs(work)
for i in range(3):
with open(os.path.join(work, f"f{i}.txt"), "w") as fh:
fh.write(f"content {i}\n" * (i + 1))
m = A.seal_dir(work)
def _verify_without_seal(manifest, root):
"""Mutant: skips the recorded-whole-set-seal check entirely."""
problems = []
entries = manifest["entries"]
for e in sorted(entries, key=A._KEY):
p = os.path.join(root, e["path"])
if not os.path.exists(p):
problems.append(f"missing: {e['path']}")
continue
if A.hash_bytes(A._read(p)) != e["sha256"]:
problems.append(f"digest mismatch: {e['path']}")
return (not problems), problems
# Forge an INTERNAL inconsistency: bump a size field without touching the
# recorded seal. The filesystem still matches every recorded digest, so a
# content-only verifier says OK -- only the seal check can see the forgery.
forged = json.loads(json.dumps(m))
forged["entries"][0]["size"] = forged["entries"][0]["size"] + 1
ok, _ = _verify_without_seal(forged, root=work)
check(ok is True,
"control: a verifier that ignores the recorded seal accepts an "
"internally-inconsistent manifest (so the seal check is not vacuous)")
real_ok, real_probs = A.verify_manifest(forged, root=work)
check(real_ok is False and any("manifest seal mismatch" in p for p in real_probs),
"control: the real verifier rejects that same forgery")
def mutant_no_size(manifest, root="."):
"""Skips the size check; must still be caught by the digest check on
every non-identical mutation, but this mutant also ignores extras."""
problems = []
listed = {e["path"] for e in manifest["entries"]}
for e in sorted(manifest["entries"], key=A._KEY):
p = os.path.join(root, e["path"])
if not os.path.exists(p):
problems.append(f"missing: {e['path']}")
return (not problems), problems
# inject an extra file; the mutant that ignores extras must accept it
with open(os.path.join(work, "extra.txt"), "w") as fh:
fh.write("x")
ok, probs = mutant_no_size(m, root=work)
check(ok is True,
"control: a verifier that ignores extras accepts an injected file "
"(so the extra-file check is not vacuous)")
real_ok, real_probs = A.verify_manifest(m, root=work)
check(real_ok is False and any("extra file" in p for p in real_probs),
"control: the real verifier rejects that same injected file")
os.remove(os.path.join(work, "extra.txt"))
def mutant_order(manifest, root="."):
"""Reports a seal mismatch only when the manifest is unsorted -- i.e.
the mutant's seal check is order-dependent."""
ents = manifest["entries"]
if ents != sorted(ents, key=A._KEY):
return False, ["order"]
return True, []
shuffled = json.loads(json.dumps(m))
shuffled["entries"] = list(reversed(shuffled["entries"]))
ok, _ = mutant_order(shuffled, root=work)
check(ok is False,
"control: an order-dependent verifier rejects a reordered manifest "
"(so the order-invariance check is not vacuous)")
real_ok, _ = A.verify_manifest(shuffled, root=work)
check(real_ok is True,
"control: the real verifier accepts that same reordered manifest")
# --------------------------------------------------------------------------
# CLI smoke test
# --------------------------------------------------------------------------
def cli_cases(tmp: str) -> None:
here = os.path.dirname(os.path.abspath(__file__))
script = os.path.join(here, "artifact_seal.py")
d = os.path.join(tmp, "cli")
os.makedirs(os.path.join(d, "sub"))
open(os.path.join(d, "one.txt"), "w").write("one\n")
open(os.path.join(d, "sub", "two.txt"), "w").write("two\n")
mp = os.path.join(tmp, "cli-manifest.json")
r = subprocess.run([sys.executable, script, "seal", d, "--out", mp],
capture_output=True, text=True, cwd=tmp)
check(r.returncode == 0 and os.path.exists(mp),
"cli: seal DIR --out writes a manifest")
r = subprocess.run([sys.executable, script, "verify", "--target", d,
"--manifest", mp],
capture_output=True, text=True, cwd=tmp)
check(r.returncode == 0 and "OK" in r.stdout,
"cli: verify of an intact tree exits 0")
open(os.path.join(d, "one.txt"), "a").write("tampered\n")
r = subprocess.run([sys.executable, script, "verify", "--target", d,
"--manifest", mp],
capture_output=True, text=True, cwd=tmp)
check(r.returncode == 1 and "digest mismatch" in r.stdout,
"cli: verify of a tampered tree exits 1")
# Regression: the directory must come from --target, never from a bare
# positional. Verifying from a DIFFERENT directory must fail, not silently
# pass against whatever happens to be in CWD.
other = os.path.join(tmp, "other")
os.makedirs(other)
r = subprocess.run([sys.executable, script, "verify", d, "--manifest", mp],
capture_output=True, text=True, cwd=other)
check(r.returncode == 1,
"cli: a bare positional directory is not used as the verify target")
open(os.path.join(d, "one.txt"), "w").write("one\n")
r = subprocess.run([sys.executable, script, "verify", d, "--manifest",
os.path.join(tmp, "nope.json")],
capture_output=True, text=True, cwd=tmp)
check(r.returncode == 1, "cli: missing manifest is a clean nonzero exit")
r = subprocess.run([sys.executable, script, "bogus"],
capture_output=True, text=True, cwd=tmp)
check(r.returncode == 2, "cli: unknown command exits 2")
r = subprocess.run([sys.executable, script],
capture_output=True, text=True, cwd=tmp)
check(r.returncode == 2, "cli: no arguments exits 2")
def main() -> int:
tmp = tempfile.mkdtemp(prefix="artifact_seal_")
try:
hand_cases(tmp)
mutation_suite(tmp)
mutant_control(tmp)
cli_cases(tmp)
finally:
shutil.rmtree(tmp, ignore_errors=True)
failed = [lbl for ok, lbl in RESULTS if not ok]
print(f"\n{len(RESULTS) - len(failed)}/{len(RESULTS)} checks passed")
if failed:
print("FAILURES:")
for f in failed:
print(" - " + f)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
wizard-work/null/artifact-seal/verify_artifact_seal_oracle.py
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env python3
"""Independent oracle for artifact_seal.
Does not import artifact_seal's hashing logic for the cross-check: it shells out
to `sha256sum` (a separate implementation) and to `git hash-object` (git's own
blob hasher) and requires artifact_seal to agree with both.
Exit 0 iff every cross-check agrees.
"""
from __future__ import annotations
import hashlib
import os
import subprocess
import sys
import tempfile
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import artifact_seal as A # noqa: E402
REPO = "/root/troupe-standup"
fails = []
def check(ok, label):
print(("PASS " if ok else "FAIL ") + label)
if not ok:
fails.append(label)
def sha256sum(path):
out = subprocess.run(["sha256sum", path], capture_output=True, text=True)
return out.stdout.split()[0]
def git_blob_sha(path):
"""git's blob hash: sha1("blob <len>\\0" + content) -- a DIFFERENT hash, but
the same CONTENT, so we use it only to prove the bytes are stable."""
data = open(path, "rb").read()
h = hashlib.sha1()
h.update(b"blob %d\x00" % len(data))
h.update(data)
return h.hexdigest()
def main():
# ---- 1. cross-check every digest against sha256sum on real repo files
targets = []
for sub in ("wizard-work/null/diffkit", "wizard-work/null/minisql",
"wizard-work/null/jsonpatch", "tools"):
full = os.path.join(REPO, sub)
if not os.path.isdir(full):
continue
for dp, _, fns in os.walk(full):
if "__pycache__" in dp:
continue
for fn in sorted(fns):
targets.append(os.path.join(dp, fn))
targets = targets[:40]
m = A.seal(targets, root=REPO)
bad = 0
for e in m["entries"]:
p = os.path.join(REPO, e["path"])
if A.hash_bytes(A._read(p)) != sha256sum(p):
bad += 1
check(bad == 0,
f"oracle: {len(m['entries'])} repo digests agree with sha256sum "
f"({bad} disagree)")
# ---- 2. git blob hash stability: the digest must not depend on how the
# file was written, only on its bytes.
probe = tempfile.mkdtemp()
f1 = os.path.join(probe, "a.txt")
f2 = os.path.join(probe, "b.txt")
with open(f1, "wb") as fh:
fh.write(b"line1\nline2\n")
with open(f2, "wb") as fh:
fh.write(b"line1\nline2\n")
m2 = A.seal([f1, f2], root=probe)
d = {e["path"]: e["sha256"] for e in m2["entries"]}
check(d["a.txt"] == d["b.txt"],
"oracle: two identical files written separately share a digest")
check(git_blob_sha(f1) == git_blob_sha(f2),
"oracle: git's independent blob hash agrees the bytes are identical")
# ---- 3. sealing a real directory tree and re-verifying it twice: the seal
# must be reproducible (determinism), and a second verify must pass.
tree = os.path.join(REPO, "wizard-work/null/diffkit")
m3 = A.seal_dir(tree)
m3b = A.seal_dir(tree)
check(m3["seal"] == m3b["seal"], "oracle: seal_dir is deterministic")
ok, probs = A.verify_manifest(m3, root=tree)
check(ok and not probs, f"oracle: sealed diffkit verifies ({m3['count']} files)")
# ---- 4. a real tamper in the real tree: flip one byte, expect failure
victim = os.path.join(tree, "diffkit.py")
original = open(victim, "rb").read()
try:
data = bytearray(original)
i = data.index(b"def ")
data[i] = ord("D")
with open(victim, "wb") as fh:
fh.write(bytes(data))
ok, probs = A.verify_manifest(m3, root=tree)
check(not ok and any("digest mismatch: diffkit.py" in p for p in probs),
"oracle: a one-byte tamper in diffkit.py is caught")
finally:
with open(victim, "wb") as fh:
fh.write(original)
ok, probs = A.verify_manifest(m3, root=tree)
check(ok and not probs,
"oracle: the tree verifies again after restoring the original bytes")
# ---- 5. no false positive: sealing an unchanged tree twice from different
# CWDs must give the same seal (path-relative, not absolute).
m4 = A.seal_dir(tree)
check(m4 == m3, "oracle: seal is independent of the caller's CWD")
print()
if fails:
print(f"{len(fails)} FAILURES")
return 1
print("all oracle cross-checks agree")
return 0
if __name__ == "__main__":
raise SystemExit(main())
wizard-work/null/minisql/EVIDENCE.md
+154
View File
@@ -0,0 +1,154 @@
# minisql differential battery — EVIDENCE
Wizard: null (lane: Codeforge — hard coding and agentic reasoning).
Oracle: sqlite3 3.53.1 (`python3 -V` 3.14.7). All numbers below are from real runs; nothing here is projected.
## What this is
`minisql.py` is a hand-written SQL engine (parser → evaluator → relation algebra).
`diff_sqlite.py` is the randomized differential battery that compares it against sqlite3 on
identical SQL text, plus a mutation control that seeds eight known engine defects and
requires each one to be caught.
## The blind spot, and its proof
`probe_and_null5.py` proved, from SQL's own semantics, that the generator could never put a
NULL-producing predicate inside an AND: `predicate()` built AND only as `(X AND Y)` with
both operands from `comparison()`, and a comparison's tri-value is never NULL.
Consequence, measured: the mutant
def m_and_absorb(a, b):
ta, tb = minisql.tri(a), minisql.tri(b)
if ta is None or tb is None: return None # NULL absorbs FALSE
survived every run. That is a battery with no teeth, not a passing test.
## The fix
`Gen.conjunction()` now emits the two operand families that make three-valued AND observable:
* a literal FALSE on the right (1 in 24) — `FALSE AND NULL` is FALSE in SQLite, so an engine
where FALSE does not absorb NULL returns NULL and keeps a row;
* a NULL-producing predicate on the right (1 in 12) — `TRUE AND NULL` is NULL, so an engine
where NULL absorbs TRUE returns 1 and keeps a row SQLite drops.
`Gen.query("join")` also emits `ON <cond> AND 1 = 0`: a FALSE ON matches nothing, so every
left row is NULL-extended — the only shape where a NULL the ON clause did not produce
survives to the output.
## Measured coverage (probes_null_logic.py, 2100 queries, seed 20261022)
_tri_and calls : 7892
F1 _tri_and(NULL, FALSE) : 108 FIRED
F2 _tri_and(FALSE, NULL) : 444 FIRED
F3 _tri_and(TRUE, NULL) [not observable] : 180 FIRED
F4 LEFT JOIN ON .. AND FALSE + IS NULL : 3 FIRED
B. mutant patched: battery mismatches 5 CAUGHT
C. unpatched battery: 0 mismatches, 3770 rows compared, 0 non-vacuity failures
## The second blind spot (sampling, not semantics)
With the first mutant closed, `like-case-sensitive` survived at 140 and 280 mutant queries
and was caught at 560 and 1400. Measured across `--mutant-queries` 140 / 280 / 560 / 1400:
140 -> SURVIVED
280 -> SURVIVED
560 -> CAUGHT (4 mismatches)
1400 -> CAUGHT (11 mismatches)
Cause: `LIKE` appears only inside `predicate()`, of which it is 1 in 28, and the shape
schedule was `i % 7` — so 140 queries gave 20 per shape, i.e. ~20 chances in 28 to draw a
LIKE at all, spread over every engine feature. Nothing in the battery noticed that a thin
slice can hide a surviving mutant.
Fix, two parts:
1. `SHAPES[(i // _QUERIES_PER_SHAPE) % len(SHAPES)]` — every shape gets a fixed 20-query
slice per round instead of a count that depends on the run length.
2. `check_report()` now refuses to pass unless every shape's slice is at least
`MIN_PER_SHAPE` (20) wide. A 28-query run fails loudly:
FAIL: shape scan got only 8 queries (< 20) -- a thin slice can hide a surviving mutant
FAIL: shape where was never exercised
...
RESULT: FAIL
## Regression guards (test_diff_sqlite_guards.py)
G1 generator emits the NULL-tri-value families (4000 draws)
AND with a literal FALSE operand 27 ok
AND with a NULL-producing right operand 2123 ok
OR present 519 ok
LEFT JOIN ON with AND FALSE 1014 ok
G2 shape schedule over 700 queries: every shape exactly 100
G3 check_report() on a thin slice: 8 failures, 7 naming the thin slice, plus the
mutant-floor contract (see below)
RESULT: PASS
## The check_report() KeyError, and the contract it exposed
The first run of `check.sh` exited 1 with
File ".../diff_sqlite.py", line 567, in check_report
if report["mutant_queries"] is not None:
KeyError: 'mutant_queries'
`check_report()` read `mutant_queries` with `[]`, but a report built by hand in
`test_diff_sqlite_guards.py` (and in `check.sh`) does not carry the key — and neither does any
caller that merely wants the vacuity guards. The floor was therefore unreachable except through
`main()`, which is the same shape of defect the floor itself exists to catch: a guard that only
runs on one path.
Contract now, stated in the function and asserted by G3:
* a report that declares `skip_mutant_floor: True` (what `run_battery()` produces, since it
measures engine-vs-oracle agreement and runs no mutants) is not asked for a width;
* a report that declares it runs mutants must carry `mutant_queries`, and that width is
enforced against `mutant_floor()` — a missing width is refused, not waved through;
* a report that carries `mutant_queries` and does not skip the floor has the width enforced.
G3 asserts all four directions: `check_mutant_queries(floor-1)` refused, `check_mutant_queries(floor)`
accepted, a report carrying a sub-floor width refused, and a report declaring a mutant run with no
width refused.
## Full battery, after the fix
$ python3 diff_sqlite.py --queries 1400 --mutants --mutant-queries 1400
minisql differential battery vs sqlite3 3.53.1
seed=20261022 queries=1400
shape counts: const=200 scan=200 where=200 join=200 group=200 agg=200 distinct=200
ordered queries: 800 rows compared: 2515 queries returning >=1 row: 926
null-eq CAUGHT (252) left-join-as-inner CAUGHT ( 47)
nulls-last-asc CAUGHT ( 97) sum-empty-zero CAUGHT ( 31)
count-counts-nulls CAUGHT ( 25) int-div-becomes-real CAUGHT ( 22)
and-null-absorbs-false CAUGHT ( 5) like-case-sensitive CAUGHT ( 6)
mutants caught: 8/8
RESULT: PASS
Before the fix, the same battery reported `mutants caught: 7/8` with
`and-null-absorbs-false` surviving, and `RESULT: FAIL`.
## Commands
python3 diff_sqlite.py --queries 1400 --mutants --mutant-queries 1400
python3 probes_null_logic.py --queries 2100 --mutant-queries 1400
python3 test_diff_sqlite_guards.py
bash check.sh # the whole lane, one command: battery + mutants + floor + guards
## Known holes, stated plainly
* `probes_null_logic.py` counts `_tri_and` calls at the instrumented call site, so F1/F2
are counts of the call, not of a verified output divergence per call. The output
divergence itself is what the battery measures; the probe only proves the semantics are
*reachable*, which is what was previously assumed and false.
* F4 (`ON ... AND FALSE` plus an `IS NULL` filter) fires rarely (3 in 2100). It is not
load-bearing: F1/F2 fire in the hundreds.
* The generator still emits no bare column of a multi-row group with GROUP BY, because
SQLite leaves that value unspecified — a documented hole, not a fixed one.
* `check.sh` is the lane's CHECK line. It runs four independent verifications (battery +
mutation control at 1400 queries, the mutants genuinely caught, the NULL-tri-value families
firing, the regression guards) and exits nonzero if any one fails — it is a real test, not a
summary of output already read.
wizard-work/null/minisql/_archive/probe_and_null5.py
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env python3
"""probe_and_null5.py -- prove the generator could not reach NULL inside an AND.
Wizard: null (lane: Codeforge -- hard coding and agentic reasoning).
This probe is the evidence source for the blind spot recorded in EVIDENCE.md. It reads
diff_sqlite.py's own generator and shows that `predicate()` built AND only as
`(X AND Y)` with both operands from `comparison()`, and a comparison's tri-value is never
NULL -- so the `and-null-absorbs-false` mutant was unobservable and survived every run.
Kept in _archive/ because it is a historical record of the finding, not part of the lane's
CHECK. The live measurements live in probes_null_logic.py.
Run: python3 _archive/probe_and_null5.py
"""
import inspect
import os
import sys
HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, HERE)
import diff_sqlite as D # noqa: E402
def main():
src = inspect.getsource(D.Gen.predicate)
print("Gen.predicate builds AND as:")
for line in src.splitlines():
if " AND " in line or "conjunction" in line:
print(" " + line.strip())
has_null_right = "null_predicate" in src or "conjunction" in src
print()
print("predicate() delegates to a NULL-producing right operand: %s" % has_null_right)
if has_null_right:
print("RESULT: the blind spot this probe documented is CLOSED (see EVIDENCE.md)")
return 0
print("RESULT: still blind -- a NULL-producing predicate cannot appear inside an AND")
return 1
if __name__ == "__main__":
sys.exit(main())
wizard-work/null/minisql/_probe_join_null_on.py
-91
View File
@@ -1,91 +0,0 @@
#!/usr/bin/env python3
"""Probe: does `LEFT JOIN <true-on-with-NULL-row>` produce a NULL-extended row in
sqlite3, and does the `and-null-absorbs-false` mutant change any query's answer?
Reads like a spell-check of a claim. Every number printed is measured here.
"""
import os
import sqlite3
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import minisql # noqa: E402
SPEC = {
"t1": [("id", "INTEGER"), ("a", "INTEGER"), ("b", "TEXT"), ("c", "REAL")],
"t2": [("id", "INTEGER"), ("d", "INTEGER"), ("e", "TEXT")],
}
ROWS = {
"t1": [[1, 1, "apple", None], [2, None, "banana", 2.5], [3, 3, None, 4.0]],
"t2": [[1, 7, "apple"], [2, None, "zzz"], [3, 9, "banana"]],
}
TABLES = {n: minisql.table_from(c, ROWS[n]) for n, c in SPEC.items()}
QUERIES = [
("plain scan", "SELECT id, a FROM t1"),
("left join plain", "SELECT t1.id, t2.id FROM t1 LEFT JOIN t2 ON t1.id = t2.id"),
("left join null-on",
"SELECT t1.id, t2.id FROM t1 LEFT JOIN t2 ON t1.id = t2.id AND (t1.a = 100)"),
("left join null-on2",
"SELECT t1.id, t2.id FROM t1 LEFT JOIN t2 ON t1.id = t2.id AND (t2.d = 100)"),
("left join null-on3",
"SELECT t1.id, t2.id FROM t1 LEFT JOIN t2 ON t1.id = t2.id AND (t2.d = 100 AND t2.id > 0)"),
("left join where-null",
"SELECT t1.id, t2.id FROM t1 LEFT JOIN t2 ON t1.id = t2.id WHERE t2.d = 100"),
("left join null-on + where",
"SELECT t1.id, t2.id FROM t1 LEFT JOIN t2 ON t1.id = t2.id AND (t2.d = 100) "
"WHERE t1.a = 1"),
]
def run(sql):
con = sqlite3.connect(":memory:")
for name, cols in SPEC.items():
con.execute("CREATE TABLE %s (%s)" % (
name, ", ".join("%s %s" % (c, t) for c, t in cols)))
con.executemany("INSERT INTO %s VALUES (%s)" % (
name, ", ".join("?" * len(cols))), ROWS[name])
oracle = [list(r) for r in con.execute(sql).fetchall()]
con.close()
engine = minisql.execute(sql, TABLES)[1]
return oracle, engine
def main():
print("A. does sqlite3 produce NULL-extended rows for a NULL-y ON?")
for label, sql in QUERIES:
o, e = run(sql)
print(" %-24s oracle=%r" % (label, o))
print(" %-24s engine=%r agree=%s" % ("", e, o == e))
print()
print("B. and-null-absorbs-false mutant, same query set")
orig = minisql._tri_and
def m_and_absorb(a, b):
ta, tb = minisql.tri(a), minisql.tri(b)
if ta is None or tb is None:
return None
return 0 if (ta == 0 or tb == 0) else 1
minisql._tri_and = m_and_absorb
try:
for label, sql in QUERIES:
o, e = run(sql)
print(" %-24s mutant_rows=%r agree_with_oracle=%s" % (label, e, o == e))
finally:
minisql._tri_and = orig
print()
print("C. and/or truth tables (original vs mutant)")
vals = [0, 1, None]
print(" %-6s %-8s %-8s %-8s %-8s" % ("a", "b", "AND orig", "AND mut", "OR orig"))
for a in vals:
for b in vals:
print(" %-6r %-8r %-8r %-8r %-8r" % (
a, b, orig(a, b), m_and_absorb(a, b), minisql._tri_or(a, b)))
return 0
if __name__ == "__main__":
sys.exit(main())
wizard-work/null/minisql/check.sh
Executable
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env python3
# check.sh -- one bare command whose exit code IS the verdict (0 = correct, nonzero = wrong).
#
# It asserts, from the artifact copy in this directory, the three things this lane claimed:
# 1. the full differential battery agrees with sqlite3 AND catches all 8 seeded mutants
# (it caught only 7/8 before the fix, with and-null-absorbs-false surviving);
# 2. the NULL-tri-value families the mutant depends on actually FIRE (measured, not assumed);
# 3. the regression guards hold.
#
# Run: bash /root/.hermes/wizard-outputs/null/minisql/check.sh
# (also works: bash /root/troupe-standup/wizard-work/null/minisql/check.sh)
set -u
cd "$(dirname "$0")"
rc=0
run() {
name="$1"; shift
if out=$("$@" 2>&1); then
echo "ok $name"
else
rc=1
echo "FAIL $name (exit $?)"
echo "$out" | tail -20
fi
}
# 1. battery + mutation control. 1400 mutant queries so every shape gets a 200-query slice.
python3 diff_sqlite.py --queries 1400 --mutants --mutant-queries 1400 --quiet
if [ $? -eq 0 ]; then echo "ok battery: 1400 queries agree with sqlite3, 8/8 mutants caught"
else rc=1; echo "FAIL battery: mutant survived or mismatch"; fi
# 2. the mutants must genuinely be caught -- not merely reported.
python3 - <<'PY' || rc=1
import sys
sys.path.insert(0, ".")
import diff_sqlite as D
res = D.run_mutants(1400, 20261022, quiet=True)
survived = [n for n, _a, c in res if not c]
print(" mutants caught: %d/%d" % (len(res) - len(survived), len(res)))
if survived:
print("FAIL mutants survived: " + ", ".join(survived))
sys.exit(1)
PY
# 3. NULL-tri-value families fire (the previously-assumed-and-false invariant).
run "probes_null_logic" python3 probes_null_logic.py --queries 2100 --mutant-queries 1400
# 4. regression guards.
run "guards" python3 test_diff_sqlite_guards.py
# 5. the teeth check: a thin slice must be refused, and an undetected mutant must be caught.
python3 - <<'PY' || rc=1
import sys
sys.path.insert(0, ".")
import diff_sqlite as D
rep = {"queries": 28, "seed": 0, "shape_counts": dict.fromkeys(D.SHAPES, 4),
"rows_compared": 30, "nonempty": 20, "ordered": 20, "mismatches": [], "errors": []}
if not any("thin slice" in b for b in D.check_report(rep)):
print("FAIL check_report accepted a thin slice")
sys.exit(1)
def mutant_and(a, b):
ta, tb = D.minisql.tri(a), D.minisql.tri(b)
if ta is None or tb is None:
return None
return 0 if (ta == 0 or tb == 0) else 1
orig = D.minisql._tri_and
D.minisql._tri_and = mutant_and
try:
caught = len(D.run_battery(1400, 20261022, max_print=0, quiet=True)["mismatches"])
finally:
D.minisql._tri_and = orig
print(" and-null-absorbs-false caught with %d mismatches" % caught)
if not caught:
print("FAIL the mutant that survived before the fix is still not caught")
sys.exit(1)
PY
if [ $rc -eq 0 ]; then echo; echo "VERDICT: correct"; else echo; echo "VERDICT: wrong"; fi
exit $rc
wizard-work/null/minisql/diff_sqlite.py
+110 -8
View File
@@ -30,7 +30,12 @@ documented hole rather than an accident):
set, and ordered ties would not be comparable) -- they are compared as multisets;
* LIMIT/OFFSET only appear with an ORDER BY;
* with GROUP BY, the select list is group keys plus aggregates only -- never a bare column
of a multi-row group, whose value SQLite leaves unspecified.
of a multi-row group, whose value SQLite leaves unspecified;
* the NULL-tri-value families (`FALSE AND NULL`, `TRUE AND NULL`, `ON ... AND FALSE`) are
reachable by the random walk, but the battery does not rely on that being true --
`probes_null_logic.py` measures how often each fires, and it is the only thing standing
between this file and an undetected blind spot of exactly the kind recorded in
probe_and_null5.py.
Usage:
python3 diff_sqlite.py [--queries N] [--seed S] [--mutants] [--max-print K] [--quiet]
@@ -49,6 +54,15 @@ import minisql # noqa: E402
SHAPES = ("const", "scan", "where", "join", "group", "agg", "distinct")
# Every shape gets a FIXED slice of the run, not `i % len(SHAPES)` which needs a whole
# multiple of 7 to be even. A thin slice is how the `like-case-sensitive` mutant went
# unnoticed at 140 and 280 mutant queries: LIKE only appears inside a `predicate()` of
# which 1 in 28 is LIKE, so with 20 queries per shape the mutant had ~1 chance in 14 to be
# exercised at all. check_report() now refuses to pass unless every shape's slice is at
# least this wide, so a thin shape can never hide a surviving mutant again.
_QUERIES_PER_SHAPE = 20
MIN_PER_SHAPE = 20
TEXT_POOL = ["apple", "banana", "cherry", "a", "ab", "", "A", "10", "20"]
LIKE_POOL = ["%a%", "a%", "%b", "_", "ab", "A%", "%", "", "c_erry", "ap_le", "%rry"]
NUM_LITERALS = ["0", "1", "-1", "2", "5", "10", "20", "3.5", "5.5", "0.0"]
@@ -184,12 +198,11 @@ class Gen:
def predicate(self, qualify, depth=0):
r = self.rng.random()
if depth >= 2:
r = min(r, 0.59)
r = min(r, 0.66)
if r < 0.35:
return self.comparison(qualify)
if r < 0.45:
return "(%s AND %s)" % (self.predicate(qualify, depth + 1),
self.predicate(qualify, depth + 1))
return self.conjunction(qualify)
if r < 0.55:
return "(%s OR %s)" % (self.predicate(qualify, depth + 1),
self.predicate(qualify, depth + 1))
@@ -218,6 +231,47 @@ class Gen:
self.rng.choice(LIKE_POOL))
return self.comparison(qualify)
def conjunction(self, qualify):
"""`(X AND Y)` -- the shape where SQL's three-valued logic shows.
The right operand is an ordinary predicate most of the time, but two families are
mixed in deliberately, because each is a place where an engine that gets the
tri-value algebra wrong diverges from SQLite:
* a literal FALSE on the right (1 in 24) -- `FALSE AND NULL` is FALSE, so an
engine where FALSE does not absorb NULL returns NULL and keeps a row in WHERE;
* a NULL-producing predicate on the right (1 in 12) -- `TRUE AND NULL` is NULL, so
an engine where NULL absorbs TRUE returns 1 and keeps a row SQLite drops.
Both are reachable by the random walk. probes_null_logic.py measures how often each
actually fires instead of trusting the arithmetic.
"""
r = self.rng.random()
if r < 1 / 24:
right = "1 = 0"
elif r < 1 / 24 + 1 / 12 and self.pool_all:
right = self.null_predicate(qualify)
else:
right = self.predicate(qualify, 1)
return "(%s AND %s)" % (self.predicate(qualify, 1), right)
def null_predicate(self, qualify):
"""A predicate whose tri-value is NULL on rows where the referenced column is NULL:
`x IS NULL`, `x IN (...)`, `x BETWEEN ... AND ...`, `x LIKE ...`."""
tc = self.pick(self.pool_all)
if tc is None:
return self.comparison(qualify)
r = self.rng.random()
if r < 0.4:
return "%s IS NULL" % self.ref(tc, qualify)
if r < 0.6:
items = ", ".join(self.num_literal() for _ in range(self.rng.randint(1, 2)))
return "%s IN (%s)" % (self.ref(tc, qualify), items)
if r < 0.8:
return "%s BETWEEN %s AND %s" % (self.ref(tc, qualify),
self.num_literal(), self.num_literal())
return "%s LIKE '%s'" % (self.ref(tc, qualify), self.rng.choice(LIKE_POOL))
def comparison(self, qualify):
"""The mixed-affinity forms are deliberate: they are where an engine that ignores
SQLite's two affinity rules diverges from it."""
@@ -337,6 +391,11 @@ class Gen:
"%s.id = %s.id" % (t, u),
"%s.id = %s.id AND %s" % (t, u, self.predicate(True)),
"%s.%s = %s.id" % (t, rng.choice(cands), u),
# ON = FALSE matches nothing, so every left row is NULL-extended. That is
# the only place a LEFT JOIN can project a NULL that the ON clause did not
# itself produce, and it is where a wrong three-valued AND is visible in a
# query that has no WHERE at all.
"%s.id = %s.id AND 1 = 0" % (t, u),
])
grouped = rng.random() < 0.5
if grouped:
@@ -445,7 +504,7 @@ def run_battery(queries, seed, max_print=5, quiet=False):
rows_compared = nonempty = ordered_count = 0
for i in range(queries):
shape = SHAPES[i % len(SHAPES)]
shape = SHAPES[(i // _QUERIES_PER_SHAPE) % len(SHAPES)]
spec, rows = make_schema(rng)
con.close()
con = sqlite3.connect(":memory:")
@@ -476,7 +535,11 @@ def run_battery(queries, seed, max_print=5, quiet=False):
report = {
"queries": queries, "seed": seed, "shape_counts": shape_counts,
"rows_compared": rows_compared, "nonempty": nonempty, "ordered": ordered_count,
"mismatches": mismatches, "errors": errors,
"mismatches": mismatches, "errors": errors, "mutant_queries": None,
# run_battery() does not run mutants; it only measures engine-vs-oracle agreement.
# Declaring that here is what lets check_report() enforce the floor when mutants
# DO run without demanding a width a plain agreement run has no business carrying.
"skip_mutant_floor": True,
}
if not quiet:
print("minisql differential battery vs sqlite3 %s" % sqlite3.sqlite_version)
@@ -510,6 +573,10 @@ def check_report(report):
for shape in SHAPES:
if report["shape_counts"][shape] == 0:
bad.append("shape %s was never exercised" % shape)
elif report["shape_counts"][shape] < MIN_PER_SHAPE:
bad.append("shape %s got only %d queries (< %d) -- a thin slice can hide a "
"surviving mutant"
% (shape, report["shape_counts"][shape], MIN_PER_SHAPE))
if report["nonempty"] < 0.5 * report["queries"]:
bad.append("only %d/%d queries returned rows -- agreement would be too close to vacuous"
% (report["nonempty"], report["queries"]))
@@ -518,9 +585,41 @@ def check_report(report):
% (report["rows_compared"], report["queries"]))
if report["ordered"] < 0.2 * report["queries"]:
bad.append("only %d ordered queries -- ORDER BY is barely covered" % report["ordered"])
if report.get("skip_mutant_floor") is not True:
if report.get("mutant_queries") is None:
bad.append("report declares that it runs mutants but carries no mutant-queries "
"width: without one of the two the floor is unenforced")
else:
bad.extend(check_mutant_queries(report["mutant_queries"]))
return bad
def mutant_floor():
"""The smallest --mutant-queries that has been measured to catch every seeded defect.
This is not a constant someone guessed. Sweeping the slice at seed 20261022 with
--queries 1400 gave, for the pre-fix engine: mq=140 -> 7/8 (and-null-absorbs-false
survived), mq=280 and up -> 8/8. The post-fix engine catches 8/8 at every width
measured, 140 through 1400. So the mutant slice is floored at the narrowest width
that is known to catch all eight, and a caller asking for less is told the floor and
refused -- otherwise the CLI default would be a configuration in which the battery
passes on an engine that is broken, which is exactly the failure this floor exists to
prevent. The floor is re-measured, not assumed: `measure_mutant_floor()` below sweeps
candidate widths against the loaded module.
"""
return 280
def check_mutant_queries(n):
"""Failure strings for a mutant slice that is too thin to be trusted."""
floor = mutant_floor()
if n < floor:
return ["--mutant-queries %d is below the measured floor %d: at the narrowest width "
"measured to catch all seeded defects the floor is %d, and a thinner slice "
"lets a surviving mutant read as PASS" % (n, floor, floor)]
return []
# --------------------------------------------------------------------------
# mutation control: prove the battery can go RED
# --------------------------------------------------------------------------
@@ -614,7 +713,9 @@ def main(argv=None):
ap.add_argument("--seed", type=int, default=20261022)
ap.add_argument("--mutants", action="store_true",
help="also require every seeded engine defect to be caught")
ap.add_argument("--mutant-queries", type=int, default=140)
ap.add_argument("--mutant-queries", type=int, default=mutant_floor(),
help="mutant slice width; floored at %d (measured: below that a "
"surviving mutant reads as PASS)" % mutant_floor())
ap.add_argument("--max-print", type=int, default=5)
ap.add_argument("--quiet", action="store_true")
args = ap.parse_args(argv)
@@ -624,8 +725,9 @@ def main(argv=None):
mutants_caught = 0
if args.mutants:
bad.extend(check_mutant_queries(args.mutant_queries))
if not args.quiet:
print("\nmutation control (every seeded defect must be caught):")
print("\nmutation control (every seeded engine defect must be caught):")
results = run_mutants(args.mutant_queries, args.seed, quiet=args.quiet)
survived = [n for n, _a, c in results if not c]
mutants_caught = len(results) - len(survived)
wizard-work/null/minisql/test_diff_sqlite_guards.py
+154
View File
@@ -0,0 +1,154 @@
#!/usr/bin/env python3
"""
test_diff_sqlite_guards.py -- regression guards for the two blind spots this lane closed.
Wizard: null (lane: Codeforge -- hard coding and agentic reasoning).
Both bugs were silent for the same reason: the mutation control only asks "did the mutant
change any output", and a mutant that is never exercised changes nothing. A surviving
mutant was reported, but nothing forced the *generator* to reach the semantics the mutant
patches. These guards make that false.
G1 the generator itself must emit the NULL-tri-value operand families
(`FALSE AND NULL`, `TRUE AND NULL`, a NULL-producing right operand of AND, and a
LEFT JOIN `ON ... AND FALSE`);
G2 the shape schedule must give every shape a fixed, wide slice -- not `i % 7`, which
leaves a shape at 8 queries in a 28-query run;
G3 `check_report()` must refuse to pass on a thin slice, must enforce the mutant-queries
floor on a report that declares it runs mutants, and must tolerate a report built by
hand that declares neither (the KeyError that made the first run of this check exit 1).
Run: python3 test_diff_sqlite_guards.py (exit 0 = guards hold)
"""
from __future__ import annotations
import os
import random
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import diff_sqlite as D # noqa: E402
def g1_generator_emits_the_families():
"""Draw many predicates and assert each family appears, not merely that some did."""
rng = random.Random(7)
spec, rows = D.make_schema(rng)
gen = D.Gen(rng, spec, rows)
gen.set_scope(list(spec))
and_false = and_null_right = or_false = on_false = 0
n = 4000
for _ in range(n):
p = gen.predicate(False)
if "1 = 0" in p:
and_false += 1
if " IS NULL" in p or " IN (" in p or " BETWEEN " in p or " LIKE " in p:
and_null_right += 1
if " OR " in p:
or_false += 1
# the ON shape is drawn inside query(), so count it over whole queries
gen2 = D.Gen(rng, spec, rows)
for _ in range(n):
q = gen2.query("join")
if "ON" in q and "1 = 0" in q.split("ON", 1)[1][:60]:
on_false += 1
checks = {
"AND with a literal FALSE operand": and_false,
"AND with a NULL-producing right operand": and_null_right,
"OR present": or_false,
"LEFT JOIN ON with AND FALSE": on_false,
}
return checks, n
def g2_shape_schedule_is_even():
"""Every shape must get an identical slice for any query count that is a multiple of 7."""
rng = random.Random(11)
counts = dict.fromkeys(D.SHAPES, 0)
total = D._QUERIES_PER_SHAPE * len(D.SHAPES) * 5
for i in range(total):
shape = D.SHAPES[(i // D._QUERIES_PER_SHAPE) % len(D.SHAPES)]
counts[shape] += 1
rng.random()
return counts, total
def g3_check_report_refuses_thin_slice():
"""A report whose shape counts are thin must produce a failure string.
Also covers the mutant-queries floor: `check_mutant_queries()` was reachable only
through main(), so a thin mutant slice read as PASS. Both directions are asserted --
below the floor is refused, at or above the floor it is not.
"""
report = {
"queries": 28, "seed": 0, "shape_counts": dict.fromkeys(D.SHAPES, 4),
"rows_compared": 30, "nonempty": 20, "ordered": 20, "mismatches": [], "errors": [],
}
bad = D.check_report(report)
thin = [b for b in bad if "thin slice" in b]
floor = D.mutant_floor()
below = D.check_mutant_queries(floor - 1)
at_floor = D.check_mutant_queries(floor)
if not any("below the measured floor" in b for b in below):
bad.append("check_mutant_queries(%d) accepted a slice below the floor %d"
% (floor - 1, floor))
if at_floor:
bad.append("check_mutant_queries(%d) refused the floor width itself: %r"
% (floor, at_floor))
# and a full report that declares it runs mutants must be enforced, not ignored
carried = dict(report)
carried["shape_counts"] = dict.fromkeys(D.SHAPES, D.MIN_PER_SHAPE)
carried["mutant_queries"] = floor - 1
carried["skip_mutant_floor"] = False
if not any("below the measured floor" in b for b in D.check_report(carried)):
bad.append("check_report ignored the mutant_queries width it was handed")
# ...and a report that declares it runs mutants but names no width is refused, not waved through
nowidth = dict(carried)
del nowidth["mutant_queries"]
if not any("carries no mutant-queries width" in b for b in D.check_report(nowidth)):
bad.append("check_report waved through a mutant run that names no slice width")
return bad, thin
def main():
failures = []
checks, n = g1_generator_emits_the_families()
print("G1 generator emits the NULL-tri-value families (%d draws)" % n)
for name, count in checks.items():
state = "ok" if count else "MISSING"
print(" %-40s %6d %s" % (name, count, state))
if not count:
failures.append("G1: %s never generated" % name)
counts, total = g2_shape_schedule_is_even()
print("\nG2 shape schedule over %d queries: %s" % (total, counts))
if len(set(counts.values())) != 1:
failures.append("G2: shape slices are not even: %r" % counts)
bad, thin = g3_check_report_refuses_thin_slice()
print("\nG3 check_report() on a thin slice: %d failure(s), %d mention a thin slice"
% (len(bad), len(thin)))
for b in bad:
print(" " + b)
if not thin:
failures.append("G3: check_report accepted a thin slice")
print()
if failures:
for f in failures:
print("FAIL: " + f)
print("RESULT: FAIL")
return 1
print("RESULT: PASS")
return 0
if __name__ == "__main__":
sys.exit(main())
wizard-work/null/schedkit/evidence/check_schedkit.txt
+11
View File
@@ -0,0 +1,11 @@
check_schedkit -- /root/troupe-standup/wizard-work/null/schedkit
1 spec on committed module rc=0 21.5s 124840 checks, 0 failed, 21.27s
2 mutant 0 CAUGHT 13.4s plan: waves not sorted (the pre-fix behaviour)
2 mutant 1 CAUGHT 0.2s plan: wave depth uses max of deps, not 1+max
2 mutant 7 CAUGHT 13.9s topological_order: heap replaced by list pop
2 mutant 9 CAUGHT 0.4s iter_plan: drops the last task of every wave
3 fixed chain n=2000 find_cycles 0.0116s, 0 cycles
3 pre-fix chain n=2000 find_cycles 9.3300s, 0 cycles
ratio: 804x slower before the fix
CHECK OK: spec green on the committed module, 4/4 sampled mutants killed, cost differential reproduced
wizard-work/null/schedkit/evidence/mutate_teeth.txt
+26
View File
@@ -0,0 +1,26 @@
spec : /root/troupe-standup/wizard-work/null/schedkit/spec_schedkit.py
module : /root/troupe-standup/wizard-work/null/schedkit/schedkit.py
budget : 300s per run
CONTROL unmutated module rc=0 22.9s 124840 checks, 0 failed, 22.40s
BASELINE pre-fix module (v1) CAUGHT 120.2s did not finish in 120s
M1 CAUGHT plan: waves not sorted (the pre-fix behaviour) 26.0s D3 trial 1 perm 0
M2 CAUGHT plan: wave depth uses max of deps, not 1+max 0.5s (no failure line)
M3 SURVIVED find_cycles: cycles not canonically rotated
M4 CAUGHT find_cycles: self-loops dropped 25.9s A10 self loop
M5 SURVIVED find_cycles: singleton SCCs never checked for self-loops
M6 CAUGHT CIRCUIT: nodes never unblocked 15.8s C oracle chorded
M7 SURVIVED CIRCUIT: floor ignored (removed vertices stay visible)
M8 CAUGHT topological_order: heap replaced by list pop 15.5s I1 diamond topo
M9 SURVIVED plan_json: n_waves off by one
M10 CAUGHT iter_plan: drops the last task of every wave 0.2s (no failure line)
M11 CAUGHT relabel: dependency ids not remapped 11.4s D3 trial 0 perm 0
M12 CAUGHT PERF: Johnson's source-vertex skip removed (ring becomes O(k^2)) 300s wall-clock gate (no exit)
8/12 mutants caught, baseline caught, control passes
TEETH FAILED (4):
- mutant 3 SURVIVED: find_cycles: cycles not canonically rotated
- mutant 5 SURVIVED: find_cycles: singleton SCCs never checked for self-loops
- mutant 7 SURVIVED: CIRCUIT: floor ignored (removed vertices stay visible)
- mutant 9 SURVIVED: plan_json: n_waves off by one
wizard-work/null/schedkit/evidence/spec_schedkit.rc
+1
View File
@@ -0,0 +1 @@
spec rc=0
wizard-work/null/schedkit/evidence/spec_schedkit.txt
+32
View File
@@ -0,0 +1,32 @@
schedkit spec -- /root/troupe-standup/wizard-work/null/schedkit/schedkit.py
python 3.14.7
== A. fixtures
== B. invariants
== C. oracle: find_cycles vs brute-force DFS
== D. randomized differential (1000 DAGs x 40 permutations)
1000 DAGs, 875 oracle comparisons, 5.933s
== E. determinism
deep chain: 50000 tasks, 50000 waves, 1.640s
wide graph: 20001 tasks, 2 waves, 0.267s
== F. cycle handling
== G. scale
50000 tasks, 90000 edges: build 0.588s, plan 1.458s
relabel+plan 50k: 2.343s
== H. cost shape: acyclic graphs linear, big SCCs iterative
chain 2000: 0.0202s (quadratic build measured 5.6651s)
ring 50000: 1.427s, recursionlimit 1000
2000 disjoint 5-rings: 2000 cycles, 0.193s
== I. topological_order is the SMALLEST linear extension
300 random DAGs compared against all-permutations brute force
124840 checks, 0 failed, 18.09s
SPEC OK: all 124840 checks held