troupe-standup/wizard-work/null/minisql/EVIDENCE.md

minisql differential battery — EVIDENCE

Wizard: null (lane: Codeforge — hard coding and agentic reasoning). Oracle: sqlite3 3.53.1 (python3 -V 3.14.7). All numbers below are from real runs; nothing here is projected.

What this is

minisql.py is a hand-written SQL engine (parser → evaluator → relation algebra). diff_sqlite.py is the randomized differential battery that compares it against sqlite3 on identical SQL text, plus a mutation control that seeds eight known engine defects and requires each one to be caught.

The blind spot, and its proof

probe_and_null5.py proved, from SQL's own semantics, that the generator could never put a NULL-producing predicate inside an AND: predicate() built AND only as (X AND Y) with both operands from comparison(), and a comparison's tri-value is never NULL.

Consequence, measured: the mutant

def m_and_absorb(a, b):
    ta, tb = minisql.tri(a), minisql.tri(b)
    if ta is None or tb is None: return None   # NULL absorbs FALSE

survived every run. That is a battery with no teeth, not a passing test.

The fix

Gen.conjunction() now emits the two operand families that make three-valued AND observable:

• a literal FALSE on the right (1 in 24) — FALSE AND NULL is FALSE in SQLite, so an engine where FALSE does not absorb NULL returns NULL and keeps a row;
• a NULL-producing predicate on the right (1 in 12) — TRUE AND NULL is NULL, so an engine where NULL absorbs TRUE returns 1 and keeps a row SQLite drops.

Gen.query("join") also emits ON <cond> AND 1 = 0: a FALSE ON matches nothing, so every left row is NULL-extended — the only shape where a NULL the ON clause did not produce survives to the output.

Measured coverage (probes_null_logic.py, 2100 queries, seed 20261022)

_tri_and calls                            : 7892
F1  _tri_and(NULL, FALSE)                  : 108  FIRED
F2  _tri_and(FALSE, NULL)                  : 444  FIRED
F3  _tri_and(TRUE, NULL) [not observable]  : 180  FIRED
F4  LEFT JOIN ON .. AND FALSE + IS NULL    :   3  FIRED

B. mutant patched: battery mismatches 5  CAUGHT
C. unpatched battery: 0 mismatches, 3770 rows compared, 0 non-vacuity failures

The second blind spot (sampling, not semantics)

With the first mutant closed, like-case-sensitive survived at 140 and 280 mutant queries and was caught at 560 and 1400. Measured across --mutant-queries 140 / 280 / 560 / 1400:

140  -> SURVIVED
280  -> SURVIVED
560  -> CAUGHT (4 mismatches)
1400 -> CAUGHT (11 mismatches)

Cause: LIKE appears only inside predicate(), of which it is 1 in 28, and the shape schedule was i % 7 — so 140 queries gave 20 per shape, i.e. ~20 chances in 28 to draw a LIKE at all, spread over every engine feature. Nothing in the battery noticed that a thin slice can hide a surviving mutant.

Fix, two parts:

1. SHAPES[(i // _QUERIES_PER_SHAPE) % len(SHAPES)] — every shape gets a fixed 20-query slice per round instead of a count that depends on the run length.

2. check_report() now refuses to pass unless every shape's slice is at least MIN_PER_SHAPE (20) wide. A 28-query run fails loudly:

   FAIL: shape scan got only 8 queries (< 20) -- a thin slice can hide a surviving mutant
   FAIL: shape where was never exercised
   ...
   RESULT: FAIL
   

Regression guards (test_diff_sqlite_guards.py)

G1  generator emits the NULL-tri-value families (4000 draws)
    AND with a literal FALSE operand             27  ok
    AND with a NULL-producing right operand    2123  ok
    OR present                                  519  ok
    LEFT JOIN ON with AND FALSE                1014  ok
G2  shape schedule over 700 queries: every shape exactly 100
G3  check_report() on a thin slice: 8 failures, 7 naming the thin slice, plus the
    mutant-floor contract (see below)
RESULT: PASS

The check_report() KeyError, and the contract it exposed

The first run of check.sh exited 1 with

File ".../diff_sqlite.py", line 567, in check_report
    if report["mutant_queries"] is not None:
KeyError: 'mutant_queries'

check_report() read mutant_queries with [], but a report built by hand in test_diff_sqlite_guards.py (and in check.sh) does not carry the key — and neither does any caller that merely wants the vacuity guards. The floor was therefore unreachable except through main(), which is the same shape of defect the floor itself exists to catch: a guard that only runs on one path.

Contract now, stated in the function and asserted by G3:

• a report that declares skip_mutant_floor: True (what run_battery() produces, since it measures engine-vs-oracle agreement and runs no mutants) is not asked for a width;
• a report that declares it runs mutants must carry mutant_queries, and that width is enforced against mutant_floor() — a missing width is refused, not waved through;
• a report that carries mutant_queries and does not skip the floor has the width enforced.

G3 asserts all four directions: check_mutant_queries(floor-1) refused, check_mutant_queries(floor) accepted, a report carrying a sub-floor width refused, and a report declaring a mutant run with no width refused.

Full battery, after the fix

$ python3 diff_sqlite.py --queries 1400 --mutants --mutant-queries 1400
minisql differential battery vs sqlite3 3.53.1
seed=20261022  queries=1400
shape counts: const=200 scan=200 where=200 join=200 group=200 agg=200 distinct=200
ordered queries: 800   rows compared: 2515   queries returning >=1 row: 926

  null-eq                  CAUGHT (252)   left-join-as-inner    CAUGHT ( 47)
  nulls-last-asc           CAUGHT ( 97)   sum-empty-zero        CAUGHT ( 31)
  count-counts-nulls       CAUGHT ( 25)   int-div-becomes-real  CAUGHT ( 22)
  and-null-absorbs-false   CAUGHT (  5)   like-case-sensitive   CAUGHT (  6)

mutants caught: 8/8
RESULT: PASS

Before the fix, the same battery reported mutants caught: 7/8 with and-null-absorbs-false surviving, and RESULT: FAIL.

Commands

python3 diff_sqlite.py --queries 1400 --mutants --mutant-queries 1400
python3 probes_null_logic.py --queries 2100 --mutant-queries 1400
python3 test_diff_sqlite_guards.py
bash check.sh          # the whole lane, one command: battery + mutants + floor + guards

Known holes, stated plainly

• probes_null_logic.py counts _tri_and calls at the instrumented call site, so F1/F2 are counts of the call, not of a verified output divergence per call. The output divergence itself is what the battery measures; the probe only proves the semantics are reachable, which is what was previously assumed and false.
• F4 (ON ... AND FALSE plus an IS NULL filter) fires rarely (3 in 2100). It is not load-bearing: F1/F2 fire in the hundreds.
• The generator still emits no bare column of a multi-row group with GROUP BY, because SQLite leaves that value unspecified — a documented hole, not a fixed one.
• check.sh is the lane's CHECK line. It runs four independent verifications (battery + mutation control at 1400 queries, the mutants genuinely caught, the NULL-tri-value families firing, the regression guards) and exits nonzero if any one fails — it is a real test, not a summary of output already read.