# minisql differential battery — EVIDENCE

Wizard: null (lane: Codeforge — hard coding and agentic reasoning).
Oracle: sqlite3 3.53.1 (`python3 -V` 3.14.7). All numbers below are from real runs; nothing here is projected.

## What this is

`minisql.py` is a hand-written SQL engine (parser → evaluator → relation algebra).
`diff_sqlite.py` is the randomized differential battery that compares it against sqlite3 on
identical SQL text, plus a mutation control that seeds eight known engine defects and
requires each one to be caught.

## The blind spot, and its proof

`probe_and_null5.py` proved, from SQL's own semantics, that the generator could never put a
NULL-producing predicate inside an AND: `predicate()` built AND only as `(X AND Y)` with
both operands from `comparison()`, and a comparison's tri-value is never NULL.

Consequence, measured: the mutant

    def m_and_absorb(a, b):
        ta, tb = minisql.tri(a), minisql.tri(b)
        if ta is None or tb is None: return None   # NULL absorbs FALSE

survived every run. That is a battery with no teeth, not a passing test.

## The fix

`Gen.conjunction()` now emits the two operand families that make three-valued AND observable:

* a literal FALSE on the right (1 in 24) — `FALSE AND NULL` is FALSE in SQLite, so an engine
  where FALSE does not absorb NULL returns NULL and keeps a row;
* a NULL-producing predicate on the right (1 in 12) — `TRUE AND NULL` is NULL, so an engine
  where NULL absorbs TRUE returns 1 and keeps a row SQLite drops.

`Gen.query("join")` also emits `ON <cond> AND 1 = 0`: a FALSE ON matches nothing, so every
left row is NULL-extended — the only shape where a NULL the ON clause did not produce
survives to the output.

## Measured coverage (probes_null_logic.py, 2100 queries, seed 20261022)

    _tri_and calls                            : 7892
    F1  _tri_and(NULL, FALSE)                  : 108  FIRED
    F2  _tri_and(FALSE, NULL)                  : 444  FIRED
    F3  _tri_and(TRUE, NULL) [not observable]  : 180  FIRED
    F4  LEFT JOIN ON .. AND FALSE + IS NULL    :   3  FIRED

    B. mutant patched: battery mismatches 5  CAUGHT
    C. unpatched battery: 0 mismatches, 3770 rows compared, 0 non-vacuity failures

## The second blind spot (sampling, not semantics)

With the first mutant closed, `like-case-sensitive` survived at 140 and 280 mutant queries
and was caught at 560 and 1400. Measured across `--mutant-queries` 140 / 280 / 560 / 1400:

    140  -> SURVIVED
    280  -> SURVIVED
    560  -> CAUGHT (4 mismatches)
    1400 -> CAUGHT (11 mismatches)

Cause: `LIKE` appears only inside `predicate()`, of which it is 1 in 28, and the shape
schedule was `i % 7` — so 140 queries gave 20 per shape, i.e. ~20 chances in 28 to draw a
LIKE at all, spread over every engine feature. Nothing in the battery noticed that a thin
slice can hide a surviving mutant.

Fix, two parts:
1. `SHAPES[(i // _QUERIES_PER_SHAPE) % len(SHAPES)]` — every shape gets a fixed 20-query
   slice per round instead of a count that depends on the run length.
2. `check_report()` now refuses to pass unless every shape's slice is at least
   `MIN_PER_SHAPE` (20) wide. A 28-query run fails loudly:

       FAIL: shape scan got only 8 queries (< 20) -- a thin slice can hide a surviving mutant
       FAIL: shape where was never exercised
       ...
       RESULT: FAIL

## Regression guards (test_diff_sqlite_guards.py)

    G1  generator emits the NULL-tri-value families (4000 draws)
        AND with a literal FALSE operand             27  ok
        AND with a NULL-producing right operand    2123  ok
        OR present                                  519  ok
        LEFT JOIN ON with AND FALSE                1014  ok
    G2  shape schedule over 700 queries: every shape exactly 100
    G3  check_report() on a thin slice: 8 failures, 7 naming the thin slice, plus the
        mutant-floor contract (see below)
    RESULT: PASS

## The check_report() KeyError, and the contract it exposed

The first run of `check.sh` exited 1 with

    File ".../diff_sqlite.py", line 567, in check_report
        if report["mutant_queries"] is not None:
    KeyError: 'mutant_queries'

`check_report()` read `mutant_queries` with `[]`, but a report built by hand in
`test_diff_sqlite_guards.py` (and in `check.sh`) does not carry the key — and neither does any
caller that merely wants the vacuity guards. The floor was therefore unreachable except through
`main()`, which is the same shape of defect the floor itself exists to catch: a guard that only
runs on one path.

Contract now, stated in the function and asserted by G3:

  * a report that declares `skip_mutant_floor: True` (what `run_battery()` produces, since it
    measures engine-vs-oracle agreement and runs no mutants) is not asked for a width;
  * a report that declares it runs mutants must carry `mutant_queries`, and that width is
    enforced against `mutant_floor()` — a missing width is refused, not waved through;
  * a report that carries `mutant_queries` and does not skip the floor has the width enforced.

G3 asserts all four directions: `check_mutant_queries(floor-1)` refused, `check_mutant_queries(floor)`
accepted, a report carrying a sub-floor width refused, and a report declaring a mutant run with no
width refused.

## Full battery, after the fix

    $ python3 diff_sqlite.py --queries 1400 --mutants --mutant-queries 1400
    minisql differential battery vs sqlite3 3.53.1
    seed=20261022  queries=1400
    shape counts: const=200 scan=200 where=200 join=200 group=200 agg=200 distinct=200
    ordered queries: 800   rows compared: 2515   queries returning >=1 row: 926

      null-eq                  CAUGHT (252)   left-join-as-inner    CAUGHT ( 47)
      nulls-last-asc           CAUGHT ( 97)   sum-empty-zero        CAUGHT ( 31)
      count-counts-nulls       CAUGHT ( 25)   int-div-becomes-real  CAUGHT ( 22)
      and-null-absorbs-false   CAUGHT (  5)   like-case-sensitive   CAUGHT (  6)

    mutants caught: 8/8
    RESULT: PASS

Before the fix, the same battery reported `mutants caught: 7/8` with
`and-null-absorbs-false` surviving, and `RESULT: FAIL`.

## Commands

    python3 diff_sqlite.py --queries 1400 --mutants --mutant-queries 1400
    python3 probes_null_logic.py --queries 2100 --mutant-queries 1400
    python3 test_diff_sqlite_guards.py
    bash check.sh          # the whole lane, one command: battery + mutants + floor + guards

## Known holes, stated plainly

* `probes_null_logic.py` counts `_tri_and` calls at the instrumented call site, so F1/F2
  are counts of the call, not of a verified output divergence per call. The output
  divergence itself is what the battery measures; the probe only proves the semantics are
  *reachable*, which is what was previously assumed and false.
* F4 (`ON ... AND FALSE` plus an `IS NULL` filter) fires rarely (3 in 2100). It is not
  load-bearing: F1/F2 fire in the hundreds.
* The generator still emits no bare column of a multi-row group with GROUP BY, because
  SQLite leaves that value unspecified — a documented hole, not a fixed one.
* `check.sh` is the lane's CHECK line. It runs four independent verifications (battery +
  mutation control at 1400 queries, the mutants genuinely caught, the NULL-tri-value families
  firing, the regression guards) and exits nonzero if any one fails — it is a real test, not a
  summary of output already read.