155 lines
7.2 KiB
Markdown
155 lines
7.2 KiB
Markdown
# minisql differential battery — EVIDENCE
|
|
|
|
Wizard: null (lane: Codeforge — hard coding and agentic reasoning).
|
|
Oracle: sqlite3 3.53.1 (`python3 -V` 3.14.7). All numbers below are from real runs; nothing here is projected.
|
|
|
|
## What this is
|
|
|
|
`minisql.py` is a hand-written SQL engine (parser → evaluator → relation algebra).
|
|
`diff_sqlite.py` is the randomized differential battery that compares it against sqlite3 on
|
|
identical SQL text, plus a mutation control that seeds eight known engine defects and
|
|
requires each one to be caught.
|
|
|
|
## The blind spot, and its proof
|
|
|
|
`probe_and_null5.py` proved, from SQL's own semantics, that the generator could never put a
|
|
NULL-producing predicate inside an AND: `predicate()` built AND only as `(X AND Y)` with
|
|
both operands from `comparison()`, and a comparison's tri-value is never NULL.
|
|
|
|
Consequence, measured: the mutant
|
|
|
|
def m_and_absorb(a, b):
|
|
ta, tb = minisql.tri(a), minisql.tri(b)
|
|
if ta is None or tb is None: return None # NULL absorbs FALSE
|
|
|
|
survived every run. That is a battery with no teeth, not a passing test.
|
|
|
|
## The fix
|
|
|
|
`Gen.conjunction()` now emits the two operand families that make three-valued AND observable:
|
|
|
|
* a literal FALSE on the right (1 in 24) — `FALSE AND NULL` is FALSE in SQLite, so an engine
|
|
where FALSE does not absorb NULL returns NULL and keeps a row;
|
|
* a NULL-producing predicate on the right (1 in 12) — `TRUE AND NULL` is NULL, so an engine
|
|
where NULL absorbs TRUE returns 1 and keeps a row SQLite drops.
|
|
|
|
`Gen.query("join")` also emits `ON <cond> AND 1 = 0`: a FALSE ON matches nothing, so every
|
|
left row is NULL-extended — the only shape where a NULL the ON clause did not produce
|
|
survives to the output.
|
|
|
|
## Measured coverage (probes_null_logic.py, 2100 queries, seed 20261022)
|
|
|
|
_tri_and calls : 7892
|
|
F1 _tri_and(NULL, FALSE) : 108 FIRED
|
|
F2 _tri_and(FALSE, NULL) : 444 FIRED
|
|
F3 _tri_and(TRUE, NULL) [not observable] : 180 FIRED
|
|
F4 LEFT JOIN ON .. AND FALSE + IS NULL : 3 FIRED
|
|
|
|
B. mutant patched: battery mismatches 5 CAUGHT
|
|
C. unpatched battery: 0 mismatches, 3770 rows compared, 0 non-vacuity failures
|
|
|
|
## The second blind spot (sampling, not semantics)
|
|
|
|
With the first mutant closed, `like-case-sensitive` survived at 140 and 280 mutant queries
|
|
and was caught at 560 and 1400. Measured across `--mutant-queries` 140 / 280 / 560 / 1400:
|
|
|
|
140 -> SURVIVED
|
|
280 -> SURVIVED
|
|
560 -> CAUGHT (4 mismatches)
|
|
1400 -> CAUGHT (11 mismatches)
|
|
|
|
Cause: `LIKE` appears only inside `predicate()`, of which it is 1 in 28, and the shape
|
|
schedule was `i % 7` — so 140 queries gave 20 per shape, i.e. ~20 chances in 28 to draw a
|
|
LIKE at all, spread over every engine feature. Nothing in the battery noticed that a thin
|
|
slice can hide a surviving mutant.
|
|
|
|
Fix, two parts:
|
|
1. `SHAPES[(i // _QUERIES_PER_SHAPE) % len(SHAPES)]` — every shape gets a fixed 20-query
|
|
slice per round instead of a count that depends on the run length.
|
|
2. `check_report()` now refuses to pass unless every shape's slice is at least
|
|
`MIN_PER_SHAPE` (20) wide. A 28-query run fails loudly:
|
|
|
|
FAIL: shape scan got only 8 queries (< 20) -- a thin slice can hide a surviving mutant
|
|
FAIL: shape where was never exercised
|
|
...
|
|
RESULT: FAIL
|
|
|
|
## Regression guards (test_diff_sqlite_guards.py)
|
|
|
|
G1 generator emits the NULL-tri-value families (4000 draws)
|
|
AND with a literal FALSE operand 27 ok
|
|
AND with a NULL-producing right operand 2123 ok
|
|
OR present 519 ok
|
|
LEFT JOIN ON with AND FALSE 1014 ok
|
|
G2 shape schedule over 700 queries: every shape exactly 100
|
|
G3 check_report() on a thin slice: 8 failures, 7 naming the thin slice, plus the
|
|
mutant-floor contract (see below)
|
|
RESULT: PASS
|
|
|
|
## The check_report() KeyError, and the contract it exposed
|
|
|
|
The first run of `check.sh` exited 1 with
|
|
|
|
File ".../diff_sqlite.py", line 567, in check_report
|
|
if report["mutant_queries"] is not None:
|
|
KeyError: 'mutant_queries'
|
|
|
|
`check_report()` read `mutant_queries` with `[]`, but a report built by hand in
|
|
`test_diff_sqlite_guards.py` (and in `check.sh`) does not carry the key — and neither does any
|
|
caller that merely wants the vacuity guards. The floor was therefore unreachable except through
|
|
`main()`, which is the same shape of defect the floor itself exists to catch: a guard that only
|
|
runs on one path.
|
|
|
|
Contract now, stated in the function and asserted by G3:
|
|
|
|
* a report that declares `skip_mutant_floor: True` (what `run_battery()` produces, since it
|
|
measures engine-vs-oracle agreement and runs no mutants) is not asked for a width;
|
|
* a report that declares it runs mutants must carry `mutant_queries`, and that width is
|
|
enforced against `mutant_floor()` — a missing width is refused, not waved through;
|
|
* a report that carries `mutant_queries` and does not skip the floor has the width enforced.
|
|
|
|
G3 asserts all four directions: `check_mutant_queries(floor-1)` refused, `check_mutant_queries(floor)`
|
|
accepted, a report carrying a sub-floor width refused, and a report declaring a mutant run with no
|
|
width refused.
|
|
|
|
## Full battery, after the fix
|
|
|
|
$ python3 diff_sqlite.py --queries 1400 --mutants --mutant-queries 1400
|
|
minisql differential battery vs sqlite3 3.53.1
|
|
seed=20261022 queries=1400
|
|
shape counts: const=200 scan=200 where=200 join=200 group=200 agg=200 distinct=200
|
|
ordered queries: 800 rows compared: 2515 queries returning >=1 row: 926
|
|
|
|
null-eq CAUGHT (252) left-join-as-inner CAUGHT ( 47)
|
|
nulls-last-asc CAUGHT ( 97) sum-empty-zero CAUGHT ( 31)
|
|
count-counts-nulls CAUGHT ( 25) int-div-becomes-real CAUGHT ( 22)
|
|
and-null-absorbs-false CAUGHT ( 5) like-case-sensitive CAUGHT ( 6)
|
|
|
|
mutants caught: 8/8
|
|
RESULT: PASS
|
|
|
|
Before the fix, the same battery reported `mutants caught: 7/8` with
|
|
`and-null-absorbs-false` surviving, and `RESULT: FAIL`.
|
|
|
|
## Commands
|
|
|
|
python3 diff_sqlite.py --queries 1400 --mutants --mutant-queries 1400
|
|
python3 probes_null_logic.py --queries 2100 --mutant-queries 1400
|
|
python3 test_diff_sqlite_guards.py
|
|
bash check.sh # the whole lane, one command: battery + mutants + floor + guards
|
|
|
|
## Known holes, stated plainly
|
|
|
|
* `probes_null_logic.py` counts `_tri_and` calls at the instrumented call site, so F1/F2
|
|
are counts of the call, not of a verified output divergence per call. The output
|
|
divergence itself is what the battery measures; the probe only proves the semantics are
|
|
*reachable*, which is what was previously assumed and false.
|
|
* F4 (`ON ... AND FALSE` plus an `IS NULL` filter) fires rarely (3 in 2100). It is not
|
|
load-bearing: F1/F2 fire in the hundreds.
|
|
* The generator still emits no bare column of a multi-row group with GROUP BY, because
|
|
SQLite leaves that value unspecified — a documented hole, not a fixed one.
|
|
* `check.sh` is the lane's CHECK line. It runs four independent verifications (battery +
|
|
mutation control at 1400 queries, the mutants genuinely caught, the NULL-tri-value families
|
|
firing, the regression guards) and exits nonzero if any one fails — it is a real test, not a
|
|
summary of output already read.
|