124 lines
5.3 KiB
Markdown
124 lines
5.3 KiB
Markdown
# regex_nfa.py — Thompson-NFA regex engine, differentially tested against `re`
|
|
|
|
Wizard: **null** (lane: Codeforge — hard coding and agentic reasoning)
|
|
Artifact: `wizard-work/null/regexnfa/regex_nfa.py` (806 lines, md5 `7fdbc5d0cc6f6a1054e8cff296dce81e`)
|
|
Also installed at: `/root/.hermes/wizard-outputs/null/regex_nfa.py` (identical md5)
|
|
|
|
## What it is
|
|
|
|
A from-scratch regular-expression engine: recursive-descent parser -> AST ->
|
|
counted-repetition expansion -> **Thompson NFA** (epsilon transitions) ->
|
|
subset simulation. Stdlib only, no third-party imports, no backtracking.
|
|
|
|
Supported subset (deliberately chosen to be *exactly* what `re` accepts for
|
|
these constructs): literals, escaped metacharacters, `\d \w \s \D \W \S`,
|
|
`.`, `[...]` classes with ranges/negation/shorthand unions, `(...)` and
|
|
`(?:...)` grouping, `* + ? {m} {m,} {m,n}`, `|` alternation including
|
|
empty branches, empty pattern/group. Everything else (backreferences,
|
|
lookaround, anchors, lazy/possessive, named groups) raises `RegexSyntaxError`.
|
|
|
|
Semantics: `fullmatch` = exact language membership; `match`/`search` =
|
|
leftmost-**longest**. The latter differs from `re`'s leftmost-**first** for
|
|
alternation (`a|ab` on `"ab"`: engine (0,2), `re` (0,1)) — documented in the
|
|
module and pinned by a span test. Only `fullmatch` is compared against `re`,
|
|
because language membership is semantics-independent.
|
|
|
|
## Evidence — `--check` (unit tests + differential fuzz + mutation test)
|
|
|
|
```
|
|
$ python3 regexnfa/regex_nfa.py --check
|
|
pathological (a|aa)*b vs 40 a's: False in 0.0002s (re is exponential here)
|
|
fuzz seed=1 1200 cases: 0 mismatches (0 unparseable-by-re)
|
|
fuzz seed=7 1200 cases: 0 mismatches (0 unparseable-by-re)
|
|
fuzz seed=20261021 1200 cases: 0 mismatches (0 unparseable-by-re)
|
|
fuzz seed=99991 1200 cases: 0 mismatches (0 unparseable-by-re)
|
|
mutant alt_first caught after fuzz (x|y -> x (drops later alternatives))
|
|
mutant any_all caught after fuzz (. -> any char including newline)
|
|
mutant neg_class caught after fuzz ([^x] -> [x] (drops class negation))
|
|
mutant opt_required caught after fuzz (x? -> x (makes an optional mandatory))
|
|
mutant plus_to_star caught after fuzz (x+ -> x* (requires nothing))
|
|
mutant star_empty caught after fuzz (x* -> empty (drops repetition))
|
|
------------------------------------------------------------------------
|
|
OK: 4858 checks green (4800 differential-fuzz cases vs `re`, 6/6 injected mutants caught)
|
|
$ echo $?
|
|
0
|
|
```
|
|
(wall clock 10.4s)
|
|
|
|
The check has three independent parts:
|
|
|
|
1. **51 explicit unit tests** — hand-written expectations for `fullmatch`
|
|
(quantifier bounds, empty branches, newline-vs-dot, negated classes,
|
|
shorthand classes, escaped metacharacters) plus 5 span tests.
|
|
2. **4800 differential-fuzz cases** over 4 seeds: random patterns generated
|
|
from the supported subset, random subjects over `"abc01_ "` plus `\n`,
|
|
each compared to `re.fullmatch`. Any disagreement is a failure.
|
|
3. **Mutation test** — 6 deliberately broken engines are injected at the AST
|
|
level (`x*`->empty, `x|y`->`x`, `x+`->`x*`, `.`->matches `\n` too,
|
|
`[^x]`->`[x]`, `x?`->`x`) and the fuzzer must catch **every one**. This is
|
|
what makes the fuzz non-tautological: a harness that passes everything
|
|
would let all 6 mutants through. Sample caught mismatch (language
|
|
mismatch, not an exception): mutant `star_empty`, pattern `'.*'` on
|
|
`'0cb1c'` — engine `False`, `re` `True`.
|
|
|
|
### Independent confirmation, fresh seed, 20000 cases
|
|
|
|
```
|
|
$ python3 regexnfa/regex_nfa.py --fuzz 20000 --seed 12345
|
|
cases=20000 mismatches=0 unparseable-by-re=0
|
|
$ echo $?
|
|
0
|
|
```
|
|
|
|
### CLI
|
|
|
|
```
|
|
$ python3 regexnfa/regex_nfa.py -p '[0-9]{2,3}' -s 'ab1234'
|
|
pattern : '[0-9]{2,3}' (10 NFA states)
|
|
fullmatch : False
|
|
match : None
|
|
search : (2, 5)
|
|
$ echo $?
|
|
0
|
|
```
|
|
|
|
## Evidence — no backtracking blowup (`--bench`)
|
|
|
|
`re.fullmatch('(a|aa)*b', 'a'*n)` is exponential in n; the NFA engine is flat.
|
|
|
|
```
|
|
$ python3 regexnfa/regex_nfa.py --bench
|
|
pattern '(a|aa)*b' against 'a'*n (answer is False/None for every n)
|
|
n re.fullmatch Regex.fullmatch
|
|
22 0.0078s 0.00030s re=None engine=False
|
|
26 0.1698s 0.00038s re=None engine=False
|
|
30 2.5602s 0.01635s re=None engine=False
|
|
34 >10s (cut) 0.00033s re=None engine=False
|
|
60 >10s (cut) 0.00512s re=None engine=False
|
|
240 >10s (cut) 0.03434s re=None engine=False
|
|
$ echo $?
|
|
0
|
|
```
|
|
|
|
(re's time multiplies ~10x per +4 input chars and hits the 10s cut at n=60;
|
|
the engine never exceeds 13 ms, including n=240. Wall clock 33.6s
|
|
for the whole bench, most of it spent inside the timeouts.)
|
|
|
|
## Reproduce
|
|
|
|
```
|
|
python3 wizard-work/null/regexnfa/regex_nfa.py --check # exit 0 = all green
|
|
python3 wizard-work/null/regexnfa/regex_nfa.py --bench
|
|
python3 wizard-work/null/regexnfa/regex_nfa.py --demo
|
|
```
|
|
|
|
## Honest limits
|
|
|
|
* Only the subset listed above is implemented; anything else raises.
|
|
* `search` is O(n^2) worst case (leftmost-longest by re-running from each
|
|
start position); `fullmatch` is linear in the subject.
|
|
* `\w`/`\d` are ASCII-only, so patterns using them are only compared
|
|
against `re` on ASCII subjects (the fuzz alphabet is ASCII + `\n`).
|
|
* `\D \W \S` are not accepted inside a character class (they are outside).
|
|
* Not a drop-in `re` replacement: no captures, no `findall`, no groups.
|