step35-cli 03cf42e29d feat(benchmarks): multi-config comparison suite (refs #29) ...

Add compare_configs.py — orchestrates running 4 Gemma4 configs in
parallel and produces side-by-side comparison table with aggregated
metrics (TTFT, tokens/sec, latency, peak memory). Picks winner by
highest tokens/sec.

Configurations:
1. Ollama gemma4 (baseline)
2. llama-server gemma4 f16 KV
3. llama-server gemma4 turbo4 KV
4. llama-server gemma4 turbo4 + layer-adaptive

Also adds comprehensive test suite (13 tests) covering ConfigEntry,
aggregation, table building, demo mode, and prompt loading.

Closes #29

Co-authored-by: step35-cli <step35-cli@timmy.foundation>

2026-04-26 00:05:24 -04:00

..

compare_configs.py

feat(benchmarks): multi-config comparison suite (refs #29)

2026-04-26 00:05:24 -04:00

perplexity_results.json

feat: wikitext-2 corpus + perplexity benchmark script (closes #21)

2026-04-12 00:39:14 -04:00

prompts.json

feat: add standardized benchmarking prompts

2026-03-30 21:14:48 +00:00

run_benchmarks.py

feat: multi-backend benchmark suite with TTFT + memory tracking (#37)

2026-04-13 14:05:17 +00:00

run_long_session.py

burn: add long-session quality test (Issue #12) (#39)

2026-04-13 19:59:22 +00:00

run_perplexity.py

feat: wikitext-2 corpus + perplexity benchmark script (closes #21)

2026-04-12 00:39:14 -04:00

test_prompts.json

Add benchmark test prompts for quality comparison (Issue #22)

2026-03-31 17:31:05 +00:00