Alexander Whitestone 5bfb9eb000 feat: multi-config benchmark comparison suite (Issue #29) ...

benchmarks/compare_configs.py:
  - Runs 4 configs (ollama, llama-f16, llama-turbo4, llama-turbo4-adaptive)
  - Aggregates TTFT, tok/s, latency, peak memory
  - Picks winner by highest tok/s
  - Outputs JSON report + human-readable table
  - --demo mode for testing without live servers

tests/test_compare_configs.py (13 tests):
  - ConfigEntry, ConfigResult, default configs
  - Aggregation logic, winner selection, table format
  - Demo mode with and without output file
  - Prompt loading from test_prompts.json

Closes #29.

2026-04-13 21:42:29 -04:00

..

compare_configs.py

feat: multi-config benchmark comparison suite (Issue #29)

2026-04-13 21:42:29 -04:00

perplexity_results.json

feat: wikitext-2 corpus + perplexity benchmark script (closes #21)

2026-04-12 00:39:14 -04:00

prompts.json

feat: add standardized benchmarking prompts

2026-03-30 21:14:48 +00:00

run_benchmarks.py

feat: multi-backend benchmark suite with TTFT + memory tracking (#37)

2026-04-13 14:05:17 +00:00

run_long_session.py

burn: add long-session quality test (Issue #12) (#39)

2026-04-13 19:59:22 +00:00

run_perplexity.py

feat: wikitext-2 corpus + perplexity benchmark script (closes #21)

2026-04-12 00:39:14 -04:00

test_prompts.json

Add benchmark test prompts for quality comparison (Issue #22)

2026-03-31 17:31:05 +00:00