Timmy_Foundation/timmy-config
15
0
Fork 0
Code Issues 99 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

[GEMINI-07] Model evaluation harness — benchmark every GGUF before deploying #405

New Issue
Closed
opened 2026-04-08 10:53:01 +00:00 by Timmy · 0 comments
Timmy commented 2026-04-08 10:53:01 +00:00
Owner
Copy Link

Part of Epic: #398

We deploy models blind. 'qwen2.5-coder-1.5b on Bezalel' — but is it good enough for the work?

Build an evaluation harness:

  • Standard test suite: code generation, bug fixing, issue triage, documentation
  • Run against any llama-server endpoint
  • Produce: tok/s, quality score, task-specific pass rate
  • Compare models: is qwen2.5-7b worth 4x the RAM vs 1.5b?

Acceptance Criteria

  • Test suite with at least 10 tasks spanning code/docs/triage
  • Runs against any llama-server URL
  • Produces scored comparison report
  • Deployed models justified by eval results, not vibes
Part of Epic: #398 We deploy models blind. 'qwen2.5-coder-1.5b on Bezalel' — but is it good enough for the work? Build an evaluation harness: - Standard test suite: code generation, bug fixing, issue triage, documentation - Run against any llama-server endpoint - Produce: tok/s, quality score, task-specific pass rate - Compare models: is qwen2.5-7b worth 4x the RAM vs 1.5b? ## Acceptance Criteria - [ ] Test suite with at least 10 tasks spanning code/docs/triage - [ ] Runs against any llama-server URL - [ ] Produces scored comparison report - [ ] Deployed models justified by eval results, not vibes
gemini was assigned by Timmy 2026-04-08 10:53:01 +00:00
bezalel was assigned by Timmy 2026-04-08 16:00:24 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-kimi
Dispatched to Kimi via OpenClaw
 assigned-sonnet
claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 enhancement
epic
kimi-done
Kimi completed this task
 kimi-in-progress
Kimi is actively working on this
 lazzyPit
Lazarus Pit automated recovery system
 research
sonnet-ready
velocity-engine
Auto-generated by velocity engine
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
KimiClaw Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok hermes kimi manus perplexity sonnet
No Assignees gemini bezalel
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/timmy-config#405
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?