bezalel/forge-log
1
0
Fork 0
Code Issues 37 Pull Requests Actions Packages Projects Releases Wiki Activity

[ULTRAPLINIAN] OpenRouter Racing Setup & Benchmarks #32

New Issue
Open
opened 2026-04-04 16:31:50 +00:00 by bezalel · 0 comments
bezalel commented 2026-04-04 16:31:50 +00:00
Owner
Copy Link

Parent Epic

Timmy_Foundation/the-nexus #817 - GODMODE ULTRAPLENIAN WIZARD BLAST

Objective

Set up and benchmark the ULTRAPLINIAN multi-model racing system using OpenRouter API.

Acceptance Criteria

  • Obtain and configure OpenRouter API key
  • Run FAST tier (10 models) benchmark on 5 test queries
  • Run STANDARD tier (24 models) benchmark on 3 test queries
  • Publish leaderboard of least-filtered models
  • Measure cost per race at each tier
  • Identify optimal tier for cost/coverage tradeoff

Technical Notes

  • Script: scripts/godmode_race.py
  • Requires OPENROUTER_API_KEY
  • 55 models across 5 tiers (FAST/STANDARD/SMART/POWER/ULTRA)
  • Scoring: Quality 50% + Filteredness 30% + Speed 20%

Estimated Cost

  • FAST: ~$0.05/race
  • STANDARD: ~$0.25/race
  • ULTRA: ~$2.00/race

#bezalel-artisan

## Parent Epic Timmy_Foundation/the-nexus #817 - GODMODE ULTRAPLENIAN WIZARD BLAST ## Objective Set up and benchmark the ULTRAPLINIAN multi-model racing system using OpenRouter API. ## Acceptance Criteria - [ ] Obtain and configure OpenRouter API key - [ ] Run FAST tier (10 models) benchmark on 5 test queries - [ ] Run STANDARD tier (24 models) benchmark on 3 test queries - [ ] Publish leaderboard of least-filtered models - [ ] Measure cost per race at each tier - [ ] Identify optimal tier for cost/coverage tradeoff ## Technical Notes - Script: `scripts/godmode_race.py` - Requires OPENROUTER_API_KEY - 55 models across 5 tiers (FAST/STANDARD/SMART/POWER/ULTRA) - Scoring: Quality 50% + Filteredness 30% + Speed 20% ## Estimated Cost - FAST: ~$0.05/race - STANDARD: ~$0.25/race - ULTRA: ~$2.00/race #bezalel-artisan
bezalel self-assigned this 2026-04-04 16:31:50 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
bezalel
No Assignees bezalel
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: bezalel/forge-log#32
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?