Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Feature: Agent "Performance Regression" Suite #1015

New Issue
Closed
opened 2026-03-22 23:04:55 +00:00 by gemini · 1 comment
gemini commented 2026-03-22 23:04:55 +00:00
Collaborator
Copy Link

Objective

Implement a standardized suite of Morrowind tasks to verify agent performance and prevent regressions after code changes.

Scope

  • Define a set of "Benchmark Scenarios" (e.g., "Walk from Seyda Neen to Balmora", "Complete the Fargoth quest").
  • Create a script to run the agent through these scenarios in a headless OpenMW instance.
  • Track metrics: time to completion, success rate, number of LLM calls, total "metabolic" cost.
  • Integrate the suite into the CI/CD pipeline.
## Objective Implement a standardized suite of Morrowind tasks to verify agent performance and prevent regressions after code changes. ## Scope - Define a set of "Benchmark Scenarios" (e.g., "Walk from Seyda Neen to Balmora", "Complete the Fargoth quest"). - Create a script to run the agent through these scenarios in a headless OpenMW instance. - Track metrics: time to completion, success rate, number of LLM calls, total "metabolic" cost. - Integrate the suite into the CI/CD pipeline.
claude was assigned by Rockachopa 2026-03-22 23:30:11 +00:00
claude commented 2026-03-22 23:55:06 +00:00
Collaborator
Copy Link

PR #1053 created.

Added a full agent performance regression benchmark suite:

  • 5 Morrowind scenarios (navigation, quest, combat, observation)
  • BenchmarkRunner executing through the heartbeat loop with MockWorldAdapter
  • Metrics: cycles, wall time, LLM calls, metabolic cost
  • Goal predicates for early success detection
  • JSONL persistence with regression comparison (compare_runs())
  • CLI script with tag filtering and baseline comparison
  • tox -e benchmark CI environment
  • 31 unit tests, all passing
PR #1053 created. Added a full agent performance regression benchmark suite: - 5 Morrowind scenarios (navigation, quest, combat, observation) - BenchmarkRunner executing through the heartbeat loop with MockWorldAdapter - Metrics: cycles, wall time, LLM calls, metabolic cost - Goal predicates for early success detection - JSONL persistence with regression comparison (`compare_runs()`) - CLI script with tag filtering and baseline comparison - `tox -e benchmark` CI environment - 31 unit tests, all passing
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees claude
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#1015
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?