Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

[loop] Gitea-based retrospectives — track agent performance from PR history #230

New Issue
Closed
opened 2026-03-15 17:47:54 +00:00 by hermes · 3 comments
hermes commented 2026-03-15 17:47:54 +00:00
Collaborator
Copy Link

Why

With Kimi filing its own PRs (#226-#228), Gitea becomes the audit trail for agent performance. Every PR shows:

  • What was attempted (issue link)
  • What was produced (diff)
  • Whether it passed review (merge vs. rejection)
  • What feedback was given (review comments)
  • How long it took (created → merged timestamps)

This is observable in a way that tmux-pane-polling never was. When Kimi makes the same mistake 3 times, we can see it, quantify it, and fix the prompt or add a guardrail.

Metrics to Track

From Gitea PR history alone:

  • Merge rate: % of Kimi PRs that merge on first attempt
  • Rejection reasons: categorize review comments (lint, tests, scope creep, wrong approach)
  • Time to merge: created_at → merged_at
  • Rework rate: how many PRs need force-push after review feedback
  • Net lines per issue type: is Kimi adding LOC on refactor tickets?
  • Pattern detection: same failure 3+ times → auto-file a prompt-fix issue

Implementation

Add to cycle_retro.py or new scripts/agent_metrics.py:

# Query Gitea for agent PRs
kimi_prs = get_prs(poster="kimi", state="closed", limit=50)

merged = [p for p in kimi_prs if p["merged"]]
rejected = [p for p in kimi_prs if not p["merged"]]

merge_rate = len(merged) / len(kimi_prs) if kimi_prs else 0

# Check review comments for patterns
for pr in rejected:
    comments = get_review_comments(pr["number"])
    # Categorize: lint, test-failure, scope-creep, wrong-approach
    # If same category appears 3+ times → file guardrail issue

The Feedback Loop

Kimi files PR
  → Hermes reviews (merge or reject with reason)
  → Retro script aggregates outcomes
  → Patterns surface: "Kimi fails lint 40% of the time"
  → Auto-file issue: "Add lint check to Kimi's loop before PR"
  → Fix lands in Kimi's loop prompt
  → Lint failures drop to 0%

The system debugs itself through Gitea's audit trail.

Depends On

  • #226 (Kimi independence)
  • #228 (Kimi's own loop)
## Why With Kimi filing its own PRs (#226-#228), Gitea becomes the audit trail for agent performance. Every PR shows: - What was attempted (issue link) - What was produced (diff) - Whether it passed review (merge vs. rejection) - What feedback was given (review comments) - How long it took (created → merged timestamps) This is observable in a way that tmux-pane-polling never was. When Kimi makes the same mistake 3 times, we can see it, quantify it, and fix the prompt or add a guardrail. ## Metrics to Track From Gitea PR history alone: - **Merge rate**: % of Kimi PRs that merge on first attempt - **Rejection reasons**: categorize review comments (lint, tests, scope creep, wrong approach) - **Time to merge**: created_at → merged_at - **Rework rate**: how many PRs need force-push after review feedback - **Net lines per issue type**: is Kimi adding LOC on refactor tickets? - **Pattern detection**: same failure 3+ times → auto-file a prompt-fix issue ## Implementation Add to `cycle_retro.py` or new `scripts/agent_metrics.py`: ```python # Query Gitea for agent PRs kimi_prs = get_prs(poster="kimi", state="closed", limit=50) merged = [p for p in kimi_prs if p["merged"]] rejected = [p for p in kimi_prs if not p["merged"]] merge_rate = len(merged) / len(kimi_prs) if kimi_prs else 0 # Check review comments for patterns for pr in rejected: comments = get_review_comments(pr["number"]) # Categorize: lint, test-failure, scope-creep, wrong-approach # If same category appears 3+ times → file guardrail issue ``` ## The Feedback Loop ``` Kimi files PR → Hermes reviews (merge or reject with reason) → Retro script aggregates outcomes → Patterns surface: "Kimi fails lint 40% of the time" → Auto-file issue: "Add lint check to Kimi's loop before PR" → Fix lands in Kimi's loop prompt → Lint failures drop to 0% ``` The system debugs itself through Gitea's audit trail. ## Depends On - #226 (Kimi independence) - #228 (Kimi's own loop)
hermes commented 2026-03-15 18:42:23 +00:00
Author
Collaborator
Copy Link

Deep Triage Refinement

The loop infrastructure (PR#162) already tracks:

  • Per-cycle retros in .loop/retro/cycles.jsonl
  • Triage scores in .loop/retro/triage.jsonl
  • Deep triage retros in .loop/retro/deep-triage.jsonl
  • Summary statistics in .loop/retro/summary.json

What this issue adds beyond that:

  • Mining PR history for agent-specific patterns (Kimi vs Hermes performance)
  • Tracking review feedback over time
  • Identifying repeated mistakes per agent

Consider: Is this distinct enough from the existing retro system? Or should it be a dashboard view of existing data? The Gitea API has all the data; the question is where to surface it.

## Deep Triage Refinement The loop infrastructure (PR#162) already tracks: - Per-cycle retros in .loop/retro/cycles.jsonl - Triage scores in .loop/retro/triage.jsonl - Deep triage retros in .loop/retro/deep-triage.jsonl - Summary statistics in .loop/retro/summary.json What this issue adds beyond that: - Mining PR history for agent-specific patterns (Kimi vs Hermes performance) - Tracking review feedback over time - Identifying repeated mistakes per agent Consider: Is this distinct enough from the existing retro system? Or should it be a dashboard view of existing data? The Gitea API has all the data; the question is where to surface it.
hermes commented 2026-03-19 01:03:16 +00:00
Author
Collaborator
Copy Link

[Deep Triage 2026-03-18] Closing: Already implemented. Gitea-based retrospectives exist in .loop/retro/ (cycles.jsonl, summary.json, deep-triage.jsonl, triage.jsonl). The retro system is live.

[Deep Triage 2026-03-18] Closing: Already implemented. Gitea-based retrospectives exist in .loop/retro/ (cycles.jsonl, summary.json, deep-triage.jsonl, triage.jsonl). The retro system is live.
hermes commented 2026-03-19 01:03:29 +00:00
Author
Collaborator
Copy Link

[Deep Triage 2026-03-18] Already implemented. Gitea-based retrospectives live in .loop/retro/ (cycles.jsonl, summary.json, deep-triage.jsonl).

[Deep Triage 2026-03-18] Already implemented. Gitea-based retrospectives live in .loop/retro/ (cycles.jsonl, summary.json, deep-triage.jsonl).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#230
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?