Timmy_Foundation/compounding-intelligence
15
0
Fork 0
Code Issues 173 Pull Requests 59 Actions Packages Projects Releases Wiki Activity

[Measure] Define metrics and build measurer.py #14

New Issue
Open
opened 2026-04-14 15:15:22 +00:00 by Timmy · 0 comments
Timmy commented 2026-04-14 15:15:22 +00:00
Owner
Copy Link

Epic: #4 (Compounding Measurement)

Task

Build the metrics engine that proves compounding is happening.

Metrics

  1. Knowledge velocity: new facts per 1M tokens (trending up = good)
  2. Knowledge coverage: % of repos with >10 facts (breadth)
  3. Hit rate: % of sessions that reference bootstrap knowledge (should increase)
  4. Error recurrence: same errors appearing in multiple sessions (should decrease)
  5. Task completion: % of sessions ending with "ok" status (should increase)
  6. First-try success: actions completed without backtracking (should increase)
  7. Knowledge age: days since facts were last confirmed (staleness indicator)

Interface

python3 measurer.py                    # Compute all metrics
python3 measurer.py --since 2026-04-01 # Time range
python3 measurer.py --repo the-nexus   # Per-repo metrics

Output

{
  "date": "2026-04-13",
  "knowledge_velocity": 0.47,
  "total_facts": 147,
  "repos_covered": 8,
  "hit_rate": 0.23,
  "error_recurrence": 0.31,
  "task_completion_rate": 0.68,
  "trend_7d": {
    "velocity_delta": "+12%",
    "error_recurrence_delta": "-8%",
    "completion_delta": "+5%"
  }
}

Acceptance Criteria

  • All 7 metrics computable from session transcripts + knowledge store
  • Runs in <10 seconds
  • Outputs JSON for dashboard + markdown for human reading
  • Tracks 7-day trend for each metric
## Epic: #4 (Compounding Measurement) ### Task Build the metrics engine that proves compounding is happening. ### Metrics 1. **Knowledge velocity**: new facts per 1M tokens (trending up = good) 2. **Knowledge coverage**: % of repos with >10 facts (breadth) 3. **Hit rate**: % of sessions that reference bootstrap knowledge (should increase) 4. **Error recurrence**: same errors appearing in multiple sessions (should decrease) 5. **Task completion**: % of sessions ending with "ok" status (should increase) 6. **First-try success**: actions completed without backtracking (should increase) 7. **Knowledge age**: days since facts were last confirmed (staleness indicator) ### Interface ```bash python3 measurer.py # Compute all metrics python3 measurer.py --since 2026-04-01 # Time range python3 measurer.py --repo the-nexus # Per-repo metrics ``` ### Output ```json { "date": "2026-04-13", "knowledge_velocity": 0.47, "total_facts": 147, "repos_covered": 8, "hit_rate": 0.23, "error_recurrence": 0.31, "task_completion_rate": 0.68, "trend_7d": { "velocity_delta": "+12%", "error_recurrence_delta": "-8%", "completion_delta": "+5%" } } ``` ### Acceptance Criteria - [ ] All 7 metrics computable from session transcripts + knowledge store - [ ] Runs in <10 seconds - [ ] Outputs JSON for dashboard + markdown for human reading - [ ] Tracks 7-day trend for each metric
Timmy added the measurermilestone:3 labels 2026-04-14 15:15:22 +00:00
hermes was assigned by Rockachopa 2026-04-15 01:50:39 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acceptance-criteria
batch-pipeline
Token masterplan batch pipeline
 bootstrapper
Pre-session context injection
 epic
Epic-level issue
 harvester
Session knowledge extraction
 measurer
Compounding metrics
 milestone:1
Milestone 1: Foundation
 milestone:2
Milestone 2: Integration
 milestone:3
Milestone 3: Measurement
 milestone:4
Milestone 4: Retroactive
 pipeline
Pipeline/integration work
 pipeline
priority:high
priority:medium
retroactive
Processing existing sessions
 throughput-10x
throughput-10x label
 token-masterplan
token-masterplan label
No Label measurer milestone:3
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude codex-agent ezra gemini google grok hermes kimi manus perplexity
No Assignees hermes
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/compounding-intelligence#14
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?