Files

Allegro 5f549bf1f6 [#79 ] JSONL Scorecard Generator for overnight loop analysis

Generates comprehensive reports from overnight loop JSONL data:

**Features:**
- Reads ~/shared/overnight-loop/*.jsonl
- Produces JSON and Markdown reports
- Pass/fail statistics with pass rates
- Duration analysis (avg, median, p95)
- Per-task breakdowns
- Hourly timeline trends
- Error pattern analysis
- Auto-generated recommendations

**Reports:**
- ~/timmy/reports/scorecard_YYYYMMDD.json (structured)
- ~/timmy/reports/scorecard_YYYYMMDD.md (human-readable)

**Usage:**
  python uni-wizard/scripts/generate_scorecard.py

Closes #79

2026-03-30 15:50:06 +00:00

2.8 KiB

Raw Blame History

Scorecard Generator Documentation

Overview

The Scorecard Generator analyzes overnight loop JSONL data and produces comprehensive reports with statistics, trends, and recommendations.

Usage

Basic Usage

# Generate scorecard from default input directory
python uni-wizard/scripts/generate_scorecard.py

# Specify custom input/output directories
python uni-wizard/scripts/generate_scorecard.py \
    --input ~/shared/overnight-loop \
    --output ~/timmy/reports

Cron Setup

# Generate scorecard every morning at 6 AM
0 6 * * * /root/timmy/venv/bin/python /root/timmy/uni-wizard/scripts/generate_scorecard.py

Input Format

JSONL files in ~/shared/overnight-loop/*.jsonl:

{"task": "read-soul", "status": "pass", "duration_s": 19.7, "timestamp": "2026-03-29T21:54:12Z"}
{"task": "check-health", "status": "fail", "duration_s": 5.2, "error": "timeout", "timestamp": "2026-03-29T22:15:33Z"}

Fields:

task: Task identifier
status: "pass" or "fail"
duration_s: Execution time in seconds
timestamp: ISO 8601 timestamp
error: Error message (for failed tasks)

Output

JSON Report

~/timmy/reports/scorecard_YYYYMMDD.json:

{
  "generated_at": "2026-03-30T06:00:00Z",
  "summary": {
    "total_tasks": 100,
    "passed": 95,
    "failed": 5,
    "pass_rate": 95.0,
    "duration_stats": {
      "avg": 12.5,
      "median": 10.2,
      "p95": 45.0,
      "min": 1.2,
      "max": 120.5
    }
  },
  "by_task": {...},
  "by_hour": {...},
  "errors": {...},
  "recommendations": [...]
}

Markdown Report

~/timmy/reports/scorecard_YYYYMMDD.md:

Executive summary with pass/fail counts
Duration statistics (avg, median, p95)
Per-task breakdown with pass rates
Hourly timeline showing performance trends
Error analysis with frequency counts
Actionable recommendations

Report Interpretation

Pass Rate Thresholds

Pass Rate	Status	Action
95%+	✅ Excellent	Continue current operations
85-94%	⚠️ Good	Monitor for degradation
70-84%	⚠️ Fair	Review failing tasks
<70%	❌ Poor	Immediate investigation required

Duration Guidelines

Duration	Assessment
<5s	Fast
5-15s	Normal
15-30s	Slow
>30s	Very slow - consider optimization

Troubleshooting

No JSONL files found

# Check input directory
ls -la ~/shared/overnight-loop/

# Ensure Syncthing is syncing
systemctl status syncthing@root

Malformed lines

The generator skips malformed lines with a warning. Check the JSONL files for syntax errors.

Empty reports

If no data exists, verify:

Overnight loop is running and writing JSONL
File permissions allow reading
Input path is correct

2.8 KiB Raw Blame History