[#79] JSONL Scorecard Generator for overnight loop analysis
Generates comprehensive reports from overnight loop JSONL data: **Features:** - Reads ~/shared/overnight-loop/*.jsonl - Produces JSON and Markdown reports - Pass/fail statistics with pass rates - Duration analysis (avg, median, p95) - Per-task breakdowns - Hourly timeline trends - Error pattern analysis - Auto-generated recommendations **Reports:** - ~/timmy/reports/scorecard_YYYYMMDD.json (structured) - ~/timmy/reports/scorecard_YYYYMMDD.md (human-readable) **Usage:** python uni-wizard/scripts/generate_scorecard.py Closes #79
This commit is contained in:
125
docs/SCORECARD.md
Normal file
125
docs/SCORECARD.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# Scorecard Generator Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
The Scorecard Generator analyzes overnight loop JSONL data and produces comprehensive reports with statistics, trends, and recommendations.
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Generate scorecard from default input directory
|
||||
python uni-wizard/scripts/generate_scorecard.py
|
||||
|
||||
# Specify custom input/output directories
|
||||
python uni-wizard/scripts/generate_scorecard.py \
|
||||
--input ~/shared/overnight-loop \
|
||||
--output ~/timmy/reports
|
||||
```
|
||||
|
||||
### Cron Setup
|
||||
|
||||
```bash
|
||||
# Generate scorecard every morning at 6 AM
|
||||
0 6 * * * /root/timmy/venv/bin/python /root/timmy/uni-wizard/scripts/generate_scorecard.py
|
||||
```
|
||||
|
||||
## Input Format
|
||||
|
||||
JSONL files in `~/shared/overnight-loop/*.jsonl`:
|
||||
|
||||
```json
|
||||
{"task": "read-soul", "status": "pass", "duration_s": 19.7, "timestamp": "2026-03-29T21:54:12Z"}
|
||||
{"task": "check-health", "status": "fail", "duration_s": 5.2, "error": "timeout", "timestamp": "2026-03-29T22:15:33Z"}
|
||||
```
|
||||
|
||||
Fields:
|
||||
- `task`: Task identifier
|
||||
- `status`: "pass" or "fail"
|
||||
- `duration_s`: Execution time in seconds
|
||||
- `timestamp`: ISO 8601 timestamp
|
||||
- `error`: Error message (for failed tasks)
|
||||
|
||||
## Output
|
||||
|
||||
### JSON Report
|
||||
|
||||
`~/timmy/reports/scorecard_YYYYMMDD.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"generated_at": "2026-03-30T06:00:00Z",
|
||||
"summary": {
|
||||
"total_tasks": 100,
|
||||
"passed": 95,
|
||||
"failed": 5,
|
||||
"pass_rate": 95.0,
|
||||
"duration_stats": {
|
||||
"avg": 12.5,
|
||||
"median": 10.2,
|
||||
"p95": 45.0,
|
||||
"min": 1.2,
|
||||
"max": 120.5
|
||||
}
|
||||
},
|
||||
"by_task": {...},
|
||||
"by_hour": {...},
|
||||
"errors": {...},
|
||||
"recommendations": [...]
|
||||
}
|
||||
```
|
||||
|
||||
### Markdown Report
|
||||
|
||||
`~/timmy/reports/scorecard_YYYYMMDD.md`:
|
||||
|
||||
- Executive summary with pass/fail counts
|
||||
- Duration statistics (avg, median, p95)
|
||||
- Per-task breakdown with pass rates
|
||||
- Hourly timeline showing performance trends
|
||||
- Error analysis with frequency counts
|
||||
- Actionable recommendations
|
||||
|
||||
## Report Interpretation
|
||||
|
||||
### Pass Rate Thresholds
|
||||
|
||||
| Pass Rate | Status | Action |
|
||||
|-----------|--------|--------|
|
||||
| 95%+ | ✅ Excellent | Continue current operations |
|
||||
| 85-94% | ⚠️ Good | Monitor for degradation |
|
||||
| 70-84% | ⚠️ Fair | Review failing tasks |
|
||||
| <70% | ❌ Poor | Immediate investigation required |
|
||||
|
||||
### Duration Guidelines
|
||||
|
||||
| Duration | Assessment |
|
||||
|----------|------------|
|
||||
| <5s | Fast |
|
||||
| 5-15s | Normal |
|
||||
| 15-30s | Slow |
|
||||
| >30s | Very slow - consider optimization |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No JSONL files found
|
||||
|
||||
```bash
|
||||
# Check input directory
|
||||
ls -la ~/shared/overnight-loop/
|
||||
|
||||
# Ensure Syncthing is syncing
|
||||
systemctl status syncthing@root
|
||||
```
|
||||
|
||||
### Malformed lines
|
||||
|
||||
The generator skips malformed lines with a warning. Check the JSONL files for syntax errors.
|
||||
|
||||
### Empty reports
|
||||
|
||||
If no data exists, verify:
|
||||
1. Overnight loop is running and writing JSONL
|
||||
2. File permissions allow reading
|
||||
3. Input path is correct
|
||||
Reference in New Issue
Block a user