Rockachopa/Timmy-time-dashboard
1
0
Fork 2
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

[Content] Highlight Detection Engine — Chat Spikes + Game Events + Audio Energy #874

New Issue
Closed
opened 2026-03-21 23:40:06 +00:00 by perplexity · 0 comments
perplexity commented 2026-03-21 23:40:06 +00:00
Collaborator
Copy Link

Why This Is High Leverage

No AI streamer has open-sourced an automated content pipeline — not even Neuro-sama at $2M/year. The V2 roadmap identifies this as the single largest gap in the AI streaming ecosystem and recommends it as Phase 7's first deliverable. The highlight detection layer is the one component that must be built from scratch (2-3 weeks). Everything downstream (clip extraction, episode composition, publishing) uses existing production tools.

Scope

Build a weighted scoring system combining three signal channels to identify memorable moments from gameplay streams:

Signal Channels

  1. Chat activity analysis — frequency spikes and sentiment shifts (validated by Twitch compilation community)
  2. Game event timestamps — TES3MP Lua layer events (combat, quest advancement, death, discovery)
  3. Audio energy peaks — FFmpeg's astats filter for sudden volume/energy changes

Scoring

  • Weighted combination of all three channels
  • Produces ranked list of candidate highlights with timestamps and confidence scores
  • Configurable thresholds for auto-clip vs. manual review

Requirements

  • src/content/detection/chat_analyzer.py — Chat frequency + sentiment scoring over sliding windows
  • src/content/detection/game_events.py — Parse TES3MP event log for significant moments
  • src/content/detection/audio_analyzer.py — FFmpeg astats wrapper for energy peak detection
  • src/content/detection/scorer.py — Weighted fusion, ranking, threshold-based classification
  • src/content/detection/pipeline.py — End-to-end: ingest stream data → output ranked highlights
  • Tests with sample data

Acceptance Criteria

  • Given a mock stream with known highlights, detection recalls >80% of them
  • Output is a JSON list of {timestamp, duration, score, signals, confidence}
  • Runs offline against recorded stream data (not requiring live stream)

Reference

  • TwitchCompilationCreator (github.com/ContentAutomation/TwitchCompilationCreator)
  • Estimated effort: 2-3 weeks

Assignee: Kimi

## Why This Is High Leverage No AI streamer has open-sourced an automated content pipeline — not even Neuro-sama at $2M/year. The V2 roadmap identifies this as the single largest gap in the AI streaming ecosystem and recommends it as Phase 7's first deliverable. The highlight detection layer is the one component that must be built from scratch (2-3 weeks). Everything downstream (clip extraction, episode composition, publishing) uses existing production tools. ## Scope Build a weighted scoring system combining three signal channels to identify memorable moments from gameplay streams: ### Signal Channels 1. **Chat activity analysis** — frequency spikes and sentiment shifts (validated by Twitch compilation community) 2. **Game event timestamps** — TES3MP Lua layer events (combat, quest advancement, death, discovery) 3. **Audio energy peaks** — FFmpeg's `astats` filter for sudden volume/energy changes ### Scoring - Weighted combination of all three channels - Produces ranked list of candidate highlights with timestamps and confidence scores - Configurable thresholds for auto-clip vs. manual review ## Requirements - [ ] `src/content/detection/chat_analyzer.py` — Chat frequency + sentiment scoring over sliding windows - [ ] `src/content/detection/game_events.py` — Parse TES3MP event log for significant moments - [ ] `src/content/detection/audio_analyzer.py` — FFmpeg astats wrapper for energy peak detection - [ ] `src/content/detection/scorer.py` — Weighted fusion, ranking, threshold-based classification - [ ] `src/content/detection/pipeline.py` — End-to-end: ingest stream data → output ranked highlights - [ ] Tests with sample data ## Acceptance Criteria - Given a mock stream with known highlights, detection recalls >80% of them - Output is a JSON list of `{timestamp, duration, score, signals, confidence}` - Runs offline against recorded stream data (not requiring live stream) ## Reference - TwitchCompilationCreator (github.com/ContentAutomation/TwitchCompilationCreator) - Estimated effort: 2-3 weeks ## Assignee: Kimi
claude was assigned by Rockachopa 2026-03-22 23:33:32 +00:00
claude added the harnessmorrowindp1-important labels 2026-03-23 13:53:35 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
Workshop: Timmy as Presence (Epic #222)
 actionable
Has a concrete code/config task extracted
 assigned-claude
Issue currently assigned to Claude agent — do not assign to another agent
 assigned-gemini
Issue currently assigned to Gemini agent — do not assign to another agent
 assigned-groq
assigned-kimi
Issue currently assigned to Kimi agent — do not assign to another agent
 assigned-manus
Issue currently assigned to Manus agent — do not assign to another agent
 claude-ready
consolidation
Part of a consolidation epic
 deprioritized
Keep open but not blocking P0 work
 deprioritized
Keep open but not blocking P0 work
 duplicate
Duplicate of another issue
 gemini-review
Auto-generated by Gemini, needs relevance review
 groq-ready
harness
Core product: agent framework, heartbeat, inference, memory
 heartbeat
Harness: Agent heartbeat loop
 inference
Harness: Inference and model routing
 infrastructure
Supporting stage: dashboard, CI/CD, deployment, DNS
 kimi-ready
Scoped and ready for Kimi to pick up
 memory-session
Harness: Memory and session crystallization
 morrowind
Harness: Morrowind embodiment
 needs-design
Needs architectural design before implementation
 needs-extraction
Philosophy with unextracted engineering work
 p0-critical
Priority 0: Must fix now
 p1-important
Priority 1: Important, next sprint
 p2-backlog
Priority 2: Backlog, do when time permits
 philosophy
Philosophical foundation — informs architecture decisions
 rejected-direction
Closed: rejected or superseded direction
 seed:know-purpose
Three Seeds: KNOW YOUR PURPOSE
 seed:serve-real
Three Seeds: SERVE THE REAL
 seed:tell-truth
Three Seeds: TELL THE TRUTH
 sovereignty
Harness: Sovereignty stack
No Label harness morrowind p1-important
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Timmy claude grok groq kimi manus perplexity Rockachopa
No Assignees claude
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Rockachopa/Timmy-time-dashboard#874
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?