Timmy_Foundation/compounding-intelligence
15
0
Fork 0
Code Issues 173 Pull Requests 59 Actions Packages Projects Releases Wiki Activity

Knowledge Mine: Parallel Batch Harvester — Process 20K Sessions #24

New Issue
Open
opened 2026-04-14 17:57:27 +00:00 by claude · 0 comments
claude commented 2026-04-14 17:57:27 +00:00
Member
Copy Link

Knowledge Mine — Sub-Issue

The core engine. Read sessions, extract knowledge, store it.

Task

Build ~/.hermes/pipelines/knowledge-mine.py:

  1. Load session scores from session-sampler
  2. For each session (high-value first):
    • Read .jsonl transcript
    • Send to LLM for knowledge extraction (prompt: extract decisions, patterns, errors, tool quirks)
    • Parse structured output
    • Store in ~/.hermes/knowledge/sessions/{session_id}.json
  3. Run 10 parallel workers
  4. Progress tracking + resume on failure

Knowledge Extraction Prompt

For each session, extract:

  • Decisions: What was decided and why
  • Patterns: Recurring approaches that worked
  • Errors: What went wrong and how it was fixed
  • Tool quirks: Non-obvious behavior of tools/APIs
  • User preferences: Corrections, style, workflow preferences
  • Environment facts: OS, hardware, installed tools, configs

Acceptance Criteria

  • Processes 20K sessions in <6 hours with 10 workers
  • Each session produces structured JSON knowledge file
  • Resumable (can restart from last processed session)
  • Token budget tracked and reported
  • Output quality spot-checked on 100 random sessions

Tokens: ~200M (10K per session × 20K sessions)

Depends On: Knowledge Mine: Session Sampler

## Knowledge Mine — Sub-Issue The core engine. Read sessions, extract knowledge, store it. ### Task Build `~/.hermes/pipelines/knowledge-mine.py`: 1. Load session scores from session-sampler 2. For each session (high-value first): - Read .jsonl transcript - Send to LLM for knowledge extraction (prompt: extract decisions, patterns, errors, tool quirks) - Parse structured output - Store in `~/.hermes/knowledge/sessions/{session_id}.json` 3. Run 10 parallel workers 4. Progress tracking + resume on failure ### Knowledge Extraction Prompt For each session, extract: - **Decisions**: What was decided and why - **Patterns**: Recurring approaches that worked - **Errors**: What went wrong and how it was fixed - **Tool quirks**: Non-obvious behavior of tools/APIs - **User preferences**: Corrections, style, workflow preferences - **Environment facts**: OS, hardware, installed tools, configs ### Acceptance Criteria - [ ] Processes 20K sessions in <6 hours with 10 workers - [ ] Each session produces structured JSON knowledge file - [ ] Resumable (can restart from last processed session) - [ ] Token budget tracked and reported - [ ] Output quality spot-checked on 100 random sessions ### Tokens: ~200M (10K per session × 20K sessions) ### Depends On: Knowledge Mine: Session Sampler
claude added the batch-pipelineharvester labels 2026-04-14 17:57:27 +00:00
hermes was assigned by Rockachopa 2026-04-15 01:50:32 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acceptance-criteria
batch-pipeline
Token masterplan batch pipeline
 bootstrapper
Pre-session context injection
 epic
Epic-level issue
 harvester
Session knowledge extraction
 measurer
Compounding metrics
 milestone:1
Milestone 1: Foundation
 milestone:2
Milestone 2: Integration
 milestone:3
Milestone 3: Measurement
 milestone:4
Milestone 4: Retroactive
 pipeline
Pipeline/integration work
 pipeline
priority:high
priority:medium
retroactive
Processing existing sessions
 throughput-10x
throughput-10x label
 token-masterplan
token-masterplan label
No Label batch-pipeline harvester
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude codex-agent ezra gemini google grok hermes kimi manus perplexity
No Assignees hermes
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/compounding-intelligence#24
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?