Timmy_Foundation/compounding-intelligence
15
0
Fork 0
Code Issues 173 Pull Requests 59 Actions Packages Projects Releases Wiki Activity

feat: add harvester.py + session_reader.py — session knowledge extractor (closes #8) #20

Closed
Rockachopa wants to merge 2 commits from burn/8-harvester-py into main
pull from: burn/8-harvester-py
merge into: Timmy_Foundation:main
Conversation 2 Commits 2 Files Changed 2 +589
Rockachopa commented 2026-04-14 17:28:20 +00:00
Owner
Copy Link

Build harvester.py that combines session_reader + extraction prompt to extract durable knowledge from session transcripts.

What was built

scripts/session_reader.py — JSONL parser for Hermes session transcripts:

  • read_session() — load full session
  • extract_conversation() — filter to user/assistant turns
  • truncate_for_context() — keep first 50 + last 50 for long sessions
  • messages_to_text() — convert to plain text for LLM
  • Handles malformed JSON gracefully

scripts/harvester.py — Main knowledge extraction script:

  • Single session: python3 harvester.py --session session.jsonl --output knowledge/
  • Batch mode: python3 harvester.py --batch --since 2026-04-01 --limit 100
  • Dry run: --dry-run to preview without writing
  • Configurable via env vars or CLI flags (API base, key, model, confidence threshold)
  • Deduplication against existing knowledge store (fingerprint + word overlap)
  • Writes to knowledge/index.json + per-repo markdown files
  • Graceful failure handling (logs errors, never crashes)
  • Auto-discovers API keys from standard locations

Acceptance Criteria

  • Processes one session (call to LLM API)
  • Deduplicates against existing knowledge
  • Writes to correct knowledge/ subdirectory
  • Handles extraction failures gracefully (logs, doesn't crash)

Depends on

  • Issue #7 (extraction prompt at templates/harvest-prompt.md) — already merged

Closes #8

Build harvester.py that combines session_reader + extraction prompt to extract durable knowledge from session transcripts. ## What was built **scripts/session_reader.py** — JSONL parser for Hermes session transcripts: - `read_session()` — load full session - `extract_conversation()` — filter to user/assistant turns - `truncate_for_context()` — keep first 50 + last 50 for long sessions - `messages_to_text()` — convert to plain text for LLM - Handles malformed JSON gracefully **scripts/harvester.py** — Main knowledge extraction script: - Single session: `python3 harvester.py --session session.jsonl --output knowledge/` - Batch mode: `python3 harvester.py --batch --since 2026-04-01 --limit 100` - Dry run: `--dry-run` to preview without writing - Configurable via env vars or CLI flags (API base, key, model, confidence threshold) - Deduplication against existing knowledge store (fingerprint + word overlap) - Writes to knowledge/index.json + per-repo markdown files - Graceful failure handling (logs errors, never crashes) - Auto-discovers API keys from standard locations ## Acceptance Criteria - [x] Processes one session (call to LLM API) - [x] Deduplicates against existing knowledge - [x] Writes to correct knowledge/ subdirectory - [x] Handles extraction failures gracefully (logs, doesn't crash) ## Depends on - Issue #7 (extraction prompt at templates/harvest-prompt.md) — already merged Closes #8
Rockachopa added 2 commits 2026-04-14 17:28:21 +00:00
feat: add session_reader.py for JSONL parsing (#8) b2a9bca162
feat: add harvester.py — session knowledge extractor (#8) 9a2135b1df
Timmy requested changes 2026-04-14 18:12:32 +00:00
Timmy left a comment
Owner
Copy Link

Review: PR #20 — feat: add harvester.py + session_reader.py (closes #8)

Recommendation: Close this PR in favor of PR #26.

This PR contains the exact same harvester.py (447 lines) and session_reader.py (142 lines) as PR #26, but is missing the test file (test_harvester_pipeline.py) that PR #26 includes. The code is character-for-character identical between the two PRs.

PR #26 is strictly superior because:

  • Same harvester.py and session_reader.py code
  • Adds test_harvester_pipeline.py with 5 passing smoke tests
  • Better branch naming (fix/8-harvester vs burn/8-harvester-py)

Please close this PR and consolidate work into PR #26. All code review feedback has been left on that PR.

## Review: PR #20 — feat: add harvester.py + session_reader.py (closes #8) **Recommendation: Close this PR in favor of PR #26.** This PR contains the exact same harvester.py (447 lines) and session_reader.py (142 lines) as PR #26, but is missing the test file (test_harvester_pipeline.py) that PR #26 includes. The code is character-for-character identical between the two PRs. PR #26 is strictly superior because: - Same harvester.py and session_reader.py code - Adds test_harvester_pipeline.py with 5 passing smoke tests - Better branch naming (fix/8-harvester vs burn/8-harvester-py) Please close this PR and consolidate work into PR #26. All code review feedback has been left on that PR.
Rockachopa commented 2026-04-14 18:56:27 +00:00
Author
Owner
Copy Link

Closing — duplicate of PR #26 which has the same harvester.py + session_reader.py plus test_harvester_pipeline.py. Per review on #26: "PR #20 should be closed in favor of this one."

Closing — duplicate of PR #26 which has the same harvester.py + session_reader.py plus test_harvester_pipeline.py. Per review on #26: "PR #20 should be closed in favor of this one."
Rockachopa closed this pull request 2026-04-14 18:56:27 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Timmy
Labels
Clear labels
acceptance-criteria
batch-pipeline
Token masterplan batch pipeline
 bootstrapper
Pre-session context injection
 epic
Epic-level issue
 harvester
Session knowledge extraction
 measurer
Compounding metrics
 milestone:1
Milestone 1: Foundation
 milestone:2
Milestone 2: Integration
 milestone:3
Milestone 3: Measurement
 milestone:4
Milestone 4: Retroactive
 pipeline
Pipeline/integration work
 pipeline
priority:high
priority:medium
retroactive
Processing existing sessions
 throughput-10x
throughput-10x label
 token-masterplan
token-masterplan label
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Rockachopa Timmy allegro antigravity bezalel claude codex-agent ezra gemini google grok hermes kimi manus perplexity
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/compounding-intelligence#20
Delete Branch "burn/8-harvester-py"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?