Timmy_Foundation/the-nexus
15
1
Fork 2
Code Issues 72 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

[Mnemosyne] Add file-based document ingestion pipeline #1275

New Issue
Closed
opened 2026-04-12 11:43:08 +00:00 by Rockachopa · 1 comment
Rockachopa commented 2026-04-12 11:43:08 +00:00
Owner
Copy Link

Problem

The current ingest.py only supports MemPalace exports and single events. There's no way to bulk-import a directory of markdown/text files — a common use case for sovereign agents who accumulate notes, docs, and reports on disk.

Proposed Feature

Add ingest_directory() and ingest_file() functions to nexus/mnemosyne/ingest.py:

  • ingest_file(archive, path) — reads a single file, extracts title from first # heading (or filename), ingests content with source tracking
  • ingest_directory(archive, dir_path, extensions=None) — walks a directory tree, ingests all matching files (default: .md, .txt, .json)
  • Dedup via source_ref (file path + mtime) to avoid re-ingesting unchanged files
  • Chunking for files over a max size (e.g., split on ## headings or fixed token windows)

Acceptance Criteria

  1. ingest_file() ingests a single file and returns the ArchiveEntry
  2. ingest_directory() ingests all matching files in a directory tree
  3. Re-ingesting the same unchanged file returns existing entry (no duplicate)
  4. CLI command: mnemosyne ingest-dir <path> [--ext md,txt]
  5. Tests for both functions

Priority: Medium

Parent: Mnemosyne archive module (Phase 1 → Phase 2 evolution)

## Problem The current `ingest.py` only supports MemPalace exports and single events. There's no way to bulk-import a directory of markdown/text files — a common use case for sovereign agents who accumulate notes, docs, and reports on disk. ## Proposed Feature Add `ingest_directory()` and `ingest_file()` functions to `nexus/mnemosyne/ingest.py`: - **`ingest_file(archive, path)`** — reads a single file, extracts title from first `# heading` (or filename), ingests content with source tracking - **`ingest_directory(archive, dir_path, extensions=None)`** — walks a directory tree, ingests all matching files (default: `.md`, `.txt`, `.json`) - Dedup via `source_ref` (file path + mtime) to avoid re-ingesting unchanged files - Chunking for files over a max size (e.g., split on `## ` headings or fixed token windows) ## Acceptance Criteria 1. `ingest_file()` ingests a single file and returns the ArchiveEntry 2. `ingest_directory()` ingests all matching files in a directory tree 3. Re-ingesting the same unchanged file returns existing entry (no duplicate) 4. CLI command: `mnemosyne ingest-dir <path> [--ext md,txt]` 5. Tests for both functions ## Priority: Medium Parent: Mnemosyne archive module (Phase 1 → Phase 2 evolution)
claude self-assigned this 2026-04-12 11:45:05 +00:00
bezalel was assigned by Timmy 2026-04-12 11:45:15 +00:00
claude commented 2026-04-12 11:49:52 +00:00
Member
Copy Link

PR created: #1276

Added ingest_file() and ingest_directory() to nexus/mnemosyne/ingest.py, plus mnemosyne ingest-dir CLI command. Dedup via source_ref (path + mtime), chunking on ## headings for large files, recursive directory walking with configurable extensions. 20 new tests, all passing.

PR created: https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus/pulls/1276 Added `ingest_file()` and `ingest_directory()` to `nexus/mnemosyne/ingest.py`, plus `mnemosyne ingest-dir` CLI command. Dedup via source_ref (path + mtime), chunking on `## ` headings for large files, recursive directory walking with configurable extensions. 20 new tests, all passing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
3d-world
CI
Continuous integration, runners, workflow issues
 QA
Quality assurance, testing, and production audit
 actionable
agent-presence
aistudio-ready
assigned-aistudio
assigned-claude
assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-gemini
assigned-groq
assigned-kimi
Dispatched to Kimi via OpenClaw
 assigned-kimi
assigned-perplexity
assigned-sonnet
blocked
Blocked by external dependency or merge conflict
 claude-ready
claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 deprioritized
duplicate
epic
Epic / umbrella issue
 gemini-api
Gemini API integration
 gemini-review
google-ai-ultra
Google AI Ultra integration work
 groq-ready
harness
identity
Timmy identity and branding
 infrastructure
kimi-done
Kimi completed this task
 kimi-in-progress
Kimi is actively working on this
 kimi-ready
lazzyPit
Lazarus Pit — automated agent resurrection and health recovery
 media-gen
AI media generation (image/video/audio)
 mnemosyne
Project Mnemosyne — The Living Holographic Archive
 modularization
needs-design
nostr
p0-critical
p1-important
p2-backlog
performance
perplexity-ready
portal
research
Deep research and planning tasks
 security
Security hardening, vulnerability fixes, access control
 sonnet-ready
sovereignty
velocity-engine
Auto-generated by velocity engine
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
KimiClaw Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok hermes kimi manus perplexity sonnet
No Assignees claude bezalel
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/the-nexus#1275
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?