[Mnemosyne] Pluggable Embedding Backend — Ollama + TF-IDF #1257

Rockachopa · 2026-04-12T05:08:14Z

Rockachopa commented

2026-04-12 05:08:14 +00:00

Pluggable Embedding Backend for Mnemosyne

Implements the high-priority embedding_backend planned feature from FEATURES.yaml.

What's New

embeddings.py — Abstract EmbeddingBackend interface with two implementations:
- OllamaEmbeddingBackend: connects to local Ollama for sovereign neural embeddings (default model: nomic-embed-text)
- TfidfEmbeddingBackend: pure-Python TF-IDF fallback — no dependencies, always available
- get_embedding_backend(): auto-detects best available backend
linker.py updated — HolographicLinker accepts optional EmbeddingBackend:
- Uses cosine similarity on vectors when embedding backend is available
- Falls back to Jaccard token similarity gracefully
- Embedding cache for performance during bulk link operations
archive.py updated — MnemosyneArchive.__init__ accepts optional EmbeddingBackend:
- Auto-detects via get_embedding_backend() by default
- semantic_search() uses embedding cosine similarity when available, with link boost
cli.py updated — --backend ollama|tfidf|auto flag on search and rebuild commands
tests/test_embeddings.py — test suite for cosine similarity, TF-IDF backend, auto-detection
__init__.py — exports new embedding classes

Configuration

MNEMOSYNE_EMBED_BACKEND env var: force ollama or tfidf
MNEMOSYNE_EMBED_MODEL env var: override embedding model (default: nomic-embed-text)
OLLAMA_URL env var: override Ollama URL (default: http://localhost:11434)

Test

cd the-nexus && python -m pytest nexus/mnemosyne/tests/test_embeddings.py -v

Closes planned feature: embedding_backend in FEATURES.yaml.

## Pluggable Embedding Backend for Mnemosyne Implements the high-priority `embedding_backend` planned feature from FEATURES.yaml. ### What's New - **`embeddings.py`** — Abstract `EmbeddingBackend` interface with two implementations: - `OllamaEmbeddingBackend`: connects to local Ollama for sovereign neural embeddings (default model: `nomic-embed-text`) - `TfidfEmbeddingBackend`: pure-Python TF-IDF fallback — no dependencies, always available - `get_embedding_backend()`: auto-detects best available backend - **`linker.py`** updated — `HolographicLinker` accepts optional `EmbeddingBackend`: - Uses cosine similarity on vectors when embedding backend is available - Falls back to Jaccard token similarity gracefully - Embedding cache for performance during bulk link operations - **`archive.py`** updated — `MnemosyneArchive.__init__` accepts optional `EmbeddingBackend`: - Auto-detects via `get_embedding_backend()` by default - `semantic_search()` uses embedding cosine similarity when available, with link boost - **`cli.py`** updated — `--backend ollama|tfidf|auto` flag on `search` and `rebuild` commands - **`tests/test_embeddings.py`** — test suite for cosine similarity, TF-IDF backend, auto-detection - **`__init__.py`** — exports new embedding classes ### Configuration - `MNEMOSYNE_EMBED_BACKEND` env var: force `ollama` or `tfidf` - `MNEMOSYNE_EMBED_MODEL` env var: override embedding model (default: `nomic-embed-text`) - `OLLAMA_URL` env var: override Ollama URL (default: `http://localhost:11434`) ### Test ```bash cd the-nexus && python -m pytest nexus/mnemosyne/tests/test_embeddings.py -v ``` Closes planned feature: embedding_backend in FEATURES.yaml.

Rockachopa added 7 commits 2026-04-12 05:08:15 +00:00

feat: add pluggable embedding backend (Ollama + TF-IDF) 3bf8d6e0a6

Implements embedding_backend from FEATURES.yaml:
- Abstract EmbeddingBackend interface
- OllamaEmbeddingBackend for local sovereign models
- TfidfEmbeddingBackend pure-Python fallback
- get_embedding_backend() auto-detection

feat: linker supports pluggable embedding backend fd7c66bd54

HolographicLinker now accepts optional EmbeddingBackend.
Uses cosine similarity on embeddings when available,
falls back to Jaccard token similarity otherwise.
Embedding cache for performance during link operations.

feat: archive.py uses embedding backend for semantic search bd27cd4bf5

- MnemosyneArchive.__init__ accepts optional EmbeddingBackend
- Auto-detects best backend via get_embedding_backend()
- semantic_search uses embedding cosine similarity when available
- Falls back to Jaccard token similarity gracefully

test: add embedding backend test suite 85dda06ff0

Tests cosine similarity, TF-IDF backend,
auto-detection, and fallback behavior.

feat: CLI --backend flag for embedding backend selection fcdb049117

- search: --backend ollama|tfidf|auto
- rebuild: --backend flag
- Auto-detects best backend when --semantic is used

docs: mark embedding_backend as shipped in FEATURES.yaml 1f8994abc9

feat: export embedding backends from mnemosyne __init__

CI / test (pull_request) Failing after 10s

Details

CI / validate (pull_request) Failing after 15s

Details

Review Approval Gate / verify-review (pull_request) Failing after 2s

Details

3fca28b1c8

Rockachopa requested review from perplexity 2026-04-12 05:08:16 +00:00

Timmy merged commit 89e19dbaa2 into main

2026-04-12 05:30:05 +00:00

Timmy referenced this issue from a commit

2026-04-12 05:30:06 +00:00

Merge PR #1257

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Timmy_Foundation/the-nexus#1257