diff --git a/intelligence/deepdive/OPERATIONAL_READINESS.md b/intelligence/deepdive/OPERATIONAL_READINESS.md new file mode 100644 index 0000000..a215f01 --- /dev/null +++ b/intelligence/deepdive/OPERATIONAL_READINESS.md @@ -0,0 +1,265 @@ +# Deep Dive — Operational Readiness Checklist + +> **Issue**: [#830](https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus/issues/830) — Deep Dive: Sovereign NotebookLM + Daily AI Intelligence Briefing +> **Location**: `intelligence/deepdive/OPERATIONAL_READINESS.md` +> **Created**: 2026-04-05 by Ezra, Archivist +> **Purpose**: Bridge the gap between "code complete" and "daily briefing delivered." This is the pre-flight checklist for making the Deep Dive pipeline operational on the Hermes VPS. + +--- + +## Executive Summary + +The Deep Dive pipeline is **code-complete and tested** (9/9 tests pass). This document defines the exact steps to move it into **daily production**. + +| Phase | Status | Blocker | +|-------|--------|---------| +| Code & tests | ✅ Complete | None | +| Documentation | ✅ Complete | None | +| Environment config | 🟡 **Needs verification** | Secrets, endpoints, Gitea URL | +| TTS engine | 🟡 **Needs install** | Piper model or ElevenLabs key | +| LLM endpoint | 🟡 **Needs running server** | `localhost:4000` or alternative | +| Systemd timer | 🟡 **Needs install** | `make install-systemd` | +| Live delivery | 🔴 **Not yet run** | Complete checklist below | + +--- + +## Step 1: Environment Prerequisites + +Run these checks on the host that will execute the pipeline (Hermes VPS): + +```bash +# Python 3.11+ +python3 --version + +# Git +git --version + +# Network outbound (arXiv, blogs, Telegram, Gitea) +curl -sI http://export.arxiv.org/api/query | head -1 +curl -sI https://api.telegram.org | head -1 +curl -sI https://forge.alexanderwhitestone.com | head -1 +``` + +**All must return HTTP 200.** + +--- + +## Step 2: Clone & Enter Repository + +```bash +cd /root/wizards/the-nexus/intelligence/deepdive +``` + +If the repo is not present: +```bash +git clone https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus.git /root/wizards/the-nexus +cd /root/wizards/the-nexus/intelligence/deepdive +``` + +--- + +## Step 3: Install Dependencies + +```bash +make install +``` + +This creates `~/.venvs/deepdive/` and installs: +- `feedparser`, `httpx`, `pyyaml` +- `sentence-transformers` + `all-MiniLM-L6-v2` model (~80MB) + +**Verify:** +```bash +~/.venvs/deepdive/bin/python -c "import feedparser, httpx, sentence_transformers; print('OK')" +``` + +--- + +## Step 4: Configure Secrets + +Export these environment variables (add to `~/.bashrc` or a `.env` file loaded by systemd): + +```bash +export GITEA_TOKEN="" +export TELEGRAM_BOT_TOKEN="" +# Optional, for cloud TTS fallback: +export ELEVENLABS_API_KEY="" +export OPENAI_API_KEY="" +``` + +**Verify Gitea connectivity:** +```bash +curl -s -H "Authorization: token $GITEA_TOKEN" \ + https://forge.alexanderwhitestone.com/api/v1/user | jq -r '.login' +``` + +Must print a valid username (e.g., `ezra`). + +**Verify Telegram bot:** +```bash +curl -s "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/getMe" | jq -r '.result.username' +``` + +Must print the bot username. + +--- + +## Step 5: TTS Engine Setup + +### Option A: Piper (sovereign, local) + +```bash +# Install piper binary (example for Linux x86_64) +mkdir -p ~/.local/bin +curl -L -o ~/.local/bin/piper \ + https://github.com/rhasspy/piper/releases/download/v1.2.0/piper_linux_x86_64.tar.gz +tar -xzf ~/.local/bin/piper -C ~/.local/bin/ +export PATH="$HOME/.local/bin:$PATH" + +# Download voice model (~2GB) +python3 -c " +from tts_engine import PiperTTS +tts = PiperTTS('en_US-lessac-medium') +print('Piper ready') +" +``` + +### Option B: ElevenLabs (cloud, premium quality) + +Ensure `ELEVENLABS_API_KEY` is exported. No local binary needed. + +### Option C: OpenAI TTS (cloud, balance) + +Update `config.yaml`: +```yaml +tts: + engine: "openai" + voice: "alloy" +``` + +Ensure `OPENAI_API_KEY` is exported. + +--- + +## Step 6: LLM Endpoint Verification + +The default config points to `http://localhost:4000/v1` (LiteLLM or local llama-server). + +**Verify the endpoint is listening:** +```bash +curl http://localhost:4000/v1/models +``` + +If the endpoint is down, either: +1. Start it: `llama-server -m model.gguf --port 4000 -ngl 999 --jinja` +2. Or change `synthesis.llm_endpoint` in `config.yaml` to an alternative (e.g., OpenRouter, Kimi, Anthropic). + +--- + +## Step 7: Dry-Run Verification + +```bash +make run-dry +``` + +Expected output includes: +- `Phase 1: Source Aggregation` with >0 items fetched +- `Phase 2: Relevance Scoring` with >0 items ranked +- `Phase 0: Fleet Context Grounding` with 4 repos, commits, issues +- `Phase 3: Synthesis` with briefing saved to `~/.cache/deepdive/` +- `Phase 4: Audio disabled` (if TTS not configured) or audio path +- `Phase 5: DRY RUN - delivery skipped` + +**If any phase errors, fix before proceeding.** + +--- + +## Step 8: First Live Run + +⚠️ **This will send a Telegram message to the configured channel.** + +```bash +make run-live +# Type 'y' when prompted +``` + +Watch for: +- Telegram text summary delivery +- Telegram voice message delivery (if TTS + audio enabled) + +--- + +## Step 9: Install Systemd Timer (Daily 06:00) + +```bash +make install-systemd +``` + +**Verify:** +```bash +systemctl --user status deepdive.timer +systemctl --user list-timers --all | grep deepdive +``` + +To trigger a manual run via systemd: +```bash +systemctl --user start deepdive.service +journalctl --user -u deepdive.service -f +``` + +--- + +## Step 10: Monitoring & Rollback + +### Monitor daily runs +```bash +journalctl --user -u deepdive.service --since today +``` + +### Check latest briefing +```bash +ls -lt ~/.cache/deepdive/briefing_*.json | head -1 +``` + +### Disable timer (rollback) +```bash +systemctl --user stop deepdive.timer +systemctl --user disable deepdive.timer +``` + +### Clean reinstall +```bash +make clean +make install +make test +``` + +--- + +## Known Gaps & Mitigations + +| Gap | Impact | Mitigation | +|-----|--------|------------| +| arXiv RSS empty on weekends | Empty briefing Sat/Sun | ArXiv API fallback is implemented | +| `feedparser` missing | RSS skipped | API fallback activates automatically | +| `localhost:4000` down | Synthesis uses template | Start LLM endpoint or update config | +| Piper model ~2GB download | First TTS run slow | Pre-download during `make install` | +| Telegram rate limits | Delivery delayed | Retry is manual; add backoff if needed | + +--- + +## Sign-Off + +| Check | Verified By | Date | +|-------|-------------|------| +| Dependencies installed | | | +| Secrets configured | | | +| TTS engine ready | | | +| LLM endpoint responding | | | +| Dry-run successful | | | +| Live run successful | | | +| Systemd timer active | | | + +--- + +*Created by Ezra, Archivist | 2026-04-05* diff --git a/intelligence/deepdive/config.yaml b/intelligence/deepdive/config.yaml index 97c9da6..3ab2b87 100644 --- a/intelligence/deepdive/config.yaml +++ b/intelligence/deepdive/config.yaml @@ -91,7 +91,7 @@ deepdive: # Phase 0: Fleet Context Grounding fleet_context: enabled: true - gitea_url: "http://143.198.27.163:3000" + gitea_url: "https://forge.alexanderwhitestone.com" token: "${GITEA_TOKEN}" # From environment owner: "Timmy_Foundation" repos: