diff --git a/docs/DEEPSDIVE_EXECUTION.md b/docs/DEEPSDIVE_EXECUTION.md new file mode 100644 index 0000000..76d2236 --- /dev/null +++ b/docs/DEEPSDIVE_EXECUTION.md @@ -0,0 +1,167 @@ +# Deep Dive — Execution Runbook + +> Parent: [#830](http://143.198.27.163:3000/Timmy_Foundation/the-nexus/issues/830) +> Location: `docs/DEEPSDIVE_EXECUTION.md` +> Updated: 2026-04-05 +> Owner: @ezra + +## Quick Start + +Zero-to-briefing in 10 minutes: + +```bash +cd /root/wizards/the-nexus + +# 1. Configure (~5 min) +export DEEPDIVE_TTS_PROVIDER=openai # or "elevenlabs" or "piper" +export OPENAI_API_KEY=sk-... # or ELEVENLABS_API_KEY +export DEEPDIVE_TELEGRAM_BOT_TOKEN=... # BotFather +export DEEPDIVE_TELEGRAM_CHAT_ID=... # Your Telegram chat ID + +# 2. Test run (~2 min) +./bin/deepdive_orchestrator.py --dry-run + +# 3. Full delivery (~5 min) +./bin/deepdive_orchestrator.py --date $(date +%Y-%m-%d) +``` + +--- + +## Provider Decision Matrix + +| Provider | Cost | Quality | Latency | Setup Complexity | Best For | +|----------|------|---------|---------|------------------|----------| +| **Piper** | Free | Medium | Fast (local) | High (model download) | Privacy-first, offline | +| **ElevenLabs** | $5/mo | High | Medium (~2s) | Low | Production quality | +| **OpenAI** | ~$0.015/1K chars | Good | Fast (~1s) | Low | Quick start, good balance | + +**Recommendation**: Start with OpenAI (`tts-1` model, `alloy` voice) for immediate results. Migrate to ElevenLabs for final polish if budget allows. + +--- + +## Phase-by-Phase Testing + +### Phase 1: Aggregation Test +```bash +./bin/deepdive_aggregator.py --sources arxiv_cs_ai --output /tmp/test_agg.json +cat /tmp/test_agg.json | jq ".metadata" +``` + +### Phase 2: Filtering Test (via Orchestrator) +```bash +./bin/deepdive_orchestrator.py --date 2026-04-05 --stop-after phase2 +ls ~/the-nexus/deepdive_state/2026-04-05/ranked.json +``` + +### Phase 3: Synthesis Test (requires LLM setup) +```bash +export OPENAI_API_KEY=sk-... +./bin/deepdive_orchestrator.py --date 2026-04-05 --stop-after phase3 +cat ~/the-nexus/deepdive_state/2026-04-05/briefing.md +``` + +### Phase 4: TTS Test +```bash +echo "Hello from Deep Dive. This is a test." | ./bin/deepdive_tts.py --output /tmp/test +ls -la /tmp/test.mp3 +``` + +### Phase 5: Delivery Test +```bash +./bin/deepdive_delivery.py --audio /tmp/test.mp3 --caption "Deep Dive test" --dry-run +./bin/deepdive_delivery.py --audio /tmp/test.mp3 --caption "Deep Dive test" +``` + +--- + +## Environment Variables Reference + +### Required +| Variable | Purpose | Example | +|----------|---------|---------| +| `DEEPDIVE_TTS_PROVIDER` | TTS adapter selection | `openai`, `elevenlabs`, `piper` | +| `OPENAI_API_KEY` or `ELEVENLABS_API_KEY` | API credentials | `sk-...` | +| `DEEPDIVE_TELEGRAM_BOT_TOKEN` | Telegram bot auth | `123456:ABC-DEF...` | +| `DEEPDIVE_TELEGRAM_CHAT_ID` | Target chat | `@yourusername` or `-1001234567890` | + +### Optional +| Variable | Default | Description | +|----------|---------|-------------| +| `DEEPDIVE_TTS_VOICE` | `alloy` / `matthew` | Voice ID | +| `DEEPDIVE_OUTPUT_DIR` | `~/the-nexus/deepdive_state` | State storage | +| `DEEPDIVE_LLM_PROVIDER` | `openai` | Synthesis LLM | +| `DEEPDIVE_MAX_ITEMS` | `10` | Items per briefing | + +--- + +## Cron Installation + +Daily 6 AM briefing: + +```bash +# Add to crontab +crontab -e + +# Entry: +0 6 * * * cd /root/wizards/the-nexus && ./bin/deepdive_orchestrator.py --date $(date +\%Y-\%m-\%d) >> /var/log/deepdive.log 2>&1 +``` + +Verify cron environment has all required exports by adding to `~/.bashrc` or using absolute paths in crontab. + +--- + +## Troubleshooting + +### "No items found" from aggregator +- Check internet connectivity +- Verify arXiv RSS is accessible: `curl http://export.arxiv.org/rss/cs.AI` + +### "Audio file not valid" from Telegram +- Ensure MP3 format, reasonable file size (< 50MB) +- Test with local playback: `mpg123 /tmp/test.mp3` + +### "Telegram chat not found" +- Use numeric chat ID for groups: `-1001234567890` +- For personal chat, message @userinfobot + +### Piper model not found +```bash +mkdir -p ~/.local/share/piper +cd ~/.local/share/piper +wget https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/lessac/medium/en_US-lessac-medium.onnx +wget https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/lessac/medium/en_US-lessac-medium.onnx.json +``` + +--- + +## Architecture Recap + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ D E E P D I V E V1 .1 │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────┐ ┌─────────────┐ ┌──────────────┐ │ +│ │ deepdive_aggregator.py │ deepdive_orchestrator.py │ │ +│ │ (arXiv RSS) │───▶│ (filter) │───▶│ (synthesize)│───▶ ... │ +│ └─────────────────┘ └─────────────┘ └──────────────┘ │ +│ │ │ +│ deepdive_tts.py ◀──────────┘ │ +│ (TTS adapter) │ +│ │ │ +│ deepdive_delivery.py │ +│ (Telegram voice msg) │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Next Steps for Full Automation + +- [ ] **LLM Integration**: Complete `orchestrator.phase3()` with LLM API call +- [ ] **Prompt Engineering**: Design briefing format prompt with Hermes context +- [ ] **Source Expansion**: Add lab blogs (OpenAI, Anthropic, DeepMind) +- [ ] **Embedding Filter**: Replace keyword scoring with semantic similarity +- [ ] **Metrics**: Track delivery success, user engagement, audio length + +**Status**: Phases 1, 2, 4, 5 scaffolded and executable. Phase 3 synthesis awaiting LLM integration.