# Deep Dive: Sovereign Daily Intelligence Briefing

> **Parent**: the-nexus#830  
> **Created**: 2026-04-05 by Ezra burn-mode triage  
> **Status**: Architecture proof, Phase 1 ready for implementation

## Executive Summary

**Deep Dive** is a fully automated, sovereign alternative to NotebookLM. It aggregates AI/ML intelligence from arXiv, lab blogs, and newsletters; filters by relevance to Hermes/Timmy work; synthesizes into structured briefings; and delivers as audio podcasts via Telegram.

This document provides the technical decomposition to transform #830 from 21-point EPIC to executable child issues.

---

## System Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  SOURCE LAYER   │───▶│  FILTER LAYER   │───▶│ SYNTHESIS LAYER │
│   (Phase 1)     │    │   (Phase 2)     │    │   (Phase 3)     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                      │                      │
         ▼                      ▼                      ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│ • arXiv RSS     │    │ • Keyword match │    │ • LLM prompt    │
│ • Blog scrapers │    │ • Embedding sim │    │ • Context inj   │
│ • Newsletters   │    │ • Ranking algo  │    │ • Brief gen     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                       │
                                                       ▼
                                              ┌─────────────────┐
                                              │  OUTPUT LAYER   │
                                              │ (Phases 4-5)    │
                                              ├─────────────────┤
                                              │ • TTS pipeline  │
                                              │ • Audio file    │
                                              │ • Telegram bot  │
                                              │ • Cron schedule │
                                              └─────────────────┘
```

---

## Phase Decomposition

### Phase 1: Source Aggregation (2-3 points)
**Dependencies**: None. Can start immediately.

| Source | Method | Rate Limit | Notes |
|--------|--------|------------|-------|
| arXiv | RSS + API | 1 req/3 sec | cs.AI, cs.CL, cs.LG categories |
| OpenAI Blog | RSS feed | None | Research + product announcements |
| Anthropic | RSS + sitemap | Respect robots.txt | Research publications |
| DeepMind | RSS feed | None | arXiv cross-posts + blog |
| Import AI | Newsletter | Manual | RSS if available |
| TLDR AI | Newsletter | Manual | Web scrape if no RSS |

**Implementation Path**:
```python
# scaffold/deepdive/phase1/arxiv_aggregator.py
# ArXiv RSS → JSON lines store
# Daily cron: fetch → parse → dedupe → store
```

**Sovereignty**: Zero API keys needed for RSS. arXiv API is public.

### Phase 2: Relevance Engine (4-5 points)
**Dependencies**: Phase 1 data store

**Embedding Strategy**:
| Option | Model | Local? | Quality | Speed |
|--------|-------|--------|---------|-------|
| **Primary** | nomic-embed-text-v1.5 | ✅ llama.cpp | Good | Fast |
| Fallback | all-MiniLM-L6-v2 | ✅ sentence-transformers | Good | Medium |
| Cloud | OpenAI text-embedding-3 | ❌ | Best | Fast |

**Relevance Scoring**:
1. Keyword pre-filter (Hermes, agent, LLM, RL, training)
2. Embedding similarity vs codebase embedding
3. Rank by combined score (keyword + embedding + recency)
4. Pick top 10 items per briefing

**Implementation Path**:
```python
# scaffold/deepdive/phase2/relevance_engine.py
# Load daily items → embed → score → rank → filter
```

### Phase 3: Synthesis Engine (3-4 points)
**Dependencies**: Phase 2 filtered items

**Prompt Architecture**:
```
SYSTEM: You are Deep Dive, an AI intelligence analyst for the Hermes/Timmy project.
Your task: synthesize daily AI/ML news into a 5-7 minute briefing.

CONTEXT: Hermes is an open-source LLM agent framework. Key interests:
- LLM architecture and training
- Agent systems and tool use
- RL and GRPO training
- Open-source model releases

OUTPUT FORMAT:
1. HEADLINES (3 items): One-sentence summaries with impact tags [MAJOR|MINOR]
2. DEEP DIVE (1-2 items): Paragraph with context + implications for Hermes
3. IMPLICATIONS: "Why this matters for our work"
4. SOURCES: Citation list

TONE: Professional, concise, actionable. No fluff.
```

**LLM Options**:
| Option | Source | Local? | Quality | Cost |
|--------|--------|--------|---------|------|
| **Primary** | Gemma 4 E4B via Hermes | ✅ | Excellent | Zero |
| Fallback | Kimi K2.5 via OpenRouter | ❌ | Excellent | API credits |
| Fallback | Claude via Anthropic | ❌ | Best | $$ |

### Phase 4: Audio Generation (5-6 points)
**Dependencies**: Phase 3 text output

**TTS Pipeline Decision Matrix**:
| Option | Engine | Local? | Quality | Speed | Cost |
|--------|--------|--------|---------|-------|------|
| **Primary** | Piper TTS | ✅ | Good | Fast | Zero |
| Fallback | Coqui TTS | ✅ | Good | Slow | Zero |
| Fallback | MMS | ✅ | Medium | Fast | Zero |
| Cloud | ElevenLabs | ❌ | Best | Fast | $ |
| Cloud | OpenAI TTS | ❌ | Great | Fast | $ |

**Recommendation**: Implement local Piper first. If quality insufficient for daily use, add ElevenLabs as quality-gated fallback.

**Voice Selection**:
- Piper: `en_US-lessac-medium` (balanced quality/speed)
- ElevenLabs: `Josh` or clone custom voice

### Phase 5: Delivery Pipeline (3-4 points)
**Dependencies**: Phase 4 audio file

**Components**:
1. **Cron Scheduler**: Daily 06:00 EST trigger
2. **Telegram Bot Integration**: Send voice message via existing gateway
3. **On-demand Trigger**: `/deepdive` slash command in Hermes
4. **Storage**: Audio file cache (7-day retention)

**Telegram Voice Message Format**:
- OGG Opus (Telegram native)
- Piper outputs WAV → convert via ffmpeg
- 10-15 minute typical length

---

## Data Flow

```
06:00 EST (cron)
    │
    ▼
┌─────────────┐
│ Run Aggregator│◄── Daily fetch of all sources
└─────────────┘
    │
    ▼ JSON lines store
┌─────────────┐
│ Run Relevance │◄── Embed + score + rank
└─────────────┘
    │
    ▼ Top 10 items
┌─────────────┐
│ Run Synthesis │◄── LLM prompt → briefing text
└─────────────┘
    │
    ▼ Markdown + raw text
┌─────────────┐
│ Run TTS     │◄── Text → audio file
└─────────────┘
    │
    ▼ OGG Opus file
┌─────────────┐
│ Telegram Send │◄── Voice message to channel
└─────────────┘
    │
    ▼
Alexander receives daily briefing ☕
```

---

## Child Issue Decomposition

| Child Issue | Scope | Points | Owner | Blocked By |
|-------------|-------|--------|-------|------------|
| the-nexus#830.1 | Phase 1: arXiv RSS aggregator | 3 | @ezra | None |
| the-nexus#830.2 | Phase 1: Blog scrapers (OpenAI, Anthropic, DeepMind) | 2 | TBD | None |
| the-nexus#830.3 | Phase 2: Relevance engine + embeddings | 5 | TBD | 830.1, 830.2 |
| the-nexus#830.4 | Phase 3: Synthesis prompts + briefing template | 4 | TBD | 830.3 |
| the-nexus#830.5 | Phase 4: TTS pipeline (Piper + fallback) | 6 | TBD | 830.4 |
| the-nexus#830.6 | Phase 5: Telegram delivery + `/deepdive` command | 4 | TBD | 830.5 |

**Total**: 24 points (original 21 was optimistic; TTS integration complexity warrants 6 points)

---

## Sovereignty Preservation

| Component | Sovereign Path | Trade-off |
|-----------|---------------|-----------|
| Source aggregation | RSS (no API keys) | Limited metadata vs API |
| Embeddings | nomic-embed-text via llama.cpp | Setup complexity |
| LLM synthesis | Gemma 4 via Hermes | Requires local GPU |
| TTS | Piper (local, fast) | Quality vs ElevenLabs |
| Delivery | Hermes Telegram gateway | Already exists |

**Fallback Plan**: If local GPU unavailable for synthesis, use Kimi K2.5 via OpenRouter. If Piper quality unacceptable, use ElevenLabs with budget cap.

---

## Directory Structure

```
the-nexus/
├── docs/deep-dive-architecture.md      (this file)
├── scaffold/deepdive/
│   ├── phase1/
│   │   ├── arxiv_aggregator.py       (proof-of-concept)
│   │   ├── blog_scraper.py
│   │   └── config.yaml               (source URLs, categories)
│   ├── phase2/
│   │   ├── relevance_engine.py
│   │   └── embeddings.py
│   ├── phase3/
│   │   ├── synthesis.py
│   │   └── briefing_template.md
│   ├── phase4/
│   │   ├── tts_pipeline.py
│   │   └── piper_config.json
│   └── phase5/
│       ├── telegram_delivery.py
│       └── deepdive_command.py
├── data/deepdive/                      (gitignored)
│   ├── raw/                            # Phase 1 output
│   ├── scored/                         # Phase 2 output
│   ├── briefings/                      # Phase 3 output
│   └── audio/                          # Phase 4 output
└── cron/deepdive.sh                    # Daily runner
```

---

## Proof-of-Concept: Phase 1 Stub

See `scaffold/deepdive/phase1/arxiv_aggregator.py` for immediately executable arXiv RSS fetcher.

**Zero dependencies beyond stdlib + feedparser** (can use xml.etree if strict).

**Can run today**: No API keys, no GPU, no TTS decisions needed.

---

## Acceptance Criteria Mapping

| Original Criterion | Implementation | Owner |
|-------------------|----------------|-------|
| Zero manual copy-paste | RSS aggregation + cron | 830.1, 830.2 |
| Daily delivery 6 AM | Cron trigger | 830.6 |
| arXiv cs.AI/CL/LG | arXiv RSS categories | 830.1 |
| Lab blogs | Blog scrapers | 830.2 |
| Relevance ranking | Embedding similarity | 830.3 |
| Hermes context | Synthesis prompt injection | 830.4 |
| TTS audio | Piper/ElevenLabs | 830.5 |
| Telegram voice | Bot integration | 830.6 |
| On-demand `/deepdive` | Slash command | 830.6 |

---

## Immediate Next Action

**@ezra** will implement Phase 1 proof-of-concept (`arxiv_aggregator.py`) to validate pipeline architecture and unblock downstream phases.

**Estimated time**: 2 hours to working fetch+store.

---

*Document created during Ezra burn-mode triage of the-nexus#830*