---
name: codebase-extraction-pipeline
description: >
  Four-phase pipeline for extracting architectural knowledge from a large
  codebase drop. Automated inventory → pattern extraction → adaptation specs
  → knowledge ingestion. Designed for studying reference implementations
  (like Claude Code 512K lines) and translating patterns into actionable
  specs for your own stack. Splits work between agents by capability.
triggers:
  - study a codebase
  - extract patterns from source
  - deep dive source code
  - analyze reference implementation
  - learn from open source drop
tags: [architecture, extraction, research, codebase-analysis, knowledge-transfer]
---

# Codebase Extraction Pipeline

## When to Use
A large codebase drop lands (open source release, fork, reference implementation)
and you need to systematically extract useful patterns, not just read randomly.

## The Four Phases

### Phase 1: Automated Inventory (Kimi/grunt agent burns this)

**Goal:** Machine-readable map of the entire codebase before any human reads it.

**Scripts to run:**

1. **File inventory with line counts:**
```bash
find src -type f -name "*.ts" -o -name "*.py" | while read f; do
  echo "$(wc -l < "$f") $f"
done | sort -rn > inventory.txt
```

2. **Dependency graph (imports/exports):**
```python
import os, re, json
from collections import defaultdict
imports = defaultdict(list)
for root, dirs, files in os.walk('src'):
    for f in files:
        path = os.path.join(root, f)
        with open(path, 'r', errors='ignore') as fh:
            for line in fh:
                m = re.match(r"import.*from ['\"](.+?)['\"]", line)
                if m: imports[path].append(m.group(1))
# Count most-imported modules to find load-bearing walls
```

3. **Symbol catalog (exported functions, classes, types):**
```python
# Regex for exports, capture kind + name + docstring
catalog = []
# re.match(r"export (function|class|const|type|interface) (\w+)", line)
```

4. **Feature flags / conditional compilation:**
```bash
grep -rn "feature(" src/ | extract flag names + surrounding context
```

**Output:** `analysis/inventory.txt`, `analysis/dependency-graph.json`, `analysis/symbol-catalog.json`, `analysis/feature-flags.txt`

**Assignment:** Give to the grunt agent (Kimi/Allegro) — pure automation, no judgment needed.

### Phase 2: Pattern Extraction (deep-read agents, parallel)

**Goal:** For each backlog ticket that maps to the reference codebase, extract the SPECIFIC architectural pattern: decision logic, prompts, algorithms, data structures.

**Template per pattern:**
```markdown
# Pattern: [Name]

## Decision Logic
[When does this fire? What thresholds? Extract the actual if-statement.]

## The Prompt (if applicable)
[Copy verbatim — prompts are the most valuable single artifact.]

## Algorithm
[Step-by-step, with code excerpts for non-obvious parts.]

## Data Structures
[What types/schemas are used? Copy the type definitions.]

## Edge Cases Handled
[What did they handle that we haven't thought of?]
```

**Assignment:** Split by domain expertise. The reasoning agent gets architecture patterns. The code agent gets integration patterns.

### Phase 3: Adaptation Specs (1-page design docs)

**Goal:** For each extracted pattern, write a 1-page spec translating it to your stack.

**Template:**
```markdown
# Adaptation Spec: [Feature Name]

## Source Pattern
[Reference to Phase 2 pattern doc]

## Our Constraints
[What's different — language, runtime, local-first, etc.]

## Design Decision
[What to adopt, what to skip, what to change]

## Implementation Plan
[Which files to modify, estimated LOC, dependencies]

## Verification
[How to test that it works]
```

**Assignment:** Builder agent writes, oracle agent reviews.

### Phase 4: Knowledge Ingestion

**Goal:** All pattern docs and specs become queryable knowledge, not just files on disk.

**Method:** Until RAG pipeline exists, store as indexed markdown with a manifest:
```json
{
  "patterns": [
    {"file": "compact.md", "tags": ["compression", "context"], "summary": "..."},
    ...
  ]
}
```

## Pitfalls

1. **Don't start Phase 2 without Phase 1.** The inventory tells you WHERE to look. Without it, agents read randomly and miss the important files.

2. **Copy prompts verbatim.** The extraction prompt, the compaction prompt, the memory prompt — these are the most valuable artifacts. Don't summarize them. Copy them.

3. **The largest files are not always the most important.** The dependency graph (Phase 1 step 2) reveals which files are load-bearing. A 500-line utility imported by 40 files matters more than a 5,000-line leaf.

4. **Feature flags reveal the roadmap.** Unreleased features behind flags tell you what the maintainers are building next. Catalog them.

5. **Private repos require explicit API paths.** `repos/search` doesn't return private repos. Use `repos/{owner}/{repo}` directly.

6. **Split work by agent capability.** Grunt work (inventory, grep, catalog) → Kimi/cheap agent. Deep reads (architecture, decisions) → strong reasoning agent. Specs → builder agent. Review → oracle agent.

7. **Translate patterns, don't transliterate code.** TypeScript patterns adapted to Python should feel Pythonic. Don't write Python that looks like TypeScript.

## Provenance
Developed during Claude Code source drop analysis (2026-03-31).
512K lines TypeScript → 14 extraction tickets across 4 phases.
Pattern proven on: compact system, memory extraction, agent forking, bridge protocol, hook system, cost tracking.
