diff --git a/docs/deepseek-cost-routing.md b/docs/deepseek-cost-routing.md new file mode 100644 index 0000000..242bed3 --- /dev/null +++ b/docs/deepseek-cost-routing.md @@ -0,0 +1,77 @@ +# DeepSeek v4-pro: direct vs OpenRouter — measurement + routing plan + +Date: 2026-10-20. Live A/B, same prompt both sides (measure-v4pro-cost.py). + +## Measured (one representative reasoning task) + +| | DeepSeek direct | OpenRouter | +|---|---|---| +| prompt tokens | 198 | 119 | +| completion tokens | 17,181 | 6,644 | +| reasoning tokens | 16,008 | 5,738 | +| actual billed | (computed) | **$0.0100731** | + +OpenRouter's own `cost` field → effective rates: +- output: $0.009966 / 6,644 = **~$1.50/M** (NOT the $0.75 headline) +- input: $0.0001071 / 119 = **~$0.90/M** (NOT the $0.25 headline) + +## Per-token comparison (our workload is ~90% output/reasoning) + +| scheme | in/M | out/M | +|---|---|---| +| DS direct off-peak | $0.66 | $1.98 | +| DS direct peak | $1.32 | $3.96 | +| OpenRouter (measured) | ~$0.90 | ~$1.50 | +| OpenRouter (headline, misleading) | $0.25 | $0.75 | + +## Conclusions + +1. OpenRouter's headline price is misleading — the real billed rate is ~$0.90/$1.50, not $0.25/$0.75. +2. Per-token: OpenRouter beats DeepSeek on OUTPUT (~24% cheaper off-peak, ~62% cheaper peak), but LOSES on INPUT (~36% dearer than off-peak). +3. For our output-heavy workload, OpenRouter is a net win mainly at PEAK (dodges the 2× surcharge). +4. YELLOW FLAG: OpenRouter emitted 2.6× fewer reasoning tokens (5,738 vs 16,008) for the identical prompt — possibly shallower reasoning (or stochastic). Quality unverified. + +## Plan + +1. **Quality gate (blocking):** 3-task benchmark, both routes, judged on answer quality. If OpenRouter answers are ≥ our bar, proceed; else drop the idea. +2. **Peak-routing:** route peak-hours v4-pro through OpenRouter (flat ~$1.50/M out vs $3.96/M peak); keep off-peak direct (our $0.66 input is cheaper, and $1.98 off-peak output is close enough to not justify the quality risk). +3. **Free-tier flash remains primary** for wizard work (opencode deepseek-v4.1-flash-free + free models) — the routing only concerns v4-pro (main agent + dreams). +4. **Track OR spend** in the existing balance watch if we route there. + +Decision gate owner: Timmy. Approver: Alexander. + +## Nous Portal check (2026-10-20) + +- **Flash discount (real):** DeepSeek V4.1 Flash 1017 is **77% off → $0.07 in / $0.50 out** (vs our direct $0.15/$0.60 off-peak, $0.30/$1.20 peak). Plus a partner promo: "V4 Flash 0731 90% off for 7 days." +- **No Pro deal:** Nous Portal V4 Pro 0813 = $0.26 in / **$8.00 out** — output is ~4× our direct off-peak ($1.98). Worse for reasoning-heavy work. +- **No Nous key configured** in env (would need a subscription to use the flash promo). + +### Revised conclusion (post quality gate) +- **Quality: comparable.** OpenRouter's v4-pro answered all 3 tasks correctly and completely; the earlier 2.6× reasoning-token drop was run-to-run variance (on the planning task OR emitted *more* tokens than DS). +- **Cost: a wash.** OR was cheaper on 2 tasks, ~2.5× dearer on 1 (it emitted more tokens). No reliable off-peak saving. +- **V4 Pro: direct DeepSeek off-peak remains the best real deal.** OpenRouter's only edge is dodging the 2× peak surcharge — but we already moved heavy v4-pro work off-peak, so little peak usage remains to capture. Not worth auto-flipping the main agent's provider (risk > reward). +- **Flash:** Nous Portal (77% off) / Deep Infra ($0.06) are cheaper, but we already run flash work on FREE models — keep as fallback only. +- **Captured wins remain:** off-peak scheduling + free wizard models + free opencode flash route. + +## Verification status (appended 2026-10-24 by Timmy, control plane) + +This file stayed untracked until today — it was written to the worktree on 2026-10-20 and +never committed, so no reader of the repository could open the document those rates came +from. It is committed now, and the ledger row that tolerated it +(`tools/path_claim_baseline.tsv`, class NOT_ADDED, added by Volt at `54961d4`) now reads +CLEARED. + +- **Arithmetic: Documented.** The billed field is the sum of its two parts ($0.009966 + + $0.0001071 = $0.0100731 exactly), and both effective rates reproduce from it: + 0.009966 / 6,644 = $1.5000/M output, 0.0001071 / 119 = $0.9000/M input. Every figure in + the two tables above follows from the quoted fields. +- **DeepSeek direct rates: Corroborated.** $0.66 input / $1.98 output off-peak and $1.32 / + $3.96 peak, with peak at 01:00-04:00 and 06:00-10:00 UTC Monday-Friday, reproduced + 2026-10-24 from three independent secondary sources (mercatus-ai.com, deepseekprice.com, + cabina.ai). The primary rate card is **Unrecovered**: the web-extract backend on this host + is search-only, so `api-docs.deepseek.com/quick_start/pricing` itself was never fetched. +- **OpenRouter billed field: Testimony only.** One live API response on 2026-10-20, not + re-fetched since; the $1.50/$0.90 effective rates are derived from that response, not from + any published OpenRouter rate sheet. +- **The quality gate above ran on 2026-10-20 and was not re-run.** The conclusion it produced + is unchanged: direct off-peak for v4-pro, free flash for wizard work, no auto-flip.