timmy-tower/TIMMY_TEST_PLAN.md

Timmy API — Test Plan & Report Prompt

What is Timmy? Timmy is a Lightning Network-gated AI agent API. Users pay Bitcoin (via Lightning) to submit requests to an AI agent (Claude). Two payment modes:

• Mode 1 — Per-Job (live): Pay per request. Eval invoice (10 sats fixed) → Haiku judges the request → work invoice (dynamic, token-based) → Sonnet executes → result delivered.
• Mode 2 — Session (live): Pre-fund a credit balance. Requests automatically debit actual compute cost (eval + work tokens × 1.4 margin, converted to sats at live BTC/USD). No per-job invoices once active.

Live base URL:

https://9f85e954-647c-46a5-90a7-396e495a805a-00-clz2vhmfuk7p.spock.replit.dev

---

Running the full test suite — one command

curl -s https://9f85e954-647c-46a5-90a7-396e495a805a-00-clz2vhmfuk7p.spock.replit.dev/api/testkit | bash

The server returns a self-contained bash script with the base URL already baked in. Requirements: curl, bash, jq — nothing else.

Note for repeat runs: Tests 7 and 8c hit GET /api/demo, which is rate-limited to 5 req/hr per IP. If you run the testkit more than once in the same hour from the same IP, those two checks will return 429. This is expected behaviour — the rate limiter is working correctly. Run from a fresh IP (or wait an hour) for a clean 20/20.

---

What the testkit covers

Mode 1 — Per-Job (tests 1–10)

#	Name	What it checks
1	Health check	GET /api/healthz → HTTP 200, status=ok
2	Create job	POST /api/jobs → HTTP 201, jobId + evalInvoice.amountSats=10
3	Poll before payment	GET /api/jobs/:id → state=awaiting_eval_payment, invoice echoed, paymentHash present in stub mode
4	Pay eval invoice	POST /api/dev/stub/pay/:hash → {"ok":true}
5	Eval state advance	Polls until state=awaiting_work_payment OR state=rejected (30s timeout)
6	Pay work + get result	Pays work invoice, polls until state=complete, result non-empty (30s timeout)
7	Demo endpoint	GET /api/demo?request=... → HTTP 200, coherent result
8a	Missing body	POST /api/jobs {} → HTTP 400
8b	Unknown job ID	GET /api/jobs/does-not-exist → HTTP 404
8c	Demo missing param	GET /api/demo → HTTP 400
8d	501-char request	POST /api/jobs with 501 chars → HTTP 400 mentioning "500 characters"
9	Rate limiter	6× GET /api/demo → at least one HTTP 429
10	Rejection path	Adversarial request goes through eval, polls until state=rejected with a non-empty reason

Mode 2 — Session (tests 11–16)

#	Name	What it checks
11	Create session	POST /api/sessions {"amount_sats":200} → HTTP 201, sessionId, state=awaiting_payment, invoice.amountSats=200
12	Poll before payment	GET /api/sessions/:id → state=awaiting_payment before invoice is paid
13	Pay deposit + activate	Pays deposit via stub, polls GET → state=active, balanceSats=200, macaroon present
14	Submit request (accepted)	POST /api/sessions/:id/request with valid macaroon → state=complete OR state=rejected, debitedSats>0, balanceRemaining decremented
15	Request without macaroon	Same endpoint, no Authorization header → HTTP 401
16	Topup invoice creation	POST /api/sessions/:id/topup {"amount_sats":500} with macaroon → HTTP 200, topup.paymentRequest present, topup.amountSats=500

---

Architecture notes for reviewers

Mode 1 mechanics

• Stub mode is active (no real Lightning node). paymentHash is exposed on GET responses so the testkit can drive the full payment flow automatically. In production (real LNbits), paymentHash is hidden.
• POST /api/dev/stub/pay/:hash is only mounted when NODE_ENV !== 'production'.
• State machine advances server-side on every GET poll — no webhooks.
• AI models: Haiku for eval (cheap gating), Sonnet for work (full output).
• Pricing: eval = 10 sats fixed. Work invoice = actual token usage (input + output) × Anthropic per-token rate × 1.4 margin, converted at live BTC/USD. This is dynamic — a 53-char request typically produces an invoice of ~180 sats, not a fixed tier. The old 50/100/250 sat fixed tiers were replaced by this model.
• Max request length: 500 chars. Rate limiter: 5 req/hr/IP on /api/demo (in-memory, resets on server restart).

Mode 2 mechanics

• Minimum deposit: 100 sats. Maximum: 10,000 sats. Minimum working balance: 50 sats.
• Session expiry: 24 hours of inactivity. Balance is forfeited on expiry. Expiry is stated in the expiresAt field of every session response.
• Auth: Authorization: Bearer <macaroon> header. Macaroon is issued on first activation (GET /sessions/:id after deposit is paid).
• Cost per request: (eval tokens + work tokens) × model rate × 1.4 margin → converted to sats. If a request starts with enough balance but actual cost pushes balance negative, the request still completes and delivers — only the next request is blocked.
• If balance drops below 50 sats, session transitions to paused. Top up via POST /sessions/:id/topup. Session resumes automatically on the next GET poll once the topup invoice is paid.
• The same POST /api/dev/stub/pay/:hash endpoint works for all invoice types (eval, work, session deposit, topup).

Eval + work latency (important for manual testers)

The eval call uses the real Anthropic API (Haiku), typically 2–5 seconds. The testkit uses polling loops (max 30s). Manual testers should poll with similar patience. The work call (Sonnet) typically runs 3–8 seconds.

---

Test results log

Date	Tester	Score	Notes
2026-03-18	Perplexity Computer	20/20 PASS	Issue #22
2026-03-18	Hermes (Claude Opus 4)	19/20 (pre-fix)	Issue #23; 1 failure = test ordering bug (8c hit rate limiter before param check). Fixed in testkit v4.
2026-03-19	Replit Agent (post-fix)	20/20 PASS	Verified on fresh server after testkit v4 — all fixes confirmed
2026-03-18	Claude Opus 4.6	14/20 (pre-fix)	Issue #25; 2 failures = same rate-limit ordering as Hermes. Fixed in testkit v4.

---

Report template

Tester: [Claude / Perplexity / Kimi / Hermes / Human / Other] Date: Base URL tested: Method: [Automated (curl … | bash) / Manual]

Mode 1 — Per-Job

Test	Pass / Fail / Skip	Latency	Notes
1 — Health check		—
2 — Create job		—
3 — Poll before payment		—
4 — Pay eval invoice		—
5 — Eval state advance		___s
6 — Pay work + result		___s
7 — Demo endpoint		___s
8a — Missing body		—
8b — Unknown job ID		—
8c — Demo missing param		—
8d — 501-char request		—
9 — Rate limiter		—
10 — Rejection path		___s

Mode 2 — Session

Test	Pass / Fail / Skip	Notes
11 — Create session
12 — Poll before payment
13 — Pay + activate
14 — Submit request
15 — Reject no macaroon
16 — Topup invoice

Overall verdict: Pass / Partial / Fail

Total: PASS=___ FAIL=___ SKIP=___

Issues found:

Observations on result quality:

Suggestions: