# Timmy API Testkit Report — Claude (Opus 4.6)

**Tester:** Claude (Opus 4.6) via browser automation
**Date:** 2026-03-18
**Base URL tested:** `https://9f85e954-647c-46a5-90a7-396e495a805a-00-clz2vhmfuk7p.spock.replit.dev`
**Method:** Manual (browser automation)
**Testkit version:** v3 (pre-fix — tested before PR #24 was merged)

---

## Mode 1 — Per-Job

| Test | Pass / Fail | Latency | Notes |
|------|------------|---------|-------|
| 1 — Health check | **PASS** | — | HTTP 200, status=ok, uptime 776s, 49 jobs total |
| 2 — Create job | **PASS** | — | HTTP 201, jobId returned, evalInvoice.amountSats=10 |
| 3a — Poll (state) | **PASS** | — | state=awaiting_eval_payment |
| 3b — Poll (paymentHash) | **PASS** | — | paymentHash present, stub mode active |
| 4 — Pay eval invoice | **PASS** | — | ok: true |
| 5 — Eval state advance | **PASS** | 3s | state=awaiting_work_payment, workInvoice.amountSats=182 |
| 6 — Pay work + result | **PASS** | 5s | state=complete, coherent LN explanation |
| 7 — Demo endpoint | **FAIL** | <1s | HTTP 429 — rate-limit exhausted by prior runs |
| 8a — Missing body | **PASS** | — | HTTP 400 |
| 8b — Unknown job ID | **PASS** | — | HTTP 404 |
| 8c — Demo missing param | **FAIL** | — | HTTP 429 instead of 400 — rate limiter fires before param validation |
| 8d — 501-char request | **PASS** | — | HTTP 400, mentions 500 chars |
| 9 — Rate limiter | **PASS** | — | All 6 requests returned 429 |
| 10 — Rejection path | **PASS** | 2s | state=rejected, reason cites ethical/legal guidelines |

## Mode 2 — Session

| Test | Pass / Fail | Notes |
|------|------------|-------|
| 11 — Create session | **PASS** | HTTP 201, state=awaiting_payment, amountSats=200 |
| 12 — Poll before payment | **PASS** | state=awaiting_payment |
| 13 — Pay + activate | **PASS** | state=active, balanceSats=200, macaroon present |
| 14 — Submit request | **PASS** | state=complete, debitedSats=179, balanceRemaining=21, latency 2s |
| 15 — Reject no macaroon | **PASS** | HTTP 401 |
| 16 — Topup invoice | **PASS** | paymentRequest present, amountSats=500 |

**Overall verdict:** PASS (failures are test-ordering artifacts, not API bugs)

**Total:** PASS=14 FAIL=2 SKIP=0

---

## Issues Found

1. **Tests 7 and 8c** — both fail due to rate-limit exhaustion from prior runs. Fixed in testkit v4: test 8c moved before tests 7 and 9; `head -n-1` replaced with `sed '$d'` for macOS compat.

2. **"Stop Claude" button in healthz response** — Claude flagged this as a possible prompt injection. It is a non-issue: this is the Anthropic web UI's own Stop button rendered in the browser tab, not anything served by our API. `GET /api/healthz` returns plain JSON.

---

## Observations on Result Quality

- Test 6 (LN explanation): accurate, correctly describes Layer 2 and payment channels.
- Test 10 (adversarial): correctly rejected with clear ethical/legal reasoning.
- Test 14 (session request): 179 sat debit for a short request — token-based pricing working as documented.
