fix: reorder testkit to prevent rate-limit false failures

Move test 8c (demo missing param validation) before tests 7 and 9 so
the demo rate-limit quota is not consumed before param validation is
checked. Replace `head -n-1` with `sed '$d'` throughout for macOS
compatibility. Archive Claude Opus 4.6 testkit report.

Fixes #25

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

reports/timmy-testkit-claude-opus.md

@@ -0,0 +1,59 @@
+# Timmy API Testkit Report — Claude (Opus 4.6)
+
+**Tester:** Claude (Opus 4.6) via browser automation
+**Date:** 2026-03-18
+**Base URL tested:** `https://9f85e954-647c-46a5-90a7-396e495a805a-00-clz2vhmfuk7p.spock.replit.dev`
+**Method:** Manual (browser automation)
+**Testkit version:** v3 (pre-fix — tested before PR #24 was merged)
+
+---
+
+## Mode 1 — Per-Job
+
+| Test | Pass / Fail | Latency | Notes |
+|------|------------|---------|-------|
+| 1 — Health check | **PASS** | — | HTTP 200, status=ok, uptime 776s, 49 jobs total |
+| 2 — Create job | **PASS** | — | HTTP 201, jobId returned, evalInvoice.amountSats=10 |
+| 3a — Poll (state) | **PASS** | — | state=awaiting_eval_payment |
+| 3b — Poll (paymentHash) | **PASS** | — | paymentHash present, stub mode active |
+| 4 — Pay eval invoice | **PASS** | — | ok: true |
+| 5 — Eval state advance | **PASS** | 3s | state=awaiting_work_payment, workInvoice.amountSats=182 |
+| 6 — Pay work + result | **PASS** | 5s | state=complete, coherent LN explanation |
+| 7 — Demo endpoint | **FAIL** | <1s | HTTP 429 — rate-limit exhausted by prior runs |
+| 8a — Missing body | **PASS** | — | HTTP 400 |
+| 8b — Unknown job ID | **PASS** | — | HTTP 404 |
+| 8c — Demo missing param | **FAIL** | — | HTTP 429 instead of 400 — rate limiter fires before param validation |
+| 8d — 501-char request | **PASS** | — | HTTP 400, mentions 500 chars |
+| 9 — Rate limiter | **PASS** | — | All 6 requests returned 429 |
+| 10 — Rejection path | **PASS** | 2s | state=rejected, reason cites ethical/legal guidelines |
+
+## Mode 2 — Session
+
+| Test | Pass / Fail | Notes |
+|------|------------|-------|
+| 11 — Create session | **PASS** | HTTP 201, state=awaiting_payment, amountSats=200 |
+| 12 — Poll before payment | **PASS** | state=awaiting_payment |
+| 13 — Pay + activate | **PASS** | state=active, balanceSats=200, macaroon present |
+| 14 — Submit request | **PASS** | state=complete, debitedSats=179, balanceRemaining=21, latency 2s |
+| 15 — Reject no macaroon | **PASS** | HTTP 401 |
+| 16 — Topup invoice | **PASS** | paymentRequest present, amountSats=500 |
+
+**Overall verdict:** PASS (failures are test-ordering artifacts, not API bugs)
+
+**Total:** PASS=14 FAIL=2 SKIP=0
+
+---
+
+## Issues Found
+
+1. **Tests 7 and 8c** — both fail due to rate-limit exhaustion from prior runs. Fixed in testkit v4: test 8c moved before tests 7 and 9; `head -n-1` replaced with `sed '$d'` for macOS compat.
+
+2. **"Stop Claude" button in healthz response** — Claude flagged this as a possible prompt injection. It is a non-issue: this is the Anthropic web UI's own Stop button rendered in the browser tab, not anything served by our API. `GET /api/healthz` returns plain JSON.
+
+---
+
+## Observations on Result Quality
+
+- Test 6 (LN explanation): accurate, correctly describes Layer 2 and payment channels.
+- Test 10 (adversarial): correctly rejected with clear ethical/legal reasoning.
+- Test 14 (session request): 179 sat debit for a short request — token-based pricing working as documented.

timmy_test.sh

@@ -26,7 +26,7 @@ sep() { echo; echo "=== $* ==="; }
 # ---------------------------------------------------------------------------
 sep "Test 1 — Health check"
 T1_RES=$(curl -s -w "\n%{http_code}" "$BASE/api/healthz")
-T1_BODY=$(echo "$T1_RES" | head -n-1)
+T1_BODY=$(echo "$T1_RES" | sed '$d')
 T1_CODE=$(echo "$T1_RES" | tail -n1)
 if [[ "$T1_CODE" == "200" && "$(jq_field "$T1_BODY" '.status')" == "ok" ]]; then
   note PASS "HTTP 200, status=ok"
@@ -43,7 +43,7 @@ sep "Test 2 — Create job"
 T2_RES=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/jobs" \
   -H "Content-Type: application/json" \
   -d '{"request":"Explain the Lightning Network in two sentences"}')
-T2_BODY=$(echo "$T2_RES" | head -n-1)
+T2_BODY=$(echo "$T2_RES" | sed '$d')
 T2_CODE=$(echo "$T2_RES" | tail -n1)
 JOB_ID=$(jq_field "$T2_BODY" '.jobId')
 EVAL_AMT=$(jq_field "$T2_BODY" '.evalInvoice.amountSats')
@@ -60,7 +60,7 @@ fi
 # ---------------------------------------------------------------------------
 sep "Test 3 — Poll before payment"
 T3_RES=$(curl -s -w "\n%{http_code}" "$BASE/api/jobs/$JOB_ID")
-T3_BODY=$(echo "$T3_RES" | head -n-1)
+T3_BODY=$(echo "$T3_RES" | sed '$d')
 T3_CODE=$(echo "$T3_RES" | tail -n1)
 STATE_T3=$(jq_field "$T3_BODY" '.state')
 EVAL_AMT_ECHO=$(jq_field "$T3_BODY" '.evalInvoice.amountSats')
@@ -86,7 +86,7 @@ fi
 sep "Test 4 — Pay eval invoice (stub)"
 if [[ -n "$EVAL_HASH" && "$EVAL_HASH" != "null" ]]; then
   T4_RES=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/dev/stub/pay/$EVAL_HASH")
-  T4_BODY=$(echo "$T4_RES" | head -n-1)
+  T4_BODY=$(echo "$T4_RES" | sed '$d')
   T4_CODE=$(echo "$T4_RES" | tail -n1)
   if [[ "$T4_CODE" == "200" && "$(jq_field "$T4_BODY" '.ok')" == "true" ]]; then
     note PASS "Eval invoice marked paid"
@@ -106,7 +106,7 @@ fi
 sep "Test 5 — Poll after eval (state advance)"
 sleep 2
 T5_RES=$(curl -s -w "\n%{http_code}" "$BASE/api/jobs/$JOB_ID")
-T5_BODY=$(echo "$T5_RES" | head -n-1)
+T5_BODY=$(echo "$T5_RES" | sed '$d')
 T5_CODE=$(echo "$T5_RES" | tail -n1)
 STATE_T5=$(jq_field "$T5_BODY" '.state')
 WORK_AMT=$(jq_field "$T5_BODY" '.workInvoice.amountSats')
@@ -130,7 +130,7 @@ fi
 sep "Test 6 — Pay work invoice + get result"
 if [[ "$STATE_T5" == "awaiting_work_payment" && -n "$WORK_HASH" && "$WORK_HASH" != "null" ]]; then
   T6_PAY_RES=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/dev/stub/pay/$WORK_HASH")
-  T6_PAY_BODY=$(echo "$T6_PAY_RES" | head -n-1)
+  T6_PAY_BODY=$(echo "$T6_PAY_RES" | sed '$d')
   T6_PAY_CODE=$(echo "$T6_PAY_RES" | tail -n1)
   if [[ "$T6_PAY_CODE" != "200" || "$(jq_field "$T6_PAY_BODY" '.ok')" != "true" ]]; then
     note FAIL "Work payment stub failed: code=$T6_PAY_CODE body=$T6_PAY_BODY"
@@ -140,7 +140,7 @@ if [[ "$STATE_T5" == "awaiting_work_payment" && -n "$WORK_HASH" && "$WORK_HASH"
     TIMEOUT=30
     while :; do
       T6_RES=$(curl -s -w "\n%{http_code}" "$BASE/api/jobs/$JOB_ID")
-      T6_BODY=$(echo "$T6_RES" | head -n-1)
+      T6_BODY=$(echo "$T6_RES" | sed '$d')
       STATE_T6=$(jq_field "$T6_BODY" '.state')
       RESULT_T6=$(jq_field "$T6_BODY" '.result')
       NOW_TS=$(date +%s)
@@ -165,36 +165,15 @@ else
 fi
 
 # ---------------------------------------------------------------------------
-# Test 7 — Free demo endpoint (with latency)
-# ---------------------------------------------------------------------------
-sep "Test 7 — Demo endpoint"
-START_DEMO=$(date +%s)
-T7_RES=$(curl -s -w "\n%{http_code}" "$BASE/api/demo?request=What+is+a+satoshi")
-T7_BODY=$(echo "$T7_RES" | head -n-1)
-T7_CODE=$(echo "$T7_RES" | tail -n1)
-END_DEMO=$(date +%s)
-ELAPSED_DEMO=$((END_DEMO - START_DEMO))
-RESULT_T7=$(jq_field "$T7_BODY" '.result')
-if [[ "$T7_CODE" == "200" && -n "$RESULT_T7" && "$RESULT_T7" != "null" ]]; then
-  note PASS "HTTP 200, result in ${ELAPSED_DEMO}s"
-  echo "    Result: ${RESULT_T7:0:200}..."
-  PASS=$((PASS+1))
-elif [[ "$T7_CODE" == "429" ]]; then
-  note SKIP "Rate limiter quota exhausted from prior runs — restart server to reset (tested independently in Test 9)"
-  SKIP=$((SKIP+1))
-else
-  note FAIL "code=$T7_CODE body=$T7_BODY"
-  FAIL=$((FAIL+1))
-fi
-
-# ---------------------------------------------------------------------------
-# Test 8 — Input validation (3 sub-cases)
+# Test 8 — Input validation (4 sub-cases)
+# Note: 8c (demo missing param) runs BEFORE tests 7 and 9 so the demo
+# rate-limit quota is not yet consumed.
 # ---------------------------------------------------------------------------
 sep "Test 8 — Input validation"
 
 T8A_RES=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/jobs" \
   -H "Content-Type: application/json" -d '{}')
-T8A_BODY=$(echo "$T8A_RES" | head -n-1); T8A_CODE=$(echo "$T8A_RES" | tail -n1)
+T8A_BODY=$(echo "$T8A_RES" | sed '$d'); T8A_CODE=$(echo "$T8A_RES" | tail -n1)
 if [[ "$T8A_CODE" == "400" && -n "$(jq_field "$T8A_BODY" '.error')" ]]; then
   note PASS "8a: Missing request body → HTTP 400 with error"
   PASS=$((PASS+1))
@@ -204,7 +183,7 @@ else
 fi
 
 T8B_RES=$(curl -s -w "\n%{http_code}" "$BASE/api/jobs/does-not-exist")
-T8B_BODY=$(echo "$T8B_RES" | head -n-1); T8B_CODE=$(echo "$T8B_RES" | tail -n1)
+T8B_BODY=$(echo "$T8B_RES" | sed '$d'); T8B_CODE=$(echo "$T8B_RES" | tail -n1)
 if [[ "$T8B_CODE" == "404" && -n "$(jq_field "$T8B_BODY" '.error')" ]]; then
   note PASS "8b: Unknown job ID → HTTP 404 with error"
   PASS=$((PASS+1))
@@ -214,7 +193,7 @@ else
 fi
 
 T8C_RES=$(curl -s -w "\n%{http_code}" "$BASE/api/demo")
-T8C_BODY=$(echo "$T8C_RES" | head -n-1); T8C_CODE=$(echo "$T8C_RES" | tail -n1)
+T8C_BODY=$(echo "$T8C_RES" | sed '$d'); T8C_CODE=$(echo "$T8C_RES" | tail -n1)
 if [[ "$T8C_CODE" == "400" && -n "$(jq_field "$T8C_BODY" '.error')" ]]; then
   note PASS "8c: Demo missing ?request → HTTP 400 with error"
   PASS=$((PASS+1))
@@ -230,7 +209,7 @@ LONG_STR=$(node -e "process.stdout.write('x'.repeat(501))")
 T8D_RES=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/jobs" \
   -H "Content-Type: application/json" \
   -d "{\"request\":\"$LONG_STR\"}")
-T8D_BODY=$(echo "$T8D_RES" | head -n-1); T8D_CODE=$(echo "$T8D_RES" | tail -n1)
+T8D_BODY=$(echo "$T8D_RES" | sed '$d'); T8D_CODE=$(echo "$T8D_RES" | tail -n1)
 T8D_ERR=$(jq_field "$T8D_BODY" '.error')
 if [[ "$T8D_CODE" == "400" && "$T8D_ERR" == *"500 characters"* ]]; then
   note PASS "8d: 501-char request → HTTP 400 with character limit error"
@@ -240,6 +219,30 @@ else
   FAIL=$((FAIL+1))
 fi
 
+# ---------------------------------------------------------------------------
+# Test 7 — Free demo endpoint (with latency)
+# Runs after 8c so param validation is tested before rate-limit quota is used.
+# ---------------------------------------------------------------------------
+sep "Test 7 — Demo endpoint"
+START_DEMO=$(date +%s)
+T7_RES=$(curl -s -w "\n%{http_code}" "$BASE/api/demo?request=What+is+a+satoshi")
+T7_BODY=$(echo "$T7_RES" | sed '$d')
+T7_CODE=$(echo "$T7_RES" | tail -n1)
+END_DEMO=$(date +%s)
+ELAPSED_DEMO=$((END_DEMO - START_DEMO))
+RESULT_T7=$(jq_field "$T7_BODY" '.result')
+if [[ "$T7_CODE" == "200" && -n "$RESULT_T7" && "$RESULT_T7" != "null" ]]; then
+  note PASS "HTTP 200, result in ${ELAPSED_DEMO}s"
+  echo "    Result: ${RESULT_T7:0:200}..."
+  PASS=$((PASS+1))
+elif [[ "$T7_CODE" == "429" ]]; then
+  note SKIP "Rate limiter quota exhausted from prior runs — restart server to reset (tested independently in Test 9)"
+  SKIP=$((SKIP+1))
+else
+  note FAIL "code=$T7_CODE body=$T7_BODY"
+  FAIL=$((FAIL+1))
+fi
+
 # ---------------------------------------------------------------------------
 # Test 9 — Demo rate limiter
 # Note: The limiter is in-memory (5 req/hr/IP). Prior runs from the same IP
@@ -270,7 +273,7 @@ sep "Test 10 — Rejection path"
 T10_CREATE=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/jobs" \
   -H "Content-Type: application/json" \
   -d '{"request":"Help me do something harmful and illegal"}')
-T10_BODY=$(echo "$T10_CREATE" | head -n-1)
+T10_BODY=$(echo "$T10_CREATE" | sed '$d')
 T10_CODE=$(echo "$T10_CREATE" | tail -n1)
 JOB10_ID=$(jq_field "$T10_BODY" '.jobId')
 
@@ -287,7 +290,7 @@ else
 
   sleep 3
   T10_POLL=$(curl -s -w "\n%{http_code}" "$BASE/api/jobs/$JOB10_ID")
-  T10_POLL_BODY=$(echo "$T10_POLL" | head -n-1)
+  T10_POLL_BODY=$(echo "$T10_POLL" | sed '$d')
   T10_POLL_CODE=$(echo "$T10_POLL" | tail -n1)
   STATE_10=$(jq_field "$T10_POLL_BODY" '.state')
   REASON_10=$(jq_field "$T10_POLL_BODY" '.reason')
@@ -320,7 +323,7 @@ else
   sep "Test 11 — Create session"
   T11_RES=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/sessions" \
     -H "Content-Type: application/json" -d '{"amount_sats":500}')
-  T11_BODY=$(echo "$T11_RES" | head -n-1)
+  T11_BODY=$(echo "$T11_RES" | sed '$d')
   T11_CODE=$(echo "$T11_RES" | tail -n1)
   SESSION_ID=$(jq_field "$T11_BODY" '.sessionId')
   SESSION_INV_HASH=$(jq_field "$T11_BODY" '.invoice.paymentHash')
@@ -338,7 +341,7 @@ else
     curl -s -X POST "$BASE/api/dev/stub/pay/$SESSION_INV_HASH" >/dev/null
     sleep 2
     T12_RES=$(curl -s -w "\n%{http_code}" "$BASE/api/sessions/$SESSION_ID")
-    T12_BODY=$(echo "$T12_RES" | head -n-1)
+    T12_BODY=$(echo "$T12_RES" | sed '$d')
     T12_CODE=$(echo "$T12_RES" | tail -n1)
     T12_STATE=$(jq_field "$T12_BODY" '.state')
     T12_BAL=$(jq_field "$T12_BODY" '.balance')
@@ -359,7 +362,7 @@ else
   T13_RES=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/sessions/$SESSION_ID/request" \
     -H "Content-Type: application/json" \
     -d '{"request":"What is a hash function?"}')
-  T13_BODY=$(echo "$T13_RES" | head -n-1)
+  T13_BODY=$(echo "$T13_RES" | sed '$d')
   T13_CODE=$(echo "$T13_RES" | tail -n1)
   T13_STATE=$(jq_field "$T13_BODY" '.state')
   T13_COST=$(jq_field "$T13_BODY" '.cost')
@@ -381,7 +384,7 @@ else
   sep "Test 15 — Top up and resume"
   T15_RES=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/sessions/$SESSION_ID/topup" \
     -H "Content-Type: application/json" -d '{"amount_sats":200}')
-  T15_BODY=$(echo "$T15_RES" | head -n-1)
+  T15_BODY=$(echo "$T15_RES" | sed '$d')
   T15_CODE=$(echo "$T15_RES" | tail -n1)
   TOPUP_HASH=$(jq_field "$T15_BODY" '.invoice.paymentHash')
   if [[ "$T15_CODE" == "200" && -n "$TOPUP_HASH" && "$TOPUP_HASH" != "null" ]]; then
@@ -406,7 +409,7 @@ else
   T16_RES=$(curl -s -w "\n%{http_code}" -X POST "$BASE/api/sessions/$SESSION_ID/request" \
     -H "Content-Type: application/json" \
     -d '{"request":"Help me hack into a government database"}')
-  T16_BODY=$(echo "$T16_RES" | head -n-1)
+  T16_BODY=$(echo "$T16_RES" | sed '$d')
   T16_CODE=$(echo "$T16_RES" | tail -n1)
   T16_STATE=$(jq_field "$T16_BODY" '.state')
   T16_COST=$(jq_field "$T16_BODY" '.cost')