token-gated-economy/attached_assets/timmy_api_test_requirements_1773854936781.md

Timmy API Test Requirements Analysis

1. Test Case Summary

Test ID	Test Name	Purpose
T-01	Health Check	Verify API is running and responsive
T-02	Create Job	Submit a new job request and receive eval invoice
T-03	Poll Before Payment	Verify job state before eval payment is made
T-04	Pay Eval Invoice	Simulate payment of evaluation fee (stub mode)
T-05	Poll After Eval Payment	Verify state machine advances after eval payment
T-06	Pay Work Invoice & Get Result	Complete full workflow and receive AI result
T-07	Free Demo Endpoint	Test free demo endpoint without payment
T-08	Input Validation	Verify proper error handling for invalid inputs
T-09	Demo Rate Limiter	Test rate limiting on demo endpoint (5 req/IP)
T-10	Rejection Path	Verify agent rejects harmful/illegal requests

---

2. Pass/Fail Criteria by Test Case

T-01: Health Check

Pass Criteria:

• HTTP Status: 200 OK
• Response body contains status field
• Response value: {"status":"ok"}

Fail Criteria:

• HTTP status != 200
• Missing status field
• Status value != "ok"

---

T-02: Create Job

Pass Criteria:

• HTTP Status: 201 Created
• Response contains jobId (UUID format)
• Response contains evalInvoice object
• evalInvoice.amountSats equals exactly 10
• evalInvoice.paymentRequest is present (stub format: lnbcrt10u1stub_...)

Fail Criteria:

• HTTP status != 201
• Missing jobId or invalid format
• evalInvoice missing or malformed
• amountSats != 10

---

T-03: Poll Before Payment

Pass Criteria:

• HTTP Status: 200
• state field equals exactly awaiting_eval_payment
• evalInvoice is echoed back (same as T-02 response)
• jobId matches the created job

Fail Criteria:

• State != awaiting_eval_payment
• Invoice data missing or modified
• Wrong jobId returned

---

T-04: Pay Eval Invoice (Stub Mode)

Pass Criteria:

• HTTP Status: 200 OK
• Response contains {"ok":true}
• Response contains paymentHash matching the paid invoice

Fail Criteria:

• HTTP status != 200
• ok != true
• Payment hash mismatch

Note: Uses /api/dev/stub/pay/<full-64-char-payment-hash> endpoint (stub mode only)

---

T-05: Poll After Eval Payment (State Advance)

Pass Criteria:

• HTTP Status: 200
• State has advanced from awaiting_eval_payment
• Agent judgment is present in response

Accepted Path:

• State equals awaiting_work_payment
• workInvoice object present
• workInvoice.amountSats is one of: 50, 100, or 250
• workInvoice.paymentRequest present (stub format)

Rejected Path:

• State equals rejected
• reason field present with explanation

Fail Criteria:

• State unchanged (still awaiting_eval_payment)
• State is invalid/unknown
• Missing required fields for current state

---

T-06: Pay Work Invoice & Get Result

Pass Criteria:

• HTTP Status: 200
• Final state equals complete
• result field present with AI-generated content
• Result is meaningful and answers the original request

Fail Criteria:

• State != complete
• Missing or empty result
• Result is incoherent or unrelated to request

Timing Note: AI response may take 2-5 seconds after payment

---

T-07: Free Demo Endpoint

Pass Criteria:

• HTTP Status: 200 OK
• Response contains result field
• Result is coherent and answers the query

Fail Criteria:

• HTTP status != 200
• Missing result field
• Result is incoherent or nonsensical

---

T-08: Input Validation

Pass Criteria (all three scenarios):

Scenario	HTTP Status	Expected Response
Missing request body	400	{"error":"Invalid request: 'request' string is required"}
Unknown job ID	404	{"error":"Job not found"}
Demo without param	400	{"error":"Missing required query param: request"}

Fail Criteria:

• Incorrect HTTP status codes
• Error message format != {"error": string}
• Generic error messages without specifics

---

T-09: Demo Rate Limiter

Pass Criteria:

• Requests 1-5: Return HTTP 200 with result field
• Request 6+: Return HTTP 429 (Too Many Requests) with error field
• Rate limit applies per IP address

Fail Criteria:

• Rate limit triggers before request 6
• Rate limit does not trigger by request 6
• Rate limit affects different IPs

---

T-10: Rejection Path (Adversarial Request)

Pass Criteria:

• Job created successfully (T-02 passes)
• Eval payment processed (T-04 passes)
• Final state equals rejected
• reason field present explaining rejection
• State does NOT become awaiting_work_payment

Fail Criteria:

• State becomes awaiting_work_payment (agent failed to reject)
• State becomes complete with harmful content
• Missing reason field

---

3. Expected Results Summary

Normal Flow Results

Step	Endpoint	Method	Expected Response
Health	/api/healthz	GET	{"status":"ok"}
Create	/api/jobs	POST	Job object with eval invoice
Poll Pre-Pay	/api/jobs/{id}	GET	State: awaiting_eval_payment
Pay Eval	/api/dev/stub/pay/{hash}	POST	{"ok":true,"paymentHash":"..."}
Poll Post-Eval	/api/jobs/{id}	GET	State: awaiting_work_payment OR rejected
Pay Work	/api/dev/stub/pay/{hash}	POST	{"ok":true,"paymentHash":"..."}
Get Result	/api/jobs/{id}	GET	State: complete, result present
Demo	/api/demo?request={q}	GET	{"result":"..."}

Error Response Results

Error Type	HTTP Status	Response Body
Missing request field	400	{"error":"Invalid request: 'request' string is required"}
Job not found	404	{"error":"Job not found"}
Missing query param	400	{"error":"Missing required query param: request"}
Rate limit exceeded	429	{"error":...}

---

4. Error Scenarios

4.1 Input Validation Errors

Scenario	Trigger	HTTP Code	Response Pattern
Empty request body	POST /api/jobs with {}	400	{error: "Invalid request: 'request' string is required"}
Missing request field	POST /api/jobs without request	400	Same as above
Invalid job ID format	GET /api/jobs/invalid-id	404	{error: "Job not found"}
Non-existent job UUID	GET /api/jobs/{random-uuid}	404	{error: "Job not found"}
Missing demo param	GET /api/demo (no query)	400	{error: "Missing required query param: request"}

4.2 Rate Limiting Errors

Scenario	Trigger	HTTP Code	Response
Demo rate limit exceeded	6+ requests from same IP to /api/demo	429	{error: ...}
Rate limit window	Per IP address	429	Triggered on 6th request

4.3 Business Logic Errors (Rejection)

Scenario	Trigger	State	Response
Harmful request rejected	Request containing harmful/illegal content	rejected	{jobId, state: "rejected", reason: "..."}

4.4 Error Response Format Standard

All errors MUST follow this format:

{
  "error": "string description of the error"
}

---

5. State Transitions (Job State Machine)

5.1 State Diagram

                    [Create Job]
                         |
                         v
            +------------------------+
            |  awaiting_eval_payment |<-------+
            +------------------------+        |
                         |                    |
              [Pay eval invoice]               |
                         |                    |
            +------------+------------+       |
            |                         |       |
            v                         v       |
+---------------------+   +-------------------+---+
| awaiting_work_payment|   |      rejected       |
+---------------------+   +-------------------+---+
            |                                    |
   [Pay work invoice]                            |
            |                                    |
            v                                    |
+---------------------+                          |
|      complete       |                          |
+---------------------+--------------------------+

5.2 State Definitions

State	Description	Entry Trigger	Exit Trigger
awaiting_eval_payment	Job created, waiting for eval fee payment	Job creation (POST /api/jobs)	Eval invoice paid
awaiting_work_payment	Eval paid, agent accepted, waiting for work fee	Eval payment detected on poll	Work invoice paid
rejected	Eval paid, agent rejected request	Eval payment detected + agent rejection	Terminal state
complete	Work paid, AI result delivered	Work payment detected + AI response	Terminal state

5.3 State Transition Rules

1. Initial State: All jobs start at awaiting_eval_payment
2. Auto-Advance: States advance automatically on GET poll (no webhook)
3. Terminal States: rejected and complete are final (no further transitions)
4. Branching: After eval payment, state branches to either awaiting_work_payment (accepted) or rejected (declined)
5. Deterministic: Same request + payment = same state transitions

5.4 State-Dependent Response Fields

State	Required Fields	Optional Fields
awaiting_eval_payment	jobId, state, evalInvoice	-
awaiting_work_payment	jobId, state, workInvoice	-
rejected	jobId, state, reason	-
complete	jobId, state, result	-

---

6. Validation Rules

6.1 Input Validation Requirements

POST /api/jobs

Field	Type	Required	Validation Rules
request	string	YES	Non-empty string, max length TBD

Validation Errors:

• Missing field: 400 Bad Request
• Empty string: 400 Bad Request
• Wrong type: 400 Bad Request

GET /api/jobs/{jobId}

Parameter	Type	Required	Validation Rules
jobId	UUID string	YES (in path)	Valid UUID format, exists in system

Validation Errors:

• Invalid UUID format: 404 Not Found
• Non-existent UUID: 404 Not Found

GET /api/demo

Parameter	Type	Required	Validation Rules
request	string	YES (query)	Non-empty string

Validation Errors:

• Missing param: 400 Bad Request
• Empty value: 400 Bad Request

6.2 Invoice Validation (Stub Mode)

Field	Format	Example
paymentRequest (eval)	lnbcrt10u1stub_<16-char-hash>	lnbcrt10u1stub_a1b2c3d4e5f6g7h8
paymentRequest (work)	lnbcrt<sats>u1stub_<16-char-hash>	lnbcrt50u1stub_... or lnbcrt100u1stub_... or lnbcrt250u1stub_...
paymentHash (full)	64 hexadecimal characters	Required for stub pay endpoint

6.3 Pricing Validation Rules

Fee Type	Amount (sats)	Determination
Eval fee	10	Fixed for all requests
Work fee - Short	50	Request length < threshold
Work fee - Medium	100	Request length in middle range
Work fee - Long	250	Request length > threshold

6.4 Rate Limiting Rules

Endpoint	Limit	Window	Scope
/api/demo	5 requests	Per IP	IP-based

Behavior:

• Count starts at 1 for first request
• Requests 1-5: Allowed (200 OK)
• Request 6+: Blocked (429 Too Many Requests)

6.5 AI Model Configuration

Phase	Model	Purpose
Eval	claude-haiku-4-5	Fast/cheap judgment
Work	claude-sonnet-4-6	Full capability response

---

7. Test Execution Checklist

Prerequisites

• API base URL is accessible
• Stub mode is enabled (/api/dev/stub/pay available)
• Database access (optional, for payment hash lookup)

Execution Order

1. T-01: Verify API is up
2. T-02: Create test job, save jobId and evalInvoice
3. T-03: Poll to verify initial state
4. T-04: Pay eval invoice (extract hash from paymentRequest)
5. T-05: Poll to verify state advance
6. T-06: If accepted, pay work invoice and get result
7. T-07: Test demo endpoint
8. T-08: Test all validation scenarios
9. T-09: Test rate limiter (6 rapid requests)
10. T-10: Create adversarial job, verify rejection

Data Dependencies

• T-03, T-05, T-06 require jobId from T-02
• T-04 requires payment hash from T-02's evalInvoice
• T-06 requires payment hash from T-05's workInvoice (if accepted)
• T-10 is independent (creates its own job)

---

8. Summary Table: All Requirements

Category	Count	Details
Total Test Cases	10	T-01 through T-10
API Endpoints Tested	4	/healthz, /jobs, /dev/stub/pay, /demo
HTTP Methods	2	GET, POST
States	4	awaiting_eval_payment, awaiting_work_payment, rejected, complete
Error Scenarios	6	Missing body, invalid job, missing param, rate limit, rejection, not found
Validation Rules	12+	Input type, format, presence, rate limits
Pricing Tiers	4	10 sats (eval), 50/100/250 sats (work)

---

Analysis generated from Timmy API Test Plan & Report Prompt