timmy-talking-turd/docs/MODEL-METRICS.md

Privacy-safe model health counters

Issue #19. This is the operator surface for "is the worker healthy?", and it is deliberately the smallest one that answers the question. The journal stays the only place a person's bowel data lives.

Endpoint

GET /api/metrics
Authorization: Bearer $TIMMY_METRICS_TOKEN

• TIMMY_METRICS_TOKEN unset → 404. A default deployment has no metrics route at all; nothing is exposed by forgetting to configure something.
• Token set, header missing or wrong → 403, with a body that contains no counters.
• Token set and matched → 200 with the snapshot below.

The snapshot is read from process memory. It survives no restart, is written to no log, and is sent to no third party.

What is in the snapshot

{
  "schema": 1,
  "generatedAt": "2026-11-01T10:00:00.000Z",
  "app":   { "uptimeMs": 0, "analyzeRequests": 0, "refusedBeforeModel": 0 },
  "queue": { "inFlight": 0, "limit": 4, "atCapacity": false, "accepted": 0 },
  "model": {
    "outcomes": { "suggestion": 0, "abstained": 0, "failure": 0 },
    "failureClasses": { "ingress_refused": 0, "rate_limited": 0, "disabled": 0,
                        "provider_unreachable": 0, "provider_rejected": 0,
                        "schema_invalid": 0, "timeout": 0, "unexpected": 0 },
    "abstentionRate": null,
    "readiness": { "phase": "checking", "ready": false, "coldLoadMs": null },
    "latency": { "count": 0, "meanMs": null, "maxMs": 0,
                 "bucketsMs": { "1000": 0, "2000": 0, "5000": 0, "15000": 0, "60000": 0 } }
  }
}

The three blocks are the three boundaries, and each counts only what it owns:

Block	Owns	Does not own
app	Every POST /api/analyze the process saw, and the ones refused before any model work (bad bytes, per-device budget, analysis disabled).	Queue occupancy or model answers.
queue	Slots handed out and released, and the configured concurrency ceiling.	Whether the request succeeded.
model	Answers (suggestion / abstained), classified failures, readiness phase, and model-work latency.	Anything decided before a slot was taken.

abstentionRate is abstained / (suggestion + abstained); it is null until the model has answered at least once. A failed request is not an abstention — an outage must not look like a cautious model.

latency.bucketsMs is cumulative: "5000" counts every sample at or below 5 s. coldLoadMs is the interval from process start to the first observation in which the pinned model was resident; a worker that flaps back to offline keeps the original figure.

readiness.phase uses the same vocabulary the browser uses for the AI affordance (checking, disabled, offline, loading, ready), and the value is produced by the same visionReadiness() function the app renders, so a metric and a screen cannot disagree.

What can never appear in it

The only strings this module stores are members of three frozen lists — outcomes, failure classes, and readiness phases — plus a fixed latency-bucket edge set. Anything else is bucketed, not copied:

• an unrecognised outcome becomes failure;
• an unrecognised failure class becomes unexpected;
• an unrecognised readiness phase becomes checking.

Therefore no note text, no image bytes, no image hash or fingerprint, no subject key, no caller address, and no session or cookie value can reach a label. tests/model-metrics.test.js asserts this by feeding hostile strings into every label position, and tests/metrics-endpoint.acceptance.test.js asserts it through the shipping server.

Reading it

curl -sS -H "Authorization: Bearer $TIMMY_METRICS_TOKEN" http://127.0.0.1:4173/api/metrics | jq .

For a quick look that does not need jq, the report renders the same snapshot as three labelled blocks:

TIMMY_METRICS_TOKEN=... node scripts/model_metrics_report.mjs --url http://127.0.0.1:4173

Exit codes are the point of the command as much as the numbers: 0 means a snapshot was rendered, 1 means the server answered non-200 (404 says the route is disabled because no token is configured, 403 says the supplied token was rejected), 2 means the command itself was called wrongly or the server could not be reached. The token is read from the environment or --token and is never echoed in the report or in an error line.

What the numbers mean to an operator:

Signal	Reading
readiness.phase: offline	The worker is not answering. Manual logging is unaffected; do not restart the app for this.
readiness.phase: loading	The worker answered but the pinned model is not resident. AI is withheld rather than offered and failed.
queue.atCapacity: true	Every model slot is in use. Watch latency.maxMs; a growing figure with no completions is a stuck inference, not load.
failureClasses.provider_unreachable climbing	Network or worker outage, not caller error.
failureClasses.schema_invalid climbing	The worker is answering off-contract. Treat as a regression in the pinned model or prompt, not a user problem.
app.refusedBeforeModel climbing with no model traffic	Abuse or a broken client; the model never saw it.

Known gap

queue.inFlight is the app's own admission depth for POST /api/analyze. The full bounded queue (concurrency limit enforcement, cancellation, overload fallback) is issue #17; this snapshot reports the slot accounting that exists today and does not claim the rest.