timmy-talking-turd/docs/MODEL-METRICS.md

# Privacy-safe model health counters

Issue [#19](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/19).
This is the operator surface for "is the worker healthy?", and it is deliberately the smallest one
that answers the question. The journal stays the only place a person's bowel data lives.

## Endpoint

```
GET /api/metrics
Authorization: Bearer $TIMMY_METRICS_TOKEN
```

- `TIMMY_METRICS_TOKEN` unset → **404**. A default deployment has no metrics route at all; nothing is
  exposed by forgetting to configure something.
- Token set, header missing or wrong → **403**, with a body that contains no counters.
- Token set and matched → **200** with the snapshot below.

The snapshot is read from process memory. It survives no restart, is written to no log, and is sent
to no third party.

## What is in the snapshot

```json
{
  "schema": 1,
  "generatedAt": "2026-11-01T10:00:00.000Z",
  "app":   { "uptimeMs": 0, "analyzeRequests": 0, "refusedBeforeModel": 0 },
  "queue": { "inFlight": 0, "limit": 4, "atCapacity": false, "accepted": 0 },
  "model": {
    "outcomes": { "suggestion": 0, "abstained": 0, "failure": 0 },
    "failureClasses": { "ingress_refused": 0, "rate_limited": 0, "disabled": 0,
                        "provider_unreachable": 0, "provider_rejected": 0,
                        "schema_invalid": 0, "timeout": 0, "unexpected": 0 },
    "abstentionRate": null,
    "readiness": { "phase": "checking", "ready": false, "coldLoadMs": null },
    "latency": { "count": 0, "meanMs": null, "maxMs": 0,
                 "bucketsMs": { "1000": 0, "2000": 0, "5000": 0, "15000": 0, "60000": 0 } }
  }
}
```

The three blocks are the three boundaries, and each counts only what it owns:

| Block | Owns | Does not own |
|---|---|---|
| `app` | Every `POST /api/analyze` the process saw, and the ones refused before any model work (bad bytes, per-device budget, analysis disabled). | Queue occupancy or model answers. |
| `queue` | Slots handed out and released, and the configured concurrency ceiling. | Whether the request succeeded. |
| `model` | Answers (`suggestion` / `abstained`), classified failures, readiness phase, and model-work latency. | Anything decided before a slot was taken. |

`abstentionRate` is `abstained / (suggestion + abstained)`; it is `null` until the model has answered
at least once. A failed request is **not** an abstention — an outage must not look like a cautious
model.

`latency.bucketsMs` is cumulative: `"5000"` counts every sample at or below 5 s. `coldLoadMs` is the
interval from process start to the first observation in which the pinned model was resident; a worker
that flaps back to `offline` keeps the original figure.

`readiness.phase` uses the same vocabulary the browser uses for the AI affordance (`checking`,
`disabled`, `offline`, `loading`, `ready`), and the value is produced by the same
`visionReadiness()` function the app renders, so a metric and a screen cannot disagree.

## What can never appear in it

The only strings this module stores are members of three frozen lists — outcomes, failure classes,
and readiness phases — plus a fixed latency-bucket edge set. Anything else is bucketed, not copied:

- an unrecognised outcome becomes `failure`;
- an unrecognised failure class becomes `unexpected`;
- an unrecognised readiness phase becomes `checking`.

Therefore no note text, no image bytes, no image hash or fingerprint, no subject key, no caller
address, and no session or cookie value can reach a label. `tests/model-metrics.test.js` asserts this
by feeding hostile strings into every label position, and
`tests/metrics-endpoint.acceptance.test.js` asserts it through the shipping server.

## Reading it

```bash
curl -sS -H "Authorization: Bearer $TIMMY_METRICS_TOKEN" http://127.0.0.1:4173/api/metrics | jq .
```

For a quick look that does not need `jq`, the report renders the same snapshot as three labelled blocks:

```bash
TIMMY_METRICS_TOKEN=... node scripts/model_metrics_report.mjs --url http://127.0.0.1:4173
```

Exit codes are the point of the command as much as the numbers: `0` means a snapshot was rendered,
`1` means the server answered non-200 (404 says the route is disabled because no token is configured,
403 says the supplied token was rejected), `2` means the command itself was called wrongly or the
server could not be reached. The token is read from the environment or `--token` and is never echoed
in the report or in an error line.

What the numbers mean to an operator:

| Signal | Reading |
|---|---|
| `readiness.phase: offline` | The worker is not answering. Manual logging is unaffected; do not restart the app for this. |
| `readiness.phase: loading` | The worker answered but the pinned model is not resident. AI is withheld rather than offered and failed. |
| `queue.atCapacity: true` | Every model slot is in use. Watch `latency.maxMs`; a growing figure with no completions is a stuck inference, not load. |
| `failureClasses.provider_unreachable` climbing | Network or worker outage, not caller error. |
| `failureClasses.schema_invalid` climbing | The worker is answering off-contract. Treat as a regression in the pinned model or prompt, not a user problem. |
| `app.refusedBeforeModel` climbing with no model traffic | Abuse or a broken client; the model never saw it. |

## Known gap

`queue.inFlight` is the app's own admission depth for `POST /api/analyze`. The full bounded queue
(concurrency limit enforcement, cancellation, overload fallback) is issue
[#17](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/17); this
snapshot reports the slot accounting that exists today and does not claim the rest.