# Corpus manifest contract

The training corpus is described by an **append-only** `manifest.jsonl`. One line is one event;
nothing is ever rewritten in place, so the trail can be audited after a withdrawal. The
implementation lives in [`scripts/corpus_manifest.py`](../scripts/corpus_manifest.py) and is
driven by [`scripts/ingest_training_photo.py`](../scripts/ingest_training_photo.py).

## Sample record (`recordType: "sample"`, schema version 2)

| Field | Meaning |
|---|---|
| `contributionId` | Opaque id minted by the app's contribution receipt. The same id addresses every derived artifact and every later tombstone. |
| `consent.version`, `consent.recordedAt` | The consent version the contributor accepted, and when. Duplicated as `consentVersion` for older readers. |
| `source.kind`, `source.inputSha256` | How the bytes arrived (`cli-ingest`) and the digest of the caller's input. The caller's path is never recorded. |
| `hashes.rawInput`, `hashes.derived` | Digest of the supplied bytes and of the re-encoded image that is actually stored. |
| `derivation` | `format`, `maxDimension`, `metadataStripped`, and the `pipeline` version that produced the derived image. |
| `review.status`, `review.reviewedAt` | `user-confirmed` or `clinician-reviewed`. A label set is never unlabelled-by-default. |
| `split` | `train`, `validation`, or `test`, derived from the salted subject key so one contributor never straddles splits. |
| `subjectKey` | 16-hex pseudonym derived from a salted subject id. Never a name, email, or account. |
| `lifecycle.state`, `lifecycle.updatedAt` | `active` while the sample is usable for training. |
| `imageId`, `relativePath` | Derived image digest and its dataset-relative path. |

## Tombstone record (`recordType: "tombstone"`)

A withdrawal never deletes history; it appends a tombstone that:

- keeps `contributionId` and `imageId` so the withdrawal can be matched to the original sample,
- records `lifecycle.state: "revoked"`, the timestamp, and the reason,
- lists `deletedArtifacts` — every derived copy it removed, addressed by contribution id with the
  digest that identifies the bytes,
- deliberately drops `labels`, `subjectKey`, and `relativePath`: a deletion receipt keeps
  identifiers and hashes, not the health label or the person.

`revoke()` removes the derived image and any raw pre-derivation copy from the active dataset
before appending the tombstone, and refuses any path that resolves outside the dataset root.

## Lifecycle rules (all covered by `tests/corpus-manifest.test.py`)

- `active` → `revoked` only, and only with a recorded reason.
- `revoked` → `active` is refused: a revoked contribution cannot return to the active dataset.
- Revoking an unknown contribution, or revoking twice, is refused and appends nothing.
- A contribution id can be ingested once; the same derived image cannot carry two label sets.
- `active_records()` returns only samples that are still `active` and were never tombstoned, so a
  trainer that reads through it cannot silently train on withdrawn data.

## Usage

```bash
export TIMMY_DATASET_SALT=...                       # >= 8 chars, never committed
python3 scripts/ingest_training_photo.py \
  --input photo.jpg --output-dir /srv/timmy-corpus \
  --subject-id subject-123 --consent-version research-v1 \
  --contribution-id "$CONTRIBUTION_ID" \
  --bristol-type 4 --color brown --quality good --reviewer user-confirmed

python3 scripts/ingest_training_photo.py --revoke "$CONTRIBUTION_ID" \
  --output-dir /srv/timmy-corpus --reason "subject withdrawal"
```

`npm run test:corpus` runs the contract suite. Every fixture is synthetic; no real medical image
is read, written, or committed.

## Review and adjudication

A manifest says what was ingested and whether the person still allows it. It does not say whether
the labels are right. That decision lives in a separate append-only ledger,
`<dataset>/reviews.jsonl`, written by [scripts/review_corpus.py](review_corpus.py) — see
[REVIEWER-TOOL.md](REVIEWER-TOOL.md). The trainer reads `trainable_records()`, which returns only
samples that are still active, carry a human decision, and were not excluded.