stackchain/timmy-talking-turd
2
0 Fork 0
Code Issues 21 Pull Requests 7 Actions 4 Packages Projects Releases 41 Wiki Activity

feat(corpus): publish the dataset card and labeling guide from the manifest (#27)
Quality gates / quality (pull_request) Canceled after 0s
Details

Browse Source 
The corpus had an append-only manifest, a reviewer tool, and an evaluation harness, but nothing a
stranger could read: what is in the corpus, what a label means, and how leakage is prevented lived in
code and in reviewers' heads.

scripts/dataset_card.py generates docs/DATASET_CARD.md from manifest.jsonl. Every count is read back
from real records — active samples, contributors, per-split contributor and sample counts, class
balance over active samples only — and every fact the pipeline does not collect (devices, contexts,
licenses) is printed as pending rather than estimated. The generator refuses to render a card for a
corpus where one contributor straddles two splits or where a recorded split is not the one the
subject key derives, and it never publishes a subject key or a withdrawn sample's labels. --check
fails when the committed page is not the generator's own output, so the card cannot drift into a
hand-written document.

docs/LABELING_GUIDE.md is the reviewer's rulebook: visible form, broad visible color, and photo
quality; visible color is not blood; nothing is diagnosed; a model suggestion is provisional input to
a human decision. Exclusions are named, and the split policy is stated where it is enforced.

The demo's new DATASET CARD scene builds a throwaway corpus with the real ingest command from
committed synthetic fixtures, publishes the card, and shows the refusal a planted contributor earns,
then re-checks the committed page. Its captions are release gates in build_release.py, which also runs
tests/dataset-card.test.py before a release exists.

tests/dataset-card.test.py covers the counts, the pending facts, the withdrawal receipt, both
refusals, determinism, the empty-corpus case, the committed-page identity, and both documents'
prohibitions. Every fixture is synthetic.

Closes #27
This commit is contained in:
timmy committed 2026-10-28 11:35:41 +00:00
1 parent b191d0427d
commit 9dabad495b
11 files changed
+1014 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitea/workflows/quality.yml
+1
View File
@@ -35,6 +35,7 @@ jobs:
python3 tests/staging-deploy.test.py -v
python3 tests/corpus-manifest.test.py -v
python3 tests/corpus-review.test.py -v
python3 tests/dataset-card.test.py -v
python3 tests/calibration-harness.test.py -v
- name: Mobile browser acceptance
run: |
README.md
+2
View File
@@ -138,6 +138,8 @@ The calibration harness ([scripts/calibration_report.py](scripts/calibration_rep
The reviewer tool ([scripts/review_corpus.py](scripts/review_corpus.py), [contract](docs/REVIEWER-TOOL.md)) is the labeling control for that corpus, and it is a command-line tool with no screen in the app. The review queue shows the contributor's proposal and withholds every model field, a reveal is refused until a human has committed their own labels, a correction must actually change the proposal, adjudication is appended beside the review it resolves instead of overwriting it, and a withdrawn contribution cannot be queued, reviewed, revealed, adjudicated, or trained on. Unreviewed and excluded samples are never training data. It records decisions; it does not measure a model.
What the corpus is, and what a label may assert, is published rather than implied: [docs/DATASET_CARD.md](docs/DATASET_CARD.md) is generated from the manifest by [scripts/dataset_card.py](scripts/dataset_card.py), so every count on it is read back from real records and every fact the pipeline does not collect (devices, contexts, licenses) is printed as `pending` instead of estimated — a card for a corpus where one contributor straddles two splits is refused, not published. [docs/LABELING_GUIDE.md](docs/LABELING_GUIDE.md) is the reviewer's rulebook: visible form, broad visible color, and photo quality only, visible color is not blood, nothing is diagnosed, and a model suggestion is provisional input to a human decision rather than the label.
## Architecture
```text
docs/DATASET_CARD.md
+104
View File
@@ -0,0 +1,104 @@
# Dataset card — Timmy consented stool-image corpus
Generated by [`scripts/dataset_card.py`](../scripts/dataset_card.py) from the append-only
corpus manifest. Do not edit this page by hand: regenerate it, or `--check` fails. Every count
below is read back from `manifest.jsonl`; a fact the pipeline does not collect is printed as
`pending` rather than estimated, so nothing here is invented.
| Field | Value |
|---|---|
| Corpus | `/srv/timmy-corpus` |
| Corpus present on this machine | no |
| Last manifest record | pending — no record yet |
| Active samples | 0 |
| Contributors | 0 |
| Withdrawals | 0 |
| Contexts | pending — not collected by ingest — no setting, time-of-day, or location fields are recorded |
| Devices | pending — not collected by ingest — no camera, model, or OS fields are recorded |
| Licenses | pending — no contributor grant is recorded yet, so no license is asserted |
| Regenerate | `python3 scripts/dataset_card.py --corpus /srv/timmy-corpus` |
## What this corpus is for
Consented photos of stool, labelled by a human reviewer, for research on a stool-form
classifier. The corpus may not be used to diagnose disease, to identify a person, or to
train a model that speaks to a patient with clinical authority. A model's output stays
provisional until the person logging confirms it: see [PRODUCT-DECISIONS.md](PRODUCT-DECISIONS.md).
This corpus holds no consented images yet. Every count of zero below is a real count of
an empty manifest, not a placeholder, and the pending facts stay pending until the
pipeline collects them.
## Contributor-level split policy
Splits are derived from the salted pseudonymous subject key, never assigned per photo: the
first byte of the key decides the split, `< 205` train, `< 230` validation, otherwise test
(the ingestion rule in `scripts/ingest_training_photo.py`). One contributor's photos all land
in one split, and this generator refuses a manifest where a subject key appears in two splits
or where a recorded split is not the one its subject key derives. Both edges are stated here
so the policy on this page is the policy in the code.
| Split | Contributors | Active samples |
|---|---:|---:|
| `train` | 0 | 0 |
| `validation` | 0 | 0 |
| `test` | 0 | 0 |
## Class balance (active samples only)
**Type**
| Type | Active samples |
|---|---:|
| Type 1 | 0 |
| Type 2 | 0 |
| Type 3 | 0 |
| Type 4 | 0 |
| Type 5 | 0 |
| Type 6 | 0 |
| Type 7 | 0 |
**Color**
| Color | Active samples |
|---|---:|
| `brown` | 0 |
| `green` | 0 |
| `yellow` | 0 |
| `pale` | 0 |
| `red` | 0 |
| `black` | 0 |
**Quality**
| Quality | Active samples |
|---|---:|
| `good` | 0 |
| `fair` | 0 |
| `poor` | 0 |
**Review status**
| Review status | Active samples |
|---|---:|
| `user-confirmed` | 0 |
| `clinician-reviewed` | 0 |
A withdrawn sample is excluded from every table above. Labels are what a reviewer saw, not a
diagnosis: [LABELING_GUIDE.md](LABELING_GUIDE.md) says what a label may and may not assert.
## Exclusions and withdrawals
A withdrawal appends a tombstone, deletes the derived image and any raw copy, and stays in the
history. Contribution ids and reasons are published here because a deletion receipt is
checkable; subject keys and labels are never published.
No withdrawals are recorded.
## Limitations
- Composition (devices, contexts, licenses) is pending, so the corpus is not yet a
representative sample of anything and must not be described as one.
- Review status is a human decision, not a clinical gold standard; `user-confirmed` means the
person who took the photo agreed with the proposal.
- A held-out split measures generalization across contributors, not across a population.
docs/LABELING_GUIDE.md
+84
View File
@@ -0,0 +1,84 @@
# Labeling guide — Timmy corpus review
This is the guide a human reviewer follows when labelling a contributed photo in
[`scripts/review_corpus.py`](../scripts/review_corpus.py). It exists so a label in the corpus means
the same thing to every reviewer, and so a model trained on those labels cannot quietly inherit a
clinical claim nobody made. The corpus it describes is summarised, from the manifest and never by
hand, in [DATASET_CARD.md](DATASET_CARD.md).
A label is an observation. It is **not** a diagnosis, not a severity assessment, and not a statement
about the person's health.
## What a reviewer may record
Three fields, all of them visible in the photo:
| Field | Vocabulary | Rule |
|---|---|---|
| `bristolType` | 1–7 | The Bristol form the sample most resembles, by shape and surface — not by how the person says they feel. |
| `color` | `brown`, `green`, `yellow`, `pale`, `red`, `black` | The **broad visible color** of the sample in the photo. |
| `quality` | `good`, `fair`, `poor` | Whether the photo supports a confident form and color call at all. |
`quality` is about the photo, not the person: glare, blur, framing, and distance. A `poor` photo
belongs in the corpus as a `poor` photo; it is not a reason to guess a higher-quality label, and
`review_corpus.py` excludes an unclear sample from training instead of filling the gap.
## Color is not blood
The single most damaging mistake available here. Visible color describes what the photo looks like;
it does not establish that blood is present:
- `red` means **the sample looks reddish in this photo**. It is **not blood**, and it is not
"bleeding". A reviewer never writes "blood" into a label, a comment, or an exclusion reason.
- `black` means **the sample looks black or very dark in this photo**. It is not "melena", not
"GI bleed", and not evidence of anything.
- Neither field may be used to reassure. A `brown` label is **not** a statement that there is no
bleeding; a photo cannot rule that out and the app does not claim it can.
- Blood, black or tarry stool, pain, urgency, fever, or vomiting are **user-reported** facts, asked
in words and routed to the deterministic red-flag path. They are never inferred from an image and
never stored as a label by this tool.
If the photo genuinely shows something a reviewer believes needs urgent attention, the correct action
is to stop labelling and escalate to the maintainers — not to encode a clinical judgement in a class.
## Never diagnose
- Do not diagnose, rule out, or name a disease, condition, or cause in any label, comment, or
exclusion reason.
- Do not infer symptoms, pain, urgency, frequency, hydration, diet, medication, or pregnancy.
- Do not grade severity ("mild", "concerning", "abnormal"). The vocabulary is form, color, and photo
quality only.
- A model suggestion is **provisional input**, never the label. The reviewer's own committed decision
is the label; `review_corpus.py` refuses to reveal the model's answer until the reviewer has
committed theirs, and records any disagreement beside the proposal rather than averaging it away.
## Exclusions
Exclude a sample when it cannot be labelled honestly:
| Situation | Action |
|---|---|
| No stool visible, or the photo is of something else | Exclude: `not-stool`. |
| Photo too blurred, glared, or distant to judge form | Exclude: `unreadable-photo`. |
| Two samples in one frame, or ambiguous framing | Exclude: `ambiguous-frame`. |
| Reviewer believes the image needs urgent clinical attention | **Stop.** Escalate to maintainers; do not label, do not exclude quietly. |
| Contributor withdrew consent | Not a review decision: the manifest tombstones the contribution and the queue drops it. |
An exclusion is recorded with a reason from the list above. An excluded sample is never training
data, and re-including it requires a new review, not an edit.
## Splits: one contributor, one split
All of a contributor's photos stay in **one split**, decided by their salted pseudonymous subject key
(`< 205` train, `< 230` validation, otherwise test — the ingestion rule in
`scripts/ingest_training_photo.py`). A reviewer never chooses a split, and a reviewer never moves a
sample between splits to balance a class. If a class is thin, the honest answer is more consented
contributors, not a leaked one. `scripts/dataset_card.py` refuses to publish a card for a corpus
where one subject key appears in two splits.
## Reporting a problem with this guide
If a label feels impossible to assign without a clinical judgement, that is a gap in this guide, not
a licence to improvise. Open an issue against
[stackchain/timmy-talking-turd](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd)
describing the case, without attaching a real medical image.
package.json
+3 -2
View File
@@ -18,7 +18,7 @@
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check src/abstention-policy.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/review_corpus.py scripts/dataset_card.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check scripts/abstention_policy_receipt.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
"check:diff": "bash scripts/check_diff.sh",
"start": "node server.mjs",
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
@@ -26,7 +26,8 @@
"test:corpus": "python3 tests/corpus-manifest.test.py",
"test:calibration": "python3 tests/calibration-harness.test.py",
"test:policy": "node --test tests/abstention-policy.test.js",
"test:review": "python3 tests/corpus-review.test.py"
"test:review": "python3 tests/corpus-review.test.py",
"test:dataset-card": "python3 tests/dataset-card.test.py"
},
"devDependencies": {
"playwright": "^1.62.1"
scripts/build_release.py
+12 -1
View File
@@ -72,6 +72,10 @@ DEMO_REQUIRED_CAPTIONS = (
"The queue shows the proposal and hides the model answer; revealing it is refused until a reviewer commits",
"The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over",
"Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all",
"The dataset card is generated from the corpus manifest, so no count on it is typed by hand",
"One contributor's photos stay in one split: a corpus that leaks a person across splits is refused",
"Devices, contexts, and licenses stay pending until the pipeline collects them",
"The card is a page, not a screen in the app — it is published with the real generator against synthetic fixtures",
)
@@ -173,6 +177,9 @@ def main() -> int:
# Issue #26's review tool decides which samples a trainer may read, so the blindness and
# withdrawal rules it enforces are a release gate too.
run(["python3", "tests/corpus-review.test.py", "-v"], tree)
# Issue #27 publishes a dataset card from the manifest, so the generator that decides what
# a stranger reads about the corpus is exercised before this release exists.
run(["python3", "tests/dataset-card.test.py", "-v"], tree)
# Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
# arithmetic are exercised before this release is allowed to exist.
run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
@@ -268,7 +275,7 @@ def main() -> int:
run(["bash", "-n", "deploy/worker/entrypoint.sh"], tree)
run(["bash", "-n", "deploy/worker/build_image.sh"], tree)
run(["bash", "-n", "deploy/worker/run_worker_container.sh"], tree)
run(["python3", "-m", "py_compile", "scripts/ingest_training_photo.py", "scripts/build_release.py", "scripts/build_vision_corpus.py", "scripts/corpus_manifest.py", "scripts/review_corpus.py"], tree)
run(["python3", "-m", "py_compile", "scripts/ingest_training_photo.py", "scripts/build_release.py", "scripts/build_vision_corpus.py", "scripts/corpus_manifest.py", "scripts/review_corpus.py", "scripts/dataset_card.py"], tree)
run(["git", "diff", "--check", commit], tree)
names = tracked_files(tree)
@@ -306,6 +313,9 @@ def main() -> int:
"- Scope note: the corpus pipeline is a command-line tool. There is no corpus screen in the app, and this release does not imply one — the recorded demo runs the real ingest and revoke commands against the committed synthetic fixture and shows their actual output\n"
"- Blind reviewer and adjudication tool (issue #26): the review queue shows the proposal and withholds every model field, revealing a model answer is refused until a human has committed their own labels, a correction must actually change the proposal, the ledger is append-only, and a second reader's adjudication is appended beside the review it resolves rather than replacing it; a withdrawn contribution cannot be queued, reviewed, revealed, adjudicated, or trained on, and unreviewed or excluded samples are never training data\n"
"- Scope note: the review tool is a command line, not a screen in the app. There is no reviewer screen, and this release does not imply one — the recorded demo runs the real queue, review, reveal, adjudicate, and trainable commands against the committed synthetic sample and shows their actual output, with a synthetic answer file that no model produced\n"
"- Published dataset card and labeling guide (issue #27): the card is generated from the corpus manifest by scripts/dataset_card.py, so every count on it is read back from the manifest and every fact the pipeline does not collect (devices, contexts, licenses) is printed as pending instead of estimated; the generator refuses to publish a card for a corpus where one contributor straddles two splits or where a recorded split is not the one the subject key derives, a withdrawn sample contributes no label to the balance tables, and no subject key ever reaches the page\n"
"- Contributor-level split policy, published where it is enforced: splits follow the salted subject key (below 205 train, below 230 validation, otherwise test), all of a contributor's photos land in one split, and docs/LABELING_GUIDE.md states that a reviewer records visible form, broad visible color, and photo quality only — visible color is not blood, nothing is diagnosed, and a model suggestion is provisional input to a human decision rather than the label\n"
"- Scope note: the dataset card is a generated document, not a screen in the app, and this release does not imply one — the recorded demo builds a throwaway corpus with the real ingest command from synthetic fixtures, publishes the card with the real generator, and shows the refusal a planted contributor earns\n"
"- Known limitation: open-weight VLM acceptance is proven, Bristol accuracy is not clinically validated\n"
"- Package excludes model weights, medical-image corpora, raw research pages, generated screenshots, and video production assets\n"
)
@@ -369,6 +379,7 @@ def main() -> int:
"training_contribution_consent": "passed",
"corpus_manifest_provenance": "passed",
"reviewer_adjudication_blind": "passed",
"dataset_card_generated": "passed",
"calibration_harness_gates": "passed",
"portable_backup_restore": "passed",
"dependency_audit_high": "passed",
scripts/dataset_card.py
+344
View File
@@ -0,0 +1,344 @@
#!/usr/bin/env python3
"""Generate the published dataset card from the append-only corpus manifest (issue #27).
A dataset card that is typed by hand drifts from the corpus it describes, and a card that estimates
its own numbers is worse than no card at all. This generator therefore only reports what the manifest
actually contains:
- counts of active samples, contributors, and withdrawals,
- the contributor-level split policy, checked before a single line is rendered,
- class balance over the active samples only,
- every fact the pipeline does not collect (devices, contexts, licenses) printed as ``pending``.
Nothing is inferred. A person's pseudonymous subject key never reaches the page, and a withdrawn
sample contributes no label to the balance tables. The output is a pure function of the manifest, so
the committed card can be regenerated and compared byte for byte.
Usage:
python3 scripts/dataset_card.py --corpus /srv/timmy-corpus
python3 scripts/dataset_card.py --corpus /srv/timmy-corpus --out docs/DATASET_CARD.md
python3 scripts/dataset_card.py --corpus /srv/timmy-corpus --check docs/DATASET_CARD.md
python3 scripts/dataset_card.py --corpus /srv/timmy-corpus --json
Exit codes: 0 the card rendered (and matched, with --check); 1 the card on disk is out of date;
2 the corpus cannot be described honestly and was refused.
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from corpus_manifest import ( # noqa: E402
COLORS,
QUALITIES,
REVIEWERS,
SAMPLE,
SPLITS,
TOMBSTONE,
CorpusError,
active_records,
load_manifest,
)
from calibration_report import expected_split # noqa: E402
# The facts no current pipeline step records. They are reported as pending rather than guessed.
PENDING_FACTS = ("contexts", "devices", "licenses")
PENDING_LABELS = {
"devices": "Devices",
"contexts": "Contexts",
"licenses": "Licenses",
}
PENDING_REASONS = {
"devices": "not collected by ingest — no camera, model, or OS fields are recorded",
"contexts": "not collected by ingest — no setting, time-of-day, or location fields are recorded",
"licenses": "no contributor grant is recorded yet, so no license is asserted",
}
class CardError(Exception):
"""The corpus cannot be described as asked."""
def assert_contributor_held_out(records: list[dict]) -> None:
"""Refuse a corpus where one contributor straddles splits or a split cannot be re-derived.
The rule is the ingestion rule, so a corpus cannot quietly relabel its own splits: a split must
follow from the pseudonymous subject key alone, and one person's photos must all land in one
split. Otherwise every held-out number downstream is measuring a person the model already met.
"""
splits_by_subject: dict[str, set[str]] = {}
for record in records:
if record.get("recordType", SAMPLE) != SAMPLE:
continue
subject = record.get("subjectKey")
if not isinstance(subject, str) or not subject:
raise CardError("every sample must carry the pseudonymous subject key")
split = record.get("split")
if split not in SPLITS:
raise CardError(f"split must be one of {', '.join(SPLITS)}: {split!r}")
splits_by_subject.setdefault(subject, set()).add(split)
for subject, splits in sorted(splits_by_subject.items()):
if len(splits) > 1:
raise CardError(
f"contributor {subject} leaks across {', '.join(sorted(splits))}: "
"one contributor's photos must stay in one split"
)
for record in records:
if record.get("recordType", SAMPLE) != SAMPLE:
continue
try:
required = expected_split(record["subjectKey"])
except Exception as exc: # noqa: BLE001 - the rule reports its own refusal
raise CardError(str(exc)) from exc
if record["split"] != required:
raise CardError(
f"split {record['split']!r} for contributor {record['subjectKey']} is not reproducible: "
f"the subject key derives {required!r}"
)
def summarize(records: list[dict]) -> dict:
"""Count what the manifest holds. Pure function of the records, in a stable order."""
assert_contributor_held_out(records)
active = active_records(records)
# A withdrawal can appear as a tombstone or as a revoked sample record (both are legal history);
# it is one withdrawn contribution either way, so count distinct contribution ids.
withdrawn: dict[str, dict] = {}
for record in records:
is_tombstone = record.get("recordType") == TOMBSTONE
is_revoked_sample = (
record.get("recordType", SAMPLE) == SAMPLE
and record.get("lifecycle", {}).get("state") == "revoked"
)
if not (is_tombstone or is_revoked_sample):
continue
contribution = record.get("contributionId")
if not isinstance(contribution, str) or not contribution:
raise CardError("a withdrawn record must carry the contribution id it revokes")
lifecycle = record.get("lifecycle", {})
entry = withdrawn.setdefault(
contribution,
{"contributionId": contribution, "withdrawnAt": None, "reason": None, "artifacts": 0},
)
stamp = lifecycle.get("updatedAt")
if stamp and (entry["withdrawnAt"] is None or stamp > entry["withdrawnAt"]):
entry["withdrawnAt"] = stamp
if lifecycle.get("reason"):
entry["reason"] = lifecycle["reason"]
if is_tombstone:
entry["artifacts"] = len(record.get("deletedArtifacts") or [])
by_split = {split: {"contributors": set(), "samples": 0} for split in SPLITS}
for record in active:
by_split[record["split"]]["contributors"].add(record["subjectKey"])
by_split[record["split"]]["samples"] += 1
bristol = {str(day): 0 for day in range(1, 8)}
color = {name: 0 for name in COLORS}
quality = {name: 0 for name in QUALITIES}
review = {name: 0 for name in REVIEWERS}
for record in active:
labels = record.get("labels", {})
bristol[str(labels.get("bristolType"))] = bristol.get(str(labels.get("bristolType")), 0) + 1
if labels.get("color") in color:
color[labels["color"]] += 1
if labels.get("quality") in quality:
quality[labels["quality"]] += 1
status = record.get("review", {}).get("status")
if status in review:
review[status] += 1
stamps = [record.get("ingestedAt") for record in records if record.get("ingestedAt")]
stamps += [entry["withdrawnAt"] for entry in withdrawn.values() if entry["withdrawnAt"]]
withdrawals_detail = [
{
"contributionId": entry["contributionId"],
"withdrawnAt": entry["withdrawnAt"],
"reason": entry["reason"],
"artifacts": entry["artifacts"],
}
for entry in withdrawn.values()
]
return {
"activeSamples": len(active),
"contributors": len({record["subjectKey"] for record in active}),
"withdrawals": len(withdrawn),
"bySplit": {split: {"contributors": len(entry["contributors"]), "samples": entry["samples"]} for split, entry in by_split.items()},
"bristol": bristol,
"colors": color,
"qualities": quality,
"review": review,
"lastRecordAt": max(stamps) if stamps else None,
"withdrawalsDetail": sorted(withdrawals_detail, key=lambda entry: (entry["withdrawnAt"] or "", entry["contributionId"] or "")),
"pending": sorted(PENDING_FACTS),
}
def _count_table(title: str, rows: list[tuple[str, int]]) -> list[str]:
lines = [f"**{title}**", "", f"| {title} | Active samples |", "|---|---:|"]
lines += [f"| {label} | {count} |" for label, count in rows]
lines.append("")
return lines
def render_card(summary: dict, *, corpus: str) -> str:
"""Render the card. Every value here comes from the manifest, or is marked pending."""
present = summary["activeSamples"] > 0 or summary["withdrawals"] > 0
lines = [
"# Dataset card — Timmy consented stool-image corpus",
"",
"Generated by [`scripts/dataset_card.py`](../scripts/dataset_card.py) from the append-only",
"corpus manifest. Do not edit this page by hand: regenerate it, or `--check` fails. Every count",
"below is read back from `manifest.jsonl`; a fact the pipeline does not collect is printed as",
"`pending` rather than estimated, so nothing here is invented.",
"",
"| Field | Value |",
"|---|---|",
f"| Corpus | `{corpus}` |",
f"| Corpus present on this machine | {'yes' if present else 'no'} |",
f"| Last manifest record | {summary['lastRecordAt'] or 'pending — no record yet'} |",
f"| Active samples | {summary['activeSamples']} |",
f"| Contributors | {summary['contributors']} |",
f"| Withdrawals | {summary['withdrawals']} |",
]
for fact in PENDING_FACTS:
lines.append(f"| {PENDING_LABELS[fact]} | pending — {PENDING_REASONS[fact]} |")
lines.append(f"| Regenerate | `python3 scripts/dataset_card.py --corpus {corpus}` |")
lines += [
"",
"## What this corpus is for",
"",
"Consented photos of stool, labelled by a human reviewer, for research on a stool-form",
"classifier. The corpus may not be used to diagnose disease, to identify a person, or to",
"train a model that speaks to a patient with clinical authority. A model's output stays",
"provisional until the person logging confirms it: see [PRODUCT-DECISIONS.md](PRODUCT-DECISIONS.md).",
"",
]
if summary["activeSamples"] == 0:
lines += [
"This corpus holds no consented images yet. Every count of zero below is a real count of",
"an empty manifest, not a placeholder, and the pending facts stay pending until the",
"pipeline collects them.",
"",
]
lines += [
"## Contributor-level split policy",
"",
"Splits are derived from the salted pseudonymous subject key, never assigned per photo: the",
"first byte of the key decides the split, `< 205` train, `< 230` validation, otherwise test",
"(the ingestion rule in `scripts/ingest_training_photo.py`). One contributor's photos all land",
"in one split, and this generator refuses a manifest where a subject key appears in two splits",
"or where a recorded split is not the one its subject key derives. Both edges are stated here",
"so the policy on this page is the policy in the code.",
"",
"| Split | Contributors | Active samples |",
"|---|---:|---:|",
]
for split in SPLITS:
entry = summary["bySplit"][split]
lines.append(f"| `{split}` | {entry['contributors']} | {entry['samples']} |")
lines += ["", "## Class balance (active samples only)", ""]
lines += _count_table("Type", [(f"Type {day}", summary["bristol"][str(day)]) for day in range(1, 8)])
lines += _count_table("Color", [(f"`{name}`", count) for name, count in summary["colors"].items()])
lines += _count_table("Quality", [(f"`{name}`", count) for name, count in summary["qualities"].items()])
lines += _count_table("Review status", [(f"`{name}`", count) for name, count in summary["review"].items()])
lines += [
"A withdrawn sample is excluded from every table above. Labels are what a reviewer saw, not a",
"diagnosis: [LABELING_GUIDE.md](LABELING_GUIDE.md) says what a label may and may not assert.",
"",
"## Exclusions and withdrawals",
"",
"A withdrawal appends a tombstone, deletes the derived image and any raw copy, and stays in the",
"history. Contribution ids and reasons are published here because a deletion receipt is",
"checkable; subject keys and labels are never published.",
"",
]
if summary["withdrawalsDetail"]:
lines += [
"| Contribution | Withdrawn at | Reason | Artifacts removed |",
"|---|---|---|---:|",
]
for entry in summary["withdrawalsDetail"]:
lines.append(
f"| `{entry['contributionId']}` | {entry['withdrawnAt'] or 'pending'} | "
f"{entry['reason'] or 'pending'} | {entry['artifacts']} |"
)
else:
lines.append("No withdrawals are recorded.")
lines += [
"",
"## Limitations",
"",
"- Composition (devices, contexts, licenses) is pending, so the corpus is not yet a",
" representative sample of anything and must not be described as one.",
"- Review status is a human decision, not a clinical gold standard; `user-confirmed` means the",
" person who took the photo agreed with the proposal.",
"- A held-out split measures generalization across contributors, not across a population.",
"",
]
return "\n".join(lines)
def load_corpus(corpus: Path) -> list[dict]:
try:
return load_manifest(corpus / "manifest.jsonl")
except (CorpusError, json.JSONDecodeError) as exc:
raise CardError(f"the manifest at {corpus / 'manifest.jsonl'} cannot be read: {exc}") from exc
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="Generate the Timmy dataset card from the corpus manifest.")
parser.add_argument("--corpus", required=True, help="dataset root holding manifest.jsonl")
parser.add_argument("--out", help="write the card here instead of stdout")
parser.add_argument("--check", help="fail if the card at this path is not what would be generated")
parser.add_argument("--json", action="store_true", help="print the summary object instead of the card")
parser.add_argument("--corpus-label", help="path to name on the card (defaults to --corpus)")
args = parser.parse_args(argv)
corpus = Path(args.corpus)
try:
records = load_corpus(corpus)
summary = summarize(records)
except CardError as exc:
print(f"refusing to render a dataset card: {exc}", file=sys.stderr)
return 2
if args.json:
print(json.dumps(summary, indent=2, sort_keys=True))
return 0
card = render_card(summary, corpus=args.corpus_label or args.corpus)
if args.check:
target = Path(args.check)
existing = target.read_text(encoding="utf-8") if target.exists() else ""
if existing != card:
print(
f"{target} is out of date: regenerate it with "
f"`python3 scripts/dataset_card.py --corpus {args.corpus_label or args.corpus} --out {target}`",
file=sys.stderr,
)
return 1
print(f"{target} matches the corpus manifest")
return 0
if args.out:
target = Path(args.out)
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(card, encoding="utf-8")
print(f"wrote {target} ({summary['activeSamples']} active samples, {summary['contributors']} contributors)")
return 0
sys.stdout.write(card)
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/record_release_demo.mjs
+125
View File
@@ -921,6 +921,131 @@ await caption('A second reviewer adjudicates, and the review being overruled is
await caption('Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all', 3800);
await page.evaluate(() => document.querySelector('#review-audit')?.remove());
// Issue #27 — the dataset card is a generated document, not a hand-written page. This scene builds a
// throwaway corpus with the real ingest command from committed synthetic fixtures, publishes the card
// with the real generator, and shows the refusal the split policy earns when one contributor is
// planted across two splits. Every count, id, and refusal line on the card is read back from those
// runs. No model labelled anything in this corpus, and the caption says so.
async function cardCli(args) {
try {
const result = await execFileAsync('python3', ['scripts/dataset_card.py', ...args], { cwd: process.cwd() });
return { code: 0, stdout: result.stdout, stderr: result.stderr };
} catch (error) {
return {
code: typeof error.code === 'number' ? error.code : 1,
stdout: error.stdout || '',
stderr: error.stderr || '',
};
}
}
async function publishDatasetCard() {
const dataset = await mkdtemp(path.join(os.tmpdir(), 'timmy-card-demo-'));
const leaked = await mkdtemp(path.join(os.tmpdir(), 'timmy-card-leak-'));
const cardPath = path.join(dataset, 'DATASET_CARD.md');
// Three distinct committed synthetic fixtures: a formed sample, a well-lit stool-like sample, and a
// glare frame that can only honestly be labelled poor quality.
const contributions = [
{ id: 'demo-card-0001-abcd', subject: 'demo-contributor-a', file: 'tests/fixtures/synthetic-type4.jpg', type: '4', color: 'brown', quality: 'good' },
{ id: 'demo-card-0002-abcd', subject: 'demo-contributor-a', file: 'tests/fixtures/vision-corpus/stool-like-formed.png', type: '3', color: 'green', quality: 'fair' },
{ id: 'demo-card-0003-abcd', subject: 'demo-contributor-b', file: 'tests/fixtures/vision-corpus/glare-overexposed.png', type: '1', color: 'brown', quality: 'poor' },
];
try {
for (const item of contributions) {
await corpusCli([
'scripts/ingest_training_photo.py',
'--input', item.file,
'--output-dir', dataset,
'--subject-id', item.subject,
'--consent-version', 'training-consent-v1',
'--contribution-id', item.id,
'--bristol-type', item.type, '--color', item.color, '--quality', item.quality,
'--reviewer', 'user-confirmed',
]);
}
// Withdraw one of the first contributor's two samples, so both contributors still have an active
// sample and the class table shows the withdrawn label dropping out of the count.
const withdrawn = contributions[1].id;
await corpusCli([
'scripts/ingest_training_photo.py', '--output-dir', dataset,
'--revoke', withdrawn, '--reason', 'participant withdrew consent',
]);
const published = await cardCli(['--corpus', dataset, '--out', cardPath]);
if (published.code !== 0) throw new Error(`the dataset card was refused: ${lastLine(published.stderr)}`);
const summary = JSON.parse((await cardCli(['--corpus', dataset, '--json'])).stdout);
if (summary.activeSamples !== 2) throw new Error('the card must count the two active samples the run ingested');
if (summary.contributors !== 2) throw new Error('the card must count both contributors');
if (summary.withdrawals !== 1) throw new Error('the card must count the withdrawal as a withdrawal');
const card = await readFile(cardPath, 'utf8');
for (const fact of ['| Devices | pending', '| Contexts | pending', '| Licenses | pending']) {
if (!card.includes(fact)) throw new Error(`an uncollected fact must be printed as pending: ${fact}`);
}
if (!card.includes(withdrawn) || card.includes('subjectKey')) {
throw new Error('the card must name the withdrawn contribution and never carry a subject key');
}
if (summary.bristol['3'] !== 0) throw new Error('a withdrawn sample must not be counted in the class balance');
// Plant one contributor in a second split and prove the policy refuses to publish a card at all.
const original = await readFile(path.join(dataset, 'manifest.jsonl'), 'utf8');
const records = original.split('\n').filter(line => line.trim()).map(line => JSON.parse(line));
const sample = records.find(record => record.recordType === 'sample' && record.contributionId === contributions[0].id);
const planted = { ...sample, contributionId: 'demo-card-0009-leak', split: sample.split === 'train' ? 'test' : 'train' };
await writeFile(path.join(leaked, 'manifest.jsonl'), `${original}${JSON.stringify(planted)}\n`);
const refused = await cardCli(['--corpus', leaked, '--out', path.join(leaked, 'DATASET_CARD.md')]);
if (refused.code === 0) throw new Error('the generator must refuse a corpus that leaks one contributor across splits');
const leakRefusal = lastLine(refused.stderr);
if (!/leaks across/.test(leakRefusal)) throw new Error(`the leak refusal was not legible: ${leakRefusal}`);
// The committed page has to be this generator's own output, not a document someone typed.
const committed = await cardCli(['--corpus', '/srv/timmy-corpus', '--check', 'docs/DATASET_CARD.md']);
if (committed.code !== 0) throw new Error("the committed dataset card must be the generator's own output");
return { summary, leakRefusal, withdrawn };
} finally {
await rm(dataset, { recursive: true, force: true });
await rm(leaked, { recursive: true, force: true });
}
}
const cardEvidence = await publishDatasetCard();
await tap('[data-view="home"]', 450);
const splits = cardEvidence.summary.bySplit;
const cardRows = [
['command', 'python3 scripts/dataset_card.py --corpus <tmp> --out DATASET_CARD.md'],
['corpus', `${cardEvidence.summary.activeSamples} active · ${cardEvidence.summary.contributors} contributors · ${cardEvidence.summary.withdrawals} withdrawn`],
['splits', `train ${splits.train.contributors}c/${splits.train.samples}s · validation ${splits.validation.contributors}c/${splits.validation.samples}s · test ${splits.test.contributors}c/${splits.test.samples}s`],
['class', `Type 1 ${cardEvidence.summary.bristol['1']} · Type 3 ${cardEvidence.summary.bristol['3']} · Type 4 ${cardEvidence.summary.bristol['4']} · withdrawn label excluded`],
['withdrawn', `${cardEvidence.withdrawn} · "participant withdrew consent" · no label survives in any table`],
['pending', 'devices · contexts · licenses — not collected, not estimated'],
['—', ''],
['command', 'publish a manifest that plants one contributor in train and test'],
['refused', cardEvidence.leakRefusal],
['card', 'not written: the policy is checked before a line is rendered'],
['—', ''],
['command', '--check docs/DATASET_CARD.md'],
['committed', 'the published page is this generator\'s own output'],
['data', 'synthetic fixtures · no model labelled anything here'],
];
const cardPanel = await evidenceCard({
id: 'dataset-card',
title: 'DATASET CARD',
subtitle: 'generated from the manifest — command line, not app UI',
rows: cardRows,
keyWidth: 70,
});
await sleep(420);
const cardProof = await evidenceCardProof('dataset-card');
if (!cardProof.ok) throw new Error(`Dataset card is not fully readable on screen: ${JSON.stringify(cardProof)}`);
if (!cardPanel.text.includes(`${cardEvidence.summary.contributors} contributors`)) {
throw new Error('the card must show the contributor count the run produced');
}
if (!cardPanel.text.includes('leaks across')) throw new Error('the card must show the leak refusal the run earned');
if (!cardPanel.text.includes('not collected')) throw new Error('the card must show the facts that stay pending');
await caption('The dataset card is generated from the corpus manifest, so no count on it is typed by hand', 3600);
await caption('One contributor\'s photos stay in one split: a corpus that leaks a person across splits is refused', 3800);
await caption('Devices, contexts, and licenses stay pending until the pipeline collects them', 3400);
await caption('The card is a page, not a screen in the app — it is published with the real generator against synthetic fixtures', 3400);
await page.evaluate(() => document.querySelector('#dataset-card')?.remove());
// Issue #29 — the release gate is a command-line harness, not a screen in the app. This scene runs
// the real harness over the committed synthetic prediction fixtures and shows the receipt it wrote:
// a blocked comparison, a passing fixture, and a refused leaked corpus. Every number and every
tests/ci-workflow.test.js
+11
View File
@@ -34,6 +34,17 @@ test('CI pins the Python image dependency required by the full unit suite', asyn
assert.match(requirements, /^Pillow==12\.3\.0$/m);
});
test('the dataset card generator and its suite are CI- and release-gated', async () => {
const workflow = await readFile(workflowPath, 'utf8');
const packageJson = JSON.parse(await readFile(packagePath, 'utf8'));
// Issue #27 publishes a page a stranger reads about the corpus, so the generator that decides its
// numbers runs in CI and is invoked by its own suite, not only by whoever remembers.
assert.match(workflow, /python3 tests\/dataset-card\.test\.py -v/);
assert.match(packageJson.scripts['test:dataset-card'], /dataset-card\.test\.py/);
assert.match(packageJson.scripts['check:syntax'], /py_compile[^&]*scripts\/dataset_card\.py/);
});
test('default unit suite includes the self-host bootstrap contract', async () => {
const packageJson = JSON.parse(await readFile(packagePath, 'utf8'));
tests/dataset-card.test.py
+300
View File
@@ -0,0 +1,300 @@
#!/usr/bin/env python3
"""Tests for the published dataset card and the contributor-level split policy.
Issue #27 asks for three things a stranger can inspect before trusting a model: a dataset card that
reports who is in the corpus, *the* labeling guide, and a contributor-level split policy that keeps a
person's photos in exactly one split. The card is generated from the append-only manifest by
[`scripts/dataset_card.py`](../scripts/dataset_card.py) instead of being written by hand, so a number
on it is a claim the corpus can be re-checked against and an unknown fact can only appear as
``pending`` — never as a plausible-looking estimate.
Every fixture here is synthetic: records are built through the real
[`scripts/corpus_manifest.py`](../scripts/corpus_manifest.py) builder and no image bytes are read or
written.
"""
from __future__ import annotations
import importlib.util
import json
import os
import re
import subprocess
import sys
import tempfile
from pathlib import Path
import unittest
ROOT = Path(__file__).resolve().parents[1]
CARD_SCRIPT = ROOT / "scripts" / "dataset_card.py"
COMMITTED_CARD = ROOT / "docs" / "DATASET_CARD.md"
LABELING_GUIDE = ROOT / "docs" / "LABELING_GUIDE.md"
CANONICAL_CORPUS = "/srv/timmy-corpus"
# Subject keys chosen so the ingestion rule (first byte of the key) puts each contributor in a known
# split: < 205 train, < 230 validation, otherwise test.
TRAIN_A = "0a" + "1" * 14
TRAIN_B = "0b" + "2" * 14
VALIDATION_C = "d0" + "3" * 14
TEST_D = "ee" + "4" * 14
def load_module():
spec = importlib.util.spec_from_file_location("corpus_manifest", ROOT / "scripts" / "corpus_manifest.py")
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
structures = load_module()
def digest(seed: str) -> str:
import hashlib
return hashlib.sha256(seed.encode()).hexdigest()
def sample(
*,
contribution_id: str,
subject_key: str,
split: str,
bristol: int = 4,
color: str = "brown",
quality: str = "good",
reviewer: str = "user-confirmed",
at: str = "2026-08-01T10:00:00+00:00",
) -> dict:
return structures.build_record(
contribution_id=contribution_id,
consent_version="training-consent-v1",
subject_key=subject_key,
split=split,
labels={"bristolType": bristol, "color": color, "quality": quality},
review_status=reviewer,
derived_sha256=digest(contribution_id + "-derived"),
raw_sha256=digest(contribution_id + "-raw"),
ingested_at=at,
)
def write_manifest(dataset: Path, records: list[dict]) -> Path:
manifest = dataset / "manifest.jsonl"
manifest.parent.mkdir(parents=True, exist_ok=True)
with manifest.open("w", encoding="utf-8") as handle:
for record in records:
handle.write(json.dumps(record, separators=(",", ":")) + "\n")
return manifest
def run_card(args: list[str]) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, str(CARD_SCRIPT), *args],
cwd=ROOT,
capture_output=True,
text=True,
env={**os.environ},
)
def build_corpus(dataset: Path) -> None:
"""Four active samples across three contributors, plus one withdrawal."""
records = [
sample(contribution_id="contrib-train-a1", subject_key=TRAIN_A, split="train", bristol=4),
sample(contribution_id="contrib-train-a2", subject_key=TRAIN_A, split="train", bristol=3, color="green"),
sample(contribution_id="contrib-train-b1", subject_key=TRAIN_B, split="train", bristol=6, quality="fair"),
sample(
contribution_id="contrib-validation-c1",
subject_key=VALIDATION_C,
split="validation",
bristol=4,
reviewer="clinician-reviewed",
),
sample(contribution_id="contrib-test-d1", subject_key=TEST_D, split="test", bristol=1, quality="poor"),
]
revoked = structures.transition(
sample(contribution_id="contrib-withdrawn-1", subject_key=TEST_D, split="test", bristol=7, color="black"),
"revoked",
"2026-08-02T09:00:00+00:00",
"participant withdrew consent",
)
write_manifest(dataset, [*records, revoked])
class DatasetCardTest(unittest.TestCase):
def setUp(self) -> None:
self.tmp = tempfile.TemporaryDirectory()
self.dataset = Path(self.tmp.name) / "corpus"
self.dataset.mkdir(parents=True, exist_ok=True)
def tearDown(self) -> None:
self.tmp.cleanup()
def test_card_reports_the_real_corpus_counts_from_the_manifest(self) -> None:
build_corpus(self.dataset)
result = run_card(["--corpus", str(self.dataset)])
self.assertEqual(result.returncode, 0, result.stderr)
card = result.stdout
# Five active samples from four contributors; the withdrawal is reported separately.
self.assertIn("| Active samples | 5 |", card)
self.assertIn("| Contributors | 4 |", card)
self.assertIn("| Withdrawals | 1 |", card)
# Contributor-level split policy: the contributor count per split is what a reader needs to
# see that one person cannot be in two splits.
self.assertIn("| `train` | 2 | 3 |", card)
self.assertIn("| `validation` | 1 | 1 |", card)
self.assertIn("| `test` | 1 | 1 |", card)
# Class balance is counted, not described.
self.assertIn("| Type 4 | 2 |", card)
self.assertIn("| Type 1 | 1 |", card)
self.assertIn("| `black` | 0 |", card, "a withdrawn sample's label must not be counted")
self.assertIn("| `clinician-reviewed` | 1 |", card)
def test_card_marks_uncollected_facts_pending_instead_of_estimating_them(self) -> None:
build_corpus(self.dataset)
card = run_card(["--corpus", str(self.dataset)]).stdout
for fact in ("Devices", "Contexts", "Licenses"):
line = next((row for row in card.splitlines() if row.startswith(f"| {fact} |")), None)
self.assertIsNotNone(line, f"the card must name {fact}")
self.assertIn("pending", line.lower(), f"{fact} cannot be reported until it is collected")
self.assertIn("rather than estimated", card.lower())
self.assertIn("nothing here is invented", card.lower())
def test_card_names_withdrawals_without_exposing_the_person_or_their_labels(self) -> None:
build_corpus(self.dataset)
card = run_card(["--corpus", str(self.dataset)]).stdout
self.assertIn("contrib-withdrawn-1", card)
self.assertIn("participant withdrew consent", card)
for subject in (TRAIN_A, TRAIN_B, VALIDATION_C, TEST_D):
self.assertNotIn(subject, card, "the published card must never carry a subject key")
self.assertNotIn(digest("contrib-withdrawn-1" + "-derived"), card)
def test_card_refuses_a_contributor_spread_across_two_splits(self) -> None:
write_manifest(
self.dataset,
[
sample(contribution_id="contrib-leak-1", subject_key=TEST_D, split="test"),
# Same person, relabelled into training: the leak the policy exists to stop.
sample(contribution_id="contrib-leak-2", subject_key=TEST_D, split="train"),
],
)
out = Path(self.tmp.name) / "card.md"
result = run_card(["--corpus", str(self.dataset), "--out", str(out)])
self.assertNotEqual(result.returncode, 0)
self.assertRegex(result.stderr, r"contributor [0-9a-f]{16} (leaks across|is in two splits)")
self.assertFalse(out.exists(), "a refused card must not be written")
def test_card_refuses_a_split_the_subject_key_does_not_derive(self) -> None:
write_manifest(
self.dataset,
[sample(contribution_id="contrib-mislabelled-1", subject_key=TEST_D, split="train")],
)
result = run_card(["--corpus", str(self.dataset)])
self.assertNotEqual(result.returncode, 0)
self.assertRegex(result.stderr, r"not reproducible")
def test_card_states_the_split_rule_it_applies(self) -> None:
build_corpus(self.dataset)
card = run_card(["--corpus", str(self.dataset)]).stdout
# The thresholds are the ingestion rule; a card that states a different rule is a different
# policy, so both edges are named on the page.
self.assertIn("205", card)
self.assertIn("230", card)
self.assertRegex(card, r"subject key")
def test_card_is_byte_identical_between_runs(self) -> None:
build_corpus(self.dataset)
first = run_card(["--corpus", str(self.dataset)])
second = run_card(["--corpus", str(self.dataset)])
# A rendered card, not two matching empty strings: the run has to succeed and say something.
self.assertEqual(first.returncode, 0, first.stderr)
self.assertIn("| Active samples | 5 |", first.stdout)
self.assertEqual(first.stdout, second.stdout)
def test_card_is_honest_when_there_is_no_corpus(self) -> None:
missing = Path(self.tmp.name) / "absent"
result = run_card(["--corpus", str(missing)])
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("| Active samples | 0 |", result.stdout)
self.assertIn("| Contributors | 0 |", result.stdout)
self.assertRegex(result.stdout, r"no consented images")
self.assertIn("pending", result.stdout.lower(), "an empty corpus reports pending facts, not blanks")
def test_card_reports_a_withdrawal_count_without_inventing_a_denominator(self) -> None:
build_corpus(self.dataset)
summary = json.loads(run_card(["--corpus", str(self.dataset), "--json"]).stdout)
self.assertEqual(summary["activeSamples"], 5)
self.assertEqual(summary["contributors"], 4)
self.assertEqual(summary["withdrawals"], 1)
self.assertEqual(summary["bySplit"]["train"], {"contributors": 2, "samples": 3})
self.assertEqual(summary["pending"], ["contexts", "devices", "licenses"])
self.assertEqual([entry["contributionId"] for entry in summary["withdrawalsDetail"]], ["contrib-withdrawn-1"])
self.assertNotIn("subjectKey", json.dumps(summary))
def test_committed_card_is_the_generator_output_not_a_hand_written_page(self) -> None:
card = COMMITTED_CARD.read_text(encoding="utf-8")
result = run_card(["--corpus", CANONICAL_CORPUS])
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(
card,
result.stdout,
"docs/DATASET_CARD.md must be regenerated with scripts/dataset_card.py, never hand-edited",
)
self.assertIn(f"--corpus {CANONICAL_CORPUS}", card)
def test_check_mode_refuses_a_hand_edited_card(self) -> None:
build_corpus(self.dataset)
stale = Path(self.tmp.name) / "stale-card.md"
stale.write_text("# Dataset card\n\n| Active samples | 9000 |\n", encoding="utf-8")
result = run_card(["--corpus", str(self.dataset), "--check", str(stale)])
self.assertNotEqual(result.returncode, 0)
self.assertRegex(result.stderr, r"out of date|differs")
def test_labeling_guide_separates_visible_color_from_blood_and_forbids_diagnosis(self) -> None:
guide = LABELING_GUIDE.read_text(encoding="utf-8")
lowered = guide.lower()
self.assertIn("color", lowered)
# Visible color is not bleeding: the guide has to say so in the reviewer's own vocabulary.
self.assertIn("not blood", lowered)
for forbidden in ("diagnose", "diagnosis"):
self.assertIn(forbidden, lowered, "the guide must name what a label may never assert")
self.assertRegex(lowered, r"do not diagnose|never diagnos")
# Exclusions and the split policy are part of the guide, not a separate folklore.
self.assertIn("exclude", lowered)
self.assertRegex(lowered, r"one split|same split")
# The card and the guide point at each other, so a reader lands on both.
self.assertIn("DATASET_CARD.md", guide)
self.assertIn("LABELING_GUIDE.md", COMMITTED_CARD.read_text(encoding="utf-8"))
def test_labeling_guide_never_claims_a_model_decides_the_label(self) -> None:
guide = LABELING_GUIDE.read_text(encoding="utf-8")
lowered = guide.lower()
self.assertRegex(lowered, r"human|reviewer")
# A model suggestion is provisional input to a human decision, never the decision.
self.assertRegex(lowered, r"provisional|suggestion")
self.assertNotRegex(lowered, r"model (decides|determines) the label")
def test_readme_points_a_newcomer_at_the_card_and_the_guide(self) -> None:
readme = (ROOT / "README.md").read_text(encoding="utf-8")
# A published card nobody links to is not published: both pages are reachable from the front page.
self.assertIn("docs/DATASET_CARD.md", readme)
self.assertIn("docs/LABELING_GUIDE.md", readme)
self.assertIn("scripts/dataset_card.py", readme)
if __name__ == "__main__":
unittest.main(verbosity=2)
tests/release-demo.test.js
+28
View File
@@ -458,3 +458,31 @@ test('release demo runs the blind review tool in its enforced order instead of d
assert.match(builder, /The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over/);
assert.match(builder, /Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all/);
});
test('release demo publishes the dataset card from a manifest it builds on camera', async () => {
const demo = await readFile(demoPath, 'utf8');
const builder = await readFile(builderPath, 'utf8');
// Issue #27: the card is a generated artifact. The scene builds a real corpus with the real ingest
// commands, publishes the card, and re-checks the committed page, instead of describing a document.
assert.match(demo, /scripts\/dataset_card\.py/);
assert.match(demo, /id: 'dataset-card'/);
assert.match(demo, /Dataset card is not fully readable on screen/);
// The split policy is proven by the refusal, not by a caption: a leaked contributor is attempted.
assert.match(demo, /leaks across/);
assert.match(demo, /the generator must refuse a corpus that leaks one contributor across splits/);
// The pending facts must be visibly pending, and the committed page must be the generator's output.
assert.match(demo, /pending/);
assert.match(demo, /the committed dataset card must be the generator's own output/);
assert.match(demo, /docs\/DATASET_CARD\.md/);
// Nothing on the card may claim a model measured or labelled anything here.
assert.match(demo, /no model labelled anything here/);
// The scene and its guide are release gates, so a later release cannot drop them silently.
assert.match(builder, /The dataset card is generated from the corpus manifest, so no count on it is typed by hand/);
assert.match(builder, /One contributor's photos stay in one split: a corpus that leaks a person across splits is refused/);
assert.match(builder, /Devices, contexts, and licenses stay pending until the pipeline collects them/);
assert.match(builder, /tests\/dataset-card\.test\.py/);
assert.match(builder, /"dataset_card_generated": "passed"/);
assert.match(builder, /scripts\/dataset_card\.py/);
});