Version ledger migrations and hardened JSON portability #61

rockachopa · 2026-08-22T20:40:57Z

rockachopa commented

2026-08-22 20:40:57 +00:00

Closes #35 (parent epic: #6)

Outcome

User-owned ledger records stay portable as schema versions and AI provenance fields evolve.

Changes

src/domain.js: version-ledger constants (SCHEMA_VERSION, KNOWN_SCHEMA_VERSIONS, MAX_IMPORT_BYTES = 2 MiB). importLedger migrates prior schema versions (v0 bare-array legacy exports and the v1 envelope), fails safely on future versions (RangeError with update guidance), malformed JSON, wrong-product envelopes, and oversized files before parsing user data. exportLedger normalizes entries through sanitizeEntry, so confirmed values and bounded provenance round-trip while unknown fields and smuggled secrets never enter the export file. sanitizeEntry keeps only a whitelisted provenance.origin (user | ai-suggestion).
src/analysis.js: mergeVisualSuggestion records { origin: 'ai-suggestion' } only when a suggestion is actually applied; abstentions record nothing and nonvisual fields stay user-owned.
app.js: import now merges into the existing ledger instead of replacing it - a failed or partial import can never silently drop user-owned records.
service-worker.js: shell cache bumped v5 -> v6 per base-path namespace so installed PWAs receive the migration code; activation purges obsolete v5 caches.
New tests/ledger-portability.acceptance.mjs browser gate: export round trip, merge import, future-version/malformed failure surfacing, root vs /timmy-staging storage isolation, and Delete Everything in both namespaces. Wired as npm run test:portability and added to CI quality gates.

Acceptance criteria evidence

Import supports current and prior schema versions: v0 array + v1 envelope migrate forward (all tests RED-verified failing first).
Round-trip preserves confirmed values and provenance without secrets: round-trip + re-export convergence tests; smuggled tokens/session cookies asserted absent from exported text.
Malformed, oversized, and future-version imports fail safely: dedicated fail-safe tests plus browser-level checks that the local ledger stays untouched after each rejected import.

Privacy / safety boundaries

No real personal or medical data anywhere: all fixtures synthetic.
Deterministic medical safety unchanged: urgent-flag detection, red-flag copy, escalation-before-chat untouched (full suite green).
User-owned local data preserved: merge-import, base-path namespace isolation, legacy-store compatibility, and Delete Everything guarantees covered by tests.
Live staging/systemd/Caddy/deployment untouched.

Verification receipts (local run)

npm test: 78/78 pass
python3 tests/staging-deploy.test.py: 20/20 OK
test:ui / test:photo / test:sleek / test:portability: PASS
npm audit --audit-level=high: 0 vulnerabilities
npm run check:syntax (+ node --check on the new acceptance file): pass
bash scripts/check_diff.sh HEAD^..HEAD: clean

Closes #35 (parent epic: #6) ## Outcome User-owned ledger records stay portable as schema versions and AI provenance fields evolve. ## Changes - `src/domain.js`: version-ledger constants (`SCHEMA_VERSION`, `KNOWN_SCHEMA_VERSIONS`, `MAX_IMPORT_BYTES` = 2 MiB). `importLedger` migrates prior schema versions (v0 bare-array legacy exports and the v1 envelope), fails safely on future versions (RangeError with update guidance), malformed JSON, wrong-product envelopes, and oversized files before parsing user data. `exportLedger` normalizes entries through `sanitizeEntry`, so confirmed values and bounded provenance round-trip while unknown fields and smuggled secrets never enter the export file. `sanitizeEntry` keeps only a whitelisted `provenance.origin` (`user` | `ai-suggestion`). - `src/analysis.js`: `mergeVisualSuggestion` records `{ origin: 'ai-suggestion' }` only when a suggestion is actually applied; abstentions record nothing and nonvisual fields stay user-owned. - `app.js`: import now merges into the existing ledger instead of replacing it - a failed or partial import can never silently drop user-owned records. - `service-worker.js`: shell cache bumped v5 -> v6 per base-path namespace so installed PWAs receive the migration code; activation purges obsolete v5 caches. - New `tests/ledger-portability.acceptance.mjs` browser gate: export round trip, merge import, future-version/malformed failure surfacing, root vs `/timmy-staging` storage isolation, and Delete Everything in both namespaces. Wired as `npm run test:portability` and added to CI quality gates. ## Acceptance criteria evidence - Import supports current and prior schema versions: v0 array + v1 envelope migrate forward (all tests RED-verified failing first). - Round-trip preserves confirmed values and provenance without secrets: round-trip + re-export convergence tests; smuggled tokens/session cookies asserted absent from exported text. - Malformed, oversized, and future-version imports fail safely: dedicated fail-safe tests plus browser-level checks that the local ledger stays untouched after each rejected import. ## Privacy / safety boundaries - No real personal or medical data anywhere: all fixtures synthetic. - Deterministic medical safety unchanged: urgent-flag detection, red-flag copy, escalation-before-chat untouched (full suite green). - User-owned local data preserved: merge-import, base-path namespace isolation, legacy-store compatibility, and Delete Everything guarantees covered by tests. - Live staging/systemd/Caddy/deployment untouched. ## Verification receipts (local run) - npm test: 78/78 pass - python3 tests/staging-deploy.test.py: 20/20 OK - test:ui / test:photo / test:sleek / test:portability: PASS - npm audit --audit-level=high: 0 vulnerabilities - npm run check:syntax (+ node --check on the new acceptance file): pass - bash scripts/check_diff.sh HEAD^..HEAD: clean

rockachopa added 1 commit 2026-08-22 20:40:57 +00:00

feat: version ledger migrations and hardened JSON portability

Quality gates / quality (pull_request) Successful in 1m42s

Details

b8532f587d

Implements #35.

- importLedger migrates prior schema versions (v0 bare-array legacy
  exports and the v1 envelope) and fails safely on future versions,
  malformed JSON, wrong-product envelopes, and oversized files with a
  new 2 MiB MAX_IMPORT_BYTES guard applied before parsing.
- exportLedger normalizes entries through sanitizeEntry so confirmed
  values and bounded provenance round-trip while smuggled secrets and
  unknown fields never enter the portable file.
- Entries may carry a whitelisted provenance origin ('user' or
  'ai-suggestion'); mergeVisualSuggestion records 'ai-suggestion' only
  when a suggestion is actually applied, keeping nonvisual fields
  user-owned.
- App import now merges into the existing ledger instead of replacing
  it, so a failed or partial import can never silently drop
  user-owned records.
- Service-worker shell cache bumped to v6 (per base-path namespace)
  so installed PWAs receive the migration code; old v5 caches are
  purged on activation.
- New tests/ledger-portability.acceptance.mjs browser gate covers
  export round trip, merge import, safe-failure surfacing, root vs
  /timmy-staging storage isolation, and Delete Everything for both
  namespaces; wired into package.json test:portability and CI quality.yml.

Deterministic medical safety unchanged: urgent-flag detection, red-flag
copy, and chat escalation paths are untouched; all fixtures synthetic.

rockachopa commented

2026-08-22 20:48:21 +00:00

Independent verification: CHANGES REQUESTED at b8532f587d8e2d8ee04b24173d7cde4305a4eaa4. Gitea CI is green, but three adversarial portability/privacy blockers reproduce:

Provenance allowlist bypass (src/domain.js:61-64). origin in KNOWN_PROVENANCE_ORIGINS includes inherited Object prototype keys. Origins toString, constructor, and __proto__ are all retained as provenance, contradicting the claimed strict vocabulary. Use an own-property-safe set/membership check and regressions for prototype names.
Timmy can reject its own valid exports. exportLedger() retains photoDataUrl, while import caps the entire document at 2 MiB. A synthetic valid JPEG data URL just over that limit exported successfully, then importLedger(exportLedger(...)) threw RangeError. The app accepts photos up to 4 MiB, so real user-owned exports can be non-importable. Establish a symmetric policy: either a cap compatible with the actual maximum export, or an explicit portable/photo-exclusion contract that preserves user expectations. Test at real image boundaries.
The byte limit is not a byte limit (src/domain.js:103). text.length counts UTF-16 code units, not UTF-8 bytes. A JSON payload of 1,200,087 JS characters measured 2,400,087 UTF-8 bytes, exceeded MAX_IMPORT_BYTES=2,097,152, and was accepted. Check File.size before File.text() in the browser and use a deterministic UTF-8 byte calculation in the domain boundary; test multibyte input.

Also harden repeated/colliding-ID merge semantics so re-import does not duplicate or ambiguously shadow user records. Do not merge until corrected and independently re-reviewed.

Independent verification: **CHANGES REQUESTED** at `b8532f587d8e2d8ee04b24173d7cde4305a4eaa4`. Gitea CI is green, but three adversarial portability/privacy blockers reproduce: 1. **Provenance allowlist bypass** (`src/domain.js:61-64`). `origin in KNOWN_PROVENANCE_ORIGINS` includes inherited Object prototype keys. Origins `toString`, `constructor`, and `__proto__` are all retained as provenance, contradicting the claimed strict vocabulary. Use an own-property-safe set/membership check and regressions for prototype names. 2. **Timmy can reject its own valid exports.** `exportLedger()` retains `photoDataUrl`, while import caps the entire document at 2 MiB. A synthetic valid JPEG data URL just over that limit exported successfully, then `importLedger(exportLedger(...))` threw `RangeError`. The app accepts photos up to 4 MiB, so real user-owned exports can be non-importable. Establish a symmetric policy: either a cap compatible with the actual maximum export, or an explicit portable/photo-exclusion contract that preserves user expectations. Test at real image boundaries. 3. **The byte limit is not a byte limit** (`src/domain.js:103`). `text.length` counts UTF-16 code units, not UTF-8 bytes. A JSON payload of 1,200,087 JS characters measured 2,400,087 UTF-8 bytes, exceeded `MAX_IMPORT_BYTES=2,097,152`, and was accepted. Check `File.size` before `File.text()` in the browser and use a deterministic UTF-8 byte calculation in the domain boundary; test multibyte input. Also harden repeated/colliding-ID merge semantics so re-import does not duplicate or ambiguously shadow user records. Do not merge until corrected and independently re-reviewed.

rockachopa commented

2026-08-22 20:54:27 +00:00

Supplemental independent review confirmed all four prior defects and found another data-loss blocker:

Base-path Delete Everything crosses namespaces. On /timmy-staging, deletion unconditionally removes global timmy-ledger-v1. With a staging ledger and separate root legacy data present, staging deletion removed both. Root scope may clean the legacy key to prevent resurrection, but a prefixed deployment must not delete another namespace’s data. Add a browser regression with nonempty root legacy data.

Also tighten malformed current-schema values: integer numeric fields, approved raster data URLs only, and safe invalid-date handling. The correction lane has been restarted at the provider’s sustainable concurrency with this complete finding set.

Supplemental independent review confirmed all four prior defects and found another data-loss blocker: 5. **Base-path Delete Everything crosses namespaces.** On `/timmy-staging`, deletion unconditionally removes global `timmy-ledger-v1`. With a staging ledger and separate root legacy data present, staging deletion removed both. Root scope may clean the legacy key to prevent resurrection, but a prefixed deployment must not delete another namespace’s data. Add a browser regression with nonempty root legacy data. Also tighten malformed current-schema values: integer numeric fields, approved raster data URLs only, and safe invalid-date handling. The correction lane has been restarted at the provider’s sustainable concurrency with this complete finding set.

rockachopa added 1 commit 2026-08-22 21:54:45 +00:00

fix: close hostile-review blockers in ledger portability

Quality gates / quality (pull_request) Failing after 1m28s

Details

dd86d6675d

- provenance origin set is own-safe exact membership (Set.has); inherited
  toString/constructor/__proto__ names can never become origins
- import/export symmetry restored with an explicit bounded policy:
  MAX_IMPORT_BYTES raised 2 MiB -> 16 MiB UTF-8 bytes, above any export
  this app can produce (photos capped at 4 MiB binary), so valid exports
  always re-import without silent data loss while hostile files stay bounded
- byte limit is byte-exact now: utf8ByteLength() measures real UTF-8 bytes
  (multibyte boundaries tested), and the browser rejects oversized files
  by File.size BEFORE File.text() reads user data
- collision-safe deterministic mergeLedgers(): existing user-owned rows
  win, incoming rows only ever added for new ids, intra-file duplicates
  collapse deterministically, every collision reported explicitly in the
  import toast (no duplicate/overwrite/shadow of user records)
- base-path Delete Everything is namespace-scoped: root still cleans/
  migrates the legacy store to prevent resurrection, /timmy-staging no
  longer erases another namespace's global legacy ledger (browser
  regression covers deletion with root legacy data present)
- strict current-schema values: Bristol 1-7 / urgency 0-4 / discomfort
  0-4 must be true integers (out-of-range falls back instead of silent
  clamping), photos restricted to JPEG/PNG/WebP base64 raster data URLs
  (SVG/GIF/non-base64 dropped), invalid dates never throw or persist
  Invalid Date values

Verification: npm test 91/91, test:ui/test:photo/test:sleek/test:portability
PASS, staging-deploy 20/20 OK, check:syntax clean, npm audit 0 high,
check_diff clean, adversarial probe battery (exact-byte boundary at cap,
prototype pollution via JSON, lone surrogates, data-URL strictness) green.

timmy requested changes 2026-08-22 22:14:08 +00:00

Dismissed

timmy left a comment

Independent exact-head hostile review of dd86d6675d5ff6f8e3e2675f36347469f7c7b927: CHANGES REQUESTED. Existing 91/91 + browser/deploy gates pass, and prior provenance/UTF-8/namespace/integer blockers are improved, but storage integrity still fails.

Import silently deletes a second existing local medical record when local storage already contains duplicate IDs; mergeLedgers dedupes local rows by dropping later records. Preserve every distinct local record, deterministically repair IDs, and never silently delete user-owned data.
Export/import symmetry fails through the UI. entries=result.merged occurs before durable saveEntries; quota failures leave in-memory entries accumulated while storage stays null. Three individually accepted imports produced a 16,778,688-byte export over the 16,777,216-byte import cap. Make save transactional/rollback-safe and enforce the portability ceiling before committing memory or storage.
Every top-level array is treated as legacy Timmy data. [{},7,{id:"",occurredAt:"bad"}] became three plausible Type-4 logs with generated IDs/current times. Require a defensible legacy shape/product contract; reject ambiguous unrelated JSON instead of inventing medical records.
Malformed existing localStorage is not sanitized on load; occurredAt:"not-a-date" blanked the app with RangeError: Invalid time value. Validate/migrate persisted data before render.
Raster validation checks only MIME/alphabet. Noncanonical abc, A, AAAAA, and SVG bytes mislabeled JPEG were accepted. Decode canonical base64, bound decoded bytes, and verify JPEG/PNG/WebP magic.
sanitizeEntry still throws on Symbol/BigInt dates despite the never-throw contract.

Exact hostile scripts and full receipts are preserved in review evidence. CI is red at this head. No merge/deploy until RED→GREEN correction and independent verification.

Independent exact-head hostile review of `dd86d6675d5ff6f8e3e2675f36347469f7c7b927`: **CHANGES REQUESTED**. Existing 91/91 + browser/deploy gates pass, and prior provenance/UTF-8/namespace/integer blockers are improved, but storage integrity still fails. 1. Import silently deletes a second existing local medical record when local storage already contains duplicate IDs; `mergeLedgers` dedupes local rows by dropping later records. Preserve every distinct local record, deterministically repair IDs, and never silently delete user-owned data. 2. Export/import symmetry fails through the UI. `entries=result.merged` occurs before durable `saveEntries`; quota failures leave in-memory entries accumulated while storage stays null. Three individually accepted imports produced a 16,778,688-byte export over the 16,777,216-byte import cap. Make save transactional/rollback-safe and enforce the portability ceiling before committing memory or storage. 3. Every top-level array is treated as legacy Timmy data. `[{},7,{id:"",occurredAt:"bad"}]` became three plausible Type-4 logs with generated IDs/current times. Require a defensible legacy shape/product contract; reject ambiguous unrelated JSON instead of inventing medical records. 4. Malformed existing localStorage is not sanitized on load; `occurredAt:"not-a-date"` blanked the app with `RangeError: Invalid time value`. Validate/migrate persisted data before render. 5. Raster validation checks only MIME/alphabet. Noncanonical `abc`, `A`, `AAAAA`, and SVG bytes mislabeled JPEG were accepted. Decode canonical base64, bound decoded bytes, and verify JPEG/PNG/WebP magic. 6. `sanitizeEntry` still throws on Symbol/BigInt dates despite the never-throw contract. Exact hostile scripts and full receipts are preserved in review evidence. CI is red at this head. No merge/deploy until RED→GREEN correction and independent verification.

rockachopa added 1 commit 2026-08-22 23:47:45 +00:00

fix: close second hostile-review round on ledger portability

Quality gates / quality (pull_request) Successful in 4m14s

Details

6f73b8551c

- collision-safe ID repair: duplicate ids inside stored data are repaired
  deterministically (first keeps id, twins get id#2, id#3, ... scanning past
  owned suffixes); every distinct local record survives, never dropped or
  silently merged; hostile id types (Symbol/BigInt/objects) repair onto
  fresh deterministic ids instead of throwing
- transactional import: parse+merge into a candidate ledger, persist first,
  then commit memory; quota/error rolls back in-memory state and localStorage
  together with explicit user feedback; total 16MiB portability budget
  enforced before mutation on export, import (post-migration expansion), and
  storage writes
- strict Timmy legacy contract for bare top-level arrays: nonempty array of
  plain rows each carrying a nonempty string id and integer Bristol 1-7;
  arbitrary unrelated arrays are rejected wholesale - no invented medical
  defaults from foreign JSON
- canonical raster photo validation: strict JPEG/PNG/WebP grammar, canonical
  base64 (linear scan, no regex on multi-MB strings), atob round-trip decode,
  declared-format magic bytes, 32B-4MiB decoded bounds; mislabeled SVG/HTML
  and noncanonical tiny junk are stripped while genuine photos survive
  byte-for-byte
- migrateStoredLedger: localStorage is validated and migrated before render;
  invalid dates become safe ISO timestamps, duplicate ids repaired, junk rows
  dropped (never fabricated into default records); healthy storage is
  byte-stable and never rewritten
- sanitizeEntry absorbs Symbol/BigInt/hostile dates/throwing toString,
  valueOf, getTime, toJSON without throwing; results stay serializable
- browser regression suite: quota rollback, pre-render migration, array
  rejection, photo contract, duplicate-ID preservation in the real app flow
- staging-health startup-rejection budget anchored to measured server
  cold-start instead of a fixed 800ms (fixes load-sensitive flake)

timmy requested changes 2026-08-22 23:57:48 +00:00

timmy left a comment

Exact-head hostile review of 6f73b8551c402e415916320d25c6a8cdf02ac618: REQUEST_CHANGES / do not merge. The corrected 49 domain tests and real ledger-portability browser journey pass, and prior local duplicate/quota/size/boot blockers improved. Three independent data-integrity failures remain:

Invalid stored medical dates are replaced with the live wall clock. Two calls to safeIsoTimestamp("not-a-real-date") 20 ms apart produced two different 2026-08-22 timestamps. Migration must not fabricate when a medical event occurred; preserve a stable explicit unknown/quarantined state or another non-invented deterministic contract that the UI can render.
Two distinct imported medical records with the same ID still lose one record. importLedger retained both notes, but mergeLedgers([], incoming) returned one row and skippedIds:["same"]. Collision-safe ID repair must preserve every distinct imported row too, not only pre-existing local duplicates.
Fake raster payloads still pass. Canonical base64 containing only JPEG/PNG/WebP signature bytes plus 600 bytes of junk was retained by sanitizeEntry for all three MIME types. Signature sniffing is not raster validity; require structurally decodable canonical images and reject truncation/junk.

These are user-owned medical-record integrity blockers despite green submitted tests. No merge or deployment.

Exact-head hostile review of `6f73b8551c402e415916320d25c6a8cdf02ac618`: **REQUEST_CHANGES / do not merge**. The corrected 49 domain tests and real ledger-portability browser journey pass, and prior local duplicate/quota/size/boot blockers improved. Three independent data-integrity failures remain: 1. Invalid stored medical dates are replaced with the live wall clock. Two calls to `safeIsoTimestamp("not-a-real-date")` 20 ms apart produced two different 2026-08-22 timestamps. Migration must not fabricate when a medical event occurred; preserve a stable explicit unknown/quarantined state or another non-invented deterministic contract that the UI can render. 2. Two distinct imported medical records with the same ID still lose one record. `importLedger` retained both notes, but `mergeLedgers([], incoming)` returned one row and `skippedIds:["same"]`. Collision-safe ID repair must preserve every distinct imported row too, not only pre-existing local duplicates. 3. Fake raster payloads still pass. Canonical base64 containing only JPEG/PNG/WebP signature bytes plus 600 bytes of junk was retained by `sanitizeEntry` for all three MIME types. Signature sniffing is not raster validity; require structurally decodable canonical images and reject truncation/junk. These are user-owned medical-record integrity blockers despite green submitted tests. No merge or deployment.

Quality gates / quality (pull_request) Successful in 4m14s

Details

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin timmy/35-version-ledger-migrations:timmy/35-version-ledger-migrations

git checkout timmy/35-version-ledger-migrations

Merge

Merge the changes and update on Gitea.

git checkout main

git merge --no-ff timmy/35-version-ledger-migrations

git checkout main

git merge --ff-only timmy/35-version-ledger-migrations

git checkout timmy/35-version-ledger-migrations

git rebase main

git checkout main

git merge --no-ff timmy/35-version-ledger-migrations

git checkout main

git merge --squash timmy/35-version-ledger-migrations

git checkout main

git merge --ff-only timmy/35-version-ledger-migrations

git checkout main

git merge timmy/35-version-ledger-migrations

git push origin main

Sign in to join this conversation.

No reviewers

No Milestone

No project

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#61