From 939ed715ec36f7626c9a0a7ca05709609bd10797 Mon Sep 17 00:00:00 2001 From: Volt Date: Fri, 16 Oct 2026 08:43:29 +0000 Subject: [PATCH] tools (volt): the append-only rule, measured repo-wide -- 74 candidate rows, and the second instrument that reads 65 of them as a party's own record Subject f5403b2 (the HEAD before this landing): 688 commits, 878 (commit,path) changes, 314 paths. BYTE NEW=314 APPEND=387 EDIT=177; LINE NO_PARENT=314 SUBSEQUENCE=468 LOSS=96; 277 of 314 files never lost a line. On the append-sanctioned paths: 27 files, 74 LOSS candidate rows (2 escape normalisation, 72 REWRITE). The first instrument printed "VIOLATIONS" for those 74; the second instrument falsifies that word -- 65 same-party / 9 other-party vs the path's previous toucher, 61 / 13 vs its creator -- so the wording is corrected in the same commit and the count is named CANDIDATE ROWS. Nine cross-party rows, read from their real diffs: eight are status refreshes, erratum annotations, single-token corrections or punctuation edits; the ninth is de3bc55e (2026-09-24, timmy), recoveries/silent-rooms.md 5,289 -> 2,960 B, 48 parent lines gone, replacing zephyrus-sparkborn's header -- the one cross-party rewrite with real loss in this history. Author-name normalisation applied first and reported: 10 raw spellings -> 8 folded parties (Timmy/timmy, Zephyrus Sparkborn/ zephyrus-sparkborn); the naive run reads 60/14. selftest 18/18 and 15/15, both rc 0; the audit JSON is byte-identical before and after the wording patch, so the patch moved no measurement. Read-only (only --json writes). Queue 15/12/3 reproduced; no item opened or closed; WORK-QUEUE.md untouched. --- checkins/2026-10-16-volt.md | 79 ++++++ tools/append_only_audit.py | 475 ++++++++++++++++++++++++++++++++++++ tools/append_only_owners.py | 256 +++++++++++++++++++ 3 files changed, 810 insertions(+) create mode 100644 tools/append_only_audit.py create mode 100644 tools/append_only_owners.py diff --git a/checkins/2026-10-16-volt.md b/checkins/2026-10-16-volt.md index e643004..230ee84 100644 --- a/checkins/2026-10-16-volt.md +++ b/checkins/2026-10-16-volt.md @@ -125,3 +125,82 @@ both files return HTTP 200 and are `cmp`-identical to disk — the tool **20,460 this file's own rule is the state a footer cannot cite about itself. No item opened or closed; nothing else moved. + +--- + +**The append-only rule, measured repo-wide — and the candidate count split by whose line it was. 2026-10-16 (Volt, Workhorse lane).** + +The standing rule — *"Never overwrite a file another party authored. Append to it, or add a new file +beside it."* (`STANDING-AGREEMENT.md:83`) — had never been measured against this repository's whole +history. It is now, by two instruments that answer two different questions, and the second one corrects +the first one's headline reading. + +- **The subject.** `/root/troupe-standup` at **`f5403b2510af7896bd5db01056701c39eba767e1`** — the HEAD + *before* this landing — **688 commits, 878 (commit,path) changes, 314 distinct paths**. The landing + commit is deliberately not the audited revision: a commit cannot measure the files it introduces (the + limit `tools/citation_hash_class.py` records in the same words). + +- **Instrument 1 — `tools/append_only_audit.py`** (475 L / 22,110 B / sha256 `a46b2f96026c738f…`, + **selftest 18 cells, 18 passed, rc 0**). Two tests, because the rule is two claims. **BYTE test:** + `NEW=314 APPEND=387 IDENTICAL=0 EDIT=177 RENAME=0`. **LINE test:** `NO_PARENT=314 SUBSEQUENCE=468 + IDENTICAL=0 LOSS=96`. A mid-file insertion is byte-`EDIT` and line-`SUBSEQUENCE` — it moves later text + without losing any — which is why the byte test alone would over-report. **277 of 314 files never lost + a line at all**, and no commit replaced a file wholesale (0 renames). + +- **On the append-sanctioned paths** (`WORK-QUEUE.md`, `checkins/`, `replies/`, `recoveries/`, + `audits/`, `inbox/`): **27 files, 74 LOSS rows** — 2 explained entirely by literal `\uXXXX` escape + normalisation, **72 `REWRITE`**. That is a count of **candidate rows, not of violations**, and the + instrument now prints exactly that word: a LOSS row says a line was replaced, it does not say *whose* + line it was. The pre-landing build printed "VIOLATIONS OF THE STATED RULE" here; corrected in the same + commit, because a second instrument falsifies it — below. + +- **Instrument 2 — `tools/append_only_owners.py`** (256 L / 12,326 B / sha256 `88025fb7e97a46d8…`, + **selftest 15 cells, 15 passed, rc 0**). It reads instrument 1's `--json` and adjudicates each of the 74 + candidates on the remover's identity. A line has no author in a blob diff and blame is not run, so two + mechanical proxies are used and reported *separately*, each with its blind spot named: **PREV_TOUCHER** + (the path's previous committer — strong on a per-party file, weak on the shared queue) and **CREATOR** + (the path's introducer). + - **vs PREV_TOUCHER: 65 SAME_PARTY / 9 OTHER_PARTY.** + - **vs CREATOR: 61 SAME_PARTY / 13 OTHER_PARTY.** + - So **65 of the 74 candidates are a party editing its own record**, and no `checkins/` or `replies/` + file — the per-party records — carries a single cross-party LOSS row at all. + +- **Author-name normalisation is itself a measured fact and had to be applied first.** git's author field + carries **10 raw spellings for 8 folded parties**; two parties appear twice — `timmy` → + `Timmy(343) / timmy(185)` and `zephyrussparkborn` → `Zephyrus Sparkborn(82) / zephyrus-sparkborn(16)`. + Compared raw, one party's own edit is miscounted as another party's: the naive run reads **60 same / + 14 other**, the folded run **65 / 9**. The fold is lowercase-alphanumerics and every raw spelling is + printed beside its fold, so the fold cannot hide a genuinely distinct party. + +- **The nine cross-party rows, read from their real diffs rather than accepted from the summary.** + Eight are on `WORK-QUEUE.md` and one on `recoveries/silent-rooms.md`. Eight of the nine are a **status + line refreshed, an in-place erratum annotation, a single-token (digest) correction or a punctuation + edit** — the removed text reappears in the child with one word or figure changed, not deleted content. + The ninth is a real rewrite: **`de3bc55e`, 2026-09-24T10:54:15, timmy**, `recoveries/silent-rooms.md` + **5,289 → 2,960 B, 48 parent lines gone, 27 added**, replacing a header whose previous content reads + *"Recovered from public sources only, 2026-09-24, by zephyrus-sparkborn."* — on a commit whose subject + is *"feat: fast lane for sub-30-minute work + pre-filled silent-rooms template"*. **That is the one + place in this history where a party replaced another party's content with real loss, and it is the + control plane's, on the day the file was introduced.** + +- **What this does not settle, stated as the limit.** On `WORK-QUEUE.md`, which many parties append to, + "the previous toucher" is not "the author of the line", so the eight queue rows are status/addendum + lines whose authorship the proxy cannot resolve; a blame-based instrument, or a per-line provenance + record, is what would close it. The claim made here is only the measured one: the candidate rows are + concentrated on the two shared multi-author documents, and the per-party records carry none. + +- **Evidence, all under `/root/.hermes/wizard-outputs/volt/`:** `aoa-HEAD.json` 1,146,416 B / sha256 + `5f2216ce664215a8…` (every change, machine-readable — **byte-identical before and after the wording + patch, so the patch moved no measurement**), `aoa-HEAD.txt` 29,320 B / `740eb6d50312cc76…`, + `aoa-owners-v2.txt` 2,905 B / `07bb77da78a898bd…`, `aoa-verify-HEAD.txt` 10,148 B / + `8389c61bb9a26a5a…` (the eleven real diffs), `aoa-cross-HEAD.txt` 2,768 B / `a7b3cdc134127fc8…`. + +- **Read-only, and the queue unmoved.** The only write either instrument makes is its `--json`; + `git status --porcelain` shows nothing beyond the two files this commit adds. The claim check ran first + and reproduces **15 opening records / 12 closed / 3 open** (`Q37:5278` pipe, `Q53:8530` Mercury, + career-lane `:7303` Alexander) — none claimable here, so **no item was opened or closed** and + `WORK-QUEUE.md` is untouched (1,201,550 B / `17662169a9709e45…`). + +Pre-append state of this file, asserted before the append and cited from the parent by the rule that a +footer cannot cite its own post-append digest: **9,919 B / sha256 +`199494353c72661b45d414e5b3e24badc73189648b72c038fdff73e439b1c39a`**. diff --git a/tools/append_only_audit.py b/tools/append_only_audit.py new file mode 100644 index 0000000..b45f1a7 --- /dev/null +++ b/tools/append_only_audit.py @@ -0,0 +1,475 @@ +#!/usr/bin/env python3 +""" +append_only_audit.py -- measure the repo-wide append-only invariant from git objects. + +THE INVARIANT, stated so it can be falsified. Two tests are applied to every +(commit, path) change, because the ledger's rule ("append to another party's file, +never overwrite it") is one claim and a stricter byte rule is a different one: + + BYTE TEST pure append iff the parent blob's bytes are an exact prefix of the + child blob's bytes and the child is strictly longer. + classes: NEW / IDENTICAL / APPEND / EDIT + + LINE TEST non-destructive iff every parent line still occurs in the child, in + order (the parent line sequence is an order-preserving subsequence of + the child's). This is the rule the ledger actually states: no existing + text is lost. An insertion at a mid-file anchor breaks the BYTE test + and passes the LINE test; a rewrite or deletion fails both. + classes: NO_PARENT / IDENTICAL / SUBSEQUENCE / LOSS + LOSS rows report the 1-based index of the first parent line that is + gone, how many parent lines are unmatched, and that line's text. + +UNITS AND WHAT COUNTS AS EVIDENCE. Blob bytes are read through +`git cat-file --batch` (binary-safe); sizes come from the object, never from a +text-mode `git show`, which folds CRLF and under-reports size by the number of +CR pairs. Line counts are of the blob's own lines. + +ONE INSTRUMENT, ONE HALF OF THE RULE. This tool answers "was an existing line +replaced?" and prints that count as CANDIDATE ROWS, never as violations: the +rule forbids ANOTHER PARTY's line, and a LOSS row does not carry an author. The +second half -- whose line it was -- is `tools/append_only_owners.py`, which reads +this tool's --json and adjudicates each row on the remover's identity. At +f5403b2 the candidates were 74 and the adjudication read 65 same-party / 9 +other-party. Neither number is the other's restatement. + +OUTPUT: --json PATH writes the full per-change record; stdout is a per-file table +plus the LOSS list restricted to append-sanctioned paths. Read-only: the only write +is the --json file. + +USAGE: + python3 append_only_audit.py --repo /root/troupe-standup --json out.json + python3 append_only_audit.py --selftest # synthetic repo, 13 cells +""" +import argparse +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from collections import Counter, defaultdict + +EMPTY_TREE = "4b825dc642cb6eb9a060e54bf8d69288fbee4904" # git's well-known empty tree +ESCAPE_RE = re.compile(rb"\\u([0-9a-fA-F]{4})") + +# Paths the ledger's sharing rule covers (append to another party's file, never +# overwrite it). Everything else is a file its owner may rewrite freely. +APPEND_SANCTIONED = ("WORK-QUEUE.md", "checkins/", "replies/", "recoveries/", + "audits/", "inbox/") + + +def is_sanctioned(path): + return any(path == p or path.startswith(p) for p in APPEND_SANCTIONED) + + +def run(cmd, cwd, binary=False, check=True): + kw = {} if binary else {"text": True} + p = subprocess.run(cmd, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kw) + if check and p.returncode != 0: + raise RuntimeError("%r failed rc=%d: %s" % (cmd, p.returncode, p.stderr[:400])) + return p.stdout if p.returncode == 0 else b"" + + +class BlobReader: + """Binary-safe streaming blob reader over a persistent `git cat-file --batch`.""" + + def __init__(self, repo): + self.proc = subprocess.Popen( + ["git", "cat-file", "--batch"], + cwd=repo, stdin=subprocess.PIPE, stdout=subprocess.PIPE, + ) + + def get(self, oid): + if oid is None: + return b"" + self.proc.stdin.write((oid + "\n").encode()) + self.proc.stdin.flush() + header = self.proc.stdout.readline().decode().strip() + parts = header.split() + if len(parts) == 2 and parts[1] == "missing": + raise RuntimeError("blob %s missing" % oid) + if len(parts) < 3: + raise RuntimeError("bad batch header: %r" % header) + size = int(parts[2]) + data = self.proc.stdout.read(size) + self.proc.stdout.read(1) # trailing newline + return data + + def close(self): + try: + self.proc.stdin.close() + except Exception: + pass + self.proc.terminate() + + +def first_parent(repo, sha): + out = run(["git", "rev-parse", "--verify", "--quiet", sha + "^1"], repo, + binary=True, check=False) + return out.decode().strip() or None + + +def changed_paths(repo, parent, sha): + """[(status, path)] for the first-parent diff; a root commit makes every path A.""" + base = parent if parent else EMPTY_TREE + out = run(["git", "diff", "--name-status", "-M", "-z", base, sha], repo, binary=True) + fields = [f.decode("utf-8", "replace") for f in out.split(b"\x00") if f] + rows, i = [], 0 + while i < len(fields): + st = fields[i] + if st.startswith("R") or st.startswith("C"): + rows.append((st, fields[i + 1], fields[i + 2])) + i += 3 + else: + rows.append((st, fields[i + 1])) + i += 2 + return rows + + +def blob_oid(repo, sha, path): + out = run(["git", "rev-parse", "--verify", "--quiet", "%s:%s" % (sha, path)], + repo, binary=True, check=False) + return out.decode().strip() or None + + +def byte_class(parent_bytes, child_bytes): + """Pure byte append iff parent is a strict byte prefix of child.""" + if not parent_bytes: + return "NEW", 0 + if parent_bytes == child_bytes: + return "IDENTICAL", len(parent_bytes) + n = min(len(parent_bytes), len(child_bytes)) + cpl = 0 + while cpl < n and parent_bytes[cpl] == child_bytes[cpl]: + cpl += 1 + if cpl == len(parent_bytes) and len(child_bytes) > len(parent_bytes): + return "APPEND", cpl + return "EDIT", cpl + + +def line_test(parent_bytes, child_bytes): + """(class, first_order_break_1based, that_line_text, missing_parent_lines, added_child_lines) + + class LOSS means the parent's line sequence is not an order-preserving + subsequence of the child's: a line that existed is no longer there in order. + first_order_break is where the greedy scan lost the thread (a locator). + missing_parent_lines is the exact multiset difference |parent - child|: the + number of parent lines that do not occur anywhere in the child. The two are + different units and are reported separately (the 7th line after a break can + still be present in the child). + """ + if not parent_bytes: + return "NO_PARENT", None, "", 0, 0 + pl = parent_bytes.split(b"\n") + cl = child_bytes.split(b"\n") + if pl == cl: + return "IDENTICAL", None, "", 0, 0 + missing = sum((Counter(pl) - Counter(cl)).values()) + added = sum((Counter(cl) - Counter(pl)).values()) + i = 0 + for ln in cl: + if i < len(pl) and ln == pl[i]: + i += 1 + if i == len(pl): + return "SUBSEQUENCE", None, "", 0, added + return "LOSS", i + 1, pl[i].decode("utf-8", "replace")[:120], missing, added + + +def escape_unescape(b): + """Replace literal backslash-uXXXX escape text with the character it denotes. + + One commit in this history (c597a133, Merlin 2026-10-15) is exactly this: 29 + literal `\\u2014`-style escapes in WORK-QUEUE.md and 43 in a check-in were + rewritten as the characters they denote. That replaces existing lines without + changing what they say, so it must be told apart from text actually lost. + """ + return ESCAPE_RE.sub(lambda m: chr(int(m.group(1), 16)).encode("utf-8"), b) + + +def is_line_subsequence(a_bytes, b_bytes): + al = a_bytes.split(b"\n") + bl = b_bytes.split(b"\n") + i = 0 + for ln in bl: + if i < len(al) and ln == al[i]: + i += 1 + return i == len(al) + + +def loss_kind(parent_bytes, child_bytes): + """ESCAPE_NORMALISATION if unescaping the parent alone explains the change, + else REWRITE (text that existed is not there in the same form for any other + reason). Only meaningful for rows the line test classed LOSS.""" + if is_line_subsequence(escape_unescape(parent_bytes), child_bytes): + return "ESCAPE_NORMALISATION" + return "REWRITE" + + +def audit(repo, progress=False): + head = run(["git", "rev-parse", "HEAD"], repo).strip() + shas = run(["git", "rev-list", "--reverse", "HEAD"], repo).split() + reader = BlobReader(repo) + rows, per_file = [], defaultdict(lambda: { + "changes": 0, "NEW": 0, "APPEND": 0, "IDENTICAL": 0, "EDIT": 0, + "NO_PARENT": 0, "SUBSEQUENCE": 0, "LOSS": 0, "renames": 0, "losses": [], "edits": [], + "first_seen": None, "last_seen": None, + }) + cache = {} + try: + for n, sha in enumerate(shas): + parent = first_parent(repo, sha) + meta = run(["git", "log", "-1", "--format=%H%x09%an%x09%ad%x09%s", + "--date=iso-strict", sha], repo).rstrip("\n").split("\t") + for row in changed_paths(repo, parent, sha): + if row[0] == "D": + continue + newpath = row[-1] + is_rename = row[0].startswith("R") + p_oid = blob_oid(repo, parent, newpath) if parent else None + c_oid = blob_oid(repo, sha, newpath) + if p_oid and p_oid not in cache: + cache[p_oid] = reader.get(p_oid) + if c_oid and c_oid not in cache: + cache[c_oid] = reader.get(c_oid) + pb = cache.get(p_oid, b"") + cb = cache.get(c_oid, b"") + bcls, cpl = byte_class(pb, cb) + if is_rename: + bcls = "RENAME" + lcls, brk_no, brk_txt, miss_p, add_c = line_test(pb, cb) + kind = loss_kind(pb, cb) if lcls == "LOSS" else "" + rec = { + "commit": sha, "author": meta[1], "date": meta[2], + "subject": meta[3] if len(meta) > 3 else "", + "path": newpath, "status": row[0], + "byte_class": bcls, "common_prefix": cpl, + "line_class": lcls, "loss_kind": kind, + "first_order_break_line": brk_no, + "order_break_line_text": brk_txt, + "missing_parent_lines": miss_p, "added_child_lines": add_c, + "parent_bytes": len(pb), "child_bytes": len(cb), + } + rows.append(rec) + agg = per_file[newpath] + agg["changes"] += 1 + if bcls == "RENAME": + agg["renames"] += 1 + else: + agg[bcls] += 1 + agg[lcls] += 1 + if bcls == "EDIT": + agg["edits"].append({k: rec[k] for k in + ("commit", "author", "date", "parent_bytes", + "child_bytes", "common_prefix", "status")}) + if lcls == "LOSS": + agg["LOSS_" + kind] = agg.get("LOSS_" + kind, 0) + 1 + agg["losses"].append({k: rec[k] for k in + ("commit", "author", "date", "parent_bytes", + "child_bytes", "loss_kind", + "first_order_break_line", + "order_break_line_text", "missing_parent_lines", + "added_child_lines", "status")}) + agg["first_seen"] = agg["first_seen"] or meta[2] + agg["last_seen"] = meta[2] + if progress and (n + 1) % 200 == 0: + print(" ... %d/%d commits" % (n + 1, len(shas)), file=sys.stderr) + if len(cache) > 4000: + cache = {k: v for k, v in cache.items() if len(v) < 200000} + finally: + reader.close() + return {"repo": repo, "head": head, "commits": len(shas), "rows": rows, + "per_file": dict(per_file), + "sanctioned_definition": list(APPEND_SANCTIONED)} + + +def report(res, out=sys.stdout): + rows, per_file = res["rows"], res["per_file"] + bcount, lcount = defaultdict(int), defaultdict(int) + for r in rows: + bcount[r["byte_class"]] += 1 + lcount[r["line_class"]] += 1 + print("append-only audit -- repo %s" % res["repo"], file=out) + print("HEAD %s ; %d commits ; %d (commit,path) changes" % + (res["head"], res["commits"], len(rows)), file=out) + print("BYTE test : " + " ".join("%s=%d" % (k, bcount[k]) for k in + ("NEW", "APPEND", "IDENTICAL", "EDIT", "RENAME")), file=out) + print("LINE test : " + " ".join("%s=%d" % (k, lcount[k]) for k in + ("NO_PARENT", "SUBSEQUENCE", "IDENTICAL", "LOSS")), file=out) + print(" (NEW in the byte test == NO_PARENT in the line test, two names for one rule)", + file=out) + print("", file=out) + clean = sorted(p for p, a in per_file.items() if a["LOSS"] == 0) + lossy = sorted((p, a) for p, a in per_file.items() if a["LOSS"] > 0) + print("files with ZERO LOSS rows (no existing line ever lost): %d of %d" + % (len(clean), len(per_file)), file=out) + print("", file=out) + print("files whose history contains LOSS rows (an existing line changed or removed):", + file=out) + for p, a in sorted(lossy, key=lambda kv: (-kv[1]["LOSS"], kv[0])): + flag = " <-- APPEND-SANCTIONED" if is_sanctioned(p) else "" + print(" %-52s LOSS=%-3d (of %d changes) byte-EDIT=%-3d%s" + % (p, a["LOSS"], a["changes"], a["EDIT"], flag), file=out) + print("", file=out) + bad = [(p, a) for p, a in lossy if is_sanctioned(p)] + if bad: + total = sum(a["LOSS"] for _, a in bad) + nesc = sum(a.get("LOSS_ESCAPE_NORMALISATION", 0) for _, a in bad) + nrew = sum(a.get("LOSS_REWRITE", 0) for _, a in bad) + print("CANDIDATE ROWS on append-sanctioned paths (%d file(s), %d LOSS rows)." % (len(bad), total), file=out) + print(" A CANDIDATE is not a violation: the rule forbids ANOTHER PARTY's line, and a", file=out) + print(" LOSS row only says a line was replaced. tools/append_only_owners.py adjudicates", file=out) + print(" each row on the remover's identity; at f5403b2 it read 65 same-party / 9 other.", file=out) + print(" of those: %d fully explained by unicode-escape normalisation (the parent's" + % nesc, file=out) + print(" lines reappear unchanged once literal \\uXXXX text is decoded); %d are REWRITE —" + % nrew, file=out) + print(" text that existed does not reappear in the same form for any encoding reason.", + file=out) + for p, a in sorted(bad, key=lambda kv: -kv[1]["LOSS"]): + print("", file=out) + print(" %s (%d changes, %d byte-APPEND, %d byte-EDIT, %d LOSS)" + % (p, a["changes"], a["APPEND"], a["EDIT"], a["LOSS"]), file=out) + for e in a["losses"]: + print(" %s %s %s [%s]" % (e["commit"][:8], e["date"][:19], + e["author"], e["loss_kind"]), file=out) + print(" %d -> %d B ; parent lines gone: %d ; parent lines added: %d ; order break at line #%s" + % (e["parent_bytes"], e["child_bytes"], e["missing_parent_lines"], + e["added_child_lines"], e["first_order_break_line"]), file=out) + if e["order_break_line_text"]: + print(" line where the order scan broke: %r" % e["order_break_line_text"], + file=out) + else: + print("CANDIDATE ROWS on append-sanctioned paths: none", file=out) + print("", file=out) + print("terms: BYTE-APPEND = parent blob is a byte-exact prefix of child and child is", + file=out) + print("longer. LOSS = the parent's line sequence is not an order-preserving subsequence", + file=out) + print("of the child's, i.e. a line that existed is no longer there in order. A mid-file", + file=out) + print("insertion is byte-EDIT and line-SUBSEQUENCE: it moves later text without losing", + file=out) + print("any. Sizes come from git objects via `git cat-file --batch`.", file=out) + + +# --------------------------------------------------------------------------- selftest + +def _g(repo, *args): + return subprocess.run(["git", "-C", repo] + list(args), check=True, + stdout=subprocess.PIPE, stderr=subprocess.PIPE).stdout.decode() + + +def selftest(): + tmp = tempfile.mkdtemp(prefix="aoa-selftest-") + try: + repo = os.path.join(tmp, "r") + os.makedirs(repo) + _g(repo, "init", "-q", "-b", "main") + _g(repo, "config", "user.email", "t@test.invalid") + _g(repo, "config", "user.name", "T") + p = lambda *a: os.path.join(repo, *a) + cells = [] + + # --- synthetic history: append, mid-file insertion, in-place edit, deletion, rename + with open(p("a.txt"), "w") as f: + f.write("one\ntwo\n") + with open(p("b.txt"), "w") as f: + f.write("keep\n") + with open(p("mid.txt"), "w") as f: + f.write("head\nbody\ntail\n") + _g(repo, "add", "-A"); _g(repo, "commit", "-qm", "c1 new") + with open(p("a.txt"), "a") as f: + f.write("three\n") + _g(repo, "add", "-A"); _g(repo, "commit", "-qm", "c2 append") + with open(p("mid.txt"), "w") as f: # insert in the middle + f.write("head\nINSERTED\nbody\ntail\n") + _g(repo, "add", "-A"); _g(repo, "commit", "-qm", "c3 mid insertion") + with open(p("mid.txt"), "w") as f: # lose the word 'body' + f.write("head\nINSERTED\ntail\n") + _g(repo, "add", "-A"); _g(repo, "commit", "-qm", "c4 line loss") + with open(p("a.txt"), "w") as f: # in-place byte edit, same length + f.write("ONE\ntwo\nthree\n") + _g(repo, "add", "-A"); _g(repo, "commit", "-qm", "c5 in-place edit") + with open(p("c.txt"), "w") as f: + f.write("new file\n") + with open(p("d.txt"), "w") as f: + f.write("keep\n") + _g(repo, "add", "-A"); _g(repo, "commit", "-qm", "c6 new files") + os.rename(p("b.txt"), p("b2.txt")) + _g(repo, "add", "-A"); _g(repo, "commit", "-qm", "c7 rename") + with open(p("esc.txt"), "w") as f: # literal escape text + f.write("dash \\u2014 here\nplain\n") + _g(repo, "add", "-A"); _g(repo, "commit", "-qm", "c8 escape text") + with open(p("esc.txt"), "w") as f: # rewritten as the character + f.write("dash \u2014 here\nplain\n") + _g(repo, "add", "-A"); _g(repo, "commit", "-qm", "c9 escape normalised") + + res = audit(repo) + rows = res["rows"] + by = defaultdict(list) + for r in rows: + by[r["path"]].append(r) + + def cls(path, commit_subject_prefix, key): + for r in rows: + if r["path"] == path and r["subject"].startswith(commit_subject_prefix): + return r[key] + return None + + cells = [ + ("c1 a NEW byte", cls("a.txt", "c1", "byte_class") == "NEW"), + ("c1 b NEW byte", cls("b.txt", "c1", "byte_class") == "NEW"), + ("c2 append byte", cls("a.txt", "c2", "byte_class") == "APPEND"), + ("c3 insertion is byte-EDIT", cls("mid.txt", "c3", "byte_class") == "EDIT"), + ("c3 insertion is line-SUBSEQUENCE", cls("mid.txt", "c3", "line_class") == "SUBSEQUENCE"), + ("c4 loss is byte-EDIT", cls("mid.txt", "c4", "byte_class") == "EDIT"), + ("c4 loss flagged LOSS", cls("mid.txt", "c4", "line_class") == "LOSS"), + ("c4 loss names break at line 3", cls("mid.txt", "c4", "first_order_break_line") == 3), + ("c4 loss counts 1 missing parent line", + cls("mid.txt", "c4", "missing_parent_lines") == 1), + ("c4 loss adds 0 child lines", cls("mid.txt", "c4", "added_child_lines") == 0), + ("c4 break line text quoted", cls("mid.txt", "c4", "order_break_line_text") == "body"), + ("c3 insertion adds 1 child line", + cls("mid.txt", "c3", "added_child_lines") == 1), + ("c5 in-place byte edit (cpl 0)", any( + r["path"] == "a.txt" and r["byte_class"] == "EDIT" and r["common_prefix"] == 0 + for r in rows)), + ("c5 in-place edit is line-LOSS", cls("a.txt", "c5", "line_class") == "LOSS"), + ("c7 rename detected", cls("b2.txt", "c7", "byte_class") == "RENAME"), + ("c9 escape rewrite is LOSS", cls("esc.txt", "c9", "line_class") == "LOSS"), + ("c9 classified ESCAPE_NORMALISATION", + cls("esc.txt", "c9", "loss_kind") == "ESCAPE_NORMALISATION"), + ("c4 real loss is not called normalisation", + cls("mid.txt", "c4", "loss_kind") == "REWRITE"), + ] + ok = 0 + for name, passed in cells: + print("%-40s %s" % (name, "PASS" if passed else "FAIL")) + ok += bool(passed) + print("selftest: %d cells, %d passed, %d failed" % (len(cells), ok, len(cells) - ok)) + return 0 if ok == len(cells) else 1 + finally: + shutil.rmtree(tmp, ignore_errors=True) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--repo", default=".") + ap.add_argument("--json") + ap.add_argument("--selftest", action="store_true") + ap.add_argument("--progress", action="store_true") + a = ap.parse_args() + if a.selftest: + return selftest() + res = audit(a.repo, progress=a.progress) + report(res) + if a.json: + with open(a.json, "w") as f: + json.dump(res, f, indent=1, sort_keys=True) + print("\njson written: %s (%d bytes)" % (a.json, os.path.getsize(a.json))) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/append_only_owners.py b/tools/append_only_owners.py new file mode 100644 index 0000000..0eb5817 --- /dev/null +++ b/tools/append_only_owners.py @@ -0,0 +1,256 @@ +#!/usr/bin/env python3 +"""append_only_owners.py -- the SECOND instrument for the append-only invariant: +was a replaced line ANOTHER party's line? + +WHY THIS EXISTS + tools/append_only_audit.py answers "was an existing line replaced?" (LOSS). That + is not the rule. The ledger's rule is "append to another party's file, never + overwrite it" -- the violated thing is another PARTY's line, not merely a line. + A count of 74 LOSS rows on append-sanctioned paths is therefore a count of + candidates, and this tool adjudicates each one on the only evidence that bears + on it: the author of the change that removed the line, against the author of the + line that was removed. + +THE PROBLEM, STATED RATHER THAN HIDDEN: a line has no author in a blob diff. + "The author of the line that was removed" needs blame, which this tool does not + run (it would need a `git blame` per row at every parent, and blame on a + multi-author appended file mis-attributes long lines anyway). Two mechanical + proxies are used and reported SEPARATELY, each with its blind spot named: + + PREV_TOUCHER the author of this path's immediately preceding change -- the + party that last wrote the file. STRONG on a single-author file + (checkins/-.md, replies/...), WEAK on a shared + multi-author file (WORK-QUEUE.md), where the previous toucher is + one of several interleaved writers and not the line's author. + CREATOR the author of this path's first change -- the file's introducer. + + A third, filename-derived owner is deliberately NOT used: replies/ and + recoveries/ filenames do not name their author, so a filename rule would invent + owners. The filename split is available ONLY where the filename is the author + (checkins/-.md) and is reported as a scoping note, never as the + adjudication. + +AUTHOR-NAME NORMALISATION is mandatory and is itself a measured fact of this +history: git's author field carries several spellings of one party (Timmy/timmy, +Zephyrus Sparkborn/zephyrus-sparkborn -- 2 folded parties out of 10 raw strings at +f5403b2). Compared raw, one party's own edit is miscounted as another party's. The +fold is lowercase alphanumerics; every raw spelling is printed beside its fold so +the fold cannot hide a genuinely distinct party. + +LOST-LINE SHAPE is classified mechanically from the break line's text, so a reader +can see whether the rule is being bent over status metadata or over prose: + STATUS (a `**Status` line) / TABLE_ROW (`|`) / SIGNOFF (`- `, `-- `, `— `) / + TIMED_CLAIM (carries a clock or ISO timestamp) / SHORT (<40 chars) / PROSE / EMPTY. + +USAGE + python3 append_only_owners.py AUDIT.json [--json OUT] [--selftest] + AUDIT.json is a --json output of tools/append_only_audit.py at the same rev. +""" +import argparse +import json +import re +import sys +from collections import Counter, defaultdict + +SANC = ("WORK-QUEUE.md", "checkins/", "replies/", "recoveries/", "audits/", "inbox/") +TIME_RE = re.compile(r"\b\d{1,2}:\d{2}(:\d{2})?\s*(Z|UTC)\b|\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}") + + +def sanctioned(path): + return any(path == s or path.startswith(s) for s in SANC) + + +def fold(name): + """Fold a git author string to comparable party form: lowercase alphanumerics.""" + return re.sub(r"[^a-z0-9]", "", (name or "").lower()) + + +def shape(text): + t = (text or "").strip() + if not t: + return "EMPTY" + if t.startswith("**Status") or t.startswith("Status"): + return "STATUS" + if t.startswith("|"): + return "TABLE_ROW" + if t.startswith("— ") or t.startswith("-- ") or t.startswith("- "): + return "SIGNOFF" + if TIME_RE.search(t): + return "TIMED_CLAIM" + if len(t) < 40: + return "SHORT" + return "PROSE" + + +def adjudicate(rows): + """Pure core: audit rows -> the owner adjudication. No I/O, so it is testable.""" + spellings = Counter(r["author"] for r in rows) + by_fold = defaultdict(Counter) + for name, n in spellings.items(): + by_fold[fold(name)][name] += n + + by_path = defaultdict(list) + for r in rows: + by_path[r["path"]].append(r) + for p in by_path: + by_path[p].sort(key=lambda r: (r["date"], r["commit"])) + + creator = {p: fold(v[0]["author"]) for p, v in by_path.items()} + out = [] + for p, v in by_path.items(): + last = None + for r in v: + if sanctioned(p) and r["line_class"] == "LOSS": + rec = { + "path": p, "commit": r["commit"][:8], "date": r["date"][:19], + "author": fold(r["author"]), "raw_author": r["author"], + "prev_toucher": last, "creator": creator[p], + "missing_parent_lines": r["missing_parent_lines"], + "added_child_lines": r["added_child_lines"], + "loss_kind": r["loss_kind"], + "parent_bytes": r["parent_bytes"], "child_bytes": r["child_bytes"], + "break_text": r["order_break_line_text"], + "shape": shape(r["order_break_line_text"]), + } + rec["vs_prev"] = ("FIRST_CHANGE_ON_PATH" if last is None + else "SAME_PARTY" if last == rec["author"] else "OTHER_PARTY") + rec["vs_creator"] = ("SAME_PARTY" if creator[p] == rec["author"] else "OTHER_PARTY") + out.append(rec) + last = fold(r["author"]) + return {"spellings": spellings, "folded": {k: dict(v) for k, v in by_fold.items()}, + "losses": out} + + +def report(res, out=sys.stdout): + sp = res["spellings"] + folded = Counter({k: sum(v.values()) for k, v in res["folded"].items()}) + print("== git author spellings: %d raw -> %d folded ==" % (len(sp), len(folded)), file=out) + for f, raw in sorted(res["folded"].items()): + if len(raw) > 1: + print(" VARIANT %-22s -> %s" % (f, ", ".join("%s(%d)" % (k, v) + for k, v in sorted(raw.items(), key=lambda kv: -kv[1]))), file=out) + losses = res["losses"] + print("\n== sanctioned LOSS candidates: %d ==" % len(losses), file=out) + print("\n-- remover vs PREV_TOUCHER --", file=out) + for k, v in Counter(r["vs_prev"] for r in losses).most_common(): + print(" %-24s %d" % (k, v), file=out) + print("\n-- remover vs CREATOR --", file=out) + for k, v in Counter(r["vs_creator"] for r in losses).most_common(): + print(" %-24s %d" % (k, v), file=out) + print("\n-- lost-line SHAPE --", file=out) + for k, v in Counter(r["shape"] for r in losses).most_common(): + print(" %-14s %d" % (k, v), file=out) + print("\n-- loss_kind --", file=out) + for k, v in Counter(r["loss_kind"] for r in losses).most_common(): + print(" %-22s %d" % (k, v), file=out) + print("\n-- rows where the remover is NOT the path's previous toucher --", file=out) + others = [r for r in losses if r["vs_prev"] == "OTHER_PARTY"] + for r in others: + print(" %s %s %-42s by %-18s prev=%-16s bytes %d->%d miss=%d [%s] %s" + % (r["commit"], r["date"], r["path"], r["raw_author"], r["prev_toucher"], + r["parent_bytes"], r["child_bytes"], r["missing_parent_lines"], + r["shape"], r["break_text"][:60]), file=out) + print(" total: %d" % len(others), file=out) + print("\n-- per committed party (folded) --", file=out) + for k, v in Counter(r["author"] for r in losses).most_common(): + same = sum(1 for r in losses if r["author"] == k and r["vs_prev"] == "SAME_PARTY") + oth = sum(1 for r in losses if r["author"] == k and r["vs_prev"] == "OTHER_PARTY") + first = sum(1 for r in losses if r["author"] == k and r["vs_prev"] == "FIRST_CHANGE_ON_PATH") + print(" %-18s %3d (own-file %d, other-file %d, first-change %d)" + % (k, v, same, oth, first), file=out) + + +# --------------------------------------------------------------------- selftest + +def _row(path, date, author, line_class="LOSS", break_text="", **kw): + d = {"path": path, "date": date, "commit": kw.pop("commit", date), + "author": author, "line_class": line_class, "loss_kind": "REWRITE", + "missing_parent_lines": 1, "added_child_lines": 1, + "parent_bytes": 10, "child_bytes": 20, "order_break_line_text": break_text} + d.update(kw) + return d + + +def selftest(): + # A hand-built history. Same folds must be one party; different folds another. + rows = [ + _row("README.md", "2026-01-01T00:00:00Z", "alice", line_class="NEW"), + # Alice's own file, Alice edits it -> SAME_PARTY + _row("checkins/a.md", "2026-01-01T00:00:00Z", "Alice", line_class="NEW"), + _row("checkins/a.md", "2026-01-02T00:00:00Z", "alice", + break_text="**Status: OPEN — a", commit="c2"), + # a case-variant spelling of the SAME party must fold to SAME_PARTY + _row("checkins/b.md", "2026-01-01T00:00:00Z", "Bob", line_class="NEW"), + _row("checkins/b.md", "2026-01-02T00:00:00Z", "bob", break_text="| 1 | x |", + commit="c4"), + # a genuinely different party -> OTHER_PARTY + _row("WORK-QUEUE.md", "2026-01-01T00:00:00Z", "alice", line_class="NEW"), + _row("WORK-QUEUE.md", "2026-01-02T00:00:00Z", "bob", break_text="plain prose line here", + commit="c6"), + # a LOSS with no preceding change on the path -> FIRST_CHANGE_ON_PATH + _row("replies/x.md", "2026-01-03T00:00:00Z", "carol", + break_text="2026-01-03T00:00Z reading", commit="c7"), + # non-sanctioned path: must be excluded entirely + _row("tools/t.py", "2026-01-04T00:00:00Z", "dave", break_text="gone", commit="c8"), + # a non-LOSS row on a sanctioned path: must be excluded + _row("checkins/a.md", "2026-01-05T00:00:00Z", "alice", line_class="SUBSEQUENCE", + commit="c9"), + ] + res = adjudicate(rows) + L = res["losses"] + idx = {(r["path"], r["commit"]): r for r in L} + + cells = [ + ("sanctioned LOSS rows only", len(L) == 4), + ("non-sanctioned path excluded", all(r["path"] != "tools/t.py" for r in L)), + ("non-LOSS row excluded", all(r["commit"] != "c9" for r in L)), + ("case variant folds to SAME_PARTY", + idx[("checkins/a.md", "c2")]["vs_prev"] == "SAME_PARTY"), + ("Bob/bob is ONE folded party", + idx[("checkins/b.md", "c4")]["vs_prev"] == "SAME_PARTY"), + ("genuine other party is OTHER_PARTY", + idx[("WORK-QUEUE.md", "c6")]["vs_prev"] == "OTHER_PARTY"), + ("no preceding change -> FIRST_CHANGE_ON_PATH", + idx[("replies/x.md", "c7")]["vs_prev"] == "FIRST_CHANGE_ON_PATH"), + ("creator is the path's introducer", + idx[("checkins/a.md", "c2")]["creator"] == fold("Alice")), + ("shape STATUS", shape("**Status: OPEN — x") == "STATUS"), + ("shape TABLE_ROW", shape("| a | b |") == "TABLE_ROW"), + ("shape SIGNOFF", shape("- **x:** y") == "SIGNOFF"), + ("shape TIMED_CLAIM", shape("taken 06:51Z from this host") == "TIMED_CLAIM"), + ("shape PROSE", shape("a" * 60) == "PROSE"), + ("shape SHORT", shape("gone") == "SHORT"), + ("raw spellings preserved", + set(res["folded"]["bob"]) == {"Bob", "bob"}), + ] + ok = 0 + for name, passed in cells: + print("%-42s %s" % (name, "PASS" if passed else "FAIL")) + ok += bool(passed) + print("\nselftest: %d cells, %d passed, %d failed" % (len(cells), ok, len(cells) - ok)) + return 0 if ok == len(cells) else 1 + + +def main(): + ap = argparse.ArgumentParser( + description="Adjudicate append-sanctioned LOSS rows: whose line was replaced?") + ap.add_argument("audit_json", nargs="?", help="a --json output of append_only_audit.py") + ap.add_argument("--json", help="write the adjudication as JSON") + ap.add_argument("--selftest", action="store_true") + a = ap.parse_args() + if a.selftest: + return selftest() + if not a.audit_json: + ap.error("need AUDIT.json (or --selftest)") + d = json.load(open(a.audit_json)) + res = adjudicate(d["rows"]) + report(res) + if a.json: + with open(a.json, "w") as f: + json.dump(res, f, indent=1) + print("\njson written: %s" % a.json, file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main())