stackchain/troupe-standup
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e4cb69c7d33aec2e714b16e1ef63a4399d07d979
troupe-standup/tools/served_drift_check.py
T
volt 46e0ad8c01 served_drift_check (volt): name the one DIFFERS that is an edit in flight, not a stale deploy 
The first live run of c56e3be graded /var/www/leathermint against HEAD and got
one DIFFERS: index.html served 26,064 B / b5136bb947dd vs tree 25,697 B /
42b2f374f5d3. That is a true row and an unhelpful one -- the serve root IS the
work tree, so an uncommitted edit reaches the host before any revision holds
it, and "the deploy fell behind" and "another lane is mid-edit" send a reader to
two different places. (It was the second: argus's commit 4f69d46 landed the
26,064 B file at 19:29:41Z, three seconds after that run at 19:29:38Z; the re-run
now reads 23/23 OK.)

Every row now carries a worktree field of "modified" or "clean", computed from
`git diff --name-only <rev>` plus `git ls-files -o --exclude-standard`, and the
summary counts the modified rows and says what they mean. The verdict stays
DIFFERS -- an uncommitted file is still not the revision, and a claim that the
host is running revision X is still false while it serves X+edit -- but the row
no longer makes the reader re-measure which of the two it is.

Three cells added (23 -> 25): an uncommitted edit is named on its own row, the
summary states the count, and a clean tree gets no such note. selftest 25/25
rc 0.
2026-10-20 19:32:06 +00:00

504 lines
21 KiB
Python
Raw Blame History

#!/usr/bin/env python3
"""served_drift_check.py -- does the host actually serve the tree at this revision?
WHY THIS EXISTS
Two lanes spend a hand-written `curl | sha256sum` loop on every leathermint
tick: Timmy's reply 45 lists six served paths "byte-exact", this lane's own
progress block lists six more, and both wrote the sizes out by hand. That is
the same fact re-measured from scratch, in prose, with no instrument that can
disagree with either of them -- the class `gate_blame.py` was built for on the
2026-10-20 gate, one layer out: a deploy is a CLAIM about a revision, and
nothing graded it.
`landing_check.py` answers the neighbouring question for the forge: it reads
`<base>/branch/<branch>/<path>`, a raw-endpoint convention, because that is
how the archive reads itself. A static host has no such convention: the URL
is the path, the served root need not be a git tree at all, and the failure
mode is not "the commit was never pushed" but "the bytes on the host are not
the bytes this revision says". This tool grades that second question with the
first tool's vocabulary, so a reader does not have to learn a third table.
WHAT IT MEASURES -- one row per tracked file at --rev, one verdict per row
OK the blob at `rev:path` and the body served at `<base>/<path>`
have the same length and the same sha256.
DIFFERS both sides returned bytes and they are not identical: the host
is running something other than this revision. Fatal, and the
row carries both lengths and both digests, because "the deploy
is stale" is only useful with the direction of the staleness.
MISSING the host answered 404. A tracked path the host does not serve.
HTTP_ERROR any other non-200 status. Not a verdict about content.
UNREACHABLE the transport failed before a status existed (DNS, refused,
TLS, timeout). Kept separate from MISSING on purpose: a host
that did not answer is not a host that lacks the file, and
collapsing them lets an outage read as a clean negative.
READ_FAILED `git cat-file` could not produce the tree's bytes. An
archive-side inconsistency, not a host one.
SKIPPED_LARGE the local blob exceeds --max-bytes; not fetched, not graded.
ADVISORY, never fatal: a size policy is not a deploy defect.
FATAL = DIFFERS, MISSING, HTTP_ERROR, UNREACHABLE, READ_FAILED. Exit 1 if any
row is fatal, else 0. A run that cannot grade at all -- not a git work tree,
a --base-url that is not http(s):// or file://, or a revision that tracks
nothing -- exits 2 with the reason, never 0.
Every row also carries `worktree`: `modified` when the deploy directory holds
an uncommitted version of that path, `clean` otherwise, and the summary counts
them. The serve root IS the work tree, so an edit in flight reaches the host
before any revision holds it. That case is named instead of being folded into
DIFFERS, because "the deploy fell behind" and "another lane is mid-edit" send
a reader to two different places -- and only the first is anybody's defect.
WHAT IT DOES NOT CLAIM
Byte equality is not provenance. Identical bytes say the host serves what this
revision holds; they say nothing about that revision being correct, reviewed,
or the one that should be live. A file no revision tracks is out of scope by
construction -- the untracked-file blindness this archive already paid for --
so a stale file that git never knew about is invisible here and is said so
here rather than left to be discovered later. `v2/` is graded like everything
else: a deprecated build that is still served is a fact, not an exemption.
READ-ONLY: GET requests only, and the single write path is `--json PATH`. No
file, branch, remote or deploy is modified; nothing is posted anywhere.
USAGE:
python3 served_drift_check.py --root /var/www/leathermint \
--base-url https://leathermint.alexanderwhitestone.com
python3 served_drift_check.py --root . --rev HEAD --prefix iso/ --quiet
python3 served_drift_check.py --root . --base-url file:///srv/copy --json out.json
python3 served_drift_check.py --selftest # fixture host, no network, 25 cells
"""
import argparse
import fnmatch
import hashlib
import json
import os
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request
FATAL = ("DIFFERS", "MISSING", "HTTP_ERROR", "UNREACHABLE", "READ_FAILED")
ADVISORY = ("SKIPPED_LARGE",)
USER_AGENT = "served_drift_check/1.0 (troupe-standup)"
class GradeError(Exception):
"""Could not grade at all: the reason is printed and rc is 2."""
class FetchError(Exception):
"""The transport failed before any status existed."""
def git(args, cwd):
return subprocess.run(["git", "-C", cwd] + list(args),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
def tracked_files(root, rev, prefix=None):
"""Every path `rev` tracks, as git spells it -- NUL-split, so a space in a
filename is a filename and not two."""
p = git(["ls-tree", "-r", "-z", "--name-only", rev], root)
if p.returncode != 0:
raise GradeError("cannot read repo at %s: %s"
% (root, p.stderr.decode("utf-8", "replace").strip()[:200]))
paths = [s for s in p.stdout.decode("utf-8", "surrogateescape").split("\0") if s]
if prefix:
paths = [x for x in paths if x.startswith(prefix)]
return sorted(paths)
def read_blob(root, rev, path):
p = git(["cat-file", "blob", "%s:%s" % (rev, path)], root)
if p.returncode != 0:
return None
return p.stdout
def check_base_url(base):
if not base:
raise GradeError("no --base-url: there is nothing to grade a tree against")
scheme = urllib.parse.urlsplit(base).scheme
if scheme not in ("http", "https", "file"):
raise GradeError("not a URL this tool can read (http, https, file): %r" % base)
return base.rstrip("/")
def url_for(base, path):
return base + "/" + urllib.parse.quote(path, safe="/")
def fetch(url, timeout):
"""Return (status, body). file:// returns (200, bytes) or (404, b'')."""
if url.startswith("file://"):
local = urllib.parse.unquote(urllib.parse.urlsplit(url).path)
try:
with open(local, "rb") as fh:
return 200, fh.read()
except FileNotFoundError:
return 404, b""
except OSError as e:
raise FetchError(str(e))
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
try:
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.getcode(), r.read()
except urllib.error.HTTPError as e:
return e.code, b""
except urllib.error.URLError as e:
raise FetchError(str(e.reason))
except (TimeoutError, OSError) as e:
raise FetchError(str(e))
def grade_one(root, rev, base, path, max_bytes, timeout):
row = {"path": path, "url": url_for(base, path), "verdict": "OK",
"tree_len": None, "served_len": None, "tree_sha": None,
"served_sha": None, "detail": ""}
blob = read_blob(root, rev, path)
if blob is None:
row.update(verdict="READ_FAILED", detail="rev:%s does not carry this path" % rev)
return row
row["tree_len"] = len(blob)
row["tree_sha"] = hashlib.sha256(blob).hexdigest()
if max_bytes is not None and len(blob) > max_bytes:
row.update(verdict="SKIPPED_LARGE",
detail="%d B over --max-bytes %d" % (len(blob), max_bytes))
return row
try:
status, body = fetch(row["url"], timeout)
except FetchError as e:
row.update(verdict="UNREACHABLE", detail="host did not answer: %s" % str(e)[:120])
return row
if status == 404:
row.update(verdict="MISSING", detail="404; tree has %d B" % len(blob))
return row
if status != 200:
row.update(verdict="HTTP_ERROR", detail="status %d" % status)
return row
row["served_len"] = len(body)
row["served_sha"] = hashlib.sha256(body).hexdigest()
if body != blob:
row.update(verdict="DIFFERS",
detail="served %d B %s vs tree %d B %s"
% (len(body), row["served_sha"][:12],
len(blob), row["tree_sha"][:12]))
return row
row["detail"] = "%d B, sha256 %s" % (len(blob), row["tree_sha"][:16])
return row
def is_fatal(row):
return row["verdict"] in FATAL
def summarise(rows):
n_fatal = sum(1 for r in rows if is_fatal(r))
n_ok = sum(1 for r in rows if r["verdict"] == "OK")
n_skip = sum(1 for r in rows if r["verdict"] in ADVISORY)
return n_ok, n_fatal, n_skip
def worktree_dirty(root, rev):
"""Paths the host would serve that `rev` does not hold: modified against rev,
plus untracked-and-not-ignored. The serve root IS the work tree, so these are
the rows where a DIFFERS is a fact about an edit in flight, not about a deploy
that fell behind -- a distinction the summary has to make or the reader has to
re-measure it by hand."""
out = set()
p = git(["diff", "--name-only", rev], root)
if p.returncode == 0:
out |= {s for s in p.stdout.decode("utf-8", "surrogateescape").splitlines() if s}
p = git(["ls-files", "-o", "--exclude-standard"], root)
if p.returncode == 0:
out |= {s for s in p.stdout.decode("utf-8", "surrogateescape").splitlines() if s}
return out
def run(root, rev, base, prefix=None, exclude=(), max_bytes=None, timeout=20.0):
if git(["rev-parse", "--is-inside-work-tree"], root).returncode != 0:
raise GradeError("not a git work tree: %s" % root)
base = check_base_url(base)
paths = tracked_files(root, rev, prefix)
paths = [p for p in paths if not any(fnmatch.fnmatch(p, pat) for pat in exclude)]
if not paths:
raise GradeError("revision %s tracks no file%s"
% (rev, " under %s" % prefix if prefix else ""))
dirty = worktree_dirty(root, rev)
rows = []
for p in paths:
row = grade_one(root, rev, base, p, max_bytes, timeout)
row["worktree"] = "modified" if p in dirty else "clean"
if row["worktree"] == "modified":
row["detail"] = (row["detail"] + "; the work tree holds an uncommitted "
"version of this path").strip("; ")
rows.append(row)
return rows
def report(rows, out=sys.stdout, rev=None):
width = max([len(r["path"]) for r in rows] + [4])
for r in rows:
out.write(" %-*s %-12s %s\n" % (width, r["path"], r["verdict"], r["detail"]))
n_ok, n_fatal, n_skip = summarise(rows)
n_dirty = sum(1 for r in rows if r.get("worktree") == "modified")
if n_dirty:
out.write("\nnote: %d path(s) are uncommitted against %s. The serve root is the"
" work tree, so a DIFFERS on those rows is an edit already on the host"
" that no revision holds yet -- still not the revision, and still fatal"
" to a claim that the host is running one.\n"
% (n_dirty, rev or "the revision graded"))
out.write("\nserved: %d files, %d OK, %d drift, %d skipped\n"
% (len(rows), n_ok, n_fatal, n_skip))
if n_fatal:
out.write("FAIL: %d file(s) are not served as this revision says\n" % n_fatal)
return 1 if n_fatal else 0
# --------------------------------------------------------------------- selftest
def _write(path, data):
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
mode = "wb" if isinstance(data, bytes) else "w"
with open(path, mode) as fh:
fh.write(data)
def _commit(repo, msg):
git(["add", "-A"], repo)
git(["-c", "user.name=t", "-c", "user.email=t@t", "commit", "-qm", msg], repo)
def selftest():
import functools
import http.server
import shutil
import tempfile
import threading
cells = [0]
bad = [0]
def cell(name, ok, detail=""):
cells[0] += 1
if not ok:
bad[0] += 1
print(" FAIL %s%s" % (name, (": " + str(detail)) if detail else ""))
tmp = tempfile.mkdtemp(prefix="served-drift-")
try:
repo = os.path.join(tmp, "repo")
served = os.path.join(tmp, "served")
os.makedirs(repo)
git(["init", "-q"], repo)
_write(os.path.join(repo, "index.html"), "<h1>v1</h1>\n")
_write(os.path.join(repo, "iso", "engine.js"), "// engine v1\n")
_write(os.path.join(repo, "journal", "index.html"), "<p>journal</p>\n")
_commit(repo, "v1")
def mirror(dst):
for dirpath, _dirs, files in os.walk(dst):
for f in files:
if f == ".git":
continue
shutil.rmtree(dst, ignore_errors=True)
shutil.copytree(os.path.join(repo, "index.html").rsplit(os.sep, 1)[0], dst,
ignore=shutil.ignore_patterns(".git"))
mirror(served)
base = "file://" + os.path.abspath(served)
rows = run(repo, "HEAD", base)
cell("c01 a mirror of the revision grades OK",
[r["verdict"] for r in rows] == ["OK"] * 3, [r["verdict"] for r in rows])
cell("c02 the row carries both digests",
rows[0]["tree_sha"] == rows[0]["served_sha"] and len(rows[0]["tree_sha"]) == 64)
cell("c03 report is rc 0 and prints the total", report(rows, _Sink()) == 0)
# the failure this tool exists for: the host runs a stale copy
_write(os.path.join(served, "iso", "engine.js"), "// engine v0\n")
rows = run(repo, "HEAD", base)
stale = [r for r in rows if r["path"] == "iso/engine.js"][0]
cell("c04 a stale served copy is DIFFERS", stale["verdict"] == "DIFFERS", stale)
cell("c05 the DIFFERS row carries both lengths and both digests",
stale["served_len"] == 13 and stale["tree_len"] == 13
and stale["served_sha"][:8] != stale["tree_sha"][:8], stale["detail"])
cell("c06 any fatal row makes the run rc 1", report(rows, _Sink()) == 1)
os.remove(os.path.join(served, "journal", "index.html"))
rows = run(repo, "HEAD", base)
cell("c07 a path the host does not serve is MISSING",
[r for r in rows if r["path"] == "journal/index.html"][0]["verdict"] == "MISSING")
# a real HTTP server, so the http branch is measured and not assumed
os.remove(os.path.join(served, "iso", "engine.js"))
class H(http.server.SimpleHTTPRequestHandler):
def log_message(self, *a):
pass
def do_GET(self):
if self.path.endswith("index.html") and "journal" in self.path:
self.send_error(500)
return
return super().do_GET()
handler = functools.partial(H, directory=served)
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
t = threading.Thread(target=srv.serve_forever, daemon=True)
t.start()
try:
http_base = "http://127.0.0.1:%d" % srv.server_address[1]
rows = run(repo, "HEAD", http_base)
by = {r["path"]: r for r in rows}
cell("c08 over HTTP a matching file is OK", by["index.html"]["verdict"] == "OK")
cell("c09 over HTTP a 500 is HTTP_ERROR, not a content verdict",
by["journal/index.html"]["verdict"] == "HTTP_ERROR", by["journal/index.html"])
cell("c10 over HTTP a removed file is MISSING",
by["iso/engine.js"]["verdict"] == "MISSING", by["iso/engine.js"])
finally:
srv.shutdown()
srv.server_close()
rows = run(repo, "HEAD", "http://127.0.0.1:1", timeout=2)
cell("c11 a refused connection is UNREACHABLE",
all(r["verdict"] == "UNREACHABLE" for r in rows), [r["verdict"] for r in rows])
cell("c12 an unreachable host is fatal, never a clean negative",
report(rows, _Sink()) == 1)
# rev discipline: grading an older revision grades THAT revision
_write(os.path.join(repo, "index.html"), "<h1>v2</h1>\n")
_commit(repo, "v2")
head_rows = run(repo, "HEAD", base)
old_rows = run(repo, "HEAD~1", base)
cell("c13 HEAD grades the new blob and HEAD~1 the old one",
[r for r in head_rows if r["path"] == "index.html"][0]["verdict"] == "DIFFERS"
and [r for r in old_rows if r["path"] == "index.html"][0]["verdict"] == "OK")
rows = run(repo, "HEAD", base, prefix="iso/")
cell("c14 --prefix grades only the subtree it names",
[r["path"] for r in rows] == ["iso/engine.js"], [r["path"] for r in rows])
rows = run(repo, "HEAD", base, exclude=("iso/*",))
cell("c15 --exclude drops the matching rows",
sorted(r["path"] for r in rows) == ["index.html", "journal/index.html"])
rows = run(repo, "HEAD", base, max_bytes=5)
cell("c16 over --max-bytes is SKIPPED_LARGE and never fatal",
all(r["verdict"] == "SKIPPED_LARGE" for r in rows)
and report(rows, _Sink()) == 0)
for label, argv, want in (
("not-a-repo", dict(root=tmp, rev="HEAD", base=base), "not a git work tree"),
("no-base-url", dict(root=repo, rev="HEAD", base=None), "no --base-url"),
("bad-scheme", dict(root=repo, rev="HEAD", base="ftp://x/y"), "not a URL"),
):
try:
run(**argv)
cell("c17 %s is refused at rc 2" % label, False, "no exception")
except GradeError as e:
cell("c17 %s is refused at rc 2" % label, want in str(e), str(e))
try:
run(root=repo, rev="HEAD~99", base=base)
cell("c18 a revision that does not resolve is rc 2", False, "no exception")
except GradeError as e:
cell("c18 a revision that does not resolve is rc 2", "cannot read repo" in str(e),
str(e))
try:
run(root=repo, rev="HEAD", base=base, prefix="nope/")
cell("c19 a prefix that tracks nothing is rc 2", False, "no exception")
except GradeError as e:
cell("c19 a prefix that tracks nothing is rc 2", "tracks no file" in str(e), str(e))
cell("c20 --json is the same rows the table printed",
_json_roundtrip(repo, base, tmp))
# the work tree is what the host serves, so an uncommitted edit is the one
# DIFFERS that is not a stale deploy -- and it must be named as such.
_write(os.path.join(repo, "index.html"), "<h1>v3 uncommitted</h1>\n")
rows = run(repo, "HEAD", base)
dr = [r for r in rows if r["path"] == "index.html"][0]
cell("c21 an uncommitted edit is named on its own row",
dr["worktree"] == "modified" and "uncommitted" in dr["detail"], dr["detail"])
cap = _Cap()
report(rows, cap, rev="HEAD")
cell("c22 the summary states how many rows are edits in flight",
"1 path(s) are uncommitted" in cap.text, cap.text[-200:])
git(["checkout", "--", "index.html"], repo)
cap = _Cap()
report(run(repo, "HEAD", base), cap, rev="HEAD")
cell("c23 a clean tree gets no such note", "uncommitted" not in cap.text)
finally:
shutil.rmtree(tmp, ignore_errors=True)
print("selftest: %d cells, %d passed, %d failed"
% (cells[0], cells[0] - bad[0], bad[0]))
return 1 if bad[0] else 0
class _Sink:
def __init__(self):
self.text = ""
def write(self, s):
self.text += s
return len(s)
_Cap = _Sink
def _json_roundtrip(repo, base, tmp):
out = os.path.join(tmp, "rows.json")
rows = run(repo, "HEAD", base)
with open(out, "w") as fh:
json.dump({"rows": rows}, fh)
with open(out) as fh:
back = json.load(fh)["rows"]
return back == rows and len(back) == 3
# ----------------------------------------------------------------------- main
def main(argv=None):
ap = argparse.ArgumentParser(
description="Grade a host's served bytes against the tree at a revision.")
ap.add_argument("--root", default=".", help="the deploy directory's git work tree")
ap.add_argument("--rev", default="HEAD")
ap.add_argument("--base-url", default=None,
help="http(s):// host root, or file:// for an offline copy")
ap.add_argument("--prefix", default=None, help="only paths under this prefix")
ap.add_argument("--exclude", action="append", default=[], metavar="GLOB")
ap.add_argument("--max-bytes", type=int, default=None, metavar="N")
ap.add_argument("--timeout", type=float, default=20.0, metavar="S")
ap.add_argument("--json", default=None, metavar="PATH")
ap.add_argument("--quiet", action="store_true")
ap.add_argument("--selftest", action="store_true")
a = ap.parse_args(argv)
if a.selftest:
return selftest()
try:
rows = run(a.root, a.rev, a.base_url, a.prefix, a.exclude,
a.max_bytes, a.timeout)
except GradeError as e:
sys.stderr.write("served_drift_check: %s\n" % e)
return 2
if a.json:
with open(a.json, "w") as fh:
json.dump({"root": os.path.abspath(a.root), "rev": a.rev,
"base_url": a.base_url, "rows": rows}, fh, indent=1)
if a.quiet:
n_ok, n_fatal, n_skip = summarise(rows)
print("served: %d files, %d OK, %d drift, %d skipped"
% (len(rows), n_ok, n_fatal, n_skip))
return 1 if n_fatal else 0
return report(rows, rev=a.rev)
if __name__ == "__main__":
sys.exit(main())
Reference in new issue View Git Blame Copy Permalink