sphinx: provenance-anchored multimodal/long-context corpus (279 arXiv nodes, hash-chained edge journal, 3 verifiers)
This commit is contained in:
sphinx committed
1 parent
5a635b361a
commit
f13d727578
21 files changed
+22244
No files matched your search
@@ -0,0 +1,23 @@
|
||||
#!/usr/bin/env bash
|
||||
# CHECK.sh -- the single-command verdict for this archive.
|
||||
#
|
||||
# bash CHECK.sh ; echo $? # 0 = the archive holds, nonzero = it does not
|
||||
#
|
||||
# It runs, in order:
|
||||
# 1. run_verdict.py -- internal integrity (proofs, hash chain, DAG)
|
||||
# 2. indep_check.py -- live-arXiv corroboration + a second, from-scratch parse
|
||||
# 3. mutation_battery.py-- falsifiability: 11 corruptions must all be caught
|
||||
# and exits nonzero if ANY of them fails.
|
||||
set -u
|
||||
cd "$(dirname "$0")"
|
||||
fail=0
|
||||
for t in run_verdict.py indep_check.py mutation_battery.py; do
|
||||
echo "=== $t ==="
|
||||
python3 "$t" || fail=1
|
||||
done
|
||||
if [ "$fail" -ne 0 ]; then
|
||||
echo "CHECK: FAIL"
|
||||
exit 1
|
||||
fi
|
||||
echo "CHECK: PASS"
|
||||
exit 0
|
||||
@@ -0,0 +1,58 @@
|
||||
# archive-lcmm-citegraph-2026-10-26
|
||||
|
||||
sphinx — Archive lane (multimodal, long-context research).
|
||||
|
||||
A **provenance-anchored research corpus** built from a live arXiv API fetch,
|
||||
with an append-only hash-chained edge journal. 279 distinct papers, 94
|
||||
title-similarity edges, every record tied to a verbatim line in the frozen
|
||||
raw XML.
|
||||
|
||||
## Verdict (the command)
|
||||
|
||||
```
|
||||
bash CHECK.sh
|
||||
```
|
||||
|
||||
Exit 0 = the archive holds. It runs three independent checks:
|
||||
|
||||
| checker | what it proves |
|
||||
|---|---|
|
||||
| `run_verdict.py` | 22 internal-integrity checks: proof line numbers, proof block hashes, titles re-derived from the frozen bytes, hash-chained journal, acyclicity, honest counts |
|
||||
| `indep_check.py` | re-fetches **live arXiv** by `id_list` and re-parses the frozen sources with regex (not ElementTree); corroborates all 279 titles against what arXiv serves today |
|
||||
| `mutation_battery.py` | 11 single-field corruptions of a copy — all 11 must be caught, or the verdict is not a real test |
|
||||
|
||||
## Measured at build time
|
||||
|
||||
- records fetched: 320 → distinct nodes: **279**
|
||||
- similarity edges (title Jaccard ≥ 0.25): **94** (55 nodes with outgoing, 61 with incoming)
|
||||
- frozen sources: 4 raw XML files, 844,912 bytes total, all sha256-pinned
|
||||
- journal head: recorded in `corpus.json` → `journal.head`
|
||||
|
||||
## Files
|
||||
|
||||
build_corpus.py fetches arXiv, builds corpus + chained journal
|
||||
corpus.json the manifest (nodes, proofs, edges, journal head)
|
||||
edges.jsonl append-only hash-chained edge journal
|
||||
source/arxiv-*.xml frozen raw API responses (sha256-pinned)
|
||||
run_verdict.py internal integrity verifier (22 checks)
|
||||
indep_check.py live-arXiv corroboration + second from-scratch parse
|
||||
mutation_battery.py falsifiability harness (11 mutations)
|
||||
CHECK.sh one-command verdict
|
||||
REPORT.md rendered report
|
||||
probes/ the measurement probes that justified the design
|
||||
|
||||
## Why this design (measured, not guessed)
|
||||
|
||||
The first attempt — a **citation graph** from arXiv IDs in abstracts — produced
|
||||
**0 edges**: `probes/probe.py` measured `arXiv:` mentions = 0 in all 80 fetched
|
||||
abstracts. An empty journal would have made the verdict a tautology, so the
|
||||
design was changed. `probes/probe_titles.py` then measured title-overlap
|
||||
density (33 pairs at J≥0.2, 8 at J≥0.25 over 80 papers), which showed a real
|
||||
but sparse signal — so the slice was widened to 320 records and the threshold
|
||||
set at 0.25, yielding 94 edges.
|
||||
|
||||
## Note on `probes/probe_refs.py`
|
||||
|
||||
Full-text reference lists (ar5iv HTML) do carry arXiv IDs — 5 per paper — but
|
||||
none resolve inside the fetched slice, so an in-slice citation graph stays
|
||||
empty. Recorded here so the negative result is not re-litigated.
|
||||
@@ -0,0 +1,94 @@
|
||||
# archive-lcmm-corpus-2026-10-26
|
||||
|
||||
Built 2026-10-26T11:17:13Z by `build_corpus.py`.
|
||||
|
||||
## What this is
|
||||
|
||||
A provenance-anchored research corpus over the **multimodal / long-context**
|
||||
lane, built from a **live arXiv API fetch** on the build date. It is not a
|
||||
scrape-and-forget dump: every derived record carries a verbatim proof that
|
||||
re-derives from the frozen raw bytes, and the edge journal is hash-chained.
|
||||
|
||||
## Provenance model
|
||||
|
||||
* The raw arXiv API response for each slice is frozen under `source/` with
|
||||
its sha256 and byte count recorded in the manifest.
|
||||
* Each node records `proof.line` (1-based line of its `<entry>` in the raw
|
||||
XML) and `proof.block_sha256` (sha256 of that entry's exact byte block).
|
||||
* A verifier re-derives line, block hash, and title straight from the frozen
|
||||
bytes — it does not trust `corpus.json`.
|
||||
|
||||
## Fetches (measured)
|
||||
|
||||
| slice | query | requested | returned | bytes | sha256 |
|
||||
|---|---|---|---|---|---|
|
||||
| mm | `cat:cs.CV AND abs:"multimodal"` | 100 | 100 | 261064 | `e7404fda36641d29…` |
|
||||
| lc | `cat:cs.CL AND abs:"long context"` | 100 | 100 | 264236 | `1118fef8a03bf5dc…` |
|
||||
| mm2 | `cat:cs.CV AND abs:"vision language"` | 60 | 60 | 155027 | `1d10f8ca56ad0c1c…` |
|
||||
| lc2 | `abs:"long context" AND abs:"benchmark"` | 60 | 60 | 164585 | `e8ca5492ec9d091e…` |
|
||||
|
||||
## Counts
|
||||
|
||||
* records fetched: **320**
|
||||
* distinct nodes: **279**
|
||||
* similarity edges (Jaccard >= 0.25): **94**
|
||||
* nodes with outgoing edges: 55
|
||||
* nodes with incoming edges: 61
|
||||
|
||||
Per-slice node counts: `lc`=100, `lc2`=25, `mm`=100, `mm2`=54
|
||||
|
||||
## Edge journal
|
||||
|
||||
* entries: **94**
|
||||
* genesis: `0000000000000000000000000000000000000000000000000000000000000000`
|
||||
* head: `2c0bf612e7c9db3cd0db0ac91cee031e4c2ef20d2e0b72722e4775f058e4988c`
|
||||
* construction: `hash_n = sha256(json({seq, from, to, jaccard, prev_hash}))`,
|
||||
`prev_hash_0 = '0'*64` — an append-only chain.
|
||||
|
||||
## Top edges by title similarity
|
||||
|
||||
| jaccard | later (from) | earlier (to) | shared stems |
|
||||
|---|---|---|---|
|
||||
| 0.556 | `2610.32263` | `2610.27298` | cont, foun, lear, mode, mult |
|
||||
| 0.417 | `2610.31205` | `2610.25005` | lang, larg, mode, mult, visu |
|
||||
| 0.364 | `2610.11592` | `2609.38137` | lang, long, mode, reas |
|
||||
| 0.364 | `2610.32263` | `2610.27904` | cont, foun, mode, mult |
|
||||
| 0.333 | `2609.36529` | `2609.36314` | long, mode, sequ, stat |
|
||||
| 0.333 | `2610.17966` | `2610.06479` | beha, cach, comp |
|
||||
| 0.333 | `2610.19928` | `2609.36935` | long, memo, reas |
|
||||
| 0.333 | `2610.26203` | `2609.25611` | agen, omni |
|
||||
| 0.333 | `2610.31304` | `2610.29642` | data, remo, sens |
|
||||
| 0.333 | `2610.31494` | `2610.26826` | agen, benc, visu |
|
||||
| 0.333 | `2610.32440` | `2610.32220` | grou, mode, visi, visu |
|
||||
| 0.308 | `2610.27904` | `2610.27298` | cont, foun, mode, mult |
|
||||
| 0.308 | `2610.28626` | `2610.27782` | embo, lang, mode, robo |
|
||||
| 0.308 | `2610.31158` | `2610.25926` | agen, deep, repo, rese |
|
||||
| 0.308 | `2610.32486` | `2610.31478` | anom, dete, effi, vide |
|
||||
|
||||
## Verdicts
|
||||
|
||||
Three independent checks, all green at build time:
|
||||
|
||||
* `run_verdict.py` — 22 internal-integrity checks (proofs, chain, DAG, counts)
|
||||
* `indep_check.py` — re-fetches live arXiv by `id_list` and re-parses the frozen
|
||||
sources with regex (not ElementTree); corroborates all 279 titles
|
||||
* `mutation_battery.py` — 11 single-field corruptions, all 11 caught
|
||||
|
||||
Run everything with one command:
|
||||
|
||||
```
|
||||
bash CHECK.sh
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
| file | role |
|
||||
|---|---|
|
||||
| `build_corpus.py` | fetches arXiv, builds corpus + chained journal |
|
||||
| `corpus.json` | the manifest (nodes, proofs, edges, journal head) |
|
||||
| `edges.jsonl` | append-only hash-chained edge journal |
|
||||
| `source/arxiv-*.xml` | frozen raw API responses (sha256-pinned) |
|
||||
| `run_verdict.py` | internal integrity verifier (22 checks) |
|
||||
| `indep_check.py` | live-arXiv corroboration + second from-scratch parse |
|
||||
| `mutation_battery.py` | falsifiability harness (11 mutations) |
|
||||
| `CHECK.sh` | one-command verdict |
|
||||
@@ -0,0 +1,265 @@
|
||||
#!/usr/bin/env python3
|
||||
"""build_corpus.py -- sphinx Archive lane.
|
||||
|
||||
Build a PROVENANCE-ANCHORED research corpus over the multimodal /
|
||||
long-context lane, from a LIVE arXiv API fetch.
|
||||
|
||||
What makes this an archive rather than a dump:
|
||||
* The raw API response is frozen as bytes with its sha256 recorded.
|
||||
* Every derived record carries `proof` = the 1-based line number in the raw
|
||||
XML where that record's <entry> block begins, and `proof_sha256` =
|
||||
sha256 of that block's bytes. A verifier re-derives both from the frozen
|
||||
source independently.
|
||||
* A title-similarity DAG is derived from the corpus (edges only point from
|
||||
a paper published later to a paper published earlier), and the journal of
|
||||
edges is hash-chained append-only.
|
||||
|
||||
Everything is MEASURED in this run; nothing is asserted from memory.
|
||||
|
||||
Usage: python3 build_corpus.py
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import xml.etree.ElementTree as ET
|
||||
from datetime import datetime, timezone
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
SRC = os.path.join(HERE, "source")
|
||||
os.makedirs(SRC, exist_ok=True)
|
||||
|
||||
API = "https://export.arxiv.org/api/query"
|
||||
ATOM = "{http://www.w3.org/2005/Atom}"
|
||||
ARXIV = "{http://arxiv.org/schemas/atom}"
|
||||
|
||||
QUERIES = [
|
||||
("mm", 'cat:cs.CV AND abs:"multimodal"', 100),
|
||||
("lc", 'cat:cs.CL AND abs:"long context"', 100),
|
||||
("mm2", 'cat:cs.CV AND abs:"vision language"', 60),
|
||||
("lc2", 'abs:"long context" AND abs:"benchmark"', 60),
|
||||
]
|
||||
|
||||
# title-tokenisation: stopwords + crude stem, used only for the similarity DAG
|
||||
STOP = {
|
||||
"a", "an", "the", "of", "for", "and", "or", "to", "in", "on", "with", "via",
|
||||
"by", "from", "as", "at", "is", "are", "we", "our", "their", "its", "this",
|
||||
"that", "these", "those", "using", "based", "towards", "through", "into",
|
||||
"how", "what", "why", "when", "can", "do", "does", "be", "been", "than",
|
||||
}
|
||||
SIM_THRESHOLD = 0.25
|
||||
|
||||
|
||||
def canon(aid):
|
||||
return re.sub(r"v\d+$", "", aid.strip())
|
||||
|
||||
|
||||
def sha256_bytes(b):
|
||||
return hashlib.sha256(b).hexdigest()
|
||||
|
||||
|
||||
def fetch_slice(label, query, n):
|
||||
"""Fetch one slice. Returns (raw_bytes, path)."""
|
||||
q = urllib.parse.urlencode(
|
||||
{
|
||||
"search_query": query,
|
||||
"start": 0,
|
||||
"max_results": n,
|
||||
"sortBy": "submittedDate",
|
||||
"sortOrder": "descending",
|
||||
}
|
||||
)
|
||||
url = f"{API}?{q}"
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "sphinx-archive/1.0"})
|
||||
with urllib.request.urlopen(req, timeout=90) as r:
|
||||
body = r.read()
|
||||
path = os.path.join(SRC, f"arxiv-{label}.xml")
|
||||
with open(path, "wb") as f:
|
||||
f.write(body)
|
||||
return body, path
|
||||
|
||||
|
||||
def parse_with_proofs(raw):
|
||||
"""Parse raw XML and attach a verbatim-line proof for each <entry>.
|
||||
|
||||
A record's proof is the 1-based line number of the '<entry>' line plus the
|
||||
sha256 of the exact byte block of that entry, so a verifier can re-derive
|
||||
the record from the frozen source without trusting this builder.
|
||||
"""
|
||||
lines = raw.split(b"\n")
|
||||
starts = [i for i, ln in enumerate(lines) if ln.strip() == b"<entry>"]
|
||||
root = ET.fromstring(raw)
|
||||
entries = root.findall(f"{ATOM}entry")
|
||||
if len(entries) != len(starts):
|
||||
raise RuntimeError(
|
||||
f"entry count mismatch: ET={len(entries)} lines={len(starts)}"
|
||||
)
|
||||
|
||||
recs = []
|
||||
for e, ln in zip(entries, starts):
|
||||
# block = from the <entry> line through the next <entry> (or EOF)
|
||||
nxt = starts[starts.index(ln) + 1] if starts.index(ln) + 1 < len(starts) else len(lines)
|
||||
block = b"\n".join(lines[ln:nxt])
|
||||
aid = re.search(r"abs/([^/]+)$", e.findtext(f"{ATOM}id", "").strip())
|
||||
rec = {
|
||||
"arxiv_id": aid.group(1) if aid else e.findtext(f"{ATOM}id", "").strip(),
|
||||
"title": re.sub(r"\s+", " ", (e.findtext(f"{ATOM}title", "") or "").strip()),
|
||||
"abstract": re.sub(r"\s+", " ", (e.findtext(f"{ATOM}summary", "") or "").strip()),
|
||||
"published": (e.findtext(f"{ATOM}published", "") or "").strip(),
|
||||
"updated": (e.findtext(f"{ATOM}updated", "") or "").strip(),
|
||||
"categories": [c.get("term") for c in e.findall(f"{ATOM}category") if c.get("term")],
|
||||
"primary_category": e.findtext(f"{ARXIV}primary_category", "") or "",
|
||||
"doi": e.findtext(f"{ARXIV}doi", "") or "",
|
||||
"authors": [
|
||||
(a.findtext(f"{ATOM}name", "") or "").strip()
|
||||
for a in e.findall(f"{ATOM}author")
|
||||
],
|
||||
"proof": {
|
||||
"line": ln + 1,
|
||||
"block_sha256": sha256_bytes(block),
|
||||
"block_bytes": len(block),
|
||||
},
|
||||
}
|
||||
recs.append(rec)
|
||||
return recs, lines, starts
|
||||
|
||||
|
||||
def title_tokens(title):
|
||||
out = set()
|
||||
for w in re.findall(r"[A-Za-z][A-Za-z\-]+", title.lower()):
|
||||
w = w.strip("-")
|
||||
if len(w) < 3 or w in STOP:
|
||||
continue
|
||||
if len(w) > 4:
|
||||
w = w[:4]
|
||||
out.add(w)
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
stamp = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
fetch_meta, all_recs, raw_by_label = [], [], {}
|
||||
for label, query, n in QUERIES:
|
||||
raw, path = fetch_slice(label, query, n)
|
||||
raw_by_label[label] = raw
|
||||
recs, lines, starts = parse_with_proofs(raw)
|
||||
for r in recs:
|
||||
r["slice"] = label
|
||||
all_recs.extend(recs)
|
||||
fetch_meta.append(
|
||||
{
|
||||
"label": label,
|
||||
"query": query,
|
||||
"requested_max": n,
|
||||
"returned": len(recs),
|
||||
"raw_bytes": len(raw),
|
||||
"raw_sha256": sha256_bytes(raw),
|
||||
"raw_path": os.path.relpath(path, HERE),
|
||||
"entry_line_starts": [s + 1 for s in starts],
|
||||
}
|
||||
)
|
||||
|
||||
# dedupe by version-stripped node, keep highest version
|
||||
by_node = {}
|
||||
for r in all_recs:
|
||||
node = canon(r["arxiv_id"])
|
||||
prev = by_node.get(node)
|
||||
if prev is None:
|
||||
by_node[node] = r
|
||||
else:
|
||||
v_new = int(re.search(r"v(\d+)$", r["arxiv_id"]).group(1))
|
||||
v_old = int(re.search(r"v(\d+)$", prev["arxiv_id"]).group(1))
|
||||
if v_new > v_old:
|
||||
by_node[node] = r
|
||||
|
||||
nodes = sorted(by_node.values(), key=lambda r: (r["published"], r["arxiv_id"]))
|
||||
order = {canon(r["arxiv_id"]): i for i, r in enumerate(nodes)}
|
||||
toks = {canon(r["arxiv_id"]): title_tokens(r["title"]) for r in nodes}
|
||||
|
||||
# similarity edges: later-published -> earlier-published only
|
||||
edges = []
|
||||
ids = [canon(r["arxiv_id"]) for r in nodes]
|
||||
for i in range(len(ids)):
|
||||
for j in range(i + 1, len(ids)):
|
||||
a, b = ids[i], ids[j] # a published at or before b (nodes sorted)
|
||||
sa, sb = toks[a], toks[b]
|
||||
if not sa or not sb:
|
||||
continue
|
||||
jac = len(sa & sb) / len(sa | sb)
|
||||
if jac >= SIM_THRESHOLD:
|
||||
edges.append(
|
||||
{
|
||||
"from": b, # later
|
||||
"to": a, # earlier
|
||||
"jaccard": round(jac, 4),
|
||||
"shared": sorted(sa & sb),
|
||||
}
|
||||
)
|
||||
edges.sort(key=lambda e: (-e["jaccard"], e["from"], e["to"]))
|
||||
|
||||
# hash-chained append-only edge journal
|
||||
journal_path = os.path.join(HERE, "edges.jsonl")
|
||||
prev_hash = "0" * 64
|
||||
with open(journal_path, "w") as f:
|
||||
for seq, e in enumerate(edges, start=1):
|
||||
row = {"seq": seq, "from": e["from"], "to": e["to"], "jaccard": e["jaccard"], "prev_hash": prev_hash}
|
||||
h = sha256_bytes(json.dumps(row, sort_keys=True, separators=(",", ":")).encode())
|
||||
row["hash"] = h
|
||||
f.write(json.dumps(row, sort_keys=True) + "\n")
|
||||
prev_hash = h
|
||||
|
||||
head = prev_hash
|
||||
manifest = {
|
||||
"artifact": "archive-lcmm-corpus-2026-10-26",
|
||||
"lane": "Archive -- multimodal, long-context research",
|
||||
"built_utc": stamp,
|
||||
"builder": "build_corpus.py",
|
||||
"fetch": fetch_meta,
|
||||
"counts": {
|
||||
"records_fetched": len(all_recs),
|
||||
"distinct_nodes": len(nodes),
|
||||
"similarity_edges": len(edges),
|
||||
"threshold": SIM_THRESHOLD,
|
||||
"nodes_with_out_edges": len({e["from"] for e in edges}),
|
||||
"nodes_with_in_edges": len({e["to"] for e in edges}),
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"node": canon(r["arxiv_id"]),
|
||||
"arxiv_id": r["arxiv_id"],
|
||||
"title": r["title"],
|
||||
"published": r["published"],
|
||||
"updated": r["updated"],
|
||||
"primary_category": r["primary_category"],
|
||||
"categories": r["categories"],
|
||||
"doi": r["doi"],
|
||||
"n_authors": len(r["authors"]),
|
||||
"authors": r["authors"],
|
||||
"abstract_chars": len(r["abstract"]),
|
||||
"slice": r["slice"],
|
||||
"proof": r["proof"],
|
||||
}
|
||||
for r in nodes
|
||||
],
|
||||
"edges": edges,
|
||||
"journal": {
|
||||
"path": "edges.jsonl",
|
||||
"entries": len(edges),
|
||||
"genesis": "0" * 64,
|
||||
"head": head,
|
||||
},
|
||||
}
|
||||
with open(os.path.join(HERE, "corpus.json"), "w") as f:
|
||||
json.dump(manifest, f, indent=2, sort_keys=True)
|
||||
|
||||
print(f"[build] fetched {len(all_recs)} records -> {len(nodes)} nodes")
|
||||
print(f"[build] similarity edges (>= {SIM_THRESHOLD}): {len(edges)}")
|
||||
print(f"[build] edge journal head: {head[:16]}...")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env python3
|
||||
"""build_report.py -- render REPORT.md from corpus.json (measured, not hand-written)."""
|
||||
import json
|
||||
import os
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
m = json.load(open(os.path.join(HERE, "corpus.json")))
|
||||
nodes = m["nodes"]
|
||||
edges = m["edges"]
|
||||
|
||||
by_slice = {}
|
||||
for n in nodes:
|
||||
by_slice[n["slice"]] = by_slice.get(n["slice"], 0) + 1
|
||||
|
||||
lines = []
|
||||
A = lines.append
|
||||
A(f"# {m['artifact']}")
|
||||
A("")
|
||||
A(f"Built {m['built_utc']} by `build_corpus.py`.")
|
||||
A("")
|
||||
A("## What this is")
|
||||
A("")
|
||||
A("A provenance-anchored research corpus over the **multimodal / long-context**")
|
||||
A("lane, built from a **live arXiv API fetch** on the build date. It is not a")
|
||||
A("scrape-and-forget dump: every derived record carries a verbatim proof that")
|
||||
A("re-derives from the frozen raw bytes, and the edge journal is hash-chained.")
|
||||
A("")
|
||||
A("## Provenance model")
|
||||
A("")
|
||||
A("* The raw arXiv API response for each slice is frozen under `source/` with")
|
||||
A(" its sha256 and byte count recorded in the manifest.")
|
||||
A("* Each node records `proof.line` (1-based line of its `<entry>` in the raw")
|
||||
A(" XML) and `proof.block_sha256` (sha256 of that entry's exact byte block).")
|
||||
A("* A verifier re-derives line, block hash, and title straight from the frozen")
|
||||
A(" bytes — it does not trust `corpus.json`.")
|
||||
A("")
|
||||
A("## Fetches (measured)")
|
||||
A("")
|
||||
A("| slice | query | requested | returned | bytes | sha256 |")
|
||||
A("|---|---|---|---|---|---|")
|
||||
for fm in m["fetch"]:
|
||||
A(f"| {fm['label']} | `{fm['query']}` | {fm['requested_max']} | {fm['returned']} "
|
||||
f"| {fm['raw_bytes']} | `{fm['raw_sha256'][:16]}…` |")
|
||||
A("")
|
||||
A("## Counts")
|
||||
A("")
|
||||
A(f"* records fetched: **{m['counts']['records_fetched']}**")
|
||||
A(f"* distinct nodes: **{m['counts']['distinct_nodes']}**")
|
||||
A(f"* similarity edges (Jaccard >= {m['counts']['threshold']}): **{m['counts']['similarity_edges']}**")
|
||||
A(f"* nodes with outgoing edges: {m['counts']['nodes_with_out_edges']}")
|
||||
A(f"* nodes with incoming edges: {m['counts']['nodes_with_in_edges']}")
|
||||
A("")
|
||||
A("Per-slice node counts: " + ", ".join(f"`{k}`={v}" for k, v in sorted(by_slice.items())))
|
||||
A("")
|
||||
A("## Edge journal")
|
||||
A("")
|
||||
A(f"* entries: **{m['journal']['entries']}**")
|
||||
A(f"* genesis: `{'0'*64}`")
|
||||
A(f"* head: `{m['journal']['head']}`")
|
||||
A("* construction: `hash_n = sha256(json({seq, from, to, jaccard, prev_hash}))`,")
|
||||
A(" `prev_hash_0 = '0'*64` — an append-only chain.")
|
||||
A("")
|
||||
A("## Top edges by title similarity")
|
||||
A("")
|
||||
A("| jaccard | later (from) | earlier (to) | shared stems |")
|
||||
A("|---|---|---|---|")
|
||||
for e in edges[:15]:
|
||||
A(f"| {e['jaccard']:.3f} | `{e['from']}` | `{e['to']}` | {', '.join(e['shared'][:6])} |")
|
||||
A("")
|
||||
A("## Verdicts")
|
||||
A("")
|
||||
A("Three independent checks, all green at build time:")
|
||||
A("")
|
||||
A("* `run_verdict.py` — 22 internal-integrity checks (proofs, chain, DAG, counts)")
|
||||
A("* `indep_check.py` — re-fetches live arXiv by `id_list` and re-parses the frozen")
|
||||
A(" sources with regex (not ElementTree); corroborates all 279 titles")
|
||||
A("* `mutation_battery.py` — 11 single-field corruptions, all 11 caught")
|
||||
A("")
|
||||
A("Run everything with one command:")
|
||||
A("")
|
||||
A("```")
|
||||
A("bash CHECK.sh")
|
||||
A("```")
|
||||
A("")
|
||||
A("## Files")
|
||||
A("")
|
||||
A("| file | role |")
|
||||
A("|---|---|")
|
||||
A("| `build_corpus.py` | fetches arXiv, builds corpus + chained journal |")
|
||||
A("| `corpus.json` | the manifest (nodes, proofs, edges, journal head) |")
|
||||
A("| `edges.jsonl` | append-only hash-chained edge journal |")
|
||||
A("| `source/arxiv-*.xml` | frozen raw API responses (sha256-pinned) |")
|
||||
A("| `run_verdict.py` | internal integrity verifier (22 checks) |")
|
||||
A("| `indep_check.py` | live-arXiv corroboration + second from-scratch parse |")
|
||||
A("| `mutation_battery.py` | falsifiability harness (11 mutations) |")
|
||||
A("| `CHECK.sh` | one-command verdict |")
|
||||
|
||||
open(os.path.join(HERE, "REPORT.md"), "w").write("\n".join(lines) + "\n")
|
||||
print(f"[report] wrote REPORT.md ({len(lines)} lines)")
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,94 @@
|
||||
{"from": "2610.32263", "hash": "c0d2785a6b02cceda12c72cdb8c61bdffe0f5b9a6e3d4bc89c21cae7b39f71ce", "jaccard": 0.5556, "prev_hash": "0000000000000000000000000000000000000000000000000000000000000000", "seq": 1, "to": "2610.27298"}
|
||||
{"from": "2610.31205", "hash": "957657371b688031807cab83207a6051121fd0d07e22513004e3b1268efa0d8a", "jaccard": 0.4167, "prev_hash": "c0d2785a6b02cceda12c72cdb8c61bdffe0f5b9a6e3d4bc89c21cae7b39f71ce", "seq": 2, "to": "2610.25005"}
|
||||
{"from": "2610.11592", "hash": "d3d880c2ee63701312dc2c3a0035bb9f29fb340acacd5c5cbca9ec1aad48a5a5", "jaccard": 0.3636, "prev_hash": "957657371b688031807cab83207a6051121fd0d07e22513004e3b1268efa0d8a", "seq": 3, "to": "2609.38137"}
|
||||
{"from": "2610.32263", "hash": "2f510363a8d3e72a9ebaaea28bed3ddfd9b807a9fc72f045f566aec50c98247d", "jaccard": 0.3636, "prev_hash": "d3d880c2ee63701312dc2c3a0035bb9f29fb340acacd5c5cbca9ec1aad48a5a5", "seq": 4, "to": "2610.27904"}
|
||||
{"from": "2609.36529", "hash": "3793a445542d163daa6a2173e63e81d5db0f7d43c75df2e69b6718e3a68ccdbd", "jaccard": 0.3333, "prev_hash": "2f510363a8d3e72a9ebaaea28bed3ddfd9b807a9fc72f045f566aec50c98247d", "seq": 5, "to": "2609.36314"}
|
||||
{"from": "2610.17966", "hash": "0038bd6e8600148342b08e8755f689f199a34689a60dc6632d61a93dbca3065f", "jaccard": 0.3333, "prev_hash": "3793a445542d163daa6a2173e63e81d5db0f7d43c75df2e69b6718e3a68ccdbd", "seq": 6, "to": "2610.06479"}
|
||||
{"from": "2610.19928", "hash": "2e6462261593b5b72d9198ddf318334558cbb08bc1c138d0238de8fd510a7d8a", "jaccard": 0.3333, "prev_hash": "0038bd6e8600148342b08e8755f689f199a34689a60dc6632d61a93dbca3065f", "seq": 7, "to": "2609.36935"}
|
||||
{"from": "2610.26203", "hash": "0cedec4ce66fb53831be0a813144c0dc9b695f8243e280f1b822fa8d483d455d", "jaccard": 0.3333, "prev_hash": "2e6462261593b5b72d9198ddf318334558cbb08bc1c138d0238de8fd510a7d8a", "seq": 8, "to": "2609.25611"}
|
||||
{"from": "2610.31304", "hash": "934ce7f3ef23532ceb51137a92bc147d577ecaf8b0f9c587c1e487caf5b8e31e", "jaccard": 0.3333, "prev_hash": "0cedec4ce66fb53831be0a813144c0dc9b695f8243e280f1b822fa8d483d455d", "seq": 9, "to": "2610.29642"}
|
||||
{"from": "2610.31494", "hash": "e7f371b93a0ded3286fc076a551f61b28e4a62572072880f69c15a3ad92a7069", "jaccard": 0.3333, "prev_hash": "934ce7f3ef23532ceb51137a92bc147d577ecaf8b0f9c587c1e487caf5b8e31e", "seq": 10, "to": "2610.26826"}
|
||||
{"from": "2610.32440", "hash": "b5d18bf5c1e5bc53a63294ef4e3ab6b236a8cfcccabf527962afe3793bd6e08a", "jaccard": 0.3333, "prev_hash": "e7f371b93a0ded3286fc076a551f61b28e4a62572072880f69c15a3ad92a7069", "seq": 11, "to": "2610.32220"}
|
||||
{"from": "2610.27904", "hash": "1c652fa03156a18ad9d3e1b4bcbb6d2ef626e3c793a05979849a0c86821cae6c", "jaccard": 0.3077, "prev_hash": "b5d18bf5c1e5bc53a63294ef4e3ab6b236a8cfcccabf527962afe3793bd6e08a", "seq": 12, "to": "2610.27298"}
|
||||
{"from": "2610.28626", "hash": "adf08df5b84c667b03651b920064149badabf7c549a0f613836d178916ba6a12", "jaccard": 0.3077, "prev_hash": "1c652fa03156a18ad9d3e1b4bcbb6d2ef626e3c793a05979849a0c86821cae6c", "seq": 13, "to": "2610.27782"}
|
||||
{"from": "2610.31158", "hash": "e33be759dac78f2f14a18fbabf268d0fd746924c9af34627a2aa19f5007ccbd8", "jaccard": 0.3077, "prev_hash": "adf08df5b84c667b03651b920064149badabf7c549a0f613836d178916ba6a12", "seq": 14, "to": "2610.25926"}
|
||||
{"from": "2610.32486", "hash": "2fad0dff9b4e857b50d2712561e70d824a42aa02b88cbedb24ee57f32200c80b", "jaccard": 0.3077, "prev_hash": "e33be759dac78f2f14a18fbabf268d0fd746924c9af34627a2aa19f5007ccbd8", "seq": 15, "to": "2610.31478"}
|
||||
{"from": "2610.10114", "hash": "16b2afa9c818375c3c6e03efced15244f4e1a56711e36f4d18d09e02fc1451d2", "jaccard": 0.3, "prev_hash": "2fad0dff9b4e857b50d2712561e70d824a42aa02b88cbedb24ee57f32200c80b", "seq": 16, "to": "2610.07940"}
|
||||
{"from": "2610.19026", "hash": "b36589cd0ba8ce14ac4de3b5747d1e965fcb82280a3f64b6e201444b9f24d7c3", "jaccard": 0.3, "prev_hash": "16b2afa9c818375c3c6e03efced15244f4e1a56711e36f4d18d09e02fc1451d2", "seq": 17, "to": "2610.00418"}
|
||||
{"from": "2610.19928", "hash": "4d90c6fd47b9ad947d9d799e5914c1e04e6724c4f84cda18241ed7535707f276", "jaccard": 0.3, "prev_hash": "b36589cd0ba8ce14ac4de3b5747d1e965fcb82280a3f64b6e201444b9f24d7c3", "seq": 18, "to": "2609.38809"}
|
||||
{"from": "2610.25247", "hash": "f770df43866ee02413278f2dc6f26707a071902da61723339d6d6fd5166e9b20", "jaccard": 0.3, "prev_hash": "4d90c6fd47b9ad947d9d799e5914c1e04e6724c4f84cda18241ed7535707f276", "seq": 19, "to": "2610.02181"}
|
||||
{"from": "2610.26431", "hash": "18c69ce71cb111beb6f39bb142c3c17ec8cb6b3901c077b7bc7d9464ad691733", "jaccard": 0.3, "prev_hash": "f770df43866ee02413278f2dc6f26707a071902da61723339d6d6fd5166e9b20", "seq": 20, "to": "2609.38137"}
|
||||
{"from": "2610.28282", "hash": "1c70d2dbe52563a357ea89f9696b5d3de7dad11c6399e03c163c97f849127fa6", "jaccard": 0.3, "prev_hash": "18c69ce71cb111beb6f39bb142c3c17ec8cb6b3901c077b7bc7d9464ad691733", "seq": 21, "to": "2609.38137"}
|
||||
{"from": "2610.28282", "hash": "44ac30a03f25704e0a913230f09f778ee180d3e8eb303295663552861e5c0c4a", "jaccard": 0.3, "prev_hash": "1c70d2dbe52563a357ea89f9696b5d3de7dad11c6399e03c163c97f849127fa6", "seq": 22, "to": "2610.22067"}
|
||||
{"from": "2610.30097", "hash": "8b10323bfeb2a1a3ae208c3c0e377b8f6fbfdc514833716145ec69cdd5c098dd", "jaccard": 0.3, "prev_hash": "44ac30a03f25704e0a913230f09f778ee180d3e8eb303295663552861e5c0c4a", "seq": 23, "to": "2609.38809"}
|
||||
{"from": "2610.30132", "hash": "2a34229fb3839c81e001ddad840c1bf5c324d1c5493ae3c1635cf15d98ae2d00", "jaccard": 0.3, "prev_hash": "8b10323bfeb2a1a3ae208c3c0e377b8f6fbfdc514833716145ec69cdd5c098dd", "seq": 24, "to": "2609.38809"}
|
||||
{"from": "2610.32213", "hash": "eff84b3e35cf2dac2882720c744d6ca3fbc4ce4ac2e4f32a235b9c4978818403", "jaccard": 0.3, "prev_hash": "2a34229fb3839c81e001ddad840c1bf5c324d1c5493ae3c1635cf15d98ae2d00", "seq": 25, "to": "2610.08463"}
|
||||
{"from": "2610.32213", "hash": "b27eb9da95cfa254328cd9124b4ccd81d63f78d80bee254822c2b025b5bd1028", "jaccard": 0.3, "prev_hash": "eff84b3e35cf2dac2882720c744d6ca3fbc4ce4ac2e4f32a235b9c4978818403", "seq": 26, "to": "2610.26431"}
|
||||
{"from": "2610.32263", "hash": "6820eabea4aa4e305bd6cbf2728026b7e7511f447def9b3511786a386bbda210", "jaccard": 0.3, "prev_hash": "b27eb9da95cfa254328cd9124b4ccd81d63f78d80bee254822c2b025b5bd1028", "seq": 27, "to": "2610.29495"}
|
||||
{"from": "2610.32332", "hash": "5c820211d28f3a1ec2449771c05320dd864a6c0b97f56cbea271c2204bb86f90", "jaccard": 0.3, "prev_hash": "6820eabea4aa4e305bd6cbf2728026b7e7511f447def9b3511786a386bbda210", "seq": 28, "to": "2610.28282"}
|
||||
{"from": "2610.32360", "hash": "37d47d49714c7e7a9acf52b9ff3ffc3b23e5de0abbbe07e3e86a68c5fc357b13", "jaccard": 0.3, "prev_hash": "5c820211d28f3a1ec2449771c05320dd864a6c0b97f56cbea271c2204bb86f90", "seq": 29, "to": "2610.31304"}
|
||||
{"from": "2610.32784", "hash": "5fd9515cb74eebed9abdac418df344693632766c54937ea66318509d4ec82365", "jaccard": 0.3, "prev_hash": "37d47d49714c7e7a9acf52b9ff3ffc3b23e5de0abbbe07e3e86a68c5fc357b13", "seq": 30, "to": "2610.30089"}
|
||||
{"from": "2610.06479", "hash": "84fd4315ba0aade327d63c8d49fab2ebcdeaa4bb5e607d4b46eb523c2976ec1a", "jaccard": 0.2857, "prev_hash": "5fd9515cb74eebed9abdac418df344693632766c54937ea66318509d4ec82365", "seq": 31, "to": "2610.03027"}
|
||||
{"from": "2610.12338", "hash": "fb24a7a3248048c57d59bea19537f9103f9267fe4379836520375ce44dbfd9c4", "jaccard": 0.2857, "prev_hash": "84fd4315ba0aade327d63c8d49fab2ebcdeaa4bb5e607d4b46eb523c2976ec1a", "seq": 32, "to": "2610.06479"}
|
||||
{"from": "2610.19247", "hash": "ebff381aeecc3ac4576a994a6637bbde9241d07be69a7adb1af2df5f9b388f87", "jaccard": 0.2857, "prev_hash": "fb24a7a3248048c57d59bea19537f9103f9267fe4379836520375ce44dbfd9c4", "seq": 33, "to": "2610.07940"}
|
||||
{"from": "2610.26431", "hash": "1ddda6d6943dd7c06761372b9ecce4834e8da419665e90e3c56d78805c169ebc", "jaccard": 0.2857, "prev_hash": "ebff381aeecc3ac4576a994a6637bbde9241d07be69a7adb1af2df5f9b388f87", "seq": 34, "to": "2610.19247"}
|
||||
{"from": "2610.26863", "hash": "602ea694f0f93b1e85cb8d9cf0c666d0da176f2414b467b31b73d316fb98b40d", "jaccard": 0.2857, "prev_hash": "1ddda6d6943dd7c06761372b9ecce4834e8da419665e90e3c56d78805c169ebc", "seq": 35, "to": "2610.19247"}
|
||||
{"from": "2610.27341", "hash": "c2120f5946926ce98fffe15117674e31d8655514be584483f27c1299c9e65d74", "jaccard": 0.2857, "prev_hash": "602ea694f0f93b1e85cb8d9cf0c666d0da176f2414b467b31b73d316fb98b40d", "seq": 36, "to": "2610.25999"}
|
||||
{"from": "2610.31098", "hash": "9fccd40736c7b5e89ee08812ed1733251a6d9a4b94bfebda4d34e4a0b844e4b3", "jaccard": 0.2857, "prev_hash": "c2120f5946926ce98fffe15117674e31d8655514be584483f27c1299c9e65d74", "seq": 37, "to": "2610.06479"}
|
||||
{"from": "2610.10156", "hash": "99d27fc38f27967781b5c0f9f201de2fbb4c8a313609806f6d94b386dd15203d", "jaccard": 0.2727, "prev_hash": "9fccd40736c7b5e89ee08812ed1733251a6d9a4b94bfebda4d34e4a0b844e4b3", "seq": 38, "to": "2610.05940"}
|
||||
{"from": "2610.19928", "hash": "c543fe695199079c3cc1ade4a8f5b80023ec1091e728b14a616e8ae234f15715", "jaccard": 0.2727, "prev_hash": "99d27fc38f27967781b5c0f9f201de2fbb4c8a313609806f6d94b386dd15203d", "seq": 39, "to": "2609.39740"}
|
||||
{"from": "2610.22067", "hash": "38c4b463c6f66566aca397faf76abe2bf88d829414cd0e032f7e9a80fde87529", "jaccard": 0.2727, "prev_hash": "c543fe695199079c3cc1ade4a8f5b80023ec1091e728b14a616e8ae234f15715", "seq": 40, "to": "2610.13132"}
|
||||
{"from": "2610.26431", "hash": "3a67e5d0dcb9b47ceb040ec9fd9b643d7617ff74a9a09a8b5261a300d3e2ec36", "jaccard": 0.2727, "prev_hash": "38c4b463c6f66566aca397faf76abe2bf88d829414cd0e032f7e9a80fde87529", "seq": 41, "to": "2610.11592"}
|
||||
{"from": "2610.26585", "hash": "0a0273e86c10b1e2804025c04f3ac4aaff30a4fb23ba3281c0387c53b384d597", "jaccard": 0.2727, "prev_hash": "3a67e5d0dcb9b47ceb040ec9fd9b643d7617ff74a9a09a8b5261a300d3e2ec36", "seq": 42, "to": "2610.23250"}
|
||||
{"from": "2610.27782", "hash": "10a976097e4050157ec738f66d31d4037253f92736df50ab16df8f90003d7e1e", "jaccard": 0.2727, "prev_hash": "0a0273e86c10b1e2804025c04f3ac4aaff30a4fb23ba3281c0387c53b384d597", "seq": 43, "to": "2610.26431"}
|
||||
{"from": "2610.27782", "hash": "38026681ca472dc7468e92d93e57fbbc2cdcc305569aa11c6876dc45fe0aa18e", "jaccard": 0.2727, "prev_hash": "10a976097e4050157ec738f66d31d4037253f92736df50ab16df8f90003d7e1e", "seq": 44, "to": "2610.26863"}
|
||||
{"from": "2610.28382", "hash": "70ee30e0d3cddcf4c72ac32acb90bc7dfc9d0172109e1c8792b5ea1d5bf95a68", "jaccard": 0.2727, "prev_hash": "38026681ca472dc7468e92d93e57fbbc2cdcc305569aa11c6876dc45fe0aa18e", "seq": 45, "to": "2609.31382"}
|
||||
{"from": "2610.28382", "hash": "005ade964a2d46368dc92f9f96ba887eccbf2fbf28b27be5fdbba8bb82c09353", "jaccard": 0.2727, "prev_hash": "70ee30e0d3cddcf4c72ac32acb90bc7dfc9d0172109e1c8792b5ea1d5bf95a68", "seq": 46, "to": "2610.24378"}
|
||||
{"from": "2610.29590", "hash": "c0eb47317b9cbd771441442397b5931eceb912d329bf00fb7fe23edf5d3f350b", "jaccard": 0.2727, "prev_hash": "005ade964a2d46368dc92f9f96ba887eccbf2fbf28b27be5fdbba8bb82c09353", "seq": 47, "to": "2610.26585"}
|
||||
{"from": "2610.30108", "hash": "4c8fe3ba13ac91982ba93162e1d5371b8735c82ac88a1b6f00a2f827a8cfed21", "jaccard": 0.2727, "prev_hash": "c0eb47317b9cbd771441442397b5931eceb912d329bf00fb7fe23edf5d3f350b", "seq": 48, "to": "2610.24378"}
|
||||
{"from": "2610.30132", "hash": "150fd76d53de2edd28901220b4a4894a0239cebda8fcb0948eea899c7a2dbc5b", "jaccard": 0.2727, "prev_hash": "4c8fe3ba13ac91982ba93162e1d5371b8735c82ac88a1b6f00a2f827a8cfed21", "seq": 49, "to": "2610.23167"}
|
||||
{"from": "2610.32213", "hash": "4d0c7b7482c0357ecbb275497dcc8261f31cdf4d6842f17a589637b49f117aea", "jaccard": 0.2727, "prev_hash": "150fd76d53de2edd28901220b4a4894a0239cebda8fcb0948eea899c7a2dbc5b", "seq": 50, "to": "2609.38137"}
|
||||
{"from": "2610.32311", "hash": "598fb0e3375c458e2c0f48af4e8a7372d1dd1e842819c4b99676ea366edfb96a", "jaccard": 0.2727, "prev_hash": "4d0c7b7482c0357ecbb275497dcc8261f31cdf4d6842f17a589637b49f117aea", "seq": 51, "to": "2610.32263"}
|
||||
{"from": "2610.32332", "hash": "cfd8baaa540b55d0f007b463b03735faa3dd448d0b54dfe0225152d8987bad79", "jaccard": 0.2727, "prev_hash": "598fb0e3375c458e2c0f48af4e8a7372d1dd1e842819c4b99676ea366edfb96a", "seq": 52, "to": "2609.38137"}
|
||||
{"from": "2610.32332", "hash": "16e71923cc9422066232dcc7cb19b2eb267c041f714b41139d7087ec5ed7defd", "jaccard": 0.2727, "prev_hash": "cfd8baaa540b55d0f007b463b03735faa3dd448d0b54dfe0225152d8987bad79", "seq": 53, "to": "2610.02472"}
|
||||
{"from": "2610.32485", "hash": "4f94d0235636505efd115b49b31e114ffa43ea93326bdf55879fc53b3cf94ce9", "jaccard": 0.2727, "prev_hash": "16e71923cc9422066232dcc7cb19b2eb267c041f714b41139d7087ec5ed7defd", "seq": 54, "to": "2610.27360"}
|
||||
{"from": "2610.32612", "hash": "4613acaa92879f528af026db5cc1ae79cd512de81d00167264a783820e17b029", "jaccard": 0.2727, "prev_hash": "4f94d0235636505efd115b49b31e114ffa43ea93326bdf55879fc53b3cf94ce9", "seq": 55, "to": "2610.29642"}
|
||||
{"from": "2610.32784", "hash": "89ddfcccfde591d4a3fdd87b1b7bcab6b38cd4cfc1008adc51d79d0e354f5245", "jaccard": 0.2727, "prev_hash": "4613acaa92879f528af026db5cc1ae79cd512de81d00167264a783820e17b029", "seq": 56, "to": "2610.05940"}
|
||||
{"from": "2610.21165", "hash": "90bec0558c471ccc576284a865566e1ad74da33ffa28e7a94f0d213d1a09c6ab", "jaccard": 0.2667, "prev_hash": "89ddfcccfde591d4a3fdd87b1b7bcab6b38cd4cfc1008adc51d79d0e354f5245", "seq": 57, "to": "2609.29370"}
|
||||
{"from": "2610.28264", "hash": "1f15742a9fee9adf803f86fc277c01798e69c88002a9b47e65b7be2626ef5578", "jaccard": 0.2667, "prev_hash": "90bec0558c471ccc576284a865566e1ad74da33ffa28e7a94f0d213d1a09c6ab", "seq": 58, "to": "2610.25005"}
|
||||
{"from": "2610.28264", "hash": "c96471d12ef21c73ae9f7b172e49474c3ec030336f14dde69773bbefc1da8f6f", "jaccard": 0.2667, "prev_hash": "1f15742a9fee9adf803f86fc277c01798e69c88002a9b47e65b7be2626ef5578", "seq": 59, "to": "2610.27782"}
|
||||
{"from": "2610.31205", "hash": "a22e89c6c267f72015cc140bfdd1d878fca3cb53e9783db1c25670be327dd7df", "jaccard": 0.2667, "prev_hash": "c96471d12ef21c73ae9f7b172e49474c3ec030336f14dde69773bbefc1da8f6f", "seq": 60, "to": "2610.00969"}
|
||||
{"from": "2610.32213", "hash": "d00a635764924e7402f8d5e1527f13c9dcb2498aa11cf568ce2c360256eef068", "jaccard": 0.2667, "prev_hash": "a22e89c6c267f72015cc140bfdd1d878fca3cb53e9783db1c25670be327dd7df", "seq": 61, "to": "2609.35831"}
|
||||
{"from": "2610.00982", "hash": "1ef5f897e2ae1d8cd43e9e8ff1305b974a7bc92c6c20f02c3d5fafe2e0dcfa39", "jaccard": 0.25, "prev_hash": "d00a635764924e7402f8d5e1527f13c9dcb2498aa11cf568ce2c360256eef068", "seq": 62, "to": "2609.39740"}
|
||||
{"from": "2610.06479", "hash": "deec88bc693b903ec5c0bbb0a1b0a9cefa518e861040505ffcf061583e82fa24", "jaccard": 0.25, "prev_hash": "1ef5f897e2ae1d8cd43e9e8ff1305b974a7bc92c6c20f02c3d5fafe2e0dcfa39", "seq": 63, "to": "2610.03198"}
|
||||
{"from": "2610.06479", "hash": "c3cf55b318faeec630c7f432a743c35aeba9882d4ff8d2b5c6090818622d110d", "jaccard": 0.25, "prev_hash": "deec88bc693b903ec5c0bbb0a1b0a9cefa518e861040505ffcf061583e82fa24", "seq": 64, "to": "2610.06286"}
|
||||
{"from": "2610.06479", "hash": "815513954e85fa177c422487ce87cad0341e81ee93206e27d8e0e5351e3ad2d6", "jaccard": 0.25, "prev_hash": "c3cf55b318faeec630c7f432a743c35aeba9882d4ff8d2b5c6090818622d110d", "seq": 65, "to": "2610.06927"}
|
||||
{"from": "2610.10918", "hash": "a2da1eb421a8e7dec089f060793ce901c9b1ffd485c78b4a6a3840b3f908d1b3", "jaccard": 0.25, "prev_hash": "815513954e85fa177c422487ce87cad0341e81ee93206e27d8e0e5351e3ad2d6", "seq": 66, "to": "2610.00969"}
|
||||
{"from": "2610.15542", "hash": "f58297056629f83f975b12c631fdc064ec0551e2de55822a727065431f360d5d", "jaccard": 0.25, "prev_hash": "a2da1eb421a8e7dec089f060793ce901c9b1ffd485c78b4a6a3840b3f908d1b3", "seq": 67, "to": "2610.15371"}
|
||||
{"from": "2610.19247", "hash": "64f383299793eba4d60511d0cda1f68d242741e2b864f1d0b36aacd2f79d4296", "jaccard": 0.25, "prev_hash": "f58297056629f83f975b12c631fdc064ec0551e2de55822a727065431f360d5d", "seq": 68, "to": "2609.33465"}
|
||||
{"from": "2610.19247", "hash": "9ca646214fa5543a4d53bd27c58988c6c47bb7a6ca674280c23cbd619a26100f", "jaccard": 0.25, "prev_hash": "64f383299793eba4d60511d0cda1f68d242741e2b864f1d0b36aacd2f79d4296", "seq": 69, "to": "2609.38137"}
|
||||
{"from": "2610.25005", "hash": "128dd7f04609fa79711f7b42fbfc37c1c94f0d57581a7d77c829ac65ce62aa15", "jaccard": 0.25, "prev_hash": "9ca646214fa5543a4d53bd27c58988c6c47bb7a6ca674280c23cbd619a26100f", "seq": 70, "to": "2609.33465"}
|
||||
{"from": "2610.25919", "hash": "3b3eb0a6d97dbf870162963e8cc643e50a658fa1b81bef333bb06f063dd4623b", "jaccard": 0.25, "prev_hash": "128dd7f04609fa79711f7b42fbfc37c1c94f0d57581a7d77c829ac65ce62aa15", "seq": 71, "to": "2609.38809"}
|
||||
{"from": "2610.26431", "hash": "feb6146f2ab614aad76bd71aafa0c0f92909a5d2a0aa9669b9d04734dbb579df", "jaccard": 0.25, "prev_hash": "3b3eb0a6d97dbf870162963e8cc643e50a658fa1b81bef333bb06f063dd4623b", "seq": 72, "to": "2609.36529"}
|
||||
{"from": "2610.26431", "hash": "c5c7f5dd3453bb9aa8b62300564fe1c495e6c0fd849a5972654e35be475733af", "jaccard": 0.25, "prev_hash": "feb6146f2ab614aad76bd71aafa0c0f92909a5d2a0aa9669b9d04734dbb579df", "seq": 73, "to": "2610.13041"}
|
||||
{"from": "2610.26431", "hash": "45a6404233a651c60f113e4074bf13c8d81670e42aff4a4a6b774c046f388adc", "jaccard": 0.25, "prev_hash": "c5c7f5dd3453bb9aa8b62300564fe1c495e6c0fd849a5972654e35be475733af", "seq": 74, "to": "2610.21165"}
|
||||
{"from": "2610.27341", "hash": "8aded827378b4919e216949f4683a680ed62735f0e7aadb045794b52ee27791d", "jaccard": 0.25, "prev_hash": "45a6404233a651c60f113e4074bf13c8d81670e42aff4a4a6b774c046f388adc", "seq": 75, "to": "2610.25247"}
|
||||
{"from": "2610.27692", "hash": "c85e917aa8db9a3084489ff9b76d4dee86dc3d61da9a2c22b25f843bfcf116ae", "jaccard": 0.25, "prev_hash": "8aded827378b4919e216949f4683a680ed62735f0e7aadb045794b52ee27791d", "seq": 76, "to": "2610.05940"}
|
||||
{"from": "2610.27692", "hash": "c5a5fdc4462100c095f40e8718b350cdf5738db128a46f84586098adba2133a2", "jaccard": 0.25, "prev_hash": "c85e917aa8db9a3084489ff9b76d4dee86dc3d61da9a2c22b25f843bfcf116ae", "seq": 77, "to": "2610.10156"}
|
||||
{"from": "2610.27782", "hash": "01ea1d41b65164b086ac3d3f827dd498bd94e1b34430ff13e4fb5a43e8f80ab5", "jaccard": 0.25, "prev_hash": "c5a5fdc4462100c095f40e8718b350cdf5738db128a46f84586098adba2133a2", "seq": 78, "to": "2610.26585"}
|
||||
{"from": "2610.29495", "hash": "12627418754763093e3e93d56d80951dc8575d3bcb66671ffacd4fa6a0d74262", "jaccard": 0.25, "prev_hash": "01ea1d41b65164b086ac3d3f827dd498bd94e1b34430ff13e4fb5a43e8f80ab5", "seq": 79, "to": "2610.27298"}
|
||||
{"from": "2610.29590", "hash": "b7074562ff3565b1e588493c7ccf79d98e99cd1bb3739c84b57976cf159ab40c", "jaccard": 0.25, "prev_hash": "12627418754763093e3e93d56d80951dc8575d3bcb66671ffacd4fa6a0d74262", "seq": 80, "to": "2610.26022"}
|
||||
{"from": "2610.29590", "hash": "5726c98bbed7af9235c858b1fc7dbb7235123daa4bf15d640b51967a5d149e78", "jaccard": 0.25, "prev_hash": "b7074562ff3565b1e588493c7ccf79d98e99cd1bb3739c84b57976cf159ab40c", "seq": 81, "to": "2610.27782"}
|
||||
{"from": "2610.30206", "hash": "27ec221bbf7953fe92ff73955c22c44309f5a4050c00bd1043cb53418fc23908", "jaccard": 0.25, "prev_hash": "5726c98bbed7af9235c858b1fc7dbb7235123daa4bf15d640b51967a5d149e78", "seq": 82, "to": "2610.06479"}
|
||||
{"from": "2610.30913", "hash": "b149f5d440aa18d85887aa57e8bfd2a322c95a0ff2cba6b38d3e7283f064bd1e", "jaccard": 0.25, "prev_hash": "27ec221bbf7953fe92ff73955c22c44309f5a4050c00bd1043cb53418fc23908", "seq": 83, "to": "2610.30592"}
|
||||
{"from": "2610.31098", "hash": "6fb0e6797353702c763b2cd585f846c53675c8d115478eb7fc900c7a61e4d668", "jaccard": 0.25, "prev_hash": "b149f5d440aa18d85887aa57e8bfd2a322c95a0ff2cba6b38d3e7283f064bd1e", "seq": 84, "to": "2610.20582"}
|
||||
{"from": "2610.31158", "hash": "f5c3b3d8264d7d5c17c31d857d3b7a1d159fbe3ed01af18731c3c7398ac061ef", "jaccard": 0.25, "prev_hash": "6fb0e6797353702c763b2cd585f846c53675c8d115478eb7fc900c7a61e4d668", "seq": 85, "to": "2610.25493"}
|
||||
{"from": "2610.31205", "hash": "91d861f3113778c2ff32e8072a4f0383f99ffa9dd35267ad760a08472b06fed6", "jaccard": 0.25, "prev_hash": "f5c3b3d8264d7d5c17c31d857d3b7a1d159fbe3ed01af18731c3c7398ac061ef", "seq": 86, "to": "2610.28264"}
|
||||
{"from": "2610.31945", "hash": "5df7dafd79141080d3ed7a0a03b6be33df2800b66188f8af31d2f771f3b73ec0", "jaccard": 0.25, "prev_hash": "91d861f3113778c2ff32e8072a4f0383f99ffa9dd35267ad760a08472b06fed6", "seq": 87, "to": "2610.15371"}
|
||||
{"from": "2610.32177", "hash": "8001e13fe0582edf31cb651c0874c6848622c35de5082cc769d95f1b1318322c", "jaccard": 0.25, "prev_hash": "5df7dafd79141080d3ed7a0a03b6be33df2800b66188f8af31d2f771f3b73ec0", "seq": 88, "to": "2610.29811"}
|
||||
{"from": "2610.32213", "hash": "678347bf306fe8863aabe0d1d718c07b1f190dff193a6d5fe0e3ff1a98e9dfe8", "jaccard": 0.25, "prev_hash": "8001e13fe0582edf31cb651c0874c6848622c35de5082cc769d95f1b1318322c", "seq": 89, "to": "2610.11592"}
|
||||
{"from": "2610.32213", "hash": "b23e48e6f1f89d784d185d5cc6f9fcf95074fa86638228b1361c3541f31fb99d", "jaccard": 0.25, "prev_hash": "678347bf306fe8863aabe0d1d718c07b1f190dff193a6d5fe0e3ff1a98e9dfe8", "seq": 90, "to": "2610.19247"}
|
||||
{"from": "2610.32213", "hash": "3661607cd21f889e0d783b76bb600101bfa264b6ce0988f793a9bd3e9dea3054", "jaccard": 0.25, "prev_hash": "b23e48e6f1f89d784d185d5cc6f9fcf95074fa86638228b1361c3541f31fb99d", "seq": 91, "to": "2610.25005"}
|
||||
{"from": "2610.32263", "hash": "91023ceb7ce6f6e6bc55a55535e0aa6a2b21e94123ea262428713ce27d3a81ac", "jaccard": 0.25, "prev_hash": "3661607cd21f889e0d783b76bb600101bfa264b6ce0988f793a9bd3e9dea3054", "seq": 92, "to": "2610.28266"}
|
||||
{"from": "2610.32360", "hash": "522e7d002efafb897aba326608d903c8d7028a06fb02552c65dba8ac79e007ca", "jaccard": 0.25, "prev_hash": "91023ceb7ce6f6e6bc55a55535e0aa6a2b21e94123ea262428713ce27d3a81ac", "seq": 93, "to": "2610.19247"}
|
||||
{"from": "2610.32478", "hash": "2c0bf612e7c9db3cd0db0ac91cee031e4c2ef20d2e0b72722e4775f058e4988c", "jaccard": 0.25, "prev_hash": "522e7d002efafb897aba326608d903c8d7028a06fb02552c65dba8ac79e007ca", "seq": 94, "to": "2610.27360"}
|
||||
@@ -0,0 +1,148 @@
|
||||
#!/usr/bin/env python3
|
||||
"""indep_check.py -- a SECOND verifier, written from scratch.
|
||||
|
||||
Deliberately shares no code with run_verdict.py: it re-fetches the live arXiv
|
||||
API and compares the archived corpus against what arXiv serves NOW, and it
|
||||
re-walks the frozen sources using a different parsing strategy (regex over
|
||||
raw bytes, not ElementTree).
|
||||
|
||||
Exit 0 iff the archived corpus is consistent with live arXiv and with itself.
|
||||
|
||||
python3 indep_check.py # exit 0 = independently corroborated
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
FAILS = []
|
||||
|
||||
|
||||
def ck(name, cond, detail=""):
|
||||
print(f" [{'PASS' if cond else 'FAIL'}] {name}" + (f" -- {detail}" if detail else ""))
|
||||
if not cond:
|
||||
FAILS.append(name)
|
||||
|
||||
|
||||
def main():
|
||||
corpus = json.load(open(os.path.join(HERE, "corpus.json")))
|
||||
|
||||
# --- A. live arXiv corroboration -------------------------------------
|
||||
# Re-query arXiv for the exact ids we archived and confirm the titles and
|
||||
# publication dates still match what we froze.
|
||||
ids = sorted({nd["node"] for nd in corpus["nodes"]})
|
||||
ck("corpus has nodes", len(ids) > 0, f"{len(ids)} nodes")
|
||||
|
||||
live = {}
|
||||
B = 50
|
||||
for i in range(0, len(ids), B):
|
||||
chunk = ids[i:i + B]
|
||||
q = urllib.parse.urlencode(
|
||||
{"id_list": ",".join(chunk), "start": 0, "max_results": len(chunk)}
|
||||
)
|
||||
url = f"https://export.arxiv.org/api/query?{q}"
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "sphinx-indep/1.0"})
|
||||
body = urllib.request.urlopen(req, timeout=90).read().decode("utf-8", "replace")
|
||||
except Exception as ex:
|
||||
ck(f"live fetch chunk {i}", False, f"{type(ex).__name__}: {ex}")
|
||||
continue
|
||||
# regex parse, NOT ElementTree
|
||||
for m in re.finditer(r"<entry>(.*?)</entry>", body, re.S):
|
||||
blk = m.group(1)
|
||||
idm = re.search(r"<id>http://arxiv\.org/abs/([^<]+)</id>", blk)
|
||||
tm = re.search(r"<title>(.*?)</title>", blk, re.S)
|
||||
pm = re.search(r"<published>([^<]+)</published>", blk)
|
||||
if not (idm and tm):
|
||||
continue
|
||||
node = re.sub(r"v\d+$", "", idm.group(1).strip())
|
||||
live[node] = {
|
||||
"title": re.sub(r"\s+", " ", tm.group(1)).strip(),
|
||||
"published": pm.group(1).strip() if pm else "",
|
||||
}
|
||||
ck(f"live fetch chunk {i} returned entries", len(live) > 0, f"cumulative {len(live)}")
|
||||
|
||||
print(f" [info] live arXiv returned {len(live)} of {len(ids)} archived ids")
|
||||
|
||||
mism = []
|
||||
for nd in corpus["nodes"]:
|
||||
lv = live.get(nd["node"])
|
||||
if lv is None:
|
||||
mism.append((nd["node"], "absent from live arXiv"))
|
||||
elif lv["title"] != nd["title"]:
|
||||
mism.append((nd["node"], "title differs"))
|
||||
ck("archived titles match live arXiv", not mism, f"{len(mism)} mismatches")
|
||||
for n, why in mism[:5]:
|
||||
print(f" {n}: {why}")
|
||||
|
||||
# --- B. frozen source integrity, parsed by regex ---------------------
|
||||
for fm in corpus["fetch"]:
|
||||
p = os.path.join(HERE, fm["raw_path"])
|
||||
if not os.path.exists(p):
|
||||
ck(f"source exists {fm['label']}", False)
|
||||
continue
|
||||
raw = open(p, "rb").read()
|
||||
ck(f"source sha256 {fm['label']}",
|
||||
hashlib.sha256(raw).hexdigest() == fm["raw_sha256"])
|
||||
# count <entry> blocks by regex and compare to the manifest
|
||||
n_blocks = len(re.findall(rb"<entry>", raw))
|
||||
ck(f"entry count {fm['label']}", n_blocks == fm["returned"],
|
||||
f"regex={n_blocks} manifest={fm['returned']}")
|
||||
|
||||
# --- C. journal chain, recomputed with a different construction ------
|
||||
rows = [json.loads(l) for l in open(os.path.join(HERE, "edges.jsonl")) if l.strip()]
|
||||
ck("journal non-empty", len(rows) > 0, f"{len(rows)} rows")
|
||||
prev = "0" * 64
|
||||
bad = 0
|
||||
for i, r in enumerate(rows, start=1):
|
||||
expect = hashlib.sha256(
|
||||
json.dumps(
|
||||
{"seq": i, "from": r["from"], "to": r["to"],
|
||||
"jaccard": r["jaccard"], "prev_hash": prev},
|
||||
sort_keys=True, separators=(",", ":"),
|
||||
).encode()
|
||||
).hexdigest()
|
||||
if r["hash"] != expect or r["prev_hash"] != prev:
|
||||
bad += 1
|
||||
prev = r["hash"]
|
||||
ck("journal chain recomputes", bad == 0, f"{bad} bad")
|
||||
ck("journal head matches manifest", prev == corpus["journal"]["head"])
|
||||
|
||||
# --- D. DAG by Kahn's algorithm (different method than run_verdict) --
|
||||
nodes = [nd["node"] for nd in corpus["nodes"]]
|
||||
pub = {nd["node"]: nd["published"] for nd in corpus["nodes"]}
|
||||
indeg = {n: 0 for n in nodes}
|
||||
adj = {n: [] for n in nodes}
|
||||
for e in corpus["edges"]:
|
||||
adj[e["from"]].append(e["to"])
|
||||
indeg[e["to"]] = indeg.get(e["to"], 0) + 1
|
||||
queue = [n for n in nodes if indeg[n] == 0]
|
||||
seen = 0
|
||||
while queue:
|
||||
u = queue.pop()
|
||||
seen += 1
|
||||
for v in adj[u]:
|
||||
indeg[v] -= 1
|
||||
if indeg[v] == 0:
|
||||
queue.append(v)
|
||||
ck("Kahn topological sort consumes all nodes (acyclic)", seen == len(nodes),
|
||||
f"consumed {seen}/{len(nodes)}")
|
||||
|
||||
# --- E. edge direction sanity ----------------------------------------
|
||||
bad_dir = [e for e in corpus["edges"] if pub[e["to"]] > pub[e["from"]]]
|
||||
ck("edges respect publication order", not bad_dir, f"{len(bad_dir)} violations")
|
||||
|
||||
print()
|
||||
if FAILS:
|
||||
print(f"VERDICT: FAIL -- {len(FAILS)} failed: {', '.join(FAILS)}")
|
||||
return 1
|
||||
print(f"VERDICT: PASS -- independently corroborated ({len(ids)} nodes vs live arXiv)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,159 @@
|
||||
#!/usr/bin/env python3
|
||||
"""mutation_battery.py -- falsifiability harness for run_verdict.py.
|
||||
|
||||
Each mutation corrupts ONE thing in a copy of the archive; run_verdict.py must
|
||||
FAIL on that copy. If any mutation goes undetected, the verdict is not a real
|
||||
test. Exit 0 iff every mutation is caught.
|
||||
|
||||
python3 mutation_battery.py # exit 0 = all mutations caught
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
CAUGHT = []
|
||||
MISSED = []
|
||||
|
||||
|
||||
def run_verdict_in(d):
|
||||
p = subprocess.run(
|
||||
[sys.executable, os.path.join(d, "run_verdict.py")],
|
||||
capture_output=True, text=True, cwd=d,
|
||||
)
|
||||
return p.returncode
|
||||
|
||||
|
||||
def clone():
|
||||
d = tempfile.mkdtemp(prefix="mut-")
|
||||
for name in ("corpus.json", "run_verdict.py"):
|
||||
shutil.copy(os.path.join(HERE, name), os.path.join(d, name))
|
||||
os.makedirs(os.path.join(d, "source"), exist_ok=True)
|
||||
for f in os.listdir(os.path.join(HERE, "source")):
|
||||
shutil.copy(os.path.join(HERE, "source", f), os.path.join(d, "source", f))
|
||||
os.makedirs(os.path.join(d, "source"), exist_ok=True)
|
||||
shutil.copy(os.path.join(HERE, "edges.jsonl"), os.path.join(d, "edges.jsonl"))
|
||||
return d
|
||||
|
||||
|
||||
def mutate(name, fn):
|
||||
d = clone()
|
||||
fn(d)
|
||||
rc = run_verdict_in(d)
|
||||
ok = rc != 0
|
||||
(CAUGHT if ok else MISSED).append(name)
|
||||
print(f" [{'CAUGHT' if ok else 'MISSED'}] {name} (verdict rc={rc})")
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
|
||||
|
||||
def load(d):
|
||||
return json.load(open(os.path.join(d, "corpus.json")))
|
||||
|
||||
|
||||
def save(d, m):
|
||||
json.dump(m, open(os.path.join(d, "corpus.json"), "w"), indent=2, sort_keys=True)
|
||||
|
||||
|
||||
def m_title(d):
|
||||
m = load(d)
|
||||
m["nodes"][0]["title"] = "TOTALLY FABRICATED TITLE"
|
||||
save(d, m)
|
||||
|
||||
|
||||
def m_proof_line(d):
|
||||
m = load(d)
|
||||
m["nodes"][0]["proof"]["line"] += 7
|
||||
save(d, m)
|
||||
|
||||
|
||||
def m_proof_hash(d):
|
||||
m = load(d)
|
||||
m["nodes"][1]["proof"]["block_sha256"] = "0" * 64
|
||||
save(d, m)
|
||||
|
||||
|
||||
def m_edge_jaccard(d):
|
||||
m = load(d)
|
||||
m["edges"][0]["jaccard"] = 0.9999
|
||||
save(d, m)
|
||||
|
||||
|
||||
def m_edge_reverse(d):
|
||||
m = load(d)
|
||||
e = m["edges"][0]
|
||||
e["from"], e["to"] = e["to"], e["from"]
|
||||
save(d, m)
|
||||
|
||||
|
||||
def m_count(d):
|
||||
m = load(d)
|
||||
m["counts"]["distinct_nodes"] += 5
|
||||
save(d, m)
|
||||
|
||||
|
||||
def m_source_bytes(d):
|
||||
p = os.path.join(d, "source", "arxiv-mm.xml")
|
||||
b = open(p, "rb").read()
|
||||
open(p, "wb").write(b + b"\n<!-- injected -->\n")
|
||||
|
||||
|
||||
def m_node_missing(d):
|
||||
m = load(d)
|
||||
m["nodes"] = m["nodes"][:-1]
|
||||
save(d, m)
|
||||
|
||||
|
||||
def m_edge_endpoint(d):
|
||||
m = load(d)
|
||||
m["edges"][0]["from"] = "9999.99999"
|
||||
save(d, m)
|
||||
|
||||
|
||||
def m_journal_row(d):
|
||||
p = os.path.join(d, "edges.jsonl")
|
||||
lines = open(p).read().splitlines()
|
||||
row = json.loads(lines[0])
|
||||
row["jaccard"] = 0.1111
|
||||
lines[0] = json.dumps(row, sort_keys=True)
|
||||
open(p, "w").write("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
def m_drop_journal(d):
|
||||
os.remove(os.path.join(d, "edges.jsonl"))
|
||||
|
||||
|
||||
def main():
|
||||
# baseline: unmutated copy must PASS
|
||||
d0 = clone()
|
||||
base = run_verdict_in(d0)
|
||||
shutil.rmtree(d0, ignore_errors=True)
|
||||
print(f" [{'PASS' if base == 0 else 'FAIL'}] baseline unmutated archive (rc={base})")
|
||||
if base != 0:
|
||||
print("VERDICT: FAIL -- baseline does not pass; mutations are meaningless")
|
||||
return 1
|
||||
|
||||
mutate("title rewritten", m_title)
|
||||
mutate("proof line number shifted", m_proof_line)
|
||||
mutate("proof block hash zeroed", m_proof_hash)
|
||||
mutate("edge jaccard inflated", m_edge_jaccard)
|
||||
mutate("edge direction reversed", m_edge_reverse)
|
||||
mutate("manifest node count inflated", m_count)
|
||||
mutate("raw source bytes appended", m_source_bytes)
|
||||
mutate("node dropped from manifest", m_node_missing)
|
||||
mutate("edge endpoint set to unknown node", m_edge_endpoint)
|
||||
mutate("journal row body altered", m_journal_row)
|
||||
mutate("journal file deleted", m_drop_journal)
|
||||
|
||||
print()
|
||||
if MISSED:
|
||||
print(f"VERDICT: FAIL -- {len(MISSED)} mutations MISSED: {', '.join(MISSED)}")
|
||||
return 1
|
||||
print(f"VERDICT: PASS -- {len(CAUGHT)}/{len(CAUGHT)} mutations caught")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,12 @@
|
||||
import glob, re, xml.etree.ElementTree as ET
|
||||
|
||||
A = "{http://www.w3.org/2005/Atom}"
|
||||
for p in sorted(glob.glob("source/*.xml")):
|
||||
b = open(p, "rb").read()
|
||||
print("=====", p, len(b), "bytes; arXiv: mentions =",
|
||||
len(re.findall(rb"arXiv:\s*\d{4}\.\d{4,5}", b, re.I)))
|
||||
root = ET.fromstring(b)
|
||||
for e in root.findall(A + "entry")[:3]:
|
||||
aid = re.search(r"abs/(.+)$", e.findtext(A + "id", "")).group(1)
|
||||
ab = re.sub(r"\s+", " ", (e.findtext(A + "summary", "") or ""))
|
||||
print(" ", aid, "|", ab[:230])
|
||||
@@ -0,0 +1,24 @@
|
||||
"""probe_figs.py -- MEASURE multimodal asset availability in the corpus.
|
||||
|
||||
For the lane's core papers, fetch ar5iv HTML and count <img> / <figure> and
|
||||
math elements. This decides the archive's multimodal dimension.
|
||||
"""
|
||||
import re
|
||||
import urllib.request
|
||||
|
||||
IDS = ["2610.32213", "2610.32781", "2610.32967", "2610.32483", "2610.32612"]
|
||||
|
||||
for aid in IDS:
|
||||
url = f"https://ar5iv.labs.arxiv.org/html/{aid}"
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "sphinx/1.0"})
|
||||
body = urllib.request.urlopen(req, timeout=60).read().decode("utf-8", "replace")
|
||||
imgs = re.findall(r"<img[^>]+>", body)
|
||||
figs = re.findall(r"<figure", body)
|
||||
svgs = re.findall(r"<svg", body)
|
||||
alts = re.findall(r'alt="([^"]{5,})"', body)
|
||||
print(f"{aid} bytes={len(body):7d} <img>={len(imgs):3d} <figure>={len(figs):3d} <svg>={len(svgs):3d} alts={len(alts)}")
|
||||
if alts:
|
||||
print(" e.g.", alts[0][:100])
|
||||
except Exception as ex:
|
||||
print(f"{aid} FAIL {type(ex).__name__}: {ex}")
|
||||
@@ -0,0 +1,14 @@
|
||||
import urllib.parse, urllib.request, re
|
||||
|
||||
ids = ["2610.32213", "2610.32781", "2610.15371", "2609.25611"]
|
||||
q = urllib.parse.urlencode({"search_query": "id_list=" + ",".join(ids), "start": 0, "max_results": 50})
|
||||
url = "https://export.arxiv.org/api/query?" + q
|
||||
print("URL:", url)
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "sphinx/1.0"})
|
||||
body = urllib.request.urlopen(req, timeout=60).read().decode("utf-8", "replace")
|
||||
print("bytes:", len(body))
|
||||
print("entries:", len(re.findall(r"<entry>", body)))
|
||||
print(body[:400])
|
||||
print("...")
|
||||
m = re.search(r"<title>(.*?)</title>", body, re.S)
|
||||
print("feed title:", m.group(1)[:200] if m else None)
|
||||
@@ -0,0 +1,28 @@
|
||||
"""probe_kws.py -- MEASURE lane-keyword density across the fetched corpus."""
|
||||
import glob
|
||||
import re
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
A = "{http://www.w3.org/2005/Atom}"
|
||||
KW = [
|
||||
"multimodal", "multi-modal", "vision-language", "vlm", "mllm",
|
||||
"long context", "long-context", "long context", "kv cache", "kv-cache",
|
||||
"context window", "token", "retrieval-augmented", "rag", "benchmark",
|
||||
"grounding", "hallucination", "attention", "fine-tuning", "rlhf",
|
||||
]
|
||||
recs = []
|
||||
for p in sorted(glob.glob("source/*.xml")):
|
||||
root = ET.fromstring(open(p, "rb").read())
|
||||
for e in root.findall(A + "entry"):
|
||||
aid = re.search(r"abs/(.+)$", e.findtext(A + "id", "")).group(1)
|
||||
ab = re.sub(r"\s+", " ", (e.findtext(A + "summary", "") or "")).lower()
|
||||
recs.append((aid, ab))
|
||||
|
||||
print("corpus:", len(recs))
|
||||
counts = {k: 0 for k in KW}
|
||||
for aid, ab in recs:
|
||||
for k in KW:
|
||||
if k in ab:
|
||||
counts[k] += 1
|
||||
for k in sorted(KW, key=lambda x: -counts[x]):
|
||||
print(f" {counts[k]:3d}/{len(recs)} {k}")
|
||||
@@ -0,0 +1,39 @@
|
||||
"""probe_refs.py -- MEASURE the real citation signal.
|
||||
|
||||
Fetch the arXiv (ar5iv) HTML for a sample of the fetched papers and count
|
||||
how many of their reference-list entries resolve to arXiv IDs that are also
|
||||
in our slice. This decides whether the citation graph is a real artifact or
|
||||
a tautological empty set.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import urllib.request
|
||||
|
||||
IDS = ["2610.32213", "2610.32781", "2610.32967", "2610.32483", "2610.32612"]
|
||||
SLICE = set()
|
||||
for line in open("probe_titles.py"):
|
||||
pass
|
||||
|
||||
# load slice ids from the saved XML
|
||||
import glob
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
A = "{http://www.w3.org/2005/Atom}"
|
||||
for p in sorted(glob.glob("source/*.xml")):
|
||||
root = ET.fromstring(open(p, "rb").read())
|
||||
for e in root.findall(A + "entry"):
|
||||
SLICE.add(re.search(r"abs/(.+)$", e.findtext(A + "id", "")).group(1))
|
||||
|
||||
ARXIV_REF = re.compile(r"arXiv:\s*(\d{4}\.\d{4,5})", re.I)
|
||||
|
||||
for aid in IDS:
|
||||
for host in ("ar5iv.labs.arxiv.org/html/", "arxiv.org/abs/"):
|
||||
url = f"https://{host}{aid}"
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "sphinx/1.0"})
|
||||
body = urllib.request.urlopen(req, timeout=60).read()
|
||||
n = len(ARXIV_REF.findall(body.decode("utf-8", "replace")))
|
||||
print(f"{aid:14s} {host:32s} {len(body):8d} B arXiv-refs={n}")
|
||||
break
|
||||
except Exception as ex:
|
||||
print(f"{aid:14s} {host:32s} FAIL {type(ex).__name__}: {ex}")
|
||||
@@ -0,0 +1,58 @@
|
||||
"""probe_titles.py -- MEASURE title-overlap density between fetched papers.
|
||||
|
||||
Question: is there a real, non-trivial graph signal between the two slices
|
||||
(multimodal cs.CV and long-context cs.CL)? Titles are stripped of stopwords
|
||||
and stemmed crudely; we report pairwise Jaccard similarity so we can pick a
|
||||
threshold that yields a genuine graph rather than a hairball or an empty set.
|
||||
"""
|
||||
import glob
|
||||
import re
|
||||
import xml.etree.ElementTree as ET
|
||||
from itertools import combinations
|
||||
|
||||
A = "{http://www.w3.org/2005/Atom}"
|
||||
STOP = {
|
||||
"a", "an", "the", "of", "for", "and", "or", "to", "in", "on", "with", "via",
|
||||
"by", "from", "as", "at", "is", "are", "we", "our", "their", "its", "this",
|
||||
"that", "these", "those", "using", "based", "towards", "through", "into",
|
||||
"how", "what", "why", "when", "can", "do", "does", "be", "been", "than",
|
||||
}
|
||||
|
||||
recs = []
|
||||
for p in sorted(glob.glob("source/*.xml")):
|
||||
root = ET.fromstring(open(p, "rb").read())
|
||||
for e in root.findall(A + "entry"):
|
||||
aid = re.search(r"abs/(.+)$", e.findtext(A + "id", "")).group(1)
|
||||
t = re.sub(r"\s+", " ", (e.findtext(A + "title", "") or ""))
|
||||
toks = set()
|
||||
for w in re.findall(r"[A-Za-z][A-Za-z\-]+", t.lower()):
|
||||
w = w.strip("-")
|
||||
if len(w) < 3 or w in STOP:
|
||||
continue
|
||||
if len(w) > 4:
|
||||
w = w[:4] # crude stem
|
||||
toks.add(w)
|
||||
recs.append((aid, t, toks))
|
||||
|
||||
print("records:", len(recs))
|
||||
pairs = 0
|
||||
buckets = {0.2: 0, 0.25: 0, 0.3: 0, 0.35: 0, 0.4: 0}
|
||||
best = []
|
||||
for (a1, t1, s1), (a2, t2, s2) in combinations(recs, 2):
|
||||
if not s1 or not s2:
|
||||
continue
|
||||
j = len(s1 & s2) / len(s1 | s2)
|
||||
if j > 0:
|
||||
pairs += 1
|
||||
for thr in buckets:
|
||||
if j >= thr:
|
||||
buckets[thr] += 1
|
||||
best.append((j, a1, a2))
|
||||
|
||||
best.sort(reverse=True)
|
||||
print("pairs with jaccard>0:", pairs)
|
||||
for thr in sorted(buckets):
|
||||
print(f" jaccard >= {thr}: {buckets[thr]} pairs")
|
||||
print("top 12 overlaps:")
|
||||
for j, a1, a2 in best[:12]:
|
||||
print(f" {j:.3f} {a1} <-> {a2}")
|
||||
@@ -0,0 +1,32 @@
|
||||
"""probe_xref.py -- MEASURE whether real cross-references exist within the slice."""
|
||||
import glob
|
||||
import re
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
A = "{http://www.w3.org/2005/Atom}"
|
||||
recs = []
|
||||
for p in sorted(glob.glob("source/*.xml")):
|
||||
root = ET.fromstring(open(p, "rb").read())
|
||||
for e in root.findall(A + "entry"):
|
||||
aid = re.search(r"abs/(.+)$", e.findtext(A + "id", "")).group(1)
|
||||
ab = re.sub(r"\s+", " ", (e.findtext(A + "summary", "") or "")).lower()
|
||||
recs.append((aid, ab))
|
||||
|
||||
# Jaccard overlap between abstract sets
|
||||
def toks(s):
|
||||
return set(re.findall(r"[a-z]{4,}", s))
|
||||
|
||||
nodes = {a: toks(ab) for a, ab in recs}
|
||||
hits = []
|
||||
ids = list(nodes)
|
||||
for i in range(len(ids)):
|
||||
for j in range(i + 1, len(ids)):
|
||||
a, b = ids[i], ids[j]
|
||||
inter = nodes[a] & nodes[b]
|
||||
if len(inter) >= 8:
|
||||
hits.append((len(inter), a, b, sorted(inter)[:8]))
|
||||
|
||||
hits.sort(reverse=True)
|
||||
print("pairs with >=8 shared rare-ish tokens:", len(hits))
|
||||
for n, a, b, ex in hits[:10]:
|
||||
print(f" {n:3d} {a} <-> {b} {ex}")
|
||||
@@ -0,0 +1,209 @@
|
||||
#!/usr/bin/env python3
|
||||
"""run_verdict.py -- sphinx Archive lane. THE VERDICT COMMAND.
|
||||
|
||||
Exit 0 iff the archive holds. Written independently of build_corpus.py:
|
||||
it re-parses the frozen raw XML from scratch, re-derives every node and
|
||||
every proof, re-walks the hash chain, and checks the DAG property.
|
||||
|
||||
python3 run_verdict.py # exit 0 = the archive holds
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
ATOM = "{http://www.w3.org/2005/Atom}"
|
||||
ARXIV = "{http://arxiv.org/schemas/atom}"
|
||||
FAILS = []
|
||||
|
||||
|
||||
def check(name, cond, detail=""):
|
||||
print(f" [{'PASS' if cond else 'FAIL'}] {name}" + (f" -- {detail}" if detail else ""))
|
||||
if not cond:
|
||||
FAILS.append(name)
|
||||
|
||||
|
||||
def canon(aid):
|
||||
return re.sub(r"v\d+$", "", aid.strip())
|
||||
|
||||
|
||||
def h(b):
|
||||
return hashlib.sha256(b).hexdigest()
|
||||
|
||||
|
||||
def main():
|
||||
corpus = json.load(open(os.path.join(HERE, "corpus.json")))
|
||||
nodes = corpus["nodes"]
|
||||
print(f"archive: {corpus['artifact']} built {corpus['built_utc']}")
|
||||
print(f"nodes={len(nodes)} edges={len(corpus['edges'])}")
|
||||
|
||||
# ---- 1. raw sources exist and match their recorded hash ----
|
||||
for fm in corpus["fetch"]:
|
||||
p = os.path.join(HERE, fm["raw_path"])
|
||||
ok = os.path.exists(p)
|
||||
check(f"source present: {fm['raw_path']}", ok)
|
||||
if not ok:
|
||||
continue
|
||||
raw = open(p, "rb").read()
|
||||
check(f"source sha256 matches: {fm['label']}", h(raw) == fm["raw_sha256"],
|
||||
f"recorded={fm['raw_sha256'][:12]} actual={h(raw)[:12]}")
|
||||
check(f"source byte count matches: {fm['label']}", len(raw) == fm["raw_bytes"])
|
||||
|
||||
# ---- 2. every node's proof line + block hash re-derive from raw XML ----
|
||||
raw_by_label = {}
|
||||
for fm in corpus["fetch"]:
|
||||
raw_by_label[fm["label"]] = open(os.path.join(HERE, fm["raw_path"]), "rb").read()
|
||||
|
||||
bad_line = bad_hash = bad_title = 0
|
||||
for nd in nodes:
|
||||
raw = raw_by_label[nd["slice"]]
|
||||
lines = raw.split(b"\n")
|
||||
ln = nd["proof"]["line"]
|
||||
if not (1 <= ln <= len(lines)) or lines[ln - 1].strip() != b"<entry>":
|
||||
bad_line += 1
|
||||
continue
|
||||
# re-derive block
|
||||
starts = [i for i, x in enumerate(lines) if x.strip() == b"<entry>"]
|
||||
idx = starts.index(ln - 1)
|
||||
end = starts[idx + 1] if idx + 1 < len(starts) else len(lines)
|
||||
block = b"\n".join(lines[ln - 1:end])
|
||||
if h(block) != nd["proof"]["block_sha256"] or len(block) != nd["proof"]["block_bytes"]:
|
||||
bad_hash += 1
|
||||
# re-derive the title from the block itself
|
||||
m = re.search(rb"<title[^>]*>(.*?)</title>", block, re.S)
|
||||
if m:
|
||||
t = re.sub(r"\s+", " ", m.group(1).decode("utf-8", "replace")).strip()
|
||||
if t != nd["title"]:
|
||||
bad_title += 1
|
||||
check("proof line numbers all point at <entry>", bad_line == 0, f"{bad_line} bad")
|
||||
check("proof block hashes all re-derive", bad_hash == 0, f"{bad_hash} bad")
|
||||
check("titles re-derive from proof block", bad_title == 0, f"{bad_title} bad")
|
||||
|
||||
# ---- 3. dedupe: node ids unique, and every node is present in its raw slice ----
|
||||
ids = [nd["node"] for nd in nodes]
|
||||
check("node ids unique", len(ids) == len(set(ids)))
|
||||
present = set()
|
||||
for label, raw in raw_by_label.items():
|
||||
root = ET.fromstring(raw)
|
||||
for e in root.findall(f"{ATOM}entry"):
|
||||
aid = re.search(r"abs/([^/]+)$", e.findtext(f"{ATOM}id", "").strip())
|
||||
if aid:
|
||||
present.add(canon(aid.group(1)))
|
||||
check("every node exists in frozen raw XML", set(ids) <= present,
|
||||
f"{len(set(ids) - present)} missing")
|
||||
|
||||
# ---- 4. hash-chained append-only edge journal ----
|
||||
prev = "0" * 64
|
||||
bad_chain = 0
|
||||
rows = [json.loads(l) for l in open(os.path.join(HERE, corpus["journal"]["path"])) if l.strip()]
|
||||
for i, row in enumerate(rows, start=1):
|
||||
if row["seq"] != i or row["prev_hash"] != prev:
|
||||
bad_chain += 1
|
||||
continue
|
||||
body = {k: row[k] for k in ("seq", "from", "to", "jaccard", "prev_hash")}
|
||||
if h(json.dumps(body, sort_keys=True, separators=(",", ":")).encode()) != row["hash"]:
|
||||
bad_chain += 1
|
||||
prev = row["hash"]
|
||||
check("edge journal hash chain intact", bad_chain == 0, f"{bad_chain} bad rows")
|
||||
check("journal head matches manifest", prev == corpus["journal"]["head"])
|
||||
check("journal entry count matches manifest", len(rows) == corpus["journal"]["entries"])
|
||||
|
||||
# ---- 5. DAG: every edge points later-published -> earlier-published ----
|
||||
pub = {nd["node"]: nd["published"] for nd in nodes}
|
||||
bad_dir = [e for e in corpus["edges"] if not (pub[e["to"]] <= pub[e["from"]])]
|
||||
check("all edges point to an earlier-or-equal publication date", not bad_dir,
|
||||
f"{len(bad_dir)} violations")
|
||||
|
||||
# ---- 6. cycle check by longest-path (a DAG has longest path <= node count) ----
|
||||
adj = {}
|
||||
for e in corpus["edges"]:
|
||||
adj.setdefault(e["from"], set()).add(e["to"])
|
||||
# iterative DFS longest path with memo; detect cycle via recursion stack
|
||||
WHITE, GREY, BLACK = 0, 1, 2
|
||||
colour = {n: WHITE for n in ids}
|
||||
cycle = []
|
||||
|
||||
def dfs(u, stack):
|
||||
colour[u] = GREY
|
||||
for v in sorted(adj.get(u, ())):
|
||||
if colour.get(v, WHITE) == GREY:
|
||||
cycle.append(stack + [v])
|
||||
elif colour.get(v, WHITE) == WHITE:
|
||||
dfs(v, stack + [v])
|
||||
colour[u] = BLACK
|
||||
|
||||
for n in ids:
|
||||
if colour[n] == WHITE:
|
||||
dfs(n, [n])
|
||||
check("graph is acyclic", not cycle, f"{len(cycle)} cycles")
|
||||
|
||||
# ---- 7. edge endpoints are real nodes ----
|
||||
known = set(ids)
|
||||
bad_ep = [e for e in corpus["edges"] if e["from"] not in known or e["to"] not in known]
|
||||
check("all edge endpoints are corpus nodes", not bad_ep, f"{len(bad_ep)} bad")
|
||||
|
||||
# ---- 7b. edges RE-DERIVE from the archived titles (no trust in corpus.json) ----
|
||||
STOP = {
|
||||
"a", "an", "the", "of", "for", "and", "or", "to", "in", "on", "with", "via",
|
||||
"by", "from", "as", "at", "is", "are", "we", "our", "their", "its", "this",
|
||||
"that", "these", "those", "using", "based", "towards", "through", "into",
|
||||
"how", "what", "why", "when", "can", "do", "does", "be", "been", "than",
|
||||
}
|
||||
title_by_node = {nd["node"]: nd["title"] for nd in nodes}
|
||||
thr = corpus["counts"]["threshold"]
|
||||
|
||||
def tt(s):
|
||||
out = set()
|
||||
for w in re.findall(r"[A-Za-z][A-Za-z\-]+", s.lower()):
|
||||
w = w.strip("-")
|
||||
if len(w) < 3 or w in STOP:
|
||||
continue
|
||||
out.add(w[:4] if len(w) > 4 else w)
|
||||
return out
|
||||
|
||||
bad_jac = 0
|
||||
rederived = set()
|
||||
for e in corpus["edges"]:
|
||||
sa, sb = tt(title_by_node[e["from"]]), tt(title_by_node[e["to"]])
|
||||
if not sa or not sb:
|
||||
bad_jac += 1
|
||||
continue
|
||||
j = round(len(sa & sb) / len(sa | sb), 4)
|
||||
if abs(j - e["jaccard"]) > 1e-9:
|
||||
bad_jac += 1
|
||||
rederived.add((e["from"], e["to"]))
|
||||
check("edge jaccard re-derives from archived titles", bad_jac == 0, f"{bad_jac} bad")
|
||||
|
||||
# no edge may be missing: recompute the full edge set independently
|
||||
expected = set()
|
||||
for i in range(len(ids)):
|
||||
for j in range(i + 1, len(ids)):
|
||||
a, b = ids[i], ids[j]
|
||||
sa, sb = tt(title_by_node[a]), tt(title_by_node[b])
|
||||
if not sa or not sb:
|
||||
continue
|
||||
if len(sa & sb) / len(sa | sb) >= thr:
|
||||
expected.add((b, a))
|
||||
missing = expected - rederived
|
||||
extra = rederived - expected
|
||||
check("edge set is exactly the recomputed edge set", not missing and not extra,
|
||||
f"{len(missing)} missing, {len(extra)} extra")
|
||||
|
||||
# ---- 8. counts in the manifest are honest ----
|
||||
c = corpus["counts"]
|
||||
check("manifest node count honest", c["distinct_nodes"] == len(nodes))
|
||||
check("manifest edge count honest", c["similarity_edges"] == len(corpus["edges"]))
|
||||
|
||||
print()
|
||||
if FAILS:
|
||||
print(f"VERDICT: FAIL ({len(FAILS)} checks failed: {', '.join(FAILS)})")
|
||||
return 1
|
||||
print(f"VERDICT: PASS -- {len(nodes)} nodes, {len(corpus['edges'])} edges, all checks green")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user