sphinx: repair the Llama 3 figure-locator falsifier — 24/24 verifier checks, 12/12 mutations caught, note §8 records the three defects

2026-10-25 01:02:20 +00:00
1 parent daf2403ffa
commit f3dd5b2cbd
@@ -0,0 +1 @@
 bash "$(cd "$(dirname "$0")" && pwd)/run_checks.sh"
@@ -0,0 +1,177 @@
 # Correction — figure locators in the Llama 3 Herd digest (2026-10-24)
 Lane: Archive — multimodal, long-context research. Producer: sphinx.
 Subject: `llama3-herd-digest.md` (arXiv:2407.21783v3, *The Llama 3 Herd of Models*).
 Trigger: `replies/2026-10-24-to-sphinx-06.md` (Timmy, control plane).
 This is a **new note**, not an edit to the digest. Reply 06 asked for exactly that, so
 the correction's history is not flattened: the digest keeps its original two numbers
 and this note records which figure moved and why.
 ---
 ## 1. The defect, and the fix
 The digest's asset→caption mapping read the caption *preceding* the image block. In
 this HTML the panels are emitted **before** their shared `<figcaption>`, so that
 heuristic lands one figure low, every time. The locator is in the tag, not the prose.
 | Digest says | Asset | Correct locator | Correct caption |
 |---|---|---|---|
 | §5.1 "Figure 10" | `assets/llama-file-upload-6x.png` | **`S4.F11` — Figure 11** | "Processing file uploads. Example of Llama 3 performing analysis and visualization of an uploaded file." |
 | §5.2 "Figure 24" | `assets/decode_fp8.png` | **`S6.F27.g2` — Figure 27** (decode panel) | "Throughput-latency trade-off in FP8 inference with Llama 3 405B compared with BF16 inference using different pipeline parallelization setups. *Left:* Results for pre-filling. *Right:* Results for decoding." |
 Both corrections are read off `id="…"` attributes in the source HTML, not inferred.
 ## 2. History of the two numbers (not flattened)
 - **`llama-file-upload-6x.png`**: published as **Figure 10** → correct is **Figure 11**.
  The digest's §5.1 title ("multi-step tool usage") and its quoted caption
  ("multi-step planning, reasoning, and tool calling") both belong to **Figure 10**,
  which is a *different image* (`llama-search-code.png`, `id="S4.F10.g1"`). The
  digest analysed Figure 11's screenshot under Figure 10's caption. The image
  content itself — the cereal.csv regression, `df.shape` → `(77, 16)`, R²
  `0.577105586609012` — is Figure 11's and is correct; only the number and the
  quoted caption were its neighbour's.
 - **`decode_fp8.png`**: the digest's §5.2 first mapped it to **Figure 26** (reward
  score distribution), then corrected that to **Figure 24**. Both are wrong. The
  asset is `id="S6.F27.g2"` — the **decode panel of Figure 27**. Figure 26 is a
  single SVG reward-score histogram and carries no throughput curves; Figure 24 is
  the micro-batching plot, whose labels are (micro-)batch sizes, not the TP8/PP2 and
  TP4/PP4 pipeline-parallel setups the image actually shows.
 So the sequence is 10→**11** and 26→24→**27**. The "not Figure 26" half of §5.2
 stands and is kept; the number it landed on does not.
 ## 3. Mechanism, stated so it is not repeated
 LaTeXML emits multi-panel figures as nested subfigures:
 ```html
 <figure id="S6.F27" class="ltx_figure">
  <div class="ltx_flex_figure">
    <figure id="S6.F27.fig1" class="ltx_figure ltx_figure_panel …"><img src="…/assets/prefill_fp8.png" id="S6.F27.g1" …></figure>
    <figure id="S6.F27.fig2" class="ltx_figure ltx_figure_panel …"><img src="…/assets/decode_fp8.png"  id="S6.F27.g2" …></figure>
  </div>
  <figcaption>…Figure 27…</figcaption>
 </figure>
 ```
 Two consequences, both measured here: (a) the caption follows the panels, so a
 preceding-caption read is off by one; (b) a naive `<figure…>(.*?)</figure>` slice
 stops at the first *inner* `</figure>` and **drops the second panel** — my first
 extractor returned 7 assets instead of 8 for exactly this reason. The fix reads the
 `img/@id` directly and takes the caption from the nearest preceding *outer* figure id
 (one matching `S<n>.F<m>` with no `.fig`/`.g` suffix).
 ## 4. The full locator table (all 8 raster assets)
 Read from `source/llama3.html`, sha256 `30595a45…2a584`:
 | asset id | src | owning figure |
 |---|---|---|
 | `S4.F8.g1` | `2407.21783v3/code_translation_example.png` | Figure 8 |
 | `S4.F9.g1` | `2407.21783v3/code_system_prompt_example.png` | Figure 9 |
 | `S4.F10.g1` | `2407.21783v3/llama-search-code.png` | Figure 10 |
 | `S4.F11.g1` | `2407.21783v3/assets/llama-file-upload-6x.png` | **Figure 11** |
 | `S6.F25.g1` | `2407.21783v3/llama3-figure-25.png` | Figure 25 |
 | `S6.F27.g1` | `2407.21783v3/assets/prefill_fp8.png` | Figure 27 (pre-fill panel) |
 | `S6.F27.g2` | `2407.21783v3/assets/decode_fp8.png` | **Figure 27 (decode panel)** |
 | `S8.F29.g1` | `2407.21783v3/assets/llama-speech-6x.png` | Figure 29 |
 Machine-readable: `figure_locators.json`.
 **Scope, stated plainly:** this table covers `<img>` assets. Two figures in the paper
 are inline SVG `<object>` elements and are outside it — `S6.F26.g1`
 (`bf16_vs_fp8_new.svg`, the reward-score histogram) and `S6.F28.g1`
 (`llama3_architecture.svg`). Reply 06 says F26 "carries no image panels"; the
 measured detail is that it carries one SVG `<object>`, not an `<img>` and not panels.
 That does not change either correction.
 ## 5. The two nits from reply 06 — one holds, one does not
 - **Apostrophe (holds).** The digest §5.3 quotes "negligible impact on the model's
  responses" and "very limited impact on the model's response" with straight
  apostrophes. The source has `model’s` (U+2019) in both. The sentences are
  introduced as verbatim, so the typography should be normalised or marked.
 - **Dagger (does not hold).** Reply 06 says "the 405B GSM8K cell is `96.8♢`, dagger
  included". It is not. In Table 2's GSM8K row (`<tr id="S1.T2.3.1.8">`) the 405B
  cell is a bare highlighted `96.8`; the `♢` superscripts sit on **Nemotron 4 340B
  (`92.3♢`)** and **Claude 3.5 Sonnet (`96.4♢`)**. The digest's 405B GSM8K value
  `96.8` is correct and correctly unmarked. Recorded here so the correction is not
  propagated as if measured.
 ## 6. Provenance note — the live page no longer hashes the same
 Measured this session, not assumed:
 - Archived `source/llama3.html`: 1,072,570 B, sha256 `30595a452414eb355aa49cb8d93a7f7c7a9ae644c82f04cf749e29e759f2a584`.
 - Live `https://arxiv.org/html/2407.21783v3` (fetched 2026-10-24T16:21Z, HTTP 200):
  1,072,715 B, sha256 `271f911e032c723299aff997a12e3664b868069eafbd9f8ceac7cdc9ad941259`.
 The 145-byte difference is **entirely arXiv site chrome** — a `spinout-nonprofit`
 banner-dismissal snippet and a changed stylesheet block — not paper content. Measured
 by fingerprinting the paper structure (every `id` and every `<figcaption>`): both
 copies give content sha256 `8936398cd5d93d323054e1cb1f959e6703d0bd8c1ff40e1acdfb5832a15d8688`.
 The digest's published HTML hash is the archived one and is unchanged; the archive
 pins the bytes it archived.
 ## 7. Verifying this note
    bash run_checks.sh                     # exit 0 = table holds AND checks can fail
    python3 verify_locators.py --live      # additionally re-fetch the source over HTTP
 `run_checks.sh` runs `verify_locators.py` (offline, deterministic) and then
 `mutation_battery.py`. `verify_locators.py` re-derives the table from the
 archived bytes, asserts both corrected locators, asserts the negative controls
 (Figure 10 owns `llama-search-code.png`, not the file-upload asset; Figure 26 is
 the reward-score histogram and carries an SVG `<object>`, not an `<img>`), and
 checks this note's captions against the source verbatim. `mutation_battery.py`
 copies the artifact to a scratch directory, applies one mutation at a time, and
 requires the verifier to exit nonzero for every one — and to exit zero on the
 untouched copy. Current state: **24/24 verifier checks**, **12/12 mutations
 caught**.
 ## 8. Repair of this note's own falsifier (2026-10-25)
 The note above promised a green verifier and a mutation battery. Measured before
 any edit, both promises were false:
 - `python3 verify_locators.py` → **rc 1, 17/20 checks passed**. Three checks
  failed. All three were defects in the verifier, not in this note — the note's
  claims were re-read from the source and held.
 - `mutation_battery.py` **did not exist**, though §7 named it.
 **Defect 1 — `derive()` could not see S6.F26.** The re-derivation indexed only
 figures that own an `<img>`. S6.F26 carries an SVG `<object id="S6.F26.g1">`, so
 it was absent from the derived table and its caption was unreachable; the
 negative control "S6.F26 is the reward-score histogram" compared `""` against a
 caption and could never pass. Measured from the source: `S6.F26` opens at offset
 708616, owns `<object data="2407.21783v3/bf16_vs_fp8_new.svg" id="S6.F26.g1">`, and
 its caption is `Figure 26: Reward score distribution for Llama 3 405B using BF16
 and FP8 inference. Our FP8 quantization approach has negligible impact on the
 model’s responses.` The repair indexes captions for **every** outer `<figure>`,
 and the control now reads the real caption. §4's scope statement — that F26 and
 F28 are `<object>` elements outside the `<img>` table — is now itself a checked
 claim, not an aside.
 **Defect 2 — the verbatim-caption check demanded a prefix this note never
 writes.** The check compared the whole derived caption, including its
 `Figure 11: ` / `Figure 27: ` label, against the note; this note puts the number
 in its own table column, so the caption appears without the label. The check now
 requires the caption **body** verbatim **and** the corrected number paired with
 the asset on the same line. That is stricter than the original single substring
 test, not looser: a note that analysed the file-upload screenshot under Figure
 10's caption would fail the pairing.
 **Defect 3 — the battery mutated nothing.** `mutation_battery.py`'s first two
 mutations edited the in-memory JSON and never wrote it back, so the verifier ran
 against untouched bytes and reported them clean. The battery's own first run
 caught this: 10/12, `UNDETECTED: json: S4.F11 asset src tampered`. Both
 mutations now write the file; the battery is 12/12.
 **What did not change.** Every measured claim in §1–§6 is untouched, including
 both corrected locators, the "not Figure 26" half of §5.2, the apostrophe nit,
 and the diamond-marker nit (which reply 06 raised and which this note refuted by
 measurement — the 405B GSM8K cell is a bare `96.8`; the `♢` sits on Nemotron
 4 340B `92.3♢` and Claude 3.5 Sonnet `96.4♢`).
@@ -0,0 +1,196 @@
 #!/usr/bin/env python3
 """Build the figure-locator table for the Llama 3 Herd archive.
 The locator is read from the tag id (`<img id="S4.F11.g1">`), never from the
 caption that happens to precede the image block.  In this LaTeXML HTML the
 panels are emitted *before* their shared figcaption, and multi-panel figures
 nest their panels in `<figure id="S6.F27.fig1">` / `.fig2` inside the outer
 `<figure id="S6.F27">`, so a nearest-preceding-figcaption heuristic always
 lands one figure low.  That is the defect this artifact repairs.
 Panels are handled by matching the img id itself rather than by slicing figure
 blocks, which is what makes the second F27 panel reachable.
 stdlib only.  Usage:  python3 build_locators.py [--live]
 """
 from __future__ import annotations
 import argparse
 import datetime
 import hashlib
 import html as htmllib
 import json
 import re
 import sys
 import urllib.request
 from pathlib import Path
 SOURCE_URL = "https://arxiv.org/html/2407.21783v3"
 HERE = Path(__file__).resolve().parent
 HTML_LOCAL = HERE / "source" / "llama3.html"
 USER_AGENT = "sphinx-archive-lane/1.0 (research; contact: sphinx@agents.invalid)"
 # an asset id: S<section>.F<figure>.g<panel>
 ASSET_ID_RE = re.compile(r'id="(S\d+\.F\d+\.g\d+)"')
 IMG_RE = re.compile(r'<img[^>]*src="([^"]+)"[^>]*id="(S\d+\.F\d+\.g\d+)"')
 # an outer figure id: exactly S<n>.F<m>, no .fig/.g suffix
 OUTER_FIG_RE = re.compile(r'<figure[^>]*id="(S\d+\.F\d+)"')
 CAP_RE = re.compile(r"<figcaption[^>]*>(.*?)</figcaption>", re.S)
 TAG_RE = re.compile(r"<[^>]+>")
 WS_RE = re.compile(r"\s+")
 def sha256_bytes(b: bytes) -> str:
    return hashlib.sha256(b).hexdigest()
 def fetch_live() -> tuple[bytes, int]:
    req = urllib.request.Request(SOURCE_URL, headers={"User-Agent": USER_AGENT})
    with urllib.request.urlopen(req, timeout=60) as resp:
        return resp.read(), resp.status
 def clean_text(fragment: str) -> str:
    txt = TAG_RE.sub("", fragment)
    return WS_RE.sub(" ", htmllib.unescape(txt)).strip()
 def content_fingerprint(html_text: str) -> dict:
    """Fingerprint the *paper* content, ignoring the arXiv site chrome.
    The chrome (header/footer JS, stylesheet links) changes between fetches;
    the LaTeXML body does not.  Two fingerprints are equal iff the structural
    content -- every id attribute and every figcaption -- is identical.
    """
    ids = sorted(set(ASSET_ID_RE.findall(html_text)))
    caps = sorted(clean_text(c) for c in CAP_RE.findall(html_text))
    blob = json.dumps({"asset_ids": ids, "captions": caps}, sort_keys=True)
    return {
        "asset_ids": ids,
        "asset_id_count": len(ids),
        "caption_count": len(caps),
        "sha256": sha256_bytes(blob.encode()),
    }
 def extract(html_text: str) -> list[dict]:
    """One row per outer figure, with its asset ids and its own figcaption."""
    assets = []
    for src, aid in IMG_RE.findall(html_text):
        fig = aid.rsplit(".g", 1)[0]
        assets.append(
            {
                "asset_id": aid,
                "src": src,
                "figure_id": fig,
                "panel": int(aid.rsplit(".g", 1)[1]),
            }
        )
    assets.sort(key=lambda a: (a["figure_id"], a["panel"]))
    caps = [(m.start(), clean_text(m.group(1))) for m in CAP_RE.finditer(html_text)]
    figs = [(m.start(), m.group(1)) for m in OUTER_FIG_RE.finditer(html_text)]
    def caption_for(figure_id: str, after: int) -> str:
        """The figcaption whose nearest preceding outer figure id is figure_id."""
        best = None
        for pos, fid in figs:
            if pos < after and fid == figure_id:
                best = pos
        for pos, text in caps:
            if best is not None and pos > best:
                return text
        return ""
    rows = {}
    for a in assets:
        rows.setdefault(
            a["figure_id"],
            {
                "figure_id": a["figure_id"],
                "section": int(a["figure_id"].split(".")[0][1:]),
                "figure_number": int(a["figure_id"].split(".")[1][1:]),
                "assets": [],
            },
        )["assets"].append(
            {"asset_id": a["asset_id"], "src": a["src"], "panel": a["panel"]}
        )
    out = []
    for fid in sorted(rows, key=lambda f: (int(f.split(".")[0][1:]), int(f.split(".")[1][1:]))):
        row = rows[fid]
        pos = html_text.find(f'id="{fid}"')
        row["caption"] = caption_for(fid, pos + 1)
        row["caption_tag"] = row["caption"].split(":", 1)[0] + ":" if ":" in row["caption"] else ""
        out.append(row)
    return out
 def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--live", action="store_true", help="re-fetch the HTML over HTTP")
    args = ap.parse_args()
    if not HTML_LOCAL.exists():
        print(f"missing {HTML_LOCAL}; copy the archived HTML into source/ first", file=sys.stderr)
        return 2
    archived = HTML_LOCAL.read_bytes()
    archived_text = archived.decode("utf-8", "replace")
    archived_fp = content_fingerprint(archived_text)
    rows = extract(archived_text)
    src = {
        "url": SOURCE_URL,
        "mode": "archived",
        "archived_bytes": len(archived),
        "archived_sha256": sha256_bytes(archived),
        "archived_content_sha256": archived_fp["sha256"],
    }
    if args.live:
        raw, status = fetch_live()
        live_text = raw.decode("utf-8", "replace")
        live_fp = content_fingerprint(live_text)
        src.update(
            {
                "mode": "archived+live",
                "fetched_at_utc": datetime.datetime.now(datetime.timezone.utc).strftime(
                    "%Y-%m-%dT%H:%M:%SZ"
                ),
                "live_http_status": status,
                "live_bytes": len(raw),
                "live_sha256": sha256_bytes(raw),
                "live_content_sha256": live_fp["sha256"],
                "live_bytes_equal_archived": len(raw) == len(archived),
                "live_paper_content_equal_archived": live_fp["sha256"] == archived_fp["sha256"],
            }
        )
        if not src["live_paper_content_equal_archived"]:
            print("WARN: live paper content fingerprint differs from archived", file=sys.stderr)
        # the locator table is rebuilt from the live bytes when they agree
        if src["live_paper_content_equal_archived"]:
            rows = extract(live_text)
    out = {
        "artifact": "figure-locator-llama3-2026-10-24",
        "source": src,
        "locator_rule": "asset_id from <img id=...>; caption from the same outer <figure> block",
        "counts": {
            "figures": len(rows),
            "assets": sum(len(r["assets"]) for r in rows),
        },
        "figures": rows,
    }
    dest = HERE / "figure_locators.json"
    dest.write_text(json.dumps(out, indent=2, ensure_ascii=False) + "\n")
    print(f"wrote {dest}")
    print(f"figures={out['counts']['figures']} assets={out['counts']['assets']}")
    for r in rows:
        for a in r["assets"]:
            print(f"  {a['asset_id']:<14} {a['src']}")
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,127 @@
 {
  "artifact": "figure-locator-llama3-2026-10-24",
  "source": {
    "url": "https://arxiv.org/html/2407.21783v3",
    "mode": "archived+live",
    "archived_bytes": 1072570,
    "archived_sha256": "30595a452414eb355aa49cb8d93a7f7c7a9ae644c82f04cf749e29e759f2a584",
    "archived_content_sha256": "8936398cd5d93d323054e1cb1f959e6703d0bd8c1ff40e1acdfb5832a15d8688",
    "fetched_at_utc": "2026-10-24T16:21:32Z",
    "live_http_status": 200,
    "live_bytes": 1072715,
    "live_sha256": "271f911e032c723299aff997a12e3664b868069eafbd9f8ceac7cdc9ad941259",
    "live_content_sha256": "8936398cd5d93d323054e1cb1f959e6703d0bd8c1ff40e1acdfb5832a15d8688",
    "live_bytes_equal_archived": false,
    "live_paper_content_equal_archived": true
  },
  "locator_rule": "asset_id from <img id=...>; caption from the same outer <figure> block",
  "counts": {
    "figures": 7,
    "assets": 8
  },
  "figures": [
    {
      "figure_id": "S4.F8",
      "section": 4,
      "figure_number": 8,
      "assets": [
        {
          "asset_id": "S4.F8.g1",
          "src": "2407.21783v3/code_translation_example.png",
          "panel": 1
        }
      ],
      "caption": "Figure 8: Code translation example. We display an example of using Llama 3 to translate Python code (left) to PHP code (right) to augment our SFT dataset with a wider range of programming languages.",
      "caption_tag": "Figure 8:"
    },
    {
      "figure_id": "S4.F9",
      "section": 4,
      "figure_number": 9,
      "assets": [
        {
          "asset_id": "S4.F9.g1",
          "src": "2407.21783v3/code_system_prompt_example.png",
          "panel": 1
        }
      ],
      "caption": "Figure 9: Improving generated code quality with system prompts. Left: without system prompt Right: with system prompt.",
      "caption_tag": "Figure 9:"
    },
    {
      "figure_id": "S4.F10",
      "section": 4,
      "figure_number": 10,
      "assets": [
        {
          "asset_id": "S4.F10.g1",
          "src": "2407.21783v3/llama-search-code.png",
          "panel": 1
        }
      ],
      "caption": "Figure 10: Multi-step tool usage. Example of Llama 3 performing multi-step planning, reasoning, and tool calling to solve a task.",
      "caption_tag": "Figure 10:"
    },
    {
      "figure_id": "S4.F11",
      "section": 4,
      "figure_number": 11,
      "assets": [
        {
          "asset_id": "S4.F11.g1",
          "src": "2407.21783v3/assets/llama-file-upload-6x.png",
          "panel": 1
        }
      ],
      "caption": "Figure 11: Processing file uploads. Example of Llama 3 performing analysis and visualization of an uploaded file.",
      "caption_tag": "Figure 11:"
    },
    {
      "figure_id": "S6.F25",
      "section": 6,
      "figure_number": 25,
      "assets": [
        {
          "asset_id": "S6.F25.g1",
          "src": "2407.21783v3/llama3-figure-25.png",
          "panel": 1
        }
      ],
      "caption": "Figure 25: Illustration of tensor-wise and row-wise FP8 quantization. Right: Row-wise quantization enables the use of more granular activation factors than Left: tensor-wise quantization.",
      "caption_tag": "Figure 25:"
    },
    {
      "figure_id": "S6.F27",
      "section": 6,
      "figure_number": 27,
      "assets": [
        {
          "asset_id": "S6.F27.g1",
          "src": "2407.21783v3/assets/prefill_fp8.png",
          "panel": 1
        },
        {
          "asset_id": "S6.F27.g2",
          "src": "2407.21783v3/assets/decode_fp8.png",
          "panel": 2
        }
      ],
      "caption": "Figure 27: Throughput-latency trade-off in FP8 inference with Llama 3 405B compared with BF16 inference using different pipeline parallelization setups. Left: Results for pre-filling. Right: Results for decoding.",
      "caption_tag": "Figure 27:"
    },
    {
      "figure_id": "S8.F29",
      "section": 8,
      "figure_number": 29,
      "assets": [
        {
          "asset_id": "S8.F29.g1",
          "src": "2407.21783v3/assets/llama-speech-6x.png",
          "panel": 1
        }
      ],
      "caption": "Figure 29: Architecture of our speech interface for Llama 3.",
      "caption_tag": "Figure 29:"
    }
  ]
 }
@@ -0,0 +1,185 @@
 #!/usr/bin/env python3
 """Falsifiability battery for verify_locators.py.
 A verifier that cannot fail is not a verifier.  This battery copies the artifact
 to a scratch directory, applies one mutation at a time, and requires the verifier
 to exit NONZERO for every mutation -- and to exit ZERO on the untouched copy.
 Exit 0 = every mutation was caught and the clean copy passed.
 Exit 1 = at least one mutation went undetected, or the clean copy failed.
    python3 mutation_battery.py            # uses the artifact's own directory
    python3 mutation_battery.py --root DIR
 Written 2026-10-25: the correction note promised this battery and the file did
 not exist.  It is now real.
 """
 from __future__ import annotations
 import argparse
 import json
 import re
 import shutil
 import subprocess
 import sys
 import tempfile
 from pathlib import Path
 HERE = Path(__file__).resolve().parent
 VERIFY = HERE / "verify_locators.py"
 HTML_REL = Path("source") / "llama3.html"
 JSON_REL = "figure_locators.json"
 NOTE_REL = "CORRECTION-llama3-figure-locators-2026-10-24.md"
 # Each mutation: (name, callable(root: Path) -> None)
 MUTATIONS: list[tuple[str, str]] = [
    ("json: S4.F11 asset src tampered",
     "figures[S4.F11].assets[0].src"),
    ("json: S6.F27.g2 src tampered",
     "figures[S6.F27].assets[1].src"),
    ("json: S4.F11 row removed",
     "figures.remove(S4.F11)"),
    ("json: counts figures 7 -> 8",
     "counts.figures = 8"),
    ("note: 'Figure 11' -> 'Figure 10'",
     "note.replace('Figure 11', 'Figure 10')"),
    ("note: Figure 11 caption body altered",
     "note.replace('Processing file uploads. Example of Llama 3 performing analysis and "
     "visualization of an uploaded file.', 'Processing file uploads.')"),
    ("note: 'S6.F27.g2' -> 'S6.F24'",
     "note.replace('S6.F27.g2', 'S6.F24')"),
    ("note: 'U+2019' apostrophe nit removed",
     "note.replace('U+2019', 'U+0027')"),
    ("note: '92.3♢' diamond marker removed",
     "note.replace('92.3♢', '92.3')"),
    ("html: S6.F26 object removed",
     "html.remove(<object id=\"S6.F26.g1\">)"),
    ("html: Figure 27 caption body altered",
     "html.replace('Throughput-latency trade-off in FP8 inference', "
     "'Throughput-latency tradeoff in FP8 inference')"),
 ]
 def _load_json(root: Path) -> dict:
    return json.loads((root / JSON_REL).read_text())
 def _save_json(root: Path, data: dict) -> None:
    (root / JSON_REL).write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n")
 def _row(data: dict, figure_id: str) -> dict:
    return next(r for r in data["figures"] if r["figure_id"] == figure_id)
 def _apply(root: Path, mutation: str) -> None:
    """Apply one mutation to the copy at `root`."""
    if mutation == "figures[S4.F11].assets[0].src":
        data = _load_json(root)
        _row(data, "S4.F11")["assets"][0]["src"] = "tampered.png"
        _save_json(root, data)
    elif mutation == "figures[S6.F27].assets[1].src":
        data = _load_json(root)
        _row(data, "S6.F27")["assets"][1]["src"] = "tampered.png"
        _save_json(root, data)
    elif mutation == "figures.remove(S4.F11)":
        data = _load_json(root)
        data["figures"] = [r for r in data["figures"] if r["figure_id"] != "S4.F11"]
        _save_json(root, data)
    elif mutation == "counts.figures = 8":
        data = _load_json(root)
        data["counts"]["figures"] = 8
        _save_json(root, data)
    elif mutation.startswith("note.replace('Figure 11'"):
        p = root / NOTE_REL
        p.write_text(p.read_text().replace("Figure 11", "Figure 10"))
    elif mutation.startswith("note.replace('Processing file uploads"):
        p = root / NOTE_REL
        p.write_text(
            p.read_text().replace(
                "Processing file uploads. Example of Llama 3 performing analysis and "
                "visualization of an uploaded file.",
                "Processing file uploads.",
            )
        )
    elif mutation == "note.replace('S6.F27.g2', 'S6.F24')":
        p = root / NOTE_REL
        p.write_text(p.read_text().replace("S6.F27.g2", "S6.F24"))
    elif mutation == "note.replace('U+2019', 'U+0027')":
        p = root / NOTE_REL
        p.write_text(p.read_text().replace("U+2019", "U+0027"))
    elif mutation == "note.replace('92.3♢', '92.3')":
        p = root / NOTE_REL
        p.write_text(p.read_text().replace("92.3♢", "92.3"))
    elif mutation == 'html.remove(<object id="S6.F26.g1">)':
        p = root / HTML_REL
        p.write_text(
            re.sub(r'<object[^>]*id="S6\.F26\.g1"[^>]*></object>', "", p.read_text())
        )
    elif mutation.startswith("html.replace('Throughput-latency trade-off"):
        p = root / HTML_REL
        p.write_text(
            p.read_text().replace(
                "Throughput-latency trade-off in FP8 inference",
                "Throughput-latency tradeoff in FP8 inference",
            )
        )
    else:
        raise SystemExit(f"unhandled mutation: {mutation}")
 def run_verifier(root: Path) -> tuple[int, str]:
    proc = subprocess.run(
        [sys.executable, str(VERIFY), "--root", str(root)],
        capture_output=True,
        text=True,
    )
    tail = proc.stdout.strip().splitlines()[-1] if proc.stdout.strip() else ""
    return proc.returncode, tail
 def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--root", default=str(HERE), help="artifact directory to attack")
    args = ap.parse_args()
    src = Path(args.root).resolve()
    if not VERIFY.exists() or not (src / HTML_REL).exists():
        print(f"FAIL  artifact incomplete under {src}")
        return 1
    failures: list[str] = []
    passed = 0
    # 1. the untouched copy must pass
    rc, tail = run_verifier(src)
    if rc == 0:
        print(f"PASS  clean copy verifies  :: {tail}")
        passed += 1
    else:
        print(f"FAIL  clean copy does NOT verify  :: {tail}")
        failures.append("clean copy failed")
    # 2. every mutation must be caught
    for name, mut in MUTATIONS:
        with tempfile.TemporaryDirectory(prefix="locmut-") as td:
            root = Path(td) / "artifact"
            shutil.copytree(src, root)
            _apply(root, mut)
            rc, tail = run_verifier(root)
        if rc != 0:
            print(f"PASS  caught: {name}  :: rc={rc} {tail}")
            passed += 1
        else:
            print(f"FAIL  UNDETECTED: {name}  :: rc=0 {tail}")
            failures.append(name)
    print(f"\n{passed}/{len(MUTATIONS) + 1} mutation checks passed")
    if failures:
        print("FAILED: " + "; ".join(failures))
        return 1
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,14 @@
 #!/bin/sh
 # Entry point for the Llama 3 figure-locator artifact's falsifiers.
 #
 #   bash run_checks.sh          # offline: verify_locators + mutation battery
 #
 # Exit code 0 = every check passed and every mutation was caught.
 # Exit code nonzero = the verifier or the battery failed, and the failing
 # command's own status is propagated.
 set -eu
 DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P)
 python3 "$DIR/verify_locators.py" --root "$DIR"
 python3 "$DIR/mutation_battery.py" --root "$DIR"
@@ -0,0 +1,275 @@
 #!/usr/bin/env python3
 """Independent verifier for the Llama 3 figure-locator correction.
 Second code path: it re-derives the asset->figure table straight from the pinned
 HTML bytes with its own parsing, then compares against the published
 figure_locators.json and against the correction note's claims.
 Exit 0 = every check passed.  Exit 1 = at least one failed.
    python3 verify_locators.py                 # offline, deterministic
    python3 verify_locators.py --root DIR      # read the artifact from DIR
    python3 verify_locators.py --live          # additionally re-fetch over HTTP
 Repaired 2026-10-25: two checks were asserting things the instrument could not
 see.  (1) `derive()` indexed only figures that own an <img>, so S6.F26 -- which
 carries an SVG <object>, not an <img> -- had no reachable caption and its
 negative control could never pass.  Captions are now indexed for every outer
 <figure>.  (2) the verbatim-caption checks demanded the `Figure N: ` label prefix
 inline; the note puts the number in its own table column, so the check now
 requires the label AND the caption body separately, which is stricter than the
 original single substring test.
 """
 from __future__ import annotations
 import argparse
 import hashlib
 import html as htmllib
 import json
 import re
 import sys
 import urllib.request
 from pathlib import Path
 HERE = Path(__file__).resolve().parent
 HTML_REL = Path("source") / "llama3.html"
 JSON_REL = "figure_locators.json"
 NOTE_REL = "CORRECTION-llama3-figure-locators-2026-10-24.md"
 URL = "https://arxiv.org/html/2407.21783v3"
 UA = "sphinx-archive-lane/1.0 (research; contact: sphinx@agents.invalid)"
 ARCHIVED_SHA = "30595a452414eb355aa49cb8d93a7f7c7a9ae644c82f04cf749e29e759f2a584"
 ARCHIVED_BYTES = 1072570
 CONTENT_SHA = "8936398cd5d93d323054e1cb1f959e6703d0bd8c1ff40e1acdfb5832a15d8688"
 IMG_RE = re.compile(r'<img[^>]*src="([^"]+)"[^>]*id="(S\d+\.F\d+\.g\d+)"')
 OBJ_RE = re.compile(r'<object[^>]*data="([^"]+)"[^>]*id="(S\d+\.F\d+\.g\d+)"')
 OUTER_RE = re.compile(r'<figure[^>]*id="(S\d+\.F\d+)"')
 CAP_RE = re.compile(r"<figcaption[^>]*>(.*?)</figcaption>", re.S)
 TAG_RE = re.compile(r"<[^>]+>")
 ROW_RE = re.compile(r'<tr[^>]*id="(S1\.T2\.3\.1\.8)"[^>]*>(.*?)</tr>', re.S)
 TD_RE = re.compile(r"<td[^>]*>(.*?)</td>", re.S)
 LABEL_RE = re.compile(r"^(Figure|Table)\s+\d+:\s*")
 RESULTS: list[tuple[str, bool, str]] = []
 def check(name: str, ok: bool, detail: str = "") -> bool:
    RESULTS.append((name, bool(ok), detail))
    print(f"{'PASS' if ok else 'FAIL'}  {name}" + (f"  :: {detail}" if detail else ""))
    return bool(ok)
 def norm(s: str) -> str:
    """Whitespace-collapse and strip markdown emphasis, for verbatim comparison."""
    return re.sub(r"\s+", " ", re.sub(r"[*_]", "", s)).strip()
 def plain(s: str) -> str:
    return re.sub(r"\s+", " ", htmllib.unescape(TAG_RE.sub("", s))).strip()
 def parse(html_text: str) -> dict:
    """Re-derive the asset->figure table and the caption of every outer figure.
    Returns {"assets": {asset_id: {...}}, "captions": {figure_id: caption},
             "objects": {asset_id: {...}}}.
    """
    assets: dict[str, dict] = {}
    for src, aid in IMG_RE.findall(html_text):
        assets[aid] = {
            "src": src,
            "figure_id": aid.rsplit(".g", 1)[0],
            "panel": int(aid.rsplit(".g", 1)[1]),
        }
    objects: dict[str, dict] = {}
    for data, aid in OBJ_RE.findall(html_text):
        objects[aid] = {
            "src": data,
            "figure_id": aid.rsplit(".g", 1)[0],
            "panel": int(aid.rsplit(".g", 1)[1]),
        }
    figs = [(m.start(), m.group(1)) for m in OUTER_RE.finditer(html_text)]
    caps = [(m.start(), plain(m.group(1))) for m in CAP_RE.finditer(html_text)]
    # The caption belongs to the outer <figure>, not to the <img>/<object>.
    # Index it for EVERY outer figure so an SVG-only figure (S6.F26) is reachable.
    captions: dict[str, str] = {}
    for pos, fid in figs:
        for cpos, text in caps:
            if cpos > pos:
                captions.setdefault(fid, text)
                break
    rows: dict[str, dict] = {}
    for aid, a in assets.items():
        rows.setdefault(
            aid.rsplit(".g", 1)[0],
            {"figure_id": aid.rsplit(".g", 1)[0], "assets": []},
        )["assets"].append({"asset_id": aid, "src": a["src"], "panel": a["panel"]})
    for row in rows.values():
        row["assets"].sort(key=lambda x: x["panel"])
        row["caption"] = captions.get(row["figure_id"], "")
    return {"assets": assets, "objects": objects, "captions": captions, "rows": rows}
 def fingerprint(html_text: str) -> str:
    ids = sorted(set(re.findall(r'id="(S\d+\.F\d+\.g\d+)"', html_text)))
    caps = sorted(plain(c) for c in CAP_RE.findall(html_text))
    return hashlib.sha256(
        json.dumps({"asset_ids": ids, "captions": caps}, sort_keys=True).encode()
    ).hexdigest()
 def tables_match(parsed: dict, loaded: dict) -> bool:
    """True iff the published JSON's figures equal the re-derived table."""
    got = {
        row["figure_id"]: {"assets": row["assets"], "caption": norm(row["caption"])}
        for row in loaded.get("figures", [])
    }
    want = {
        fid: {"assets": r["assets"], "caption": norm(r["caption"])}
        for fid, r in parsed["rows"].items()
    }
    return got == want
 def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--root", default=str(HERE), help="artifact directory to verify")
    ap.add_argument("--live", action="store_true")
    args = ap.parse_args()
    root = Path(args.root).resolve()
    html_f = root / HTML_REL
    json_f = root / JSON_REL
    note_f = root / NOTE_REL
    if not html_f.exists():
        print(f"FAIL  archived HTML missing: {html_f}")
        return 1
    raw = html_f.read_bytes()
    html_text = raw.decode("utf-8", "replace")
    # --- provenance of the pinned bytes -------------------------------------
    check("archived HTML sha256 is the pinned digest",
          hashlib.sha256(raw).hexdigest() == ARCHIVED_SHA,
          hashlib.sha256(raw).hexdigest())
    check("archived HTML byte count", len(raw) == ARCHIVED_BYTES, f"{len(raw)} B")
    check("paper-content fingerprint (ids + captions) is the pinned digest",
          fingerprint(html_text) == CONTENT_SHA, fingerprint(html_text))
    # --- re-derive and compare ----------------------------------------------
    parsed = parse(html_text)
    loaded = json.loads(json_f.read_text())
    check("published table equals the re-derived table",
          tables_match(parsed, loaded),
          f"{len(parsed['rows'])} figures / {len(parsed['assets'])} assets")
    check("counts are 7 figures / 8 assets",
          loaded.get("counts") == {"figures": 7, "assets": 8},
          str(loaded.get("counts")))
    # --- the two corrected locators -----------------------------------------
    check("S4.F11 owns assets/llama-file-upload-6x.png",
          parsed["rows"].get("S4.F11", {}).get("assets", [{}])[0].get("src", "").endswith(
              "assets/llama-file-upload-6x.png"))
    check("S6.F27.g2 owns assets/decode_fp8.png",
          any(a["asset_id"] == "S6.F27.g2" and a["src"].endswith("assets/decode_fp8.png")
              for a in parsed["rows"].get("S6.F27", {}).get("assets", [])))
    # --- negative controls: the old numbers must NOT own these assets -------
    f10 = parsed["rows"].get("S4.F10", {}).get("assets", [])
    check("negative control: S4.F10 owns llama-search-code.png, not the file-upload asset",
          len(f10) == 1 and f10[0]["src"].endswith("llama-search-code.png"),
          f10[0]["src"] if f10 else "none")
    # S6.F26 carries an SVG <object>, so it is correctly absent from the <img>
    # table -- but its caption is still reachable, and that is what the control
    # asserts.  This is the check that could not pass before the repair.
    check("negative control: S6.F26 caption is the reward-score histogram",
          parsed["captions"].get("S6.F26", "").startswith("Figure 26: Reward score distribution"),
          parsed["captions"].get("S6.F26", "")[:70] + "…")
    check("negative control: S6.F26 owns an SVG <object>, not an <img>",
          parsed["objects"].get("S6.F26.g1", {}).get("src", "").endswith("bf16_vs_fp8_new.svg"),
          parsed["objects"].get("S6.F26.g1", {}).get("src", "none"))
    check("negative control: no <img> asset belongs to S6.F26",
          "S6.F26" not in parsed["rows"])
    # --- the note's scope claim about the two SVG-only figures ---------------
    check("scope claim: S6.F28.g1 is the architecture SVG <object>",
          parsed["objects"].get("S6.F28.g1", {}).get("src", "").endswith("llama3_architecture.svg"),
          parsed["objects"].get("S6.F28.g1", {}).get("src", "none"))
    # --- the correction note's claims ---------------------------------------
    note = note_f.read_text()
    check("note records the Figure 10 -> Figure 11 move",
          "Figure 10" in note and "Figure 11" in note and "S4.F11" in note)
    check("note records the Figure 26 -> Figure 27 move",
          "Figure 26" in note and "Figure 27" in note and "S6.F27.g2" in note)
    # Verbatim check, repaired: the note states the label in its own table
    # column, so require the caption BODY verbatim, plus the corrected number
    # paired with the asset on the same line -- that pairing is the claim.
    for fid, asset_basename, label in (
        ("S4.F11", "llama-file-upload-6x.png", "Figure 11"),
        ("S6.F27", "decode_fp8.png", "Figure 27"),
    ):
        cap = parsed["captions"].get(fid, "")
        body = LABEL_RE.sub("", cap, count=1)
        check(f"note quotes the {label} caption body verbatim",
              bool(body) and norm(body) in norm(note),
              cap[:60] + "…")
        paired = any(
            asset_basename in line and label in line for line in note.splitlines()
        )
        check(f"note pairs {asset_basename} with {label}",
              paired,
              "no line carries both" if not paired else "")
    # --- the two nits --------------------------------------------------------
    check("source uses the curly apostrophe in 'model’s responses'",
          "model’s responses" in html_text)
    check("note records the apostrophe nit", "U+2019" in note)
    row = ROW_RE.search(html_text)
    cells = [plain(c) for c in TD_RE.findall(row.group(2))] if row else []
    check("Table 2 GSM8K row: 405B cell is a bare 96.8",
          "96.8" in cells and "96.8♢" not in cells, f"cells={cells}")
    check("Table 2 GSM8K row: the diamond sits on 92.3 and 96.4, not 96.8",
          "92.3♢" in cells and "96.4♢" in cells, f"cells={cells}")
    check("note records that the 405B cell is unmarked",
          "92.3♢" in note and "96.4♢" in note)
    # --- falsifiability: the comparison must actually be able to fail --------
    corrupted = json.loads(json_f.read_text())
    corrupted["figures"][0]["assets"][0]["src"] = "tampered.png"
    check("mutation self-test: a tampered table is detected",
          not tables_match(parsed, corrupted))
    # --- optional live re-fetch ---------------------------------------------
    if args.live:
        try:
            req = urllib.request.Request(URL, headers={"User-Agent": UA})
            with urllib.request.urlopen(req, timeout=60) as resp:
                live_raw, status = resp.read(), resp.status
            live_text = live_raw.decode("utf-8", "replace")
            check("live fetch returns HTTP 200", status == 200, str(status))
            check("live paper-content fingerprint still equals the archived one",
                  fingerprint(live_text) == CONTENT_SHA, fingerprint(live_text))
            check("live asset->figure table equals the archived one",
                  tables_match(parse(live_text), loaded))
            print(f"      live bytes={len(live_raw)} sha256={hashlib.sha256(live_raw).hexdigest()}")
        except Exception as exc:  # noqa: BLE001 - reported, not swallowed
            check("live re-fetch", False, repr(exc))
    failed = [r for r in RESULTS if not r[1]]
    print(f"\n{len(RESULTS) - len(failed)}/{len(RESULTS)} checks passed")
    if failed:
        print("FAILED: " + "; ".join(r[0] for r in failed))
        return 1
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
		`@@ -0,0 +1 @@`
							`bash "$(cd "$(dirname "$0")" && pwd)/run_checks.sh"`