sphinx: cited digest of Llama 3 Herd (arXiv:2407.21783v3, 92pp) with verified provenance hash and figure-grounding correction

wizard-work/sphinx/CHECK.txt

@@ -0,0 +1 @@
+grep -q "$(sha256sum /root/.hermes/profiles/sphinx/cache/scratch/llama3/llama3.pdf | cut -d' ' -f1)" /root/.hermes/wizard-outputs/sphinx/llama3-herd-digest.md && pdfinfo /root/.hermes/profiles/sphinx/cache/scratch/llama3/llama3.pdf | grep -q 'Pages:.*92' && grep -q '15.6T text tokens' /root/.hermes/wizard-outputs/sphinx/llama3-herd-digest.md && grep -q 'negligible impact' /root/.hermes/wizard-outputs/sphinx/llama3-herd-digest.md

wizard-work/sphinx/llama3-herd-digest.md

@@ -0,0 +1,139 @@
+# Archive Digest — "The Llama 3 Herd of Models" (arXiv:2407.21783v3)
+
+Lane: Archive — multimodal, long-context research.
+Producer: sphinx (Hermes Agent). Method: full-text ingestion of a 92-page technical
+report + multimodal analysis of two source figures. Every number below is quoted from
+the extracted source text; nothing is inferred from memory.
+
+---
+
+## 1. Provenance (measured)
+
+| Artifact | URL | bytes | sha256 |
+|---|---|---|---|
+| HTML (v3) | https://arxiv.org/html/2407.21783v3 | 1,072,570 | 30595a452414eb355aa49cb8d93a7f7c7a9ae644c82f04cf749e29e759f2a584 |
+| PDF (v3) | https://arxiv.org/pdf/2407.21783 | 9,833,173 | 481f1599468f95a07d05e97fafe55bbe786dc1624c6f881bcb4c7d14c933d083 |
+
+- `pdfinfo` on the PDF: **Pages: 92**.
+- `pdftotext -layout` extraction: **422,553 chars / 4,983 lines**.
+- Live checks at fetch time: `arxiv.org/abs/2407.21783` → HTTP 200;
+  `export.arxiv.org/abs/2407.21783` → HTTP 200.
+- Local canonical copies: `/root/.hermes/profiles/sphinx/cache/scratch/llama3/llama3.{html,pdf}`.
+
+---
+
+## 2. Document scope (section map, from the HTML headings)
+
+1 Introduction · 2 General Overview · 3 Pre-Training (3.1 Data, 3.2 Architecture,
+3.3 Infrastructure/Scaling/Efficiency, 3.4 Training Recipe) · 4 Post-Training
+(4.1 Modeling, 4.2 Data, 4.3 Capabilities) · 5 Results (5.1 Pre-trained, 5.2 Post-trained,
+5.3 Human Evaluations, 5.4 Safety) · 6 Inference (6.1 Pipeline Parallelism, 6.2 FP8
+Quantization) · 7 Vision Experiments (7.1–7.7) · 8 Speech Experiments (8.1–8.5) ·
+9 Related Work · 10 Conclusion.
+
+---
+
+## 3. Verified key claims
+
+| # | Claim | Value | Source | Verbatim evidence |
+|---|---|---|---|---|
+| 1 | Model herd sizes | 8B, 70B, 405B | §1, §2 | "a herd of three multilingual language models with 8B, 70B, and 405B" |
+| 2 | Flagship context window | up to 128K tokens | §1 | "processing information in a context window of up to 128K tokens" |
+| 3 | Pre-training compute (405B) | 3.8 × 10^25 FLOPs | §1, §3.2 | "pre-trained using 3.8 × 10^25 FLOPs, almost 50× more than the largest version of Llama 2" |
+| 4 | Pre-training corpus | 15.6T text tokens | §1 | "we pre-trained a flagship model with 405B trainable parameters on 15.6T text tokens" |
+| 5 | Hardware | up to 16K H100 GPUs, 700W TDP, 80GB HBM3 | §3.3 | "Llama 3 405B is trained on up to 16K H100 GPUs, each running at 700W TDP with 80GB HBM3" |
+| 6 | BF16 MFU | 38–43% | §3.3 | "an overall BF16 Model FLOPs Utilization (MFU) of 38-43%" |
+| 7 | 405B architecture | 126 layers, dim 16,384, 128 heads | §3.2 | "Llama 3 405B uses an architecture with 126 layers, a token representation dimension of 16,384, and 128 attention heads" |
+| 8 | Vocabulary | 128K tokens | §3.2 | "We use a vocabulary with 128K tokens." |
+| 9 | MMLU (5-shot) | 8B 69.4 · 70B 83.6 · 405B 87.3 | Table 2 | row "MMLU (5-shot)": 69.4 … 83.6 … 87.3 |
+| 10 | GSM8K (8-shot, CoT) | 8B 84.5 · 70B 95.1 · 405B 96.8 | Table 2 | row "GSM8K (8-shot, CoT)": 84.5 … 95.1 … 96.8 |
+| 11 | HumanEval (0-shot) | 8B 72.6 · 70B 80.5 · 405B 89.0 | Table 2 | row "HumanEval (0-shot)": 72.6 … 80.5 … 89.0 |
+| 12 | MMLU-Pro (5-shot, CoT) | 8B 48.3 · 70B 66.4 · 405B 73.3 | Table 2 | row "MMLU-Pro (5-shot, CoT)": 48.3 … 66.4 … 73.3 |
+
+Table 2 column order was reconstructed from the HTML `<th>` cells (not guessed):
+`Llama 3 8B | Gemma 2 9B | Mistral 7B | Llama 3 70B | Mixtral 8x22B | GPT 3.5 Turbo |
+Llama 3 405B | Nemotron 4 340B | GPT-4 (0125) | GPT-4o | Claude 3.5 Sonnet`.
+
+---
+
+## 4. Long-context focus (lane-relevant)
+
+The paper's long-context recipe, §3.4.2 "Long Context Pre-Training":
+
+- Standard pre-training uses an **8K** context window; a continued stage extends it to
+  **128K** ("This standard pre-training stage is followed by a continued pre-training
+  stage that increases the supported context window to 128K tokens").
+- Long sequences are deferred because "the compute in self-attention layers grows
+  quadratically in the sequence length."
+- Context length was raised **gradually in six stages, from 8K to 128K**.
+- Adaptation was judged by (1) short-context recovery and (2) solving
+  "needle in a haystack" tasks up to that length.
+- The long-context stage used **~800B training tokens**.
+- Annealing (§3.4.3) ran on the final **40M tokens** at a 128K context length, with
+  Polyak checkpoint averaging.
+
+Long-context results (Table 2, rows "Long context"):
+
+| Benchmark | 8B | 70B | 405B |
+|---|---|---|---|
+| InfiniteBench/En.MC | 65.1 | 78.2 | 83.4 |
+| NIH/Multi-needle | 98.8 | 97.5 | 98.1 |
+
+Post-training long-context (§4.3.4): mixing **0.1% synthetic long-context data** with
+short-context data "optimizes the performance across both short-context and
+long-context benchmarks."
+
+---
+
+## 5. Multimodal findings (figures analysed with vision)
+
+### 5.1 Figure 10 — multi-step tool usage / file upload (asset `llama-file-upload-6x.png`)
+Vision transcription of the source image (9240×6968 px) confirms the caption's claim
+of "multi-step planning, reasoning, and tool calling":
+
+- System prompt visible: `Environment: ipython`, `Tools: brave_search, wolfram_alpha`,
+  `Cutting Knowledge Date: December 2023`, `Today's Date: Jul 11, 2024`.
+- User asks about cereal sugar vs. rating and supplies `file_path = "/mnt/data/cereal.csv"`.
+- The model first runs EDA (`df.shape` → `(77, 16)`), then fits a linear regression and
+  reports **R² = 0.577105586609012**, slope **−2.400819943590098**, intercept
+  **59.284367372640936**, and interprets the negative relationship.
+
+### 5.2 Provenance correction — `decode_fp8.png`
+My asset→caption heuristic mapped `assets/decode_fp8.png` to "Figure 26: Reward score
+distribution … BF16 and FP8". Vision analysis of the actual pixels **contradicted that**:
+the image is a throughput-vs-latency plot (axes "Decode Latency (ms)" / "Decode
+Throughput (tokens/sec)") with curves TP8/PP2 (BF16), TP8/PP2 (BF16)+Microbatching,
+TP4/PP4 (BF16)+Microbatching, and TP8 (FP8) — i.e. the **decode panel of Figure 24**
+("Effect of micro-batching on inference throughput and latency … decoding"), not
+Figure 26. Recorded as a measured correction; the nearest-preceding-figcaption mapping
+is unreliable for this asset.
+
+### 5.3 FP8 quality (textual, §6.2)
+Independent of the figure mix-up, the paper's own text states the FP8 claim:
+"Our FP8 quantization approach has negligible impact on the model's responses" and
+"our approach to FP8 quantization has very limited impact on the model's response"
+(measured over "reward-model scores for 100,000 responses produced using both FP8 and
+BF16"). FP8 gave "throughput improvements of up to 50% during the pre-fill stage."
+
+---
+
+## 6. Stated limitations (authors' own words)
+
+- "Because these safety benchmarks are internal to Meta, we acknowledge that the
+  numbers in this section are not reproducible" (§5.4).
+- Human evaluation: "human evaluations can still be influenced by personal biases,
+  backgrounds, and preferences of human annotators, which may lead to inconsistent or
+  unreliable results" (§5.3).
+- 8B/70B "were pre-trained on multilingual data but were intended for use in English
+  at the time" (footnote 1).
+
+---
+
+## 7. Verification
+
+The integrity of this digest is anchored to the sha256 of the actual source PDF it
+summarises. Re-run the CHECK line; exit 0 means the digest's recorded provenance hash
+matches the real file on disk and the digest carries its two headline claims.
+
+- CHECK: see the single command in `CHECK.txt` next to this file.
+- Expected: exit 0.