From 304463eb682b8d880fb2941130a3c611983f81fb Mon Sep 17 00:00:00 2001 From: sphinx Date: Sat, 24 Oct 2026 05:15:16 +0000 Subject: [PATCH] sphinx: cited digest of Llama 3 Herd (arXiv:2407.21783v3, 92pp) with verified provenance hash and figure-grounding correction --- wizard-work/sphinx/CHECK.txt | 1 + wizard-work/sphinx/llama3-herd-digest.md | 139 +++++++++++++++++++++++ 2 files changed, 140 insertions(+) create mode 100644 wizard-work/sphinx/CHECK.txt create mode 100644 wizard-work/sphinx/llama3-herd-digest.md diff --git a/wizard-work/sphinx/CHECK.txt b/wizard-work/sphinx/CHECK.txt new file mode 100644 index 0000000..d403451 --- /dev/null +++ b/wizard-work/sphinx/CHECK.txt @@ -0,0 +1 @@ +grep -q "$(sha256sum /root/.hermes/profiles/sphinx/cache/scratch/llama3/llama3.pdf | cut -d' ' -f1)" /root/.hermes/wizard-outputs/sphinx/llama3-herd-digest.md && pdfinfo /root/.hermes/profiles/sphinx/cache/scratch/llama3/llama3.pdf | grep -q 'Pages:.*92' && grep -q '15.6T text tokens' /root/.hermes/wizard-outputs/sphinx/llama3-herd-digest.md && grep -q 'negligible impact' /root/.hermes/wizard-outputs/sphinx/llama3-herd-digest.md diff --git a/wizard-work/sphinx/llama3-herd-digest.md b/wizard-work/sphinx/llama3-herd-digest.md new file mode 100644 index 0000000..dddb584 --- /dev/null +++ b/wizard-work/sphinx/llama3-herd-digest.md @@ -0,0 +1,139 @@ +# Archive Digest — "The Llama 3 Herd of Models" (arXiv:2407.21783v3) + +Lane: Archive — multimodal, long-context research. +Producer: sphinx (Hermes Agent). Method: full-text ingestion of a 92-page technical +report + multimodal analysis of two source figures. Every number below is quoted from +the extracted source text; nothing is inferred from memory. + +--- + +## 1. Provenance (measured) + +| Artifact | URL | bytes | sha256 | +|---|---|---|---| +| HTML (v3) | https://arxiv.org/html/2407.21783v3 | 1,072,570 | 30595a452414eb355aa49cb8d93a7f7c7a9ae644c82f04cf749e29e759f2a584 | +| PDF (v3) | https://arxiv.org/pdf/2407.21783 | 9,833,173 | 481f1599468f95a07d05e97fafe55bbe786dc1624c6f881bcb4c7d14c933d083 | + +- `pdfinfo` on the PDF: **Pages: 92**. +- `pdftotext -layout` extraction: **422,553 chars / 4,983 lines**. +- Live checks at fetch time: `arxiv.org/abs/2407.21783` → HTTP 200; + `export.arxiv.org/abs/2407.21783` → HTTP 200. +- Local canonical copies: `/root/.hermes/profiles/sphinx/cache/scratch/llama3/llama3.{html,pdf}`. + +--- + +## 2. Document scope (section map, from the HTML headings) + +1 Introduction · 2 General Overview · 3 Pre-Training (3.1 Data, 3.2 Architecture, +3.3 Infrastructure/Scaling/Efficiency, 3.4 Training Recipe) · 4 Post-Training +(4.1 Modeling, 4.2 Data, 4.3 Capabilities) · 5 Results (5.1 Pre-trained, 5.2 Post-trained, +5.3 Human Evaluations, 5.4 Safety) · 6 Inference (6.1 Pipeline Parallelism, 6.2 FP8 +Quantization) · 7 Vision Experiments (7.1–7.7) · 8 Speech Experiments (8.1–8.5) · +9 Related Work · 10 Conclusion. + +--- + +## 3. Verified key claims + +| # | Claim | Value | Source | Verbatim evidence | +|---|---|---|---|---| +| 1 | Model herd sizes | 8B, 70B, 405B | §1, §2 | "a herd of three multilingual language models with 8B, 70B, and 405B" | +| 2 | Flagship context window | up to 128K tokens | §1 | "processing information in a context window of up to 128K tokens" | +| 3 | Pre-training compute (405B) | 3.8 × 10^25 FLOPs | §1, §3.2 | "pre-trained using 3.8 × 10^25 FLOPs, almost 50× more than the largest version of Llama 2" | +| 4 | Pre-training corpus | 15.6T text tokens | §1 | "we pre-trained a flagship model with 405B trainable parameters on 15.6T text tokens" | +| 5 | Hardware | up to 16K H100 GPUs, 700W TDP, 80GB HBM3 | §3.3 | "Llama 3 405B is trained on up to 16K H100 GPUs, each running at 700W TDP with 80GB HBM3" | +| 6 | BF16 MFU | 38–43% | §3.3 | "an overall BF16 Model FLOPs Utilization (MFU) of 38-43%" | +| 7 | 405B architecture | 126 layers, dim 16,384, 128 heads | §3.2 | "Llama 3 405B uses an architecture with 126 layers, a token representation dimension of 16,384, and 128 attention heads" | +| 8 | Vocabulary | 128K tokens | §3.2 | "We use a vocabulary with 128K tokens." | +| 9 | MMLU (5-shot) | 8B 69.4 · 70B 83.6 · 405B 87.3 | Table 2 | row "MMLU (5-shot)": 69.4 … 83.6 … 87.3 | +| 10 | GSM8K (8-shot, CoT) | 8B 84.5 · 70B 95.1 · 405B 96.8 | Table 2 | row "GSM8K (8-shot, CoT)": 84.5 … 95.1 … 96.8 | +| 11 | HumanEval (0-shot) | 8B 72.6 · 70B 80.5 · 405B 89.0 | Table 2 | row "HumanEval (0-shot)": 72.6 … 80.5 … 89.0 | +| 12 | MMLU-Pro (5-shot, CoT) | 8B 48.3 · 70B 66.4 · 405B 73.3 | Table 2 | row "MMLU-Pro (5-shot, CoT)": 48.3 … 66.4 … 73.3 | + +Table 2 column order was reconstructed from the HTML `` cells (not guessed): +`Llama 3 8B | Gemma 2 9B | Mistral 7B | Llama 3 70B | Mixtral 8x22B | GPT 3.5 Turbo | +Llama 3 405B | Nemotron 4 340B | GPT-4 (0125) | GPT-4o | Claude 3.5 Sonnet`. + +--- + +## 4. Long-context focus (lane-relevant) + +The paper's long-context recipe, §3.4.2 "Long Context Pre-Training": + +- Standard pre-training uses an **8K** context window; a continued stage extends it to + **128K** ("This standard pre-training stage is followed by a continued pre-training + stage that increases the supported context window to 128K tokens"). +- Long sequences are deferred because "the compute in self-attention layers grows + quadratically in the sequence length." +- Context length was raised **gradually in six stages, from 8K to 128K**. +- Adaptation was judged by (1) short-context recovery and (2) solving + "needle in a haystack" tasks up to that length. +- The long-context stage used **~800B training tokens**. +- Annealing (§3.4.3) ran on the final **40M tokens** at a 128K context length, with + Polyak checkpoint averaging. + +Long-context results (Table 2, rows "Long context"): + +| Benchmark | 8B | 70B | 405B | +|---|---|---|---| +| InfiniteBench/En.MC | 65.1 | 78.2 | 83.4 | +| NIH/Multi-needle | 98.8 | 97.5 | 98.1 | + +Post-training long-context (§4.3.4): mixing **0.1% synthetic long-context data** with +short-context data "optimizes the performance across both short-context and +long-context benchmarks." + +--- + +## 5. Multimodal findings (figures analysed with vision) + +### 5.1 Figure 10 — multi-step tool usage / file upload (asset `llama-file-upload-6x.png`) +Vision transcription of the source image (9240×6968 px) confirms the caption's claim +of "multi-step planning, reasoning, and tool calling": + +- System prompt visible: `Environment: ipython`, `Tools: brave_search, wolfram_alpha`, + `Cutting Knowledge Date: December 2023`, `Today's Date: Jul 11, 2024`. +- User asks about cereal sugar vs. rating and supplies `file_path = "/mnt/data/cereal.csv"`. +- The model first runs EDA (`df.shape` → `(77, 16)`), then fits a linear regression and + reports **R² = 0.577105586609012**, slope **−2.400819943590098**, intercept + **59.284367372640936**, and interprets the negative relationship. + +### 5.2 Provenance correction — `decode_fp8.png` +My asset→caption heuristic mapped `assets/decode_fp8.png` to "Figure 26: Reward score +distribution … BF16 and FP8". Vision analysis of the actual pixels **contradicted that**: +the image is a throughput-vs-latency plot (axes "Decode Latency (ms)" / "Decode +Throughput (tokens/sec)") with curves TP8/PP2 (BF16), TP8/PP2 (BF16)+Microbatching, +TP4/PP4 (BF16)+Microbatching, and TP8 (FP8) — i.e. the **decode panel of Figure 24** +("Effect of micro-batching on inference throughput and latency … decoding"), not +Figure 26. Recorded as a measured correction; the nearest-preceding-figcaption mapping +is unreliable for this asset. + +### 5.3 FP8 quality (textual, §6.2) +Independent of the figure mix-up, the paper's own text states the FP8 claim: +"Our FP8 quantization approach has negligible impact on the model's responses" and +"our approach to FP8 quantization has very limited impact on the model's response" +(measured over "reward-model scores for 100,000 responses produced using both FP8 and +BF16"). FP8 gave "throughput improvements of up to 50% during the pre-fill stage." + +--- + +## 6. Stated limitations (authors' own words) + +- "Because these safety benchmarks are internal to Meta, we acknowledge that the + numbers in this section are not reproducible" (§5.4). +- Human evaluation: "human evaluations can still be influenced by personal biases, + backgrounds, and preferences of human annotators, which may lead to inconsistent or + unreliable results" (§5.3). +- 8B/70B "were pre-trained on multilingual data but were intended for use in English + at the time" (footnote 1). + +--- + +## 7. Verification + +The integrity of this digest is anchored to the sha256 of the actual source PDF it +summarises. Re-run the CHECK line; exit 0 means the digest's recorded provenance hash +matches the real file on disk and the digest carries its two headline claims. + +- CHECK: see the single command in `CHECK.txt` next to this file. +- Expected: exit 0.