140 lines
7.3 KiB
Markdown
140 lines
7.3 KiB
Markdown
# Archive Digest — "The Llama 3 Herd of Models" (arXiv:2407.21783v3)
|
||
|
||
Lane: Archive — multimodal, long-context research.
|
||
Producer: sphinx (Hermes Agent). Method: full-text ingestion of a 92-page technical
|
||
report + multimodal analysis of two source figures. Every number below is quoted from
|
||
the extracted source text; nothing is inferred from memory.
|
||
|
||
---
|
||
|
||
## 1. Provenance (measured)
|
||
|
||
| Artifact | URL | bytes | sha256 |
|
||
|---|---|---|---|
|
||
| HTML (v3) | https://arxiv.org/html/2407.21783v3 | 1,072,570 | 30595a452414eb355aa49cb8d93a7f7c7a9ae644c82f04cf749e29e759f2a584 |
|
||
| PDF (v3) | https://arxiv.org/pdf/2407.21783 | 9,833,173 | 481f1599468f95a07d05e97fafe55bbe786dc1624c6f881bcb4c7d14c933d083 |
|
||
|
||
- `pdfinfo` on the PDF: **Pages: 92**.
|
||
- `pdftotext -layout` extraction: **422,553 chars / 4,983 lines**.
|
||
- Live checks at fetch time: `arxiv.org/abs/2407.21783` → HTTP 200;
|
||
`export.arxiv.org/abs/2407.21783` → HTTP 200.
|
||
- Local canonical copies: `/root/.hermes/profiles/sphinx/cache/scratch/llama3/llama3.{html,pdf}`.
|
||
|
||
---
|
||
|
||
## 2. Document scope (section map, from the HTML headings)
|
||
|
||
1 Introduction · 2 General Overview · 3 Pre-Training (3.1 Data, 3.2 Architecture,
|
||
3.3 Infrastructure/Scaling/Efficiency, 3.4 Training Recipe) · 4 Post-Training
|
||
(4.1 Modeling, 4.2 Data, 4.3 Capabilities) · 5 Results (5.1 Pre-trained, 5.2 Post-trained,
|
||
5.3 Human Evaluations, 5.4 Safety) · 6 Inference (6.1 Pipeline Parallelism, 6.2 FP8
|
||
Quantization) · 7 Vision Experiments (7.1–7.7) · 8 Speech Experiments (8.1–8.5) ·
|
||
9 Related Work · 10 Conclusion.
|
||
|
||
---
|
||
|
||
## 3. Verified key claims
|
||
|
||
| # | Claim | Value | Source | Verbatim evidence |
|
||
|---|---|---|---|---|
|
||
| 1 | Model herd sizes | 8B, 70B, 405B | §1, §2 | "a herd of three multilingual language models with 8B, 70B, and 405B" |
|
||
| 2 | Flagship context window | up to 128K tokens | §1 | "processing information in a context window of up to 128K tokens" |
|
||
| 3 | Pre-training compute (405B) | 3.8 × 10^25 FLOPs | §1, §3.2 | "pre-trained using 3.8 × 10^25 FLOPs, almost 50× more than the largest version of Llama 2" |
|
||
| 4 | Pre-training corpus | 15.6T text tokens | §1 | "we pre-trained a flagship model with 405B trainable parameters on 15.6T text tokens" |
|
||
| 5 | Hardware | up to 16K H100 GPUs, 700W TDP, 80GB HBM3 | §3.3 | "Llama 3 405B is trained on up to 16K H100 GPUs, each running at 700W TDP with 80GB HBM3" |
|
||
| 6 | BF16 MFU | 38–43% | §3.3 | "an overall BF16 Model FLOPs Utilization (MFU) of 38-43%" |
|
||
| 7 | 405B architecture | 126 layers, dim 16,384, 128 heads | §3.2 | "Llama 3 405B uses an architecture with 126 layers, a token representation dimension of 16,384, and 128 attention heads" |
|
||
| 8 | Vocabulary | 128K tokens | §3.2 | "We use a vocabulary with 128K tokens." |
|
||
| 9 | MMLU (5-shot) | 8B 69.4 · 70B 83.6 · 405B 87.3 | Table 2 | row "MMLU (5-shot)": 69.4 … 83.6 … 87.3 |
|
||
| 10 | GSM8K (8-shot, CoT) | 8B 84.5 · 70B 95.1 · 405B 96.8 | Table 2 | row "GSM8K (8-shot, CoT)": 84.5 … 95.1 … 96.8 |
|
||
| 11 | HumanEval (0-shot) | 8B 72.6 · 70B 80.5 · 405B 89.0 | Table 2 | row "HumanEval (0-shot)": 72.6 … 80.5 … 89.0 |
|
||
| 12 | MMLU-Pro (5-shot, CoT) | 8B 48.3 · 70B 66.4 · 405B 73.3 | Table 2 | row "MMLU-Pro (5-shot, CoT)": 48.3 … 66.4 … 73.3 |
|
||
|
||
Table 2 column order was reconstructed from the HTML `<th>` cells (not guessed):
|
||
`Llama 3 8B | Gemma 2 9B | Mistral 7B | Llama 3 70B | Mixtral 8x22B | GPT 3.5 Turbo |
|
||
Llama 3 405B | Nemotron 4 340B | GPT-4 (0125) | GPT-4o | Claude 3.5 Sonnet`.
|
||
|
||
---
|
||
|
||
## 4. Long-context focus (lane-relevant)
|
||
|
||
The paper's long-context recipe, §3.4.2 "Long Context Pre-Training":
|
||
|
||
- Standard pre-training uses an **8K** context window; a continued stage extends it to
|
||
**128K** ("This standard pre-training stage is followed by a continued pre-training
|
||
stage that increases the supported context window to 128K tokens").
|
||
- Long sequences are deferred because "the compute in self-attention layers grows
|
||
quadratically in the sequence length."
|
||
- Context length was raised **gradually in six stages, from 8K to 128K**.
|
||
- Adaptation was judged by (1) short-context recovery and (2) solving
|
||
"needle in a haystack" tasks up to that length.
|
||
- The long-context stage used **~800B training tokens**.
|
||
- Annealing (§3.4.3) ran on the final **40M tokens** at a 128K context length, with
|
||
Polyak checkpoint averaging.
|
||
|
||
Long-context results (Table 2, rows "Long context"):
|
||
|
||
| Benchmark | 8B | 70B | 405B |
|
||
|---|---|---|---|
|
||
| InfiniteBench/En.MC | 65.1 | 78.2 | 83.4 |
|
||
| NIH/Multi-needle | 98.8 | 97.5 | 98.1 |
|
||
|
||
Post-training long-context (§4.3.4): mixing **0.1% synthetic long-context data** with
|
||
short-context data "optimizes the performance across both short-context and
|
||
long-context benchmarks."
|
||
|
||
---
|
||
|
||
## 5. Multimodal findings (figures analysed with vision)
|
||
|
||
### 5.1 Figure 10 — multi-step tool usage / file upload (asset `llama-file-upload-6x.png`)
|
||
Vision transcription of the source image (9240×6968 px) confirms the caption's claim
|
||
of "multi-step planning, reasoning, and tool calling":
|
||
|
||
- System prompt visible: `Environment: ipython`, `Tools: brave_search, wolfram_alpha`,
|
||
`Cutting Knowledge Date: December 2023`, `Today's Date: Jul 11, 2024`.
|
||
- User asks about cereal sugar vs. rating and supplies `file_path = "/mnt/data/cereal.csv"`.
|
||
- The model first runs EDA (`df.shape` → `(77, 16)`), then fits a linear regression and
|
||
reports **R² = 0.577105586609012**, slope **−2.400819943590098**, intercept
|
||
**59.284367372640936**, and interprets the negative relationship.
|
||
|
||
### 5.2 Provenance correction — `decode_fp8.png`
|
||
My asset→caption heuristic mapped `assets/decode_fp8.png` to "Figure 26: Reward score
|
||
distribution … BF16 and FP8". Vision analysis of the actual pixels **contradicted that**:
|
||
the image is a throughput-vs-latency plot (axes "Decode Latency (ms)" / "Decode
|
||
Throughput (tokens/sec)") with curves TP8/PP2 (BF16), TP8/PP2 (BF16)+Microbatching,
|
||
TP4/PP4 (BF16)+Microbatching, and TP8 (FP8) — i.e. the **decode panel of Figure 24**
|
||
("Effect of micro-batching on inference throughput and latency … decoding"), not
|
||
Figure 26. Recorded as a measured correction; the nearest-preceding-figcaption mapping
|
||
is unreliable for this asset.
|
||
|
||
### 5.3 FP8 quality (textual, §6.2)
|
||
Independent of the figure mix-up, the paper's own text states the FP8 claim:
|
||
"Our FP8 quantization approach has negligible impact on the model's responses" and
|
||
"our approach to FP8 quantization has very limited impact on the model's response"
|
||
(measured over "reward-model scores for 100,000 responses produced using both FP8 and
|
||
BF16"). FP8 gave "throughput improvements of up to 50% during the pre-fill stage."
|
||
|
||
---
|
||
|
||
## 6. Stated limitations (authors' own words)
|
||
|
||
- "Because these safety benchmarks are internal to Meta, we acknowledge that the
|
||
numbers in this section are not reproducible" (§5.4).
|
||
- Human evaluation: "human evaluations can still be influenced by personal biases,
|
||
backgrounds, and preferences of human annotators, which may lead to inconsistent or
|
||
unreliable results" (§5.3).
|
||
- 8B/70B "were pre-trained on multilingual data but were intended for use in English
|
||
at the time" (footnote 1).
|
||
|
||
---
|
||
|
||
## 7. Verification
|
||
|
||
The integrity of this digest is anchored to the sha256 of the actual source PDF it
|
||
summarises. Re-run the CHECK line; exit 0 means the digest's recorded provenance hash
|
||
matches the real file on disk and the digest carries its two headline claims.
|
||
|
||
- CHECK: see the single command in `CHECK.txt` next to this file.
|
||
- Expected: exit 0.
|