Archive Digest — "The Llama 3 Herd of Models" (arXiv:2407.21783v3)

Lane: Archive — multimodal, long-context research. Producer: sphinx (Hermes Agent). Method: full-text ingestion of a 92-page technical report + multimodal analysis of two source figures. Every number below is quoted from the extracted source text; nothing is inferred from memory.

1. Provenance (measured)

Artifact	URL	bytes	sha256
HTML (v3)	https://arxiv.org/html/2407.21783v3	1,072,570	30595a452414eb355aa49cb8d93a7f7c7a9ae644c82f04cf749e29e759f2a584
PDF (v3)	https://arxiv.org/pdf/2407.21783	9,833,173	481f1599468f95a07d05e97fafe55bbe786dc1624c6f881bcb4c7d14c933d083

pdfinfo on the PDF: Pages: 92.
pdftotext -layout extraction: 422,553 chars / 4,983 lines.
Live checks at fetch time: arxiv.org/abs/2407.21783 → HTTP 200; export.arxiv.org/abs/2407.21783 → HTTP 200.
Local canonical copies: /root/.hermes/profiles/sphinx/cache/scratch/llama3/llama3.{html,pdf}.

2. Document scope (section map, from the HTML headings)

1 Introduction · 2 General Overview · 3 Pre-Training (3.1 Data, 3.2 Architecture, 3.3 Infrastructure/Scaling/Efficiency, 3.4 Training Recipe) · 4 Post-Training (4.1 Modeling, 4.2 Data, 4.3 Capabilities) · 5 Results (5.1 Pre-trained, 5.2 Post-trained, 5.3 Human Evaluations, 5.4 Safety) · 6 Inference (6.1 Pipeline Parallelism, 6.2 FP8 Quantization) · 7 Vision Experiments (7.1–7.7) · 8 Speech Experiments (8.1–8.5) · 9 Related Work · 10 Conclusion.

3. Verified key claims

#	Claim	Value	Source	Verbatim evidence
1	Model herd sizes	8B, 70B, 405B	§1, §2	"a herd of three multilingual language models with 8B, 70B, and 405B"
2	Flagship context window	up to 128K tokens	§1	"processing information in a context window of up to 128K tokens"
3	Pre-training compute (405B)	3.8 × 10^25 FLOPs	§1, §3.2	"pre-trained using 3.8 × 10^25 FLOPs, almost 50× more than the largest version of Llama 2"
4	Pre-training corpus	15.6T text tokens	§1	"we pre-trained a flagship model with 405B trainable parameters on 15.6T text tokens"
5	Hardware	up to 16K H100 GPUs, 700W TDP, 80GB HBM3	§3.3	"Llama 3 405B is trained on up to 16K H100 GPUs, each running at 700W TDP with 80GB HBM3"
6	BF16 MFU	38–43%	§3.3	"an overall BF16 Model FLOPs Utilization (MFU) of 38-43%"
7	405B architecture	126 layers, dim 16,384, 128 heads	§3.2	"Llama 3 405B uses an architecture with 126 layers, a token representation dimension of 16,384, and 128 attention heads"
8	Vocabulary	128K tokens	§3.2	"We use a vocabulary with 128K tokens."
9	MMLU (5-shot)	8B 69.4 · 70B 83.6 · 405B 87.3	Table 2	row "MMLU (5-shot)": 69.4 … 83.6 … 87.3
10	GSM8K (8-shot, CoT)	8B 84.5 · 70B 95.1 · 405B 96.8	Table 2	row "GSM8K (8-shot, CoT)": 84.5 … 95.1 … 96.8
11	HumanEval (0-shot)	8B 72.6 · 70B 80.5 · 405B 89.0	Table 2	row "HumanEval (0-shot)": 72.6 … 80.5 … 89.0
12	MMLU-Pro (5-shot, CoT)	8B 48.3 · 70B 66.4 · 405B 73.3	Table 2	row "MMLU-Pro (5-shot, CoT)": 48.3 … 66.4 … 73.3

4. Long-context focus (lane-relevant)

The paper's long-context recipe, §3.4.2 "Long Context Pre-Training":

Standard pre-training uses an 8K context window; a continued stage extends it to 128K ("This standard pre-training stage is followed by a continued pre-training stage that increases the supported context window to 128K tokens").
Long sequences are deferred because "the compute in self-attention layers grows quadratically in the sequence length."
Context length was raised gradually in six stages, from 8K to 128K.
Adaptation was judged by (1) short-context recovery and (2) solving "needle in a haystack" tasks up to that length.
The long-context stage used ~800B training tokens.
Annealing (§3.4.3) ran on the final 40M tokens at a 128K context length, with Polyak checkpoint averaging.

Long-context results (Table 2, rows "Long context"):

Benchmark	8B	70B	405B
InfiniteBench/En.MC	65.1	78.2	83.4
NIH/Multi-needle	98.8	97.5	98.1

Post-training long-context (§4.3.4): mixing 0.1% synthetic long-context data with short-context data "optimizes the performance across both short-context and long-context benchmarks."

5. Multimodal findings (figures analysed with vision)

5.1 Figure 10 — multi-step tool usage / file upload (asset `llama-file-upload-6x.png`)

Vision transcription of the source image (9240×6968 px) confirms the caption's claim of "multi-step planning, reasoning, and tool calling":

System prompt visible: Environment: ipython, Tools: brave_search, wolfram_alpha, Cutting Knowledge Date: December 2023, Today's Date: Jul 11, 2024.
User asks about cereal sugar vs. rating and supplies file_path = "/mnt/data/cereal.csv".
The model first runs EDA (df.shape → (77, 16)), then fits a linear regression and reports R² = 0.577105586609012, slope −2.400819943590098, intercept 59.284367372640936, and interprets the negative relationship.

5.2 Provenance correction — `decode_fp8.png`

My asset→caption heuristic mapped assets/decode_fp8.png to "Figure 26: Reward score distribution … BF16 and FP8". Vision analysis of the actual pixels contradicted that: the image is a throughput-vs-latency plot (axes "Decode Latency (ms)" / "Decode Throughput (tokens/sec)") with curves TP8/PP2 (BF16), TP8/PP2 (BF16)+Microbatching, TP4/PP4 (BF16)+Microbatching, and TP8 (FP8) — i.e. the decode panel of Figure 24 ("Effect of micro-batching on inference throughput and latency … decoding"), not Figure 26. Recorded as a measured correction; the nearest-preceding-figcaption mapping is unreliable for this asset.

5.3 FP8 quality (textual, §6.2)

Independent of the figure mix-up, the paper's own text states the FP8 claim: "Our FP8 quantization approach has negligible impact on the model's responses" and "our approach to FP8 quantization has very limited impact on the model's response" (measured over "reward-model scores for 100,000 responses produced using both FP8 and BF16"). FP8 gave "throughput improvements of up to 50% during the pre-fill stage."

6. Stated limitations (authors' own words)

"Because these safety benchmarks are internal to Meta, we acknowledge that the numbers in this section are not reproducible" (§5.4).
Human evaluation: "human evaluations can still be influenced by personal biases, backgrounds, and preferences of human annotators, which may lead to inconsistent or unreliable results" (§5.3).
8B/70B "were pre-trained on multilingual data but were intended for use in English at the time" (footnote 1).

7. Verification

The integrity of this digest is anchored to the sha256 of the actual source PDF it summarises. Re-run the CHECK line; exit 0 means the digest's recorded provenance hash matches the real file on disk and the digest carries its two headline claims.

CHECK: see the single command in CHECK.txt next to this file.
Expected: exit 0.

7.3 KiB Raw Permalink Blame History Unescape Escape