. Accuracy† denotes prompt-level loose accuracy.
Benchmark
Metric
Type
CoT
#Shots
Chat
Turns
#Langs
Knowledge and Reasoning
MMLU (zero-shot)
Exact Match
MCQ (Gen)
✓
0
✓
×\times
1
MMLU (few-shot)
Exact Match
MCQ (Gen)
✓
4
×\times
×\times
1
MMLU-Pro
Exact Match
MCQ (Gen)
✓
5
✓
✓
1
AGIEval
Accuracy
MCQ (Gen/LLH)
×\times
0
✓
×\times
2
ARC Challenge Chat
Exact Match
MCQ (Gen)
×\times
0
✓
×\times
1
TruthfulQA (MC2)
Accuracy
MCQ (LLH)
×\times
0
✓
×\times
1
CommonsenseQA
Accuracy
MCQ (LLH)
×\times
0
✓
×\times
1
SQuAD v2
F1
Gen
×\times
0
✓
×\times
1
HellaSwag
Accuracy (norm.)
MCQ (LLH)
×\times
0
✓
×\times
1
PIQA
Accuracy (norm.)
MCQ (LLH)
×\times
0
×\times
×\times
1
BBH
Exact Match
Gen
✓
3
✓
✓
1
GPQA Main
Exact Match
MCQ (Gen)
✓
0
✓
×\times
1
ACPBench
Exact Match
MCQ (Gen)
✓
2
✓
✓
1
DROP
F1
Gen
×\times
3
✓
✓
1
LAMBADA
Accuracy
LLH
×\times
0
×\times
×\times
1
Instruction Following and Chat
IFEval
Accuracy†
Gen
×\times
0
✓
×\times
1
Multi-IFEval
Accuracy†
Gen
×\times
0
✓
×\times
8
AlpacaEval 2
Length-controlled win rate
Gen
×\times
0
✓
×\times
1
Arena-Hard v0.1
Arena-Hard score
Gen
×\times
0
✓
×\times
1
Arena-Hard v2
Arena-Hard score
Gen
×\times
0
✓
×\times
1
Math
GSM8K (CoT)
Exact Match
Gen
✓
8
✓
✓
1
GSM8K (base)
Exact Match
Gen
✓
5
×\times
×\times
1
Hendrycks MATH
Math Verify
Gen
✓
6
✓
✓
1
Minerva MATH
Math Verify
Gen
✓
4
✓
✓
1
MathQA
Accuracy
MCQ (LLH)
×\times
0
✓
×\times
1
GSM8K Platinum (CoT)
Exact Match
Gen
✓
0
✓
×\times
1
GSM8K Platinum (base)
Exact Match
Gen
✓
5
×\times
×\times
1
MATH-500 (Minerva)
Math Verify
Gen
✓
4
✓
✓
1
AIME 2024
Exact Match
Gen
×\times
0
✓
×\times
1
AIME 2025
Exact Match
Gen
×\times
0
✓
×\times
1
Coding
HumanEval (instruct)
Pass@10
Gen
×\times
0
✓
×\times
1
HumanEval (base)
Pass@1
Gen
×\times
0
×\times
×\times
1
MBPP (instruct)
Pass@1
Gen
×\times
3
✓
✓
1
MBPP (base)
Pass@1
Gen
×\times
3
×\times
×\times
1
Table L.2: Benchmark configurations used in the Apertus evaluation suites (continued). Accuracy‡ denotes prompt-level strict accuracy.
Benchmark
Metric
Type
CoT
#Shots
Chat
Turns
#Langs
Multilingual and Cultural
Global-MMLU
Exact Match
MCQ (Gen)
×\times
0
✓
×\times
15
MGSM (English CoT)
Exact Match
Gen
✓
0
✓
×\times
11
TruthfulQA Multilingual (MC2)
Accuracy
MCQ (LLH)
×\times
0
✓
×\times
31
INCLUDE-44 (generative)
Exact Match
MCQ (Gen)
×\times
0
✓
×\times
44
INCLUDE-44 (base)
Accuracy
MCQ (LLH)
×\times
0
×\times
×\times
44
INCLUDE-45
Exact Match
MCQ (Gen)
×\times
0
✓
×\times
45
MLogiQA
Exact Match
MCQ (Gen)
×\times
0
✓
×\times
10
XNLI
Accuracy
MCQ (LLH)
×\times
0
✓
×\times
15
ARC Multilingual
Accuracy
MCQ (LLH)
×\times
0
✓
×\times
31
HellaSwag Multilingual
Accuracy (norm.)
MCQ (LLH)
×\times
0
×\times
×\times
30
LAMBADA Multilingual
Accuracy
LLH
×\times
0
×\times
×\times
5
XCOPA
Accuracy
MCQ (LLH)
×\times
0
×\times
×\times
11
BLEND (sample)
Accuracy (norm.)
MCQ (LLH)
×\times
0
✓
×\times
1
CulturalBench
Accuracy (norm.)
MCQ (LLH)
×\times
0
✓
×\times
1
SwitzerlandQA
Exact Match
MCQ (Gen)
×\times
0
✓
×\times
5
Safety and Alignment
BBQ
Accuracy
MCQ (LLH)
×\times
0
✓
×\times
1
ToxiGen
Accuracy
MCQ (LLH)
×\times
0
✓
×\times
1
WMDP
Accuracy
MCQ (LLH)
×\times
0
✓
×\times
1
OR-Bench
Refusal rate
Gen
×\times
0
✓
×\times
1
HarmBench
Score
Gen
×\times
0
✓
×\times
1
PolygloToxicityPrompts (small)
Score
Gen
×\times
0
✓
×\times
7
RealToxicityPrompts (small)
Score
Gen
×\times
0
✓
×\times
1
HalluLens PreciseWiki
Hallucination rate
Gen
×\times
0
✓
×\times
1
HalluLens LongWiki
Precision
Gen
×\times
0
✓
×\times
1
HalluLens Nonsensical
Abstention rate
Gen
×\times
0
✓
×\times
1
Aya Redteaming
Score
Gen
×\times
0
✓
×\times
8
RealGuardrails S-IFEval
Accuracy‡
Gen
×\times
0
✓
×\times
1
RealGuardrails TensorTrust
Pass rate
Gen
×\times
0
✓
×\times
1
RealGuardrails S-RuLES
Pass rate
Gen
×\times
0
✓
✓
1
Swiss AI Charter Alignment
Length-controlled win rate
Gen
×\times
0
✓
×\times
1
Tool Use
BFCL v3
Accuracy
Gen
×\times
0
✓
✓
1
Table L.3: Full post-training lineage of the Apertus 1.5 models on the development benchmark suite, all 37 benchm