n is at or below chance (16.0% and 5.0% against 16.7% for six classes).
Table 35: Audio evaluation of the released Apertus 8B and 70B checkpoints against open audio models. Word error rates (WER) are in percent, lower is better; accuracies are in percent and BLEU on a 0 to 100 scale, both higher is better. Best per row in bold. All models decode greedily, Qwen2-Audio with the repetition penalty (Keskar et al., 2019) of 1.1 from its generation configuration, which its authors’ evaluation code (QwenLM/Qwen2-Audio) also applies, and use each task’s output cap; where a task sets none, all models share a cap of 4,096 tokens, which TED-LIUM long form also receives because its own cap of 256 tokens would cut off transcripts of up to 4,164 words; on that task, Qwen2.5-Omni uses PyTorch SDPA attention, whose fused kernels do not store the full attention matrix (Dao et al., 2022; Rabe & Staats, 2021), because with the default eager attention its longest talks exceed GPU memory.
Benchmark
Apertus 1.5
8B
Apertus 1.5
70B
Qwen2.5-
Omni
Qwen2-
Audio
Kimi-
Audio
Parameters
8B
70B
7B
7B
7B
English Speech Recognition (WER, lower is better)
LibriSpeech test-clean
6.6
6.2
2.3
4.2
1.3
LibriSpeech test-other
19.9
19.3
4.4
8.2
2.5
VoxPopuli
11.2
10.3
11.9
14.0
8.2
SPGISpeech
18.4
12.6
5.5
6.0
3.5
TED-LIUM long form
36.0
47.3
67.7
99.6
61.8
Multilingual Speech Recognition (FLEURS, WER, lower is better)
English
15.2
14.1
7.4
8.4
4.4
German
16.2
17.8
8.3
16.8
23.7
French
22.2
21.0
10.6
13.8
17.7
Italian
7.6
7.3
6.8
12.1
13.1
Spanish
8.7
7.9
8.0
11.5
14.9
Polish
43.9
44.7
87.1
98.6
65.3
Ukrainian
34.9
31.4
57.6
99.3
108.2
Speech Translation (BLEU, higher is better)
CoVoST2 English-Chinese
15.3
13.1
44.3
38.1
0.3
Audio Understanding and Classification (accuracy, higher is better)
MMAU
55.1
55.9
67.8
57.7
66.4
MuChoMusic
51.2
57.3
62.5
67.6
68.7
Clotho-AQA
53.0
57.2
87.0
75.7
83.9
VocalSound
16.0
5.0
92.2
93.7
65.0
5.7 Long Context
We evaluate long-context ability with two benchmarks: RULER (Hsieh et al., 2024) measures synthetic retrieval and aggregation, while HELMET (Yen et al., 2025) spans seven categories of downstream tasks.99
9
RULER evaluation is run with swiss-ai/evals-post-train; HELMET with swiss-ai/HELMET. Table 36 reports both at every stage of the Apertus 1.5 8B and 70B pipelines. The context extension is the decisive step: on RULER it puts Apertus 1.5 ahead of Apertus 1.0 at every length and at both sizes, and it carries the model out to 256k tokens. Short-context SFT then undoes much of that, collapsing retrieval on long inputs, and the dedicated long-context SFT stage described in Section 4.3 restores it. RL and preference optimization add further gains on RULER even though neither stage trains on long documents, a point we return to in Section 3.4. HELMET, in the right-hand columns, shows the same collapse and the same recovery, with the preference-optimization stages strongest at every length. Through 128k the 70B pipeline stays ahead of the 8B on both benchmarks.
Table 36: Long-context evaluation on RULER and HELMET, in percent, higher is better. RULER cells average 13 subtasks; HELMET cells average seven task-group scores, HELMET’s own aggregation. Three HELMET settings use ROUGE in place of the reference LLM judge. Base and post-trained models are shown separately, fully open models above open-weight ones; bold marks the best value per column within a panel. “–” marks a length beyond the model’s position ceiling.
RULER
HELMET
Model
8k
16k
32k
64k
128k
256k
8k
16k
32k
64k
128k
Base models
Fully open
Apertus 1.0 8B
82.0
75.7
71.2
55.5
–
–
49.2
47.2
44.5
37.3
–
Apertus 1.0 70B
79.8
78.1
71.6
57.3
–
–
48.5
47.2
44.4
37.7
–
Apertus 1.5 8B
85.9
81.3
76.1
67.5
61.5
54.8
45.1
42.1
40.7
37.6
36.5
Apertus 1.5 70B
86.8
85.5
78.8
71.9
61.8
51.0
49.6
49.0
44.2
41.1
37.8
OLMo 3 7B
85.5
77.8
70.9
62.4
–
–
48.7
47.0
44.6
39.7
–
OLMo 3 32B
92.7
89.4
83.9
76.3
–
–
52.7
50.8
48.8
46.2
–
Open-weight
Gemma-3-12B
91.3
90.0
89.4
85.1
78.8
–
50.0
49.3
47.5
