52.7
51.3
55.3
48.5
45.0
54.8
51.3
58.1
49.4
45.5
SLAKE
77.4
79.6
70.9
63.7
71.4
77.9
71.4
80.0
69.7
52.2
VQA-RAD
68.1
66.5
68.1
65.3
67.3
75.7
74.5
72.5
71.3
78.5
Robustness and Bias
MMVP
69.0
76.0
70.3
86.3
73.7
79.7
77.0
83.0
68.0
70.3
VLMs-Are-Biased
21.5
20.9
18.8
25.2
18.9
13.9
17.1
3.9
12.1
10.5
VLMsAreBlind
68.9
54.5
56.8
58.9
47.1
59.8
52.0
66.0
47.2
15.5
Hallucination and Safety
POPE
86.1
86.0
86.8
88.7
88.7
88.8
87.8
81.8
85.5
81.9
MM-Safety*
62.0
65.8
71.5
74.9
47.9
78.0
57.6
82.0
42.1
33.9
5.6 Audio Understanding
The audio suite centers on multilingual speech recognition, the primary target of the audio training, complemented by speech translation and by audio understanding beyond transcription, from audio and music question answering to vocal-sound classification. Table 35 reports a representative subset against openly released audio models, namely Qwen2.5-Omni (Xu et al., 2025), Qwen2-Audio (Chu et al., 2024), and Kimi-Audio (Kimi Team et al., 2025), all evaluated with our fork of LMMs-Eval (Zhang et al., 2025). It covers speech recognition on LibriSpeech (Panayotov et al., 2015), VoxPopuli (Wang et al., 2021a), SPGISpeech (O’Neill et al., 2021), TED-LIUM (Hernandez et al., 2018), and FLEURS (Conneau et al., 2023); speech translation on CoVoST2 (Wang et al., 2021b); and audio understanding on MMAU (Sakshi et al., 2025), MuChoMusic (Weck et al., 2024), Clotho-AQA (Lipping et al., 2022), and VocalSound (Gong et al., 2022).
We evaluate every model ourselves, on the same tasks with the same scoring and each task’s instruction as the prompt, so that the values are comparable. Qwen2-Audio keeps the default of its chat template, and Kimi-Audio is prompted as in its authors’ own evaluation (MoonshotAI/Kimi-Audio-Evalkit), with the instruction in the user turn and no system prompt, which its inference code (MoonshotAI/Kimi-Audio) does not accept. Qwen2.5-Omni receives the task-category prompts of its authors’ examples (QwenLM/Qwen2.5-Omni) for speech recognition, speech translation, and vocal-sound classification; for audio question answering, for which its authors publish no evaluation prompt, it receives a short audio-understanding prompt, because with its default prompt it appends conversation to one-word answers. We report MMAU on its test-mini split, the split with public answers (Sakshi et al., 2025). One CoVoST2 clip cannot be decoded in the pipeline of the three peers; we score their empty answers as wrong, so that every model is scored on the same samples.
On English speech recognition of short utterances, the dedicated audio models lead, by wide margins on LibriSpeech and SPGISpeech, while Apertus 1.5 70B comes within 2.1 points of the best VoxPopuli error rate and ahead of Qwen2.5-Omni and Qwen2-Audio. On FLEURS, Apertus 1.5 70B holds the best word error rates on Spanish and Ukrainian, and Apertus 1.5 8B the best on Polish, where Qwen2.5-Omni reaches 87.1%, Qwen2-Audio 98.6%, and Kimi-Audio 65.3%; the peers lead English, German, French, and Italian. On TED-LIUM long form, whose eight talks run 3 to 22 minutes, Apertus 1.5 8B reaches 36.0% and Apertus 1.5 70B 47.3%, against 61.8% for Kimi-Audio, 67.7% for Qwen2.5-Omni, and 99.6% for Qwen2-Audio, which processes audio in 30-second windows and returns at most 26 words per talk. On CoVoST2 English-Chinese, the Qwen models lead with 44.3 BLEU for Qwen2.5-Omni and 38.1 for Qwen2-Audio, against 15.3 for Apertus 1.5 8B, 13.1 for Apertus 1.5 70B, and 0.3 for Kimi-Audio. On audio understanding beyond speech, Apertus trails the peers throughout, and its vocal-sound classification is at or below chance (16.0% and 5.0% against 16.7% for six classes).
Table 35: Audio evaluation of the released Apertus 8B and 70B checkpoints against open audio models. Word error rates (WER) are in percent, lower is better; accuracies are in percent and BLEU on a 0 to 100 scale, both higher is better. Best per row in bold. All models decode greedily, Qwen2-Audio with the repetition penalty (K