67.3
69.8
61.4
71.2
66.3
74.2
77.8
74.3
68.0
Table 31: Automatic evaluation of the SwissGov translation tasks with COMETKiwi22.
5.4.1 Machine Translation
Being able to translate well is a major sign of a model’s multilingual capability, and machine translation remains a challenging NLP task (Zouhar et al., 2026a). To this end, we evaluate several comparable models across many languages on WMT24++ (Kocmi et al., 2024; Deutsch et al., 2025) and its Romansh extension (Vamvas et al., 2025), WMT25 (Kocmi et al., 2025), and SwissGov (Wastl et al., 2026). This evaluation pays particular attention to, but is not limited to, German, Swiss German, French, Italian, and Romansh.
Table 32: Automatic evaluation on the news domain of the WMT24, WMT24++, and WMT25 translation tasks with the ChrF metric.
Table 33: Human evaluation results for the translation tasks on a 0% to 100% cESA scale.
Automatic Evaluation.
The results of automatic evaluation in Table 32 and Table 31, based on the ChrF (Popović, 2015) and COMETKiwi22 (Rei et al., 2022) metrics respectively, show that the average performance of Apertus models correlates across languages with other models, indicating that they do not tend to have an inherent preference for a particular language. An exception is translation from German into Romansh varieties, where the targeted post-training (Section 4.2.3) enables Apertus 1.5 8B to surpass comparable models by a large margin.
Human Evaluation.
Automatic evaluation can sometimes be misaligned with human perception of quality (Lavie et al., 2025; Kocmi et al., 2025). To this end, we also run human evaluation of translation quality, where bilingual annotators (recruited from authors and their professional network) are blindly presented with multiple translations and rate them from 0% to 100%. We carry out the annotations using the cESA annotation protocol (Zouhar et al., 2026b; Zouhar & Kocmi, 2026) with four model outputs shown next to each other: Apertus 1.0 8B, Apertus 1.5 8B, EuroLLM 22B, and Ministral 3 8B. In Table 33 the human evaluation confirms that Apertus 1.5 8B avoids critical failure on Switzerland-specific language pairs. The data collected by human annotators are available publicly.88
8
swiss-ai/apertus-translate
Automatic Evaluation with SwissGov.
SwissGov is a multiparallel document-level dataset of Swiss government texts in three official national languages and English, providing a domain-specific evaluation that approximates a realistic use case for Apertus. For language pairs also covered by WMT, evaluation with SwissGov yields the same relative ranking, suggesting that the observed performance differences generalize beyond the news domain.
Last Translation Benchmark.
We also evaluate Apertus models on the hard translation examples (Zouhar et al., 2026a) over more than 100 languages and 3000+ texts.
Apertus v1.5 8B shows promising performance, achieving 1.32% on LTBv1-eval (911 samples), matching the GPT-OSS-20B score and outperforming models like Claude Haiku 4.5 or Google Translate. In comparison, the Apertus v1 8B model (Apertus-8B-Instruct-2509) and the 70B model (Apertus-70B-Instruct-2509) achieve 0.77% and 1.21%, respectively, while the v1.5 70B model shows a worse result with 0.66%. We believe that our bigger model is better adapted to English-centric, high-performance tasks, while the 8B model is a fitting candidate for translation. Moreover, the results give a strong signal toward more focus on multilingual and multicultural post-training, which we plan to target in future versions with greater effort.
5.5 Image Understanding
The image suite spans around 70 unique benchmarks, from general visual question answering and OCR to spatial reasoning, remote sensing, and medical imagery; Table 34 reports a representative subset for the released checkpoints against openly released vision-language models, evaluated with our forks of LMMs-Eval (Zhang et al., 2025) and VLMEvalKit (Duan et al., 2024). Decoding is greedy with a