ding
The image suite spans around 70 unique benchmarks, from general visual question answering and OCR to spatial reasoning, remote sensing, and medical imagery; Table 34 reports a representative subset for the released checkpoints against openly released vision-language models, evaluated with our forks of LMMs-Eval (Zhang et al., 2025) and VLMEvalKit (Duan et al., 2024). Decoding is greedy with a 16,384-token budget; thinking samples at temperature 0.6 and top-p 0.95 with a 32,768-token budget. Spatial-intelligence benchmarks follow the EASI protocol (Cai et al., 2025). Within the fully open group, Apertus 1.5 8B leads on document and chart understanding (ChartQA, DocVQA) and general VQA (GQA), while Molmo2-8B leads general perception. The strongest open-weight models, Qwen3-VL and Qwen2.5-VL, stay ahead overall, most visibly on OCR-heavy tasks. Thinking lifts reasoning-heavy benchmarks by roughly 12 points (MMMU 39.8 to 52.2) but costs some extraction-style tasks about as much (ChartQA 78.7 to 66.0). Apertus 1.5 70B adds little over the 8B on most image benchmarks.
Table 34: Image understanding of the released checkpoints against open vision-language models, on the 33 image benchmarks of the released model card plus MM-Safety. 8Bth is the 8B model with thinking enabled. All values in percent, higher is better, best per group in bold. Dashes mark cells with no valid run. *For MM-Safety we report the safety rate instead of the attack rate. †EuroVLM answers AI2D in prose rather than the requested option letter in 74.6% of responses, whereas every other model complies in essentially all of them; the harness credits only bare-letter replies, so this cell largely measures instruction following. A prose-tolerant extractor scores the same generations at 64.3.
Fully open
Open-weight
Apertus 1.5
Molmo2
EuroVLM
Qwen3-
VL
Qwen2.5-
VL
Gemma
4
Pixtral
Emu3
Benchmark
8B
8Bth
70B
8B
9B
8B
7B
12B
12B
8B
General VQA and Perception
GQA
59.2
52.8
58.4
56.6
47.9
61.6
60.9
38.0
58.8
60.3
MMStar
45.2
63.1
47.0
64.5
45.5
63.9
62.4
67.9
55.8
44.4
RealWorldQA
60.0
63.5
64.3
71.1
49.9
68.8
68.9
64.7
63.4
54.0
VQAv2
76.9
71.5
75.3
77.7
71.7
81.8
83.0
56.5
78.8
73.7
V*Bench
63.4
68.6
60.2
72.8
56.5
84.8
80.1
51.3
52.9
52.4
CountBench
84.9
86.0
90.8
88.4
88.6
91.5
85.7
91.5
3.7
65.4
Documents and OCR
ChartQA
78.7
66.0
72.8
73.1
71.2
85.2
86.0
51.0
83.8
69.1
DocVQA
84.8
86.2
82.6
69.1
71.3
95.7
95.1
80.3
88.9
72.8
InfoVQA
53.0
56.4
55.5
53.6
43.4
83.6
82.3
52.0
52.1
36.6
SEED-Bench-
2-Plus
55.4
60.4
56.6
70.1
50.9
71.2
71.1
68.6
68.3
44.8
OmniDocBench
33.5
35.9
36.4
11.6
10.7
54.7
37.2
22.8
26.8
30.7
Visual Math and Logic
MathVista
61.0
66.7
61.4
63.9
46.5
79.6
67.4
75.6
58.0
45.6
MathVision
25.0
27.4
23.9
26.4
14.6
41.6
25.6
57.1
17.5
12.8
BabyVision
10.6
5.2
11.6
10.6
9.5
6.4
12.1
18.6
13.1
8.0
STEM and Knowledge
AI2D
74.3
76.8
74.1
84.5
15.8†
83.7
82.6
75.1
78.0
70.1
MMMU
39.8
52.2
44.2
52.6
36.8
53.9
51.9
63.7
50.0
29.8
MMMU-Pro
23.5
39.0
26.9
37.5
22.9
39.9
37.5
46.8
33.5
15.3
ScienceQA
84.7
90.6
85.1
94.2
52.5
92.9
88.8
86.5
88.8
72.9
Spatial Intelligence
EmbSpatial
66.9
75.2
65.4
68.8
20.7
78.0
71.5
70.8
28.4
53.5
MMSI
32.5
26.4
28.9
29.3
25.2
30.5
27.3
29.4
24.2
21.8
ViewSpatial
43.6
43.2
41.2
41.5
15.4
41.9
37.3
38.3
16.1
21.5
MindCube
43.9
36.1
42.5
41.1
37.9
30.0
33.6
48.6
29.9
32.1
Remote Sensing
FRIEDA
16.1
15.1
15.7
12.0
15.1
19.6
15.3
5.4
10.4
8.3
GeoBench
28.3
32.2
31.1
46.1
30.8
42.1
41.7
41.3
34.2
28.6
VRSBench-VQA
51.0
47.9
53.1
45.1
50.3
51.9
50.2
37.4
47.5
43.7
Medical Imaging
PathVQA
68.8
70.7
62.4
63.4
61.6
71.0
65.8
69.5
64.8
55.0
PMC-VQA
52.7
51.3
55.3
48.5
45.0
54.8
51.3
58.1
49.4
45.5
SLAKE
77.4
79.6
70.9
63.7
71.4
77.9
71.4
80.0
69.7
52.2
VQA-RAD
68.1
66.5
68.1
65.3
67.3
75.7
74.5
72.5
71.3
78.5
Robustness and Bias
MMVP
69.0
76.0
70.3
86.3
73.7
79.7
77.0
83.0
68.0
70.3
VLMs-Are-Biased
21.5
20.9
18.8
25.2
18.9
13.9
17.1
3.9
12.1
10.5
VLMsAreBlind
68.9
54.5
56.8
58.9
47.1
59.8
52.0
66.0
47.2
15.5
Hallucination and Safety
POPE
86.