ntation taxonomy. In turn, this could lead to a filtering scheme that balances the vision data in appropriate ways.
Taxonomy. We therefore design a taxonomy consisting of 61 fine-grained labels organized along two orthogonal dimensions: graphics and photographs. We also organize these fine-grained labels into five coarse-grained labels for graphics and seven coarse-grained labels for photographs for easier analysis. We plot this taxonomy in Figure D.1 and Figure D.2. Using CLIP ViT-L/14, we classify approximately 1% of the full datasets in a single forward pass, enabling efficient large-scale analysis while preserving the overall dataset statistics. The taxonomy uses a flat one-pass design: all 61 labels are scored in a single CLIP forward pass (image ↔\leftrightarrow text prompt), so classification adds no extra compute beyond the embedding computation. We focus on LLaVA-OneVision, BLIP3, DataComp-1B, MINT-1T-ArXiv, MINT-1T-HTML, and MINT-1T-PDF. These are some of the largest datasets by token count (LLaVA-OneVision, BLIP3, DataComp-1B) that consist of large image piles sourced from the general internet. We also explore a selection of more specialized datasets (MINT-1T-ArXiv, MINT-1T-HTML, and MINT-1T-PDF). These MINT-1T subsets still form a large part of the overall token count. At the same time, the image source is constrained (PDF documents for both the arXiv and PDF subparts), and the domain for MINT-1T-ArXiv is further constrained to scientific works. We separate our analysis into quantitative and qualitative steps.
Figure D.1: Graphic axis of the taxonomy, subdivided into five coarse groups and then into 26 fine-grained groups.
Figure D.2: Photograph axis of the taxonomy, subdivided into seven coarse groups and then into 35 fine-grained groups.
Quantitative breakdown. The first step provides a quantitative exploration of the datasets by applying our taxonomy-based image classification pipeline. The resulting distributions over the taxonomy classes reveal the diversity of visual content within and across datasets while demonstrating that the proposed taxonomy effectively explores both graphics and photographs.
We plot the results of our analysis in Figure D.3, Figure D.5, Figure D.5, Figure D.6, and Figure D.7. This analysis yields several interesting conclusions that can also inspire future work. First, in our coarse-grained analysis, we observe that “Composite Graphic” is the most frequent graphic. At the same time, “Artifacts/Objects” appears as the most frequent photograph. This seems intuitive, given that these datasets provide great internet coverage; logos and related material should dominate due to commercial actors and advertisements. At the same time, in our fine-grained analysis, we observe that in most datasets, the “Artifacts/Objects” class consists of “Clothing”, “Food”, “Vehicle”, “Electronic Device”, and “Home Goods”, which figure prominently across all datasets. In contrast, specialized domains such as insects, astronomy, and historic sites make up a relatively small percentage of all images, while potentially appearing prominently in user queries to the model. We quantify this imbalance in Table D.1. These imbalances raise the prospect of rebalancing the dataset to make the image distribution more equal across different domains. Finally, we observe that the image distribution remains relatively similar across different datasets except the MINT-1T dataset. Here, especially for the arXiv subset, documents and research graphs dominate the distribution in accordance with our data source. Similarly, in the MINT-1T-PDF subset, we observe diagrams and charts dominating. This aligns with how these datasets are sourced. It also provides quantitative motivation for including diversified datasets in our data mixture to address limitations in larger but more general image pools.
Figure D.3: Breakdown of the main image datasets into Graphic and Photograph categories. We observe that most datasets are roughly evenly 