From Open Text Models to Open Multimodal Reasoners Apertus v1.5 Technical Report
Report GitHub Issue
×
Title:
Content selection saved. Describe the issue below:
Description:
Submit without GitHub
Submit in GitHub
arXiv is now an independent nonprofit!
Learn more
×
Back to arXiv
Why HTML?
Report Issue
Back to Abstract
Download PDF
Abstract
1 Introduction
1.1 Released Models, Code, and Datasets
Models.
Code.
Datasets.
2 Multimodality
2.1 Unified Representation
Vocabulary Extension.
2.1.1 Images
2.1.2 Audio
2.2 Multimodal Training Data
2.2.1 Curation and Processing
Acquisition and Normalization.
Deduplication.
Caption Generation and Validation.
Processing at Scale.
2.2.2 Vision Training Data
2.2.3 Vision Data Composition
2.2.4 Audio Training Data
Interleaved Speech and Text.
Speech SFT Data.
Long-Context SFT Data.
2.3 Tokenization and Training Pipelines
2.3.1 Image Tokenization
2.3.2 Audio Tokenization
2.4 Ablations
Image Tokenizer.
Two-Stage Recipe.
3 Continued Pretraining Recipe
3.1 Pretraining Data
3.1.1 Compliance and Provenance
3.1.2 Text Data Selection
3.2 Optimizer and Training Recipe
3.3 Long-Context Extension
3.4 Pretraining Run Retrospective
4 Post-Training
4.1 Chat Template
4.2 Supervised Finetuning
Capability Filtering.
Data.
Tool-Use Data.
Multilingual and Safety Data.
Decontamination.
4.2.1 Image
Categorical Answer Balance.
4.2.2 Audio
4.2.3 Variety-Aware Translation Case Study
4.3 Long-Context Supervised Finetuning
Data.
4.4 Improving Instruction Following
Verifiable IF.
Improving Non-verifiable Instruction Following.
Reinforcement Learning.
Preference Alignment.
4.5 Reinforcement Learning
Data.
4.6 Alignment
4.6.1 Direct Preference Optimization
Preference Data.
Offline DPO.
Online DPO.
Results.
4.6.2 Charter Alignment with SDPO
Data.
Feedback.
Training.
4.7 SFT Ablations
Q1–Q3.
Takeaway.
Q4.
Takeaway.
5 Evaluations
5.1 Evaluation Setup
Known Pitfalls.
5.2 Base Models
5.3 Post-Trained Models
5.3.1 Reasoning and Thinking Mode
5.3.2 Post-Training Lineage
5.4 Multilingual and Cultural Evaluation
5.4.1 Machine Translation
Automatic Evaluation.
Human Evaluation.
Automatic Evaluation with SwissGov.
Last Translation Benchmark.
5.5 Image Understanding
5.6 Audio Understanding
5.7 Long Context
5.8 Tool Use
5.9 Security, Safety, and Interpretability
5.9.1 Safety Benchmark Performance
5.9.2 Over-Refusal and Hallucination
5.9.3 Interpretability
Exploding Activations.
Degeneration Probes.
Multilingual Geometry.
5.10 Qualitative Spot-Testing
6 Infrastructure, Scaling, and Efficiency
6.1 Infrastructure
6.2 Full Training Run Performance
6.3 Energy Consumption
6.4 Engineering Challenges and Solutions
Software Environment and Node Validation.
Communication at Scale.
Storage Capacity and Data Placement.
Network Safeguards for Data Ingestion.
Reinforcement Learning Infrastructure.
7 Responsible Release and Governance
7.1 License and Acceptable Use
7.2 Limitations
7.3 Broader Impact
7.4 Community and External Evaluation
8 Conclusion
References
A Contributions
B Vision Datasets by Training Stage
C Audio Datasets by Training Stage
D Image Dataset Exploration Detailed Breakdown
E Tokenizer Reconstruction by Category
F Long-Context Data Mixture Tables
G Training Configurations
H Supervised Finetuning Details
H.1 Per-Benchmark SFT Ablation Results
H.2 SFT Data Decontamination Details
I Preference Alignment Data
I.1 Judge Rubric Prompts
I.2 Prompt Set Composition
I.3 Response Model Pool
I.4 Completion Generation
I.5 Score Distribution
J Alignment Ablations
J.1 DPO Ablations
J.1.1 Apertus 1.5 8B
Offline DPO, standard loss.
Offline DPO, length-normalized loss.
Offline DPO, transfer to Apertus 1.5 8B.
Online DPO.
Combining offline and online DPO.
J.1.2 Apertus 1.5 70B
Transferring the Apertus 1.5 8B hyperparameters.
Offline and mixture.
J.2 Judge Ablations
J.2.1 Judging Aspects Simplification
J.2.2 Choice of Judge Model
K Chatbot System Prompt
L Evaluation Details
L.1 Full Post-Training Lineage
M Pre-Release Manual Evaluations
M.1 General Inform