World Model on Million-Length Video And Language With Blockwise RingAttention
Skip to main content
Report GitHub Issue
×
Title:
Content selection saved. Describe the issue below:
Description:
Submit without GitHub
Submit in GitHub
Why HTML?
Report Issue
Back to Abstract
Download PDF
Contents
Reading mode
Colour theme
Abstract
1 Introduction
2 Method overview
3 Stage I: Learning Long-Context Language Models
3.1 Progressive Training Towards Long Context
3.2 Model-Generated Question-Answering For Effective Context
3.3 Language Evaluation Results
3.3.1 Short Context Tasks
3.3.2 Retrieval Task: Single Information
3.3.3 Retrieval Task: Multiple Information
3.3.4 Evaluation on LOFT
4 Stage II: Extending to Long-Context Vision-Language
4.1 Architectural Modifications For Vision
4.2 Training Steps
4.3 Vision-Language Evaluation Results
4.3.1 Long Video Understanding
4.3.2 Image Understanding and Short Video Understanding
4.3.3 Image and Video Generation
5 Related Works
6 Conclusion
References
A Further Details
B Ablation Studies
B.1 Masked Sequence Packing
B.2 Mixing Synthetic And Chat Data
C More Single-Needle Retrieval Results
D Details Of The Multi-needle Evaluations
E More Image Understanding Examples
F More Video Understanding Examples
G Details Of Qualitative Video Understanding Evaluation
H More Image Generation Examples
I More Video Generation Examples
J Training Hyperparameters
License: arXiv.org perpetual non-exclusive license
arXiv:2402.08268v4 [cs.LG] 03 Feb 2025
World Model on Million-Length Video And
Language With Blockwise RingAttention
Hao Liu    Wilson Yan11footnotemark:
1
Matei Zaharia    Pieter Abbeel
††thanks: Equal contribution. Email: hao.liu@cs.berkeley.edu, wilson1.yan@berkeley.edu
Code and models of Large World Model (LWM) are available at largeworldmodel.github.io/lwm/.
Affiliation: UC Berkeley
Abstract
Enabling long-context understanding remains a key challenge in scaling existing sequence models – a crucial component in developing generally intelligent models that can process and operate over long temporal horizons that potentially consist of millions of tokens. In this paper, we aim to address these challenges by providing a comprehensive exploration of the full development process for producing 1M context language models and video-language models, setting new benchmarks in language retrieval and new capabilities in long video understanding. We detail our long context data curation process, progressive context extension from 4K to 1M tokens, and present an efficient open-source implementation for scalable training on long sequences. Additionally, we open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens.
1 Introduction
Figure 1: Comparison of context size in state-of-the-art LLMs. Our model and concurrent work Gemini 1.5 both achieve a 1M context size, significantly outperforming other LLMs.
Figure 2: Retrieval comparisons against Gemini Pro and GPT-4. Needle retrieval comparisons against Gemini Pro and GPT-4 for each respective max context length – 32K and 128K. Our model performs competitively while being able to extend to 8x longer context length. Note that in order to show fine-grained results, the x-axis is log-scale from 0-128K, and linear-scale from 128K-1M.
Enabling long-context understanding remains a key challenge in scaling existing sequence models—a crucial step toward developing generally intelligent models that can process and operate over extended temporal horizons, potentially involving millions of tokens. Current modeling approaches are predominantly limited to processing short sequences, whether in the form of language, images, or video clips (Brown et al., 2020, Touvron et al., 2023a, Touvron et al., 2023b, OpenAI, 2023, Brooks et al., 2024, Team et al., 2023). As a result, these models fall short when tasked with understanding complex, long-form language and visual contexts.
However, training models to process sequences that e