we employ classifier-free guidance (Ho and Salimans, 2022) on the logits, similar to previous works (Yu et al., 2022, Gafni et al., 2022). In the unconditional branch, we initialize each sequence with <bos><vision>. For additional image and video generation examples, please refer to Appendices H and I, respectively.
Takeaway: LWM excels in long video understanding by processing significantly more frames than previous state-of-the-arts, resulting in better understanding.
Moreover, its long-context enabled unified any-to-any architecture allows for versatile image and video and text understanding and generation.
Figure 7: LWM’s ability to generate both static images and dynamic videos from text is shown. The top row illustrates image, while the bottom rows show video.
5 Related Works
Our research builds upon existing efforts to extend the context windows of language models, enabling them to process more tokens (Chen et al., 2023b, Tworkowski et al., 2023, Liu et al., 2023c). These approaches often employ innovative extrapolation techniques to expand pretrained positional encodings, followed by model finetuning on longer context data. In contrast, our model takes a straightforward approach by incrementally increasing θ\theta in RoPE positional encodings alongside expanding the training context window sizes, which we found to be effective. Additionally, there have been investigations into architectures that avoid modeling pairwise interactions, such as sparse attention and sliding window techniques (Child et al., 2019, Beltagy et al., 2020). Prior research has explored sequence parallelization (Li et al., 2021, Korthikanti et al., 2022, inter alia), though it is not optimized for blockwise transformers or compatible with memory-efficient attention, both of which are critical for large context training. Our work further leverages large context transformer techniques (Liu et al., 2024, Liu and Abbeel, 2023) to capture exact pairwise interactions in extended sequences for enhanced performance. Load-balancing strategies, such as skipping causal masked computation (Brandon et al., 2023, Li et al., 2023) offer room for further optimization. Concurrent developments like Gemini 1.5 (Reid et al., 2024) reach 1M tokens context size in language and video.
Additionally, our approach relates closely to advances in instruction tuning (Taori et al., 2023, Chiang et al., 2023, Geng et al., 2023), which focus on finetuning models with conversational data to boost their performance across diverse language tasks. We aim to extend these capabilities to the domain of long-sequence understanding in both video and language tasks. To achieve this, we extend the model’s context size by training on comprehensive datasets, including books and long videos, and finetune on model-generated question-answering datasets to enhance its ability to handle extended conversational sequences.
Furthermore, our research draws from work on integrating vision capabilities into language models (Liu et al., 2023b, Lin et al., 2023, Awadalla et al., 2023, Zhang et al., 2023, Jin et al., 2023b, Aiello et al., 2023). These efforts frequently utilize continuous embeddings (Radford et al., 2021, Li et al., 2022) to encode visual information into embeddings for inputting into language models. While these approaches benefit from CLIP’s cross-modal understanding to encode textual information from images, their ability to predict text from visual input is limited, as is their capacity to learn from diverse visual-language formats.
In contrast, our autoregressive model, which processes "tokens in, tokens out," allows greater flexibility in modeling various formats, including image-text, text-image, text-video, video-text, and pure formats like video, image, or text. Our method is compatible with these prior works, making it an interesting future direction to combine continuous embeddings as input with discrete tokens and a long-context autoregressive model.
6 Conclusion
In conclusion,