tage, we train on the following data:
•
LWM-1K: We train on large set of text-image dataset comprising of a mix of LAION-2B-en (Schuhmann et al., 2022) and COYO-700M (Byeon et al., 2022). The datasets were filtered to only include images with at least 256 resolution – in total roughly 1B text-image pairs. During training, we concatenate the text-image pairs and randomly swap the order of the modalities to model both text-image generation, unconditional image generation, and image captioning. We pack text-image pairs to sequences of 1K tokens.
•
LWM-8K: We train on a text-video dataset mix of WebVid10M (Bain et al., 2021) and 3M InternVid10M (Wang et al., 2023) examples. Similar to prior works (Ho et al., 2022a, Ho et al., 2022b, Villegas et al., 2022), we jointly train on both images and video with a 50-50 ratio of each modality. We pack images to sequences of 8K tokens, and 30 frame videos at 4FPS. Similar to image training, we randomly swap the order of modalities for each text-video pair.
•
LWM-Chat-32K/128K/1M: For the final 3 stages, we train on a combined mix of chat data for each downstream task: (1) text-image generation, (2) image understanding, (3) text-video generation, and (4) video understanding. We construct a simple version of text-image and text-video chat data by sampling random subsets of the pretraining data augmented with chat format. For image understanding, we using the image chat instruct data from ShareGPT4V (Chen et al., 2023a). Lastly, for the video understanding chat data, we use a combined mix of Valley-Instruct-73K (Luo et al., 2023) and Video-ChatGPT-100K instruct data (Maaz et al., 2023). For all short context data (image generation, image understanding, video generation), we pack sequences to the training context length. During packing, we found it crucial to mask out the attention so that each text-vision pair only attends to itself, as well as re-weighting losses to make computation identical to training in a non-packed + padding training regime. For video understanding data, we uniformly sample a max number of frames to fit the training context length of the model if the video is too long. During training, We allocate 25% of each batch to each of the 4 downstream tasks.
Figure 6: LWM excels in answering questions about a 1-hour YouTube video. This figure compares LWM-Chat-1M with proprietary models like Gemini Pro Vision and GPT-4V, along with open-source models. The test involves answering questions based on an hour-long YouTube compilation containing over 500 video clips. LWM demonstrates superior performance in providing accurate answers requiring comprehension of extended video content.
For the first two stages of training (LWM-1K and LWM-8K), we additionally mix 16% of the batch to be pure text data from OpenLLaMA (Geng and Liu, 2023), as we found it beneficial to preserve language capabilities while training on vision data.
Summary: Stage II training incorporates image and video. Building on Stage I, it gradually increases sequence lengths of vision and text input. Importantly, we found our masked sequence packing and mixing synthetic and chat data crucial to retain short context performance during our progressive training. Appendix B shows ablations when not using our training method on instruction-following and text-image understanding benchmarks.
4.3 Vision-Language Evaluation Results
4.3.1 Long Video Understanding
Although vision-language model (Lin et al., 2023, OpenAI, 2023, Team et al., 2023) can ingest long videos, this is commonly done by performing large temporal subsampling of video frames due to limited context length. For example, Video-LLaVA (Lin et al., 2023) is restricted to uniformly sampling 8 frames from a video, no matter how long the original video may be. As such, models may lose more fine-grained temporal information that is important for accurately answering any questions about the video. In contrast, our model is trained on long sequences of 1M tokens, and as a 