frames due to limited context length. For example, Video-LLaVA (Lin et al., 2023) is restricted to uniformly sampling 8 frames from a video, no matter how long the original video may be. As such, models may lose more fine-grained temporal information that is important for accurately answering any questions about the video. In contrast, our model is trained on long sequences of 1M tokens, and as a result, can simultaneously attend thousands of frames of videos to retrieve fine-grained information over short time intervals. Table 4 shows long video evaluations on the Video-MME (Fu et al., 2024) benchmark, demonstrating our model as the best performing model among its size class. Figure 6 shows an example of our model correctly answering questions about a long, 1-hour YouTube compilation consisting of more than 500 individual clips. Our baseline methods, on the other hand, generally have difficulty answering the questions due to a limited number of frames. More results are shown in Figure 18 and Appendix F.
Table 4: Long Video-MME Benchmark. * denotes models after the completion of this paper.
Method
Parameters
Frames
Medium (4min-15min)
Long (30min-60min)
Gemini 1.5 Pro*
Unknown
≤1800\leq 1800
74.374.3
67.467.4
GPT-4o*
Unknown
384384
70.370.3
65.365.3
LLaVA-Video*
72B
6464
68.968.9
61.561.5
VideoLLaMA 2*
72B
3232
59.959.9
57.657.6
Long-LLaVA*
7B
6464
51.451.4
45.445.4
Video-LLaVA
7B
88
38.138.1
36.236.2
LWM-1M
7B
≤1800\leq 1800
63.763.7
60.860.8
Table 5: Image Understanding Benchmarks (left) and Video Understanding Benchmarks (right)
Method
Visual Token
VQAv2
GQA
SQA
MiniGPT-4
CLIP
-
30.8
25.4
Otter
CLIP
-
38.1
27.2
InstructBLIP
CLIP
-
49.2
60.5
LLaVA-1.5
CLIP
78.5
62.0
66.8
LWM (ours)
VQGAN
55.8
44.8
47.7
Method
MSVD
MSRVTT
TGIF
VideoChat
56.3
45
34.4
LLaMA-Adapte
54.9
43.8
-
Video-LLaMA
51.6
29.6
-
Video-ChatGPT
64.9
49.3
51.4
LWM (ours)
55.9
44.1
40.9
4.3.2 Image Understanding and Short Video Understanding
We evaluate LWM on standard benchmarks for image and short video understanding, with results presented in Table 5. Our model performs comparably to baselines but falls short of state-of-the-art (SOTA) models. This performance gap is not unexpected, given that SOTA models leverage vision backbones that have undergone extensive CLIP training (Radford et al., 2021). In contrast, LWM utilizes discrete tokens from an off-the-shelf model (Patil et al., 2024). Discrete tokens result in greater information loss, particularly for OCR-like textual data, compared to continuous CLIP embeddings. Moreover, our model learns text-image alignment from scratch, while CLIP-based models benefit from large-scale pretraining.
This work primarily focuses on long-context methodology, and we defer additional training to future work due to computational constraints. A straightforward approach to improving benchmark scores would be to incorporate CLIP embeddings as additional input.
Despite not achieving SOTA scores on these short video benchmarks, we believe LWM provides valuable insights for future long-context language and video understanding and generation. The model’s performance could be enhanced through additional training and minor modifications.
We include qualitative image understanding examples in Appendix E and qualitative video understanding examples in Appendix F.
4.3.3 Image and Video Generation
Thanks to a unified any-to-any architecture, our model can not only perform image/video captioning and question-answering but also generate images and videos from text. Figure 7 demonstrates examples of these capabilities. For autoregressive sampling, we employ classifier-free guidance (Ho and Salimans, 2022) on the logits, similar to previous works (Yu et al., 2022, Gafni et al., 2022). In the unconditional branch, we initialize each sequence with <bos><vision>. For additional image and video generation examples, please refer to Appendices H and I, respectively.
Takeaway: LWM excels in long video understanding by processing significantly more 