
Index Terms:
Video Understanding, Vision-Language Models, Multi-modal Learning
I Introduction
Fig. 1: Kangaroo achieves SOTA performance across various comprehensive video understanding benchmarks, surpassing other 7B/8B models.
Fig. 2: Architecture of Kangaroo. The model consists of a vision encoder, a spatial-temporal patchify module, a multi-modal projector and an LLM. Frames extracted from the input video are encoded by the vision encoder and augmented with temporal embedding. Special tokens, together with text tokens are embedded into the embedding space of LLM, and then fused with visual features before being fed to the LLM.
Leveraging the robust context processing and reasoning capability of Large Language Models (LLMs) [1, 2, 3, 4, 5], Large Multi-modal Models (LMMs) [6, 7, 8, 9, 10] have made substantial progress in vision-language tasks. As a critical medium of visual information, video plays an indispensable role in interacting with real-world scenarios. Pioneer studies [11, 12, 13, 14] begin to explore the extension of video inputs in LMMs. Despite demonstrating potential on certain benchmarks, existing methods still struggle with effectively processing long videos. We conduct an in-depth analyze of related research and identify two primary factors contributing to this issue. First, the scarcity of high-quality video datasets limits the efficacy of current approaches, leading to insufficient video-language alignment and suboptimal performance in video-related benchmarks. Second, numerous methods, as exemplified by studies such as [15, 16, 17, 18], incorporate aggressive resampling modules to exceedingly reduce the number of visual tokens. This aggressive compression results in a substantial loss of critical visual information, further exacerbating the problem.
In this paper, we propose Kangaroo, a powerful Video LMM aimed at ameliorating these issues. To overcome the obstacle posed by the limited availability of video data, a high-quality dataset is established for vision-language pre-training and instruction tuning. Utilizing this curated dataset, we devise a curriculum training strategy that progressively endows the LLM basement with the capacity to comprehend long videos. Moreover, recognizing the significance of visual information, we increase the resolution and number of input frames to improve the perception of global context and visual details in long videos. The evaluation results across diverse benchmarks for video understanding verify the superior performance of our model. Specifically, our key contributions in this paper are summarized as follows:
•
Data Curation System: We develop a data curation system encompassing data collection, filtering and caption generation. Initially, we collect a substantial amount of image and video data from various sources. To ensure high-quality video pre-training, low-quality videos are excluded prior to the implementation of the video caption generation procedure. Additionally, we establish a video instruction tuning dataset that spans multiple tasks, with task-specific prompts designed to enhance the model’s instruction-following ability.
•
Curriculum Training: We design a curriculum training strategy that is characterized by progressively increasing complexity and difficulty. The process initiates with image-text pre-training to establish fundamental vision-language correlations, followed by video-text pre-training to incorporate temporal perception. After pre-training, a high-quality subset is curated for refinement using high-resolution inputs across both spatial and temporal dimensions. Subsequently, we increase the number of input frames and conduct comprehensive fine-tuning across multiple instruction-following tasks. Finally, an additional phase is introduced, utilizing extensive input frames specifically optimized for longer videos.
•
Long Context Input:
To accommodate longer videos during instruction tuning, the maximum number of input frames is incrementally 