al dimensions. Subsequently, we increase the number of input frames and conduct comprehensive fine-tuning across multiple instruction-following tasks. Finally, an additional phase is introduced, utilizing extensive input frames specifically optimized for longer videos.
•
Long Context Input:
To accommodate longer videos during instruction tuning, the maximum number of input frames is incrementally increased to 64 and 160, corresponding to context lengths of 10K and 22K tokens, respectively. With higher resolution and extensive input frames, our model is expected to effectively process extremely long videos.
•
Outstanding Performance: We evaluate on various video understanding benchmarks. The results indicate that, with 8B parameters, Kangaroo achieves state-of-the-art performance on most benchmarks and reaches a competitive level on others. Notably, on benchmarks customized for long videos, Kangaroo outperforms certain larger models with over 10B parameters and proprietary models.
II Related Work
II-A Large Multi-modal Models
The extraordinary performance of LMMs has greatly propelled the development of multi-modal models. Pioneer studies primarily concentrate on developing techniques for modality alignment. Kosmos-1 [19] implements an end-to-end framework that directly merges visual inputs into LLM in a unified training process. Flamingo [20, 21] integrates visual features with linguistic features in each layer of LLM through a cross attention mechanism. BLIP-2 [22] designs a Q-Former module to fuse visual and linguistic features via learnable queries. MiniGPT-4 [23] and LLaVA [7, 24] circumvent complex cross-modal fusion modules and by directly projecting visual features into the LLM embedding space through a lightweight multi-layer perceptron (MLP).
Subsequent works aim to explore the application of LMMs in wider multi-modal tasks. Shikra [25], Kosmos-2 [26], MiniGPT-v2 [27] and CogVLM [10] integrate visual grounding tasks, significantly enhancing the spatial perception ability of LMMs. Moreover, mPlug-DocOwl [28] and Kosmos-2.5 [29] construct specialized datasets to optimize document understanding. Recent works aim to unify diverse tasks into a general model, such as the LLaVA-NeXT [30], InternLM-XComposer2 [31], InternVL [8] and Qwen-VL [9]. These general LMMs employ additional optimization policies, high-quality datasets covering multiple tasks and intricate training strategies to advance performance across comprehensive vision-language tasks.
II-B Large Multi-modal Models for Video Understanding
Advancements have been made to extend LMMs to video understanding tasks. To accommodate video inputs, video LMMs typically extract frames from the input video and rearrange the frame features encoded by the vision encoder to serve as the final video features. Some works [15, 16, 32] operate on the Q-Former module proposed by BLIP-2 to aggregate visual features and text features. On the other hand, methods including Video-LLaVA [11], Valley [33] and MiniGPT4-Video [23] adhere to the architecture of LLaVA, directly concatenating all frame features and projecting them through a simple fully connected layer.
Certain research efforts investigate the use of spatial or temporal downsampling modules to compress visual tokens. For instance, Video-ChatGPT [12] employs a pooling module to downsample visual tokens along both temporal and spatial dimensions. Chat-Uivi [17] represents videos with a set of dymamic visual tokens to percieve high-level and low-level semantic details. Additionally, PLLaVA [13] introduces an adaptive pooling module for projected visual tokens to accommodate more video frames while minimizing the computational load.
Recent LMMs [34, 30, 35] expand to support multi-image inputs. This enhancement enables the models to leverage extensive existing image-related datasets to elevate video comprehension.
II-C Long Video Understanding
The methods mentioned above impose restrictions on the number of input frames, presenting 