2, VaTEX, Charades, TextVR
Detailed description
35.5K
ShareGPTVideo, ShareGPT4Video, ShareGPT-4o
Chinese detailed description
50K
Generated detailed description on internal videos
Multi-choice video QA
616.1K
TV-QA, Cinepile, NExT-QA, TGIF-QA(MC), CLEVRER, SSthV2, Kinetics-710
Open-ended video QA
209.9K
Ego-QA, NExT-OE, TGIF-QA(OE), Webvid-QA, MovieChat
Single-round conversation
655.7K
VideoChatGPT, ShareGPTVideo
Multi-round conversation
35.6K
VideoChat, VideoChat2, Valley (VaTEX subset)
Chinese conversation
547.7K
Generated conversation on ChinaOpen and internal videos
Long Video Tuning
Long videos
700K
Videos longer than 64s and short videos from instruction tuning dataset
III-E Instruction Tuning Dataset
Instruction-following data improves the model’s generalization ability to various tasks [53]. To enable the model’s versatility to different tasks, we compile a large-scale video instruction tuning dataset comprising 2.24M samples from distinct public and internal sources. Detailed composition of the dataset is reported in Table I. The curated instruction tuning dataset covers a wide range of video understanding tasks and can be divided into the following categories:
III-E1 Video Caption
including short video caption and detailed video description data in both English and Chinese. Videos with short captions are collected from VaTEX [54], YouCook2 [55], Charades [56], TextVR [57] and detailed description data from ShareGPTVideo, ShareGPT-4o [58], ShareGPT4Video [59]. Each video from VaTEX and Charades corresponds to multiple short captions. Consequently, we leverage GPT-4 [60] with custom designed prompts to summarize all short captions into one overall caption. In addition, for Chinese understanding, a subset of Chinese detailed description data from the video pre-training dataset is incorporated in this stage. We formulate corresponding instructions for each type of caption to convert captions into QA pairs.
III-E2 Video QA
including multi-choice VQA and open-ended VQA data collected from various sources. This subset encompasses fundamental VQA: NExT-QA [61], TGIF-QA [62], WebvidVQA [63], egocentric VQA: Ego-4d [64], reasoning VQA: CLEVRER [65], action recognition: Something-Something-V2 [66, 67], Kinetics-710 [68] and VQA with long videos from TV-QA [69], Cinepile [70], MovieChat. For WebvidVQA, Ego-4d and action classification subsets, we adopt the partition and instructions from MVBench [16]. We additionally design specialized templates for multi-choice VQA data to combine each question and the corresponding options.
III-E3 Conversation
including single/multi-round conversation in both Chinese and English. We include ShareGPTVideo, Video-ChatGPT and integrate single-round conversations from VideoChat, VideoChat2 and Valley into multi-round format. To improve the Chinese comprehension ability of our model, we select a subset from the video pre-training dataset and and utilize GPT-4 to generate human-like conversations based on the high-quality Chinese video description.
After collection and organization, we perform a statistical analysis of the resulting dataset and illustrate the distribution in Figure 5. Data with video length less than 5 seconds is excluded and all videos are resampled to 4 fps to accelerate data loading during the training process. .
IV Methodology
In this section, we present the overall architecture of our Kangaroo model and systematically elaborate on our curriculum training strategy.
IV-A Architecture
We utilize a widely adopted multi-modal architecture that bridges visual and linguistic features through a lightweight linear projector. As depicted in Figure 2, Kangaroo is composed of a vision encoder, a multi-modal projector, a spatial-temporal patchify module and an LLM.
We initialize the vision encoder from EVA-CLIP-L [71] and choose Llama-3-8B-Instruct [2] as our base LLM. For each input video V∈ℝT×3×H×WV\in\mathbb{R}^{T\times 3\times H\times W}, we start with uniformly sampling a series of n f