l elements. First, we utilize the re-captioned image dataset for image pre-training. Each image is represented as a single-frame video and is randomly assigned a timestamp for temporal position embedding. We then extend to video pre-training based on the curated large-scale video-caption dataset. For each input video, we uniformly extract 8 frames and calculate the respective timestamps. During this stage, the visual encoder and multi-modal projector are trained via back-propagation, guided by the subsequent frozen LLM.
Stage II: Pre-training Refinement.
Due to the inconsistent quality of the initial video captions, we introduce an additional pre-training refinement stage. This stage aims to mitigate the effects of the fixed format and repetitive content present in the video pre-training dataset. To align with higher-quality data, the number and resolution of input frames are simultaneously increased to 16×448×44816\times 448\times 448. Furthermore, we implement a spatial-temporal patchify module using a 3D depthwise convolution operation to condense visual tokens, ensuring that the number of tokens fed into the LLM remains unchanged.
Stage III: Instruction Tuning.
After the preliminary alignment of visual and linguistic features, we fine-tune the entire model to enhance the instruction-following abilities across various tasks with the subsequent strategies:
•
Visual Augmentation.
With the aim of preserving visual information, we maintain a high resolution for input frames and increase the maximum number of frames to 64. Under this setting, the maximum context length for the input of LLM is extended to 10K tokens in support of long-context video inputs.
•
Dynamic Frame Sampling.
We implement a a dynamic frame sampling strategy with a variable number of input frames. The number of input frames are initially set between 16 and 64, and then adjusted according to the video’s duration. This approach allows for more frames to be captured from long videos to cover the overall content, while avoiding excessive frames sampled from short videos, which leads to information redundancy and reduces training efficiency.
•
Sequence Packing.
Instead of padding all sequences to a standard length, we employ an advanced sequence packing strategy to process the fused multi-modal sequences of varying lengths. By aggregating multiple sequences of different lengths into one composite instance with appropriate attention masks, the sequence packing strategy reduces unnecessary computation of extra padding tokens within a batch.
Stage IV: Long Video Tuning.
We introduce an additional stage to fine-tune the model with videos longer than 64 seconds selected from the instruction tuning dataset. The maximum number of input frames is further increased to 160 to accommodate extremely long videos. To maintain comprehension across different video lengths, a subset of short videos from the instruction tuning dataset is incorporated for balance. During this stage, we focus on generalizing the LLM to process longer context inputs, so we freeze the vision encoder and fine-tune other components.
V Experiments
Fig. 7: Qualitative results of reasoning capability. Kangaroo responds accurately and provides explicit reasons for the answer, such as inferring camera angles, recognizing actions and speculating about counterfactual scenarios.
Fig. 8: Qualitative results of multi-round and bilingual dialogues. Kangaroo exhibits a deep comprehension of contextual contents in multi-round dialogues and supports bilingual switching within the same conversation.
Fig. 9: Qualitative results of long videos. Kangaroo effectively grasps the overall content in long videos (category, scene), precisely locates the relevant video segments and captures visual details (count, action) in specific segments from a long video.
V-A Implementation
For all training stages, we use AdamW optimizer with the weight cay of 0.05 and cosine learning rate schedule. Layer-wise Learning Rate Decay strategy 