s of long videos. Kangaroo effectively grasps the overall content in long videos (category, scene), precisely locates the relevant video segments and captures visual details (count, action) in specific segments from a long video.
V-A Implementation
For all training stages, we use AdamW optimizer with the weight cay of 0.05 and cosine learning rate schedule. Layer-wise Learning Rate Decay strategy is applied for the vision encoder with the decay factor of 0.9. We only calculate cross-entropy loss for autoregressive text generation. Other detailed training settings are reported in Table II.
V-B Evaluation
We conduct standardized evaluation with greedy decoding and compare with other open-source and proprietary model. The best performance among the open-source models on each benchmark is highlighted in bold and the second-best results are underlined. Items that Kangaroo outperforms proprietary models are denoted in gray.
To comprehensively explore the effectiveness of our model, we evaluate on a series of benchmarks covering a wide range of tasks, which can be categorized into three aspects:
V-B1 General Video Understanding
We conduct evaluation on MVBench [16], MMBench-Video [73], VideoVista [74], VideoMME [75] and SeedBench [76] and report the results in Table III. MMBench-Video encompasses videos of varying durations with open-ended questions. VideoVista primarily assesses the model’s reasoning ability across diverse content categories. Compared to other open-source LMMs with similar sizes, Kangaroo achieves state-of-the-art (SOTA) performance on these two benchmarks, demonstrating our model’s robust adaptability to different types of videos and strong reasoning capacity. Kangaroo approaches SOTA level and surpasses GPT-4V on MVBench, partly due to the relatively short video lengths in this benchmark.
To illustrate the specific performance on different sub-tasks, we take VideoMME as an example. The detailed results in Table IV indicate that Kangaroo excels in all sub-tasks and outclasses some larger models with over 10B parameters. Additionally, we compare with other multi-image LMMs across video-related tasks from SeedBench. Given the short durations, only 8 frames of each video are provided in this benchmark, so we duplicate each set of 8 frames to 16 frames for evaluation. As demonstrated in Table V, with the same number of input frames, Kangaroo outranks most multi-image LMMs including Claude-3 and performs close to GPT-4V.
V-B2 Specialized Tasks
In addition to general video understanding benchmarks, we also test our model on the egocentric QA benchmark EgoSchema [77] and the temporal reasoning benchmark TempCompass [78]. The results verify that as a general video understanding model, Kangaroo achieves SOTA performance among open-source models in these specialized tasks, and outperforms GPT-4V on EgoSchema.
V-B3 Long Video Understanding
We focus on benchmarks tailored for long videos, involving MLVU [79], LVBench [80] and LongVideoBench [81]. As reported in Table III, Kangaroo considerably outperforms other 7B/8B models on these benchmarks. Notably, our model surpasses some larger models with over 20B parameters and proprietary models on certain metrics. This outstanding performance corresponds with our initial motivation, confirming the superiority of the curated high-quality dataset and curriculum training pipeline.
TABLE III: Evaluation results on comprehensive video understanding benchmarks
Model
Size
MVBench
MLVU
MMBench-Video
LVBench
EgoSchema
VideoVista
TempCompass
LongVideoBench
VideoChatGPT [12]
7B
32.7
31.3
0.93
-
-
36.65
43.5
-
Video-LLaVA [11]
7B
43.5
47.3
1.05
-
38.4
56.96
49.8
37.6
VideoChat2-HD [16]
7B
62.3
47.9
-
-
55.8
61.58
-
41.2
LLaMA-VID [18]
7B
41.9
33.2
-
23.9
38.5
56.87
45.6
-
PLLaVA [13]
7B
46.6
-
1.03
-
54.4
60.36
-
39.2
VideoLLaMA2 [82]
7B
54.6
48.5
-
-
51.7
60.47
-
-
LLaVA-NeXT-Video [14]
7B
53.1
-
-
-
43.9
56.66
53.7
43.5
LongVA [40]
7B
-
56.3
-
-
-
67.36
56.9
-
TimeChat [37]
7B
-
30.9
-
22.3
3