 present Kangaroo, a powerful Large Multi-modal Model that excels at long video understanding. We develop a data curation system and establish a large-scale video-based dataset. To improve the capacity for processing long videos, we adopt high-resolution inputs and extend the number of input frames to keep intact visual details. Extensive evaluation results demonstrate that Kangaroo achieves superior performance on diverse comprehensive video understanding benchmarks. Remarkably, on benchmarks specialized for long videos, Kangaroo outperforms larger models with over 10B parameters and proprietary models on certain metrics.
Building upon these promising results, we envision future works focusing on the following aspects:
•
We aim to extend Kangaroo to additional modalities like audio, charts and interleaved image-video-text data. The integration of multi-modal information is expected to provide more robust and comprehensive understanding of real-world scenarios.
•
We intend to scale up the model size and equip with more powerful LLMs. Larger LLMs are expected to empower Kangaroo to capture complex patterns in data and improve multi-modal reasoning.
•
We will investigate methods to generalize to more complex tasks. For challenging tasks including counting, visual grounding and temporal localization, specialized modules and high-quality data act as the vital role in pushing the boundaries of model’s application domains.
References
[1]
OpenAI, “Introducing chatgpt,” https://openai.com/chatgpt/, 2023.
[2]
AI@Meta, “Llama 3 model card,” https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md, 2024.
[3]
T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Rojas, G. Feng, H. Zhao, H. Lai et al., “Chatglm: A family of large language models from glm-130b to glm-4 all tools,” arXiv preprint arXiv:2406.12793, 2024.
[4]
Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu et al., “Internlm2 technical report,” arXiv preprint arXiv:2403.17297, 2024.
[5]
A. Yang, B. Xiao, B. Wang, B. Zhang, C. Bian, C. Yin, C. Lv, D. Pan, D. Wang, D. Yan et al., “Baichuan 2: Open large-scale language models,” arXiv preprint arXiv:2309.10305, 2023.
[6]
OpenAI, “Gpt-4v(ision) system card,” https://cdn.openai.com/papers/GPTV_System_Card.pdf, 2023.
[7]
H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, 2024.
[8]
Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu et al., “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24 185–24 198.
[9]
J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” arXiv preprint arXiv:2308.12966, 2023.
[10]
W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song et al., “Cogvlm: Visual expert for pretrained language models,” arXiv preprint arXiv:2311.03079, 2023.
[11]
B. Lin, B. Zhu, Y. Ye, M. Ning, P. Jin, and L. Yuan, “Video-llava: Learning united visual representation by alignment before projection,” arXiv preprint arXiv:2311.10122, 2023.
[12]
M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video-chatgpt: Towards detailed video understanding via large vision and language models,” Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
[13]
L. Xu, Y. Zhao, D. Zhou, Z. Lin, S. K. Ng, and J. Feng, “Pllava: Parameter-free llava extension from images to videos for video dense captioning,” arXiv preprint arXiv:2404.16994, 2024.
[14]
Y. Zhang, B. Li, h. Liu, Y. j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li, “Llava-next: A strong zero-shot video understanding model,” https://llava-vl.github.io/blog/2024-04-30-llava-next-video/, 2024.
[15]
K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. W