Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input
Report GitHub Issue
×
Title:
Content selection saved. Describe the issue below:
Description:
Submit without GitHub
Submit in GitHub
arXiv is now an independent nonprofit!
Learn more
×
Back to arXiv
Why HTML?
Report Issue
Back to Abstract
Download PDF
Abstract
I Introduction
II Related Work
II-A Large Multi-modal Models
II-B Large Multi-modal Models for Video Understanding
II-C Long Video Understanding
III Data Curation
III-A Image Re-Captioning
III-B Video Preprocessing
III-B1 Text Coverage Filter
III-B2 Face Coverage Filter
III-B3 Static Scene Filter
III-B4 Scene Cut
III-B5 Category Balance
III-C Video Caption Generation
III-D Video Caption Refinement
III-E Instruction Tuning Dataset
III-E1 Video Caption
III-E2 Video QA
III-E3 Conversation
IV Methodology
IV-A Architecture
IV-B Curriculum Training
V Experiments
V-A Implementation
V-B Evaluation
V-B1 General Video Understanding
V-B2 Specialized Tasks
V-B3 Long Video Understanding
V-C Qualitative Results
VI Conclusion
References
License: CC BY 4.0
arXiv:2408.15542v1 [cs.CV] 28 Aug 2024
Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input
Jiajun Liu*
Yibing Wang*
Hanghang Ma†
Xiaoping Wu†
Xiaoqi Ma†
Xiaoming Wei
Jianbin Jiao
Enhua Wu
and Jie Hu‡
††thanks: Jiajun˜Liu, Hanghang˜Ma, Xiaoping˜Wu, Xiaoqi˜Ma, and Xiaoming˜Wei are with Meituan Group, Beijing 100102, China (E-mail: {liujiajun18, mahanghang, wuxiaoping03, maxiaoqi05, weixiaoming}@meituan.com).††thanks: Yibing˜Wang and Jianbin˜Jiao are with the School of Electronic, Electrical and Communication Engineering, the University of Chinese Academy of Sciences, (UCAS), Beijing, 100049, China. (E-mail:˜wangyibing18@mails.ucas.ac.cn).††thanks: Jie˜Hu and Enhua˜Wu are with the Key Laboratory of System Software (Chinese Academy of Sciences) and State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, 100190, China, and also with the University of Chinese Academy of Sciences, Beijing, 100049, China.
Jie Hu is also with Meituan Group (E-mail:˜hujie@ios.ac.cn; ehwu@um.edu.mo).††thanks: The code and model are openly accessible to the public and can be found at https://github.com/KangarooGroup/Kangaroo.††thanks: * Joint first authors. $†$ Key contributors.††thanks: $‡$ Corresponding author and Project lead.
Abstract
Rapid advancements have been made in extending Large Language Models (LLMs) to Large Multi-modal Models (LMMs). However, extending input modality of LLMs to video data remains a challenging endeavor, especially for long videos. Due to insufficient access to large-scale high-quality video data and the excessive compression of visual features, current methods exhibit limitations in effectively processing long videos. In this paper, we introduce Kangaroo, a powerful Video LMM aimed at addressing these challenges. Confronted with issue of inadequate training data, we develop a data curation system to build a large-scale dataset with high-quality annotations for vision-language pre-training and instruction tuning. In addition, we design a curriculum training pipeline with gradually increasing resolution and number of input frames to accommodate long videos. Evaluation results demonstrate that, with 8B parameters, Kangaroo achieves state-of-the-art performance across a variety of video understanding benchmarks while exhibiting competitive results on others. Particularly, on benchmarks specialized for long videos, Kangaroo excels some larger models with over 10B parameters and proprietary models.
Index Terms:
Video Understanding, Vision-Language Models, Multi-modal Learning
I Introduction
Fig. 1: Kangaroo achieves SOTA performance across various comprehensive video understanding benchmarks, surpassing other 7B/8B models.
Fig. 2: Architecture of Kangaroo. The model consists of a vision encoder, a spatial-temporal patchify module, a multi-modal projector and an LLM. Frames extracted from th