ernal out-of-box video classifier to annotate all videos. By resampling the majority categories that exceed 1% of the total dataset, we reduce their proportion to below 1%. This approach ensures a relatively balanced category distribution in the final dataset as demonstrated in Figure 5.
Fig. 4: Distribution of Video Pre-training Dataset After Category Balance (Top 20)
Fig. 5: Data Distribution of Instruction Tuning Dataset
III-C Video Caption Generation
Existing video LMMs are deficient in description accuracy, richness and diversity. Here we designed a comprehensive video captioning pipeline to address these shortcomings. Firstly, we evenly divide each video into five segments and randomly sample one frame from each segment. Next, a multi-modal model is employed to generate caption for the extracted frames individually. Following that, an off-the-shelf LLM is leveraged to synthesize these frame-level captions into a cohesive video-level caption. In practice, we compare the results of several open-source LMMs and ultimately take InternLM-XComposer2 [31] as the frame-level captioner. To avoid introducing bias by relying on a single model, we randomly select either InternLM2-Chat-20B [4] or Baichuan2-13B-Chat [5] to serve as the caption summarizer. Specialized prompts are designed to instruct the LLM to generate high-quality holistic descriptions. Upon completion of the video preprocessing and caption generation procedures, we establish a large-scale video pre-training dataset comprising over 60 million video-caption pairs.
III-D Video Caption Refinement
Despite the specialized design of summary prompt, we discover that some generated video captions do not meet our expectations. In certain instances, the LLM tends to neglect the instruction from the summary prompt and simply concatenates individual frame captions without semantically restructuring. In addition, in some cases, the captions of adjacent frames from a video are highly correlated, resulting in repetitive sentences in the LLM outputs. These captions contain semantic redundancy and fail to present informative description of the entire video.
To mitigate the impact of repetitive contents, we curate a subset of 15M high-quality data from the 60M video-text dataset to refine video pre-training. We propose a metric to measure the similarity between sentences in a caption. For a video caption with NN sentences, we use jieba [51] for Chinese word segmentation and split English sentence into words. Let sis_{i} represent the word set of the i-th sentence in the caption. For each pair {si,sj}\{s_{i},s_{j}\} of sentences, the similarity between these two sentences is quantified by the Intersection over Union (IoU) metric, defined as:
Si​j=|si∩sj||si∪sj|\displaystyle S_{ij}=\frac{|s_{i}\cap s_{j}|}{|s_{i}\cup s_{j}|}
(2)
In this way, the sentence similarity of the entire video caption is denoted as the maximum similarity of all sentence pairs in the caption. To avoid semantic redundancy in the video captions, we remove data with sentence similarity exceeding the predefined threshold. The filtered video-caption data, along with dense video caption data from ShareGPTVideo [52] is used for pre-training refinement.
TABLE I: Summary of instruction tuning dataset
Stage
Type
Size
Source
Image Pre-training
Image-text pairs
300M
LAION, COYO and Wukong
Video Pre-training
Video-text pairs
60M
Webvid, Panda-70M, Youku-mlpug, ChinaOpen and internal videos
Pre-training Refinement
Detailed description
6.9M
ShareGPTVideo, refined caption on internal videos
Instruction Tuning
Short caption
89.7K
YouCook2, VaTEX, Charades, TextVR
Detailed description
35.5K
ShareGPTVideo, ShareGPT4Video, ShareGPT-4o
Chinese detailed description
50K
Generated detailed description on internal videos
Multi-choice video QA
616.1K
TV-QA, Cinepile, NExT-QA, TGIF-QA(MC), CLEVRER, SSthV2, Kinetics-710
Open-ended video QA
209.9K
Ego-QA, NExT-OE, TGIF-QA(OE), Webvid-QA, MovieChat
Single-round conversation
655.7K
VideoChatGP