e images are discarded due to low quality. Instead, we employ an open-source multi-modal model to generate high-quality captions. Following a detailed evaluation of several off-the-shelf models considering factors including completeness, diversity, hallucination, image-text relevance and response speed, we ultimately opt for InternLM-XComposer2 [31] as the image captioner. The prompt is crafted to limit the generated English and Chinese captions within a preset length.
III-B Video Preprocessing
To build a high-quality dataset for video pre-training, we gather videos from Webvid [43], Panda-70M [44] with English captions and Youku-mplug [45], ChinaOpen [46] with Chinese captions. In addition, a number of internal videos are incorporated to enrich the diversity of video data.
However, subsequent examination of the large amount of video data reveals the low quality of certain videos. To mitigate the degradation of model performance caused by the inclusion of these videos, we propose several video filtering mechanisms, as outlined below:
III-B1 Text Coverage Filter
A common indicator of low-quality videos is the occlusion by large areas of irrelevant text elements, complicating the task of describing the video content accurately. Thus, we extract three frames from each video and employ PaddleOCR [47] to detect text regions in these frames. Given the text bounding boxes {B1,B2,…​Bn}\{B_{1},B_{2},\ldots B_{n}\} in the i-th frame, the text coverage area of this frame tit_{i} is represented as the union area of all text regions, denoted as:
ti\displaystyle t_{i}
=A​r​e​a​(⋃j=1nBj)\displaystyle=Area\left(\bigcup_{j=1}^{n}B_{j}\right)
(1)
The text coverage area of each video is defined as the maximum text coverage area among the three frames. Under this measurement criterion, we remove videos with a text coverage area surpassing the preset threshold.
III-B2 Face Coverage Filter
Another characteristic of inferior video quality is the the excessive proportion of facial content, as some videos feature bloggers speaking or interacting with the audience in front of the camera. Despite the richness of audio content, the visual information in these videos tends to be highly homogeneous and of low relevance. Similar to text coverage, we design a filter to screen these low-quality videos with large facial areas. We uniformly extract five frames from each video and employ YOLOv8 [48] to detect facial regions. Videos with excessively large face coverage area in these frames are excluded.
III-B3 Static Scene Filter
Videos primarily containing static scenes with minimal temporal dynamics provide insufficient variability and context for robust video-text pre-training. To quantify the temporal variability, we apply Optical Flow [49] algorithm to calculate motion vectors between consecutive frames of the video. To accelerate this procedure, the videos are first resized to a low resolution of 128×\times128. Videos with optical flow magnitude below a certain threshold are classified as close-to-static and filtered out from the dataset.
III-B4 Scene Cut
The original datasets include videos spanning a wide range of durations, from short videos of only 1-2 seconds to the longest videos exceeding 20 minutes. As a consequence, we impose a limit on the video duration by removing overly short videos and segmenting short clips from long videos via PySceneDetect [50].
III-B5 Category Balance
The highly imbalanced category distribution in a dataset poses a long-tail challenge and affects the model’s generalization capability. Confronted with this problem, we employ an internal out-of-box video classifier to annotate all videos. By resampling the majority categories that exceed 1% of the total dataset, we reduce their proportion to below 1%. This approach ensures a relatively balanced category distribution in the final dataset as demonstrated in Figure 5.
Fig. 4: Distribution of Video Pre-training Dataset After Category Balance (Top 20)
Fig. 5: Data Distribution of