jected visual tokens to accommodate more video frames while minimizing the computational load.
Recent LMMs [34, 30, 35] expand to support multi-image inputs. This enhancement enables the models to leverage extensive existing image-related datasets to elevate video comprehension.
II-C Long Video Understanding
The methods mentioned above impose restrictions on the number of input frames, presenting challenges for understanding long videos. To handle hour-long videos, LLaMA-VID [18] adopts a context attention module to integrate multi-modal features and downsamples visual tokens into two tokens for each frame. MovieChat [36] designs a short-term and long-term memory module to consolidate local details and overall contents within the video. Similarly, TimeChat [37] introduces a timestamp-aware frame encoder to bind visual content with temporal information and uses a sliding Q-Former to integrate video tokens. Although these methods intend to enhance the perception of both local and global information in long videos, the excessive compression of visual tokens results in suboptimal results.
In contrast, some research adopts a stepwise strategy to incrementally adapt to long video inputs. LWM [38] gradually extends the context size from 4K to 1M tokens by optimizing the implementation of Blockwise RingAttention [39] and other features for millions-length sequence training. LongVA [40] attempts to leverage the long context generalization capability of LLMs for extremely long videos. The multi-modal model is trained based on an LLM initially generalized to millions of tokens. Training on long-context sequences ensures the entirety of visual information. However, the limited access to high-quality video data remains an obstacle to achieving ideal performance.
Up to date, long video understanding remains a challenging task. Our proposed method aims to preserve the fidelity of the original visual information and elevate performance in long video understanding tasks by large-scale video-text datasets within a curriculum learning framework.
III Data Curation
Fig. 3: We develop a sophisticated data curation system tailored for video pre-training. To ensure high-quality input, we introduce a series of filters aimed at excluding low-quality videos. An off-the-shelf multi-modal model is subsequently employed to generate captions for frames extracted from each video. Following this, we utilize a well-trained LLM to summarize all frame-level captions into a coherent video description. For the pre-training refinement stage, we implement an additional strategy to discard video descriptions exhibiting repetitive content.
In this section, we provide a detailed exposition of the data curation system. We start with the introduction of the image re-captioning procedure. Next, we elaborate on detailed steps of the video preprocessing and caption generation pipeline, setting the stage for robust video-text pre-training. In the last part, we demonstrate the composition of the multi-task instruction tuning dataset in detail. The schematic of our data curation process is depicted in Figure 3.
III-A Image Re-Captioning
Images form the foundation for bridging vision and language modalities. However, the captions accompanying images in existing public datasets are often noisy and unsuitable for pre-training. To address this issue, we implement a re-captioning procedure to create a high-quality dataset for image pre-training. Specifically, we select images from LAION-5B-en [41] for English data and Wukong [42] for Chinese data. The original captions associated with these images are discarded due to low quality. Instead, we employ an open-source multi-modal model to generate high-quality captions. Following a detailed evaluation of several off-the-shelf models considering factors including completeness, diversity, hallucination, image-text relevance and response speed, we ultimately opt for InternLM-XComposer2 [31] as the image captioner. The prompt is crafted to