a lightweight linear projector. As depicted in Figure 2, Kangaroo is composed of a vision encoder, a multi-modal projector, a spatial-temporal patchify module and an LLM.
We initialize the vision encoder from EVA-CLIP-L [71] and choose Llama-3-8B-Instruct [2] as our base LLM. For each input video V∈ℝT×3×H×WV\in\mathbb{R}^{T\times 3\times H\times W}, we start with uniformly sampling a series of n frames {f0,f1,f2,…,fn}\{f_{0},f_{1},f_{2},\ldots,f_{n}\}. Inspired by position embedding proposed in [72], we design a temporal position embedding (TPE) to encode the temporal order of extracted frames in the video, implemented by sinusoidal position encoding:
T​P​E​(t)\displaystyle TPE(t)
=(s​i​n​(tθ0/d)c​o​s​(tθ1/d)…s​i​n​(tθ(d−2)/d)c​o​s​(tθ(d−1)/d))\displaystyle=\begin{pmatrix}sin(\frac{t}{\theta^{0/d}})\\
cos(\frac{t}{\theta^{1/d}})\\
\ldots\\
sin(\frac{t}{\theta^{(d-2)/d}})\\
cos(\frac{t}{\theta^{(d-1)/d}})\end{pmatrix}
(3)
Zft^\displaystyle\hat{Z_{f}^{t}}
=Zft+T​P​E​(t)\displaystyle=Z_{f}^{t}+TPE(t)
(4)
where tt is the timestamp of each frame and Zft^\hat{Z_{f}^{t}} denotes the visual feature augmented with temporal information. Here we use the actual float-type timestamp of a frame instead of its index to incorporate the video’s meta information.
Ultimately, we concatenate visual features of all frames along the time dimension with special tokens interleaved among the sequence to model the temporal inter-dependencies. The resulting visual sequence is then projected into LLM embedding space via the multi-modal projector as the final output of the visual branch, denoted as ZV∈ℝ(h×w×n)×cZ_{V}\in\mathbb{R}^{(h\times w\times n)\times c}:
ZV=P​r​o​j​e​c​t​o​r​(Zf0^⊕Zf1^⊕…⊕Zfn^)\displaystyle Z_{V}=Projector(\hat{Z_{f}^{0}}\oplus\hat{Z_{f}^{1}}\oplus\ldots\oplus\hat{Z_{f}^{n}})
(5)
where ⊕\oplus stands for the concatenate operation.
Fig. 6: Training pipeline of Kangaroo. The training process is composed of five stages: (a) image pre-training stage and video pre-training stage, (b) pre-training refinement stage, (c) instruction tuning stage, (d) long video tuning stage. Trainable and frozen components in each stage are marked in the figure
TABLE II: Detailed training settings of Kangaroo
Configuration
Image Pre-training
Video Pre-training
Pre-training Refinement
Instruction Tuning
Long Video Tuning
ViT init
EVA02-CLIP-L-14
Image Pre-training
Video Pre-training
Pre-training Refinement
Instruction Tuning
LLM init
Llama3-8B-Instruct
Llama3-8B-Instruct
Llama3-8B-Instruct
Llama3-8B-Instruct
Instruction Tuning
Projector init
Random
Image Pre-training
Video Pre-training
Pre-training Refinement
Instruction Tuning
Patchify module init
-
-
Uniform
Pre-training Refinement
Instruction Tuning
Video/Image Resolution
224×\times224
224×\times224
448×\times448
448×\times448
448×\times448
ViT sequence length
256
256
1024
1024
1024
LLM sequence length
512
2560
2560
10K
22K
Maximum frame count
1
8
16
64 (maximum)
16064 (maximum)
Warm-up steps
1000
1000
0
0
0
Global batch size
4096
128
128
128
128
Pipeline parallelism
1
1
1
4
4
Sequence packing
✗
✗
✗
✓
✓
Initial learning rate
1e-4
3e-5
3e-5
1e-5
1e-5
Minimum learning rate
1e-6
Gradient clip
1.0
Numerical precision
bfloat16
ViT Drop path rate
0
IV-B Curriculum Training
As shown in Figure 6, we devise a curriculum training framework with progressively increasing task complexity and training difficulty to incrementally equips a text-based LLM with the capability to process long video inputs.
Stage I: Visual-language Pre-training.
We start with image/video pre-training to connect fundamental language concepts and visual elements. First, we utilize the re-captioned image dataset for image pre-training. Each image is represented as a single-frame video and is randomly assigned a timestamp for temporal position embedding. We then extend to video pre-training based on the curated large-scale video-caption dataset. For each input video, we uniformly extract 8 frames and calculate the respective timestamps. During th