umorous video of a giraffe diving into a pool.
((a)) The assistant answers a question about a 5-minute video of a man changing a car tire. The process involves using a jack and a spare tire.
((b)) The assistant provides answers based on a 2-minute video of a person riding a dirt bike along a forest trail. The rider wears a helmet and red gloves, traveling at high speed.
Figure 18: The system (LWM) successfully answers questions about video content.
Appendix G Details Of Qualitative Video Understanding Evaluation
For qualitative evaluation of our videos, we source various videos from YouTube that cover a range of topics, such as ego-centric camera, how to videos, interviews, and animations. We evaluate all videos at 1FPS, and sample uniformly a max number of frames for videos that are longer than what our video can support at 1 FPS. Videos are additionally resized and center cropped to 256×256256\times 256 resolution before inputting into the model.
Appendix H More Image Generation Examples
Figure 19: Images generation using LWM, showcasing various scenes and objects.
Appendix I More Video Generation Examples
Figure 20: Video sequences generated using LWM, showing various scenes.
Appendix J Training Hyperparameters
Table 12: LWM-Text Training Stages
32K
128K
256K
512K
1M
Parameters
7B
7B
7B
7B
7B
Initialize From
LLaMA-2 7B
Text-32K
Text-128K
Text-256K
Text-512K
Precision
float32
float32
float32
float32
float32
Sequence Length
2152^{15}
2172^{17}
2182^{18}
2192^{19}
2202^{20}
RoPE θ\theta
1M
10M
10M
25M
50M
Tokens per Batch
4M
4M
4M
4M
4M
Total Tokens
4.8B
12B
12B
3B
1.8B
Total Steps
1200
3000
3000
720
450
LR Schedule
Constant
Constant
Constant
Constant
Constant
LR Warmup Steps
100
200
200
50
25
LR
4×10−54\times 10^{-5}
4×10−54\times 10^{-5}
4×10−54\times 10^{-5}
4×10−54\times 10^{-5}
4×10−54\times 10^{-5}
Compute (TPU)
v4-512
v4-512
v4-512
v4-512
v4-512
Mesh Sharding
1,-1,4,1
1,-1,8,1
1,-1,16,1
1,-1,16,2
1,-1,16,4
Table 13: LWM-Text-Chat Training Details
128K
256K
512K
1M
Parameters
7B
7B
7B
7B
Initialize From
Text-128K
Text-256K
Text-512K
Text-1M
Precision
float32
float32
float32
float32
Sequence Length
2172^{17}
2182^{18}
2192^{19}
2202^{20}
RoPE θ\theta
10M
10M
25M
50M
Tokens per Batch
4M
4M
4M
4M
Total Tokens
1.2B
1.2B
1.2B
1.2B
Total Steps
300
300
300
300
LR Schedule
Constant
Constant
Constant
Constant
LR Warmup Steps
25
25
25
25
LR
4×10−54\times 10^{-5}
4×10−54\times 10^{-5}
4×10−54\times 10^{-5}
4×10−54\times 10^{-5}
Compute (TPU)
v4-512
v4-512
v4-512
v4-512
Mesh Sharding
1,-1,4,1
1,-1,8,1
1,-1,16,1
1,-1,16,2
Table 14: LWM / LWM-Chat Training Stages
1K
8K
32K
128K
1M
Parameters
7B
7B
7B
7B
7B
Initialize From
Text-1M
1K
8K
32K
128K
Precision
float32
float32
float32
float32
float32
Sequence Length
2102^{10}
2132^{13}
2152^{15}
2172^{17}
2202^{20}
RoPE θ\theta
50M
50M
50M
50M
50M
Tokens per Batch
8M
8M
8M
8M
8M
Total Tokens
363B
107B
10B
3.5B
0.4B
Total Steps
45000
14000
1200
450
50
LR Schedule
Cosine
Cosine
Cosine
Cosine
Cosine
LR Warmup Steps
1000
500
100
50
5
Max LR
6×10−46\times 10^{-4}
6×10−46\times 10^{-4}
8×10−58\times 10^{-5}
8×10−58\times 10^{-5}
8×10−58\times 10^{-5}
Min LR
6×10−56\times 10^{-5}
6×10−56\times 10^{-5}
8×10−58\times 10^{-5}
8×10−58\times 10^{-5}
8×10−58\times 10^{-5}
Compute (TPU)
v4-1024
v4-1024
v4-1024
v4-1024
v4-1024
Mesh Sharding
1,-1,1,1
1,-1,1,1
1.-1.4,1
1.-1.8,1
1,-1,16,4
Experimental support, please
view the build logs
for errors. Generated by
L
A
T
E
xml
.
Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile
support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the
methods listed below:
Click the "Report Issue" (
) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we
may not