hat-128k
4.62
LWM-Text-Chat-256k
5
LWM-Text-Chat-512k
4.83
LWM-Text-Chat-1M
4.19
Table 11: Relationship between the mix of chat and fact retrieval tasks and the performance on MT-Bench score and Needle Retrieval accuracy.
Chat / QA Mix
MT-Bench
Needle Acc
0% / 100%
2.42
100%
40% / 60%
4.14
100%
70% / 30%
4.62
96%
90% / 10%
5.1
55%
100% / 0%
5.8
31%
Appendix C More Single-Needle Retrieval Results
Figure 11: Needle retrieval task using the LWM-Text-Chat-1M model. The model demonstrates near-perfect retrieval accuracy across various positions within the 1M context window, as reflected by consistently high scores at different depth percentages and context lengths.
Figure 12: Single needle retrieval accuracy for the LWM-Text-Chat-256K model. The model achieves near-perfect retrieval performance across various positions in the 256K context window, as shown by consistently high scores across all depth percentages and context lengths.
Figure 13: Single needle retrieval accuracy for the LWM-Text-Chat-512K model. The model demonstrates near-perfect retrieval performance across different positions within the 512K context window, as indicated by consistently high scores at varying depth percentages and context lengths.
Appendix D Details Of The Multi-needle Evaluations
We built on top of the original Needle in a Haystack problem (see Section 3.3.2 for details), which was to retrieve a single magic number randomly inserted into some long context. In this case, magic numbers are associated with random cities (“The magic number for San Francisco is 2521233” →\rightarrow “What is the magic number for San Francisco?”). We extend the task by introducing two variables N and R, where N is the number of needles (magic numbers + different cities) randomly inserted into the context, and R is the random subset of magic numbers asked to retrieve (“What are the magic numbers for San Francisco, …, and Bangkok?”). Correctness is computed by extracting out the numbers retrieved for each cities and checked with string matching.
Appendix E More Image Understanding Examples
Figure 14: Question answering based on image input using LWM. The assistant provides detailed responses to questions about capturing the essence of a serene lakeside scene and interpreting a creative twist on a classic artwork featuring a dog.
Appendix F More Video Understanding Examples
Figure 15: LWM demonstrates video understanding over a 1-hour video. The figure shows a comparison of different AI systems responding to a question about the number of lemons in a person’s car. While GPT-4V, Gemini Pro Vision, and Video-LLAVA fail to provide a correct response, LWM accurately identifies that there are three lemons in the car.
Figure 16: LWM demonstrates video understanding over a 1-hour video. The figure compares AI systems’ responses to the question ’What animal was standing on a piano?’ While GPT-4V, Gemini Pro Vision, and Video-LLAVA provide incorrect or incomplete answers, LWM correctly identifies that the animal standing on the piano was a cat.
((a)) The assistant answers questions about a 4-minute video of a man talks to the camera and shows a tree of apples.
((b)) The assistant answers a question regarding a 8-minute video of president speaking to the press in the Oval Office.
((c)) The assistant answers a question about a 30-second video of a giraffe.
Figure 17: Answering questions about videos using LWM. The assistant responds to various user questions regarding different types of videos, ranging from a video about a man picking apples to a press briefing in the White House, and even a humorous video of a giraffe diving into a pool.
((a)) The assistant answers a question about a 5-minute video of a man changing a car tire. The process involves using a jack and a spare tire.
((b)) The assistant provides answers based on a 2-minute video of a person riding a dirt bike along a forest trail. The rider wears a helmet and red gloves, traveling at high speed.
Figure 18: The system (LWM)