s in, tokens out," allows greater flexibility in modeling various formats, including image-text, text-image, text-video, video-text, and pure formats like video, image, or text. Our method is compatible with these prior works, making it an interesting future direction to combine continuous embeddings as input with discrete tokens and a long-context autoregressive model.
6 Conclusion
In conclusion, this paper tackles the critical challenge of enabling long-context understanding in sequence models, which is vital for developing generally intelligent systems capable of processing large temporal sequences. By exploring the development of 1M context language and video-language models, the work sets new benchmarks in language retrieval and long video understanding. We outline approaches to data curation and progressive context extension, accompanied by an efficient open-source implementation for scalable training on long sequences. Moreover, we open-source a family of 7B parameter models capable of handling over 1M tokens in text and video.
Limitations. While this work
successfully develop a large large context of over 1M text and video tokens, and demonstrate promising results in processing hour-long videos and long documents, there are still some limitations that need to be addressed:
•
Improved tokenization and embedding. This work uses a vanilla image tokenizer for images and frame-by-frame tokenization for videos. Future work could explore video tokenization that takes time redundancy into account, as well as including continuous embeddings as input to enrich image understanding.
•
Limited scale. Our models use more tokens per parameter than Chinchilla’s recommendation, but being much smaller than current large language models (100B+ parameters), our findings may not directly apply to them. Extrapolating to larger scales should be done cautiously, as different scaling behaviors could emerge at those larger sizes.
Acknowledgments
This project is supported in part by Office of Naval Research grant N00014-21-1-2769 and ARO MURI (2023) on Neuro-Inspired Distributed
Deep Learning. We thank Google TPU Research Cloud for granting us access to TPUs, and thank Google Cloud for granting us research credits for storage. Pieter Abbeel holds concurrent appointments as a Professor at UC Berkeley and as an Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon.
References
Aiello et al. (2023)
Emanuele Aiello, Lili Yu, Yixin Nie, Armen Aghajanyan, and Barlas Oguz.
Jointly training large autoregressive multimodal models.
arXiv preprint arXiv:2309.15564, 2023.
ArizeAI (2023)
ArizeAI.
Needle in a haystack - pressure testing llms.
https://github.com/Arize-ai/LLMTest_NeedleInAHaystack, 2023.
Awadalla et al. (2023)
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al.
Openflamingo: An open-source framework for training large autoregressive vision-language models.
arXiv preprint arXiv:2308.01390, 2023.
Bain et al. (2021)
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.
Frozen in time: A joint video and image encoder for end-to-end retrieval.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021.
Beltagy et al. (2020)
Iz Beltagy, Matthew E Peters, and Arman Cohan.
Longformer: The long-document transformer.
arXiv preprint arXiv:2004.05150, 2020.
Bradbury et al. (2018)
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.
JAX: composable transformations of Python+NumPy programs, 2018.
URL http://github.com/google/jax.
Brandon et al. (2023)
William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley.
Striped attention: Faster ring attention for causal transformers.
arXiv preprint arXiv:23