 et al. (2020)
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.
The pile: An 800gb dataset of diverse text for language modeling.
arXiv preprint arXiv:2101.00027, 2020.
Geng and Liu (2023)
Xinyang Geng and Hao Liu.
Openllama: An open reproduction of llama.
URL: https://github. com/openlm-research/open_llama, 2023.
Geng et al. (2023)
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song.
Koala: A dialogue model for academic research.
Blog post, April, 1, 2023.
gkamradt (2023)
gkamradt.
Needle in a haystack - pressure testing llms.
https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main, 2023.
[Online; accessed 7-Feb-2024].
Ha and Schmidhuber (2018)
David Ha and Jürgen Schmidhuber.
World models.
arXiv preprint arXiv:1803.10122, 2018.
Ho and Salimans (2022)
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint arXiv:2207.12598, 2022.
Ho et al. (2022a)
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.
Imagen video: High definition video generation with diffusion models.
arXiv preprint arXiv:2210.02303, 2022a.
Ho et al. (2022b)
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.
Video diffusion models.
arXiv preprint arXiv:2204.03458, 2022b.
Jin et al. (2023a)
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Chia-Yuan Chang, and Xia Hu.
Growlength: Accelerating llms pretraining by progressively growing training length.
arXiv preprint arXiv:2310.00576, 2023a.
Jin et al. (2023b)
Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al.
Unified language-vision pretraining with dynamic discrete visual tokenization.
arXiv preprint arXiv:2309.04669, 2023b.
Korthikanti et al. (2022)
Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro.
Reducing activation recomputation in large transformer models.
arXiv preprint arXiv:2205.05198, 2022.
Lee et al. (2024)
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al.
Can long-context language models subsume retrieval, rag, sql, and more?
arXiv preprint arXiv:2406.13121, 2024.
Li et al. (2023)
Dacheng Li, Rulin Shao, Anze Xie, Eric P Xing, Joseph E Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang.
Lightseq: Sequence level parallelism for distributed training of long context transformers.
arXiv preprint arXiv:2310.03294, 2023.
Li et al. (2022)
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
Li et al. (2021)
Shenggui Li, Fuzhao Xue, Yongbin Li, and Yang You.
Sequence parallelism: Making 4d parallelism possible.
arXiv preprint arXiv:2105.13120, 2021.
Lin et al. (2023)
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.
Video-llava: Learning united visual representation by alignment before projection.
arXiv preprint arXiv:2311.10122, 2023.
Liu and Abbeel (2023)
Hao Liu and Pieter Abbeel.
Blockwise parallel transformer for large context models.
Advances in neural information processing systems, 2023.
Liu et al. (2024)
Hao Liu, Matei Zaharia, and Pieter Abbeel.
Ring attention with blockwise transformers for near-infinite context.
International Conference on Learning Representations(ICLR), 2024.
Liu et al. (2023a)
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning.
arXiv preprint arXiv:2310.03744, 2023a.
Liu et al. (2023b)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning