long video understanding benchmark.
arXiv preprint arXiv:2406.08035.
Cited by: Table 1.
Wang et al. (2024b)
W. Wang, S. Zhang, Y. Ren, Y. Duan, T. Li, S. Liu, M. Hu, Z. Chen, K. Zhang, L. Lu, et al.
Needle in a multimodal haystack.
arXiv preprint arXiv:2406.07230.
Cited by: §1,
§2.1.
West et al. (2023)
P. West, X. Lu, N. Dziri, F. Brahman, L. Li, J. D. Hwang, L. Jiang, J. Fisher, A. Ravichander, K. Chandu, et al.
The generative ai paradox:" what it can create, it may not understand".
arXiv preprint arXiv:2311.00059.
Cited by: §1.
Wu et al. (2025)
H. Wu, D. Li, B. Chen, and J. Li
Longvideobench: a benchmark for long-context interleaved video-language understanding.
Advances in Neural Information Processing Systems 37, pp. 28828–28857.
Cited by: §2.1.
Wu et al. (2024)
Z. Wu, L. Qiu, A. Ross, E. Akyürek, B. Chen, B. Wang, N. Kim, J. Andreas, and Y. Kim
Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks.
In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers),
pp. 1819–1862.
Cited by: §1.
Xiao et al. (2024)
J. Xiao, A. Yao, Y. Li, and T. Chua
Can i trust your answer? visually grounded video question answering.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pp. 13204–13214.
Cited by: §1.
Yang (2024)
Z. Yang
Retrieval or holistic understanding? dolce: differentiate our long context evaluation tasks.
arXiv preprint arXiv:2409.06338.
Cited by: §1.
Zhang et al. (2024a)
P. Zhang, K. Zhang, B. Li, G. Zeng, J. Yang, Y. Zhang, Z. Wang, H. Tan, C. Li, and Z. Liu
Long context transfer from language to vision.
arXiv preprint arXiv:2406.16852.
Cited by: §4.1.
Zhang et al. (2024b)
Y. Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li
Video instruction tuning with synthetic data.
arXiv preprint arXiv:2410.02713.
Cited by: Appendix C,
§4.1.
Zheng et al. (2023)
L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
Advances in Neural Information Processing Systems 36, pp. 46595–46623.
Cited by: Appendix C.
Zhou et al. (2024)
J. Zhou, Y. Shu, B. Zhao, B. Wu, S. Xiao, X. Yang, Y. Xiong, B. Zhang, T. Huang, and Z. Liu
Mlvu: a comprehensive benchmark for multi-task long video understanding.
arXiv preprint arXiv:2406.04264.
Cited by: §2.1,
Table 1.
NeurIPS Paper Checklist
1.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: In the abstract, we clearly state our primary contribution, which is the design of a novel set of benchmark questions specifically created to evaluate models’ abilities in long-context video understanding. In the introduction, we highlight two key limitations of existing video understanding datasets to underscore the necessity and effectiveness of our proposed benchmark. We further summarize the performance of contemporary video large language models and identify their strengths and weaknesses. Thus, these claims represent the primary contributions of our paper, and these contributions are also summarized explicitly in the introduction.
Guidelines:
•
The answer NA means that the abstract and introduction do not include the claims made in the paper.
•
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
•
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
•
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2.
Limitations
Question: Does the paper discuss the limi