CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video

Dingli Liang Affiliation: University College London Email: mailto:miruku.hzhang@gmail.commiruku.hzhang@gmail.com Yiqiao Xie Affiliation: Imperial College London Email: mailto:zhangzhensong@huawei.comzhangzhensong@huawei.com Yukai Huang Affiliation: Durham University Zhaokai Wang Affiliation: Shanghai Jiao Tong University Weitong Cai Affiliation: Queen Mary University of London Guangwen Feng Affiliation: Independent Researcher^*Equal contribution. ^†Corresponding authors. Jifei Song Affiliation: Huawei Noah’s Ark Lab Zhensong Zhang Affiliation: Huawei Noah’s Ark Lab Hang Zhang Affiliation: Independent Researcher^*Equal contribution. ^†Corresponding authors.

Abstract

Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token costs, and long-context retrieval failures. Under these practical constraints, we study whether textual captions can serve as reusable episodic memory. We define the Episodic Memory Video Caption QA task and introduce CapMem, a human-annotated benchmark with 75 videos totaling 33.7 hours, and 1,000 multiple-choice questions across 16 scenarios. On long videos ( $>20$ min), full-coverage CaptionQA with 30s and 60s caption windows outperforms direct VideoQA for 10/12 and 8/12 models, respectively. On the same video subset, a matched-frame control across six Qwen models retains mean accuracy gains of 3.22 and 2.55 points, respectively. Our caption-guided retrieve-and-verify harness further improves accuracy by up to 5.3 points. These results support the effectiveness of caption memory for episodic reasoning over long egocentric video.

1 Introduction

Recent advances in vision-language models (VLMs) have turned wearable vision devices from passive recorders into proactive assistants, supporting tasks from vision-based navigation Tokmurziyev et al. (2026) to laboratory procedure assistance Cong et al. (2025). Yet such assistants still struggle to retrieve information from continuous egocentric video streams, such as locating a misplaced object or recalling a specific action from hours of visual history Bärmann and Waibel (2022); Mangalam et al. (2023). This challenge is fundamentally about episodic memory: recovering what happened, where it happened, and when it happened from past experience Grauman et al. (2022); Majumdar et al. (2024); Cheng et al. (2024). Extending this ability to egocentric video stream is challenging: processing continuous visual input is computationally expensive Song et al. (2024); Liu et al. (2024a), while visual-token budgets limit the number of frames VLMs can process per inference, resulting in sparser temporal coverage and a greater risk of missing relevant evidence Lu et al. (2025); Tang et al. (2025). Even with relevant frames, long multimodal contexts can exacerbate fine-grained perception failures Fu et al. (2024) and the “lost-in-the-middle” effect Liu et al. (2024b); Tian et al. (2025); Fei et al. (2026) (Figure 2). An alternative is to convert visual streams into textual memory. Although caption-based representations have shown promise for static images Yang et al. (2025b); Xing et al. (2026), whether they can support long-form egocentric episodic memory is largely unexplored.

We address this gap by defining Episodic Memory Video Caption QA, a task that evaluates whether self-generated captions can serve as episodic memory in place of direct visual input for egocentric QA. Existing egocentric benchmarks Grauman et al. (2022); Grauman et al. (2025); Yang et al. (2025a) are not designed for this purpose: they emphasize retrieval or direct VideoQA and often lack questions probing fine-grained visual details and long-range temporal recall. Moreover, VLM-generated questions Yang et al. (2025b); Maaz et al. (2024); Chen et al. (2024); Li et al. (2024); Zhang et al. (2025) can inherit the spatial and temporal weaknesses the benchmark seeks to measure. To provide a cleaner testbed, we introduce CapMem, a human-annotated benchmark for egocentric episodic memory. It contains 1,000 human-authored questions grounded in 75 public egocentric videos totaling 33.7 hours across 16 scenarios. CapMem supports three task settings, a two-stage evaluation pipeline with a CleanQA safeguard, and five diagnostic dimensions. Evidence timestamps and fine-grained visual tags enable precise error analysis.

Refer to caption — Figure 1: CapMem scenario coverage and examples where CaptionQA succeeds but VideoQA fails, percentage for video share.

We also filter out questions that can be answered without visual evidence.

Using CapMem, we evaluate 12 proprietary and open-weight VLMs. Across the full benchmark, full-coverage CaptionQA outperforms direct VideoQA for 8/12 models with 30-second captions and 6/12 with 60-second captions. These gains concentrate on videos longer than 20 minutes, where CaptionQA outperforms direct VideoQA for 10/12 and 8/12 models, while direct VideoQA remains competitive on shorter videos. After matching visual frames, CaptionQA retains mean long-video gains of 3.22 and 2.55 points across six Qwen models, with the clearest statistical evidence in the 30-second setting. Cross-captioning shows that performance depends jointly on caption quality and downstream text reasoning. Caption memory also amortizes repeated visual processing across multiple queries but may omit subtle visual details. We therefore introduce a retrieve-and-verify harness that uses captions as a semantic index and selectively revisits sparse visual evidence, improving most open-weight models. Together, the task, benchmark, controlled analyses, and harness support that caption memory is an effective approach to long-form egocentric episodic reasoning under current VLM constraints.

2 Related Work

2.1 Egocentric Datasets and Benchmarks

Egocentric vision has progressed from clip-level action recognition Damen et al. (2022) to large-scale datasets Grauman et al. (2022); Grauman et al. (2025); Engel et al. (2023); Yang et al. (2025a). Existing assistant-oriented datasets and QA benchmarks, including QAEGO4D Bärmann and Waibel (2022), HoloAssist Wang et al. (2023), and EgoSchema Mangalam et al. (2023), mainly focus on short or minute-scale video contexts. More recent egocentric QA efforts, such as MM-Ego Ye et al. (2025) and EgoGazeVQA Peng et al. (2025), scale annotation with automatic or MLLM-generated QA pairs, but may inherit generation-induced language bias. In contrast, CapMem provides a human-annotated, long-form video benchmark.

2.2 Episodic Memory Tasks

Episodic memory is central to wearable assistants Engel et al. (2023); Grauman et al. (2022). Ego4D established this paradigm with memory-oriented tasks such as NLQ, VQ, and MQ Grauman et al. (2022). Subsequent work extends this line to egocentric episodic-memory QA Bärmann and Waibel (2022), narration-supervised query localization Ramakrishnan et al. (2023), and cross-view memory reasoning Grauman et al. (2025); Liu et al. (2026). While these efforts mainly focus on visual retrieval or direct VideoQA, we evaluate whether caption-based memory can support long-form egocentric QA (Appendix A).

2.3 Video Caption and Harness Engineering

Recent work tackles the memory and token limits of long-horizon videos by abstracting visual content into textual representations. SiLVR Zhang et al. (2026a) and MR. Video Pang and Wang (2025) use dense clip-level captions for long-context reasoning with LLMs, while LLMVS Lee et al. (2025) applies frame-level captions to LLM-based video summarization. However, the effectiveness of caption-based reasoning in continuous, long-form egocentric settings remains underexplored. Our task and benchmark fill this gap, providing a dedicated testbed for, to our knowledge, the first systematic study of how harness design can enhance caption-centric reasoning.

3 Task and Evaluation Setup

To evaluate the capability of VLMs to use self-generated captions as egocentric episodic memory, we introduce a QA benchmark, whose evaluation framework features three distinct task settings, a two-stage evaluation pipeline, and a five-dimensional analysis of the capabilities of models.

3.1 Task Formulation

We define Episodic Memory Video Caption QA as a multiple-choice QA task requiring episodic recall, where models answer solely by self-generated timestamped captions from an egocentric video. To assess this capability, we use direct VideoQA as the primary baseline and CleanQA as a safeguard against parametric priors:

CleanQA. The model receives only textual questions and options without visual input, serving as a blind baseline to quantify parametric priors and rule out blind guessing from pre-trained knowledge.
Direct VideoQA. The model answers each question directly from uniformly sampled video frames through its native visual interface, without constructing or using an intermediate textual memory. For brevity, we refer to Direct VideoQA as VideoQA throughout the paper.
CaptionQA. The evaluated model first converts temporally segmented video frames into timestamped captions. It then answers each question using only this textual memory, without access to visual inputs or prior conversation history.

3.2 Benchmark Scope and Statistics

A benchmark for evaluating caption-based memory in downstream real-world applications should reflect both the diversity of authentic human routines and the temporal demands of long-horizon video understanding. To this end, CapMem comprises 75 egocentric videos totaling 33.7 hours, spanning four high-value domains: Daily Life & Domestic, Outdoor & Mobility, Work & Procedural, and Pet Related. Collected across diverse cultural backgrounds, the benchmark covers 16 precise professional and domestic scenarios, ranging from industrial machinery repair to outdoor activities, sports, and human–pet interaction (Figure 1). Beyond scenario coverage, the benchmark is intentionally length-challenging: 37.3% of the videos are short clips ( $\leq 20$ mins) for verifying foundational video understanding, whereas 62.7% are long videos ( $>20$ mins, up to 1 hour) to stress episodic memory over extended contexts. This emphasis is further reflected in the QA distribution, where 79.2% of all questions are assigned to the long videos (Table 2). In total, the benchmark contains 1,000 multiple-choice questions, averaging 13.3 questions per video. Each question includes four options, one ground-truth answer, and an annotated evidence timestamp indicating where the supporting visual evidence occurs in the video. Table 1 summarizes the distribution of questions across the four domains and QA taxonomy axes (Sec. 4.2), while Table 2 reports the distribution of video duration, evidence location, and fine-grained visual tags. We categorize evidence by its relative temporal position in the video as beginning (0–20%), middle (20–80%), or end (80–100%). The fine-grained visual tags (Sec. 4.2) characterize the visual elements incorporated into the questions and answer options to increase perceptual difficulty.

Table 1: QA statistics. Queries per domain are categorized along two orthogonal axes: subject of inquiry (EGO/EXO) and query intent (4W1H).

Category	#QA	(EGO \| EXO)	WHO	WHAT	WHERE	WHEN	HOW
Daily Life	404	(93 \| 311)	20	203	54	40	87
Outdoor	298	(49 \| 249)	11	151	19	44	73
Pet	115	(32 \| 83)	1	56	16	25	17
Work	183	(50 \| 133)	14	104	8	18	39
Total	1000	(224 \| 776)	46	514	97	127	216

3.3 Evaluation Framework and Dimensions

We design a two-stage evaluation framework:

Stage 1: Baseline Safeguard. CleanQA is used as a gating test for semantic bias and data contamination. Only models performing near chance level (15%–35%) proceed to the main evaluation. To preserve the validity and fairness of downstream comparisons, models falling outside this range are excluded from the subsequent VideoQA–CaptionQA evaluation. Because guessable questions were filtered during benchmark construction (Sec. 4.3), unusually high performance at this stage may indicate training-data contamination.
Stage 2: Comparative Assessment. Models that pass the initial gate are evaluated under VideoQA and CaptionQA, as defined in Sec. 3.1. The comparison measures how well self-generated textual memory can substitute for direct visual input in egocentric episodic memory tasks. We analyze the performance along the following dimensions:

Overall QA Accuracy. Accuracy over the full question set.
Computational Cost. Average per-question token consumption.
Fine-grained Visual Accuracy. Accuracy reported separately for the four tag groups.
Evidence-location QA Accuracy. Accuracy across the evidence-location groups defined in Sec. 3.2, testing sensitivity to the “lost-in-the-middle” phenomenon.
Duration-wise QA Accuracy. Accuracy across $\leq 20$ -minute and $>20$ -minute video subsets.

Table 2: Dataset statistics. Fine-grained tags are multi-label.

Aspect	Category	Count
Video duration	$\leq 20$ mins videos	208 Qs (20.8%)
Video duration	$>20$ mins videos	792 Qs (79.2%)
Evidence location	Begin	348 Qs (34.8%)
	Middle	507 Qs (50.7%)
	End	145 Qs (14.5%)
Fine-grained tag	Object	962 Qs (96.2%)
	Attribute	708 Qs (70.8%)
	Spatial	581 Qs (58.1%)
	Action	844 Qs (84.4%)

4 Benchmark Construction

4.1 Video Collection

Our video pool is drawn from seven egocentric datasets: Ego4D Grauman et al. (2022), EgoLife Yang et al. (2025a), EgoPet Bar et al. (2024), EPIC-KITCHENS Damen et al. (2018), HoloAssist Wang et al. (2023), ENIGMA-51 Ragusa et al. (2024), and CASTLE2024 Rossetto et al. (2025). We release the CapMem annotations and associated metadata for non-commercial research purposes only, without redistributing the source videos or frames; dataset-specific licensing conditions are detailed in Appendix G.

4.2 Question Generation

We generate QA pairs through a taxonomy-guided annotation pipeline. The taxonomy is defined along two axes: the subject of inquiry, distinguishing questions about the camera wearer and their actions or attributes (Ego) from questions about external entities in view (Exo), and the query intent, organized by the 4W1H forms (Who, What, Where, When, How). Based on this taxonomy, we design 50 question templates covering common episodic reasoning patterns, including object counting, state transitions, and ego–exo interactions (Appendix B). For each video, human annotators select suitable templates and instantiate them with concrete entities or events from the video. To increase fine-grained discrimination difficulty, both questions and answer options are enriched with fine-grained cues spanning object identity, visual attributes, spatial relations, and actions, while the options are constructed with distractors that require models to distinguish subtle visual details.

4.3 Question Filtering

After the generation of QA pairs, we adopt a two-stage filtering pipeline to discard flawed data. More quality control detail can be found in Appendix E.

Step 1: Text-Only Filtering. VLMs may answer questions without visual reasoning by exploiting linguistic priors, commonsense knowledge, or option-level shortcuts Li et al. (2023); Liu et al. (2024c); Zhang et al. (2026b). We therefore run a text-only blind test with a four-model ensemble (GPT-5.2, Gemini 3, Qwen3.5, and InternVL). Questions answered correctly by at least three models are considered visually bypassable and removed. Residual biases are further controlled by the CleanQA gate in our evaluation framework.
Step 2: Visual Solvability Filtering. Human validators verify each query using only the source video. Queries with ambiguous or subjective visual evidence are discarded.

5 Experiment and Discussion

5.1 Experiment Setting

Our evaluation encompasses 12 models, including representative proprietary models (Gemini 3 Flash Google DeepMind (2025) and GPT-5.2 OpenAI (2025)) and a comprehensive suite of open-weight models spanning various parameter scales: InternVL3.5 Wang et al. (2025), Qwen3-VL Bai et al. (2025), and Qwen3.5 Qwen Team (2026). We test the models under distinct settings:

CleanQA. We run strictly single-turn, zero-shot inference. Each query is processed independently with an empty context window, clearing the KV-cache and conversation history to prevent cross-query contamination.
VideoQA. We extract frames at 1 FPS and uniformly subsample when the sequence exceeds context limits. Sequences are capped at 128 frames for InternVL3.5 and 768 frames for Qwen; for cross-model comparability, closed-source APIs are also capped at 768 frames. Models process visual inputs by their native modality interfaces in a single-turn manner. Due to closed-source API budget constraints, Gemini 3 Flash VideoQA is taken on a 786-question subset; all other reported settings use the full benchmark. This setting represents the standard direct video-input interface of general-purpose VLMs under their supported frame budgets. Our goal is to evaluate CaptionQA against this standard baseline, rather than claim superiority over all visual-reasoning pipelines.
Full-coverage CaptionQA. Models deterministically caption the entire 1-FPS frame sequence in consecutive 30- or 60-second windows (temperature = 0), without a video-level frame cap. They concatenate the captions with timestamps as textual episodic memory and answer each query using only this text. Visual inputs are unavailable at QA time, and the captioning and QA prompts are shared across models (Appendix C)).

Frame-aligned Control. Because CaptionQA may process more frames than VideoQA, we conduct a frame-aligned control on six Qwen models to determine whether its gains arise from caption-based memory rather than greater visual coverage. Both settings use all 1-FPS frames up to 768, or the same 768 uniformly sampled frames otherwise; CaptionQA groups the selected frames by timestamp into 30- or 60-second windows to construct the caption for QA.

Cross-captioning Control. Because self-caption QA jointly reflects caption-generation quality and text-QA capacity, we conduct a cross-captioning control to decouple these two factors. Captions from six Qwen models are paired with Qwen3.5-2B and Qwen3.5-27B as QA models. We evaluate every captioner–QA pairing using the same frames uniformly sampled over the full videos and capped at 768 per video.

Model	Clean	Cap30	Cap60	Video
InternVL3.5-8B	29.3	35.8	34.3	39.4
InternVL3.5-38B	28.4	42.1	39.8	42.8
Qwen3-VL-2B	27.0	36.3	35.2	35.4
Qwen3-VL-4B	26.9	42.3	40.3	37.1
Qwen3-VL-8B	25.8	39.6	39.0	37.2
Qwen3-VL-32B	26.1	47.7	44.9	42.8
Qwen3.5-2B	26.6	37.2	34.7	37.2
Qwen3.5-4B	27.4	41.8	39.6	39.6
Qwen3.5-9B	29.4	42.8	43.7	42.3
Qwen3.5-27B	30.9	49.3	46.9	45.0
GPT-5.2	25.6	52.0	51.4	37.3
Gemini 3 Flash	29.5	55.7	53.2	58.8^∗

Table 3: Overall model accuracy across evaluation formats. ^∗ means partial test due to API budget.

5.2 Main Results

We evaluate full-coverage CaptionQA for overall accuracy, video duration, amortized visual processing, and evidence position, and use frame-aligned and cross-captioning controls to decouple the effects of visual coverage, caption quality, and text-QA capacity.

Full-coverage CaptionQA is competitive under current VLM frame limits.

Table 3 reports the end-to-end results of full-coverage CaptionQA. CleanQA remains near chance (25.6%–30.9%), providing a sanity check. With each model’s better caption window, full-coverage CaptionQA outperforms VideoQA for 8 of the 12 models. For example, 30-second CaptionQA improves Qwen3-VL-32B from 42.8% to 47.7%, Qwen3.5-27B from 45.0% to 49.3%, and GPT-5.2 from 37.3% to 52.0%. The 30-second setting also generally outperforms the 60-second setting, suggesting that shorter windows better preserve local details. However, VideoQA remains stronger for both InternVL3.5 models and Gemini 3 Flash. Overall, caption memory provides an effective alternative under current frame and context constraints, but does not universally outperform direct visual input.

Full-coverage CaptionQA gains concentrate on videos longer than 20 minutes.

Figure 3 reveals a clear duration-dependent pattern . On videos of at most 20 minutes, CaptionQA generally trails direct VideoQA, indicating that captioning may discard useful visual details when the visual context is not long. On videos longer than 20 minutes, the gaps become positive for most models, reaching +7.2 points for Qwen3-VL-32B with 30-second captions and +15.2 points for GPT-5.2 with 60-second captions. Thus, full-coverage caption memory is particularly useful for long-video reasoning under current VLM frame and context limits, rather than uniformly beneficial across durations.

CaptionQA retains a mean advantage after frame alignment, particularly on long videos.

Table 4 and Figure 4 control for the broader visual coverage of full-coverage CaptionQA. Relative to full coverage, frame alignment lowers mean accuracy by 0.80 points at 30 seconds and leaves it nearly unchanged at 60 seconds (+0.17 points). The resulting mean gaps over direct VideoQA are +2.12 and +0.92 points on the full benchmark, increasing to +3.22 and +2.55 points on the long videos. We estimate uncertainty from question-weighted video-level gaps by 10,000 duration-stratified video-cluster bootstrap samples (Table 5, detail in Appendix F.5). The 30s confidence interval is positive on long videos, $[+0.45,+6.07]$ , whereas the 60s interval is negative on short videos, $[-9.17,-1.29]$ , favoring direct VideoQA. The clearest controlled evidence therefore appears in the 30-second long-video setting. Broader visual coverage contributes to some outcomes but does not fully explain the long-video advantage.

Figure 3: CaptionQA–VideoQA gap by video length before frame alignment.

Caption quality and text-QA capacity both affect CaptionQA.

Table 6 decouples the two capabilities in CaptionQA task. With the captioner fixed, Qwen3.5-27B outperforms Qwen3.5-2B by an average of 8.1 points at 30 seconds and 6.7 points at 60 seconds, showing that a stronger QA model can better use the same captions. With the QA model fixed, varying the captioner makes a max–min accuracy spread of 2.2–6.1 points. Self-captioning is not always optimal: Qwen3.5-2B improves from 36.2% with its own 30-second captions to 39.8% with captions from Qwen3-VL-32B. These results show CaptionQA depends on both memory construction and downstream reasoning.

Model	VideoQA	Cap30		Cap60
		Orig.	Align.	Orig.	Align.
Qwen3-VL-2B	35.4	36.3	38.5	35.2	36.7
Qwen3-VL-4B	37.1	42.3	40.6	40.3	38.2
Qwen3-VL-32B	42.8	47.7	46.3	44.9	43.8
Qwen3.5-2B	37.2	37.2	36.2	34.7	38.3
Qwen3.5-4B	39.6	41.8	40.1	39.6	38.4
Qwen3.5-27B	45.0	49.3	48.1	46.9	47.2
Mean	39.5	42.4	41.6	40.3	40.4

Table 4: Overall accuracy (%) before/after frame alignment.

Figure 4: CaptionQA–VideoQA gap by video length after frame alignment.

Scope	Int.	Mean	95% CI
Full	30s	$+2.12$	$[-0.27,+4.60]$
Full	60s	$+0.92$	$[-1.38,+3.18]$
$>20$ min	30s	$+3.22$	$\mathbf{[+0.45,+6.07]}$
$>20$ min	60s	$+2.55$	$[-0.11,+5.23]$
$\leq 20$ min	30s	$-2.08$	$[-6.35,+2.24]$
$\leq 20$ min	60s	$-5.29$	$\mathbf{[-9.17,-1.29]}$

Table 5: Frame-aligned mean gaps and 95% video-cluster bootstrap confidence intervals.

	30s		60s
Captioner	2B QA	27B QA	2B QA	27B QA
Qwen3-VL-2B	37.6	45.1	36.8	41.8
Qwen3-VL-4B	37.4	46.5	38.0	44.3
Qwen3-VL-32B	39.8	48.9	37.4	46.6
Qwen3.5-2B	36.2	42.8	38.3	42.5
Qwen3.5-4B	38.1	45.2	36.4	43.1
Qwen3.5-27B	38.8	48.1	38.6	47.2

Table 6: Cross-captioning accuracy (%) on the full benchmark; self-captioning results are underlined.

Caption memory amortizes visual processing across queries.

Table 7 illustrates the reuse of 30-second caption memory. For $N$ questions sharing one memory, $C_{\mathrm{Cap}}(N)=(T_{\mathrm{caption}}+\sum_{i=1}^{N}T_{\mathrm{QA},i})/N$ . CaptionQA uses logged construction and QA tokens, whereas VideoQA uses an analytical visual-patch proxy (selected frames $\times 1{,}024$ plus text). Under this accounting, Qwen CaptionQA decreases from 330k–335k per question at 1Q/video to 14k–20k at 32Q/video. The advantage is reusable visual processing: even constructing both caption granularities requires 101,814 frame inputs, versus 725,973 for Direct VideoQA, reducing repeated visual-frame processing by $7.1\times$ (Table 14). Realized token and runtime costs remain processor-dependent (Appendix F.6).

Model	CaptionQA30s						VideoQA
	1Q	2Q	4Q	8Q	16Q	32Q
InternVL3.5-8B	429	217	111	58	32	18	131
InternVL3.5-38B	429	217	111	58	32	18	131
Qwen3-VL-2B	333	170	89	48	28	18	743
Qwen3-VL-4B	333	171	89	48	28	18	743
Qwen3-VL-8B	332	169	88	47	27	16	743
Qwen3-VL-32B	335	172	91	50	30	20	743
Qwen3.5-2B	330	167	85	45	24	14	743
Qwen3.5-4B	333	170	88	48	27	17	743
Qwen3.5-9B	334	171	89	48	28	18	743
Qwen3.5-27B	334	171	90	49	29	18	743
GPT-5.2	394	200	104	56	31	19	743
Gemini 3 Flash	1775	891	450	228	118	63	760^∗

Table 7: Amortized per-question workload (k token units) for 30-second CaptionQA: logged input tokens for CaptionQA and an analytical visual-patch proxy for VideoQA.

Caption memory can reduce middle-position degradation

Figure 5 shows lower VideoQA accuracy for middle-position evidence in several models, consistent with the “lost-in-the-middle” phenomenon. Full-coverage CaptionQA produces a flatter three-bin accuracy profile for several models. For GPT-5.2, it largely removes the observed middle-position drop, while 30-second CaptionQA both reduces the drop and improves accuracy for Qwen3.5-27B. Some InternVL models exhibit flatter profiles despite limited gains in absolute accuracy. Although this pattern is not universal and does not establish a causal mechanism, it suggests that caption memory may make temporally intermediate evidence easier to retrieve for some models. Fine-grained results are provided in Appendix F.2.

5.3 Harness Engineering

Although captions are efficient for long-video reasoning, they are a lossy abstraction of the visual stream. Our harness uses captions to localize candidate moments and consults sparse frames only when they can reliably correct a caption-based answer. It has three stages: first, a deterministic weighted retrieval rule selects relevant timestamped caption windows using BM25, hash-based cosine similarity, word overlap, and temporal hints, from which the model produces a retrieved-caption answer. Second, sparse frames from the selected intervals are used to verify which option is visually supported. Third, the baseline CaptionQA answer is kept by default and overridden only when verification returns a valid option with strong support and no explicit clock-time cue, since sparse frames may miss the referenced moment (Appendix D).

Figure 6: Accuracy of open-weight models before and after retrieve-and-verify harnessing.

The harness enhances textual memory baselines.

Figure 6 shows consistent gains across all open-weight models and both caption settings. The effect is strongest for smaller models, with Qwen3.5-2B improving by +4.8 at 30s and +5.3 at 60s, while Qwen3.5-27B reaches 49.4% at 30s. These results show that captions provide useful temporal indices, while sparse frames recover visual details missed by textual memory.

6 Conclusion

We introduced CapMem, a human-annotated benchmark for caption-based egocentric episodic memory. Across 12 VLMs, CaptionQA gains concentrate on long videos; under matched frames, the mean gain across 6 selected Qwen models remains positive. Retrieve-and-verify improves most open-weight models, supporting caption memory as an effective approach under current VLM constraints.

Limitations

CapMem contains 75 public egocentric videos and 1,000 QA pairs, providing diverse long-form coverage while necessarily covering only a subset of real-world wearable scenarios. Although videos were selected for scenario coverage rather than source-dataset balance, Ego4D contributes 56 of 75 videos and 742 of 1,000 questions, so aggregate results may partly reflect its capture distribution. Our primary visual comparison is limited to standard Direct VideoQA under each model’s supported frame budget; we do not evaluate stronger chunked, retrieval-first, or hierarchical visual-memory pipelines. The frame-aligned and cross-captioning controls are restricted to Qwen models, and their controlled effects are heterogeneous, with several confidence intervals including zero, limiting generalization across model families. CaptionQA further depends on caption quality, as missed objects, attributes, or short-lived actions may propagate to downstream QA. Finally, our retrieve-and-verify harness is evaluated as an inference-time strategy rather than an end-to-end trained memory system. Future work should expand the benchmark scale and evaluate broader visual-memory baselines and model families.

Ethical Considerations

CapMem is constructed from previously released egocentric video datasets and does not involve collecting new personal recordings. Our annotations are limited to observable events, objects, actions, and spatiotemporal evidence, and avoid inferring identities, sensitive attributes, or personal intentions. Nevertheless, source videos may contain identifiable people and private environments. We therefore do not redistribute source videos or frames; users must obtain the source media separately and comply with the original datasets’ licenses, access conditions, and privacy requirements. CapMem releases only its annotations and associated metadata, intended for non-commercial research use, under the license specified in Appendix G. This license does not supersede any applicable restrictions imposed by the source datasets. Deployment of caption-based memory systems additionally requires appropriate consent, data minimization, and access control.

References

Bai et al. (2025) S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu Qwen3-VL technical report. arXiv preprint arXiv:2511.21631. External Links: 2511.21631, Link Cited by: §5.1.
Bar et al. (2024) A. Bar, A. Bakhtiar, D. Tran, A. Loquercio, J. Rajasegaran, Y. LeCun, A. Globerson, and T. Darrell EgoPet: egomotion and interaction data from an animal’s perspective. In Computer Vision – ECCV 2024, pp. 377–394. External Links: Document, Link Cited by: §4.1.
Bärmann and Waibel (2022) L. Bärmann and A. Waibel Where did i leave my keys? – episodic-memory-based question answering on egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1560–1568. External Links: Document, Link Cited by: Appendix A, §1, §2.1, §2.2.
Bell and Gemmell (2009) G. Bell and J. Gemmell Total recall: how the e-memory revolution will change everything. Dutton, New York. Cited by: Appendix A.
Chen et al. (2024) L. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, B. Lin, Z. Tang, L. Yuan, Y. Qiao, D. Lin, F. Zhao, and J. Wang ShareGPT4Video: improving video understanding and generation with better captions. In Advances in Neural Information Processing Systems, Vol. 37, pp. 19472–19495. External Links: Document, Link Cited by: §1.
Cheng et al. (2024) S. Cheng, Z. Guo, J. Wu, K. Fang, P. Li, H. Liu, and Y. Liu EgoThink: evaluating first-person perspective thinking capability of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14291–14302. External Links: Document, Link Cited by: §1.
Cong et al. (2025) L. Cong, D. Smerkous, X. Wang, D. Yin, Z. Zhang, R. Jin, Y. Wang, M. Gerasimiuk, R. K. Dinesh, A. Smerkous, L. Shi, J. Zheng, I. Lam, X. Wu, S. Liu, P. Li, Y. Zhu, N. Zhao, M. Parakh, S. Serrao, I. A. Mohammad, C. Chen, X. Xie, T. Chen, D. Weinstein, G. Barbone, B. Caglar, J. B. Sunwoo, F. Li, J. Deng, J. C. Wu, S. Wu, and M. Wang LabOS: the AI-XR co-scientist that sees and works with humans. arXiv preprint arXiv:2510.14861. External Links: 2510.14861, Link Cited by: Appendix A, §1.
Conway and Pleydell-Pearce (2000) M. A. Conway and C. W. Pleydell-Pearce The construction of autobiographical memories in the self-memory system. Psychological Review 107 (2), pp. 261–288. External Links: Document, Link Cited by: Appendix A.
Damen et al. (2018) D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray Scaling egocentric vision: the EPIC-KITCHENS dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 720–736. Cited by: Appendix A, §4.1.
Damen et al. (2022) D. Damen, H. Doughty, G. M. Farinella, A. Furnari, E. Kazakos, J. Ma, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100. International Journal of Computer Vision 130 (1), pp. 33–55. External Links: Document, Link Cited by: Appendix A, §2.1.
Datta et al. (2022) S. Datta, S. Dharur, V. Cartillier, R. Desai, M. Khanna, D. Batra, and D. Parikh Episodic memory question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19097–19106. External Links: Document, Link Cited by: Appendix A.
Doherty et al. (2012) A. R. Doherty, K. Pauly-Takacs, N. Caprani, C. Gurrin, C. J. A. Moulin, N. E. O’Connor, and A. F. Smeaton Experiences of aiding autobiographical memory using the SenseCam. Human–Computer Interaction 27 (1–2), pp. 151–174. External Links: Document, Link Cited by: Appendix A.
Engel et al. (2023) J. Engel, K. Somasundaram, M. Goesele, A. Sun, A. Gamino, A. Turner, A. Talattof, A. Yuan, B. Souti, B. Meredith, C. Peng, C. Sweeney, C. Wilson, D. Barnes, D. DeTone, D. Caruso, D. Valleroy, D. Ginjupalli, D. Frost, E. Miller, E. Mueggler, E. Oleinik, F. Zhang, G. Somasundaram, G. Solaira, H. Lanaras, H. Howard-Jenkins, H. Tang, H. J. Kim, J. Rivera, J. Luo, J. Dong, J. Straub, K. Bailey, K. Eckenhoff, L. Ma, L. Pesqueira, M. Schwesinger, M. Monge, N. Yang, N. Charron, N. Raina, O. Parkhi, P. Borschowa, P. Moulon, P. Gupta, R. Mur-Artal, R. Pennington, S. Kulkarni, S. Miglani, S. Gondi, S. Solanki, S. Diener, S. Cheng, S. Green, S. Saarinen, S. Patra, T. Mourikis, T. Whelan, T. Singh, V. Balntas, V. Baiyya, W. Dreewes, X. Pan, Y. Lou, Y. Zhao, Y. Mansour, Y. Zou, Z. Lv, Z. Wang, M. Yan, C. Ren, R. D. Nardi, and R. Newcombe Project aria: a new tool for egocentric multi-modal AI research. arXiv preprint arXiv:2308.13561. External Links: 2308.13561, Link Cited by: Appendix A, §2.1, §2.2.
Fei et al. (2026) J. Fei, J. Chen, Z. Liu, Y. Xiong, C. Zhou, W. Wen, J. Han, M. Zhuge, S. Suri, Q. Qian, S. Liu, L. Wu, R. Krishnamoorthi, V. Chandra, M. Elhoseiny, and C. Zhu Small vision-language models are smart compressors for long video understanding. arXiv preprint arXiv:2604.08120. External Links: 2604.08120, Link Cited by: §1.
Fu et al. (2024) X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W. Ma, and R. Krishna BLINK: multimodal large language models can see but not perceive. In Computer Vision – ECCV 2024, pp. 148–166. External Links: Document, Link Cited by: §1.
Gemmell et al. (2006) J. Gemmell, G. Bell, R. Lueder, S. Drucker, and C. Wong MyLifeBits: a personal database for everything. Communications of the ACM 49 (1), pp. 88–95. External Links: Document, Link Cited by: Appendix A.
Google DeepMind (2025) Google DeepMind Gemini 3 flash. Note: https://ai.google.dev/gemini-api/docs/modelsAccessed: 2026-06-17 Cited by: §5.1.
Grauman et al. (2022) K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Z. Zhao, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, C. Fuegen, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik Ego4D: around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18995–19012. External Links: Document, Link Cited by: Appendix A, §1, §1, §2.1, §2.2, §4.1.
Grauman et al. (2025) K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, E. Byrne, Z. Chavis, J. Chen, F. Cheng, F. Chu, S. Crane, A. Dasgupta, J. Dong, M. Escobar, C. Forigua, A. Gebreselasie, S. Haresh, J. Huang, M. M. Islam, S. Jain, R. Khirodkar, D. Kukreja, K. J. Liang, J. Liu, S. Majumder, Y. Mao, M. Martin, E. Mavroudi, T. Nagarajan, F. Ragusa, S. K. Ramakrishnan, L. Seminara, A. Somayazulu, Y. Song, S. Su, Z. Xue, E. Zhang, J. Zhang, A. Castillo, C. Chen, X. Fu, R. Furuta, C. González, P. Gupta, J. Hu, Y. Huang, Y. Huang, W. Khoo, A. Kumar, R. Kuo, S. Lakhavani, M. Liu, M. Luo, Z. Luo, B. Meredith, A. Miller, O. Oguntola, X. Pan, P. Peng, S. Pramanick, M. Ramazanova, F. Ryan, W. Shan, K. Somasundaram, C. Song, A. Southerland, M. Tateno, H. Wang, Y. Wang, T. Yagi, M. Yan, X. Yang, Z. Yu, S. C. Zha, C. Zhao, Z. Zhao, Z. Zhu, J. Zhuo, P. Arbeláez, G. Bertasius, D. Crandall, D. Damen, J. Engel, G. M. Farinella, A. Furnari, B. Ghanem, J. Hoffman, C. V. Jawahar, R. Newcombe, H. S. Park, J. M. Rehg, Y. Sato, M. Savva, J. Shi, M. Z. Shou, and M. Wray Ego-Exo4D: understanding skilled human activity from first- and third-person perspectives. International Journal of Computer Vision 133 (12), pp. 8356–8435. External Links: Document, Link Cited by: Appendix A, §1, §2.1, §2.2.
Hodges et al. (2006) S. Hodges, L. Williams, E. Berry, S. Izadi, J. Srinivasan, A. Butler, G. Smyth, N. Kapur, and K. Wood SenseCam: a retrospective memory aid. In UbiComp 2006: Ubiquitous Computing, Lecture Notes in Computer Science, Vol. 4206, pp. 177–193. External Links: Document, Link Cited by: Appendix A.
Lee et al. (2025) M. J. Lee, D. Gong, and M. Cho Video summarization with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18981–18991. External Links: Document, Link Cited by: §2.3.
Li et al. (2023) Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292–305. External Links: Document, Link Cited by: 1st item.
Li et al. (2024) Y. Li, X. Chen, B. Hu, L. Wang, H. Shi, and M. Zhang VideoVista: a versatile benchmark for video understanding and reasoning. arXiv preprint arXiv:2406.11303. External Links: 2406.11303, Link Cited by: §1.
Liu et al. (2024a) H. Liu, W. Yan, M. Zaharia, and P. Abbeel World model on million-length video and language with blockwise RingAttention. arXiv preprint arXiv:2402.08268. External Links: 2402.08268, Link Cited by: §1.
Liu et al. (2024b) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, pp. 157–173. External Links: Document, Link Cited by: §1.
Liu et al. (2026) R. Liu, J. Zheng, Y. Chen, D. Wen, S. Quan, C. Wu, J. Zhang, K. Yang, K. Peng, and R. Stiefelhagen EgoExoMem: cross-view memory reasoning over synchronized egocentric and exocentric videos. arXiv preprint arXiv:2605.18734. External Links: 2605.18734, Link Cited by: §2.2.
Liu et al. (2024c) Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin MMBench: is your multi-modal model an all-around player?. In Computer Vision – ECCV 2024, pp. 216–233. External Links: Document, Link Cited by: 1st item.
Lu et al. (2025) Z. Lu, Z. Yin, M. He, Z. Wang, Z. Liu, Z. Wang, and K. Hu B-VLLM: a vision large language model with balanced spatio-temporal tokens. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 24549–24558. External Links: Document, Link Cited by: §1.
Maaz et al. (2024) M. Maaz, H. Rasheed, S. Khan, and F. S. Khan Video-chatgpt: towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12585–12602. External Links: Document, Link Cited by: §1.
Majumdar et al. (2024) A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, M. Henaff, S. Silwal, P. Mcvay, O. Maksymets, S. Arnaud, K. Yadav, Q. Li, B. Newman, M. Sharma, V. Berges, S. Zhang, P. Agrawal, Y. Bisk, D. Batra, M. Kalakrishnan, F. Meier, C. Paxton, A. Sax, and A. Rajeswaran OpenEQA: embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16488–16498. External Links: Document, Link Cited by: §1.
Mangalam et al. (2023) K. Mangalam, R. Akshulakov, and J. Malik EgoSchema: a diagnostic benchmark for very long-form video language understanding. In Advances in Neural Information Processing Systems, Vol. 36, pp. 46212–46244. External Links: Document, Link Cited by: Appendix A, §1, §2.1.
OpenAI (2025) OpenAI GPT-5.2. Note: https://openai.com/index/introducing-gpt-5-2/Accessed: 2026-06-17 Cited by: §5.1.
Pang and Wang (2025) Z. Pang and Y. Wang MR. Video: MapReduce as an effective principle for long video understanding. In Advances in Neural Information Processing Systems, Vol. 38, pp. 7428–7460. External Links: Document, Link Cited by: §2.3.
Peng et al. (2025) T. Peng, J. Hua, M. Liu, and F. Lu In the eye of mllm: benchmarking egocentric video intent understanding with gaze-guided prompting. In Advances in Neural Information Processing Systems, Vol. 38, pp. 121404–121424. Note: Datasets and Benchmarks Track External Links: Document, Link Cited by: §2.1.
Qwen Team (2026) Qwen Team Qwen3.5. Note: https://huggingface.co/QwenAccessed: 2026-06-17 Cited by: §5.1.
Ragusa et al. (2024) F. Ragusa, R. Leonardi, M. Mazzamuto, C. Bonanno, R. Scavo, A. Furnari, and G. M. Farinella ENIGMA-51: towards a fine-grained understanding of human behavior in industrial scenarios. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 7065–7075. External Links: Document, Link Cited by: §4.1.
Ramakrishnan et al. (2023) S. K. Ramakrishnan, Z. Al-Halah, and K. Grauman NaQ: leveraging narrations as queries to supervise episodic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18994–19004. External Links: Document, Link Cited by: Appendix A, §2.2.
Rossetto et al. (2025) L. Rossetto, W. Bailer, D. Dang-Nguyen, G. Healy, B. Þ. Jónsson, O. Kongmeesub, H. Le, S. Rudinac, K. Schöffmann, F. Spiess, A. Tran, M. Tran, Q. Tran, and C. Gurrin The CASTLE 2024 dataset: advancing the art of multimodal understanding. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 12629–12635. External Links: Document, Link Cited by: §4.1.
Song et al. (2024) E. Song, W. Chai, G. Wang, Y. Zhang, H. Zhou, F. Wu, H. Chi, X. Guo, T. Ye, Y. Zhang, Y. Lu, J. Hwang, and G. Wang MovieChat: from dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18221–18232. External Links: Document, Link Cited by: §1.
Tang et al. (2025) X. Tang, J. Qiu, L. Xie, Y. Tian, J. Jiao, and Q. Ye Adaptive keyframe sampling for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29118–29128. External Links: Document, Link Cited by: §1.
Tian et al. (2025) X. Tian, S. Zou, Z. Yang, and J. Zhang Identifying and mitigating position bias of multi-image vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10599–10609. External Links: Document, Link Cited by: §1.
Tokmurziyev et al. (2026) I. Tokmurziyev, M. Altamirano Cabrera, M. H. Khan, Y. Mahmoud, and D. Tsetserukou LLM-Glasses: GenAI-driven glasses with haptic feedback for navigation of visually impaired people. In Companion Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction, pp. 994–998. External Links: Document, Link Cited by: Appendix A, §1.
Tulving and Thomson (1973) E. Tulving and D. M. Thomson Encoding specificity and retrieval processes in episodic memory. Psychological Review 80 (5), pp. 352–373. External Links: Document, Link Cited by: Appendix A.
Tulving (1972) E. Tulving Episodic and semantic memory. In Organization of Memory, E. Tulving and W. Donaldson (Eds.), pp. 381–403. Cited by: Appendix A.
Tulving (2002) E. Tulving Episodic memory: from mind to brain. Annual Review of Psychology 53 (1), pp. 1–25. External Links: Document, Link Cited by: Appendix A.
Wang et al. (2025) W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, Z. Wang, Z. Chen, H. Zhang, G. Yang, H. Wang, Q. Wei, J. Yin, W. Li, E. Cui, G. Chen, Z. Ding, C. Tian, Z. Wu, J. Xie, Z. Li, B. Yang, Y. Duan, X. Wang, Z. Hou, H. Hao, T. Zhang, S. Li, X. Zhao, H. Duan, N. Deng, B. Fu, Y. He, Y. Wang, C. He, B. Shi, J. He, Y. Xiong, H. Lv, L. Wu, W. Shao, K. Zhang, H. Deng, B. Qi, J. Ge, Q. Guo, W. Zhang, S. Zhang, M. Cao, J. Lin, K. Tang, J. Gao, H. Huang, Y. Gu, C. Lyu, H. Tang, R. Wang, H. Lv, W. Ouyang, L. Wang, M. Dou, X. Zhu, T. Lu, D. Lin, J. Dai, W. Su, B. Zhou, K. Chen, Y. Qiao, W. Wang, and G. Luo InternVL3.5: advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. External Links: 2508.18265, Link Cited by: §5.1.
Wang et al. (2023) X. Wang, T. Kwon, M. Rad, B. Pan, I. Chakraborty, S. Andrist, D. Bohus, A. Feniello, B. Tekin, F. V. Frujeri, N. Joshi, and M. Pollefeys HoloAssist: an egocentric human interaction dataset for interactive AI assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20270–20281. External Links: Document, Link Cited by: §2.1, §4.1.
Wang et al. (2026) Z. Wang, Y. Zhang, S. Yu, C. Zhang, Z. Zhao, J. Yoon, H. Lee, G. Bertasius, and M. Bansal EgoMemReason: a memory-driven reasoning benchmark for long-horizon egocentric video understanding. arXiv preprint arXiv:2605.09874. External Links: 2605.09874, Link Cited by: Appendix A.
Xing et al. (2026) L. Xing, X. Dong, Y. Zang, Y. Cao, J. Liang, Q. Huang, J. Wang, F. Wu, and D. Lin CapRL: stimulating dense image caption capabilities via reinforcement learning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1.
Yang et al. (2025a) J. Yang, S. Liu, H. Guo, Y. Dong, X. Zhang, S. Zhang, P. Wang, Z. Zhou, B. Xie, Z. Wang, B. Ouyang, Z. Lin, M. Cominelli, Z. Cai, Y. Zhang, P. Zhang, F. Hong, J. Widmer, F. Gringoli, L. Yang, B. Li, and Z. Liu EgoLife: towards egocentric life assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 28885–28900. External Links: Document, Link Cited by: §1, §2.1, §4.1.
Yang et al. (2025b) S. Yang, Y. Liu, B. Zhai, X. Sun, Z. Liu, E. Barsoum, M. Li, and C. Xu CaptionQA: is your caption as useful as the image itself?. arXiv preprint arXiv:2511.21025. External Links: 2511.21025, Link Cited by: §1, §1.
Ye et al. (2025) H. Ye, H. Zhang, E. Daxberger, L. Chen, Z. Lin, Y. Li, B. Zhang, H. You, D. Xu, Z. Gan, J. Lu, and Y. Yang MM-EGO: towards building egocentric multimodal LLMs for video QA. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2.1.
Zhang et al. (2026a) C. Zhang, Y. Lin, Z. Wang, M. Bansal, and G. Bertasius SiLVR: a simple language-based video reasoning framework. Transactions on Machine Learning Research. External Links: Link Cited by: §2.3.
Zhang et al. (2025) Y. Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li LLaVA-Video: video instruction tuning with synthetic data. Transactions on Machine Learning Research. External Links: Link Cited by: §1.
Zhang et al. (2026b) Y. Zhang, E. Hwang, H. Zhang, P. Du, Y. Jia, D. Jiang, X. He, S. Zhang, P. Nie, P. West, and K. R. Allen Watch before you answer: learning from visually grounded post-training. arXiv preprint arXiv:2604.05117. External Links: 2604.05117, Link Cited by: 1st item.

Appendix A An Introduction to Episodic Memory

The notion of episodic memory originates from cognitive psychology. Tulving introduced episodic memory as memory for personally experienced events anchored in specific temporal and spatial contexts, contrasting it with semantic memory, which captures general world knowledge independent of a particular episode Tulving (1972); Tulving (2002). Later theories further clarified that episodic memory is not simply a passive storage system but a constructive retrieval process. For instance, the encoding specificity principle argues that successful recall depends on whether retrieval cues overlap with the original encoded experience Tulving and Thomson (1973). Similarly, autobiographical memory research views human memories as a hierarchical structure, organized across lifetime periods, general events, and event-specific details Conway and Pleydell-Pearce (2000).

When bridged to multi-modal artificial intelligence, these cognitive concepts translate naturally into the engineering challenges of long egocentric video understanding. In this domain, a wearable assistant is rarely asked only to retrieve semantic knowledge—such as recognizing whether a generic cup or dog appears in a frame. Instead, it must act as an externalized episodic memory, answering situated questions about the user’s own experience: where a specific object was left, what action happened before another event, or which person interacted with the camera wearer. Because a continuous video stream contains dense, unrefined low-level visual observations, useful machine recall cannot rely on unstructured scanning. For example, the query “Where did I leave my keys?” may be answered only if the system can use object identity, location, action history, and temporal cues to retrieve a small span from a long, high-bandwidth visual record. Thus, establishing episodic memory in video understanding is fundamentally a problem of structured encoding (video-to-text translation), cue-driven retrieval (textual localization), and evidence-grounded answer generation (fine-grained visual verification).

Wearable memory and lifelogging.

The connection between episodic memory and wearable devices predates modern VLMs. Early lifelogging systems such as MyLifeBits aimed to create a searchable digital archive of a person’s documents, photos, audio, and daily activities Gemmell et al. (2006); Bell and Gemmell (2009). Wearable camera systems such as SenseCam Hodges et al. (2006) and later lifelogging devices explored the use of first-person visual records as memory aids, particularly for supporting autobiographical recall in users with memory impairment. Doherty et al. (2012). These systems revealed both the promise and the core difficulty of externalized memory: recording is increasingly easy, but retrieval is hard. A lifetime archive is valuable only if the user can recover the right episode at the right time.

Modern AR glasses and wearable sensing platforms revive this question at a larger scale. Project Aria, for example, frames future AR devices as always-available, context-aware systems that collect egocentric multimodal streams for personalized AI applications Engel et al. (2023). Recent wearable assistant applications, from navigation for visually impaired users to laboratory procedure assistance, also depend on persistent contextual memory rather than isolated frame recognition Tokmurziyev et al. (2026); Cong et al. (2025). In such settings, episodic memory is not an auxiliary capability; it is the mechanism that allows an assistant to connect present queries to past observations. Without episodic memory, the device can perceive the current scene but cannot answer what happened before, where an object was last seen, or how the user’s current situation relates to previous steps.

Episodic memory in egocentric video benchmarks.

Egocentric video research has gradually moved from short action recognition toward memory-centered understanding. Earlier egocentric datasets such as EPIC-KITCHENS emphasized fine-grained hand-object interactions and daily activities Damen et al. (2018); Damen et al. (2022). Ego4D marked a major shift by collecting thousands of hours of first-person video across diverse daily scenarios. Crucially, it was the first to formally introduce the cognitive concept of episodic memory into the computer vision community, explicitly defining benchmark tasks around past, present, and future understanding Grauman et al. (2022). Its episodic memory tasks, including natural language queries, visual queries, and moment queries, formalize the problem of locating or retrieving information from a camera wearer’s past experience. This design established egocentric episodic memory as a central challenge for first-person perception.

Subsequent work extended this direction in several ways. QAEGO4D, framed episodic-memory-based question answering on egocentric videos and highlighted the need for systems that can answer user-centric questions over long visual experience Bärmann and Waibel (2022). Episodic Memory Question Answering further studied memory representations for embodied agents and AR assistants in indoor environments, requiring models to ground answers in spatio-temporal scene memory Datta et al. (2022). NaQ showed that narrations can be repurposed as supervision for natural-language episodic memory queries, improving query localization in Ego4D Ramakrishnan et al. (2023). EgoSchema then pushed video-language understanding toward longer egocentric clips and diagnostic multiple-choice QA Mangalam et al. (2023). More recently, Ego-Exo4D introduced synchronized first- and third-person recordings of skilled activities, enabling research on memory and reasoning across viewpoints Grauman et al. (2025), while EgoMemReason studies memory-driven reasoning over ultra-long egocentric videos Wang et al. (2026).

Together, these works show that episodic memory has become a defining problem for egocentric AI. However, most existing benchmarks either focus on temporal localization, direct VideoQA, or specialized reasoning settings. They do not directly ask whether a compact textual memory, generated from the video itself, can replace raw visual input for long-horizon QA. This question is important because wearable devices face severe constraints: long egocentric streams are expensive to store, transmit, and process as visual tokens, while downstream users need fast and interpretable access to past events.

Why caption-based episodic memory matters.

Caption-based memory offers a natural bridge between psychological theories of retrieval and practical constraints of long video understanding. In cognitive terms, captions act as retrieval cues: they preserve discrete descriptions of objects, actions, locations, and timestamps that can later be matched to a user’s question. In systems terms, captions compress high-bandwidth visual streams into searchable textual logs, reducing the cost of querying hours of video and mitigating the long-context decay caused by severe visual-token overload. This is especially attractive for wearable assistants, where the device may need to support many downstream tasks, such as object finding, safety reminders, procedural guidance, navigation, health monitoring, and human-robot collaboration.

At the same time, captions are lossy. They may omit subtle attributes, spatial relations, transient gestures, or small objects that are visually present but not selected by the captioner. This creates a central trade-off for episodic memory systems: textual memory improves scalability and accessibility, but raw visual evidence remains necessary for fine-grained verification. This trade-off motivates our benchmark design. CapMem evaluates whether self-generated timestamped captions can serve as episodic memory for long egocentric QA, while retaining evidence timestamps and fine-grained visual tags to diagnose when captions succeed or fail. The retrieve-and-verify harness further reflects a practical memory strategy: use captions as a semantic index to identify likely moments, then selectively revisit sparse visual evidence when details matter. In this sense, CapMem connects the cognitive view of episodic memory as cue-driven recall with the engineering need for scalable, evidence-grounded wearable intelligence.

Appendix B Guiding Taxonomy and Question Template for QA Generation

A central design goal of CapMem is to evaluate the capability of VLMs to utilize textual captions as episodic memory for long egocentric videos in a way that is both comprehensive and diagnostically meaningful. To this end, to systematically assist human annotators in brainstorming and constructing high-quality QA pairs, we build our question generation pipeline around a guiding taxonomy with two orthogonal axes: subject of inquiry and query intent. The first axis distinguishes whether a question concerns the camera wearer and their actions, states, or attributes (Ego), or instead targets external entities visible in the scene (Exo). This distinction is important because egocentric memory naturally involves both self-centered and scene-centered recall. Many real-world assistant queries are inherently ego-centric, such as what the wearer picked up, where they placed an object, or what action they performed before a later event. At the same time, long-form first-person videos also contain rich exo-centric information about other people, objects, animals, tools, and environmental changes. Separating Ego from Exo therefore prevents the benchmark from collapsing into a purely self-action dataset and enables analysis of whether models fail differently when tracking the wearer versus the surrounding world.

The second axis organizes questions by 4W1H intent: Who, What, Where, When, and How. This axis is motivated by the observation that episodic memory queries are not homogeneous. Some questions ask for identity (Who), others for object or event content (What), spatial placement (Where), temporal order or timing (When), or process, state transition, and manner (How). These forms correspond to different retrieval demands. For example, “Where” questions emphasize spatial grounding, “When” questions emphasize temporal localization and sequence memory, and “How” questions often require integrating multiple observations into a coherent procedural or relational interpretation. By combining the Ego/Exo and 4W1H axes, the taxonomy provides human annotators with a structured cognitive space of question types that spans both who or what is being queried and what kind of recall operation is required. This design makes the benchmark more interpretable than a flat collection of questions, because errors can be attributed to specific memory dimensions rather than only to overall accuracy.

To operationalize this taxonomy for annotators, we design 50 standardized question templates, listed in Table 8, as scaffolds for manual annotation. The templates are not intended to rigidly constrain linguistic form; instead, they provide controlled structural patterns that encourage broad coverage of episodic reasoning behaviors. In practice, they capture recurring forms of memory queries that arise in long egocentric videos, including object counting, state changes, action destinations, ego–exo interactions, attribute comparison, spatial relations, and temporally ordered events. This template-based design provides several benefits. First, it improves coverage: annotators are explicitly guided to consider diverse memory phenomena rather than overproducing only the most obvious object or action questions. Second, it improves consistency: questions across different scenarios follow comparable structural principles, making benchmark difficulty less dependent on arbitrary annotator style. Third, it improves diagnostic value: because each template is tied to a known query form, downstream evaluation can more clearly relate performance differences to specific episodic reasoning skills.

The use of templates is particularly important for CapMem because our goal is not merely to test whether a model can answer generic video questions, but whether it can recover fine-grained and time-specific details from long first-person experience. Open-ended human annotation without structural guidance often drifts toward either overly easy questions, which can be solved from commonsense or coarse scene priors, or overly idiosyncratic questions, which are difficult to compare across videos. The 50-template design helps strike a balance between naturalness and control. Human annotators still instantiate each template with concrete entities, actions, and events from the source video, preserving realism and grounding, but the template skeleton ensures that the resulting dataset remains systematically organized. Because the questions are generated under explicit structural guidance, they can be paired with downstream filtering to remove linguistic shortcuts and ambiguous cases. Moreover, the taxonomy interacts naturally with our fine-grained visual tags, evidence timestamps, and evaluation dimensions. They define what kinds of episodic memory behaviors are tested, how those behaviors are distributed, and how model failures can later be interpreted.

Table 8: The 50 foundational question templates organized by 4W1H intent and Ego/Exo perspective.

ID	Intent	Perspective	Question Template
1	How	Exo	How many {object_x_plural} were there when {person_z} {action} {object_y}?
2	How	Exo	How many {object_x_plural} are at {place_y}?
3	How	Exo	How many {object_x_plural} appear during {time_interval_y}?
4	How	Exo	How many {object_x_plural} were around when {object_y} {state_or_action}?
5	How	Ego	How long did I spend doing {activity_x}?
6	How	Exo	How long did {someone_x} spend doing {activity_x}?
7	How	Ego	How did I {action} {object_y}?
8	Where	Exo	Where was {object_x} when I {action} with {person_y}?
9	Where	Ego	Where was I when I {action} with {person_y}?
10	Where	Exo	Where was {object_x} when I {action} with {object_y}?
11	Where	Exo	Where was {object_x} when {someone_x} {action} {object_y}?
12	Where	Ego	Where was I when I {action} {object_y}?
13	Where	Ego	Where did I spend most of my time during {interval_or_activity_x}?
14	Where	Exo	Where did {someone_x} spend most of their time during {interval_or_activity_x}?
15	Where	Ego	Where did {object_x} end up after I {action} it?
16	When	Ego	When did I {action} with {person_y}?
17	When	Ego	When did I {action} at {place_y}?
18	When	Ego	When did I {action} with {object_y}?
19	When	Exo	When did {someone_x} {action} with {person_y}?
20	When	Exo	When did {someone_x} {action} at {place_y}?
21	When	Exo	When did {someone_x} {action} with {object_y}?
22	When	Exo	When did {object_x} become {state_y}?
23	What	Ego	What did I do when I was with {person_y}?
24	What	Ego	What did I do at {place_y}?
25	What	Ego	What did I do with {object_y}?
26	What	Ego	What did I do during {interval_x}?
27	What	Exo	What did {someone_x} do when {person_z} {action}?
28	What	Exo	What did {someone_x} do when {I/someone} {action} at {place_y}?
29	What	Exo	What did {someone_x} do with {object_y}?
30	What	Exo	What did {someone_x} do during {interval_x}?
31	What	Exo	What {object_x} did I {action}?
32	What	Exo	Which {object_x} is {state_y}?
33	What	Exo	What is the state of {object_x}?
34	What	Exo	What is the relation between {object_x} and {object_y}?
35	What	Exo	What (object) did I use for {activity_x}?
36	What	Exo	What object did I use/hold when {action} with {person_x}?
37	What	Exo	What color is {object_x}?
38	What	Exo	What text/symbol is visible on {object_x}?
39	What	Ego	Which hand did I use to {action} with {object_x}?
40	What	Ego	What did I pick up from {place_y}?
41	Who	Ego	Who did I {action} at {place_y}?
42	Who	Ego	Who did I {action} {object_y} with?
43	Who	Ego	Who {action} with me when I {action} with {person_y}?
44	Who	Ego	Who did I {action} with during {interval_x}?
45	Who	Exo	Who did {someone_x} {action} at {place_y}?
46	Who	Exo	Who did {someone_x} {action} {object_y} with?
47	Who	Exo	Who {action} {someone_x} when {someone_z} {action} with {person_y}?
48	Who	Ego	Who was {person_y} to me during/at {event_or_time_x}?
49	Who	Ego	Who was looking at me when I {action}?
50	Who	Exo	Who was holding/using {object_x} at {place_y}?

Appendix C Prompts

To ensure a fair comparison across model families, we use the same task-specific prompts for all evaluated models whenever the input setting is the same. In particular, the caption generation prompt is shared across open-source and closed-source models, and the same CaptionQA, VideoQA, and CleanQA prompts are used for their corresponding evaluation settings. The only differences across the 30s and 60s CaptionQA settings are the caption window length and the resulting caption content; the prompt template itself remains unchanged. The propmts can be found in Figure 7.

Figure 7: Evaluation prompts used for caption generation, CaptionQA, VideoQA, and CleanQA. Caption generation uses shared instructions with setting-specific frame descriptions: consecutive 1-FPS frames for full-coverage CaptionQA and uniformly sampled source frames for the frame-aligned and cross-captioning controls.

Appendix D Harness Engineering

We implement a caption-primary retrieve-and-verify harness to test whether sparse visual evidence can repair errors made by caption-based long-video reasoning. As summarized in Figure 8, the harness is intentionally conservative: the baseline CaptionQA prediction remains the default answer, and visual verification is only allowed to override it under strong evidence.

Inputs.

For each evaluated question, the harness uses three inputs: the question with four answer options, cached timestamped captions generated under a fixed caption specification, and the initial CaptionQA baseline prediction (i.e., the original answer derived exclusively from the text-based reasoning stage before any visual verification). We do not mix caption granularities: results using 30s captions are compared only against the corresponding 30s CaptionQA baseline, and likewise for 60s captions.

Caption-window retrieval.

For each multiple-choice question, we form a retrieval query by concatenating the question and all answer options. The harness then scores each timestamped caption window using a deterministic weighted retrieval rule:

	$\displaystyle s={}$	$\displaystyle 0.65\cdot s_{\text{BM25}}+0.20\cdot s_{\text{hash-cos}}$
		$\displaystyle+0.15\cdot s_{\text{overlap}}+0.35\cdot s_{\text{time}}.$

We decouple the retrieval score into four distinct components because relying on a single metric is brittle when querying self-generated captions. Each term targets a different failure mode of textual retrieval:

•

BM25 ( $s_{\text{BM25}}$ ) acts as the primary retrieval backbone. By leveraging term frequency and inverse document frequency (TF-IDF), it heavily rewards exact matches of rare, discriminative keywords (e.g., a specific object name) while discounting ubiquitous stop words.
•

Hash-Cosine ( $s_{\text{hash-cos}}$ ) provides a lightweight, fuzzy similarity signal based on n-gram hashing. Unlike the strict exact-match requirement of BM25, this term improves robustness against slight phrasing variations, synonyms, or morphological differences between the question and the generated caption.
•

Word Overlap ( $s_{\text{overlap}}$ ) computes the unweighted density of shared content words. This serves as a critical counterbalance to BM25: it prevents the system from over-indexing on a single extremely rare word while ignoring the rest of the query, ensuring that the retrieved window broadly covers the multiple entities mentioned in the prompt.
•

Time-Hint ( $s_{\text{time}}$ ) acts as a temporal grounding mechanism. Pure lexical models are notoriously blind to temporal logic. This term explicitly boosts windows that match temporal markers (e.g., before”, after”, first”, last”) or explicit timestamps present in the query, which is essential for episodic memory tasks involving sequence and state changes.

Together, these four signals form a robust ensemble that accurately localizes relevant moments without the computational overhead of deploying a neural bi-encoder for every frame.

We use the following retrieval configuration:

		$\displaystyle\texttt{top\_k}=3,$
		$\displaystyle\texttt{nms\_seconds}=45,$
		$\displaystyle\texttt{neighbor}=1.$

Here, top_k=3 means that the retriever keeps up to three highest-scoring seed caption windows for each question. To avoid selecting near-duplicate temporal evidence, nms_seconds=45 suppresses additional seed windows that fall within 45 seconds of an already selected seed. Finally, neighbor=1 expands each selected seed by adding one adjacent caption window before and after it, when available, as supporting evidence. Thus, the retrieved evidence contains at most nine caption windows: up to three temporally separated seed windows, each expanded with one neighboring window on both sides. Given the caption cache, question/options, and retrieval configuration, this evidence set is fixed and does not depend on model-generated timestamps. The length of each caption window follows the caption cache granularity: 30 seconds for the 30s setting and 60 seconds for the 60s setting, except for boundary windows such as the final segment.

Retrieved-caption answering.

After retrieval, the evaluated model answers the question based only on the caption text associated with the retrieved video windows. It outputs a retrieved-caption answer and a confidence label:

		$\displaystyle\texttt{ANSWER}\in\{A,B,C,D\},$
		$\displaystyle\texttt{CONFIDENCE}\in\{\text{low},\text{medium},\text{high}\}.$

This answer is not used as the final prediction. Instead, it serves as an intermediate candidate answer that focuses the subsequent visual verification step.

Sparse-frame verification.

The harness then constructs short visual clips from the same retrieved windows. We use:

		$\displaystyle\texttt{max\_clips}=3,$
		$\displaystyle\texttt{clip\_neighbor}=1,$
		$\displaystyle\texttt{expand\_sec}=5,$
		$\displaystyle\texttt{frames\_per\_clip}=4.$

Here, max_clips=3 limits verification to at most three visual intervals. Although the retrieved caption evidence may contain up to nine caption windows after neighbor expansion, the visual verification stage is centered only on the retrieved seed windows. Since top_k=3 selects at most three seed windows, max_clips=3 yields at most three visual intervals. For each visual interval, clip_neighbor=1 merges one adjacent caption window before and after the seed window when available. The merged window is then expanded by expand_sec=5 seconds on both sides and clipped to the video boundaries. Finally, frames_per_clip=4 sparse frames are uniformly sampled from each expanded interval. Thus, each question uses at most $3\times 4=12$ frames for visual verification. The same evaluated model is then called again with a verification-specific prompt. This prompt provides the sparse frames, the question/options, and the retrieved-caption answer as context for judging the visual evidence. The goal is to determine which option is supported by the sampled frames and how strong the support is, rather than asking the model to solve the full long-video task from scratch. Output of this step is:

		$\displaystyle\texttt{VISUAL\_ANSWER}\in\{A,B,C,D,\text{UNKNOWN}\},$
		$\displaystyle\texttt{VISUAL\_SUPPORT}\in\{\text{strong},\text{weak},\text{none}\}.$

Conservative caption-primary policy.

The final decision follows a conservative policy named visual_strong_notime. Let $a_{\text{base}}$ be the initial CaptionQA baseline prediction. By default, the final answer is:

a_{\text{final}}=a_{\text{base}}.

The harness overrides this baseline only when all of the following conditions hold: the visual support is strong, the visual answer is an option in $\{A,B,C,D\}$ , and the question/options do not contain an explicit clock-time cue such as 01:23 or 01:23:45. Formally,

\displaystyle a_{\text{final}}=\begin{cases}a_{\text{visual}},&\begin{aligned} &\text{if support is strong},\\ &a_{\text{visual}}\in\{A,B,C,D\},\\ &\text{and no clock-time cue exists},\end{aligned}\\ a_{\text{base}},&\text{otherwise}.\end{cases}

We block overrides for explicit clock-time questions because sparse frames may miss the referenced moment.

CapMem currently contains a single 1,000-question evaluation split and does not provide a separate development set. The retrieval weights, $top\_k=3$ , $nms\_seconds=45$ , and visual-verification budget were set as heuristic defaults rather than selected through an accuracy-based search. However, the final override policy, including the clock-time rule, was chosen after inspecting errors and comparing policy variants on the full benchmark. The resulting retrieve-and-verify gains should therefore be interpreted as a post-hoc proof of concept rather than an unbiased estimate of out-of-sample improvement.

Reproducibility.

The released script takes the CaptionQA baseline prediction result JSON, the raw retrieval-frame harness JSON, and the question file as inputs. It reports the final accuracy, the CaptionQA baseline accuracy, the accuracy delta, the number of overrides, and whether each override changes a baseline-wrong example into a correct final prediction or a baseline-correct example into an incorrect final prediction. This separation keeps the final evaluation auditable: retrieval and sparse-frame verification provide candidate correction signals, while the final answer remains caption-primary unless the visual evidence is strong.

Appendix E Annotation and Quality-Control Pipeline

CapMem prioritizes manually verified questions rather than automatic benchmark expansion. Table 9 summarizes the complete construction pipeline. Twelve trained annotators first created 2,109 candidate QA pairs using the taxonomy and templates described in Appendix B. Two reviewers then checked the solvability, answers and template and category assignments, resolved disagreements by consensus, and retained 1,276 candidates.

A four-model text-only ensemble subsequently identified 352 questions that could be answered from the question and options without video access, leaving 924 candidates. Because this automated filter could also reject visually grounded questions, the reviewers re-examined all rejected candidates and restored 76 false positives. They finally verified that each of the resulting 1,000 questions was unambiguously answerable from the source video.

Stage	Input	Procedure	Output
Candidate creation	–	Twelve trained annotators manually constructed QA pairs.	2,109
Internal review	2,109	Two reviewers checked answers and template/category assignments and resolved disagreements by consensus.	1,276
Text-only filtering	1,276	A four-model ensemble removed questions answerable without video access.	924
Manual adjudication and visual verification	924 retained + 352 rejected	Reviewers re-examined the automatically rejected candidates, restored 76 false positives, and verified visual solvability.	1,000

Table 9: Annotation and quality-control pipeline. The final benchmark retains 1,000 of 2,109 candidate questions.

Fine-grained visual tags were assigned according to explicit definitions and checked during review. Specifically, Object denotes questions requiring identification, recognition, or counting of visible entities; Attribute covers visually observable properties such as color, shape, state, text, or number; Spatial captures location, direction, containment, and relative spatial relations; and Action denotes visible actions, interactions, or state transitions. Tags are multi-label and are assigned according to the fine-grained visual information explicitly queried by each question. Reviewer disagreements were resolved by consensus.

The final benchmark covers 33.7 hours, 16 scenarios, and four application domains. Its scale reflects the cost of manual construction, review, shortcut filtering, and visual-solvability verification. It should therefore be interpreted as a quality-controlled evaluation set rather than a comprehensive sample of all wearable-video settings. Release artifacts are detailed in Appendix G.

Appendix F Supplementary Experimental Settings and Results

This appendix reports the model, deployment, decoding, and API configurations required to reproduce the experiments. It then provides video-clustered bootstrap uncertainty estimates for the main paired comparisons, complete fine-grained and evidence-location results, examines whether the observed duration crossover is directly caused by the VideoQA frame cap, and reports measured frame, runtime, token, and cost accounting.

F.1 Supplementary Experimental Settings

Evaluation pipeline.

All visual inputs are drawn from a 1-FPS frame cache. In full-coverage CaptionQA, consecutive frames are grouped into non-overlapping 30- or 60-second windows, and each window is captioned once. The timestamped captions are then reused by every question associated with the video; no visual frames are accessed during downstream QA. Direct VideoQA instead receives deterministically and uniformly sampled frames for each question. In the frame-aligned control, CaptionQA uses exactly the same selected frames as Direct VideoQA, with all other model and decoding settings unchanged.

Model	Hugging Face model ID	Exact revision
InternVL3.5-8B	OpenGVLab/InternVL3_5-8B	9bb6a56ad9cc69db95e2d4eeb15a52bbcac4ef79
InternVL3.5-38B	OpenGVLab/InternVL3_5-38B	de99855be3642cd44fe97c9b72d70e5ce2c07f69
Qwen3-VL-2B	Qwen/Qwen3-VL-2B-Instruct	89644892e4d85e24eaac8bacfd4f463576704203
Qwen3-VL-4B	Qwen/Qwen3-VL-4B-Instruct	ebb281ec70b05090aa6165b016eac8ec08e71b17
Qwen3-VL-8B	Qwen/Qwen3-VL-8B-Instruct	0c351dd01ed87e9c1b53cbc748cba10e6187ff3b
Qwen3-VL-32B	Qwen/Qwen3-VL-32B-Instruct	0cfaf48183f594c314753d30a4c4974bc75f3ccb
Qwen3.5-2B	Qwen/Qwen3.5-2B	15852e8c16360a2fea060d615a32b45270f8a8fc
Qwen3.5-4B	Qwen/Qwen3.5-4B	851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Qwen3.5-9B	Qwen/Qwen3.5-9B	c202236235762e1c871ad0ccb60c8ee5ba337b9a
Qwen3.5-27B	Qwen/Qwen3.5-27B	b7ca741b86de18df552fd2cc952861e04621a4bd

Table 10: Exact checkpoints used for the open-weight model evaluations. The checkpoints were retrieved on February 15, 2026 for InternVL3.5 and Qwen3-VL, March 4, 2026 for Qwen3.5-27B, and March 11, 2026 for the remaining Qwen3.5 models (UTC).

Open-weight models.

Table 10 lists the exact Hugging Face checkpoint identifiers and revisions, while Table 11 summarizes the hardware, parallelism, context lengths, frame caps, and maximum observed input lengths. Open-weight inference uses vLLM 0.19.0 and Transformers 4.57.6 with Python 3.10.20, PyTorch 2.10.0, CUDA 12.8, and cuDNN 9.10.2.21. Models run in BF16 without quantization on NVIDIA H200 GPUs. InternVL3.5-8B uses one GPU, InternVL3.5-38B uses two-way tensor parallelism, and all Qwen models use one GPU. No data or pipeline parallelism is used. Prefix caching is enabled.

Caption generation batches eight windows per llm.generate call. CaptionQA submits the 1,000 questions in one application-level call, while VideoQA groups questions by video. Caption generation, CaptionQA, and VideoQA use output limits of 220, 200, and 200 tokens, respectively. All use greedy decoding with temperature 0, $\texttt{top\_p}=1.0$ , and $\texttt{top\_k}=0$ ; Qwen3.5 thinking is disabled. Every observed input remains below the corresponding context limit, and no application-level input truncation is applied.

Closed-source models.

Table 13 summarizes the retained OpenRouter request parameters, retry behavior, and scoring policy. The main GPT-5.2 and Gemini 3 Flash experiments use OpenRouter through the /api/v1/chat/completions endpoint. The requested aliases resolve to openai/gpt-5.2-20251211 and google/gemini-3-flash-preview-20251217, respectively, on May 22, 2026. Requests use temperature 0 and minimal reasoning effort. No candidate count, decoding seed, stop sequence, explicit thinking budget, JSON schema, or custom safety setting is specified. Only the first returned candidate is consumed, and provider-default safety settings are retained. Responses are returned as plain text and parsed into one of $\{A,B,C,D\}$ .

The caption output limit is 220 tokens. The retained results indicate CaptionQA limits of 512 tokens for GPT-5.2 and 64 tokens for Gemini 3 Flash. Because the original complete launch commands were not preserved, these two values are reported as recovered configurations.

Retry and scoring policy.

Closed-source requests use a 180-second timeout and up to nine total attempts. HTTP 408, 409, 429, 500, 502, 503, and 504 responses, as well as network exceptions, are retried with exponential backoff and random jitter. A successful but empty, refused, safety-blocked, or unparsable QA response is not semantically re-requested and is counted as incorrect. Empty captions are retained, and downstream QA proceeds without a semantic retry. Raw outputs, parsed predictions, correctness, token counts, and finish reasons are stored for auditing.

Model	GPUs	TP	Context length	VideoQA cap	Max. input tokens
InternVL3.5-8B	$1\times$ H200	1	40,960	128	33,442
InternVL3.5-38B	$2\times$ H200	2	40,960	128	33,442
Qwen3-VL-2B	$1\times$ H200	1	131,072	768	20,677
Qwen3-VL-4B	$1\times$ H200	1	131,072	768	16,164
Qwen3-VL-8B	$1\times$ H200	1	131,072	768	14,897
Qwen3-VL-32B	$1\times$ H200	1	131,072	768	20,011
Qwen3.5-2B	$1\times$ H200	1	131,072	768	14,908
Qwen3.5-4B	$1\times$ H200	1	131,072	768	14,908
Qwen3.5-9B	$1\times$ H200	1	131,072	768	16,783
Qwen3.5-27B	$1\times$ H200	1	131,072	768	18,642

Table 11: Runtime configuration of the open-weight models. TP denotes tensor parallelism; no data or pipeline parallelism is used. The last column reports the maximum observed input length across the evaluated requests. All observed inputs remain below the corresponding context limit; therefore, no application-level input truncation is applied.

F.2 Fine-Grained Visual Detail Results

CaptionQA shows model-dependent performance across fine-grained visual tags.

Figure F.6 reports the full model-wise accuracy across four fine-grained visual tags: object identity, visual attributes, spatial relations, and actions. The relative performance of CaptionQA and Direct VideoQA varies across models and categories, indicating that caption-based representation does not provide a uniform advantage for fine-grained visual reasoning.

GPT-5.2 shows the clearest positive pattern: its 30-second CaptionQA setting outperforms Direct VideoQA across all four tags. This result indicates that GPT-5.2 can effectively use the fine-grained information preserved in its generated captions. In contrast, Direct VideoQA remains stronger than CaptionQA across the four tags for Gemini 3 Flash, showing that strong native visual processing does not necessarily benefit from conversion into textual memory.

The open-weight models exhibit similarly heterogeneous results. Several Qwen models improve or remain competitive under CaptionQA for particular object, attribute, spatial, or action subsets, but the direction and magnitude of the differences vary by architecture and tag. InternVL3.5 generally shows smaller or negative CaptionQA differences. These variations are consistent with CaptionQA depending jointly on what information is retained during caption generation and how effectively the downstream model reasons over the resulting text.

Caption memory introduces a representational trade-off: text can compactly support downstream QA but may omit small objects, subtle attributes, spatial relations, or brief actions. Our analysis does not show that textual retrieval is uniformly easier than visual reasoning. This motivates retrieve-and-verify, which consults sparse visual evidence to resolve details missing or ambiguous in captions.

F.3 Full Evidence-Location Analysis

Figure 9 provides the full model-wise breakdown of QA accuracy by evidence location. Supporting evidence is grouped into the beginning (0–20%), middle (20–80%), and end (80–100%) portions of each video. Several models exhibit lower Direct VideoQA accuracy for middle-position evidence, producing a U-shaped three-bin profile consistent with the “lost-in-the-middle” phenomenon. This pattern is visible for a number of Qwen and InternVL variants, but is not shared uniformly across all models.

CaptionQA produces a flatter evidence-location profile in several cases. GPT-5.2 provides the clearest example: its Direct VideoQA results contain a pronounced middle-position drop, whereas CaptionQA keeps middle-evidence accuracy closer to its beginning- and end-evidence performance. Qwen3.5-27B shows a related pattern under the 30-second setting, which both improves overall accuracy and reduces the observed middle-position drop. These comparisons show that caption-based representation can be associated with less middle-position degradation for particular models and settings.

A flatter profile does not necessarily imply higher absolute accuracy. For some InternVL and smaller Qwen models, CaptionQA reduces the variation across evidence-location bins while remaining below Direct VideoQA overall. The two interfaces therefore exhibit different error profiles in these cases. CaptionQA is additionally limited by information omitted or imprecisely represented during caption generation. The 30-second setting produces a flatter profile than the 60-second setting for several models, which is consistent with shorter caption windows retaining finer temporal detail, although the pattern is not universal.

Overall, CaptionQA reduces middle-position degradation for some models, but the pattern is not universal and does not establish a causal retrieval mechanism. The differing profiles motivate the retrieve-and-verify harness, in which captions provide candidate temporal locations and sparse frames recover details that the captions may omit.

F.4 Duration Regimes and the VideoQA Frame Cap

At 1 FPS, the 768-frame VideoQA cap corresponds to approximately 12.8 minutes. We therefore divide the benchmark into $\leq 12$ minutes, $>12$ – $\leq 20$ minutes, and $>20$ minutes to test whether CaptionQA’s positive gap begins as soon as Direct VideoQA reaches its frame limit. Table 12 reports question-weighted mean CaptionQA–Direct VideoQA gaps across the six Qwen models used in the frame-aligned control.

If exceeding the frame cap alone caused the crossover, a positive aligned gap should already appear in the intermediate $>12$ – $\leq 20$ -minute group. Instead, the aligned gap in this group is zero at 30 seconds and negative at 60 seconds. Positive mean gaps appear only in the strict $>20$ -minute group, where CaptionQA exceeds Direct VideoQA by 3.22 and 2.55 points at the two intervals. Exceeding the 768-frame limit is therefore not sufficient to explain the observed duration pattern.

One possible explanation is that longer videos contain more temporally distributed or repeated evidence, which captions may compress into shorter textual records. This interpretation remains hypothetical: because we do not sweep multiple frame caps, the $>20$ -minute result should be treated as an empirical regime in CapMem rather than a universal or cap-invariant threshold.

		Original		Aligned
Duration	#QA	30s	60s	30s	60s
$\leq 12$ min	148	$-3.93$	$-4.87$	$-2.93$	$-5.52$
$>12$ – $\leq 20$ min	60	$-2.52$	$-4.72$	$+0.00$	$-4.72$
$>20$ min	792	$+4.62$	$+2.22$	$+3.22$	$+2.55$

Table 12: Mean CaptionQA–Direct VideoQA gaps (pp) across six Qwen models by video duration.

F.5 Uncertainty Estimation

Because frame alignment reduces CaptionQA’s mean gains relative to the full-coverage setting, we further assess whether the remaining controlled effects are distinguishable from zero. We estimate uncertainty over videos rather than treating the 1,000 questions as independent observations, because questions associated with the same video share visual content and may have correlated errors. The analysis is based on the six Qwen models included in the frame-aligned control.

Let $n_{v}$ denote the number of questions associated with video $v$ . For model $m$ and caption interval $s\in\{30,60\}$ , we first compute the paired video-level accuracy gap

d_{v,m,s}=100\,\frac{C^{\mathrm{Cap}}_{v,m,s}-C^{\mathrm{Video}}_{v,m}}{n_{v}},

where $C^{\mathrm{Cap}}_{v,m,s}$ and $C^{\mathrm{Video}}_{v,m}$ are the numbers of correct CaptionQA and Direct VideoQA predictions on video $v$ . We then average the gap across the six models:

\bar{d}_{v,s}=\frac{1}{6}\sum_{m=1}^{6}d_{v,m,s}.

For a set of videos $\mathcal{V}$ , the reported mean gap is weighted by the number of questions associated with each video:

\widehat{\Delta}_{s}=\frac{\sum_{v\in\mathcal{V}}n_{v}\bar{d}_{v,s}}{\sum_{v\in\mathcal{V}}n_{v}}.

This weighting makes the point estimate equivalent to aggregating the paired predictions over all questions and six models while retaining the video as the unit of resampling.

We obtain 95% confidence intervals using 10,000 duration-stratified video-cluster bootstrap samples. For the full-benchmark analysis, we independently resample with replacement the 28 videos in the $\leq 20$ -minute stratum and the 47 videos in the $>20$ -minute stratum, preserving the original number of videos in each group. For duration-specific analyses, videos are resampled only within the corresponding stratum. Whenever a video is sampled, all of its questions and all six model results are retained as one cluster. The question-weighted mean gap is recomputed for every bootstrap sample, and the 2.5th and 97.5th percentiles of the 10,000 replicates form the reported 95% confidence interval.

The resulting intervals are reported in Table 5. These intervals quantify sampling uncertainty across the videos represented in CapMem; they do not measure uncertainty over model families, since the set of six Qwen models is held fixed.

Parameter	Caption generation	GPT-5.2 CaptionQA	Gemini 3 Flash CaptionQA
Request configuration
Request unit	One 30- or 60-second video window	One question	One question
Caption reuse	Reused by all questions associated with the video	–	–
Output limit	220 tokens	512 tokens^†	64 tokens^†
Temperature	0	0	0
Reasoning effort	Minimal	Minimal	Minimal
Explicit thinking budget	Not set	Not set	Not set
Concurrency	16	64	64
Timeout	180 s	180 s	180 s
Candidate count	Not explicitly set; only choices[0] is consumed	Same	Same
Seed	Not set	Not set	Not set
Stop sequences	Not set	Not set	Not set
Response format	Plain text; JSON mode disabled	Plain-text answer letter	Plain-text answer letter
Safety settings	Provider defaults	Provider defaults	Provider defaults
Retry and scoring policy
Maximum attempts	One initial request plus up to eight retries (nine attempts in total).
Retried failures	HTTP 408, 409, 429, 500, 502, 503, and 504 responses, together with network exceptions.
Backoff	Exponential backoff with random jitter; the base delay is capped at 120 seconds.
Refusal or safety block	Counted as incorrect when no answer in $\{A,B,C,D\}$ can be extracted.
Empty or unparsable QA	Counted as incorrect.
Semantic re-query	No re-query is issued after a successful API response, even when the response is empty, refused, or unparsable.
Empty caption	Retained; downstream QA proceeds without a semantic retry	–	–

Table 13: Retained OpenRouter request and failure-handling configurations for caption generation and CaptionQA. ^† The two CaptionQA output limits were recovered from truncation boundaries in the retained results because the original complete launch commands were not preserved.

Evaluation scope	Task	Selected frame identities	Total frame inputs
1,000 QA / 75 videos	CaptionQA 30s	50,907	50,907
	CaptionQA 60s	50,907	50,907
	CaptionQA 30s + 60s	50,907	101,814
	Direct VideoQA	50,907	725,973
100 QA / 8 videos	CaptionQA 30s	5,207	5,207
	CaptionQA 60s	5,207	5,207
	CaptionQA 30s + 60s	5,207	10,414
	Direct VideoQA	5,207	70,026

Table 14: Frame accounting in the frame-aligned evaluations. Total frame inputs count repeated use across caption intervals or question-level Direct VideoQA evaluations.

F.6 Frame Accounting, Runtime, and Cost

Frame accounting.

Let $T_{v}$ denote the number of 1-FPS frames in video $v$ , $Q_{v}$ the number of questions associated with that video, and $F_{m}$ the Direct VideoQA frame cap of model $m$ . In full-coverage CaptionQA, one caption interval processes approximately $T_{v}$ frames once during memory construction; evaluating both the 30- and 60-second settings therefore processes approximately $2T_{v}$ frame inputs. Caption generation is performed window by window and is not constrained by a video-level frame cap.

Direct VideoQA uses at most $\min(T_{v},F_{m})$ selected frames for each question. Its total number of question-level visual-frame inputs for video $v$ is therefore

N^{\mathrm{Video}}_{v,m}=Q_{v}\min(T_{v},F_{m}).

The cap $F_{m}$ is model-specific: it is 128 for InternVL3.5 and 768 for the Qwen models. In the frame-aligned Qwen experiment, CaptionQA also uses $\min(T_{v},768)$ frames, with exactly the same frame identities as Direct VideoQA.

Table 14 reports aggregate frame counts in the frame-aligned evaluations. Selected frame identities count unique frames selected across the videos. Total frame inputs additionally count repeated use across the two caption intervals or across question-level Direct VideoQA evaluations.

CaptionQA processes each selected frame once when constructing the memory for a given interval, after which downstream QA is text-only. Direct VideoQA instead associates its selected visual frames with each question-level evaluation. These counts describe the logical visual inputs; implementation-level caching may reduce repeated physical encoding.

Measured computation and cost.

We measure Qwen3.5-2B and Qwen3.5-27B over all 75 videos and 1,000 questions on one NVIDIA H200. Gemini 3 Flash is measured in a separate frame-aligned official-API timing run on the first 100 questions from eight videos. The CaptionQA measurements combine the independent 30- and 60-second caption-generation and downstream QA runs. Token counts include both memory construction and QA for CaptionQA and all question-level requests for Direct VideoQA. Table 15 summarizes the measured runtime, H200-hours, API cost, and token usage. The reported cost analysis covers the base CaptionQA pipeline and does not include the additional retrieval and visual-verification calls used by the retrieve-and-verify harness.

We report H200-hours rather than converting them into monetary cost because the applicable local hourly rate was not preserved. The Gemini timing experiment uses the official API and should not be conflated with the 1,000-question OpenRouter main evaluation. Its Direct VideoQA runtime includes rate-limit retries and failed attempts, and the recorded API charge is therefore a lower bound.

Measured costs are model-dependent: both local Qwen models require more time, GPU-hours, and input tokens for the two CaptionQA settings than for one Direct VideoQA run, whereas Gemini requires fewer tokens, less time, and lower API cost. The structural advantage is reuse: caption memory is constructed once per video, while Direct VideoQA reprocesses visual input for every question. Thus CaptionQA becomes increasingly amortized as more queries share the same memory.

Model	Scope	Task	Wall-clock	H200-hours	API cost	Input / output tokens
Qwen3.5-2B	1,000 QA	CaptionQA 30s+60s	00:40:36.88	0.6769	–	27,617,234 / 319,999
Qwen3.5-2B	1,000 QA	Direct VideoQA	00:15:03.74	0.2510	–	13,525,676 / 3,009
Qwen3.5-27B	1,000 QA	CaptionQA 30s+60s	01:35:54	1.5983	–	33,206,180 / 665,839
Qwen3.5-27B	1,000 QA	Direct VideoQA	00:33:01.63	0.5505	–	13,525,676 / 2,001
Gemini 3 Flash	100 QA	CaptionQA 30s+60s	00:14:02.19	–	$5.876472	11,815,823 / 29,344
Gemini 3 Flash	100 QA	Direct VideoQA	$\approx$ 03:11:53	–	$\geq$ $7.101995	76,269,813 / 151

Table 15: Measured computation, API cost, and token use. The Qwen measurements cover the full benchmark on one H200, whereas the Gemini measurements come from a separate official-API run on 100 questions.

Appendix G Licensing and Release Details

This appendix specifies the scope of the CapMem release and distinguishes CapMem-authored artifacts from third-party source media. We first describe the released artifacts and source-dataset terms, and then specify the licenses governing the CapMem-authored data and code.

G.1 Release Composition and Source-Dataset Licensing

CapMem is constructed from 75 egocentric videos totaling 33.7 hours, selected from seven previously released datasets. The released benchmark consists only of independently authored CapMem annotations and associated metadata. It contains 1,000 multiple-choice QA records, each including a CapMem question identifier, source-dataset and source-video identifiers, the question and four answer options, the ground-truth answer, supporting evidence timestamps, domain and scenario labels, Ego/Exo and 4W1H categories, evidence-location labels, and multi-label fine-grained visual tags. We additionally release the official evaluation split, source-video manifest, evaluation prompts, evaluation, test scripts, and retrieve-and-verify scripts.

The release contains no source videos, frames, audio, or copied source-dataset annotations. Users must obtain the corresponding media from the original providers and comply with their respective licenses, access conditions, attribution requirements, and privacy restrictions. Table 16 lists the source-dataset composition and video identifiers, while Table 17 summarizes the governing source-media terms and their treatment in the CapMem release.

Source dataset	#Videos	#QA	Hours	Source video identifiers
Ego4D	56	742	22.489	a887acbe-efd5-48e2-ad8b-8e22a3c654d2.mp4, e303a071-7d0b-40e7-b8df-dea6e79ad7e8.mp4, 23164be8-8b5e-492f-8417-a56f4f44d9db.mp4, 01111831-9107-43c4-bf0e-6b26e9b32a2b.mp4, f71187bc-cead-407d-ad14-62812993b6a3.mp4, 3906d25a-d0a4-4a7c-8906-a87cea106c66.mp4, 80c74a5b-81e0-4bc1-bc09-2f146351fe70.mp4, 962fc84c-470e-430b-beef-ec892f936f5c.mp4, f3b1d2c0-728b-40c4-bc8f-236b7a558dcc.mp4, 406c7c13-4390-4c19-99c6-68861245da0d.mp4, 6f3107a6-bd6a-4761-a44a-f53c40b47f7d.mp4, b0a30641-ad93-4d6a-92a9-f277c78f1aa5.mp4, f60c7255-97fb-4f8b-a12d-f8f805423bfe.mp4, 267aa640-f3d0-4288-96b2-6ecf32b23198.mp4, b79c342b-db1e-407e-958a-e1585c6c3ddd.mp4, 0075dfd4-03b2-48c3-bde1-d7851bc8e367.mp4, 3b3ecf0c-ee8d-4518-bf1c-3301ba44c743.mp4, 9155f5b9-6235-484f-990d-fbd3010a791e.mp4, 1680e3e9-d3fc-4903-94d8-aa8e74016d74.mp4, 320f0ec0-0e73-43aa-8375-1a030793eb1e.mp4, f41432c6-1ef3-48f6-8646-887ab70cb031.mp4, 07536df6-9813-42d9-93df-35835407d8ee.mp4, 114c1877-5b5b-4ca3-9399-bd535d55669a.mp4, 9bbaa1a1-2d12-4e85-b9ce-fc5a738bf38a.mp4, f8c36c51-1cca-4ed3-81db-c190b72d5c49.mp4, 6099730e-7a7c-41ce-8266-83a9f00cacec.mp4, ce19b572-7341-426a-8c2f-fe067344c98d.mp4, 7a07a236-aaba-48ee-90a8-598fb6626cc2.mp4, 820d4b9e-6fba-472f-aa45-25b27241d4af.mp4, e99a32fa-19c9-41e6-be66-37869bd26583.mp4, 6b6d73b9-e92c-4698-86df-bbe687a9e95e.mp4, 11586867-f4d9-4341-9532-4de21a47585d.mp4, a6f99fa0-6ad4-4b1c-b7bd-68d69e708736.mp4, a8197601-7474-4cb0-8d08-e8a8febcecfc.mp4, 02ca6897-bf08-4cfb-bbc7-7810dbf5cca2.mp4, 1ebbf490-8a85-46b9-99c9-cd0f7e3b634b.mp4, 60bfacbb-063f-4e3f-bc74-fe993c7a521a.mp4, 8e84c762-b0f9-4a7d-982e-010f27999435.mp4, bc16dcef-d666-42c8-8a83-6babdf0e045f.mp4, a01446fe-4495-4424-b729-b3f5661140e5.mp4, cb87d2ca-dc4f-4e1c-a13f-a8fe7dc952e9.mp4, e750d46b-2c3a-468e-88db-4169a9d65a9a.mp4, b7420366-105d-4583-b1d2-ecfbf2166733.mp4, def26f88-2696-4b70-a545-33ad481ca696.mp4, fdf36515-a08a-4575-9a79-65d0ec260cc7.mp4, 7c925095-c62a-4a7a-a2ca-118ca3f0176a.mp4, dced4894-d13a-4222-941b-5e4ecf1e3299.mp4, f22be6c4-8951-47f3-a8f4-d1a61912b4ff.mp4, af215e0f-56c7-4356-9aad-a3a5fb7bbc4f.mp4, 033520cd-e7e4-4649-b551-83bb451a4802.mp4, 41309ada-0a81-47b2-bdd1-453a178db18a.mp4, 3f8dbb4f-ad5e-4d54-bf95-3ddbe968a85d.mp4, d51cd3c2-dc8a-42ad-90c7-a130f6481b7d.mp4, 51ab3f5f-f17f-4726-8f2c-b8e504e47377.mp4, 542bd8e2-37a1-49be-8f4c-0fedeb1cc9f1.mp4, 5c520abb-dee8-4b65-9a2a-3112a42e79a9.mp4
EPIC-KITCHENS	4	61	1.967	EPIC-KITCHENS_P01_03.MP4, EPIC-KITCHENS_P01_05.MP4, EPIC-KITCHENS_P01_105.MP4, EPIC-KITCHENS_P01_09.MP4
EgoLife	5	89	3.328	DAY1_A1_JAKE_11493000–DAY1_A1_JAKE_12290000, DAY2_A3_TASHA_11200000, DAY5_A3_TASHA_15193000–DAY5_A3_TASHA_16160000, DAY6_A1_JAKE_12420000, DAY7_A4_LUCIA_14243000
CASTLE2024	3	51	3.000	CASTLE2024_day1_Allie_12.mp4, CASTLE2024_day1_Bjorn_09.mp4, CASTLE2024_day2_Allie_15.mp4
EgoPet	5	39	2.284	EgoPet_edited_3a628da0633ba2685a871f84e840129e7885d0571e00377024f25255d5638ef6_segment_25.mp4, EgoPet_edited_f0596f2b564265bb5f34dc2042148a61fbc62e3bba7089c79f4c40ccb12e40d3_segment_011.mp4, EgoPet_edited_2601ab6126bb4fdc6aa41e307134b48a14b317228b4bb1e566adb7886e762899_segment_002.mp4, EgoPet_edited_ca553b127e28b214fc5130f384ca5341912318ef4e23d6e4935388451bdf819a_segment_003.mp4, EgoPet_edited_e639364fb4e2e34c2c4dab43cba2c1809cf8f0de12dad84b80cc849b6dc34390_segment_1.mp4
ENIGMA-51	1	4	0.278	ENIGMA51_141.mp4
HoloAssist	1	14	0.371	z015-june-17-22-rashult_assemble
Total	75	1,000	33.717	—

Table 16: Source-dataset composition and source-video identifiers used by CapMem.

Source dataset	Source-media license or access terms	CapMem release treatment
Ego4D	Ego4D Dataset License Agreement; access requires acceptance of the official agreement.	CapMem releases source identifiers and independently authored annotations only; no source media or copied source annotations.
EPIC-KITCHENS	Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0).	Source media are excluded; users obtain them from the original provider and comply with attribution and non-commercial-use terms.
EgoLife	S-Lab License 1.0; non-commercial use subject to its attribution and redistribution conditions.	Source media are excluded.
CASTLE2024	Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0), together with the project Terms of Use.	CapMem releases no source media or copied annotations; users remain subject to the original access and non-commercial-use terms.
EgoPet	Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0).	Source media are excluded; CapMem releases only source identifiers and independently authored annotations.
ENIGMA-51	No explicit public dataset license is stated on the official project page.	Source media are excluded.
HoloAssist	Community Data License Agreement–Permissive 2.0 (CDLA-Permissive-2.0).	Source media are excluded; CapMem releases only source identifiers and independently authored annotations.

Table 17: Licensing and release treatment of the source media represented in CapMem.

G.2 CapMem License

The independently authored CapMem QA annotations and associated metadata are released under the Creative Commons Attribution-NonCommercial 4.0 International license (CC BY-NC 4.0). This release includes the questions and answer options, ground-truth answers, evidence timestamps, taxonomy and visual-tag labels, the official evaluation split, source-video manifest, and evaluation prompts. These materials may be shared and adapted for non-commercial purposes with appropriate attribution.

The evaluation, test scripts, and retrieve-and-verify scripts are released separately under the Apache License 2.0. The corresponding license texts and attribution instructions are included in the release as LICENSE-DATA and LICENSE-CODE.

These licenses cover only CapMem-authored artifacts. Source media and source-dataset annotations are not redistributed and remain governed by their original terms; users must obtain the media from the original providers.