for memory management (M2A, M3-Agent, M3C, MemAgent-7B, and Memory-T1). While the
frozen-backbone agents preserve much of the abstention behavior of the underlying LLM, the others
collapse to 9–22% AR. M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score
81.82% under direct inference, and the backbone ablation in Appendix G.6 confirms that stronger
backbones alone do not close this gap. This suggests that the reward design of current RL/SFT
fine-tuning on memory agent backbones optimizes mainly answer correctness and retrieval success,
providing no signal that refusing an unanswerable question is correct, so abstention degrades after
training. Future agent designs should therefore optimize memory access, answer accuracy, and
evidence-sensitive abstention jointly [74], rather than treating memory management as independent
of hallucination control.

LVLMs and memory agents show complementary scaling trade-offs. Figure 4a plots accuracy
across five abilities from 32K to 128K for both LVLMs and memory agents. The two approaches
respond to context scaling in structurally different ways. LVLMs degrade substantially on retrieval-
heavy types, with IE and KU losing ∼20% and ∼12% respectively as evidence images become harder
to locate among growing filler content. MSR shows apparent flatness that reflects a floor effect near
30% rather than genuine robustness. AR shows the steepest LVLM decline (∼75% at 32K to ∼45%
at 128K), suggesting that growing filler content erodes abstention and pushes LVLMs to hallucinate
on unanswerable questions rather than refuse them. Six of seven memory agents, by contrast, stay
within ±7% from 32K to 256K because their retrieve-then-reason pipeline is length-invariant by
construction. The two failure modes are orthogonal: LVLMs lose to context length, while memory
agents lose to lossy multimodal compression at storage time. Because each architecture covers only
one axis, scaling along that axis leaves the other failure mode untreated, motivating hybrid designs
that span both axes.

Memory ability correlations reveal distinct sources of difficulty.
We analyze pairwise Spearman correlations among the five question         IE 1.00 -0.19 0.12 0.65 0.36 0.23
                                                                                                        0.31



types at 32K (Figure 5). Correlations vary substantially across type MSR -0.19 1.00 0.33 0.21 -0.11 0.060.21

pairs, indicating that the types do not measure a single capability. IE
and KU form the most consistent retrieval-oriented pair: they have       TR 0.12 0.33 1.00 0.29 -0.00 0.18
                                                                                                        0.13



the strongest correlation at 32K and remain significantly correlated     KU 0.65 0.21 0.29 1.00 0.59 0.43
                                                                                                        0.19


across context lengths, reflecting their shared need to first locate
the relevant evidence image. KU also correlates with AR at 32K,          AR 0.36 -0.11-0.00 0.59 1.00 0.21
                                                                                                        0.28



suggesting that some KU questions still depend on accurate evidence           IE MSR TR KU AR Avgstd

selection. In contrast, MSR shows weak correlations with IE and Figure 5: Spearman rank correla-
AR at all lengths, implying that its main challenge is aggregating tion (ρ) at 32K across all 34 evalu-
information across multiple evidence pieces rather than retrieving ated LVLMs and memory agents.
a single image. These findings reveal two complementary difficulty
axes: an evidence-retrieval axis (IE, KU) and an aggregation axis (MSR). They also illustrate how
a single aggregate score can obscure differences among long-context abilities, motivating per-type
evaluation alongside overall performance [43, 44]. Full correlation matrices at 64K and 128K appear
in Appendix G.5.

Error analysis. Figure 4b decomposes wrong answers at 128K across the four answerable types,
with the category definitions provided in Appendix G.4. On IE and KU, nearly 90% of errors fall in
the Visual category, showing that the model fails to locate or read the evidence image; once the image
is reached, the answer is usually extracted correctly. TR errors split between Mixed and Reasoning,
reflecting both image-supportive grounding and small closed-set selection. MSR is the only type
whose errors are dominated by the Reasoning category (73%). We further conduct an oracle-retrieval
diagnostic in Appendix G.7, which restores accuracy on MSR to 90–100% when only the required
sessions are supplied. This suggests a reasoning error at long context is largely induced by upstream
retrieval failure, since the model computes correctly over the wrong subset of evidence. LVLMs
and memory agents fail differently. LVLMs lose visual evidence as filler content accumulates in the
context window, whereas memory agents lose it through lossy cross-modal compression at storage
time. As memory grows, near-misses at 32K shift toward total misses at 128K (Appendix G.4) [75],
suggesting that growth weakens evidence retrieval rather than reasoning. Future memory designs
should therefore focus more on cross-modal evidence-retrieval fidelity instead of solely reasoning
improvements.


                                                    9
