From 0c92d81f083c21e0887e76548e1345bba6ba01a1 Mon Sep 17 00:00:00 2001 From: sphinx Date: Tue, 20 Oct 2026 23:25:48 +0000 Subject: [PATCH] sphinx: grounded long-context reasoning archive (12 arXiv papers, 25 verbatim-quote records, falsifiable verify.py) --- .../README.md | 27 + .../build_archive.py | 123 + .../candidates.json | 442 ++ .../claims.json | 54 + .../corpus/2409.02465.txt | 1 + .../corpus/2501.15089.txt | 1 + .../corpus/2502.13127.txt | 1 + .../corpus/2505.09561.txt | 1 + .../corpus/2506.02483.txt | 1 + .../corpus/2507.06261.txt | 1 + .../corpus/2507.06415.txt | 1 + .../corpus/2511.02817.txt | 1 + .../corpus/2512.04307.txt | 1 + .../corpus/2603.20586.txt | 1 + .../corpus/2604.05114.txt | 1 + .../corpus/2609.33485.txt | 1 + .../falsify_verify.py | 42 + .../fetch_corpus.py | 183 + .../manifest.json | 207 + .../mine_quotes.py | 50 + .../records.json | 218 + .../render_synthesis.py | 99 + .../sources.json | 3727 +++++++++++++++++ .../synthesis.md | 181 + .../verify.py | 79 + 25 files changed, 5444 insertions(+) create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/README.md create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/build_archive.py create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/candidates.json create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/claims.json create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2409.02465.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2501.15089.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2502.13127.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2505.09561.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2506.02483.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2507.06261.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2507.06415.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2511.02817.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2512.04307.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2603.20586.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2604.05114.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2609.33485.txt create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/falsify_verify.py create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/fetch_corpus.py create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/manifest.json create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/mine_quotes.py create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/records.json create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/render_synthesis.py create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/sources.json create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/synthesis.md create mode 100644 wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/verify.py diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/README.md b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/README.md new file mode 100644 index 0000000..2cee27a --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/README.md @@ -0,0 +1,27 @@ +# README — longctx-arxiv-archive + +Grounded research archive for the Archive lane (long-context research). + +## Contents + +- `synthesis.md` — themed synthesis; each evidence record carries a verbatim quote + arXiv source id. +- `corpus/.txt` — normalized plain text of each paper (single-space collapsed). +- `manifest.json` — per-paper sha256, char/word counts, retrieval method, title, authors, date. +- `candidates.json` — mined candidate sentences with stable (source, idx) keys. +- `claims.json` — operator-written claims keyed by (source, idx); quotes are never hand-typed. +- `records.json` — flattened evidence records. +- `verify.py` — independent checker; exit 0 = all quotes verbatim and corpus intact. +- `falsify_verify.py` — mutation tests: the verifier must exit nonzero on a fabricated quote, a + corrupted corpus file, and a missing corpus file (measured: all three exit 1). + +## Reproduce + +``` +python3 fetch_corpus.py # arXiv HTML/PDF -> corpus/ + sources.json +python3 build_archive.py normalize # collapse whitespace, recompute sha256 -> manifest.json +python3 mine_quotes.py # -> candidates.json +python3 render_synthesis.py # -> synthesis.md +python3 verify.py # exit code is the verdict +``` + +Coverage: 12/12 sources cited; 25 records; 964,298 corpus characters. diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/build_archive.py b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/build_archive.py new file mode 100644 index 0000000..c74a209 --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/build_archive.py @@ -0,0 +1,123 @@ +#!/usr/bin/env python3 +"""Build the grounded long-context archive: normalize corpus, pick verbatim +quote candidates, render the synthesis. Three modes: normalize | pick | render.""" +import hashlib, json, os, re, sys + +BASE = "/root/.hermes/wizard-outputs/sphinx/longctx-arxiv-archive-2026-10-20" +CORPUS = os.path.join(BASE, "corpus") + +# Per-source keyword specs: first sentence containing ALL terms (case-insensitive). +SPECS = { + "2507.06261": [("frontier", ["frontier", "long context"], ["gemini 2.5"])], + "2501.15089": [("benchmark", ["synthetic", "context expansion"], ["benchmark"])], + "2502.13127": [("cot", ["chain-of-thought", "reasoning"], ["supervised"])], + "2511.02817": [("aggregation", ["aggregation", "long context"], ["capabilities"])], + "2409.02465": [("novels", ["detective", "novels"], ["reasoning"])], + "2604.05114": [("structure", ["structure", "reasoning data"], ["improves"])], + "2603.20586": [("memory", ["memory-keyed", "attention"], ["efficient"])], + "2609.33485": [("disaggregation", ["disaggregation", "grounding"], ["scaling"])], + "2512.04307": [("webagents", ["webagent", "long-context"], ["evaluating"])], + "2507.06415": [("test-time", ["test-time", "learning"], ["long-context"])], + "2505.09561": [("diffusion", ["past-token", "prediction"], ["long-context"])], + "2506.02483": [("neurosymbolic", ["neurosymbolic", "reasoning"], ["multilingual"])], +} + + +def norm(t): + t = t.replace("\xa0", " ").replace("\u2019", "'").replace("\u2013", "-").replace("\u2014", "-") + t = re.sub(r"\s+", " ", t) + return t.strip() + + +def sentences(t): + return [s.strip() for s in re.split(r"(?<=[.!?])\s+", t) if s.strip()] + + +def load_corpus(): + out = {} + for fn in sorted(os.listdir(CORPUS)): + if fn.endswith(".txt"): + with open(os.path.join(CORPUS, fn), encoding="utf-8") as fh: + out[fn[:-4]] = fh.read() + return out + + +def mode_normalize(): + meta = json.load(open(os.path.join(BASE, "sources.json"))) + sources = meta["sources"] + manifest = [] + for s in sources: + t = norm(load_corpus()[s["arxiv_id"]]) + path = os.path.join(CORPUS, s["arxiv_id"] + ".txt") + with open(path, "w", encoding="utf-8") as fh: + fh.write(t) + sha = hashlib.sha256(t.encode()).hexdigest() + s.update({"text_sha256": sha, "text_chars": len(t), "text_words": len(t.split())}) + manifest.append({"arxiv_id": s["arxiv_id"], "title": s["title"], "sha256": sha, + "chars": len(t), "words": len(t.split()), "method": s["method"], + "url": s["source_url"], "published": s["published"], + "authors": s["authors"][:6]}) + json.dump(manifest, open(os.path.join(BASE, "manifest.json"), "w"), indent=2) + json.dump(meta, open(os.path.join(BASE, "sources.json"), "w"), indent=2) + print(f"normalized {len(manifest)} sources; total chars {sum(m['chars'] for m in manifest)}") + + +def mode_pick(): + corpus = load_corpus() + pool = [] + for aid, specs in SPECS.items(): + if aid not in corpus: + print("MISSING source", aid) + continue + for label, terms, tags in specs: + hit = None + for s in sentences(corpus[aid]): + low = s.lower() + if all(t in low for t in terms) and 60 <= len(s) <= 600: + hit = s + break + pool.append({"id": f"{aid}::{label}", "arxiv_id": aid, "label": label, + "terms": terms, "quote": hit}) + json.dump(pool, open(os.path.join(BASE, "quotes_pool.json"), "w"), indent=2) + got = sum(1 for p in pool if p["quote"]) + print(f"picked {got}/{len(pool)} quotes") + for p in pool: + print(("OK " if p["quote"] else "MISS") , p["id"], "|", (p["quote"] or "")[:150]) + + +def mode_render(claims_path): + pool = {p["id"]: p for p in json.load(open(os.path.join(BASE, "quotes_pool.json")))} + manifest = json.load(open(os.path.join(BASE, "manifest.json"))) + by_id = {m["arxiv_id"]: m for m in manifest} + claims = json.load(open(claims_path)) + used = set() + recs, skipped = [], [] + for c in claims: + p = pool.get(c["quote_id"]) + if not p or not p["quote"]: + skipped.append(c["quote_id"]) + continue + q = norm(p["quote"]) + if "\n" in q or len(q) < 60 or len(q) > 600: + skipped.append(c["quote_id"]) + continue + m = by_id[p["arxiv_id"]] + used.add(p["arxiv_id"]) + recs.append({"n": len(recs) + 1, "claim": c["claim"], "quote": q, + "arxiv_id": p["arxiv_id"], "label": c.get("label", p["label"]), + "chars": m["chars"], "title": m["title"], "method": m["method"]}) + json.dump({"records": recs, "skipped": skipped}, + open(os.path.join(BASE, "records.json"), "w"), indent=2) + print(f"rendered {len(recs)} records; skipped={skipped}; sources covered={len(used)}/{len(manifest)}") + + +if __name__ == "__main__": + mode = sys.argv[1] + if mode == "normalize": + mode_normalize() + elif mode == "pick": + mode_pick() + elif mode == "render": + mode_render(sys.argv[2]) + else: + raise SystemExit("usage: build_archive.py normalize|pick|render ") diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/candidates.json b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/candidates.json new file mode 100644 index 0000000..fac0fad --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/candidates.json @@ -0,0 +1,442 @@ +[ + { + "source": "2409.02465", + "idx": 0, + "quote": "To facilitate this research, we propose DetectiveQA, a dataset specifically designed for narrative reasoning within long contexts." + }, + { + "source": "2409.02465", + "idx": 1, + "quote": "Furthermore, we introduce a step-wise reasoning metric, which enhances the evaluation of LLMs' reasoning processes." + }, + { + "source": "2409.02465", + "idx": 2, + "quote": "Inspired by his words, we propose DetectiveQA, a long-context narrative reasoning dataset with three features." + }, + { + "source": "2409.02465", + "idx": 3, + "quote": "\u2022 We propose DetectiveQA, a human-annotated evaluation of narrative reasoning in long contexts, averaging over 100k tokens, with 1200 questions in English and Chinese, each paired with reference steps, averaging over 8 steps." + }, + { + "source": "2409.02465", + "idx": 4, + "quote": "\u2022 Furthermore, we introduce a step-wise reasoning metric that evaluates the LLM's reasoning process, which evaluates the evidence retrieval capability of long-context LLMs and, more importantly, reflects the logical coherence of their reasoning process." + }, + { + "source": "2409.02465", + "idx": 5, + "quote": "\u2022 We evaluate mainstream LLMs, including GPT-4, Claude, and LLaMA, revealing challenges in long-context reasoning." + }, + { + "source": "2409.02465", + "idx": 6, + "quote": "For long-context reasoning, the most commonly used metrics remain multiple-choice accuracy (Wang et al., 2024) or output matching (Kuratov et al., 2024)." + }, + { + "source": "2409.02465", + "idx": 7, + "quote": "Fortunately, we find a group of detective novels categorized as orthodox school (Saito, 2007), which emphasize logical puzzles and provide readers with the same evidence as the detective, making them suitable for our benchmark." + }, + { + "source": "2501.15089", + "idx": 0, + "quote": "To address this gap and enable a more comprehensive evaluation of the long-context reasoning capabilities of current LLMs, we propose a new synthetic benchmark, LongReason, which is constructed by synthesizing long-context reasoning questions from a varied set of short-context reasoning questions through context expansion." + }, + { + "source": "2501.15089", + "idx": 1, + "quote": "To address these challenges, we introduce a new long-context reasoning benchmark, LongReason, featuring diverse and realistic reasoning tasks to assess the long-context reasoning abilities of LLMs." + }, + { + "source": "2501.15089", + "idx": 2, + "quote": "To assess the current progress in the long-context reasoning abilities of existing LLMs, we evaluated 21 models of varying scales and architectures, sourced from both open-source and closed-source communities." + }, + { + "source": "2501.15089", + "idx": 3, + "quote": "Consequently, state-of-the-art language models [openai2024gpt4o, openai2024gpt4omini, reid2024gemini, anthropic2024claud35sonnet, llama3-1, mistrallarge2, qwen2.5, glm4] now support context windows ranging from 128K to millions of tokens, enabling the exploration of reasoning abilities over extensive contexts with LLMs." + }, + { + "source": "2501.15089", + "idx": 4, + "quote": "For each question, we evaluate multiple versions of the synthesized question for comparison, including the original question QshortQ_{\\text{short}}, the expanded version QexpandedQ_{\\text{expanded}}, and long-context versions with context lengths ranging from 8\u200bK8K to 128\u200bK128K." + }, + { + "source": "2501.15089", + "idx": 5, + "quote": "Evaluation setup We evaluate all models on LongReason, which comprises 794 questions, each featuring multiple variations, including the original version, expanded versions, and long-context versions with context lengths of 8K, 16K, 32K, 64K, and 128K." + }, + { + "source": "2501.15089", + "idx": 6, + "quote": "For long-context reasoning performance, Gemini-1.5 Pro outperforms all other closed-source models, exhibiting negligible performance drop when extending the context length from 8K to 128K." + }, + { + "source": "2501.15089", + "idx": 7, + "quote": "Additionally, we observe that for these problem types, Claude 3.5 shows much lower performance when the clues are distributed separately throughout the context, compared to when the clues are grouped together." + }, + { + "source": "2502.13127", + "idx": 0, + "quote": "To achieve this, we introduce LongFinanceQA, a synthetic dataset in the financial domain designed to improve long-context reasoning." + }, + { + "source": "2502.13127", + "idx": 1, + "quote": "Unlike existing long-context synthetic data, LongFinanceQA includes intermediate CoT reasoning before the final conclusion, which encourages LLMs to perform explicit reasoning, improving accuracy and interpretability in long-context understanding." + }, + { + "source": "2502.13127", + "idx": 2, + "quote": "To generate synthetic CoT reasoning, we propose Property-based Agentic Inference (PAI), an agentic framework that simulates human-like reasoning steps, including property extraction, retrieval, and summarization." + }, + { + "source": "2502.13127", + "idx": 3, + "quote": "To validate this hypothesis, we introduce LongFinanceQA, a novel long-context synthetic dataset constructed using financial data." + }, + { + "source": "2502.13127", + "idx": 4, + "quote": "To generate reliable reasoning-augmented answers, we propose Property-based Agentic Inference (PAI), a comprehensive agentic framework." + }, + { + "source": "2502.13127", + "idx": 5, + "quote": "Following this methodology, we propose the Property-based Agentic Inference (PAI) framework, illustrated in Figure 2, containing three steps: (1) property extraction, (2) property-based retrieval, and (3) summarization." + }, + { + "source": "2502.13127", + "idx": 6, + "quote": "For evaluation, Loong employs GPT-4-Turbo as a judge, scoring model responses based on accuracy, hallucinations, and completeness on a scale of 0 to 100." + }, + { + "source": "2502.13127", + "idx": 7, + "quote": "To assess the quality of these synthetic answers, we evaluate the annotator, PAI, on the Long benchmark, using GPT-4o-mini as the agent within the PAI, referred to as GPT-4o-mini w/ PAI." + }, + { + "source": "2505.09561", + "idx": 0, + "quote": "As context length increases, training becomes increasingly expensive due to rising memory demands, and policy performance often degrades as a result of spurious correlations." + }, + { + "source": "2505.09561", + "idx": 1, + "quote": "To address this, we introduce Past-Token Prediction (PTP), an auxiliary task in which the policy learns to predict past action tokens alongside future ones." + }, + { + "source": "2505.09561", + "idx": 2, + "quote": "Policies that latch onto such information may diverge from expert behavior during deployment, leading to performance degradation [10, 34]." + }, + { + "source": "2505.09561", + "idx": 3, + "quote": "In this paper, we introduce a simple and effective approach for learning long-context robot policies, illustrated in Fig." + }, + { + "source": "2505.09561", + "idx": 4, + "quote": "Crucially, we find that the benefits of PTP primarily emerge in the policy head for sequence modeling, rather than the visual encoder." + }, + { + "source": "2505.09561", + "idx": 5, + "quote": "Building upon this analysis, we introduce a multi-stage training recipe: first, pre-train the visual encoder in a short-context setting, where the policy learns to predict a chunk of future actions from only a few past frames [47, 8], and subsequently fine-tune a long-context decoder that jointly predicts past and future actions from precomputed image embeddings." + }, + { + "source": "2505.09561", + "idx": 6, + "quote": "Notably, recent works [8, 48] have empirically found that image-conditioned specialist and generalist policies degrade with history, leading many works to exclude history altogether [37, 4, 6, 47, 14, 38, 17]." + }, + { + "source": "2505.09561", + "idx": 7, + "quote": "To this end, we evaluate our method on the modern diffusion-based policy [8], in comparison with the classical regression-based policy." + }, + { + "source": "2506.02483", + "idx": 0, + "quote": "To address this challenge, we introduce NeuroSymbolic Augmented Reasoning (NSAR), which combines the benefits of neural and symbolic reasoning during inference." + }, + { + "source": "2506.02483", + "idx": 1, + "quote": "Our results highlight the effectiveness of combining explicit symbolic operations with neural inference for robust, interpretable, and scalable reasoning in multilingual settings." + }, + { + "source": "2506.02483", + "idx": 2, + "quote": "By integrating symbolic logic into a retrieval-based framework, NSAR improves interpretability, consistency, and accuracy when working with multilingual and long-context data." + }, + { + "source": "2506.02483", + "idx": 3, + "quote": "\u2022 We show that combining Chain of Thought (CoT), ReAct, and Self-Reflection prompting strategies with neurosymbolic reasoning leads to further performance gains." + }, + { + "source": "2506.02483", + "idx": 4, + "quote": "\u2022 Through extensive experiments on cross-lingual, long-context tasks, we demonstrate that NSAR substantially outperforms purely retrieval-based and neural-only methods." + }, + { + "source": "2506.02483", + "idx": 5, + "quote": "Hybrid systems like LINC (Olausson et al., 2023) and PAL (Gao et al., 2023) produce executable code, improving transparency and accuracy." + }, + { + "source": "2506.02483", + "idx": 6, + "quote": "Context length From the results presented in Figure 2, we observe that both GPT-4o-mini and Llama 3.2 exhibit declining accuracy as context length increases in LLM only (solid lines) scenario, with accuracy dropping to almost 0% as context lengths reach 128K words." + }, + { + "source": "2506.02483", + "idx": 7, + "quote": "We observe that RAG-Vanilla consistently outperforms the LLM-only baseline across all tested languages, including those with non-Latin scripts such as Persian, Russian, Hindi, and Arabic." + }, + { + "source": "2507.06261", + "idx": 0, + "quote": "Gemini Team Google Abstract In this report, we introduce the Gemini 2.X model family: Gemini 2.5 Pro and Gemini 2.5 Flash, as well as our earlier Gemini 2.0 Flash and Flash-Lite models." + }, + { + "source": "2507.06261", + "idx": 1, + "quote": "While this still increases training data throughput and storage demands by a factor of k, we find this to be a worthwhile trade-off given the significant quality improvement distillation has on our smaller models, leading to high-quality models with a reduced serving cost (see Figure 2)." + }, + { + "source": "2507.06261", + "idx": 2, + "quote": "The model has learned to use search and other tools, reason about the outputs, and issue additional, detailed follow-up queries to expand the information available to it and to verify the factual accuracy of the response." + }, + { + "source": "2507.06261", + "idx": 3, + "quote": "These models exhibit state-of-the-art performance across a suite of factuality benchmarks, including SimpleQA for parametric knowledge (Wei et al., 2024), FACTS Grounding for faithfulness to provided documents (Jacovi et al., 2024; Jacovi et al., 2025), and the Vectara Hallucination Leaderboard (Hughes et al., 2023), cementing Gemini as the model of choice for information-seeking demands." + }, + { + "source": "2507.06261", + "idx": 4, + "quote": "Deep Think enables Gemini to creatively produce multiple hypotheses and carefully critique them before arriving at the final answer, achieving state-of-the-art performances in challenging benchmarks such as Olympiad math (USAMO 2025), competitive coding (LiveCodeBench), and multimodality (MMMU), see more details at (Doshi, 2025b)." + }, + { + "source": "2507.06261", + "idx": 5, + "quote": "Audio Understanding In Table 5, we showcase the performance of the Gemini 2.5 model family at audio understanding, and compare the performance of these models to earlier Gemini models, as well as to GPT models." + }, + { + "source": "2507.06261", + "idx": 6, + "quote": "As can be seen, Gemini 2.5 Pro achieves state-of-the-art performance on key video understanding benchmarks, surpassing recent models like GPT 4.1 under comparable testing conditions (same prompt and video frames)." + }, + { + "source": "2507.06261", + "idx": 7, + "quote": "Performance is measured by string-match accuracy for multiple-choice VideoQA, LLM-based accuracy for open-ended VideoQA, R1@0.5 for moment retrieval and CIDEr for captioning." + }, + { + "source": "2507.06415", + "idx": 0, + "quote": "In this work, we propose PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for learning to encode long contexts using gradient updates at test time." + }, + { + "source": "2507.06415", + "idx": 1, + "quote": "These challenges arise because longer contexts contain more irrelevant distractor information, and may also be attached to more complex reasoning problems (i.e., multiple hops), both of which degrade LLM performance (Chen et al., 2023b)." + }, + { + "source": "2507.06415", + "idx": 2, + "quote": "In this work, we address these challenges of long-context reasoning by reframing it as test-time learning, enabling models to encode context at inference time using gradient-based parameter adaptation." + }, + { + "source": "2507.06415", + "idx": 3, + "quote": "PERK also outperforms specialized long-context LLMs (Gao et al., 2025a; Yang et al., 2025) with over 7B parameters that have undergone extensive long-context training, achieving the highest accuracy when generalizing to contexts up to 128k tokens." + }, + { + "source": "2507.06415", + "idx": 4, + "quote": "In these settings, PERK models again outperform the FT-ICR baselines, and even surpass commercial LLMs (e.g., GPT, Gemini) on benchmarks for long-context symbolic reasoning." + }, + { + "source": "2507.06415", + "idx": 5, + "quote": "PERK trained on 8K-token contexts maintains decent performance on 64K and 128K windows, consistently surpassing FT-ICR length generalization, and even outperforming models with native 512K and 1M context windows." + }, + { + "source": "2507.06415", + "idx": 6, + "quote": "(5) 4 Experiments We empirically evaluate PERK's long-context reasoning performance in three scenarios: reasoning over (1) Needles-in-a-Haystack (NIAH, Kuratov et al., 2024), (2) Open-domain, multi-document QA (Multi-Doc), and (3) Drops-in-an-Ocean (DIO), a novel long-context evaluation setting we propose." + }, + { + "source": "2507.06415", + "idx": 7, + "quote": "To address NIAH's fundamental limitation, where target information is stylistically distinct from distractors and thus artificially simpler to identify, we evaluate in two additional settings." + }, + { + "source": "2511.02817", + "idx": 0, + "quote": "(2024), inter alia), we find that models struggle with information aggregation as the input length grows." + }, + { + "source": "2511.02817", + "idx": 1, + "quote": "75.7% of traces end in an incomplete sentence.66 6 Note that in some cases, the model provides a candidate answer, then begins to double-check the answer and then exceeds the maximum output length; we evaluate the last candidate answer we are able to parse." + }, + { + "source": "2511.02817", + "idx": 2, + "quote": "We observe that both models, and especially Gemini 2.5 Pro, increasingly exceeds its maximum output length during reasoning for long Oolong-synth inputs." + }, + { + "source": "2511.02817", + "idx": 3, + "quote": "This is because GPT 5.2 is much more likely to refuse to perform the task; at 256K, GPT-5 attempts 83% of queries and answers 44.5% of these correctly, while GPT-5.2 attempts only 32% with an accuracy of 52.8%." + }, + { + "source": "2511.02817", + "idx": 4, + "quote": "As expected, adding labels improves accuracy; however, this improvement ranges from 10.9 points to only 0.79 points." + }, + { + "source": "2511.02817", + "idx": 5, + "quote": "Additionally, we show complex discussions surrounding dice rolls (Figure 13, Figure 15) and damage calculation (Figure 14)." + }, + { + "source": "2512.04307", + "idx": 0, + "quote": "Through extensive evaluation of four popular models, Claude-3.7, GPT-4.1, Llama 4, and o4-mini, we observe a dramatic performance degradation as context length increases, with success rates dropping from 40-50% in baseline conditions to less than 10% in long context scenarios." + }, + { + "source": "2512.04307", + "idx": 1, + "quote": "Please ensure the accuracy of your output, as we will execute subsequent steps based on the \u2018action\u2018, \u2018action_input\u2018 and \u2018element_id\u2018 you provide." + }, + { + "source": "2603.20586", + "idx": 0, + "quote": "We propose Memory-Keyed Attention (MKA), a hierarchical attention mechanism that integrates multi-level KV caches-local, session, and long-term-and learns to route attention across them dynamically." + }, + { + "source": "2603.20586", + "idx": 1, + "quote": "In this paper, we propose Memory-Keyed Attention (MKA), a novel attention mechanism that hierarchically organizes memory into three levels-local (L1), session (L2), and long-term (L3)-and dynamically learns how to route each query token across these sources." + }, + { + "source": "2603.20586", + "idx": 2, + "quote": "\u2022 We introduce Memory-Keyed Attention (MKA), a dynamic routing mechanism across local, session, and long-term memory with hardware-friendly implementations, we further accelerate MKA training and inference by designing broadcast routing (FastMKA)." + }, + { + "source": "2603.20586", + "idx": 3, + "quote": "Our approach supports query-dependent access, soft selection (via learned weights), and slot reuse-achieving high memory efficiency without sacrificing accuracy." + }, + { + "source": "2603.20586", + "idx": 4, + "quote": "To alleviate this, we propose Route-Fused MKA (FastMKA), a simplified yet effective variant that performs route-fusion-a token-wise soft fusion of hierarchical memory levels before attention-thereby avoiding multiple attention paths and significantly reducing runtime cost." + }, + { + "source": "2603.20586", + "idx": 5, + "quote": "Long-Context Benchmark Evaluation To validate MKA's effectiveness on long-context reasoning tasks, we evaluate on LongBench (Bai et al., 2024) and RULER (Sun et al., 2024) benchmarks, which test various long-context capabilities including QA, summarization, and long-distance retrieval." + }, + { + "source": "2603.20586", + "idx": 6, + "quote": "To address this, we propose Route-Fused MKA (FastMKA), a route-fusion variant that fuses memory levels before attention computation." + }, + { + "source": "2603.20586", + "idx": 7, + "quote": "It offers a practical balance between accuracy, latency, and memory usage, and serves as a strong drop-in replacement for traditional attention in long-context scenarios." + }, + { + "source": "2604.05114", + "idx": 0, + "quote": "Through deeper analyses, we observe that reasoning style contributes little, while faithful reasoning patterns discovered by back translation and grounded realistic long context, as \u03c02\\pi^{2} is designed for, are crucial for the improvement." + }, + { + "source": "2604.05114", + "idx": 1, + "quote": "We observe that structured representations, such as tables, is a natural medium that bridges complex reasoning and real long-context open-domain data, in which the data construction and verification are otherwise difficult." + }, + { + "source": "2604.05114", + "idx": 2, + "quote": "Through deeper analyses, we observe that reasoning style contributes little, while faithful reasoning patterns discovered by back translation and grounded realistic long context are crucial for the improvement." + }, + { + "source": "2604.05114", + "idx": 3, + "quote": "Finally, by the direct comparison with a previous long-context reasoning dataset, we show that \u03c02\\pi^{2} closes the gap between previous works and real-world demands." + }, + { + "source": "2604.05114", + "idx": 4, + "quote": "Our contributions are as follows: \u2219\\bullet We propose \u03c02\\pi^{2}, a QA curation pipeline that leverages tables as structured intermediate representations to generate verifiable, long-context, complex reasoning data across domains (\u00a72)." + }, + { + "source": "2604.05114", + "idx": 5, + "quote": "For reasoning traces generation, as we observed different reasoning \"styles\" of models from different families, we adaptively select the teacher model according to the target student model." + }, + { + "source": "2604.05114", + "idx": 6, + "quote": "We evaluate our models and baselines on in-domain benchmarks LongSeal, OfficeQA, and our \u03c02\\pi^{2}-Bench; as well as out-of-domain benchmarks LongBenchV2 and Oolong." + }, + { + "source": "2604.05114", + "idx": 7, + "quote": "Notably, we use the provided LLM grader (with backbone gpt-oss-120b) to report the accuracy on \u03c02\\pi^{2}-Bench and LongSeal, while for others, we report scores from the deterministic graders." + }, + { + "source": "2609.33485", + "idx": 0, + "quote": "Furthermore, it outperforms full-context models by up to 9.8 points on LongBench v2 and matches frontier models like Gemini-3-Pro-Preview while reducing inference costs by over 80%, establishing a highly efficient paradigm for robust long-context inference." + }, + { + "source": "2609.33485", + "idx": 1, + "quote": "Conversely, sequential multi-agent workflows (Zhang et al., 2024b; Qian et al., 2024; Zhao et al., 2024; Zhou et al., 2024) apply full LLM-grade understanding to each chunk, but process them serially---incurring linear latency and fragile cross-step state management that degrades over multiple hops." + }, + { + "source": "2609.33485", + "idx": 2, + "quote": "To realize this disaggregation, we introduce DIStributed long COntext scaling (DISCO), an architecture directly inspired by distributed computing frameworks like Apache Spark (Zaharia et al., 2010; Zaharia et al., 2016)." + }, + { + "source": "2609.33485", + "idx": 3, + "quote": "Experimental results demonstrate that DISCO effectively eliminates context rot: on RULER-QA (1M tokens), DISCO with Qwen3-8B (Qwen, 2025) maintains 78.4% accuracy while standard retrieval baselines collapse to 10.9%." + }, + { + "source": "2609.33485", + "idx": 4, + "quote": "On reasoning-intensive benchmarks, it outperforms Full Long Context baselines by up to 9.8 points (Qwen3-14B) on LongBench v2 (Long subset), with RL tuning further amplifying gains." + }, + { + "source": "2609.33485", + "idx": 5, + "quote": "This coupling creates a critical bottleneck: as LL grows, the complexity of the grounding term P\u0398\u200b(\ud835\udc9b\u2223\ud835\udc99,q)P_{\\Theta}(\\bm{z}\\!\\mid\\!\\bm{x},q) degrades rapidly due to architectural limitations (e.g., context rot and the \u201clost-in-the-middle\u201d phenomenon), where attention mechanisms fail to distinguish signal from noise." + }, + { + "source": "2609.33485", + "idx": 6, + "quote": "To circumvent the limitations of directly modeling the full sequence, we introduce Distributed Context Scaling (DISCO), a paradigm that structurally decouples contextual grounding via context partitioning." + }, + { + "source": "2609.33485", + "idx": 7, + "quote": "(7) The Accuracy Reward Racc\u2208{0,1}R_{\\text{acc}}\\in\\{0,1\\} evaluates the correctness of the final response \ud835\udc9a{\\bm{y}} against the ground truth." + } +] \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/claims.json b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/claims.json new file mode 100644 index 0000000..af31478 --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/claims.json @@ -0,0 +1,54 @@ +{ + "sections": [ + { + "title": "1. The core finding: reasoning degrades as context grows", + "records": [ + {"source": "2512.04307", "idx": 0, "label": "webagent-degradation", "claim": "Four popular agentic models lose almost all success once context grows: success falls from 40-50% in baseline conditions to under 10% in long-context scenarios."}, + {"source": "2511.02817", "idx": 0, "label": "aggregation-struggle", "claim": "Frontier models themselves struggle specifically at information aggregation as input length grows."}, + {"source": "2501.15089", "idx": 7, "label": "distributed-clues", "claim": "Performance depends on where the evidence sits: Claude 3.5 is much weaker when clues are spread through the context than when they are grouped together."}, + {"source": "2506.02483", "idx": 6, "label": "near-zero-accuracy", "claim": "Without retrieval support, LLM-only accuracy on long multilingual context falls to almost 0% at 128K words for both GPT-4o-mini and Llama 3.2."}, + {"source": "2501.15089", "idx": 6, "label": "flat-model", "claim": "Not every model degrades: Gemini-1.5 Pro shows negligible drop from 8K to 128K and outperforms other closed-source models on this axis."} + ] + }, + { + "title": "2. Why it degrades: distractors, spurious correlation, lost-in-the-middle", + "records": [ + {"source": "2507.06415", "idx": 1, "label": "distractors", "claim": "PERK attributes the degradation to two causes: more irrelevant distractor information in longer contexts and more complex multi-hop reasoning problems."}, + {"source": "2609.33485", "idx": 5, "label": "context-rot", "claim": "DISCO names the failure mode 'context rot' and ties it to the lost-in-the-middle phenomenon, where attention cannot separate signal from noise as sequence length grows."}, + {"source": "2505.09561", "idx": 0, "label": "spurious-correlation", "claim": "In robot policy learning, longer context raises memory cost and degrades performance through spurious correlations the policy latches onto."}, + {"source": "2505.09561", "idx": 4, "label": "where-ptp-helps", "claim": "The benefit of Past-Token Prediction concentrates in the policy head that does sequence modelling, not in the visual encoder."} + ] + }, + { + "title": "3. Benchmarks and datasets built to measure this", + "records": [ + {"source": "2409.02465", "idx": 3, "label": "detectiveqa-size", "claim": "DetectiveQA scales narrative long-context reasoning to over 100k tokens per instance, 1200 English and Chinese questions, each with about 8 reference reasoning steps."}, + {"source": "2501.15089", "idx": 0, "label": "longreason-construction", "claim": "LongReason synthesizes long-context reasoning questions by context expansion of short-context reasoning questions, giving controlled 8K-128K variants."}, + {"source": "2502.13127", "idx": 1, "label": "cot-in-synthetic-data", "claim": "LongFinanceQA embeds intermediate chain-of-thought reasoning before the final answer, which the authors argue improves accuracy and interpretability."}, + {"source": "2604.05114", "idx": 2, "label": "what-actually-helps", "claim": "Ablation in pi-squared shows reasoning style contributes little, while faithful reasoning patterns from back translation and grounded realistic long context are what matter."}, + {"source": "2604.05114", "idx": 4, "label": "table-pipeline", "claim": "The pi-squared pipeline uses tables as structured intermediate representations to produce verifiable long-context reasoning data across domains."} + ] + }, + { + "title": "4. Architectural and training fixes", + "records": [ + {"source": "2507.06415", "idx": 2, "label": "test-time-learning", "claim": "PERK reframes long-context reasoning as test-time learning: the model encodes the context at inference time via gradient-based parameter adaptation."}, + {"source": "2507.06415", "idx": 5, "label": "length-generalization", "claim": "A model trained on 8K contexts keeps working at 64K and 128K and beats models with native 512K and 1M context windows."}, + {"source": "2603.20586", "idx": 1, "label": "mka-routing", "claim": "Memory-Keyed Attention splits memory into local, session and long-term levels and learns to route each query token across them."}, + {"source": "2505.09561", "idx": 1, "label": "past-token-prediction", "claim": "Past-Token Prediction adds an auxiliary task in which the policy predicts past action tokens as well as future ones."}, + {"source": "2506.02483", "idx": 4, "label": "neurosymbolic", "claim": "NSAR combines symbolic logic with neural inference and substantially outperforms purely retrieval-based and neural-only methods on cross-lingual long-context tasks."}, + {"source": "2609.33485", "idx": 3, "label": "disco-rot-fix", "claim": "On RULER-QA at 1M tokens, DISCO holds 78.4% accuracy while standard retrieval baselines collapse to 10.9%, i.e. it removes the context-rot penalty."} + ] + }, + { + "title": "5. Cost, serving, and model behaviour at scale", + "records": [ + {"source": "2609.33485", "idx": 0, "label": "disco-cost", "claim": "DISCO beats full-context models by up to 9.8 points on LongBench v2 while cutting inference cost by over 80%."}, + {"source": "2511.02817", "idx": 3, "label": "refusal-behaviour", "claim": "Model behaviour changes with length: at 256K GPT-5 attempts 83% of queries and answers 44.5% correctly, while GPT-5.2 attempts only 32% with 52.8% accuracy on those attempts."}, + {"source": "2507.06261", "idx": 4, "label": "gemini-deepthink", "claim": "Gemini 2.5's Deep Think mode reaches state-of-the-art results on Olympiad math (USAMO 2025), competitive coding (LiveCodeBench) and multimodality (MMMU)."}, + {"source": "2507.06261", "idx": 3, "label": "gemini-factuality", "claim": "Gemini 2.5 is reported at state-of-the-art on factuality suites including SimpleQA, FACTS Grounding and the Vectara Hallucination Leaderboard."}, + {"source": "2502.13127", "idx": 5, "label": "pai-pipeline", "claim": "The PAI framework generates reasoning-augmented answers in three steps: property extraction, property-based retrieval, and summarization."} + ] + } + ] +} diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2409.02465.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2409.02465.txt new file mode 100644 index 0000000..8bcd909 --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2409.02465.txt @@ -0,0 +1 @@ +Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Related Work Long-Context Reasoning Reasoning Metrics 3 Creating DetectiveQA 3.1 Data Source 3.2 Data Annotation 4 Evaluating with DetectiveQA 4.1 Metric Settings Multiple-Choice Accuracy Step-wise Reasoning Metric 4.2 Context Settings 5 Experiment 5.1 Validation Results Statistics. Validation on data annotations. Validation on metrics. 5.2 Evaluation Results Models. Main results. Analysis on data contamination. 5.3 Identifying Performance Bottlenecks by Ablating Long Context and Reasoning 5.3.1 Analysis on evidence retrieval over long context. 5.3.2 Analysis on reasoning capability. 6 Conclusion References A Annotation Guideline Data format. Annotation procedure. Agent Workflow B Details in Reasoning Metric C More Validation Results Depth Distribution D More Evaluation Results License: arXiv.org perpetual non-exclusive license arXiv:2409.02465v2 [cs.CL] 14 Mar 2025 DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels Zhe Xu ††thanks: Equal contribution. Affiliation: School of Computer Science, Fudan University Email: zxu24@m.fudan.edu.cn Jiasheng Ye11footnotemark: 1 Affiliation: School of Computer Science, Fudan University Email: jsye23@m.fudan.edu.cn Xiaoran Liu11footnotemark: 1 Affiliation: School of Computer Science, Fudan University Email: xrliu24@m.fudan.edu.cn Xiangyang Liu11footnotemark: 1 Affiliation: School of Computer Science, Fudan University Tianxiang Sun Affiliation: School of Computer Science, Fudan University Zhigeng Liu Affiliation: School of Computer Science, Fudan University Qipeng Guo Affiliation: Shanghai AI Laboratory Linlin Li Affiliation: Huawei Noah's Ark Lab Qun Liu Affiliation: Huawei Noah's Ark Lab Xuanjing Huang Affiliation: School of Computer Science, Fudan University Xipeng Qiu ††thanks: Corresponding author. Affiliation: School of Computer Science, Fudan University Affiliation: Shanghai AI Laboratory Abstract Recently, significant efforts have been devoted to enhancing the long-context capabilities of Large Language Models (LLMs), particularly in long-context reasoning. To facilitate this research, we propose DetectiveQA, a dataset specifically designed for narrative reasoning within long contexts. We leverage detective novels, averaging over 100k tokens, to create a dataset containing 1200 human-annotated questions in both Chinese and English, each paired with corresponding reference reasoning steps. Furthermore, we introduce a step-wise reasoning metric, which enhances the evaluation of LLMs' reasoning processes. We validate our approach and evaluate the mainstream LLMs, including GPT-4, Claude, and LLaMA, revealing persistent long-context reasoning challenges and demonstrating their evidence-retrieval challenges. Our findings offer valuable insights into the study of long-context reasoning and lay the base for more rigorous evaluations. The evaluation code are publicly accessible via our GitHub repository 1 Introduction The long-context capabilities of Large Language Models (LLMs) (OpenAI, 2023; Anthropic, 2024a; Touvron et al., 2023; Sun et al., 2024; Bai et al., 2023a; Cai et al., 2024; Zeng et al., 2023b), particularly long-context reasoning(Kociský et al., 2018; Sprague et al., 2024; Wang et al., 2024; Karpinska et al., 2024), are a key competitive advantage in the current landscape. Recently, OpenAI released the O1 model (OpenAI, 2024), which not only supports a context length of 128k but can also generate extensive reasoning chains, effectively solving complex reasoning problems in intricate scenarios. As the long-context reasoning capabilities of LLMs improve, there is a growing demand for more challenging and realistic long-context reasoning evaluations(Sprague et al., 2024; Kuratov et al., 2024; Zhang et al., 2024; Wang et al., 2024; Karpinska et al., 2024). Among these, narrative reasoning based on detective novels provides a sufficiently realistic and challenging setting(Sprague et al., 2024; Gu et al., 2024). As Ilya, the core developer of GPT-4, has stated, if LLMs possess real understanding when presented with a detective novel with a complicated plot, a storyline, different characters, numerous events, and mysteries like clues, LLMs should be able to predict who commits the crime at the last page of the book based on the context11 1 Ilya Sutskever - GPT4 predicts the next word better - Now upgraded to the more powerful GPT4o https://www.youtube.com/watch?v=1OsHC1vbpc0. Inspired by his words, we propose DetectiveQA, a long-context narrative reasoning dataset with three features. First, DetectiveQA provides detailed annotation information. The dataset includes 1200 reasoning questions from English and Chinese detective novels, with an average length exceeding 100k tokens. As shown in Figure 1, we offer not only the questions, options, and answers but also the reference steps, which are the reference reasoning chains for the question, taken by detectives(Wang et al., 2024). Importantly, these reference steps include the explicit evidence, the evidence in the text, and the implicit evidence, the inference made by detectives. Our DetectiveQA has the highest average reasoning step number compared with other reasoning datasets as shown in Table 1. We also specify the paragraph locations of the evidence and indicate where the detectives give the answer. Furthermore, DetectiveQA features step-wise reasoning metric. Besides the stable assessment results through multiple-choice questions, we design an LLM-judged metric to evaluate whether LLM's reasoning processes aligns with the steps taken by detectives. This approach offers a more challenging and feasible method for assessing long-context reasoning(An et al., 2023). Finally, DetectiveQA can provide an in-depth analysis for long-context reasoning. We adjust the evaluation results and the context used during the assessment to explore the LLM's evidence retrieval capability(Kamradt, 2023; Li et al., 2024), the impact of data contamination(Li et al., 2023; Karpinska et al., 2024), and the differences in reasoning abilities between long and short contexts. To sum up, our contributions can be summarized as follows. • We propose DetectiveQA, a human-annotated evaluation of narrative reasoning in long contexts, averaging over 100k tokens, with 1200 questions in English and Chinese, each paired with reference steps, averaging over 8 steps. • Furthermore, we introduce a step-wise reasoning metric that evaluates the LLM's reasoning process, which evaluates the evidence retrieval capability of long-context LLMs and, more importantly, reflects the logical coherence of their reasoning process. • We evaluate mainstream LLMs, including GPT-4, Claude, and LLaMA, revealing challenges in long-context reasoning. We also identify data contamination issues and differences in reasoning between long and short contexts. Our findings provide valuable insights for research on long-context LLMs and long-context reasoning assessments. Figure 1: An example of annotation in DetectiveQA. We highlight the explicit evidence of reasoning in blue and implicit evidence in green. The whole reference steps include both. In contrast, in the Evidence Position field, the part corresponding to the explicit evidence will be the paragraph index in the novel, while that corresponding to the implicit evidence will be -1. 2 Related Work Table 1: Comparison of DetectiveQA with other reasoning datasets. To our knowledge, no previous dataset encompasses all of these qualities. ∼\thicksim denotes datasets that partially qualify for the property. Dataset Reasoning Natural Long Real- World Process Evaluation Reasoning Step Num NarrativeQA (Kociský et al., 2018) ✓ ✓ ✓ ✗ 1 HotpotQA (Yang et al., 2018) ✓ ✗ ✗ ✗ 2 BABILong (Kuratov et al., 2024) ✓ ✗ ✗ ✗ 2.2 NovelQA (Wang et al., 2024) ✓ ✓ ∼\thicksim ∼\thicksim - NoCha (Karpinska et al., 2024) ✓ ✓ ∼\thicksim ✗ - MuSR (Sprague et al., 2024) ✓ ✗ ✓ ✗ - DetectBench (Gu et al., 2024) ✓ ✗ ✓ ∼\thicksim - DetectiveQA (Ours) ✓ ✓ ✓ ✓ 8.5 Long-Context Reasoning Classical long-context benchmark (Bai et al., 2023b; Zhang et al., 2024; Kamradt, 2023) primarily focuses on tasks like QA, summarization, and retrieval, lacking an evaluation of long-context reasoning in real-world scenarios. Traditional long-context reasoning tasks, such as NarrativeQA (Kociský et al., 2018), cover limited clues, while HotpotQA (Yang et al., 2018), despite pioneering multi-hop reasoning, relies on synthetic data that does not provide a realistic context. Recently, NovalQA (Wang et al., 2024) and NoCha (Karpinska et al., 2024) have offered more challenging long-context reasoning evaluations through QA in long novels; however, their question designs are not sufficiently natural and are relatively rare in real-world situations. Among various genres of novels, detective novels are widely regarded as the most distinctive in terms of reasoning features (Gu et al., 2024; Sprague et al., 2024; Del & Fishel, 2023). Detective reasoning questions can authentically reflect LLM's understanding of context and its reasoning capability in real-world scenarios. Although detective novels have seen successful applications in short-context evaluations, such as MuSR (Sprague et al., 2024) and DetectBench (Gu et al., 2024), it has not yet been applied to long-context reasoning evaluation. In response to this gap, we propose DetectiveQA and provide detailed annotation information for classic long-form detective novels. Reasoning Metrics To measure the reasoning capability of long-context LLMs, the metric focused on the quality of the reasoning process is necessary. However, the commonly used ROUGE metric (Lin, 2004) generally fails to do so (An et al., 2023). Therefore, G-Eval utilizes LLMs like GPT-4 to assess the quality of NLG outputs (Liu et al., 2023), showing a higher correlation with human judgments in summarization and dialogue. Additionally, for mathematical reasoning (Mondorf & Plank, 2024), ReasonEval (Xia et al., 2024) introduces a method for evaluating the reasoning steps in math problems, emphasizing the validity and redundancy of each step and still using LLMs for automatic assessment. However, these studies have not addressed the reasoning process evaluation in narrative reasoning, particularly in long-context reasoning. For long-context reasoning, the most commonly used metrics remain multiple-choice accuracy (Wang et al., 2024) or output matching (Kuratov et al., 2024). In response to this, we draw inspiration from other process evaluations and, considering the characteristics of detective novels, propose an LLM-judged step-wise reasoning metric. 3 Creating DetectiveQA 3.1 Data Source Detective novels are valuable for studying language models' ability to handle long contexts due to their reasoning-heavy content. Therefore, we consider detective novels as promising candidates to be data sources of our benchmark. However, many prioritize storytelling over rigorous reasoning. Fortunately, we find a group of detective novels categorized as orthodox school (Saito, 2007), which emphasize logical puzzles and provide readers with the same evidence as the detective, making them suitable for our benchmark. Therefore, we collect orthodox detective novels as sources of long context and use questions related to the puzzles in the novels to test the language models. To ensure a smooth gradient of difficulty, we collect novels ranging from 100k to 250k words. We also limit our collection to Chinese and English versions, aligning with the language proficiency of the research team and data annotators. Figure 2: Illustration of DetectiveQA. The center shows the main annotation process, where human annotators annotate reasoning problems based on various information. On the left, AI-assisted information extraction offers summaries to help annotators quickly understand novels and locate key information. The right side, the most critical part, involves evaluating models using DetectiveQA, where reasoning metric and answer accuracy are measured. 3.2 Data Annotation Data annotation by hiring workers to read long novels (Wu et al., 2021) and generate questions is time-consuming and labor-intensive, taking an average of 3.5 hours to read a 100k-word novel, making it challenging to scale, thus prompting the need for a more efficient alternative. Our solution is to build an agent workflow using existing LLMs with strong long-context capabilities to assist human annotators. The key insight is that, for a complete detective novel, identifying reasoning questions and their corresponding answers can be viewed as an information extraction task, which is a simpler problem that state-of-the-art LLMs have shown promising performance (Zhang et al., 2024). The specific workflow involves the following steps: (i) Novel Comprehension. We first let the LLM summarize the novel to help the annotator grasp the overall story quickly. We input the entire novel again and ask the model to extract the detective's reasoning within the novel. Allow the annotator to understand the reasoning of the novel better. (ii) Question proposition. Next, Human annotators propose reasoning questions based on the reasoning process extracted. (iii) Human refinement. To aid human annotators in verifying the extracted content, we prepend indices to each paragraph in the novel and instruct the model to output the locations of the extracted reference steps. To ensure data quality, the human annotator performs discriminative filtering and refinement of the extracted reasoning with reference to the corresponding location in the original text. The above process enhances annotation efficiency while ensuring quality. Please refer to Appendix A for annotation details and Section 5.1 for data validation. 4 Evaluating with DetectiveQA 4.1 Metric Settings In this section, we present the evaluation metrics for DetectiveQA, including multiple-choice accuracy for a stable assessment of reasoning results and the step-wise reasoning metric for a detailed evaluation of the reasoning process. (a) Illustration on the reasoning metrics. The metric evaluates the ability of the models to recall evidence by comparing the models' answers to the reference steps using GPT4. (b) Context+Question (c) Question-Only (d) Evidence+Question (e) Three context settings for the evaluation on DetectiveQA. Context+Question is the standard setting that requires models to answer questions based on the full long context. Question-only setting prompt without the context, serving for a test of data contamination. Evidence+Question provides the golden evidence instead of the long context, aiming to ablate the reasoning ability of the models without long context. Figure 3: Reasoning metrics with its illustration and three settings. Multiple-Choice Accuracy Similar to previous evaluations based on multiple-choice questions (Hendrycks et al., 2021; Huang et al., 2023; An et al., 2023), we provide long-context LLM a question with four options and require LLM to output a letter corresponding to the selected option. At this point, we calculate the percentage of correctly answered questions as the score. Step-wise Reasoning Metric For narrative reasoning in DetectiveQA, simply assessing the answer is insufficient; it is important to measure the logical coherence, namely whether LLM presents sufficient evidence and articulates a complete reasoning process behind the correct answer. However, the automated evaluation of long outputs from LLMs remains challenging(An et al., 2023). Fortunately, thanks to the annotation, where each question is linked to corresponding reference steps, we can assess the logical coherence of LLMs' reasoning processes with how many correct reference steps are included, as shown in Figure 3(a). Therefore, we introduce a step-wise reasoning metric, the average score across all questions, to reflect the LLM's reasoning performance on DetectiveQA. To access the containment relationship, we use GPT-4 to review and count the reference steps provided in its responses. The specific prompt used is detailed in Appendix B. Importantly, unlike traditional multi-target information retrieval(Kamradt, 2023; Li et al., 2024) or multi-hop reasoning tasks(Yang et al., 2018; Kociský et al., 2018), DetectiveQA also evaluates whether the model can provide implicit evidence beyond the evidence present in the context, making it a more challenging and realistic evaluation. We can also visualize evidence retrieval with heatmaps in NIAH(Kamradt, 2023), which will be discussed in Section 5.2. 4.2 Context Settings In addition to two metric settings, we provide three context settings, as shown in Figure 3(e), to analyze different issues in LLMs' long-context reasoning. 1.Context+Question Concatenating the complete context before the answer appears in a detective novel, serving as the basic setting to test LLMs' long-context reasoning capability. 2.Question-Only Only the title and author of the detective novel are provided before the question. This approach addresses potential data contamination issues since our annotated detective novels are classics and may be included in the LLM's pre-training data. By comparing this setting with the previous one, we can assess whether long-context reasoning truly relies on contextual information. 3.Evidence+Question Only relevant evidence from the context is concatenated before the question, transforming long-context reasoning into short-context reasoning. This allows us to compare long-context and short-context reasoning across different LLMs. It is important to note that only the explicit evidence from the context is provided; the implicit evidence from the detective reference steps is not included. 5 Experiment We conduct experiments to validate the data quality of DetectiveQA and study the capability of prominent large language models with it. 5.1 Validation Results Figure 4: The distribution of the context tokens of samples in DetectiveQA. The novel content for each question is truncated before the answer appears. Statistics. We examine the statistics of DetectiveQA to ensure the distribution of both questions and answers satisfies our desiderata. We confirm that (1) DetectiveQA features long context, whose context lengths are distributed from 5k to 363k (Figure 4), with an average of 118k tokens (Table 2); (2) The number of clues distributed at each depth exceeded 100, and the exact distribution is shown in the Appendix C; and (3) the questions are reasoning-intensive, whose answers involve an average of 8.48 reference steps or near 400 tokens (Table 2). These results support DetectiveQA as a qualified and challenging evaluation for reasoning over long contexts. Table 2: Statistics of evidence and inferences in DetectiveQA.Our dataset also has richer corpus information in terms of responses. We count the lengths in words. And using the GPT4 tokenizer. Max. Min. Avg. context tokens 363k 5k 118k reference steps 27 3 8.48 reference tokens 1448 96 394.95 Validation on data annotations. Three authors of this work double-check the data to validate the annotation quality. Specifically, we sample annotated data from 10 novels and the three authors inspect (1) whether the questions are reasoning questions whose answers can be derived from the novel; (2) whether the reference steps are consistent with the original novel and all the evidence exists in the novel; and (3) whether the annotated answers are correct. The results in Table 3 indicate that the quality of almost all the data satisfies our requirement, thus supporting the validity of the outcome dataset. More supporting evidence will be provided in the Appendix C. Table 3: We asked three authors to score each of the ten novels using agent workflow to accelerate the labeling of questions in terms of question reliability, answer accuracy, and reasonableness of the reference step, and used a voting mechanism to arrive at a final score, and computed the Jaccard correlation between this result and the full 1-sequence. Validation part question reasoning answer Jaccard similarity 0.97 0.91 0.94 Validation on metrics. We verify the reliability of LLM as a judge for the step-wise reasoning metric. Specifically, we manually annotate the reasoning process of 100 answers and calculate the correlation between the judgment from the human judges and GPT4. As shown in Table 4, GPT4 shows high agreement with human judges. In comparison, Rouge, an n-gram-based automatic metric for content recall, despite being much cheaper, fails to correlate with human evaluation well (Table 5). Therefore, we consider it appropriate to apply GPT4 as an automatic evaluation of the reasoning process. Table 4: Using whether evidence is present in the chain of reasoning as a discrete judgment problem, we analyze the consistency of human judgments and GPT4 judgments using human judgments as the gold standard. Human vs GPT4 accuracy kappa Agreement 0.92 0.83 Overall, the above validation on both data and evaluation metrics ensures DetectiveQA is an evaluation that satisfies our needs for long-context reasoning ability for large language models. We then apply it to evaluate mainstream models and study their capabilities. Table 5: Correlation coefficient table. We randomly selected 100 reasoning processes generated by the model and compared them with human-labeled reference steps. For each process, we computed Rouge-1, Rouge-2, and Rouge-L scores, along with human and GPT ratings. The scores for the 100 reasoning processes, based on these four evaluation metrics, were then correlated with the human ratings to determine the correlation coefficients. Ours Rouge-1 Rouge-2 Rouge-L Corr. 0.91 0.52 0.61 0.58 5.2 Evaluation Results Models. We evaluate both closed-source and open-source LLMs featuring long-context capability. For closed-source models, we include GPT4-1106-preview-128k, OpenAI-O1-mini-128k (OpenAI, 2024), Claude3-opus-20240229-200k (Anthropic, 2024b), and KimiChat-200k. For open-source models, we evaluate LLaMA3.1-8B-Instruct-128k (Dubey et al., 2024), ChatGLM3-6B-128k (Zeng et al., 2023a), GLM4-9B-chat-1M (Zeng et al., 2023a), IntermLM2-7B-chat-200k (Cai et al., 2024), InternLM2.5-7B-chat-1M (InternLM, 2024) and Qwen2.5-7B-Instruct-128k (Team, 2024). Main results. We first report the multiple-choice accuracy and reasoning metric under the Question+Context setting in Table 6. Comparing the metrics, we find the multiple-choice accuracy and reasoning metrics show consistent trends across models, while gaps in numbers exist. This indicates a large number of questions are answered without a perfect reasoning chain, further highlighting the necessity to perform fine-grained evaluation on reference steps to study long-context reasoning capabilities. Table 6: Win rate was calculated for model responses based on the Question Only setting and the Question+Context setting, and G.M. is the geometric mean of the answer accuracy and reasoning scores. Models Question+Context Question-Only Win Rate Answer Reasoning G.M. Answer Reasoning G.M. GPT-4-1106-preview-128k 73.99 27.43 45.05 43.16 10.99 21.77 84.34 OpenAI-O1-mini-128k 60.83 23.80 38.05 41.67 11.64 22.03 70.65 KimiChat-200k 64.13 27.79 42.21 45.07 9.64 20.84 67.27 Claude3-Opus-200k 81.95 37.33 55.30 23.43 16.22 19.49 94.61 LLaMA3.1-8B-Instruct-128k 28.17 21.15 24.41 39.42 8.08 17.84 69.44 GLM3-6B-128k 40.58 22.08 33.63 33.63 7.16 15.51 63.47 GLM4-9B-chat-1M 59.00 24.07 37.68 40.33 8.06 18.03 74.02 Qwen2.5-7B-Instruct-128k 61.75 21.16 36.15 40.58 9.09 19.21 76.86 InternLM2-7B-chat-200k 57.95 23.94 37.24 36.97 12.65 21.62 81.69 InternLM2.5-7B-chat-1M 60.92 22.45 36.98 39.17 7.76 17.44 77.99 Comparing across models, results indicate obvious discrepancies, which we summarize as follows. • Open-source models still lag behind esteem closed-source models. While open-source models have made significant progress in recent development, claiming to approach state-of-the-art closed-source models (Dubey et al., 2024), the performance gaps still exist in long-context reasoning capabilities. • Distinction also exists among closed-source models. Despite being known as a strong reasoner, OpenaAI-O1-mini-128k does not show distinctively superior performance. Instead, Claude3 performed the best among the closed-source models. • Most open-source models perform on par, while Llama3.1 lags behind others. Our subsequent analysis (Section 5.3) attributes this to its failure on over 100k-long contexts. Analysis on data contamination. The use of detective novels may raise concerns about data contamination issues, which we investigate the potential impact through the question-only setting. The main idea is that the model is able to answer the question correctly even without the context if data contamination occurs. Therefore, we perform a question-wise comparison between the Question-only and Question+Context settings. We summarize the results in terms of win rate22 2 Win rate is defined as the score comparison of each question under two different settings, with scores comprising accuracy (0 or 1) and step-wise reasoning metric (0-1), where the higher score wins. in Figure 6(a). The results suggest the data contamination issues are mild. The question-only setting merely wins on a small proportion of questions, and the proportions are similar for different models. Besides, as shown in Table 6, our main results on question+context settings correlate with the win rate. Therefore, we consider the data contamination is not severe and does not affect the validity of our evaluation with DetectiveQA. (a) Llama3.1 (b) Qwen2.5 (c) GLM4 (d) InternLM2.5 (e) GPT4 (f) Claude3 Figure 5: Multi-needle-in-a-haystack test results for different models. We treat each clue in the reference steps found in the article as a ”needle” and determine whether the needle is detected by checking if it is included in the model's reasoning process. We define ”depth” as the percentage of the problem's total character count where the evidence appears far from the beginning of the document. Our analysis focuses on recall based on varying context lengths and clue depths. 5.3 Identifying Performance Bottlenecks by Ablating Long Context and Reasoning DetectiveQA evaluates the intersection of long-context processing and reasoning, two crucial abilities for large language models, simultaneously. To help identify the performance bottlenecks and provide insight to help improve model capability, we disentangle the effect of the two capabilities. (a) Stacked bar charts for analyzing data contamination. The figure contrasts model perf. in Q-only vs. C+Q modes using win rate (Sec. 5.2). If both settings yield incorrect answers, they're not compared, categorized as “both lose” for calc. (b) Stacked bar charts for reasoning analysis. The figure shows model perf. in evidence+Q and context+Q settings, focusing only on answer accuracy, as reasoning score comparison is unfair to context+Q. Figure 6: Figures of two different analyses. 5.3.1 Analysis on evidence retrieval over long context. To understand whether the evaluated models are proficient in long-context processing, we study the models' success rate in retrieving evidence in different positions over the context of different lengths, inspired by the multi-needle in a haystack task (Li et al., 2024). Results are in figure 5, which unveils that some models underperform others due to their limitations in long-context processing. For instance, the overall performance of Llama3.1 in our main setting lags behind others. Correspondingly, Figure 5 shows that Llama3.1 almost fails to retrieve any clues when the context length exceeds 100k tokens, while the others still show a descent recall rate. This showcases our annotation on evidence and their position being helpful to ablate the long-context abilities in long-context reasoning. 5.3.2 Analysis on reasoning capability. For how reasoning affects model performance on DetectiveQA, we study the comparison between Question+Context and Evidence+Context settings. The latter provides golden evidence instead of the long context, thus eliminating the influence of the long-context capability of the models. Similar to the comparison between question-only and Question+Context settings, we again conduct question-wise comparisons and summarize the win rates in the two settings, shown in figure 6(b). We summarize our findings from different types of answers, respectively. The ”evidence win” takes up a number of answers for all the models, indicating the need to enhance long-context capability for all models. Notably, the answers from Llama 3.1 take up a large proportion in this part, aligning with our previous findings on its limitation in long-context processing (Section 5.3.1). Additionally, comparing the proportions of being ”both wrong”, we find GLM3 stands out to have more answers belonging to this type, implying its less advanced reasoning ability. We also notice there exist a non-neglectable number of cases in ”context win”, possibly due to that dropping most of the context may harm the understanding of the story thus harming performance33 3 This may relate to the comparison between retrieval-based methods (Xu et al., 2023) and context-based methods for handling massive information. We leave this for future study. . 6 Conclusion We introduced DetectiveQA to test the models' ability to reason narratively over long contexts, the first benchmark for narrative reasoning with an average context length of 100k. We challenged the models' ability to reason over long texts as well as narrative reasoning using detective novels, the real-world texts. For each model, our test gives two scores (answer accuracy and reasoning score) in three settings. With a rich experimental setup, we can deeply analyze the performance of the model and find that the current model still faces challenges in long text comprehension, information extraction and narrative reasoning. We hope that our dataset will facilitate future improvements in model reasoning ability, leading to more robust AI applications and the highest machine intelligence. Limitations Our dataset only serves as an evaluation benchmark on long-context reasoning ability, while how to improve the model capability remains an open question. Meanwhile, our benchmark contains only data from detective novels and mainly serves narrative reasoning. More diverse scenarios can be included in the future. Ethics Statement We are committed to ensuring that DetectiveQA is used only for academic and scientific purposes, and therefore we have rigorously copyright-checked all of the reasoning novels used in Detective's annotations to ensure that the individual novels are not designed to create copyright problems in non-commercial areas. Through these screening tools, we aim to respect the principle of ‘fair use' under copyright protection and ensure that our project navigates within legal and ethical boundaries in a responsible manner. References An et al. (2023) Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. CoRR, abs/2307.11088, 2023. doi: 10.48550/ARXIV.2307.11088. URL https://doi.org/10.48550/arXiv.2307.11088. Anthropic (2024a) Anthropic. Model card and evaluations for claude models, 2024a. URL https://www-cdn.anthropic.com/bd2a28d2535bfb0494cc8e2a3bf135d2e7523226/Model-Card-Claude-2.pdf. Anthropic (2024b) Anthropic. Introducing the next generation of claude, 2024b. URL https://www.anthropic.com/news/claude-3-family. Bai et al. (2023a) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. CoRR, abs/2309.16609, 2023a. doi: 10.48550/ARXIV.2309.16609. URL https://doi.org/10.48550/arXiv.2309.16609. Bai et al. (2023b) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. CoRR, abs/2308.14508, 2023b. doi: 10.48550/ARXIV.2308.14508. URL https://doi.org/10.48550/arXiv.2308.14508. Cai et al. (2024) Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, and et al. Internlm2 technical report. CoRR, abs/2403.17297, 2024. doi: 10.48550/ARXIV.2403.17297. URL https://doi.org/10.48550/arXiv.2403.17297. Del & Fishel (2023) Maksym Del and Mark Fishel. True detective: A deep abductive reasoning benchmark undoable for gpt-3 and challenging for gpt-4. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (* SEM 2023), pp. 314-322, 2023. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Gu et al. (2024) Zhouhong Gu, Lin Zhang, Xiaoxuan Zhu, Jiangjie Chen, Wenhao Huang, Yikai Zhang, Shusen Wang, Zheyu Ye, Yan Gao, Hongwei Feng, and Yanghua Xiao. Detectbench: Can large language model detect and piece together implicit evidence?, 2024. URL https://arxiv.org/abs/2406.12641. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ. Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/c6ec1844bec96d6d32ae95ae694e23d8-Abstract-Datasets_and_Benchmarks.html. InternLM (2024) InternLM. Internlm2.5-7b, July 2024. URL https://huggingface.co/internlm/internlm2_5-7b. Kamradt (2023) Greg Kamradt. Needle in a haystack - pressure testing llms, 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack. Karpinska et al. (2024) Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. One thousand and one pairs: A” novel” challenge for long-context language models. arXiv preprint arXiv:2406.16264, 2024. Kociský et al. (2018) Tomás Kociský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Trans. Assoc. Comput. Linguistics, 6:317-328, 2018. doi: 10.1162/TACL\_A\_00023. URL https://doi.org/10.1162/tacl_a_00023. Kuratov et al. (2024) Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. arXiv preprint arXiv:2406.10149, 2024. Li et al. (2023) Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? CoRR, abs/2311.04939, 2023. doi: 10.48550/ARXIV.2311.04939. URL https://doi.org/10.48550/arXiv.2311.04939. Li et al. (2024) Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024. Lin (2004) Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004. Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2522, 2023. Mondorf & Plank (2024) Philipp Mondorf and Barbara Plank. Beyond accuracy: Evaluating the reasoning behavior of large language models-a survey. arXiv preprint arXiv:2404.01869, 2024. OpenAI (2023) OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. OpenAI (2024) OpenAI. Introducing openai o1, 2024. URL https://openai.com/o1/. Saito (2007) Satomi Saito. Culture and authenticity : the discursive space of japanese detective fiction and the formation of the national imaginary. 2007. URL https://api.semanticscholar.org/CorpusID:190048951. Sprague et al. (2024) Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of-thought with multistep soft reasoning, 2024. URL https://arxiv.org/abs/2310.16049. Sun et al. (2024) Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanjing Huang, Yu-Gang Jiang, and Xipeng Qiu. Moss: An open conversational large language model. Machine Intelligence Research, 2024. ISSN 2731-5398. doi: 10.1007/s11633-024-1502-8. URL https://doi.org/10.1007/s11633-024-1502-8. Team (2024) Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023. doi: 10.48550/ARXIV.2307.09288. URL https://doi.org/10.48550/arXiv.2307.09288. Wang et al. (2024) Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, and Yue Zhang. Novelqa: A benchmark for long-range novel question answering. CoRR, abs/2403.12766, 2024. doi: 10.48550/ARXIV.2403.12766. URL https://doi.org/10.48550/arXiv.2403.12766. Wu et al. (2021) Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul F. Christiano. Recursively summarizing books with human feedback. CoRR, abs/2109.10862, 2021. URL https://arxiv.org/abs/2109.10862. Xia et al. (2024) Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy. arXiv preprint arXiv:2404.05692, 2024. Xu et al. (2023) Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. CoRR, abs/2310.03025, 2023. doi: 10.48550/ARXIV.2310.03025. URL https://doi.org/10.48550/arXiv.2310.03025. Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pp. 2369-2380. Association for Computational Linguistics, 2018. doi: 10.18653/V1/D18-1259. URL https://doi.org/10.18653/v1/d18-1259. Zeng et al. (2023a) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130B: an open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL https://openreview.net/pdf?id=-Aw0rrrPUF. Zeng et al. (2023b) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130B: an open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL https://openreview.net/pdf?id=-Aw0rrrPUF. Zhang et al. (2024) Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. ∞\inftybench: Extending long context evaluation beyond 100k tokens. CoRR, abs/2402.13718, 2024. doi: 10.48550/ARXIV.2402.13718. URL https://doi.org/10.48550/arXiv.2402.13718. Appendix A Annotation Guideline Data format. For each novel, we require the annotators to annotate (1) several multiple-choice questions involving reasoning with (2) the answers to the question and (3) the multi-step reasoning chains. For each step in reasoning chains, we annotate (4) tags indicating whether the step is a piece of evidence in the original novel and the corresponding position. An example of our annotated data is in Figure 7. { ”question”:”Which of the following is the reason for the disappearance of Sainsbury Seale?”, ”options”: ”A”: ”left voluntarily.”, ”B”: ”met an untimely end.”, ”C”: ”eloped with someone.”, ”D”: ”Sudden memory loss.” ”answer”:”B”, ”reasoning”: [ ”Ms Sainsbury-Seal did not take her luggage with her when she disappeared.”, ”This does not appear to be a voluntary departure.”, ”Ms Seale had a dinner date with a friend to play solitaire.”, ”Normally at the appointed time she would have been back at the hotel.”, ”Therefore, based on the above evidences, it is surmised that it was Sainsbury Seale who met an untimely end.” ], ”evidence_position”:[740,-1,734,-1,-1], ”answer_position”: 1202 } Figure 7: An example of a multiple-choice annotation in DetectiveQA. We highlight the evidences of reasoning in blue italics, and inference in green plain typeface . the ”reasoning” part includes evidences and inferences, while in the ”evidence_position” field, the part corresponding to the evidence will be the paragraph in which the evidence occurs in the article, while the part corresponding to the reference will be -1. Annotation procedure. We visualize the annotation process in Figure 8. With the collected novels, we first apply our agents to summarize the novels, locating rationales from the detective in the novel and proposing candidate questions for each rationale. The prompts we use are in Fig. 7 With the outputs from the agent, we require the annotators to refine the data with the following procedure. (i) On questions. The annotator should determine whether a question requires reasoning to answer. This involves assessing if answering the question necessitates extracting and synthesizing clues from the original text. Questions that can be answered through simple retrieval of a specific answer from the text, without requiring any additional contextual information, should be filtered out. Additionally, the validity of the question should be evaluated, with manual modifications applied to the original question as needed. (ii) On options. The question must be accompanied by four options, including the correct answer and a minimum of one distractor. (iii) On evidence. The annotators evaluate whether the model-provided evidence is relevant to the question, complete, and accurate. They should eliminate incorrect or irrelevant evidence and supplement missing ones. (iv) On answers and rationales. The rationales should be separated into steps with each step being an evidence or inferences. (v) On positions. The annotators should ensure the answer positions and the evidence positions are correct. Agent Workflow Figure 8: Annotation process Figure 8 illustrates the specific process of accelerated annotation through agent workflow. LLM extracts a wide variety of key information, which a human then integrates and constructs into a complete annotation file for the reasoning problem. Appendix B Details in Reasoning Metric In our step-wise reasoning metric, we mentioned the use of GPT4 for the number of contained leads, and we used the following prompt template 9.We use LLMs such as Kimi, Claude3, and others. In order to evaluate a question-answering (QA) system's reasoning process regarding a particular inference question, specifically whether the reasoning process correctly includes certain reference steps, multiple reference steps will be provided. Due to the nature of the inference question being based on a detective novel, the reasoning process may involve some sensitive content. However, for the purpose of this evaluation, please focus solely on determining whether the reasoning process explicitly or implicitly includes the provided reference steps. The QA system's output for the reasoning process may not explicitly mention the provided reference steps, but it might implicitly incorporate them. In such cases, it should still be considered as correct including the provided reference steps. The reasoning process output by the QA system and the reference reference steps to be considered are presented below. Please objectively assess whether the QA system's reasoning process explicitly or implicitly includes the provided reference steps, and clearly state which reference steps are included. Reasoning Process: [Reasoning Process] reference steps: [reference steps] Provide an initial sentence explaining whether the reasoning process explicitly or implicitly includes each reasoning step. Then, in the second line, specify the indices of the included reference steps in a list format, such as [0, 1, 2, 3, …]. Your response should maintain this format: Explanation: ¡One-sentence explanation¿ Included Reference Steps: [Indices of the included reference steps] Figure 9: GPT-4 Questioning Template: Replace bolded font with evidence and Model's Inference Process in Query. This template asks enough questions to get usable responses without adding additional samples for a few shots to help answer. Appendix C More Validation Results Table 7 lists the statistics of the novel parts that were manual annotation and accelerated through agent workflow-assisted annotation. While there is an increase in the length of questions for agent workflow-assisted accelerated annotation, the evidence and reasoning sections of manual annotation are more detailed and have slightly less coverage than agent workflow-assisted accelerated annotation. Overall, the difference in quality between manual and agent workflow-assisted accelerated annotation was minimal, suggesting that the use of agent workflow-assisted accelerated annotation is feasible. Statistic w/ agent workflow w/o agent workflow (Max/Min/Avg) (Max/Min/Avg) context length 148k/4k/81k 167k/4k/94k reference steps 26 / 4 / 7.65 21 / 3 / 7.80 reasoning length 606 / 78 / 251.14 1073 / 92 / 301.94 total questions 308 338 Table 7: Annotation w/ agent workflow vs. Annotation w/o agent workflow Statistics Comparison where context length refers to the length of the problem and the meaning of the remaining metrics is detailed in Section 5.1. Depth Distribution The exact distribution can be seen in Figure 10, where there will be more clues at 100% depth since the longer dependent questions will also have valid clues in the closer locations. Figure 10: Distribution of different evidence depths. For each piece of evidence, we take as depth the percentage of the total number of words from the beginning of the word in which it occurs Appendix D More Evaluation Results We have done quite a lot of experiments under the 32K input length model, and we can find that the experimental results of the 32K model are all relatively unsatisfactory. The experimental results are shown in Table 8 However, a larger number of parameters would allow the model to make full use of the information in the 32K text and its own reasoning power to mitigate the problem. Table 8: Performance of models supporting context lengths of 32K or less Models Question+Context Question-Only Win Rate Answer Reasoning G.M. Answer Reasoning G.M. LongChat-v1.5-7B-32k 29.33 11.07 18.01 27.83 5.72 12.59 33.58 Vicuna-v1.5-7B-16k 30.33 12.63 19.57 27.67 6.69 13.60 32.57 Qwen1.5-7B-8k 49.50 10.09 22.34 35.33 7.74 16.53 60.71 Qwen1.5-72B-32K 70.67 19.69 37.30 44.67 10.55 21.70 76.51 Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" () button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. We gratefully acknowledge support from our major funders, member institutions, , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2501.15089.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2501.15089.txt new file mode 100644 index 0000000..4ce923b --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2501.15089.txt @@ -0,0 +1 @@ +1 Introduction 2 Related Work 3 Our Benchmark: LongReason 3.1 Long-context Reasoning Question Construction via Context Expansion 3.2 The Statistics of LongReason 4 Exerperiments & Results 4.1 Experimental setup 4.2 Main Results 4.3 Further Analysis 5 Conclusion and Limitations 6 Model Information 7 Hyperparameters for LongReason Construction 8 Human Annotator 9 Prompts 1]ByteDance Seed 2]UC San Diego 3]University of Illinois Urbana-Champaign \contribution[*]Work done at ByteDance Seed \contribution[†]Corresponding authors LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion Zhan Ling Kang Liu Kai Yan Yifan Yang Weijian Lin Ting-Han Fan Lingfeng Shen Zhengyin Du Jiecao Chen [ [ [ z6ling@ucsd.edu jiecao.chen@bytedance.com (November 17, 2025) Abstract Large language models (LLMs) have demonstrated remarkable progress in understanding long-context inputs. However, benchmarks for evaluating the long-context reasoning abilities of LLMs fall behind the pace. Existing benchmarks often focus on a narrow range of tasks or those that do not demand complex reasoning. To address this gap and enable a more comprehensive evaluation of the long-context reasoning capabilities of current LLMs, we propose a new synthetic benchmark, LongReason, which is constructed by synthesizing long-context reasoning questions from a varied set of short-context reasoning questions through context expansion. LongReason consists of 794 multiple-choice reasoning questions with diverse reasoning patterns across three task categories: reading comprehension, logical inference, and mathematical word problems. We evaluate 21 LLMs on LongReason, revealing that most models experience significant performance drops as context length increases. Our further analysis shows that even state-of-the-art LLMs still have significant room for improvement in providing robust reasoning across different tasks. We have open-sourced LongReason to support the comprehensive evaluation of LLMs' long-context reasoning capabilities. \correspondence Zhan Ling at , Jiecao Chen at \checkdata[Dataset]https://huggingface.co/datasets/lz1bytedance/LongReason 1 Introduction Benchmark Avg Len Light Human Effort Realistic Tasks Broad Tasks Controllable Context ZeroSCROLLS [zeroscrolls] ∼\sim10K ✗ ✓ ✓ ✗ L-Eval [leval] ∼\sim8K ✗ ✓ ✓ ✗ BAMBOO [dong2023bamboo] ∼\sim16K ✗ ✓ ✓ ✗ LongBench [bai2023longbench] ∼\sim8K ✗ ✓ ✓ ✗ LooGLE [li2023loogle] ∼\sim20K ✗ ✓ ✓ ✗ InfiniteBench [zhang2024infty] ∼\sim200K ✗ ✓ ✓ ✗ Loong [wang2024leave] ∼\sim250K ✗ ✓ ✓ ✗ Needle-in-a-haystack [needleinhaystack] any ✓ ✗ ✗ ✓ RULER [hsieh2024ruler] any ✓ ✗ ✓ ✓ LongReason (Ours) any ✓ ✓ ✓ ✓ Table 1: Comparison of LongReason with other long-context benchmarks. LongReason offers controllable context lengths and incorporating diverse and realistic tasks without the need for human annotation on long text. In recent years, large language models (LLMs) [openai2023gpt4, gemini, claude3, llama3-1, mistral, qwen2.5] have demonstrated remarkable advances in diverse natural language processing tasks. The ability to comprehend and reason over long inputs is essential for downstream applications, including multi-turn conversations [tan2024peer], document understanding [masry2024longfin] retrieval-augmented generation [yao2022react, xu2023retrieval], and language agents [zhao2024longagent, zhang2024chain]. Meanwhile, extensive efforts in deep learning system [dao2022flashattention, dao2023flashattention, chen2023longlora, ratner2022parallel] research have been devoted to optimizing computational overhead to support increasing numbers of input tokens, which has led to growing attention on long-context LLMs. Now, both proprietary and open-source LLMs can support up to millions of input tokens [gemini, mistralnemo, glm4]. However, despite the rapid development of long-context language models, benchmarks have lagged behind. One of the key challenges is dataset construction, as long-context question-answering data is relatively scarce on the internet. To address this, prevalent long-context benchmarks have utilized synthetic tasks like passkey retrieval [mohtashami2023landmark], needle-in-a-haystack (NIAH) [needleinhaystack, zhao2024longagent], and variable tracking [hsieh2024ruler] to evaluate long-context LLMs. However, these tasks are often unrealistic and involve reasoning processes that differ significantly from those in real-world applications. Alternatively, some research efforts have involved human annotation of realistic questions and gold answers over one or multiple long documents [bamboo, wang2024leave, li2023loogle]. However, creating realistic long-context tasks from extensive texts is both challenging and time-consuming, even for human experts [wang2024leave]. This limitation restricts the expansion of datasets to accommodate arbitrary context lengths and the ability to support controllable context. As shown in Table 1, existing benchmarks either rely on a limited number of synthetic tasks, demand significant human effort to read long contexts, or lacking controllable contexts and support for arbitrary context lengths. Furthermore, existing datasets [bamboo, wang2024leave, li2023loogle] often utilize documents from specific domains, such as financial reports or legal cases, as input, which can inherently limit the diversity of task categories. Consequently, they tend to focus on a narrow set of tasks, such as comparison or classification, rather than evaluating more complex and challenging tasks that require chain-of-thought reasoning. To address these challenges, we introduce a new long-context reasoning benchmark, LongReason, featuring diverse and realistic reasoning tasks to assess the long-context reasoning abilities of LLMs. To create the dataset efficiently and effectively, we first had human annotators collect short reasoning questions from the internet, cleaning them to avoid data contamination and forming the seed dataset. This seed dataset contains reasoning questions with diverse patterns from three major task categories: reading comprehension, logical inference, and mathematical word problems. We chose to use multiple-choice problems for easy evaluation, avoiding the use of LLMs or inaccurate metrics like Rouge score and F1 to assess the correctness of reasoning. Then, we utilize an automatic pipeline that synthesizes multi-hop long-context reasoning questions from the collected short-context problems. To ensure quality, we leverage LLMs to automatically verify the generated questions, ensuring they retain the same logic as their shorter counterparts. Ultimately, we retain 794 questions that pass these checks. For each question, we can generate long-context versions of arbitrary lengths; however, since most existing models support contexts up to 128K tokens, we focus our evaluation within this limit. This synthetic pipeline supports converting one short reasoning question into different lengths, enabling fine-grained assessment of LLMs across various context lengths and reasoning tasks. Figure 1: Overview of our pipeline for constructing LongReason. Givem a short reasoning question QshortQ_{\text{short}}, the pipeline first separates it into a background context CshortC_{\text{short}} and a final question II. Next, multiple paragraphs are synthesized from the background context CshortC_{\text{short}}. These synthesized paragraphs are then embedded within irrelevant passages to create a long-context background. Finally, the constructed context is combined with the final question to generate the long-context reasoning question QlongQ_{\text{long}}. To assess the current progress in the long-context reasoning abilities of existing LLMs, we evaluated 21 models of varying scales and architectures, sourced from both open-source and closed-source communities. While most of these models achieve near-saturated performance on previous synthetic tasks such as NIAH, nearly all exhibit significant performance degradation on LongReason as the context length increases. Further analysis reveals that even state-of-the-art LLMs show varying degrees of performance decline across different task categories, underscoring the importance of evaluating diverse reasoning tasks to fully understand the long-context reasoning capabilities of LLMs. Our key contributions are summarized as follows: • We present LongReason, a new synthetic long-context reasoning benchmark that encompasses a diverse range of task categories and supports controllable context lengths. • We propose an innovative synthesis algorithm that generates long-context reasoning questions from existing short questions, reducing the need for labor-intensive human annotation for long-context data. • We perform an extensive analysis of current LLMs, benchmarking their performance in long-context reasoning and offering valuable insights to enhance long-context reasoning capabilities. 2 Related Work Long-Context Large Language Models Recent advancements in deep learning system have significantly propelled the development of long-context large language models (LLMs). One of the key challenges in scaling these models is the quadratic time and space complexity inherent in computing self-attention over long sequences. To mitigate this computational burden, efficient self-attention algorithms [dao2022flashattention, dao2023flashattention, liu2023ring] have been introduced, reducing memory overhead, and novel training methods [li2021sequence, chen2023longlora] facilitate the training of these long-context models. As Rotary Position Embedding (RoPE) [su2024roformer] is widely used for positional encoding in many open-source models [llama3-1, qwen2.5, mistrallarge2], recent research [pi, xiong2023effective, peng2023yarn, liu20242, ding2024longrope, pose] has focused on adapting RoPE from pre-trained short-context models to effectively handle longer sequences. Moreover, new architectures [mamba, rwkv, bulatov2022recurrent, rmt, botev2024recurrentgemma] have been developed to efficiently process long-context inputs. Consequently, state-of-the-art language models [openai2024gpt4o, openai2024gpt4omini, reid2024gemini, anthropic2024claud35sonnet, llama3-1, mistrallarge2, qwen2.5, glm4] now support context windows ranging from 128K to millions of tokens, enabling the exploration of reasoning abilities over extensive contexts with LLMs. Figure 2: An illustrative example in LongReason. The original question is first decomposed into a separate background passage and an inquiry based on it. The inquiry includes keywords such as “Jack's father's age” and a time reference like “on a sunny afternoon” from the background passage, ensuring a clear connection to the passage. Subsequently, the background passage is expanded into multiple independent materials while preserving these key keywords. Finally, these independent materials are combined with some unrelated passages to create the final long-context reasoning question. Long-Context Benchmarks As the context window of current LLMs expands rapidly, numerous benchmarks have been proposed to evaluate their capabilities. In early benchmarks such as ZeroSCROLLS [zeroscrolls], L-Eval [leval], BAMBOO [bamboo], LongBench [bai2023longbench], and LooGLE [li2023loogle], the average input length remains under 25K tokens, which is far shorter than the context window size supported by existing LLMs. Recently, some research has begun to explore using synthetic datasets, which can support controllable context lengths, to evaluate the long-context abilities of LLMs. Needle-in-a-Haystack and its variants [needleinhaystack, zhao2024longagent] primarily evaluate retrieval abilities by inserting relevant information into extensive irrelevant corpora and testing the LLMs' capacity to extract it. Additionally, RULER [hsieh2024ruler] constructs synthetic tasks based on code-like flexible configurations to assess LLM performance over long contexts. While synthetic tasks can support the evaluation of arbitrarily long contexts, they are limited in scope, focusing on a narrow set of tasks and failing to comprehensively evaluate the reasoning abilities of LLMs in realistic scenarios. Other benchmarks like InfiniteBench [zhang2024infty] and Loong [wang2024leave] use human annotations to create questions from given long texts, which contain more diverse tasks but are both time-consuming and costly. Our proposed benchmark, LongReason, focuses on evaluating the long-context reasoning abilities of LLMs, which are created automatically from short reasoning questions without heavy human effort in reading the long context. We conduct a detailed comparison with existing benchmarks in Table 1. 3 Our Benchmark: LongReason In this section, we provide a detailed overview of LongReason, our synthetic long-context reasoning benchmark. This includes the problem formulation, the dataset construction process, and an analysis of the statistics of LongReason. 3.1 Long-context Reasoning Question Construction via Context Expansion Problem Formulation The primary goal of LongReason is to assess the long-context reasoning abilities of LLMs. To achieve this, we first define the reasoning task as follows: Given a reasoning question QQ, LLMs to need reason over QQ to produce a reasoning chain SS that leads to the final answer AA. In this work, the focus is on scenarios where the question QQ can be divided into a background context CC and a final inquiry II based on that context, denoted as Q=(C,I)Q=(C,I). In LongReason, the context CC can be long, comprising multiple paragraphs from diverse sources, while only a small subset of the information in the context CC is directly relevant to answering II. To simplify evaluation, LongReason employs close-ended multiple-choice questions for II. The dataset construction begins with a set of questions Qshort\textbf{Q}_{\text{short}}, consisting of questions QshortQ_{\text{short}} with relatively short question statements. For each QshortQ_{\text{short}}, our proposed context expansion pipeline utilizes LLMs to automatically generate a long-context version of the question, Qlong=(Clong,I)Q_{\text{long}}=(C_{\text{long}},I). The detailed construction pipeline is illustrated in Figure 1. Short-Context Reasoning Question Collection We begin by asking human annotators to create a dataset Qshort\textbf{Q}_{\text{short}}, comprising short questions QshortQ_{\text{short}} across various domains and diverse task categories. Annotators collect example questions from the internet and utilize an LLM to refine these questions, ensuring they are free from data contamination. To ensure that each short question require reasoning, we prompt an LLM to evaluate the number of reasoning steps in its corresponding ground-truth reasoning chain, denoted as S¯\bar{S}. We include only those questions that require at least two reasoning steps to arrive at the final answer in LongReason, thereby filtering out straightforward common-sense problems that lack significant reasoning depth. Automatic Short-Context Reasoning Question Decomposition with LLMs For each short reasoning question QshortQ_{\text{short}}, we prompt an LLM to decompose the question into a background context CshortC_{\text{short}} and an inquiry II. This decomposition needs to ensure that the final inquiry II is clearly linked to the background context CshortC_{\text{short}} , enabling the LLM to relate them and answer the inquiry based on the context. To have the better performance, we prompt the LLM to perform the decomposition in a chain-of-thought manner. Specifically, the LLM first extracts key elements such as keywords, time, main characters, and event names from the original short question and incorporates them into both the background context CshortC_{\text{short}} and the final inquiry II during the decomposition process. To ensure the quality of the decomposition, we introduce a self-verification stage after generating the decomposed question. We ask the LLM to verify whether the decomposed question, Qdecomposed=(Cshort,I)Q_{\text{decomposed}}=(C_{\text{short}},I), retains the same meaning as the original question QshortQ_{\text{short}}. For each question, we use a sampling temperature of 0.7 and generate up to 5 decompositions with the LLM. We retain only the decomposition that successfully passes the self-verification process conducted by the LLM. In our experiments, we found that over 99.34% of questions could be successfully decomposed within 5 samples, demonstrating the effectiveness of our question decomposition pipeline. Automatic Background Context Decomposition with LLMs To evaluate the ability to aggregate key information and reason across different positions within a long context, we further decompose the background context CshortC_{\text{short}} in the question QdecomposedQ_{\text{decomposed}} into multiple information pieces. Specifically, we use an LLM to first analyze all key information points within CshortC_{\text{short}} and then, for each information point, generate an independent and complete passage C′C^{\prime}. These generated passages retain certain keywords similar to those used during the question decomposition stage, ensuring that all passages are closely related to the final inquiry II. This process results in Cexpanded=(C1′,C2′,⋯)C_{\text{expanded}}=(C^{\prime}_{1},C^{\prime}_{2},\cdots) , where the passages are coherent and can be correctly associated with the final inquiry II. To ensure the quality of the expanded context, we introduce a self-verification stage. After generating the expanded question Qexpanded=(Cexpanded,I)Q_{\text{expanded}}=(C_{\text{expanded}},I), we prompt the LLM to verify whether QexpandedQ_{\text{expanded}} retains the same meaning as the original question QshortQ_{\text{short}}. For the background in the each question, we use a sampling temperature of 0.7 and generate up to 5 decompositions with the LLM. Only the decompositions that successfully pass the self-verification process are retained. In our experiments, we observed that over 94.67% of the background contexts were successfully decomposed within 5 samples. Automatic Background Decomposition with LLMs To evaluate the ability to aggregate key information and reason across different positions within a long context, we further decompose the background context CshortC_{\text{short}} in the question QdecomposedQ_{\text{decomposed}} into multiple information pieces. Specifically, we use an LLM to first analyze all key information points within CshortC_{\text{short}} and then, for each information point, generate an independent and complete passage C′C^{\prime}. These generated passages retain certain keywords similar to those used during the question decomposition stage, ensuring that all passages are closely related to the final inquiry II. This process results in C¯expanded=(Ce1,Ce2,⋯)\bar{C}_{\text{expanded}}=(C_{e}^{1},C_{e}^{2},\cdots) , where the passages are coherent and can be correctly associated with the final inquiry II. To ensure the quality of the expanded context, we introduce a self-verification stage. After generating the expanded question Qexpanded=(C¯expanded,I)Q_{\text{expanded}}=(\bar{C}_{\text{expanded}},I), we prompt the LLM to verify whether QexpandedQ_{\text{expanded}} retains the same meaning as the original question QshortQ_{\text{short}}. For the background context in the each question, we use a sampling temperature of 0.7 and generate up to 5 decompositions with the LLM. Only the decompositions that successfully pass the self-verification process are retained. In our experiments, we observed that over 94.67% of the background contexts were successfully decomposed within 5 samples. Figure 3: The number of reasoning steps in the ground-truth analysis for questions in LongReason. Models Length Q-O Q-E 8K 16K 32K 64K 128K Avg. Random - 25.21 25.21 25.21 25.21 25.21 25.21 25.21 25.21 closed-source models Gemini-1.5 Pro - 90.42 84.11 77.81 79.70 77.81 78.94 78.81 78.56 Gemini-1.5 Flash - 90.16 80.20 75.91 76.29 75.79 75.66 76.92 75.91 GPT-4o - 90.42 85.62 77.30 76.80 74.91 74.02 73.39 75.76 GPT-4o mini - 79.95 74.40 68.73 66.83 65.45 62.67 61.66 65.92 Claude-3.5 Sonnet - 84.36 78.18 73.01 70.11 68.47 68.22 65.95 69.95 Claude-3.5 Haiku - 77.05 71.75 64.44 64.44 63.93 60.03 59.90 63.21 open-source models Llama-3.1-70B 128K 80.83 74.27 68.22 66.46 61.16 63.30 48.30 64.78 Llama-3.1-8B 128K 58.13 57.12 53.47 51.20 51.45 49.94 46.53 51.52 Mistral Large 2 128K 83.73 81.97 72.89 70.11 64.69 52.46 0.00 00footnotemark: 0 65.04 Mixtral 8x22B 64K 64.69 63.30 50.95 52.21 49.31 48.68 - 50.29 Mistral Nemo 1M 56.12 52.96 50.57 43.00 42.37 38.21 29.51 43.54 Mistral Small 32K 50.32 64.94 56.75 50.32 37.70 - - 48.26 Mistral-7B 32K 41.61 40.86 44.77 43.25 42.75 - - 43.59 Qwen2.5-72B 128K 89.16 85.75 76.67 77.43 74.27 74.53 69.48 75.72 Qwen2.5-32B 128K 84.24 81.59 78.44 74.91 72.76 71.75 67.34 74.46 Qwen2.5-14B 128K 84.11 76.17 71.88 70.87 68.10 66.20 62.30 69.26 Qwen2.5-7B 128K 76.42 73.01 66.33 62.42 62.17 58.76 54.22 62.42 Qwen2.5-3B 32K 61.29 58.76 49.81 49.56 45.65 - - 48.34 Phi-3.5-MoE 128K 65.32 66.46 48.42 56.37 53.72 48.80 49.56 51.83 Phi-3.5-mini 128K 55.99 60.53 50.69 48.80 49.81 45.40 24.97 48.68 glm-4-9b 128K 59.90 63.43 48.68 46.15 44.14 38.97 39.60 44.48 Table 2: Performance (%) of selected LLMs on LongReason. All the scores are computed by averaging the accuracy across 794 questions in LongReason. Q-O represents the performance of the original short question QshortQ_{\text{short}}, and Q-E denotes the performance of the expanded question QexpandedQ_{\text{expanded}} mention in Section 3.1. For long-context questions, the final inquiry is placed after the background context, positioning it at the end of the context. The average score (Avg.) represents the mean performance across context lengths spanning from 8K to 128K. Long-Context Reasoning Question Construction Through Context Expansion Finally, we construct the long-context version of each question by embedding each passage CeiC_{e}^{i} from the expanded context C¯​expanded\bar{C}{\text{expanded}} at random positions within a set of irrelevant passages C¯​irrelevant\bar{C}{\text{irrelevant}}, forming the final long-context reasoning questions. To create C¯​irrelevant\bar{C}{\text{irrelevant}} , we first collect passages from the Pile [gao2020pile] and use an LLM to rewrite each passage to minimize stylistic differences between the synthesized background passages and the irrelevant passages. These rewritten passages are then compiled to form the set of irrelevant passages C¯irrelevant\bar{C}_{\text{irrelevant}}. In LongReason, GPT-4 is used for all data synthesis and self-verification. For each question, we evaluate multiple versions of the synthesized question for comparison, including the original question QshortQ_{\text{short}}, the expanded version QexpandedQ_{\text{expanded}}, and long-context versions with context lengths ranging from 8​K8K to 128​K128K. Furthermore, similar to NIAH [needleinhaystack], our pipeline is capable of generating reasoning questions with even longer contexts by incorporating additional irrelevant information. 3.2 The Statistics of LongReason 00footnotetext: Mistral Large 2 generates an empty response when the question context length reaches 128K. LongReason comprises 794 multiple-choice reasoning questions encompassing diverse reasoning patterns across three task categories: 280 reading comprehension questions, 347 logical inference questions, and 167 mathematical word problems. We only keep the questions that require at least 2 reasoning steps, the reasoning steps of the questions range from 2 to more than 10 reasoning steps. The average reasoning steps of the questions is 4.47. More detailed statistics of the number of the reasoning steps are shown in Figure 3. 4 Exerperiments & Results We conduct a comprehensive set of experiments to evaluate a broad set of LLMs using LongReason. In this section, we present the experimental setup, main results, and additional analysis. 4.1 Experimental setup Models & Inference Setup We select a set of representative LLMs that support long context windows, including 6 closed-source models from 3 model families (GPT, Gemini and Claude) and 15 open-source models spanning a wide range of model sizes (3B to 123B) and claimed context lengths (8K to 2M). Detailed information about these models can be found in Appendix 6. For open-source models, we utilize vLLM [vllm], which enables efficient KV cache memory management during inference time. All inferences are performed using bfloat16 precision on 8 NVIDIA A100 GPUs with greedy decoding (temperature=0). Evaluation setup We evaluate all models on LongReason, which comprises 794 questions, each featuring multiple variations, including the original version, expanded versions, and long-context versions with context lengths of 8K, 16K, 32K, 64K, and 128K. Each input is constructed using a predefined zero-shot chain-of-thought template that combines the background context, followed by the corresponding final inquiry. Detailed information about the prompt template is provided in Appendix 9. To assess the reasoning performance of the LLMs, we extract the predicted choice by identifying the first character following the phrase “the answer is” and compare it to the ground-truth option for accuracy. 4.2 Main Results The results of 21 LLMs are presented in Table 2. From the table, we first observe a significant performance drop across nearly all models when evaluated on QexpandedQ_{\text{expanded}} compared to QshortQ_{\text{short}}. To ensure this decline is not caused by the quality of the synthetic questions, we manually examine 20 failure cases from Gemini-1.5 Pro, where correct answers on QshortQ_{\text{short}} turn incorrect on QexpandedQ_{\text{expanded}}. Only 3 cases involve ambiguity or errors introduced by context expansion. Similarly, when comparing QexpandedQ_{\text{expanded}} to Q8​KQ_{8K} , a large performance drop persists. Among 20 failure cases from Gemini-1.5 Pro where correct answers on Q8​KQ_{8K} turn incorrect on QexpandedQ_{\text{expanded}}, only 2 cases are affected by added irrelevant information. For long-context reasoning performance, Gemini-1.5 Pro outperforms all other closed-source models, exhibiting negligible performance drop when extending the context length from 8K to 128K. In contrast, the long-context reasoning capabilities of open-source LLMs lag behind those of the most advanced closed-source models in LongReason. For example, the best-performing open-source model, Qwen2.5-72B, experiences a significant performance drop (5.05%) when the input context length increases from 64K to 128K. Furthermore, a comparison of Qwen2.5 models of different sizes, shown in Figure 4, reveals that performance declines at a similar rate across all model sizes as context length increases. Smaller models perform worse overall, primarily due to their weaker reasoning abilities, even in shorter-context scenarios. 4.3 Further Analysis We conduct further analysis on LongReason to provide a deeper understanding of the long-context reasoning performance of existing LLMs. Does the position of the final inquiry influence model performance? As shown in Table 3, the performance of state-of-the-art language models is highly sensitive to the position of the final inquiry. Although Gemini-1.5 Pro demonstrates excellent long-context reasoning performance when the final inquiry is placed after the background context, it still struggles when the inquiry is positioned at the beginning of the input, before the background context. Meanwhile, GPT-4o demonstrates similar performance in both cases, particularly when the context length is short. However, as the input length increases, GPT-4o's performance declines significantly for questions with the final inquiry is placed before the background context. Model 8K 16K 32K 64K 128K I-L I-F I-L I-F I-L I-F I-L I-F I-L I-F GPT-4o 77.30 75.41 76.80 72.89 74.91 69.10 74.02 65.32 73.39 65.95 Gemini-1.5 Pro 77.81 68.22 79.70 70.37 77.81 68.60 78.94 66.96 78.81 66.71 Claude-3.5 Sonnet 73.01 68.60 70.11 67.84 68.47 66.46 68.22 64.69 65.95 66.20 Table 3: Ablation study on the position of the final inquiry for selected models evaluated at context lengths ranging from 8K to 128K. I-L represents questions where the final inquiry is placed after the background context, while I-F represents questions where the inquiry is placed before the background context. Figure 4: Performance of the Qwen2.5 series on LongReason, with model sizes ranging from 7B to 72B. Figure 5: Comparison of the long-context reasoning performance between Gemini-1.5 Pro and Claude 3.5-Sonnet across different task categories. In the figure, the dotted line represents the single-hop version of the synthesized questions, where all clues are placed together in the context. The solid line represents the multi-hop version, which is the standard format used in LongReason, where clues are distributed separately throughout the context. Do LLMs have similar long-context reasoning performance over different tasks and clue placement in LongReason? As shown in Figure 5, both Gemini-1.5 Pro and Claude 3.5 demonstrate strong long-context reasoning performance on reading comprehension problems. However, for logic and math problems, Claude 3.5 significantly underperforms compared to Gemini. Additionally, we observe that for these problem types, Claude 3.5 shows much lower performance when the clues are distributed separately throughout the context, compared to when the clues are grouped together. Figure 6: An example illustrating how Gemini-1.5 Pro provides incorrect reasoning for a long-context question but correct reasoning for the original short question. The key difference in reasoning is underlined in the figure. Error Cases Analysis We analyze 20 randomly sampled error cases from Gemini-1.5 Pro on questions with a 128K context. Among these, we find only 3 instances where the errors are caused by missing critical information in the background context during reasoning, while the remaining cases were attributed to reasoning errors. A detailed example is provided in Figure 6. 5 Conclusion and Limitations In this work, we introduce LongReason, a synthetic reasoning benchmark designed to evaluate the long-context reasoning capabilities of large language models (LLMs). Using LongReason, we evaluate the long-context reasoning performance of 21 LLMs across context sizes ranging from 8K to 128K. Our experiments and analyses reveal that existing LLMs still have significant room for improvement in delivering robust long-context reasoning. Additionally, several limitations of LongReason remain, as discussed below. Lack of evaluation for complex reasoning Current LongReason primarily focuses on evaluating reasoning questions that require only a few reasoning steps. However, this is insufficient to fully understand the performance of LLMs when dealing with challenging problems that demand many reasoning steps over a long context. Lack of evaluation for tasks requiring full context Similar to most existing work, LongReason focuses on tasks that do not require understanding the entire contexts for finishing the tasks. All the questions in LongReason are derived from short reasoning problems that can be solved by examining only a small portion of the context. \beginappendix 6 Model Information We select in total 21 large language models (LLMs) for evaluation and analysis. We only include the aligned models including 6 clouse-source models like GPT-4o, Gemini-1.5, and Claude-3.5 and also 17 open-source models with dense and MoE architectures like Llama and Mixtral using LongReason. Model Aligned Size Context Length Huggingface [huggingface] / API GPT-4o [openai2024gpt4o] ✓ - 128K gpt-4o-2024-08-06 GPT-4o-mini [openai2024gpt4omini] ✓ - 128K gpt-4o-mini-2024-07-18 Gemini-1.5-Pro [gemini] ✓ - 2M gemini-1.5-pro-002 Gemini-1.5-Flash [gemini] ✓ - 2M gemini-1.5-flash-002 Claude-3.5-Sonnet[anthropic2024claud35sonnet] ✓ - 200K claude-3-5-sonnet-20240620 Claude-3.5-Haiku[anthropic2024claud35sonnet] ✓ - 200K claude-3-5-haiku-20241022 Mistral-Large2 [mistrallarge2] ✓ 123B 128K mistralai/Mistral-Large-Instruct-2407 Mixtral-8 ×\times22B [jiang2024mixtral] ✓ 39B/8 ×\times22B 64K mistralai/Mixtral-8x22B-Instruct-v0.1 Mistral-Small [mistraltec] ✓ 22B 32K mistralai/Mistral-Small-Instruct-2409 Mistral-Nemo [mistralnemo] ✓ 12B 1M mistralai/Mistral-Nemo-Instruct-2407 Mistral-7B [mistral] ✓ 7B 32K mistralai/Mistral-7B-Instruct-v0.3 Llama3.1 [llama3-1] ✓ 70B 128K meta-llama/Meta-Llama-3.1-70B-Instruct Llama3.1 [llama3-1] ✓ 8B 128K meta-llama/Meta-Llama-3.1-8B-Instruct Qwen2.5 [qwen2.5] ✓ 72B 128K Qwen/Qwen2.5-72B-Instruct Qwen2.5 [qwen2.5] ✓ 32B 128K Qwen/Qwen2.5-32B-Instruct Qwen2.5 [qwen2.5] ✓ 14B 128K Qwen/Qwen2.5-14B-Instruct Qwen2.5 [qwen2.5] ✓ 7B 128K Qwen/Qwen2.5-7B-Instruct Qwen2.5 [qwen2.5] ✓ 3B 32K Qwen/Qwen2.5-3B-Instruct GLM4-9B [glm4] ✓ 9B 128K THUDM/glm-4-9b-chat Phi3.5-MoE [phi3] ✓ 6.6B/16 ×\times3.8B 128K microsoft/Phi-3.5-MoE-instruct Phi3.5-mini [phi3] ✓ 14B 128K microsoft/Phi-3.5-mini-instruct Table 4: Information of evaluated and analyzed models in LongReason. 7 Hyperparameters for LongReason Construction In LongReason, we utilize gpt-4o-2024-08-06 to synthesize our dataset, with the total cost of creating the datasets being under $200. 8 Human Annotator To create the short reasoning questions with human annotations, we trained five researchers from our research group following the requirements outlined in the paper. 9 Prompts ## Question {question} ## Instruction Please break down the question below into a background passage and a question related to this background passage according to the following requirements: 1. The background passage should contain a complete context and retain as much of the original wording as possible, but do not add any extra facts, as this may affect the correct answer to the final question. Do not include options in the background passage! 2. Since the background passage will be mixed in with a large amount of other materials, the question about the background passage needs to have a clear signal pointing to this specific background passage. This signal is preferably the name of a character, location, or event from the background passage, but it should not include the core content of the story and should contain as little information from the background passage as possible. 3. Do not use any pronouns in the question about the background passage. 4. If the original question contains options, the question about the background passage should also include options. When writing options, list each option on a new line, using letters such as A, B, C, D, and E as labels. Do not include any blank options from the original question. 5. Please write the background passage and the question about the background passage in English. ## Answer Format ### Analysis Please provide a brief analysis of how to perfectly break down this question into a background passage and a question related to this passage, particularly focusing on what signal should be used in the question to refer to the background passage. ### Broken Down Question #### Background Passage A background passage that meets my requirements. Please use English and do not include any additional information. #### Question About the Background Passage A question about the background passage that meets my requirements. Please use English and do not include any additional information. ### Analysis of the Question About the Background Passage Briefly analyze the question about the background passage and judge whether, if the background passage is mixed in with a large amount of other materials, you would be able to find the related background passage after reading the "question about the background passage." ### Judge Whether the Question About the Background Passage Meets the Requirements Yes or No. Do not provide any additional information. ### Analyze Whether the Broken Down Question Matches the Original Question in Meaning Briefly analyze whether the broken down question is consistent in meaning with the original question. ### Judge Whether the Broken Down Question Matches the Original Question in Meaning Yes or No. Do not provide any additional information. ## Respond to My Instructions According to the Above Format Table 5: Zero-shot prompt for separating a short reasoning quesiton into a background context and a final inquiry. ## Material {context} ## Question about the Material {final_question} ## Instructions Please expand the above material into multiple independent paragraphs with around 200 words in English, while meeting the following requirements. 1. Ensure that every key piece of information from the material appears in one paragraph of the expanded text. Try to place the key information in the middle of the expanded paragraphs. 2. The expanded material need to avoid introducing additional knowledge, reasoning, or any content that might influence the answer to the "Question about the material." 3. The expanded text should clearly relate to the "Question about the material." Please include hints or references to the question within each paragraph to maintain this connection. However, do not use words like "question" or "query" explicitly in the expanded text. 4. Each paragraph should be a standalone piece of text, comprehensible without needing to refer to other paragraphs. Minimize the use of pronouns, particularly those referring to other paragraphs. 5. Do not reference or imply any possible answer choices that might be part of the "Question about the material." 6. The style of the expanded text should match the specified target genre: {target_genre}. ## Response Format ### Analysis Please analyze how to appropriately add background information to expand the material into multiple independent paragraphs based on the given requirements. Additionally, assess whether the provided material can be easily divided into multiple paragraphs. If not please only provide only one paragraph in the expanded material. ### Expanded material Present the expanded material as a series of independent paragraphs that meet the above requirements. Add the index like "1." (do not use any format here) at the beginning of each paragraph (starting from 1), use English, and do not include any extra information. ## Respond to My Instructions According to the Above Format Table 6: Zero-shot prompt for expanding the given short context into several independent passages. ## Original material {context} ## Question about the material {final_question} ### Expanded material {expanded_context} ## Instructions Please compare the expanded material with the original material and answer the following questions: 1. Does the expanded material contain all the key information from the original material? 2. Does the expanded material contain information that will affect the answer to the question? 3. Do you think all the paragraphs in the expanded material are related to the main topic/character of the question? ## Response Format ### Analysis Please analyze the expanded material and compare it with the original material. Then, combine the question and analyze the three questions above. ### Does the expanded material contain all the key information from the original material based on the analysis? Yes or No. Do not provide any additional information. ### Does the expanded material contain information that will affect the answer to the question based on the analysis? Yes or No. Do not provide any additional information. ### Are all the paragraphs in the expanded material are related to the main topic/character of the question based on the analysis? Yes or No. Do not provide any additional information. ## Respond to My Instructions According to the Above Format Table 7: Zero-shot prompt for assessing the quality of the synthesized background context. ### Question {question} Please answer the above question based on the background information! ### Answer Please analyze step by step, and provide the final answer in the last line using "The answer is" + option (represented by ABCDE)! Table 8: Zero-shot chain-of-thought prompt for answering the given question. ### Background Information {context} ### Question about the Background Information {final_question} Please answer the above question based on the background information! ### Answer Please analyze step by step, and provide the final answer in the last line using "The answer is" + option (represented by ABCDE)! Table 9: Zero-shot prompt chain-of-thought for answering the given question based on the background context. Generated on Mon Nov 17 22:00:05 2025 by LaTeXML \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2502.13127.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2502.13127.txt new file mode 100644 index 0000000..692b45d --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2502.13127.txt @@ -0,0 +1 @@ +Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Related Work 3 Methodology 3.1 Problem Formulation 3.2 LongFinanceQA Dataset 3.3 Supervised CoT Reasoning 4 Experiments 4.1 Experimental Setup 4.2 Main Results 4.3 Discussion 5 Conclusion References A Appendix A.1 Trend of Long-Context LLMs A.2 Prompts of Property-based Agentic Inference (PAI) A.3 LongFinanceQA Dataset Statistics A.4 Synthetic Question Generation A.5 More Empirical Results A.6 Full Performances License: arXiv.org perpetual non-exclusive license arXiv:2502.13127v2 [cs.CL] 01 Dec 2025 Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning Jingyang Lin Andy Wong Affiliation: University of Rochester, NY, United States Affiliation: PAII Inc., CA, United States Tian Xia Affiliation: PAII Inc., CA, United States Shenghua He Affiliation: PAII Inc., CA, United States Hui Wei Affiliation: PAII Inc., CA, United States Mei Han Affiliation: PAII Inc., CA, United States Jiebo Luo Affiliation: University of Rochester, NY, United States Affiliation: University of California, Merced, CA, United Stateshttps://long-pai.github.io/ Abstract Recent advances in Large Language Models (LLMs) have enabled them to process increasingly longer sequences, ranging from 2K to 2M tokens and even beyond. However, simply extending the input sequence length does not necessarily lead to effective long-context understanding. In this study, we integrate Chain-of-Thought (CoT) reasoning into LLMs in a supervised manner to facilitate effective long-context understanding. To achieve this, we introduce LongFinanceQA, a synthetic dataset in the financial domain designed to improve long-context reasoning. Unlike existing long-context synthetic data, LongFinanceQA includes intermediate CoT reasoning before the final conclusion, which encourages LLMs to perform explicit reasoning, improving accuracy and interpretability in long-context understanding. To generate synthetic CoT reasoning, we propose Property-based Agentic Inference (PAI), an agentic framework that simulates human-like reasoning steps, including property extraction, retrieval, and summarization. We evaluate PAI's reasoning capabilities by assessing GPT-4o-mini w/ PAI on the Loong benchmark, outperforming standard GPT-4o-mini by 20.0%. Furthermore, we fine-tune LLaMA-3.1-8B-Instruct on LongFinanceQA, achieving a 28.0% gain on Loong's financial subset. ††footnotetext: Corresponding author: 🖂 jlin81@ur.rochester.edu.††footnotetext: Work was done during the internship at PAII Inc. 1 Introduction Figure 1: The hype of long-context large language models. The results shown are from the Loong benchmark, where green points refer to open-source LLMs and red points indicate closed-source LLMs. Our GPT-4o-mini w/ PAI stands out as the red open circle. Long context understanding remains an evolving challenge Kočiský et al. (2018); Wu et al. (2021); Bai et al. (2024); Wang et al. (2024) in natural language processing (NLP). Achieving long-context understanding requires processing long-form textual information, thereby enhancing a model's ability to generate coherent, accurate, and contextually relevant responses IBM Research (2024). Practical long context understanding has a potential impact on numerous applications, such as private document analysis Mukherjee et al. (2023), large codebase understanding Nam et al. (2024), and multimodal content understanding Chandrasegaran et al. (2024); Lin et al. (2023); Tang et al. (2023); Chen et al. (2024c). Recent advancements in large language models Ouyang et al. (2022); Reid et al. (2024); Dubey et al. (2024) have significantly extended the input sequence length, ranging from 2K to 2M tokens, as shown in Figure 7 in Appendix A. However, simply increasing the input sequence length does not necessarily improve the ability to comprehend the long content Yang et al. (2024a); Goldman et al. (2024). In particular, Figure 1 presents the performance of various long-context LLMs on the Loong benchmark Wang et al. (2024). The results suggest that, regardless of the maximum sequence length models can process, their performance remains similar, typically between 45% and 55%. This phenomenon exposes a hype that despite considerably enlarging context window size, the state-of-the-art LLMs still fail to perform satisfactorily in practical long-context problem-solving tasks. Therefore, instead of merely increasing the input sequence length, achieving effective long-context understanding remains an open challenge. In parallel with advanced long-context architectures and techniques Peng et al. (2024); Liu et al. (2024c); Dao and Gu (2024), constructing high-quality long-context training data remains essential yet underexplored. Given the scarcity and high annotation costs of long-context data, generating high-quality synthetic data for long-context modeling is valuable and urgent. Early attempts Raffel et al. (2020); Fu et al. (2024) simply pack all short-length data into long-context chunks without considering document boundaries. Later works Zhang et al. (2024a); He et al. (2024) have introduced long-context QA tasks from both single- and multi-source perspectives. Qwen-Agent Yang et al. (2024a) develops an agentic system to further improve the quality of synthetic answers. However, existing synthetic long-context data typically pair challenging questions with brief final answers for model training. This way overlooks a key difference between long-context and traditional QA tasks: practical long-context questions often require multi-step reasoning throughout the long content. Without intermediate reasoning, LLMs struggle to learn effective patterns from the paired complex questions and brief answers. We hypothesize that directly guiding models to generate brief answers without intermediate reasoning steps for long-context modeling will lead to suboptimal training. Instead, incorporating intermediate reasoning into synthetic data will help LLMs learn effective patterns and enhance training optimization. To validate this hypothesis, we introduce LongFinanceQA, a novel long-context synthetic dataset constructed using financial data. Each sample in this dataset consists of a practical long-context question and the corresponding augmented answer along with intermediate chain-of-thought (CoT) reasoning steps. In particular, we first collect 6,911 bilingual financial annual reports (i.e., English and Chinese) published before 2022. We then build a financial metric pool comprising key metrics commonly found in these reports, such as profit, cash flow, and debt. Based on these financial metrics, we generate 46,457 long-context questions that require single- or multi-source evidence. To generate reliable reasoning-augmented answers, we propose Property-based Agentic Inference (PAI), a comprehensive agentic framework. PAI leverages LLM-based agents to simulate human-like reasoning and operates in three steps: 1) a property extraction agent decomposes complex queries by identifying key properties, where each property consists of a metric and the corresponding subject; 2) a property-based retrieval agent retrieves relevant information from long documents for each identified property, and then generates intermediate findings by leveraging the property-based retrieved content; 3) a summarization agent synthesizes the intermediate findings into a coherent conclusion. The synthetic reasoning results are formed by integrating outputs from the property extraction and retrieval stages, while the conclusion is derived from the summarization stage. Finally, LongFinanceQA comprises 46,457 long-context QA pairs with CoT reasoning over 6,911 financial reports. Although PAI performs human-like reasoning in long-context scenarios, it relies on human-crafted design and multi-step inference. To simplify this inference process, we intend to transfer PAI's long-context reasoning ability to a large language model, LLaMA-3.1, via supervised fine-tuning on LongFinanceQA. The enhanced model, LongPAI, leverages CoT reasoning to handle long-context problems in a single step. This fine-tuning procedure is termed as Supervised CoT Reasoning. Empirically, we first evaluate the outcome of PAI on the Loong benchmark Wang et al. (2024), involving challenging long-context tasks on three different domains. The results show that equipping GPT-4o-mini with PAI achieves a substantial 20% improvement on the Loong as shown in Figure 1. Moreover, the effectiveness of PAI guarantees the quality of the synthetic data in LongFinanceQA. Meanwhile, the enhanced LLaMA-3.1 model, LongPAI, achieves a 28.0% improvement on the Financial subset of Loong. Notably, in several scenarios, LongPAI even surpasses its teacher model PAI. This phenomenon emphasizes the importance of long-context modeling, contradicting recent arguments that the long-context problem can be solved by short language models Qian et al. (2024); Chen et al. (2024b). Our main contributions are three-fold: 1) we introduce LongFinanceQA, a long-context synthetic dataset for fine-tuning with 46,457 QA pairs featuring high-quality CoT reasoning from 6,911 bilingual financial annual reports; 2) to generate reasoning-augmented answers, we propose an agentic framework, Property-based Agentic Inference, to mimic human behaviors; and 3) empirical results validate the effectiveness of PAI and supervised CoT reasoning, the quality of LongFinanceQA, and the importance of long-context modeling. 2 Related Work Long-context Synthetic Data. The scarcity of well-annotated long-context data makes high-quality synthetic data generation a valuable research direction. Early works Raffel et al. (2020); Fu et al. (2024) concatenate short data into long-context fragments without considering the boundaries of the document. Large World Model Liu et al. (2024b) addresses this boundary issue with masked sequence packing to keep attention within documents. Subsequent studies Zhang et al. (2024a); He et al. (2024) construct single- and multi-source long-context QA pairs requiring evidence retrieval across multiple document positions. Qwen2-Agent Yang et al. (2024a) leverages multiple agents to enhance answer quality. Beyond conventional QA pairs, our study augments answers with intermediate reasoning steps, explicitly guiding language models in learning reasoning abilities for practical long-context scenarios. Long-context Large Language Models. Two main approaches enhance long-context problem-solving: reduction-based and extension-based methods. Reduction-based methods compress input by preserving essential information, allowing LLMs to focus on relevant content. Techniques like Retrieval-Augmented Generation (RAG) Lewis et al. (2020) and task decomposition for book summarizationWu et al. (2021) follow this approach. Extension-based methods expand the context window directly. RoPE Su et al. (2024) introduces a foundational positional encoding widely used in long-context LLMs Peng et al. (2024); Yang et al. (2024a); Dubey et al. (2024). Parallelism techniques Ren et al. (2021); Liu et al. (2024c) scale context capacity, enabling fully fine-tuning for long-context LLMs. This study leverages parallelism techniques to achieve long-context training. Chain-of-Thought Resoning. The CoT technique improves the reasoning abilities of language models by incorporating intermediate reasoning steps. Early works Wei et al. (2022) present that prompting LLMs with step-by-step reasoning significantly improves performance on complex reasoning tasks. Following works Zelikman et al. (2022); Yao et al. (2024) explore structured CoT approaches, such as tree-based and self-consistent reasoning. This work incorporates CoT reasoning steps into fine-tuning data instead of well-crafted prompts. This approach enables the augmented data to explicitly supervise language models in reasoning skills. Agentic RAG. Agentic RAG frameworks can be grouped into two paradigms Liang et al. (2025): predefined reasoning and agentic reasoning RAG. Our proposed PAI framework is a hybrid of the two paradigms. Like predefined RAG systems Press et al. (2023); Sarthi et al. (2024), PAI follows a structured, multi-step reasoning workflow. However, the components of PAI employ the techniques used in agentic reasoning RAG. Concretely, Property Extraction Agent and Proper-based Retrieval Agent use prompt-based approaches (i.e., function calling Eleti et al. (2023)) to generate structured outputs. Agentic RAG systems Li et al. (2024); Yao et al. (2023); Eleti et al. (2023) typically treat follow-up queries as free-form text, which invites query drift, irrelevant results, and hard-to-audit reasoning. Different from agentic RAG, PAI instead converts each sub-query into a well-formed property, enabling verifiable retrieval, interpretable reasoning traces, and compact supervised training. Beyond agentic RAG, this study further investigates transferring the capability of long-context reasoning into a lightweight language model via supervised CoT, resulting in the LongPAI model. 3 Methodology In this section, we first introduce the problem formulation of the QA task enhanced by intermediate CoT reasoning in Section 3.1. Then, we present the procedure of LongFinanceQA dataset construction in Section 3.2. After that, we describe the fine-tuning details in Section 3.3. Figure 2: Overview of Property-based Agentic Inference (PAI), containing three stages. A Property Extraction Agent identifies key properties 𝒑i\mbox{\boldmath$p$}_{i} from the given query 𝐐\mathbf{Q}, where each property consists of a measurable metric and its corresponding subject. Given the selected properties, a Property-based Retrieval Agent first transforms each property into a sub-query 𝒒i\mbox{\boldmath$q$}_{i} to retrieve relevant content chunks from long documents, yielding intermediate findings. A Summarization Agent integrates these intermediate findings to generate a comprehensive conclusion 𝐀\mathbf{A}. After finishing PAI, we incorporate the output from the above three agents to produce reasoning-augmented answers. These augmented answers serve as the core contribution of the LongFinanceQA. 3.1 Problem Formulation Traditional QA tasks require language models to generate outputs 𝐀\mathbf{A} directly from a given query 𝐐\mathbf{Q} and the corresponding input content 𝐗\mathbf{X} by modeling the conditional probability: pθ​(𝐀|𝐗,𝐐),p_{\theta}(\mathbf{A}|\mathbf{X},\mathbf{Q}), (1) where θ\theta is the parameter of the language models. Compared to traditional QA tasks, practical long-context QA tasks often require intermediate reasoning steps to analyze multiple pieces of evidence across long documents, and then derive the final answer. Thus, we formalize long-context QA tasks as a joint conditional probability of intermediate reasoning results 𝐑\mathbf{R} and the final answer 𝐀\mathbf{A}: pθ(𝐑,𝐀|𝐗,𝐐).p_{\theta}(\mathbf{R},\mathbf{A}|\mathbf{X},\mathbf{Q}). (2) As shown in Eq.(2), this study aims to facilitate long-context understanding by guiding LLMs to first predict intermediate CoT reasoning steps before generating the final answer. To achieve this, we construct a long-context synthetic dataset that explicitly supervises language models in learning intermediate reasoning. The data construction process can be factorized as follows: pθ(𝐑,𝐀|𝐗,𝐐)=pθ(𝐑|𝐗,𝐐)⋅pθ(𝐀|𝐗,𝐐,𝐑),p_{\theta}(\mathbf{R},\mathbf{A}|\mathbf{X},\mathbf{Q})=p_{\theta}(\mathbf{R}|\mathbf{X},\mathbf{Q})\cdot p_{\theta}(\mathbf{A}|\mathbf{X},\mathbf{Q},\mathbf{R}), (3) where pθ​(𝐑|𝐗,𝐐)p_{\theta}(\mathbf{R}|\mathbf{X},\mathbf{Q}) indicates the generation of intermediate reasoning steps 𝐑\mathbf{R} given a query 𝐐\mathbf{Q} and input content 𝐗\mathbf{X}, and then pθ​(𝐀|𝐗,𝐐,𝐑)p_{\theta}(\mathbf{A}|\mathbf{X},\mathbf{Q},\mathbf{R}) is the process of incorporating the query and summarizing reasoning steps to produce the answer 𝐀\mathbf{A}. The data construction follows the principle of Eq.(3). 3.2 LongFinanceQA Dataset LongFinanceQA dataset is designed to generate practical long-context QA pairs with reasoning steps to effectively analyze long content. The finance domain was chosen for several reasons. First, annual financial reports are readily accessible and present complex long-context reasoning challenges. Moreover, finance is a data-driven field where accurate insights can drive critical decisions, making advancements in AI particularly valuable for real-world applications. We will introduce the data construction pipeline of LongFinanceQA as follows. Data Collection. To begin, we collect bilingual financial annual reports (i.e., English and Chinese) dated before 2022 from open-source official websites, specifically the SEC-10-K11 1 https://www.sec.gov/ and cninfo22 2 http://www.cninfo.com.cn/ platforms. Documents from companies included in the Loong benchmark are then excluded. Next, we filter reports based on a token length range of 20K to 80K, determined using the GPT-4o tokenizer. Moreover, we prioritize companies with consistent annual reports over the years. In the end, 6,911 bilingual financial reports are selected. Diverse Query Generation. We first construct a financial metric pool containing key metrics commonly found in financial reports, such as profit, revenue, and cash flow. Then, we randomly pick several metrics in the metric pool and select a combination of financial reports. Given these financial metrics and the metadata from the selected documents (e.g., company name and year), we generate various long-context questions that require either single- or multi-source evidence. Next, we filter out questions whose corresponding combined documents exceed 256K tokens, as this surpasses the maximum token limit of our model. Finally, we obtain 46,457 practical long-context questions. Please refer to Appendix A for more details. Augmented Answer Generation with CoT Reasoning. Given long-context questions and their corresponding documents, our goal is to generate answers with CoT reasoning, following the principle of Eq. (3). In particular, we first generate step-by-step reasoning based on the question and documents, then integrate the reasoning steps into a conclusion. Generally, long-context questions are challenging as they require models to retrieve multiple pieces of evidence scattered throughout long content and then integrate them for a global understanding Wang et al. (2024); Edge et al. (2024). Inspired by this phenomenon, we intend to first extract key evidence points that support answering the question, then retrieve relevant information based on these points, and finally aggregate them into the conclusion. We term these supporting evidence points as “Properties”. Following this methodology, we propose the Property-based Agentic Inference (PAI) framework, illustrated in Figure 2, containing three steps: (1) property extraction, (2) property-based retrieval, and (3) summarization. Step 1: Property Extraction. This step focuses on extracting a set of properties {𝒑i}i=1Np\{\mbox{\boldmath$p$}_{i}\}^{N_{p}}_{i=1} from the given query 𝐐\mathbf{Q}, where NpN_{p} denotes the number of properties. Each property consists of a metric (a measurable factor being analyzed) and its corresponding subject mentioned in the query 𝐐\mathbf{Q}. For instance, given the query shown in Figure 2, the metric is “revenue” and subjects are “'s annual reports from different years (e.g., 2018, 2020, and 2022)”, which serve as the sources of information to determine the trend. Step 2: Property-based Retrieval. After extracting the properties {𝒑i}i=1Np\{\mbox{\boldmath$p$}_{i}\}^{N_{p}}_{i=1} from the given query, this step aims to retrieve relevant information based on these properties. Specifically, each property is first transformed into a sub-query 𝒒i\mbox{\boldmath$q$}_{i}, which is then matched against relevant content chunks. These chunks are derived from the original long documents 𝐗\mathbf{X}, with each chunk limited to 1,024 tokens. Based on the sub-queries and their corresponding retrieved chunks, we can derive several intermediate findings (i.e., sub-answers). Step 3: Summarization. In this final step, the original query 𝐐\mathbf{Q} and all intermediate findings from the second step are integrated to generate a comprehensive conclusion 𝐀\mathbf{A}. Figure 3: Token length distribution of answers with and without CoT reasoning from Multi-Source QA pairs in the proposed LongFinanceQA dataset. The PAI serves as a low-cost annotator for the LongFinanceQA dataset. Using PAI, we produce augmented answers with CoT reasoning by combining the intermediate reasoning results from the first two steps with the conclusion. Figure 3 illustrates the token length difference between answers with and without CoT reasoning, showing that reasoning-augmented answers are nearly 200 tokens longer on average.33 3 The answers are tokenized by the LLaMA-3.1 tokenizer. Please check out more data statistics in the Appendix A. 3.3 Supervised CoT Reasoning Although PAI can behave like a human in long-context scenarios, it requires human-crafted design and multi-step inference. To address this limitation, we seek to transfer the long-context reasoning capability of PAI to existing language models, enabling them to analyze long content in a single-step inference. To this end, we fine-tune a language model on LongFinanceQA to explicitly guide it in learning CoT reasoning. This fine-tuning procedure is termed as Supervised CoT Reasoning. Specifically, we use LLaMA-3.1-8B-Instruct Dubey et al. (2024) as the base language model. Following Fu et al. (2024), we first extend the context window of LLaMA-3.1 from 128K to 262K through continued pretraining on 1.6B tokens, which consist of packed documents from Slimpajama Soboleva et al. (2023). After extending context length, we fine-tune the extended LLaMA-3.1 model (parameterized by θ\theta) on LongFinanceQA with reasoning-augmented answers. Through this process, we obtain an enhanced LongPAI model by maximizing the log-likelihood of the predicted answer 𝐘\mathbf{Y} along with the CoT reasoning 𝐑\mathbf{R}, conditioned on the given long documents 𝐗\mathbf{X} and the query 𝐐\mathbf{Q}. The objective is calculated as: ℒ(θ)=∑i=1Nr+Nylogpθ(𝐑,𝐘|𝐗,𝐐),\mathcal{L(\theta)}=\sum_{i=1}^{N_{r}+N_{y}}\log p_{\theta}(\mathcal{\mathbf{R},\mathbf{Y}|\mathbf{X},\mathbf{Q}}), (4) where NrN_{r} is the number of reasoning tokens and NyN_{y} indicates the answering token, including those in the properties, sub-answers, and final answers. We mask non-answer positions during the fine-tuning, allowing for more efficient learning. Although we only consider the LLaMA-3.1 model here, LongFinanceQA can also be used to fine-tune other language models (See Section 4.1 for details). 4 Experiments 4.1 Experimental Setup Evaluation Benchmarks. We evaluate long-context understanding using two practical benchmarks: Loong Wang et al. (2024) and ∞\inftyBench Zhang et al. (2024b). Loong focuses on real-world multi-document question answering and comprises 1,600 test samples across four categories, including Spotlight Locating, Comparison, Clustering, and Chain of Reasoning. These tasks assess distinct capabilities in handling long-context tasks. ∞\inftyBench facilitates multilingual evaluation, assessing models on English (En.QA) and Chinese (Zh.QA) question-answering tasks that require long-range dependency and reasoning beyond short-passage retrieval. Table 1: Data quality assessment for the long-context synthetic dataset (LongFinanceQA) by measuring the performance of PAI on the Loong benchmark. AS denotes Average Scores (0-100), and PR represents the Perfect Rate (0-1). Green highlights the remarkable improvements over the base model (GPT-4o-mini). Model Context Spotlight Locating Comparison Clustering Chain of Reasoning Overall Length AS PR AS PR AS PR AS PR AS PR Open-Source Long-Context Large Language Models LLaMA-3.1-8B-Instruct 128K 62.42 0.52 39.13 0.21 25.96 0.01 44.20 0.22 38.79 0.18 DeepSeek-R1-Qwen-32B 128K 51.68 0.41 49.25 0.34 41.53 0.16 45.00 0.30 45.45 0.27 Qwen2-72B-Instruct 128K 54.17 0.36 42.38 0.20 36.71 0.04 47.76 0.18 43.29 0.15 Qwen2.5-72B-Instruct 128K 65.08 0.55 51.90 0.30 46.07 0.08 64.43 0.40 54.83 0.28 Qwen2.5-14B-Instruct-1M 1000K 67.50 0.58 55.12 0.35 39.05 0.04 57.81 0.31 51.30 0.25 Closed-Source Long-Context Large Language Models Kimi-Chat 200K 60.98 0.50 34.74 0.13 28.76 0.04 38.52 0.15 37.49 0.16 Claude3.5-Sonnet 200K 58.45 0.49 54.21 0.35 45.77 0.07 43.92 0.25 48.85 0.23 GPT-4o 128K 73.95 0.62 50.50 0.28 44.29 0.09 57.95 0.28 53.47 0.26 Gemini-1.5-pro 1000K 75.02 0.56 49.94 0.27 44.10 0.09 64.97 0.37 55.37 0.27 GPT-4o-mini (Base) 128K 59.46 0.49 51.90 0.27 34.55 0.04 64.28 0.39 49.25 0.24 GPT-4o-mini w/ PAI (ours) 128K 79.74 0.67 67.60 0.46 62.80 0.27 75.46 0.57 69.58 0.44 Evaluation Metrics. For evaluation, Loong employs GPT-4-Turbo as a judge, scoring model responses based on accuracy, hallucinations, and completeness on a scale of 0 to 100. Meanwhile, it introduces the Perfect Rate, measuring the proportion of responses achieving a perfect score. In ∞\inftyBench, model performance is measured by the F1 score Zhang et al. (2024b). Base Models. We adopt GPT-4o-mini Achiam et al. (2023) and LLaMA-3.1-8B-Instruct Dubey et al. (2024) as our base models. Specifically, GPT-4o-mini serves as the agent of the proposed Property-based Agentic Inference (PAI), while LLaMA-3.1-8B is used as the base model for LongPAI, which is fine-tuned on the LongFinanceQA. Implementation Details. To enable training on long sequences (> 250K), we employ several optimization techniques, including flash-attention-2 Dao (2023) and ring-attention Liu et al. (2024c). Furthermore, we adopt a zigzag sharding approach Zhu (2024) within ring attention for more effective load distribution across multiple GPUs. This training setup allows us to fine-tune the large language models fully. Using 8 A100 GPUs, the long-context training is completed in three days for fine-tuning. In addition, in training the long-context LLMs, we adopt the rotary base scaling approach Liu et al. (2024d) and scale up the base value (i.e., rotary base of 1,247,820) to adapt RoPE to a longer context. For optimization, we use a constant learning rate of 1e-5 for the entire training procedure. Following the common practice Fu et al. (2024), we set the batch size to 16M tokens as mentioned in Dubey et al. (2024). During inference, we set the temperature as zero to eliminate the randomness. We also increase the maximum output tokens to 1,024 since the CoT reasoning requires more output tokens. 4.2 Main Results In this section, we first assess the quality of our long-context synthetic data (LongFinanceQA) by evaluating the performance of the proposed PAI on the Loong benchmark. Next, we compare the enhanced long-context language model (LongPAI) with its base LLaMA-3.1 model and other state-of-the-art LLMs on the Finance subset of Loong. Data Quality Assessment. Reasoning-augmented answers in LongFinanceQA are automatically generated by the PAI framework. To assess the quality of these synthetic answers, we evaluate the annotator, PAI, on the Long benchmark, using GPT-4o-mini as the agent within the PAI, referred to as GPT-4o-mini w/ PAI. Table 1 shows that the GPT-4o-mini-based PAI framework significantly enhances the base model's performance, demonstrating the effectiveness of the PAI. In particular, compared to the standard GPT-4o-mini model, the overall average score improves by 20.3%, with substantial gains in key tasks such as spotlight locating (+20.2%), comparison (+15.7%), clustering (+29.2%), and chain of reasoning (+10.2%). Although the basic GPT-4o-mini is not the strongest model, GPT-4o-mini w/ PAI outperforms the state-of-the-art closed-source model, Geneni-1.5-pro Reid et al. (2024), by over 15%. Moreover, the superior performance in the Spotlight Locating task (i.e., single-source QA task) highlights the quality of intermediate reasoning results generated by the PAI, as predictions in this task serve as essential reasoning steps for the other three multi-source QA tasks. Consequently, the strong performance on the Loong demonstrates the capability of PAI as a reliable annotator, ensuring high-quality synthetic data in long-context scenarios. Human Evaluation on LongFinanceQA. To further quantify the data quality, we manually annotated 200 randomly selected LongFinanceQA questions, including both single-source and multi-source questions. Then, 10 volunteers score each response on a 1-to-5 scale (1 = completely incorrect, 5 = completely correct). Furthermore, volunteers rate the correctness of intermediate reasoning steps on a 1-to-5 scale for each multi-source question. The averaged scores above 4.0 can be interpreted as largely correct. Table 2 shows that both final answers and their supporting reasoning trajectories are consistently reliable. It provides concrete evidence that the performance improvements primarily arise from high-quality synthesized data, rather than from patterns in noisy samples. Method Comparisons on Existing Benchmarks. To evaluate the effectiveness of supervised CoT reasoning on long-context modeling, we measure the performance of the enhanced LongPAI model on two well-known long-context understanding benchmarks, including Loong and ∞\inftyBench. First, we evaluate the LongPAI on an in-domain benchmark, namely the Finance subset of Loong. Table 3 presents that supervised CoT reasoning enables LongPAI to outperform its base model, LLaMA-3.1-8B-Instruct, by 28.0% in overall results. Furthermore, LongPAI exhibits a 30% improvement on subsets with longer content (i.e., 200K-250K). Also, LongPAI achieves a competitive performance against existing state-of-the-art language models. Remarkably, LongPAI is comparable to its teacher model, GPT-4o-mini w/ PAI (73.94% vs. 75.56%). In some settings, LongPAI even surpasses its teacher model. This finding highlights the significance of long-context modeling. Meanwhile, this finding strongly challenges the recent claim that the long-context problem can be adequately addressed by short language models Qian et al. (2024); Chen et al. (2024b). In other words, certain long-context problems require long-context modeling, as short language models struggle to analyze and reason effectively over reduced or retrieved information. At the same time, we evaluate the LongPAI on ∞\inftyBench. The results in Table 4 show that the LongPAI outperforms its base model even on the out-of-domain benchmark. Table 2: Human evaluation on LongFinanceQA dataset. Subset Score Single-source QA 4.45 ±\pm 0.21 Multi-source QA 4.24 ±\pm 0.23 Intermediate Reasoning Trajectories 4.30 ±\pm 0.22 Overall 4.29 ±\pm 0.19 Table 3: Performance on Financial subset of Loong benchmark. AS represents Avg Scores (0~100) and PR denotes Perfect Rate (0~1). Green highlights improvements over the base model. Full results are shown in Appendix A. Model Context Spotlight Locating Comparison Clustering Chain of Reasoning Overall Length AS PR AS PR AS PR AS PR AS PR 𝙰𝚕𝚕​𝚂𝚎𝚝\mathtt{All\ Set} (10K-250K) Qwen2-72B-Instruct 128K 59.80 0.47 61.12 0.43 34.32 0.06 74.68 0.50 53.20 0.32 Qwen2.5-72B-Instruct 128K 71.07 0.63 59.14 0.41 38.23 0.08 81.09 0.60 57.36 0.36 Qwen-2.5-14B-Instruct-1M 1000K 78.83 0.72 65.27 0.50 36.24 0.07 79.10 0.64 59.78 0.41 GPT-4o 128K 88.23 0.84 62.90 0.48 45.51 0.17 69.40 0.43 63.05 0.44 GPT-4o-mini (Base) 128K 70.90 0.59 59.37 0.38 36.33 0.06 79.58 0.58 56.50 0.34 GPT-4o-mini w/ PAI (ours) 128K 91.07 0.83 74.40 0.58 61.55 0.32 89.63 0.78 75.56 0.57 LLaMA-3.1-8B-Instruct (Base) 128K 67.84 0.56 47.12 0.30 24.62 0.02 63.63 0.34 45.88 0.26 LongPAI (ours) 262K 89.79 0.84 71.69 0.60 59.71 0.32 90.28 0.83 73.94 0.58 𝚂𝚎𝚝𝟹\mathtt{Set3} (100K-200K) Qwen2-72B-Instruct 128K 47.00 0.33 48.07 0.27 25.79 0.00 69.37 0.34 42.98 0.20 Qwen2.5-72B-Instruct 128K 60.47 0.48 49.00 0.28 30.61 0.01 76.54 0.46 48.99 0.26 Qwen-2.5-14B-Instruct-1M 1000K 74.33 0.68 54.64 0.35 30.72 0.01 73.71 0.51 53.47 0.33 GPT-4o 128K 87.25 0.83 46.00 0.31 36.68 0.08 64.57 0.40 54.79 0.36 GPT-4o-mini (Base) 128K 63.05 0.53 53.48 0.24 29.80 0.01 72.37 0.46 50.03 0.26 GPT-4o-mini w/ PAI (ours) 128K 94.08 0.83 74.13 0.63 55.78 0.20 87.71 0.77 74.21 0.55 LLaMA-3.1-8B-Instruct (Base) 128K 63.38 0.47 36.04 0.19 20.28 0.00 62.49 0.26 40.45 0.20 LongPAI (ours) 262K 94.17 0.90 63.76 0.49 51.83 0.24 88.66 0.77 70.00 0.54 𝚂𝚎𝚝𝟺\mathtt{Set4} (200K-250K) Qwen2-72B-Instruct 128K 41.85 0.19 39.75 0.15 29.17 0.03 41.67 0.07 37.23 0.11 Qwen2.5-72B-Instruct 128K 57.48 0.44 49.50 0.30 27.33 0.00 55.00 0.13 45.51 0.22 Qwen-2.5-14B-Instruct-1M 1000K 59.44 0.41 37.00 0.20 27.33 0.00 31.67 0.00 39.57 0.16 GPT-4o 128K 69.26 0.56 50.50 0.35 30.70 0.00 50.67 0.07 49.58 0.25 GPT-4o-mini (Base) 128K 48.37 0.26 50.00 0.30 28.70 0.00 48.33 0.07 42.30 0.15 GPT-4o-mini w/ PAI (ours) 128K 82.78 0.70 63.50 0.35 48.00 0.17 76.00 0.53 66.14 0.42 LLaMA-3.1-8B-Instruct (Base) 128K 40.74 0.30 35.85 0.20 19.77 0.00 28.73 0.00 30.88 0.13 LongPAI (ours) 262K 71.48 0.59 56.50 0.45 46.83 0.17 76.00 0.67 60.92 0.43 Table 4: Comparison on En.QA and Zh.QA of ∞\inftyBench. ∗ indicates results borrowed from Zhang et al. (2024b). En.QA Zh.QA YaRN-Mistra∗ 9.55 16.98 Kimi-Chat∗ 16.52 18.62 Claude 2∗ 11.97 10.53 GPT-4∗ 22.22 23.06 LLaMA-3.1-8B-Instruct (Base) 27.11 29.77 LongPAI (ours) 32.65 32.88 4.3 Discussion Ablation Study on Supervised CoT Reasoning. To further analyze the impact of supervised CoT reasoning, we fine-tune the base LLaMA-3.1 model on LongFinanceQA while excluding CoT reasoning steps from the augmented answers, resulting in a new model, LongPAI§. Unlike the original LongPAI, LongPAI§ directly predicts the final answer, skipping intermediate reasoning steps. Table 5 shows that LongPAI significantly outperforms LongPAI§ over different input lengths. This result strongly supports the hypothesis mentioned in Section 1, which argues that directly guiding models to generate brief answers without intermediate reasoning steps for long-context modeling will lead to suboptimal training (finding 1). Moreover, Table 5 presents several interesting findings. First, while LongPAI§ performs comparably to LongPAI on short content (10K-50K tokens), its performance declines significantly on longer content, which means CoT reasoning is necessary for long-context modeling (finding 2). Furthermore, LongPAI§ performs well in single-source questions (Spotlight Locating) but struggles with multi-source questions (Comparison, Clustering, and Chain of Reasoning) as input length increases, demonstrating that CoT reasoning benefits complex long-context problem-solving (finding 3). In sum, all these findings reaffirm the importance of the reasoning capability for long-context modeling. Comparison of Various Inference Frameworks. We compare PAI with four relevant inference frameworks: PAI-, Self-Route Li et al. (2024), Self-Ask Press et al. (2023), and RAG Lewis et al. (2020). PAI- is a variant of PAI that generates sub-questions directly, rather than first extracting properties and then generating sub-questions. Table 6 presents that PAI- generally outperforms the base GPT-4o-mini, except on 𝚂𝚎𝚝𝟷\mathtt{Set1}. However, there is a gap between PAI- and PAI, highlighting the superiority of the Property Extraction Agent. RAG follows a two-step process: it first retrieves the top KK chunks relevant to a given query, and then uses the retrieved chunks to generate an answer. Following Loong Wang et al. (2024), we adopt the BGE Chen et al. (2024a) as the embedding choice and set KK to 50, selecting from 5, 10, 30, and 50. Table 6 shows that RAG struggles with long-context problems, a conclusion also reached by previous work Wang et al. (2024). Furthermore, we compare the proposed PAI framework with both predefined reasoning RAG (Self-Route) and agentic reasoning RAG (Self-Ask). The results demonstrate that PAI consistently outperforms the other two agentic RAG methods, especially on longer contexts (Set3 and Set4), highlighting a significant superiority of the PAI framework under the scenario of long-context understanding. Table 5: Ablation study on Supervised CoT Reasoning. Symbol § represents LongPAI without supervised CoT reasoning during fine-tuning. Average Scores (0-100) are evaluated by GPT-4-Turbo. Green highlights the remarkable improvements over the base LLaMA-3.1-8B, while Red indicates a decline. Abbreviations: S.L. (Spotlight Locating), Comp. (Comparison), Clust. (Clustering), and Chain. (Chain of Reasoning). Method S.L. Comp. Clust. Chain. Overall 𝚂𝚎𝚝𝟷\mathtt{Set1} (10K-50K) LLaMA-3.1-8B 89.13 72.33 31.77 74.0 60.50 LongPAI§ 93.39 68.67 79.38 83.80 79.82 LongPAI 97.30 90.17 72.88 94.00 85.42 𝚂𝚎𝚝𝟸\mathtt{Set2} (50-100K) LLaMA-3.1-8B 80.58 51.11 27.39 75.12 51.13 LongPAI§ 82.12 40.40 11.24 50.25 38.11 LongPAI 91.25 76.27 66.02 96.12 78.19 𝚂𝚎𝚝𝟹\mathtt{Set3} (100K-200K) LLaMA-3.1-8B 63.38 36.04 20.28 62.49 40.45 LongPAI§ 59.58 30.43 5.47 37.43 29.46 LongPAI 94.17 63.76 51.83 88.66 70.00 𝚂𝚎𝚝𝟺\mathtt{Set4} (200K-250K) LLaMA-3.1-8B 40.74 35.85 19.77 28.73 30.88 LongPAI§ 43.37 22.00 5.60 17.20 22.14 LongPAI 71.48 56.50 46.83 76.00 60.92 Efficiency Analysis on PAI and LongPAI. The PAI framework relies on multi-step inference, whereas LongPAI achieves results with a single inference step. We adopt the number of input tokens processed on the Loong benchmark (i.e., 1,600 samples) as a metric to compare the efficiency between PAI and LongPAI. In this comparison, PAI processes 3.53B tokens in total, determined by the GPT-4o tokenizer, whereas LongPAI requires only 112M tokens, less than 3% of PAI's total. Despite PAI delivering the highest overall performance, LongPAI stands out as the far more efficient approach, demonstrating a dramatic reduction in computational cost. 5 Conclusion In this work, we introduce LongFinanceQA, a novel long-context synthetic dataset featuring reasoning-augmented answers for practical long-context questions. To generate these enriched answers, we develop the Property-based Agentic Inference (PAI) framework, which simulates human-like reasoning through property extraction, retrieval, and summarization. Empirical analysis demonstrates the effectiveness of PAI and ensures the high quality of the LongFinanceQA dataset. Beyond PAI, we explicitly guide a lightweight language model in learning CoT reasoning via fine-tuning on LongFinanceQA. A series of empirical results underscores the importance of reasoning-augmented long-context modeling. Table 6: Comparison of inference frameworks on the Loong Benchmark. Performance is evaluated by GPT-4-Turbo across four sets with different context sizes: 𝚂𝚎𝚝𝟷\mathtt{Set1} (10-50K), 𝚂𝚎𝚝𝟸\mathtt{Set2} (50-100K), 𝚂𝚎𝚝𝟹\mathtt{Set3} (100-200K), and 𝚂𝚎𝚝𝟺\mathtt{Set4} (200-250K). Green indicates an improvement over base GPT-4o-mini. Red denotes a decline. Method 𝚂𝚎𝚝𝟷\mathtt{Set1} 𝚂𝚎𝚝𝟸\mathtt{Set2} 𝚂𝚎𝚝𝟹\mathtt{Set3} 𝚂𝚎𝚝𝟺\mathtt{Set4} Overall GPT-4o-mini 65.05 50.63 45.41 31.61 49.25 w/ PAI 78.69 69.62 68.95 58.12 69.58 w/ PAI- 60.52 57.69 54.89 37.99 54.56 w/ Self-Ask 62.44 52.37 46.91 34.53 50.17 w/ Self-Route 68.79 55.24 49.16 34.44 53.13 w/ RAG 64.67 41.21 31.80 22.08 40.35 Limitations In this study, we investigate the effectiveness of supervised CoT reasoning using reasoning-augmented long-context synthetic data (LongFinanceQA). While effective, our approach has certain limitations. First, although the reasoning-enhanced LongPAI model demonstrates significant improvement in the financial domain, its ability to generalize to broader long-context scenarios remains uncertain since overfitting is an inherent issue that stems from supervised fine-tuning Li et al. (2025). To address this, our future work will explore the impact of diverse data sources and different data scales. Also, we might explore potential techniques to solve this issue, such as regularization Li et al. (2025) and reinforcement learning Schulman et al. (2017); Shao et al. (2024); Rafailov et al. (2023) Second, while the PAI framework achieves strong performance on the Loong benchmark, it still relies on domain-specific human-crafted prompts to guide agents in maintaining structured reasoning. For example, prompts may explicitly direct the model to focus on concepts such as “profit, revenue, or cash flow” when processing financial data, or “reference” and “citation” when handling scientific texts. Therefore, we plan to explore inference methods that enable autonomous reasoning with long-context inputs, minimizing reliance on extensive human-crafted prompts. Despite the current limitations of PAI and LongPAI, their ability to achieve significant performance gains through minimal domain-specific guidance highlights their strong practical potential for specialized applications. References Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv:2303.08774. Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The falcon series of open language models. arXiv:2311.16867. Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv:2305.10403. Anthropic Team (2024) Anthropic Team. 2024. Introducing the next generation of claude. https://www.anthropic.com/news/claude-3-family. Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv:2309.16609. Bai et al. (2024) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multitask benchmark for long context understanding. In ACL. Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS. Chandrasegaran et al. (2024) Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. 2024. Hourvideo: 1-hour video-language understanding. In NeurIPS. Chen et al. (2024a) Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In ACL. Chen et al. (2024b) Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, and Min Yang. 2024b. Long context is not long at all: A prospector of long-dependency data for large language models. arXiv:2405.17915. Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Chen et al. (2024c) Wenting Chen, Linlin Shen, Jingyang Lin, Jiebo Luo, Xiang Li, and Yixuan Yuan. 2024c. Fine-grained image-text alignment in medical imaging enables explainable cyclic image-report generation. In ACL. Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research. Chung et al. (2024) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research. Dao (2023) Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv:2307.08691. Dao and Gu (2024) Tri Dao and Albert Gu. 2024. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In ICML. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv:2407.21783. Edge et al. (2024) Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv:2404.16130. Eleti et al. (2023) Atty Eleti, Jeff Harris, and Logan Kilpatrick. 2023. Function calling and other api updates. Fu et al. (2024) Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128k context. arXiv:2402.10171. Goldman et al. (2024) Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. 2024. Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp. arXiv:2407.00402. Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948. He et al. (2024) Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qianguosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. 2024. Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. In ACL. Hurst et al. (2024) Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv:2410.21276. IBM Research (2024) IBM Research. 2024. What's an llm context window and why is it getting larger? https://research.ibm.com/blog/larger-context-window. Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv:2401.04088. Kočiský et al. (2018) Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics. Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Li et al. (2024) Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Retrieval augmented generation or long-context LLMs? a comprehensive study and hybrid approach. In EMNLP: Industry Track. Li et al. (2025) Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. 2025. Preserving diversity in supervised fine-tuning of large language models. In ICLR. Liang et al. (2025) Jintao Liang, Gang Su, Huifeng Lin, You Wu, Rui Zhao, and Ziyue Li. 2025. Reasoning rag via system 1 or system 2: A survey on reasoning agentic retrieval-augmented generation for industry challenges. arXiv:2506.10408. Lin et al. (2023) Jingyang Lin, Hang Hua, Ming Chen, Yikang Li, Jenhao Hsiao, Chiuman Ho, and Jiebo Luo. 2023. Videoxum: Cross-modal visual and textural summarization of videos. IEEE Transactions on Multimedia. Liu et al. (2024a) Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv:2412.19437. Liu et al. (2024b) Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024b. World model on million-length video and language with blockwise ringattention. arXiv:2402.08268. Liu et al. (2024c) Hao Liu, Matei Zaharia, and Pieter Abbeel. 2024c. Ringattention with blockwise transformers for near-infinite context. In ICLR. Liu et al. (2024d) Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. 2024d. Scaling laws of roPE-based extrapolation. In ICLR. Mistral AI team (2024a) Mistral AI team. 2024a. Large enough. https://mistral.ai/en/news/mistral-large-2407. Mistral AI team (2024b) Mistral AI team. 2024b. Mistral nemo. https://mistral.ai/en/news/mistral-nemo. Mukherjee et al. (2023) Pritam Mukherjee, Benjamin Hou, Ricardo B Lanfredi, and Ronald M Summers. 2023. Feasibility of using the privacy-preserving large language model vicuna for labeling radiology reports. Radiology. Nam et al. (2024) Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. In ICSE. Nijkamp et al. (2023a) Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. 2023a. Codegen2: Lessons for training llms on programming and natural languages. arXiv:2305.02309. Nijkamp et al. (2023b) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023b. Codegen: An open large language model for code with multi-turn program synthesis. In ICLR. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS. Peng et al. (2024) Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In ICLR. Pichai et al. (2024) Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. 2024. Introducing gemini 2.0: our new ai model for the agentic era. https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/. Press et al. (2023) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of EMNLP. Priyanshu et al. (2024) Aman Priyanshu, Yash Maurya, and Zuofei Hong. 2024. Ai governance and accountability: An analysis of anthropic's claude. arXiv preprint arXiv:2407.01557. Qian et al. (2024) Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Yujia Zhou, Xu Chen, and Zhicheng Dou. 2024. Are long-llms a necessity for long-context tasks? arXiv:2405.15318. Qwen Team (2024) Qwen Team. 2024. Introducing qwen1.5. https://qwenlm.github.io/blog/qwen1.5/. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS. Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research. Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530. Ren et al. (2021) Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. Zero-offload: Democratizing billion-scale model training. In USENIX ATC. Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, et al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR. Sarthi et al. (2024) Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. 2024. Raptor: Recursive abstractive processing for tree-organized retrieval. In ICLR. Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06347. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300. Soboleva et al. (2023) Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://huggingface.co/datasets/cerebras/SlimPajama-627B. Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. Tang et al. (2023) Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. 2023. Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432. Tay et al. (2023) Yi Tay, Jason Wei, Hyung Chung, Vinh Tran, David So, Siamak Shakeri, Xavier Garcia, et al. 2023. Transcending scaling laws with 0.1% extra compute. In EMNLP. Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv:2312.11805. Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Wang et al. (2024) Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024. Leave no document behind: Benchmarking long-context LLMs with extended multi-doc QA. In EMNLP. Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS. Wu et al. (2021) Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv:2109.10862. Yang et al. (2024a) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024a. Qwen2 technical report. arXiv:2407.10671. Yang et al. (2024b) An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024b. Qwen2.5 technical report. arXiv:2412.15115. Yao et al. (2024) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS. Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In ICLR. Young et al. (2024) Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. 2024. Yi: Open foundation models by 01. ai. arXiv:2403.04652. Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In NeurIPS. Zhang et al. (2024a) Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao, Hongjin Qian, Qiwei Ye, and Zhicheng Dou. 2024a. Extending llama-3's context ten-fold overnight. arXiv preprint arXiv:2404.19553. Zhang et al. (2024b) Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024b. ∞\inftyBench: Extending long context evaluation beyond 100K tokens. In ACL. Zhu (2024) Zilin Zhu. 2024. Ring flash attention. https://github.com/zhuzilin/ring-flash-attention. Appendix A Appendix A.1 Trend of Long-Context LLMs In this section, we present the context window sizes of closed-source and open-source LLMs in Figure 7. It indicates the gap between closed-source and open-source LLMs becomes smaller. In particular, the LLMs listed in Figure 7 include T5 Raffel et al. (2020), GPT-3 Brown et al. (2020), Codex Chen et al. (2021), T0 Sanh et al. (2022), Anthropic Priyanshu et al. (2024), InstructGPT Ouyang et al. (2022), CodeGen Nijkamp et al. (2023b), PaLM Chowdhery et al. (2023), U-PaLM Tay et al. (2023), Flan-T5 Chung et al. (2024), GPT-3.5 Turbo, LLaMA Touvron et al. (2023), GPT-4 Achiam et al. (2023), Claude 1.3, CodeGen2 Nijkamp et al. (2023a), PaLM2 Anil et al. (2023), Claude 2, LLaMA-2 Touvron et al. (2023), Qwen Bai et al. (2023), Kimi-Chat-200K, Yi-34B Young et al. (2024), Falcon 180B Almazrouei et al. (2023), Gemini Ultra Team et al. (2023), Mixtral Jiang et al. (2024), Qwen1.5 Qwen Team (2024), Gemini 1.5 Pro, Claude 3.0 Anthropic Team (2024), Kimi-Chat-2M, GPT-4 Turbo Achiam et al. (2023), LLaMA-3 Dubey et al. (2024), GPT-4o Hurst et al. (2024), Qwen2 Yang et al. (2024a), GPT-4o-mini Hurst et al. (2024), Mistral NeMo Mistral AI team (2024b), LLaMA-3.1 Dubey et al. (2024), Mistral Large 2 Mistral AI team (2024a), o1, Qwen-2.5 Yang et al. (2024b), Gemini 2.0 Pichai et al. (2024), DeepSeek-V3 Liu et al. (2024a), DeepSeek-R1 Guo et al. (2025), Qwen-2.5-1M Yang et al. (2024b), and o3-mini. A.2 Prompts of Property-based Agentic Inference (PAI) In this section, we present the prompts for the three agents in the PAI framework: the property extraction agent, the property-based retrieval agent, and the summarization agent. Property Extraction Agent. To extract properties, we employ the function-calling API of GPT-4o-mini to selectively retrieve the relevant metric and its corresponding subject from a given query. Figure 4 illustrates the property extraction process using function calling. In addition, we incorporate domain-specific examples within the function call to enhance accuracy across different domains. For instance, in the finance domain, we use “profit”, “revenue”, and “debt” as metric examples, while the “financial document title” serves as a subject example. In the legal domain, “verdict” represents the metric, and “legal judgment” serves as the subject. Similarly, in the academic domain, “reference” and “citation” are used as metric examples, with “paper title” as the subject. Figure 4: Function calling details of property extraction agent (top) and property-based retrieval (bottom). Figure 5: The histogram of input token length in the LongFinanceQA (left). The proportion of single-source and multi-source QA tasks in the LongFinanceQA (right). Figure 6: The pipeline of long-context synthetic question generation for the LongFinanaceQA dataset. Figure 7: The trend of the context window size of recent large language models. The green dots refer to open-source LLMs, while the red dots denote closed-source LLMs. Property-based Retrieval Agent. After obtaining the metric and its corresponding subject, we generate a sub-question in the format: “What was the of the ?”. Then, the long-context input document is divided into a list of 1024-token chunks. Each chunk is evaluated to determine its relevance to the sub-questions, as shown in Figure 4 (bottom). After that, each sub-question is assigned relevant chunks. Next, we pack these relevant chunks and generate a sub-answer to the corresponding sub-question. Thus, this agent functions similarly to RAG Lewis et al. (2020). Summarization Agent. Given the original query, the summarization agent summarizes a conclusion based on the sub-answers generated by the property-based retrieval agent. A.3 LongFinanceQA Dataset Statistics Figure 5 presents a histogram of input token lengths in the LongFinanceQA dataset, which generally range from 50K to 250K tokens. Most input documents contain approximately 200K tokens. At the same time, we provide the proportion of single-source and multi-source QA pairs in the proposed LongFinanceQA dataset, where single-source QA pairs make up 45.6%, while multi-source QA pairs account for 54.4%. Figure 8: Case study on two representative long-context QA problems. A.4 Synthetic Question Generation Figure 6 illustrates the procedure of synthetic Question generation. First, We create a financial metric pool with key metrics like profit, revenue, and cash flow (represented as the grey box). Then, we randomly select metrics and metadata of financial reports (e.g., company name and year) to generate long-context questions requiring single- or multi-source evidence (represented as the blue box). In addition, the types of multi-source questions are inspired by Loong Wang et al. (2024) and include comparison, clustering, trend analysis, etc. A.5 More Empirical Results Case Study on Practical Long-Context Problems. As shown in Figure 8, we present two representative long-context QA examples. Both questions are particularly challenging, as they require models to consider multiple pieces of evidence from the input text. Specifically, the case study includes the user query (white box), ground truth (yellow box), predictions from the base LLaMA-3.1-8B-Instruct model Dubey et al. (2024) (white box), and predictions from our LongPAI model (green box). In addition, incorrect predictions are highlighted in red, while correct ones are marked in green. The results demonstrate the superiority of the proposed LongPAI model, which benefits from supervised CoT reasoning. Furthermore, Figure 8 illustrates that LongPAI offers significantly greater interpretability than its base model. A.6 Full Performances Table 7 presents the full results of the Table 3 in the main manuscript. Also, we involve more popular LLMs Yang et al. (2024b); Guo et al. (2025) into the Table 7. Table 7: Full results of Table 3. AS represents Avg Scores (0~100) and PR denotes Perfect Rate (0~1). Green indicates improvements compared to the base model; Red denotes a decrease compared to the base model. Model Context Spotlight Locating Comparison Clustering Chain of Reasoning Overall Length AS PR AS PR AS PR AS PR AS PR 𝙰𝚕𝚕​𝚂𝚎𝚝\mathtt{All\ Set} (10K-250K) DeepSeek-R1-Qwen-32B 128K 53.66 0.46 52.19 0.39 39.76 0.17 65.15 0.51 49.92 0.34 Qwen2-72B-Instruct 128K 59.80 0.47 61.12 0.43 34.32 0.06 74.68 0.50 53.20 0.32 Qwen2.5-72B-Instruct 128K 71.07 0.63 59.14 0.41 38.23 0.08 81.09 0.60 57.36 0.36 LLaMA-3-8B-Instruct-262K 262K 58.60 0.41 33.12 0.16 20.04 0.01 35.10 0.09 34.41 0.15 GLM4-9B-Chat 1000K 72.69 0.60 49.31 0.32 23.41 0.02 60.77 0.28 46.71 0.27 Qwen-2.5-14B-Instruct-1M 1000K 78.83 0.72 65.27 0.50 36.24 0.07 79.10 0.64 59.78 0.41 GPT-4o 128K 88.23 0.84 62.90 0.48 45.51 0.17 69.40 0.43 63.05 0.44 GPT-4o-mini (Base) 128K 70.90 0.59 59.37 0.38 36.33 0.06 79.58 0.58 56.50 0.34 GPT-4o-mini w/ PAI (ours) 128K 91.07 0.83 74.40 0.58 61.55 0.32 89.63 0.78 75.56 0.57 LLaMA-3.1-8B-Instruct (Base) 128K 67.84 0.56 47.12 0.30 24.62 0.02 63.63 0.34 45.88 0.26 LongPAI (ours) 262K 89.79 0.84 71.69 0.60 59.71 0.32 90.28 0.83 73.94 0.58 𝚂𝚎𝚝𝟷\mathtt{Set1} (10K-50K) DeepSeek-R1-Qwen-32B 128K 41.73 0.33 39.56 0.23 25.67 0.06 55.14 0.37 37.35 0.21 Qwen2-72B-Instruct 128K 88.04 0.83 89.33 0.83 43.00 0.17 93.50 0.80 71.46 0.57 Qwen2.5-72B-Instruct 128K 88.70 0.87 84.67 0.80 43.92 0.10 88.00 0.80 70.07 0.54 LLaMA-3-8B-Instruct-262K 262K 75.82 0.64 40.83 0.23 20.68 0.03 68.00 0.40 43.14 0.25 GLM4-9B-Chat 1000K 88.26 0.83 73.17 0.57 24.65 0.00 87.30 0.50 59.07 0.40 Qwen-2.5-14B-Instruct-1M 1000K 96.09 0.96 88.67 0.80 49.00 0.20 100.00 1.00 76.02 0.62 GPT-4o 128K 100.00 1.00 88.50 0.80 55.25 0.28 98.50 0.90 79.13 0.65 GPT-4o-mini (Base) 128K 97.39 0.96 81.83 0.67 46.50 0.15 100.00 1.00 73.35 0.56 GPT-4o-mini w/ PAI (ours) 128K 95.87 0.91 91.50 0.83 78.38 0.60 96.80 0.70 87.89 0.75 LLaMA-3.1-8B-Instruct (Base) 128K 89.13 0.87 72.33 0.60 31.77 0.05 74.00 0.60 60.50 0.45 LongPAI (ours) 262K 97.30 0.91 90.17 0.80 72.88 0.47 94.00 0.90 85.42 0.71 𝚂𝚎𝚝𝟸\mathtt{Set2} (50K-100K) DeepSeek-R1-Qwen-32B 128K 41.73 0.33 39.56 0.23 25.67 0.06 55.14 0.37 37.35 0.21 Qwen2-72B-Instruct 128K 74.88 0.65 68.60 0.51 40.70 0.09 87.00 0.72 62.38 0.41 Qwen2.5-72B-Instruct 128K 86.00 0.82 61.64 0.41 46.94 0.16 93.12 0.85 65.36 0.46 LLaMA-3-8B-Instruct-262K 262K 72.44 0.54 44.03 0.26 24.11 0.02 37.25 0.12 40.40 0.20 GLM4-9B-Chat 1000K 82.12 0.68 52.73 0.36 26.04 0.04 76.83 0.42 51.66 0.31 Qwen-2.5-14B-Instruct-1M 1000K 88.75 0.85 74.07 0.61 39.06 0.09 96.38 0.90 67.24 0.51 GPT-4o 128K 95.75 0.95 72.87 0.57 54.94 0.27 73.38 0.47 70.10 0.51 GPT-4o-mini (Base) 128K 82.65 0.70 58.77 0.41 40.89 0.08 92.50 0.78 61.61 0.40 GPT-4o-mini w/ PAI (ours) 128K 89.38 0.85 70.73 0.51 64.37 0.36 94.62 0.90 75.34 0.57 LLaMA-3.1-8B-Instruct (Base) 128K 80.58 0.70 51.11 0.33 27.39 0.02 75.12 0.47 51.13 0.30 LongPAI (ours) 262K 91.25 0.88 76.27 0.67 66.02 0.37 96.12 0.93 78.19 0.63 𝚂𝚎𝚝𝟹\mathtt{Set3} (100K-200K) DeepSeek-R1-Qwen-32B 128K 41.73 0.33 39.56 0.23 25.67 0.06 55.14 0.37 37.35 0.21 Qwen2-72B-Instruct 128K 47.00 0.33 48.07 0.27 25.79 0.00 69.37 0.34 42.98 0.20 Qwen2.5-72B-Instruct 128K 60.47 0.48 49.00 0.28 30.61 0.01 76.54 0.46 48.99 0.26 LLaMA-3-8B-Instruct-262K 262K 54.14 0.40 22.93 0.05 15.43 0.00 29.00 0.00 28.58 0.11 GLM4-9B-Chat 1000K 74.75 0.65 41.63 0.24 21.99 0.01 49.86 0.17 43.58 0.25 Qwen-2.5-14B-Instruct-1M 1000K 74.33 0.68 54.64 0.35 30.72 0.01 73.71 0.51 53.47 0.33 GPT-4o 128K 87.25 0.83 46.00 0.31 36.68 0.08 64.57 0.40 54.79 0.36 GPT-4o-mini (Base) 128K 63.05 0.53 53.48 0.24 29.80 0.01 72.37 0.46 50.03 0.26 GPT-4o-mini w/ PAI (ours) 128K 94.08 0.83 74.13 0.63 55.78 0.20 87.71 0.77 74.21 0.55 LLaMA-3.1-8B-Instruct (Base) 128K 63.38 0.47 36.04 0.19 20.28 0.00 62.49 0.26 40.45 0.20 LongPAI (ours) 262K 94.17 0.90 63.76 0.49 51.83 0.24 88.66 0.77 70.00 0.54 𝚂𝚎𝚝𝟺\mathtt{Set4} (200K-250K) DeepSeek-R1-Qwen-32B 128K 18.33 0.11 10.25 0.05 11.27 0.00 26.00 0.07 15.52 0.05 Qwen2-72B-Instruct 128K 41.85 0.19 39.75 0.15 29.17 0.03 41.67 0.07 37.23 0.11 Qwen2.5-72B-Instruct 128K 57.48 0.44 49.50 0.30 27.33 0.00 55.00 0.13 45.51 0.22 LLaMA-3-8B-Instruct-262K 262K 34.19 0.07 20.00 0.05 20.31 0.00 20.71 0.00 24.61 0.03 GLM4-9B-Chat 1000K 40.85 0.19 29.50 0.05 18.13 0.00 25.73 0.00 28.51 0.07 Qwen-2.5-14B-Instruct-1M 1000K 59.44 0.41 37.00 0.20 27.33 0.00 31.67 0.00 39.57 0.16 GPT-4o 128K 69.26 0.56 50.50 0.35 30.70 0.00 50.67 0.07 49.58 0.25 GPT-4o-mini (Base) 128K 48.37 0.26 50.00 0.30 28.70 0.00 48.33 0.07 42.30 0.15 GPT-4o-mini w/ PAI (ours) 128K 82.78 0.70 63.50 0.35 48.00 0.17 76.00 0.53 66.14 0.42 LLaMA-3.1-8B-Instruct (Base) 128K 40.74 0.30 35.85 0.20 19.77 0.00 28.73 0.00 30.88 0.13 LongPAI (ours) 262K 71.48 0.59 56.50 0.45 46.83 0.17 76.00 0.67 60.92 0.43 Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" () button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. We gratefully acknowledge support from our major funders, member institutions, , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2505.09561.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2505.09561.txt new file mode 100644 index 0000000..ddc2eab --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2505.09561.txt @@ -0,0 +1 @@ +Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Related Work Imitation Learning. Long-Context Policies. Test-Time Scaling. 3 Preliminaries Problem Setting. Practical Challenges. 4 Method 4.1 Past-Token Prediction 4.2 Memory-Efficient Training with PTP 4.3 Test-Time Verification with PTP 5 Experiments 5.1 Simulation Experiments 5.2 Real-World Experiments 6 Conclusion 7 Limitations and Discussion. References A Additional Experiments A.1 Which component of the policy benefits most from PTP? A.2 Does our method reduce reliance on action chunking? A.3 Is PTP still critical when conditioning on past actions? B Environmental Details B.1 Real-World Tasks B.2 Simulation Tasks B.3 Implementation Details B.3.1 Policy Architecture B.3.2 Subsampling Rate B.3.3 Context Length B.3.4 Action Dependency Metric C Additional Results License: CC BY 4.0 arXiv:2505.09561v2 [cs.RO] 19 May 2025 Learning Long-Context Diffusion Policies via Past-Token Prediction Marcel Torne ††thanks: Equal contribution Andy Tang11footnotemark: 1 Yuejiang Liu11footnotemark: 1 Chelsea Finn Affiliation: Stanford University August 24, 2026 Abstract Reasoning over long sequences of observations and actions is essential for many robotic tasks. Yet, learning effective long-context policies from demonstrations remains challenging. As context length increases, training becomes increasingly expensive due to rising memory demands, and policy performance often degrades as a result of spurious correlations. Recent methods typically sidestep these issues by truncating context length, discarding historical information that may be critical for subsequent decisions. In this paper, we propose an alternative approach that explicitly regularizes the retention of past information. We first revisit the copycat problem in imitation learning and identify an opposite challenge in recent diffusion policies: rather than over-relying on prior actions, they often fail to capture essential dependencies between past and future actions. To address this, we introduce Past-Token Prediction (PTP), an auxiliary task in which the policy learns to predict past action tokens alongside future ones. This regularization significantly improves temporal modeling in the policy head, with minimal reliance on visual representations. Building on this observation, we further introduce a multistage training strategy: pre-train the visual encoder with short contexts, and fine-tune the policy head using cached long-context embeddings. This strategy preserves the benefits of PTP while greatly reducing memory and computational overhead. Finally, we extend PTP into a self-verification mechanism at test time, enabling the policy to score and select candidates consistent with past actions during inference. Experiments across four real-world and six simulated tasks demonstrate that our proposed method improves the performance of long-context diffusion policies by 3× and accelerates policy training by more than 10×. Videos are available at https://long-context-dp.github.io. 1 Introduction Many robotic tasks are inherently non-Markovian: an appropriate choice of action may depend not only on the current observation but also on past observations and actions [21, 47, 15, 48]. For example, consider manipulation tasks where the robot arm occludes critical parts of the scene, or multi-stage tasks where early steps inform later strategies [24]. Likewise, past actions can prescribe a style of execution, such as speed, curvature, or strategy, that shapes how future actions should unfold [8, 19]. Despite the importance of historical observations, learning long-context robotic policies through imitation learning remains difficult. First, longer observation histories often introduce features that spuriously correlate with actions in the training data. Policies that latch onto such information may diverge from expert behavior during deployment, leading to performance degradation [10, 34]. Second, conditioning on high-dimensional image sequences imposes a rapidly growing memory and computation burden, making end-to-end training excessively expensive at scale [48, 16]. To cope with these challenges, recent methods typically limit the amount of historical information the policy sees - either by truncating the context length [8, 5] or by engineering past observations into compact representations, such as selecting key frames [42] and summarizing observations [48]. While these strategies reduce memory requirement, they risk discarding information critical to subsequent decisions. In this paper, we introduce a simple and effective approach for learning long-context robot policies, illustrated in Fig. 1. At the core of our method is to explicitly regularize the information preserved from past observations. Specifically, we start with an analysis on the discrepancy between recent diffusion policies and their corresponding demonstrations [8, 15]. We observe that action sequences generated by learned policies often exhibit weaker temporal dependencies than those in expert data. To address this, we present past-token prediction (PTP), an auxiliary task where the policy learns to predict past actions alongside future ones. This regularizer encourages the model to attend more effectively to past context, significantly boosting performance. Crucially, we find that the benefits of PTP primarily emerge in the policy head for sequence modeling, rather than the visual encoder. Building upon this analysis, we introduce a multi-stage training recipe: first, pre-train the visual encoder in a short-context setting, where the policy learns to predict a chunk of future actions from only a few past frames [47, 8], and subsequently fine-tune a long-context decoder that jointly predicts past and future actions from precomputed image embeddings. This design enables the policy to capture long-range temporal dependencies while substantially reducing memory and computational overhead. Beyond training, we further leverage PTP as a self-verification mechanism during inference. At each time step, the policy generates multiple candidate actions and selects the one most consistent with its previously executed actions. Figure 1: We propose a simple framework for learning long-context diffusion policies from human demonstrations. Our method leads to 3× gains in performance while reducing the training expense by more than 10×. In summary, our main contributions are twofold: (i) identify a critical discrepancy in temporal action dependencies between learned policies and expert demonstrations (Sec. 3), (ii) propose a training and inference method for long-context imitation learning via past-token prediction (Sec. 4). Empirically, we validate our method on diffusion-based policies [8] across six simulation and four real-world tasks (Sec. 5). On average, our method increases the success rate of long-context policies by 3× while reducing training overhead by over 10 times. Notably, it enables policies to achieve 80% success on history-critical tasks where existing methods fail entirely. 2 Related Work Imitation Learning. Imitation learning has long served as a simple yet powerful paradigm for robot learning [1, 26, 45]. Early approaches typically framed it as a supervised learning problem, where the policy learns to map a given observation to the target action [28]. More recent works have shifted toward modeling the distribution of demonstrations [47, 8, 15, 46, 3, 40, 12, 19]. This approach has recently achieved remarkable success towards generalist robot policies [7, 4]. However, imitation learning remains highly susceptible to covariate shift [10, 41, 32], e.g. Ross et al. [29] and Spencer et al. [34] characterize compounding errors in a feedback loop once the learned policy diverges from the demonstration manifold. This problem is exacerbated by high-dimensional visual inputs, where less robust features might be learned due to underspecification [23]. Notably, recent works [8, 48] have empirically found that image-conditioned specialist and generalist policies degrade with history, leading many works to exclude history altogether [37, 4, 6, 47, 14, 38, 17]. Our work introduces and analyzes a training recipe that counteracts this degradation. Long-Context Policies. Handling long sequences of high-dimensional observations has been a persistent challenge in robot learning. A common strategy is to reduce the input history-by discarding parts of the past via adversarial regularization [41], information bottlenecks [30], or selecting salient subsets through techniques like keyframes [42] and motion tracks [27]. Other methods construct higher-level summaries, such as sketch synthesis [36] or visual trace prompting [48], especially for generalist policies. These approaches rely on the assumption that much of the historical context is irrelevant-a simplification that may break down in temporally complex tasks. An alternative line of work attempts to model the full context in an autoregressive manner using action tokens [25, 11]. Yet, designing action tokenizers that can effectively capture long-range temporal structure remains an open problem [39]. Our method takes an orthogonal approach: we explicitly regularize diffusion policies to retain information about past actions that would otherwise be lost from historical context. Test-Time Scaling. Recent research in language modeling, image generation, and robotics has shown that inference-time compute may allow models to improve their performance [2, 20, 22]. Some seek to build an additional verifier to re-rank the output samples [9, 43, 18, 44], while others propose to leverage the internal knowledge to improve reasoning through self-verification [35]. Our method echoes the latter paradigm in the robotic context: our policy is trained to predict accurate past actions before predicting the present action and can self-verify at test-time through past action accuracy. Similarly to how it may be more compute-efficient to use test-time compute on a small LLM [33], we show that checkpoints trained for fewer epochs or at shorter histories can approach the performance of optimal checkpoints by using more test-time compute. 3 Preliminaries Problem Setting. We consider the problem of imitation learning, where a robot learns to perform complex tasks from expert demonstrations. At each time step tt, the robot receives a visual observation oto_{t} and executes an action ata_{t}. Crucially, we assume that each observation oto_{t} contains only partial information about the underlying state sts_{t}, but the complete information about sts_{t} can be inferred from the history of observations. This setting encapsulates practical challenges commonly encountered in robotic tasks, such as latent strategies in the demonstrations (e.g., expert preference), temporal context (e.g., stage within a task), and perceptual limitations (e.g., visual occlusions). Given a dataset of NN expert demonstrations 𝒟={τi}i=1N\mathcal{D}=\{\tau_{i}\}_{i=1}^{N}, where each demonstration trajectory τi\tau_{i} consists of a sequence of observation-action pairs, our goal is to learn a long-context policy πθ(𝐚t:t+l|𝐨t−k:t)\pi_{\theta}(\mathbf{a}_{t:t+l}|\mathbf{o}_{t-k:t}) that takes as input the current observation along with the history 𝐨t−k:t=(ot−k,…,ot)\mathbf{o}_{t-k:t}=(o_{t-k},\dots,o_{t}) over the past kk time steps, and predicts the current and future actions 𝐚t:t+l=(at,…,at+l)\mathbf{a}_{t:t+l}=(a_{t},\dots,a_{t+l}) spanning the next ll time steps. While increasing the context length kk provides richer historical information, the resulting long-context policies often suffer from substantial performance declines [8, 48]. Practical Challenges. One central challenge in long-context imitation learning arises from the prevalence of spurious features in observation history. As context length increases, the model is exposed to a growing set of input features, some of which correlate with but do not causally influence the expert actions. Policies relying on these spurious features in observation history may reach high prediction accuracy within the training distribution but generalize poorly during deployment [10]. One notable manifestation is the copycat behavior [41], where the learned policy simply mimics previous actions as predictions for future ones, ignoring current state observations. Does this phenomenon persist in modern imitation learning methods? To understand this, we evaluate temporal action dependencies by measuring how predictable the current action is from prior actions alone. Specifically, given a set of demonstrations, we first train long-context policies with varying observation history lengths. We then collect policy rollouts and train a simple two-layer MLP ϕ⁡(at|at−1)\phi(a_{t}|a_{t-1}) to predict the current action based solely on the previous action. We measure the mean-squared error ϵπ\epsilon_{\pi} of the MLP predictor on holdout rollouts and similarly obtain ϵπ∗\epsilon_{\pi^{*}} for expert demonstrations. Following [41, 31], we define the action predictability ratio as ϵπ∗/ϵπ\epsilon_{\pi^{*}}/\epsilon_{\pi}. Intuitively, a ratio greater than 1 indicates an over-reliance on previous actions (i.e., copycat behavior), while a ratio less than 1 indicates weaker-than-expert action dependency. Fig. 3 shows the action predictability ratios for classical regression-based policies and modern diffusion-based policies [8] across three simulation tasks in RoboMimic [21]. Interestingly, the two approaches exhibit opposite failure modes: The regression-based policies indeed exhibit high action predictability, even exceeding that of the expert demonstrations. In contrast, modern diffusion-based policies yield predictability ratios significantly below 1, indicating a surprising underuse of past action information despite conditioning on long observation histories. Ideally, an effective imitator should not only learn to accurately predict expert actions in the training set, but also reach a similar level of temporal action dependencies in its rollouts. We will next introduce a method designed to explicitly bridge this gap. Figure 2: Comparison of regression-based and diffusion-based policies in temporal action dependency, normalized by that in demonstrations. Figure 3: Illustration of past-token prediction. The policy head is trained to jointly predict both past and future action tokens, encouraging the model to capture the temporal dependencies that are otherwise lost between past and future actions. 4 Method In this section, we introduce a long-context imitation learning method, aiming to improve both policy performance and training efficiency. We will first describe a simple but crucial auxiliary task to enhance temporal dependencies in sequential decision-making (Sec. 4.1). We will then present a multi-stage training recipe that preserves the benefit of this auxiliary task while reducing memory consumption (Sec. 4.2). Finally, we will introduce an inference technique that leverages the auxiliary task to effectively self-verify sampled predictions at test time (Sec. 4.3). 4.1 Past-Token Prediction One common design choice in imitation learning is next-token prediction, where the policy predicts only the immediate next action token at each time step. To better capture temporal dependencies, recent methods have extended this to predict a chunk of future action tokens [47, 8]. However, as shown in Sec. 3, this design alone remains insufficient for modeling the critical dependencies between past and future decisions. We address this issue through Past-Token Prediction (PTP), an auxiliary objective that tasks the policy to predict past action tokens alongside future ones. Formally, given a sequence of observations 𝐨t−k:t\mathbf{o}_{t-k:t}, the policy is trained to jointly predict the action tokens from the past time step t−kt-k to the upcoming time step t+ht+h: 𝐚^t−k:t+h=πθ(𝐨t−k:t).\hat{\mathbf{a}}_{t-k:t+h}=\pi_{\theta}(\mathbf{o}_{t-k:t}). (1) As illustrated in Fig. 3, this objective expands the prediction window in both temporal directions, explicitly encouraging the policy to preserve information about past actions from the history context. 4.2 Memory-Efficient Training with PTP Recent imitation learning approaches typically train visuomotor policies end-to-end, jointly optimizing both the visual encoder and the policy head. However, this strategy incurs memory costs that grow linearly with context length, making it prohibitively expensive to train long-context policies. To address this, we propose a multi-stage training recipe that decouples visual representation learning from policy optimization. Our training process consists of three specific stages: Figure 4: Overview of multistage training with embedding caching. As PTP acts on the decoder, caching embeddings substantially improves inference speed without sacrificing performance. We use a visual encoder from a short-range policy with low validation loss to compute the embeddings of the images in the buffer and cache them in the buffer. With the cached embeddings we can train the long-horizon policy much faster. At test time we take the original encoder. 1. Encoder Training: We first train the visual encoder with a short observation context but a long prediction horizon, encouraging it to extract representations that retain information critical for predicting ll subsequent steps. 2. Feature Caching: We then freeze the encoder and precompute embeddings for all frames in the training set. This caching step eliminates redundant computation during policy training. 3. Policy Training: Finally, we train the policy head conditioned on long-context observations represented by the cached embeddings. This enables the policy to model long-range dependencies without repeatedly processing visual inputs. As shown in Fig. 4, this multistage training approach retains a computational footprint similar to short-context training while enabling efficient scaling to longer observation contexts. In Appendix A.1, we show in more detail how the features of a short-history policy are sufficient to support strong long-context performance. 4.3 Test-Time Verification with PTP Figure 5: Test-time verification. Multiple action sequences are sampled from the same observation, and the policy selects the sequence that is most consistent compared to ground-truth previous actions. Another common challenge in recent diffusion policies lies in the robustness of sampled predictions. Often, not all samples are equally good at capturing the critical temporal dependencies. Recent work has explored re-ranking sampled predictions based on consistency with past predictions [19]. However, when the previous prediction for future actions is suboptimal, e.g. because of unexpected environmental changes, this approach may propagate errors rather than correct them. To address this shortcoming, we cast Past-Token Prediction as a self-verification mechanism during deployment. At each inference step, we sample a batch of BB candidate action sequences: 𝒜={𝐚^(1),…,𝐚^(B)},𝐚^(i)∼πθ(𝐨t−k:t),\mathcal{A}=\{\hat{\mathbf{a}}^{(1)},\dots,\hat{\mathbf{a}}^{(B)}\},\quad\hat{\mathbf{a}}^{(i)}\sim\pi_{\theta}(\mathbf{o}_{t-k:t}), (2) where each sampled candidate 𝐚^(i)=(at−k,…,at+h)(i)\hat{\mathbf{a}}^{(i)}=(a_{t-k},\dots,a_{t+h})^{(i)} includes both reconstructed past actions and predicted future actions. Since the first k−1k-1 actions have already been executed, we use them as a ground-truth reference and select the candidate whose reconstructed past actions best match the executed ones: 𝐚^∗=arg⁡min⁡∑τ=t−kt−1𝐚^∈𝒜⁡‖a^τ−aτ‖2\hat{\mathbf{a}}^{*}=\arg\min_{\hat{\mathbf{a}}\in\mathcal{A}}\sum_{\tau=t-k}^{t-1}\|\hat{a}_{\tau}-a_{\tau}\|^{2} (3) As illustrated in Fig. 5, this sample selection procedure is fully parallelizable on GPU devices, enabling self-verification of temporal action dependencies with minimal computational overhead. 5 Experiments In this section, we evaluate the proposed method for learning long-context diffusion policies. We seek to answer the following questions regarding policy performance and training efficiency: 1. How effectively does PTP mitigate the lack of temporal action dependencies shown in Sec. 3? 2. How well do the resulting policies perform on tasks that require history-aware decision-making? 3. To what extent does the proposed multi-stage training recipe accelerate policy learning? 4. Could PTP verification further mitigate deficiencies in temporal dependencies at test time? 5. Finally, how do these findings generalize to history-critical tasks in the real world? Figure 6: Effect of PTP on temporal action dependency and policy performance. Increasing the amount of past-token supervision aligns the learner more closely with expert action dependencies, resulting in higher success rates. Figure 7: Effect of history observations on PTP-trained diffusion policies. Increasing the context length progressively enhances policy performance, especially in history-critical tasks such as Long-Horizon Square Figure 8: Effect of feature caching. Caching speeds up training by over 5× without hurting performance. On complex tasks like Tool Hang, long-context policies fail to perform even after two days without caching. To this end, we evaluate our method on the modern diffusion-based policy [8], in comparison with the classical regression-based policy. By default, both policies receive visual and proprioceptive observations from the past 16 time steps as conditional input. We compare policies trained with PTP against two baselines: no-history policies that take only the current and past single frame as input, and no-PTP policies that are trained without PTP. Unless otherwise specified, all policies are trained using the multistage recipe with feature caching and evaluated under a single-sample inference setting. The effect of test-time verification is evaluated separately across multiple checkpoints under varying sample budgets. Additional results are presented in Appendix A, with implementation details provided in Appendix B. 5.1 Simulation Experiments We first evaluate our method across six simulated tasks. Four of these are sourced from existing benchmarks: square, tool hang, and transport from RoboMimic [21], each provided by multi-human demonstration datasets, and Push-T from Chi et al. [8]. These tasks feature diverse strategies in demonstrations, requiring the policy to infer and commit to consistent behaviors over time based on historical context. In addition, we introduce two new long-horizon simulation tasks: long-horizon square, where the robot must place and remove a square onto the peg twice before finally dropping it in the peg; and long-horizon aloha, where one arm must pick up a block, move it to the center of the field of view, and return it precisely to its original location. Success in these new tasks critically depends on the ability to recall and act upon information observed earlier in the episode. Each policy-task pair is evaluated over 100 episodes across three random seeds. We next summarize the key findings from these simulation experiments. Takeaway 1: PTP mitigates deficiencies in modeling temporal action dependencies. To validate the effect of PTP on modeling temporal action dependencies, we use the same set of tasks as in Sec. 3 and train policies to predict a variable number of past tokens {a^t−c−1,…,a^t}\{\hat{a}_{t-c-1},\dots,\hat{a}_{t}\}, where cc denotes the number of actions included in the prediction target. Specifically, we compare three variants: (i) no-PTP with c=1c=1, equivalent to the vanilla next-token prediction baseline; (ii) half-PTP with c=8c=8, which predicts action tokens corresponding to half the observation window; and (iii) full-PTP with c=16c=16. As shown in Fig. 8, PTP consistently increases the action predictability and gets closer to that observed in the expert demonstrations. Notably, the non-PTP baseline exhibits approximately 10× to 100× weaker action predictability ratios compared to expert behavior, whereas full-PTP yields temporal dependencies comparable to demonstrations. Figure 9: Comparison of different policies across six simulation tasks. Unlike classical regression-based policies, modern diffusion-based policies exhibit a clear drop in performance when conditioned on historical observations. Our method achieves an average improvement of over 30% compared to no-history diffusion policies, and over 60% compared to no-PTP diffusion policies. The gains are especially pronounced on history-critical tasks such as long-horizon aloha and long-horizon square. Takeaway 2: PTP significantly improves the performance of modern policies. To assess the impact of PTP on task performance, we compare our method against the no-history and no-PTP baselines on two classes of policies: diffusion-based versus regression-based. All models are evaluated following the protocol from [8], with action chunking set to 8 time steps. As shown in Fig. 9, while the no-history baseline already performs competitively on some existing tasks, PTP matches or surpasses its performance. The advantage of PTP is particularly pronounced in long-horizon tasks: both the no-history and no-PTP baselines struggle with success rates below 30%, whereas our method achieves near-perfect performance on the long-horizon tasks. Averaged across all six simulation tasks, PTP yields an average 50% improvement for diffusion-based policies when conditioned on long contexts, and outperforms the regression counterpart by nearly 20%. Takeaway 3: PTP-trained policies benefit from longer contexts. To further understand the role of historical contexts, we evaluate PTP-trained diffusion-based policies conditioned on observation histories of varying lengths, ranging from 2 to 16 time steps. As shown in Fig. 8, longer histories generally lead to improved performance. For relatively simple tasks such as square, gains tend to saturate beyond 4 steps; however, for more complex tasks, such as transport, long-horizon square, and long-horizon aloha, longer contexts provide substantial performance boosts. Takeaway 4: Embedding caching accelerates PTP training without sacrificing performance. To assess the effectiveness of the proposed multistage training strategy, we train history-conditioned diffusion policies with and without embedding caching for two days on the three tasks used above (Sec. 3), evaluating checkpoints saved every 50 epochs. As shown in Fig. 8, the vanilla training recipe without caching completes only a limited number of epochs within the time budget. In contrast, our caching-based approach matches performance in just 20% of the training time and surpasses it within 40% of the compute budget. Takeaway 5: PTP verification boosts performance in challenging settings at test time. To validate the potential of self-verification through PTP, we evaluate history-conditioned policies on three challenging tasks, including Tool Hang, Transport, and Long Square, trained under constrained compute budgets and tested with varying sampling budgets {1, 3, 5, 10}. As shown in Fig. 11, PTP-guided sample selection provides notable performance gains. Notably, increasing the number of sampled candidates from 1 to 5 results in approximately 5% improvement in success rate on all these tasks. Figure 10: Effect of PTP self-verification. Increasing sampling budgets yields a 5% gain in challenging closed-loop settings. Figure 11: Comparison of different policies on four real-world tasks that critically depend on historical context. Our method yields over a 55% improvement compared to baselines. 5.2 Real-World Experiments We next examine our method on four history-critical tasks across two robot platforms in the real world: Franka block switch: move a block from one side to another, where history is needed to correctly infer which side to place the block; Franka two scoops, transport two scoops to the target, where history is needed to count scoops; Franka mug replacement and ALOHA tape replacement: replace one mug or tape by another, where history is needed to distinguish old and new objects. Across all tasks, we use diffusion-based policies with a context length of 16 and a chunk size of 8. Due to different ranges of temporal dependency in these tasks, we apply task-specific subsampling rates detailed in Appendix B. Quantitatively, PTP outperforms baselines by over 4× in the real world. As shown in Fig. 11, the no-history baseline is limited to an average success rate of 15% due to the absence of critical history information. The no-PTP baseline, which simply conditions on history without PTP, yields near-zero success on three of four tasks. In contrast, our method achieves an average 70% success rate. Notably, on Tape Replacement, one of the most challenging tasks across the board, our method achieves 80% success, while the two baselines fail entirely. Qualitatively, PTP-trained long-context policies excel at both high-level and low-level memory. As shown in the videos on the website, the two baselines exhibit distinct failure modes: the no-history policies often fail at high-level decision-making, such as replacing the wrong object or miscounting scoops, whereas the no-PTP baseline struggles with low-level motor control, such as unsuccessful grasps and inaccurate placements. In comparison, policies trained with our method demonstrate improvement in both high-level planning and low-level control, resulting in more coherent and reliable behavior across tasks. 6 Conclusion We have presented Past Token Prediction (PTP), a simple yet effective auxiliary objective for learning history-conditioned diffusion policies from demonstrations. We have shown that PTP can effectively strengthen temporal action dependencies that are often lost in recent diffusion policies. In addition, we have introduced a multistage training strategy and a self-verification mechanism that allow for effective use of PTP during both training and inference. Extensive experiments across ten manipulation tasks in both simulations and the real world demonstrate its advantages in efficiency and effectiveness. 7 Limitations and Discussion. Our work has focused on extending context length specifically for diffusion policies, motivated by their growing prevalence in the robot learning community. Nevertheless, the effectiveness of our method may generalize to other classes of modern policies as well. In fact, concurrently with our work, Vuong et al. [39] observes similar challenges in tokenization-based policies. Extending our approach to such settings, and more broadly, designing action tokenizers that better preserve temporal structure, can be an exciting avenue for future research. Another practical challenge our method faces is inference overhead. While we have shown that caching and reusing visual embeddings can substantially reduce memory consumption and speed up policy training, inference overhead remains a practical bottleneck for closed-loop operations. To make inference time manageable, we followed common practices from recent literature by downsampling observation history and extending action chunk. However, these adjustments are known to compromise policy reactivity. Designing strategies to further accelerate inference-particularly given the growing scale of VLA models-could be another fruitful direction for future research. Acknowledgments We thank Sergey Levine and the members of the Stanford IRIS lab for their insightful discussions. We thank Joey Hejna, Moo Jin Kim, Alec Lessing, and Amber Xie for their support with the real world experiments. This work was supported in part by The Robotics and AI Institute, DARPA, and the ONR grant N00014-22-1-2621. C. Finn is a CIFAR fellow. Y. Liu is an SNSF postdoc fellow. References [1] Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469-483, May 2009. [2] Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q. Tran, and Mehran Kazemi. Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling. arXiv preprint arXiv:2408.16737, August 2024. [3] Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4788-4795, May 2024. [4] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. $ π​_\pi\_0$: A Vision-Language-Action Flow Model for General Robot Control. arXiv preprint arXiv:2410.24164, November 2024a. [5] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π0\pi_{0}: A vision-language-action flow model for general robot control, 2024b. URL https://arxiv.org/abs/2410.24164. [6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv preprint arXiv:2307.15818, July 2023a. [7] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics Transformer for Real-World Control at Scale. In Robotics: Science and Systems XIX. Robotics: Science and Systems Foundation, July 2023b. ISBN 978-0-9923747-9-2. [8] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Robotics: Science and Systems XIX. Robotics: Science and Systems Foundation, July 2023. ISBN 978-0-9923747-9-2. [9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168, November 2021. [10] Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal Confusion in Imitation Learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. [11] Letian Fu, Huang Huang, Gaurav Datta, Lawrence Yunliang Chen, William Chung-Ho Panitch, Fangchen Liu, Hui Li, and Ken Goldberg. In-Context Imitation Learning via Next-Token Prediction. arXiv preprint arXiv:2408.15980, September 2024. [12] Siddhant Haldar, Zhuoran Peng, and Lerrel Pinto. BAKU: An Efficient Transformer for Multi-Task Policy Learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, November 2024. [13] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R. Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J. Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. arXiv preprint arXiv:2403.12945, March 2024. [14] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P. Foster, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An Open-Source Vision-Language-Action Model. In 8th Annual Conference on Robot Learning, September 2024. [15] Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior Generation with Latent Actions. arXiv preprint arXiv:2403.03181, March 2024. [16] Xinghang Li, Peiyan Li, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma, Tao Kong, Hanbo Zhang, and Huaping Liu. Towards Generalist Robot Policies: What Matters in Building Vision-Language-Action Models. arXiv preprint arXiv:2412.14058, December 2024a. [17] Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, and Ankit Goyal. HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation. In The Thirteenth International Conference on Learning Representations, October 2024b. [18] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step. In The Twelfth International Conference on Learning Representations, October 2023. [19] Yuejiang Liu, Jubayer Ibn Hamid, Annie Xie, Yoonho Lee, Maximilian Du, and Chelsea Finn. Bidirectional Decoding: Improving Action Chunking via Closed-Loop Resampling. arXiv preprint arXiv:2408.17355, December 2024. [20] Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, and Saining Xie. Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps. arXiv preprint arXiv:2501.09732, January 2025. [21] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. In Proceedings of the 5th Conference on Robot Learning, pages 1678-1690. PMLR, January 2022. [22] Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance. In 8th Annual Conference on Robot Learning, September 2024. [23] Soroush Nasiriany, Sean Kirmani, Tianli Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, and Ted Xiao. Rt-affordance: Affordances are versatile intermediate representations for robot manipulation, 2024a. URL https://arxiv.org/abs/2411.02704. [24] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots. arXiv preprint arXiv:2406.02523, June 2024b. [25] Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot Learning with Sensorimotor Pre-training. In Proceedings of The 7th Conference on Robot Learning, pages 683-693. PMLR, December 2023. [26] Harish Ravichandar, Athanasios S. Polydoros, Sonia Chernova, and Aude Billard. Recent Advances in Robot Learning from Demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3(Volume 3, 2020):297-330, May 2020. [27] Juntao Ren, Priya Sundaresan, Dorsa Sadigh, Sanjiban Choudhury, and Jeannette Bohg. Motion tracks: A unified representation for human-robot transfer in few-shot imitation learning, 2025. URL https://arxiv.org/abs/2501.06994. [28] Stephane Ross and Drew Bagnell. Efficient Reductions for Imitation Learning. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 661-668. JMLR Workshop and Conference Proceedings, March 2010. [29] Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011. URL https://arxiv.org/abs/1011.0686. [30] Seokin Seo, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Regularized behavior cloning for blocking the leakage of past action information. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 2128-2153. Curran Associates, Inc., 2023a. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/06b71ad997f7e3e4b2e2f2ea12e5a759-Paper-Conference.pdf. [31] Seokin Seo, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Regularized Behavior Cloning for Blocking the Leakage of Past Action Information. Advances in Neural Information Processing Systems, 36:2128-2153, December 2023b. [32] Daqian Shao, Thomas Kleine Buening, and Marta Kwiatkowska. A Unifying Framework for Causal Imitation Learning with Hidden Confounders. arXiv preprint arXiv:2502.07656, February 2025. [33] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv.org/abs/2408.03314. [34] Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J. Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift, 2021. URL https://arxiv.org/abs/2102.02872. [35] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks. arXiv preprint arXiv:2402.08115, August 2024. [36] Priya Sundaresan, Quan Vuong, Jiayuan Gu, Peng Xu, Ted Xiao, Sean Kirmani, Tianhe Yu, Michael Stark, Ajinkya Jain, Karol Hausman, Dorsa Sadigh, Jeannette Bohg, and Stefan Schaal. Rt-sketch: Goal-conditioned imitation learning from hand-drawn sketches, 2024. URL https://arxiv.org/abs/2403.02709. [37] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An Open-Source Generalist Robot Policy. arXiv preprint arXiv:2405.12213, May 2024. [38] Marcel Torne, Arhan Jain, Jiayi Yuan, Vidaaranya Macha, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, and Abhishek Gupta. Robot Learning with Super-Linear Scaling. arXiv preprint arXiv:2412.01770, December 2024. [39] An Dinh Vuong, Minh Nhat Vu, Dong An, and Ian Reid. Action Tokenizer Matters in In-Context Imitation Learning. arXiv preprint arXiv:2503.01206, March 2025. [40] Dian Wang, Stephen Hart, David Surovik, Tarik Kelestemur, Haojie Huang, Haibo Zhao, Mark Yeatman, Jiuguang Wang, Robin Walters, and Robert Platt. Equivariant Diffusion Policy. In 8th Annual Conference on Robot Learning, September 2024. [41] Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting Copycat Agents in Behavioral Cloning from Observation Histories. In Advances in Neural Information Processing Systems, volume 33, pages 2564-2575. Curran Associates, Inc., 2020. [42] Chuan Wen, Jierui Lin, Jianing Qian, Yang Gao, and Dinesh Jayaraman. Keyframe-focused visual imitation learning, 2021. URL https://arxiv.org/abs/2106.06452. [43] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large Language Models are Better Reasoners with Self-Verification. In The 2023 Conference on Empirical Methods in Natural Language Processing, December 2023. [44] Fei Yu, Anningzhe Gao, and Benyou Wang. OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL 2024, pages 858-875, Mexico City, Mexico, June 2024. Association for Computational Linguistics. [45] Maryam Zare, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges. IEEE Transactions on Cybernetics, 54(12):7173-7186, December 2024. [46] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3D Diffusion Policy. arXiv preprint arXiv:2403.03954, March 2024. [47] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. arXiv preprint arXiv:2304.13705, April 2023. [48] Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies, 2024. URL https://arxiv.org/abs/2412.10345. Appendix A Additional Experiments In addition to the main results presented in Sec. 5.1, we conduct three experiments to further validate the design decisions behind our proposed method. A.1 Which component of the policy benefits most from PTP? To identify which part of the policy is most influenced by PTP, we compare a fully PTP-trained long-context policy against two ablated variants: Encoder PTP, where we first train the visual encoder with PTP, then freeze it and train the action decoder without PTP; Decoder PTP, where we conversely train the encoder without PTP, freeze it, and then apply PTP only during decoder training. As shown in Fig. 12, Decoder PTP achieves performance on par with the fully trained PTP policy, whereas Encoder PTP performs significantly worse. This result suggests that the benefits of PTP primarily stem from improved temporal modeling in the action decoder, rather than from changes to the visual encoder, directly motivating our multi-stage training recipe that decouples encoder pretraining from long-context policy learning. Figure 12: Performance of ablated PTP variants on Push-T. Applying PTP only to the decoder recovers the full PTP policy performance, whereas encoder-only PTP does not. A.2 Does our method reduce reliance on action chunking? Existing short-context policies typically rely on action chunking compensate for limited access to past observations. However, this common design choice comes at the cost of reduced reactivity. To assess whether our method alleviates this limitation, we compare the performance of three policy variants: (i) short-context short-chunk, which receives the past 2 frames as input and outputs single-step actions (chunk size 1); (ii) long-context short-chunk, which receives the past 16 frames and also outputs single-step actions; and (iii) long-context long-chunk, which receives the past 16 frames and outputs action chunks of size 8. As shown in Fig. 13, the long-context short-chunk policies trained by our method substantially outperform the short-context counterparts and recover most of the performance of the long-context long-chunk policies. This result demonstrates the effectiveness of our method in reducing reliance on open-loop action chunking. Figure 13: Comparison of policies with different context lengths and chunk sizes. Long-context policies trained with PTP perform significantly better than short-context policies when run in a fully closed-loop setting (chunk size 1). Moreover, they achieve performance comparable to long-context policies that use open-loop chunking (chunk size 8), indicating reduced reliance on chunking during execution. A.3 Is PTP still critical when conditioning on past actions? Our earlier analysis in Fig. 8 has shown the importance of PTP in capturing temporal action dependencies when the policy is conditioned on past observations. A natural question is whether PTP remains necessary when the model also has direct access to past actions. To understand this, we augment the input to diffusion policies with the previous 16 actions and compare performance with and without PTP. As shown in Fig. 14, even with access to past actions, the vanilla baseline performs poorly without PTP, while our method consistently yields substantially better results. Consistent with our previous findings, this result highlights the critical role of PTP in enabling diffusion policies to effectively model temporal structure, even when past actions are explicitly provided. Figure 14: Comparison of adding past actions into the context history without PTP and our baseline of PTP and no PTP without actions. We observe that adding past actions in the observation doesn't improve performance. Appendix B Environmental Details B.1 Real-World Tasks We conduct real-world experiments using two robot platforms: a Franka Panda arm set up, and the ALOHA bimanual system. For the Franka setup, we follow the DROID hardware configuration [13], using a single arm with wrist-mounted RGB cameras and proprioceptive sensing. The observation space includes RGB images and end-effector pose, while the action space consists of 6-DoF Cartesian displacements and gripper commands. We collect 50-200 human demonstrations for each task. For the ALOHA platform, we use the bimanual robot as described in [47], with RGB camera inputs and proprioceptive feedback. The observation space consists of dual-arm RGB views and proprioception, and the action space includes joint displacements for both arms and gripper states. We collect 150 demonstrations for the tape replacement task. B.2 Simulation Tasks Our simulation tasks include the Push-T task [8], three existing tasks in the Robomimic benchmark [21], along with two new long-horizon tasks that we introduce: • Long-Horizon Square: A variant of the RoboMimic square task, where the robot must place a block onto the farthest peg from its initial location. We collect 100 noisy scripted demonstrations to prevent the policy from inferring the goal using current pose information alone, thus requiring memory of the initial state. • Long-Horizon ALOHA: A simulated bimanual task where one arm picks up a block, moves it to the center of the workspace, and places it back at its original location. Success requires remembering the block's starting position, highlighting the need for long-term memory. Table 1: Hyper-parameters in simulation and real-world experiments. Hyperparameter LH Square LH Aloha Block Move Two Scoops Mug Repl. Tape Repl. Epochs 500 1500 500 500 500 1500 # Demos 100 50 50 200 200 150 # Subsampled frames 20 20 1 20 1 24 # Observations 20 32 10 20 32 20 B.3 Implementation Details B.3.1 Policy Architecture We build upon the transformer-based Diffusion Policy codebase [8], which supports training and evaluation across multiple Robomimic tasks. All policies are trained for 500 epochs by default, using visual encoders and chunked action prediction. For long-horizon ALOHA tasks, we train for 1500 epochs to accommodate the added difficulty of bimanual coordination and higher-frequency control. To reduce training overhead, all long-context policies are initialized with a frozen short-context encoder, pretrained on 2-frame inputs. This design choice is supported by the analysis in Fig. 12, which shows that freezing the encoder does not impair performance. To further improve training efficiency, we cache visual embeddings during data preprocessing and load them at runtime. This avoids repeatedly passing observations through the encoder and speeds up training. B.3.2 Subsampling Rate Real-world tasks often require longer history horizons, but full-length observation sequences can be computationally expensive. To reduce inference latency, we apply temporal subsampling to the input sequence. Specifically, instead of feeding all TT observations t0,t1,…,tT−1{t_{0},t_{1},...,t_{T-1}}, we sample every KKth frame, i.e., tK−1,t2​K−1,…,tT−1{t_{K-1},t_{2K-1},...,t_{T-1}}, where TT is a multiple of KK. This reduces the effective observation size while retaining broad temporal coverage. Subsampling values are listed in Table 1. B.3.3 Context Length When increasing the observation history, we scale the prediction horizon (past and future tokens) proportionally. We empirically find that the prediction length of future tokens has only a minor effect on task performance. Detailed context length configurations are provided in Table 2. Table 2: Settings for different context lengths Observation Length 2 4 8 16 Horizon 16 20 24 32 Future Tokens 14 16 16 16 B.3.4 Action Dependency Metric To quantify how well a policy captures temporal action structure, we use action predictability as a proxy metric [41]. Specifically, we measure how accurately the current action ata_{t} can be predicted from a window of K=15K=15 past actions, defined as p(at∣at−K:t−1)p(a_{t}\mid a_{t-K:t-1}). We compute this quantity over policy rollouts and compare it to the same metric evaluated on the expert demonstrations. Higher predictability indicates stronger temporal action dependencies captured by the learned policy. Appendix C Additional Results In addition to the results reported in Sec. 5.1, we report per-task success rates across varying temporal context lengths, training conditions, and chunking configurations. Table 3 compares diffusion-based and regression-based baselines on six benchmark tasks, evaluated under different history lengths and with or without PTP. Table 4 presents results in the closed-loop setting (chunk size = 1) across different context lengths. Table 3: Success rate (%) of diffusion-based and regression-based policies on simulation tasks under different training and history conditions. Results are reported as mean ± standard deviation across 3 seeds. Method Push-T Square Tool-Hang Transport ALOHA Long Square Diffusion (PTP) 0.62 ± 0.02 0.89 ± 0.01 0.75 ± 0.10 0.67 ± 0.08 0.98 ± 0.01 0.93 ± 0.02 Diffusion (no-PTP) 0.59 ± 0.01 0.17 ± 0.01 0.00 ± 0.00 0.00 ± 0.00 0.20 ± 0.19 0.03 ± 0.02 Diffusion (no-hist) 0.67 ± 0.03 0.79 ± 0.06 0.51 ± 0.14 0.60 ± 0.08 0.28 ± 0.04 0.12 ± 0.05 Regression (PTP) 0.40 ± 0.10 0.74 ± 0.02 0.41 ± 0.01 0.63 ± 0.04 0.99 ± 0.02 0.90 ± 0.05 Regression (no-PTP) 0.31 ± 0.31 0.78 ± 0.03 0.40 ± 0.05 0.51 ± 0.01 1.00 ± 0.00 0.89 ± 0.01 Regression (no-hist) 0.64 ± 0.08 0.19 ± 0.01 0.14 ± 0.04 0.48 ± 0.03 0.43 ± 0.03 0.00 ± 0.00 Table 4: Success rate (%) on simulation tasks under closed-loop execution (chunk size = 1) . Observations Push-T Square Tool Hang Transport Long Square Mean 2 0.53 0.13 0.62 0.053 0.02 0.37 4 0.53 0.68 0.84 0.13 0.02 0.51 8 0.59 0.83 0.86 0.48 0.11 0.63 16 0.64 0.85 0.82 0.51 0.81 0.77 Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" () button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. We gratefully acknowledge support from our major funders, member institutions, , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2506.02483.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2506.02483.txt new file mode 100644 index 0000000..90ef2ac --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2506.02483.txt @@ -0,0 +1 @@ +Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Related Works Long-Context and Cross-Lingual Challenges Advanced Prompting Strategies Neurosymbolic Reasoning 3 System Architecture and Approach 3.1 Retrieval Component Tokenization and Embedding Candidate Selection and Model Input 3.2 NeuroSymbolic Augmented Reasoning (NSAR) 1. Symbolic Fact Extraction 2. Python Code Generation 3. Final Answer Extraction 4 Experiments 4.1 Dataset 4.2 Evaluation 4.3 Models and Baseline Methods 5 Results and Analysis 5.1 LLM-Only vs. RAG-Vanilla Context length Accuracy by context language Error analysis 5.2 NeuroSymbolic Reasoning (NSAR) Accuracy by context language Effect of kk on performance Error analysis for NSAR and NSAR+3 6 Conclusion References A Prompt and Needle Templates A.1 Needle Template A.2 Prompt Templates License: CC BY 4.0 arXiv:2506.02483v1 [cs.CL] 03 Jun 2025 Enhancing Large Language Models with Neurosymbolic Reasoning for Multilingual Tasks Sina Bagheri Nezhad Email: sina.bagherinezhad@pdx.edu Ameeta Agrawal Email: ameeta@pdx.edu Affiliation: Portland State University, USA Abstract Large language models (LLMs) often struggle to perform multi-target reasoning in long-context scenarios where relevant information is scattered across extensive documents. To address this challenge, we introduce NeuroSymbolic Augmented Reasoning (NSAR), which combines the benefits of neural and symbolic reasoning during inference. NSAR explicitly extracts symbolic facts from text and generates executable Python code to handle complex reasoning steps. Through extensive experiments across seven languages and diverse context lengths, we demonstrate that NSAR significantly outperforms both a vanilla RAG baseline and advanced prompting strategies in accurately identifying and synthesizing multiple pieces of information. Our results highlight the effectiveness of combining explicit symbolic operations with neural inference for robust, interpretable, and scalable reasoning in multilingual settings. 1 Introduction Large language models (LLMs) have achieved impressive progress in natural language processing, yet their ability to handle long-context, cross-lingual reasoning remains limited (Agrawal et al., 2024). In real-world multilingual scenarios, information is often scattered across languages and embedded within lengthy documents, complicating retrieval and coherent reasoning. This challenge is exacerbated by the lost-in-the-middle phenomenon, where LLMs struggle to retain and integrate relevant information from extended contexts (Liu et al., 2023). Although retrieval-augmented generation (RAG) frameworks help narrow the context window, LLMs still frequently fail to combine dispersed facts into a consistent chain of reasoning, resulting in hallucinations and inconsistencies (Zhu et al., 2024; Jiang et al., 2024). In this work, we introduce NeuroSymbolic Augmented Reasoning (NSAR)-a neurosymbolic method that explicitly merges symbolic reasoning with neural inference. NSAR centers on a neurosymbolic prompt that instructs LLMs to extract structured symbolic facts and generate executable code, thus offering a verifiable and interpretable reasoning pathway for complex tasks. By integrating symbolic logic into a retrieval-based framework, NSAR improves interpretability, consistency, and accuracy when working with multilingual and long-context data. The main contributions of our work are as follows: • We propose NSAR, a neurosymbolic reasoning framework that couples retrieval with explicit symbolic reasoning. We develop a dedicated NSAR prompt to structure information into symbolic representations and guide the model in generating executable Python code for final verification. • We show that combining Chain of Thought (CoT), ReAct, and Self-Reflection prompting strategies with neurosymbolic reasoning leads to further performance gains. • Through extensive experiments on cross-lingual, long-context tasks, we demonstrate that NSAR substantially outperforms purely retrieval-based and neural-only methods. 2 Related Works Long-Context and Cross-Lingual Challenges Large language models (LLMs) frequently exhibit a lost-in-the-middle phenomenon in extended contexts (Liu et al., 2023; Xu et al., 2024), causing them to overlook crucial information. Retrieval-augmented frameworks such as LONGEMBED (Zhu et al., 2024), LongRAG (Jiang et al., 2024) and DR-RAG (Hei et al., 2024) alleviate this issue but mostly target monolingual settings. McCrolin (Limkonchotiwat et al., 2024) extends these ideas to cross-lingual retrieval, yet it remains computationally intensive. Bridging queries and contexts in different languages is still difficult (Li et al., 2024; Hengle et al., 2024). Methods such as OPTICAL (Huang et al., 2023) and XAMPLER (Lin et al., 2024) improve multilingual retrieval but face scalability limitations in very long contexts. Parameter-efficient solutions, such as Sparse Fine-Tuning Masks (SFTMs) (Litschko et al., 2022) and adapters, achieve strong multilingual retrieval while reducing resource overhead, but they do not fully address multi-target reasoning. Advanced Prompting Strategies Prompting methods like Chain-of-Thought (CoT) (Wei et al., 2022), ReAct (Yao et al., 2023b), and Self-Reflection (Renze and Guven, 2024) improve LLM reasoning by encouraging more explicit intermediate steps. The Tree-of-Thoughts (ToT) approach (Yao et al., 2023a) further expands these pathways, enhancing structured reasoning. However, such methods rely on text-based explanations that may lack verifiability. Neurosymbolic Reasoning Neurosymbolic approaches offer interpretability and robustness by coupling neural networks with symbolic logic. Hybrid systems like LINC (Olausson et al., 2023) and PAL (Gao et al., 2023) produce executable code, improving transparency and accuracy. Similarly, incorporating symbolic modules into decision-making (Fang et al., 2024) enhances reliability. 3 System Architecture and Approach Our system, as visualized in Figure 1, is composed of two distinct layers. First, a Retrieval Component efficiently narrows down the long, multilingual context using a two-phase Retrieval-Augmented Generation (RAG) framework. Then, the NeuroSymbolic Augmented Reasoning (NSAR) component builds upon this retrieved context to perform explicit neurosymbolic reasoning. NSAR employs a specialized prompt that instructs the language model to extract structured symbolic facts and generate executable Python code for deterministic reasoning. Figure 1: High-level overview of the system, illustrating the two layers: the retrieval component (left) and the neurosymbolic reasoning component (NSAR, right). First, the long context and query are tokenized and embedded to select the most relevant sentences. Next, the NSAR prompt directs the LLM to extract symbolic facts and generate Python code, which is then executed to produce the final answer. 3.1 Retrieval Component The retrieval component extracts the most relevant chunks of information from extensive, multilingual contexts using a Retrieval-Augmented Generation framework. Tokenization and Embedding The input context, which may comprise hundreds of thousands of words across multiple languages, is first segmented into sentences and embedded to capture semantic nuances. We use the Punkt tokenizer (Kiss and Strunk, 2006) for segmentation and the multilingual bge-m3 model (Chen et al., 2024) to generate sentence embeddings in 1024-dimensional embedding size. Candidate Selection and Model Input Next, we compute the semantic distance between each sentence embedding and the query, selecting the top kk most relevant sentences based on a tunable hyperparameter (with kk values of 3, 5, 10, 20, and 50 evaluated in our experiments). These sentences are then passed to the language model as input for the subsequent neurosymbolic reasoning process. 3.2 NeuroSymbolic Augmented Reasoning (NSAR) Purely neural approaches to long-context question answering often struggle with reliability, interpretability, and logically integrating multiple pieces of information. While recent prompting techniques (e.g., Chain-of-Thought, ReAct, and Self-Reflection) have improved reasoning, they remain reliant on implicit neural processes, leaving little room for explicit verification or modular correction. To address these limitations, we introduce the NeuroSymbolic Augmented Reasoning (NSAR) component which integrates structured symbolic representations within the neural architecture by extracting symbolic facts and generating executable Python code for reasoning. This approach bridges the gap between the flexibility and fluency of LLMs and the interpretability and rigor of symbolic methods. Neural systems excel at language understanding and generation, but often fail in complex scenarios requiring multiple reasoning steps, such as comparing multiple facts, deducing the “largest” or “smallest” value, or verifying constraints across scattered pieces of information. Symbolic methods, by contrast, excel in structured reasoning but can be brittle when parsing unstructured text. By coupling a language model with a symbolic layer, NSAR enhances interpretability by providing an explicit record of extracted facts and logical steps, enabling users to audit and verify the model's reasoning. Additionally, it improves reliability by reducing errors in compositional tasks by systematically comparing and fusing pieces of information through symbolic code execution. We design an NSAR prompt to guide the reasoning process through three distinct stages: 1. Symbolic Fact Extraction First, the model is instructed to identify all relevant facts in the provided context and represent them in a structured, symbolic format. For instance, if the context contains lines such as “The special magic Cairo number is: 1234567” and “The special magic Mumbai number is: 9999999”, the model generates: FACT("Cairo", "special_magic_number", 1234567) FACT("Mumbai", "special_magic_number", 9999999) 2. Python Code Generation Next, the model is prompted to produce concise, executable Python code that uses the extracted symbolic facts to answer the question. Instead of implicitly inferring logic through text, the Python code can contain explicit comparisons (>, <, ==), data structures (lists, dictionaries), or domain-specific libraries. In the case of identifying the largest special magic number, this code might look like: numbers = [1234567, 9999999] answer = max(numbers) The logic here can be arbitrarily extended to handle more complex reasoning steps (e.g., filtering facts, applying constraints, or computing aggregates). Once the LLM generates the Python code, it is executed in a controlled environment. 3. Final Answer Extraction Finally, the answer is determined by executing the generated Python code. This guarantees a concise, verified response and prevents any contradictory or incoherent rationales that might arise from purely text-based reasoning. In other words, while the LLM might propose a final textual answer, the actual answer delivered to the user is the deterministic output of the code execution. As such, NSAR prompt structure ensures that the language model provides an interpretable chain of reasoning, resulting in a Python snippet that can be independently executed and verified. The template of NSAR prompt is shown below: In this work, “neurosymbolic” denotes the hybrid of explicit fact extraction with deterministic code execution. While our current triples support only simple attribute logic, extending to richer formalisms (e.g. first-order rules or constraint solvers) would more fully realize the neurosymbolic ideal. 4 Experiments 4.1 Dataset We evaluate the proposed approach on a question answering task encompassing long contexts and multiple languages. We adopt and extend the mLongRR dataset (Agrawal et al., 2024) by increasing the maximum context length to 512,000 words and expanding the language set to English, Vietnamese, Swahili, Persian, Russian, Hindi, and Arabic. This setting incorporates multiple scripts (e.g., Cyrillic, Devanagari) and introduces a cross-lingual challenge: the query is always in English while the context (haystack) is in one of the seven languages. In addition, we randomly place three “needles” (target sentences) within each context haystack, requiring the model to retrieve and compare multiple pieces of relevant information. The contexts consist of news articles ranging in size from 2k, 8k, 16k, 32k, 64k, 128k, 256k, and 512k words. Embedded within these contexts are needles such as: “The special magic {city} number is: {number}”, where {city} is randomly chosen from 23 city names translated into all seven target languages, and {number} is a random 7-digit number (Agrawal et al., 2024; Team, 2024; Anthropic, 2024). Needles appear in the same language as the haystack, ensuring a consistent linguistic setting. Additional details on the needle translation and city selection process can be found in Appendix A.1. 4.2 Evaluation To assess the effectiveness of NSAR in handling complex, cross-lingual reasoning tasks, we adopt an evaluation protocol based on a multi-target scenario-the 3-needles test. Previous studies have reported that the 3-needle scenario is particularly challenging for LLMs operating in long-context settings and low-resource languages (Agrawal et al., 2024). In this test, three needles are randomly placed throughout the context, and the model is prompted with the query: “What is the largest special magic number?” This setup requires the model to locate and compare multiple pieces of information to produce the correct answer, reflecting real-world scenarios where critical information is scattered. We run experiments across eight different context lengths (2k, 8k, 16k, 32k, 64k, 128k, 256k, and 512k words) and five values of kk for retrieved sentences (3, 5, 10, 20, and 50). For each setting, the three needles are randomly positioned in the text. We use accuracy as our main metric, defined as the percentage of cases in which the model correctly identifies the correct response by effectively integrating multiple retrieved facts. The results are reported in terms of average accuracy computed over two runs. 4.3 Models and Baseline Methods We evaluate our approach using two large language models. GPT-4o-mini is a smaller, resource-efficient variant of GPT-4o (OpenAI et al., 2024), designed to trade off some generative capacity for faster inference and lower memory usage, and Llama 3.2 (90B) which is an expanded successor to Llama 2 (Touvron et al., 2023), with refined training procedures and a broader pretraining corpus. We include three relevant baseline prompting strategies like Chain-of-Thought (Wei et al., 2022), ReAct (Yao et al., 2023b), and Self-Reflection (Renze and Guven, 2024) in our evaluation which have demonstrated success in boosting reasoning performance. \subfigure [Accuracy vs. Context Length] \subfigure[Radar Plots of Retrieval Accuracy] Figure 2: Accuracy as a function of context length and across seven context languages. 5 Results and Analysis 5.1 LLM-Only vs. RAG-Vanilla We compare an LLM-only approach, where the entire context is fed directly to the model, with a RAG-Vanilla approach that employs the retrieval component to narrow the context (by identifying the top- kk most relevant sentences) before further processing with the LLM. This selective context narrowing not only improves retrieval accuracy but also significantly reduces computational costs by minimizing the total token count processed by the model. In both methods, we use the same prompt adapted from previous work (see Appendix A.2). The performance results for the RAG-Vanilla approach shown in the figures represent the average accuracy across all k-values and evaluated languages. Context length From the results presented in Figure 2, we observe that both GPT-4o-mini and Llama 3.2 exhibit declining accuracy as context length increases in LLM only (solid lines) scenario, with accuracy dropping to almost 0% as context lengths reach 128K words. In contrast, narrowing the input to a small set of top-relevant sentences using RAG-Vanilla (dashed lines) enables these models to maintain high accuracy even in long-context scenarios (over half a million words). Accuracy by context language Figure 2 shows radar plots comparing accuracy across seven context languages using English prompts for the 3-Needles test. We observe that RAG-Vanilla consistently outperforms the LLM-only baseline across all tested languages, including those with non-Latin scripts such as Persian, Russian, Hindi, and Arabic. Notably, even languages like Swahili and Vietnamese, which differ significantly from English in morphology, benefit from the narrowed context, highlighting the method's cross-lingual robustness. \subfigure [Retrieval failures (red) versus LLM failures (blue) for GPT-4o-mini and Llama 3.2.] \subfigure[Retrieval errors and LLM errors as kk (the number of retrieved sentences) increases.] Figure 3: A combined view of errors in RAG-Vanilla (failure rate % of total queries). Error analysis Although the retrieval-based approach demonstrates significant improvements in accuracy, a closer examination of incorrect cases reveals important insights into its limitations. We analyze the error rates, distinguishing between errors occurring during the embedding retrieval stage and those during the language model's response generation. The errors are categorized into two types: • Retrieval error: The correct sentence is not included among the top- kk retrieved segments, indicating that the embedding model failed to identify the relevant content. • LLM error: The language model fails to extract or correctly reason about the target information, even when the relevant sentence is retrieved. Error rates for each category are calculated as the percentage of total queries that result in that specific type of failure. Figure 3 shows the distribution of retrieval errors (red) versus LLM errors (blue) for GPT-4o-mini and Llama 3.2 90b using RAG-Vanilla. Although retrieval errors remain relatively low (4.6% for both models), the LLM error rates are quite substantial for GPT-4o-mini (63.4%) and Llama 3.2 90b (12.7%), indicating that while RAG-Vanilla effectively narrows the context, the language model itself continues to struggle with multi-target reasoning. We also evaluated how increasing kk (the number of retrieved sentences) influences retrieval errors, as illustrated in Figure 3. Although a larger kk reduces the likelihood of missing relevant information, it introduces additional distractors that can complicate the model's reasoning process. Based on these results, we identify several key factors contributing to LLM failures: • Increased distractors: Multiple similar sentences can confuse the model's reasoning process. • Inconsistent answer prioritization: Determining which label is “largest” or “most relevant” can be nontrivial when multiple valid options exist. • Ambiguity in sentence ranking: Even with successful retrieval, the model may incorrectly prioritize semantically similar sentences when generating its final response. These findings underscore the need for more advanced methods-such as neurosymbolic approaches-to improve multi-target reasoning beyond the gains offered by simple context narrowing. 5.2 NeuroSymbolic Reasoning (NSAR) Although RAG narrows down the context and alleviates many challenges inherent to long input passages, it alone does not guarantee robust multi-target reasoning. To address this shortcoming, we enhance our baseline RAG system with NSAR component. As baselines, besides RAG-Vanilla, we evaluate three prompting-based methods-Chain-of-Thought (CoT), ReAct, and Self-Reflection. We also experiment with a hybrid approach (NSAR+3) that combines NSAR with all three prompting strategies (Chain-of-Thought, ReAct, Self-Reflection, NSAR, NSAR+3). Figure 4: Overall accuracy of GPT-4o-mini (left) and Llama 3.2 90b (right) under different reasoning strategies (RAG-Vanilla, CoT, ReAct, Self-Reflection, NSAR, and a combined approach which combines NSAR with other reasoning methods (NSAR+3)). As shown in Figure 4, the RAG-Vanilla baseline lags behind in multi-target reasoning, confirming that narrowing the context alone does not suffice to fuse and compare multiple pieces of information. In contrast, our proposed approach NSAR substantially improves accuracy by leveraging explicit symbolic extraction and Python-based reasoning. Moreover, combining NSAR with CoT, ReAct, and Self-Reflection (NSAR+3) yields the highest accuracy overall. Notably, for GPT-4o-mini, NSAR achieves 91.1%, followed closely by NSAR+3 and CoT, both at 90.2%, whereas for Llama 3.2, NSAR+3 attains the highest performance at 93.8%. These findings suggest that integrating explicit symbolic reasoning can fill critical gaps in retrieval-augmented generation, particularly for complex tasks that demand robust compositional inference. Accuracy by context language Figures 5 and 5 provide a more granular perspective on how each approach performs across seven context languages. Each cell represents the accuracy (%) of a particular approach-language pair, revealing where certain strategies excel or fall short. \subfigure [GPT-4o mini.] \subfigure[Llama 3.2 90b.] Figure 5: Heatmaps illustrating the accuracy (%) of different approaches (rows) across seven context languages (columns). Darker cells indicate lower accuracy, while lighter cells indicate higher accuracy. Figure 6: Accuracy versus kk (the number of retrieved sentences) for GPT-4o-mini (left) and Llama 3.2 (right). Across both heatmaps, NSAR and NSAR+3 maintain high accuracy in most languages, confirming the benefits of explicit symbolic reasoning for multi-target retrieval tasks. In contrast, baseline methods (RAG-Vanilla) and single-step prompting strategies (Chain-of-Thought, ReAct, Self-Reflection) exhibit more variability and struggle in specific languages. Notably, languages such as Swahili and Arabic appear more challenging, yet neurosymbolic approaches still achieve competitive performance. These language-specific patterns underscore the importance of robust, compositional reasoning-particularly in cross-lingual or lower-resource settings. Effect of kk on performance Figure 6 shows how different reasoning strategies perform as we vary the number of retrieved sentences kk (3, 5, 10, 20, and 50) for GPT-4o-mini (left) and Llama 3.2 90b(right). Several trends are evident. First, at low values of kk, all methods tend to have lower accuracy, likely due to the increased chance of missing key information during retrieval. As kk increases, accuracy generally improves up to a point. However, very large kk (e.g., 50) can introduce additional distractors, leading to a decline in performance. This aligns with our earlier observations that, while a broader retrieval scope reduces the risk of overlooking relevant facts, it can also complicate the model's reasoning by introducing more non-essential content. Comparing across models, Llama 3.2 shows more pronounced fluctuations as kk increases, suggesting it is more sensitive to context size and potential distractors. In contrast, GPT-4o-mini maintains relatively stable performance at intermediate kk values. Notably, NSAR+3 consistently outperforms purely neural prompting methods in Llama 3.2, whereas GPT-4o-mini exhibits closer competition among NSAR and Chain-of-Thought. Overall, these findings highlight the importance of carefully tuning kk to balance retrieval breadth and processing load, while also demonstrating that neurosymbolic reasoning can mitigate many of the challenges introduced by larger context windows. Error analysis for NSAR and NSAR+3 Figure 7 illustrates the distribution of error types for GPT-4o-mini and Llama 3.2 under the NSAR and NSAR+3 methods, categorizing failures into two types: • Facts: The model retrieved the correct segments but failed to extract the target fact from the input. • Code: Although the target fact was extracted correctly, the model produced incorrect or incomplete Python code, leading to an erroneous final answer. Figure 7: Failure rate (% of total queries) for fact extraction vs. code generation errors under NSAR and NSAR+3. The distribution of fact-extraction versus code-generation failures varies notably between the two models and across the two neurosymbolic methods. In NSAR for GPT-4o-mini, most errors stem from fact extraction, whereas Llama 3.2 primarily struggles with code generation. This suggests that GPT-4o-mini's neurosymbolic pipeline struggles in locating the correct sentences to extract facts, while Llama 3.2 successfully extracts facts but sometimes produces flawed Python code. Moving from NSAR to NSAR+3 reverses this trend for Llama 3.2, significantly reducing code-generation failures but leading to more fact-extraction issues. Meanwhile, GPT-4o-mini exhibits a small increase in fact-extraction errors and a modest rise in code-generation errors when switching to NSAR+3. Overall, these results imply that while neurosymbolic reasoning substantially mitigates retrieval shortcomings, the balance between accurate fact extraction and correct code generation can shift depending on the underlying model and the specific prompting strategy. 6 Conclusion We presented a neurosymbolic reasoning module (NSAR) to address the challenges of multi-target reasoning in long-context, cross-lingual settings. While RAG effectively narrows the context and reduces computational overhead, purely neural models often struggle to integrate and compare multiple pieces of information. NSAR overcomes these limitations by extracting symbolic facts and generating executable Python code, enabling transparent, verifiable reasoning. Our experiments across seven languages demonstrate that NSAR significantly outperforms both the vanilla RAG baseline and advanced prompting strategies (Chain-of-Thought, ReAct, and Self-Reflection). Future work will expand NSAR's symbolic scope beyond numeric lookups-into graph queries, set operations, and constraint satisfaction-and evaluate on broader tasks to test generalizability of our reasoning pipeline. Acknowledgments We thank the anonymous reviewers for their valuable feedback. References Agrawal et al. (2024) Ameeta Agrawal, Andy Dang, Sina Bagheri Nezhad, Rhitabrat Pokharel, and Russell Scheinberg. Evaluating multilingual long-context models for retrieval and reasoning, 2024. URL https://arxiv.org/abs/2409.18006. Anthropic (2024) Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf. Chen et al. (2024) Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 2318-2335, Bangkok, Thailand, August 2024. Association for Computational Linguistics. 10.18653/v1/2024.findings-acl.137. URL https://aclanthology.org/2024.findings-acl.137. Fang et al. (2024) Meng Fang, Shilong Deng, Yudi Zhang, Zijing Shi, Ling Chen, Mykola Pechenizkiy, and Jun Wang. Large language models are neurosymbolic reasoners. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 17985-17993, 2024. Gao et al. (2023) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models, 2023. URL https://arxiv.org/abs/2211.10435. Hei et al. (2024) Zijian Hei, Weiling Liu, Wenjie Ou, Juyi Qiao, Junming Jiao, Guowen Song, Ting Tian, and Yi Lin. Dr-rag: Applying dynamic document relevance to retrieval-augmented generation for question-answering, 2024. URL https://arxiv.org/abs/2406.07348. Hengle et al. (2024) Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models, 2024. URL https://arxiv.org/abs/2408.10151. Huang et al. (2023) Zhiqi Huang, Puxuan Yu, and James Allan. Improving cross-lingual information retrieval on low-resource languages via optimal transport distillation. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM '23, page 1048-1056. ACM, February 2023. 10.1145/3539597.3570468. URL http://dx.doi.org/10.1145/3539597.3570468. Jiang et al. (2024) Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval-augmented generation with long-context llms, 2024. URL https://arxiv.org/abs/2406.15319. Kiss and Strunk (2006) Tibor Kiss and Jan Strunk. Unsupervised multilingual sentence boundary detection. Computational Linguistics, 32(4):485-525, 12 2006. ISSN 0891-2017. 10.1162/coli.2006.32.4.485. URL https://doi.org/10.1162/coli.2006.32.4.485. Li et al. (2024) Bryan Li, Samar Haider, Fiona Luo, Adwait Agashe, and Chris Callison-Burch. Bordirlines: A dataset for evaluating cross-lingual retrieval-augmented generation, 2024. URL https://arxiv.org/abs/2410.01171. Limkonchotiwat et al. (2024) Peerat Limkonchotiwat, Wuttikorn Ponwitayarat, Lalita Lowphansirikul, Potsawee Manakul, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, and Sarana Nutanong. McCrolin: Multi-consistency cross-lingual training for retrieval question answering. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2780-2793, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-emnlp.157. Lin et al. (2024) Peiqin Lin, André F. T. Martins, and Hinrich Schütze. Xampler: Learning to retrieve cross-lingual in-context examples, 2024. URL https://arxiv.org/abs/2405.05116. Litschko et al. (2022) Robert Litschko, Ivan Vulić, and Goran Glavaš. Parameter-efficient neural reranking for cross-lingual and multilingual retrieval. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na, editors, Proceedings of the 29th International Conference on Computational Linguistics, pages 1071-1082, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https://aclanthology.org/2022.coling-1.90. Liu et al. (2023) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023. URL https://arxiv.org/abs/2307.03172. Olausson et al. (2023) Theo Olausson, Alex Gu, Ben Lipkin, Cedegao Zhang, Armando Solar-Lezama, Joshua Tenenbaum, and Roger Levy. LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5153-5176, Singapore, December 2023. Association for Computational Linguistics. 10.18653/v1/2023.emnlp-main.313. URL https://aclanthology.org/2023.emnlp-main.313/. OpenAI et al. (2024) OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian O'Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276. Renze and Guven (2024) Matthew Renze and Erhan Guven. Self-reflection in llm agents: Effects on problem-solving performance, 2024. URL https://arxiv.org/abs/2405.06682. Team (2024) Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530. Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv.org/abs/2307.09288. Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022. Xu et al. (2024) Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=xw5nxFWMlo. Yao et al. (2023a) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809-11822, 2023a. Yao et al. (2023b) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=WE_vluYUL-X. Zhu et al. (2024) Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey, 2024. URL https://arxiv.org/abs/2308.07107. Appendix A Prompt and Needle Templates This appendix presents the templates and data used in our experiments. For the corresponding versions of Needles in other languages (Swahili, Vietnamese, Persian, Hindi, Arabic, and Russian), please refer to the anonymous repository 11 1 https://anonymous.4open.science/r/prompts-template-D729. A.1 Needle Template "The special magic {city} number is {number}." Here, number represents a randomly generated 7-digit number, and city is selected at random from the list below (with the city names translated into the context language): • Chicago • Yangon • Antananarivo • Colombo • Almaty • Sydney • Mexico City • Seattle • Lagos • Amsterdam • Belgrade • Cairo • Baghdad • Damascus • Kigali • Dakar • Sofia • Victoria • Tashkent • Mumbai • Barcelona • Amman • Toronto A.2 Prompt Templates This appendix presents the complete set of prompt templates used in our experiments, each designed to elicit concise and direct answers from a large language model (LLM). All prompts share the convention that contextual information is enclosed between #CONTEXT and #ENDCONTEXT, followed by a specific question. • Vanilla Prompts: Adapted from Agrawal et al. (2024), provide a baseline retrieval-focused setup without any specialized reasoning strategy. • Chain-of-Thought (CoT): Encourages the LLM to articulate its reasoning steps in natural language before arriving at a final answer, aiming to improve transparency and correctness. • ReAct: Combines “thinking” (Thought) and “acting” (Action) phases, prompting the LLM to explicitly delineate its internal reasoning and the subsequent steps taken to reach an answer. • Self-Reflection: Prompts the model to not only provide a step-by-step solution but also critically review its reasoning for potential errors, thus refining the final answer. • NSAR: Introduces a neurosymbolic prompt that extracts symbolic facts and generates executable Python code to produce a deterministic, verifiable answer. • NSAR+3: A hybrid method that combines neurosymbolic reasoning with Chain-of-Thought, ReAct, and Self-Reflection, offering a comprehensive approach to both interpretability and robust logic. Below, we detail each prompt template: Each of these templates is tailored to evaluate different aspects of the LLM's reasoning and retrieval capabilities. The Vanilla prompts test basic and multi-target retrieval scenarios, while Chain-of-Thought, ReAct, and Self-Reflection aim to improve intermediate reasoning steps. Meanwhile, NSAR introduces an explicit neurosymbolic approach for verifiable multi-target logic, and NSAR+3 extends this by integrating advanced prompting strategies for even more robust reasoning. Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" () button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. We gratefully acknowledge support from our major funders, member institutions, , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2507.06261.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2507.06261.txt new file mode 100644 index 0000000..9142944 --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2507.06261.txt @@ -0,0 +1 @@ +Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Model Architecture, Training and Dataset 2.1 Model Architecture 2.2 Dataset 2.3 Training Infrastructure 2.4 Post-training 2.5 Thinking 2.6 Capability-specific improvements 2.7 The path to Gemini 2.5 3 Quantitative evaluation 3.1 Methodology 3.2 Core capability quantitative results 3.3 Evaluation of Gemini 2.5 Pro against other large language models 4 Example use cases of Gemini 2.5 Pro 4.1 Gemini Plays Pokémon Long Context Agentic Tooling Long Horizon Task Coherence Screen reading Long Context Reasoning 4.2 What else can Gemini 2.5 do? 4.3 Gemini in Google Products 5 Safety, Security, and Responsibility Guideline for Navigating This Section 5.1 Our Process 5.2 Policies and Desiderata 5.3 Training for Safety, Security, and Responsibility 5.4 Results on Training/Development Evaluations 5.5 Automated Red Teaming 5.6 Memorization and Privacy 5.7 Assurance Evaluations and Frontier Safety Framework 5.8 External Safety Testing 6 Discussion References 7 Contributors and Acknowledgments 8 Appendix 8.1 Evaluation additional details 8.2 Gemini Plays Pokémon Additional Details Agent Panic 8.3 Frontier Safety Framework Evaluations Additional Details: Frontier Safety Correctness Tests 8.4 Image to Code Demo 8.5 Long Context Video Understanding Demo License: CC BY 4.0 arXiv:2507.06261v6 [cs.CL] 19 Dec 2025 \uselogo Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. Gemini Team Google Abstract In this report, we introduce the Gemini 2.X model family: Gemini 2.5 Pro and Gemini 2.5 Flash, as well as our earlier Gemini 2.0 Flash and Flash-Lite models. Gemini 2.5 Pro is our most capable model yet, achieving SoTA performance on frontier coding and reasoning benchmarks. In addition to its incredible coding and reasoning skills, Gemini 2.5 Pro is a thinking model that excels at multimodal understanding and it is now able to process up to 3 hours of video content. Its unique combination of long context, multimodal and reasoning capabilities can be combined to unlock new agentic workflows. Gemini 2.5 Flash provides excellent reasoning abilities at a fraction of the compute and latency requirements and Gemini 2.0 Flash and Flash-Lite provide high performance at low latency and cost. Taken together, the Gemini 2.X model generation spans the full Pareto frontier of model capability vs cost, allowing users to explore the boundaries of what is possible with complex agentic problem solving. 1 Introduction We present our latest family of natively multimodal models with advanced reasoning through thinking, long context and tool-use capabilities: Gemini 2.5 Pro and 2.5 Flash and our earlier Gemini 2.0 Flash and Gemini 2.0 Flash-Lite models. Together these form a new family of highly-capable models representing our next generation of AI models, designed to power a new era of agentic systems. Building upon the foundation of the Gemini 1.5 series (Gemini Team, 2024), this Gemini 2.X generation brings us closer to the vision of a universal AI assistant (Hassabis, 2025). The Gemini 2.X series are all built to be natively multimodal, supporting long context inputs of >1 million tokens and have native tool use support. This allows them to comprehend vast datasets and handle complex problems from different information sources, including text, audio, images, video and even entire code repositories. These extensive capabilities can also be combined to build complex agentic systems, as happened in the case of Gemini Plays Pokémon11 1 Pokémon is a trademark of Nintendo Co., Ltd., Creatures Inc., and Game Freak Inc. (Zhang, 2025). Different models in the series have different strengths and capabilities: (1) Gemini 2.5 Pro is our most intelligent thinking model, exhibiting strong reasoning and code capabilities. It excels at producing interactive web applications, is capable of codebase-level understanding and also exhibits emergent multimodal coding abilities. (2) Gemini 2.5 Flash is our hybrid reasoning model with a controllable thinking budget, and is useful for most complex tasks while also controlling the tradeoff between quality, cost, and latency. (3) Gemini 2.0 Flash is our fast and cost-efficient non-thinking model for everyday tasks and (4) Gemini 2.0 Flash-Lite is our fastest and most cost-efficient model, built for at-scale usage. A full comparison of the models in the Gemini 2.X model family is provided in Table 1. Taken together, the Gemini 2.X family of models cover the whole Pareto frontier of model capability vs cost, shifting it forward across a large variety of core capabilities, applications and use-cases, see Figure 1. FlashGemini 1.5 ProGemini 1.5 Flash-LiteGemini 2.0 FlashGemini 2.0 FlashGemini 2.5 ProGemini 2.5 Input modalities Video, AudioText, Image, Video, AudioText, Image, Video, AudioText, Image, Video, AudioText, Image, Video, AudioText, Image, Video, AudioText, Image, Input length 1M 2M 1M 1M 1M 1M Output modalities Text Text Text Text, Image* Text, Audio* Text, Audio* Output length 8K 8K 8K 8K 64K 64K Thinking No No No Yes* Dynamic Dynamic Supports tool use? No No No Yes Yes Yes Knowledge cutoff November 2023 November 2023 June 2024 June 2024 January 2025 January 2025 Table 1: Comparison of Gemini 2.X model family with Gemini 1.5 Pro and Flash. Tool use refers to the ability of the model to recognize and execute function calls (e.g., to perform web search, complete a math problem, execute code). *currently limited to Experimental or Preview, see Section 2.7. Information accurate as of publication date. The Gemini 2.5 family of models maintain robust safety metrics while improving dramatically on helpfulness and general tone compared to their 2.0 and 1.5 counterparts. In practice, this means that the 2.5 models are substantially better at providing safe responses without interfering with important use cases or lecturing end users. We also evaluated Gemini 2.5 Pro's Critical Capabilities, including CBRN, cybersecurity, machine learning R&D, and deceptive alignment. While Gemini 2.5 Pro showed a significant increase in some capabilities compared to previous Gemini models, it did not reach any of the Critical Capability Levels in any area. Our report is structured as follows: we begin by briefly describing advances we have made in model architecture, training and serving since the release of the Gemini 1.5 model. We then showcase the performance of the Gemini 2.5 models, including qualitative demonstrations of its abilities. We conclude by discussing the safety evaluations and implications of this model series. Figure 1: Cost-performance plot. Gemini 2.5 Pro is a marked improvement over Gemini 1.5 Pro, and has an LMArena score that is over 120 points higher than Gemini 1.5 Pro. Cost is a weighted average of input and output tokens pricing per million tokens. Source: LMArena, imported on 2025-06-16. Figure 2: Number of output tokens generated per second (after the first chunk has been received from the API) for different models. Source: ArtificialAnalysis.ai, imported on 2025-06-15. 2 Model Architecture, Training and Dataset 2.1 Model Architecture The Gemini 2.5 models are sparse mixture-of-experts (MoE) (Clark et al., 2022; Du et al., 2021; Fedus et al., 2021; Lepikhin et al., 2020; Shazeer et al., 2017; Roller et al., 2021; Riquelme et al., 2021; Jiang et al., 2024) transformers (Vaswani et al., 2017) with native multimodal support for text, vision, and audio inputs. Sparse MoE models activate a subset of model parameters per input token by learning to dynamically route tokens to a subset of parameters (experts); this allows them to decouple total model capacity from computation and serving cost per token. Developments to the model architecture contribute to the significantly improved performance of Gemini 2.5 compared to Gemini 1.5 Pro (see Section 3). Despite their overwhelming success, large transformers and sparse MoE models are known to suffer from training instabilities (Chowdhery et al., 2022; Dehghani et al., 2023; Zhang et al., 2022; Molybog et al., 2023; Wortsman et al., 2023; Liu et al., 2020; Zhai et al., 2023; Fedus et al., 2021; Lepikhin et al., 2020). The Gemini 2.5 model series makes considerable progress in enhancing large-scale training stability, signal propagation and optimization dynamics, resulting in a considerable boost in performance straight out of pre-training compared to previous Gemini models. Gemini 2.5 models build on the success of Gemini 1.5 in processing long-context queries, and incorporate new modeling advances allowing Gemini 2.5 Pro to surpass the performance of Gemini 1.5 Pro in processing long context input sequences of up to 1M tokens (see Table 3). Both Gemini 2.5 Pro and Gemini 2.5 Flash can process pieces of long-form text (such as the entirety of “Moby Dick” or “Don Quixote”), whole codebases, and long form audio and video data (see Appendix 8.5). Together with advancements in long-context abilities, architectural changes to Gemini 2.5 vision processing lead to a considerable improvement in image and video understanding capabilities, including being able to process 3-hour-long videos and the ability to convert demonstrative videos into interactive coding applications (see our recent blog post by Baddepudi et al., 2025). The smaller models in the Gemini 2.5 series - Flash size and below - use distillation (Hinton et al., 2015; Anil et al., 2018), as was done in the Gemini 1.5 series (Gemini Team, 2024). To reduce the cost associated with storing the teacher's next token prediction distribution, we approximate it using a k-sparse distribution over the vocabulary. While this still increases training data throughput and storage demands by a factor of k, we find this to be a worthwhile trade-off given the significant quality improvement distillation has on our smaller models, leading to high-quality models with a reduced serving cost (see Figure 2). 2.2 Dataset Our pre-training dataset is a large-scale, diverse collection of data encompassing a wide range of domains and modalities, which includes publicly available web documents, code (various programming languages), images, audio (including speech and other audio types) and video, with a cutoff date of June 2024 for 2.0 and January 2025 for 2.5. Compared to the Gemini 1.5 pre-training dataset we also utilized new methods for improved data quality for both filtering, and deduplication. Our post-training dataset, like Gemini 1.5, consists of instruction tuning data that is carefully collected and vetted. It is a collection of multimodal data with paired instructions and responses, in addition to human preference and tool-use data. 2.3 Training Infrastructure This model family is the first to be trained on TPUv5p architecture. We employed synchronous data-parallel training to parallelise over multiple 8960-chip pods of Google's TPUv5p accelerators, distributed across multiple datacenters. The main advances in software pre-training infrastructure compared with Gemini 1.5 were related to elasticity and mitigation of SDC (Silent Data Corruption) errors: 1. Slice-Granularity Elasticity: Our system now automatically continues training with fewer “slices” of TPU chips when there is a localized failure, and this reconfiguration results in tens of seconds of lost training time per interruption, compared with the 10 or more minute delay waiting for healthy machines to be rescheduled without elasticity; the system continues training at around 97% throughput while the failed slice is recovering. At the scale of this training run we see interruptions from hardware failures multiple times per hour, but our fault tolerance machinery is designed to tolerate the higher failure rates expected at much larger scales. 2. Split-Phase SDC Detection: On previous large-scale runs it could take many hours to detect and localize machines with SDC errors, requiring both downtime while debugging, and rollback/replay of a large number of potentially corrupt training steps. We now use lightweight deterministic replay to immediately repeat any step with suspicious metrics, and compare per-device intermediate checksums to localize the root cause of any data corruption. Empirically, accelerators that start to exhibit intermittent SDCs are identified within a few minutes, and quickly excluded from the job. During this run, around 0.25% of steps were replayed due to suspected SDCs and 6% of these replays turned out to be genuine hardware corruption. Both of the above techniques were relatively simple to implement due to the single-controller design of the Pathways system (Barham et al., 2022), which allows all accelerators to be coordinated from a single python program with a global view of the system state. The controller can make use of parallel ‘remote python' operations on TPU workers to monitor training metrics, track performance stragglers, and root-cause SDC errors. Overall during the run, 93.4% of the time was spent performing TPU computations; the remainder was approximately spent half in elastic reconfigurations, and half in rare tail cases where elasticity failed. Around 4.5% of the computed steps were replays or rollbacks for model debugging interventions. 2.4 Post-training Since the initial announcement of Gemini 1.5, significant advancements have been made in our post-training methodologies, driven by a consistent focus on data quality across the Supervised Fine-Tuning (SFT), Reward Modeling (RM), and Reinforcement Learning (RL) stages. A key focus has been leveraging the model itself to assist in these processes, enabling more efficient and nuanced quality control. Furthermore, we have increased the training compute allocated to RL, allowing deeper exploration and refinement of model behaviors. This has been coupled with a focus on verifiable rewards and model-based generative rewards to provide more sophisticated and scalable feedback signals. Algorithmic changes to the RL process have also improved stability during longer training. These advancements have enabled Gemini 2.5 to learn from more diverse and complex RL environments, including those requiring multi-step actions and tool use. The combination of these improvements in data quality, increased compute, algorithmic enhancements, and expanded capabilities has contributed to across-the-board performance gains (as described in Section 3) , notably reflected in the significant increase in the model's LMArena Elo scores, with both Gemini 2.5 Flash and Pro gaining more than 110 points over their Gemini 1.5 counterparts (122 for Gemini 2.5 Pro and 111 for Gemini 2.5 Flash, see Figure 1), along with significant improvements on several other frontier benchmarks. 2.5 Thinking Figure 3: Impact of “Thinking” on Gemini's performance on AIME 2025 (Balunović et al., 2025), LiveCodeBench (corresponding to 10/05/2024 - 01/04/2025 in the UI) (Jain et al., 2024) and GPQA diamond (Rein et al., 2024) benchmarks. Past Gemini models produce an answer immediately following a user query. This constrains the amount of inference-time compute (Thinking) that our models can spend reasoning over a problem. Gemini Thinking models are trained with Reinforcement Learning to use additional compute at inference time to arrive at more accurate answers. The resulting models are able to spend tens of thousands of forward passes during a “thinking” stage, before responding to a question or query. Our training recipe has evolved from the original experimental thinking model, Gemini 2.0 Flash Thinking (launched in December 2024), to the Gemini 2.5 Thinking series, which incorporates Thinking natively across all domains. The result is a single model that can achieve stronger reasoning performance across the board, and is able to scale up its performance further as a function of inference time (see Figure 3 for an example of the impact of Thinking). We integrated Thinking with other Gemini capabilities, including native multimodal inputs (images, text, video, audio) and long context (1M+ tokens). For any of these capabilities, the model decides for itself how long to think before providing an answer. We also provide the ability to set a Thinking budget, constraining the model to respond within a desired number of tokens. This allows users to trade off performance with cost. To demonstrate this capability, we conducted experiments where we systematically varied the thinking budget, measured in the number of tokens the model is allowed to use for internal computation. As shown in Figure 4, increasing this budget allows the model to scale its performance and achieve significantly higher accuracy. Figure 4: Impact of thinking budget on performance on AIME 2025 (Balunović et al., 2025), LiveCodeBench (corresponding to 10/05/2024 - 01/04/2025 in the UI) (Jain et al., 2024) and GPQA diamond (Rein et al., 2024) benchmarks. 2.6 Capability-specific improvements While most of the changes made to our training architecture and recipe since Gemini 1.5 have resulted in improvements across all capabilities, we have also made changes that have resulted in some capability-specific wins. We will now discuss these for code, factuality, long context, multilinguality, audio, video, and agentic use cases (with a particular focus on Gemini Deep Research). Code Gemini 2.0 and 2.5 represent a strategic shift of our development priorities towards delivering tangible real-world value, empowering users to address practical challenges and achieve development objectives within today's complex, multimodal software environments. To realize this, concerted efforts have been undertaken across both pre-training and post-training phases since Gemini 1.5. In pre-training, we intensified our focus on incorporating a greater volume and diversity of code data from both repository and web sources into the training mixture. This has rapidly expanded coverage and enabled the development of more compute-efficient models. Furthermore, we have substantially enhanced our suite of evaluation metrics for assessing code capabilities aligned with downstream use cases, alongside improving our ability to accurately predict model performance. During post-training, we developed novel training techniques incorporating reasoning capabilities and curated a diverse set of engineering tasks, with the aim to equip Gemini with effective problem-solving skills crucial for addressing modern engineering challenges. Key applications demonstrating these advancements include IDE functionalities, code agent use cases for complex, multi-step operations within full repositories, and multimodal, interactive scenarios such as end-to-end web and mobile application development. Collectively, these efforts have yielded broad and significant improvements in Gemini's coding capabilities. This progress is evidenced by superior performance on established benchmarks: performance on LiveCodeBench (Jain et al., 2024) increased from 30.5% for Gemini 1.5 Pro to 74.2% for Gemini 2.5 Pro, while that for Aider Polyglot (Gauthier, 2025) went from 16.9% to 82.2%. Performance on SWEBench-verified (Jimenez et al., 2024; Chowdhury et al., 2024) went from 34.2% to 67.2%, see Table 3 and Figure 5 in Section 3.2. Furthermore, Gemini 2.5 Pro obtained an increase of over 500 Elo over Gemini 1.5 Pro on the LMArena WebDev Arena (LMArena Team, 2025; Chiang et al., 2024), resulting in meaningful enhancements in practical applications, including UI and web application development (Doshi, 2025a), and the creation of sophisticated agentic workflows (Kilpatrick, 2025). Factuality Within the context of generative models, ensuring the factuality of model responses to information-seeking prompts remains a core pillar of Gemini model development. With Gemini 1.5, our research was concentrated on enhancing the model's world knowledge and its ability to provide answers faithfully grounded in the context provided within the prompt. This effort culminated in the December 2024 release of FACTS Grounding (Jacovi et al., 2025), now an industry-standard benchmark for evaluating an LLM's capacity to generate responses grounded in user-provided documents. With Gemini 2.0 and 2.5, we have significantly expanded our scope to address multimodal inputs, long-context reasoning, and model-retrieved information. At the same time, the landscape and user expectations for factuality have evolved dramatically, shaped in part by Google's deployment of AI Overviews and AI Mode (Stein, 2025). To meet these demands, Gemini 2.0 marked a significant leap as our first model family trained to natively call tools like Google Search, enabling it to formulate precise queries and synthesize fresh information with sources. Building on this, Gemini 2.5 integrates advanced reasoning, allowing it to interleave these search capabilities with internal thought processes to answer complex, multi-hop queries and execute long-horizon tasks. The model has learned to use search and other tools, reason about the outputs, and issue additional, detailed follow-up queries to expand the information available to it and to verify the factual accuracy of the response. Our latest models now power the experiences of over 1.5B monthly active users in Google's AI Overviews and 400M users in the Gemini App. These models exhibit state-of-the-art performance across a suite of factuality benchmarks, including SimpleQA for parametric knowledge (Wei et al., 2024), FACTS Grounding for faithfulness to provided documents (Jacovi et al., 2024; Jacovi et al., 2025), and the Vectara Hallucination Leaderboard (Hughes et al., 2023), cementing Gemini as the model of choice for information-seeking demands. Long context Modeling and data advances helped us improve the quality of our models' responses to queries utilizing our one million-length context window, and we reworked our internal evaluations to be more challenging to help steer our modeling research. When hill-climbing, we targeted challenging retrieval tasks (like LOFT of Lee et al., 2024), long-context reasoning tasks (like MRCR-V2 of Vodrahalli et al., 2024), and multimodal tasks (like VideoMME of Fu et al., 2025). According to the results in Table 6, the new 2.5 models improve greatly over previous Gemini 1.5 models and achieve state-of-the-art quality on all of those. An example showcasing these improved capabilities for video recall can be seen in Appendix 8.5, where Gemini 2.5 Pro is able to consistently recall a 1 second visual event out of a full 46-minute video.22 2 For further discussion on long context capabilities, challenges, and future outlook, the Release Notes podcast episode “Deep Dive into Long Context” provides additional insights and discussion: https://youtu.be/NHMJ9mqKeMQ. Multilinguality Gemini's multilingual capabilities have also undergone a profound evolution since 1.5, which already encompassed over 400 languages via pretraining. This transformation stems from a holistic strategy, meticulously refining pre- and post-training data quality, advancing tokenization techniques, innovating core modeling, and executing targeted capability hillclimbing. The impact is particularly striking in Indic and Chinese, Japanese and Korean languages, where dedicated optimizations in data quality and evaluation have unlocked dramatic gains in both quality and decoding speed. Consequently, users benefit from significantly enhanced language adherence, responses designed to faithfully respect the requested output language, and a robust improvement in generative quality and factuality across languages, solidifying Gemini's reliability across diverse linguistic contexts. Audio While Gemini 1.5 was focused on native audio understanding tasks such as transcription, translation, summarization and question-answering, in addition to understanding, Gemini 2.5 was trained to perform audio generation tasks such as text-to-speech or native audio-visual to audio out dialog. To enable low-latency streaming dialog, we incorporated causal audio representations that also allow streaming audio into and out of Gemini 2.5. These capabilities derive from an increased amount of pre-training data spanning over 200 languages, and development of improved post-training recipes. Finally, through our improved post-training recipes, we have integrated advanced capabilities such as thinking, affective dialog, contextual awareness and tool use into Gemini's native audio models. Video We have significantly expanded both our pretraining and post-training video understanding data, improving the audio-visual and temporal understanding capabilities of the model. We have also trained our models so that they perform competitively with 66 instead of 258 visual tokens per frame, enabling using about 3 hours of video instead of 1h within a 1M tokens context window33 3 This is referred to as low media resolution in the API: https://ai.google.dev/api/generate-content#MediaResolution.. Two new applications that were not previously possible, but that have been unlocked as a result of these changes are: creating an interactive app from a video (such as a quiz to test students' understanding of the video content) and creating a p5.js animation to show the key concepts from the video. Our recent blog post (Baddepudi et al., 2025) shows examples of these applications. Gemini as an Agent: Deep Research Gemini Deep Research (Gemini Team, Google, 2024) is an agent built on top of the Gemini 2.5 Pro model designed to strategically browse the web and provide informed answers to even the most niche user queries. The agent is optimized to perform task prioritization, and is also able to identify when it reaches a dead-end when browsing. We have massively improved the capabilities of Gemini Deep Research since its initial launch in December 2024. As evidence of that, performance of Gemini Deep Research on the Humanity's Last Exam benchmark (Phan et al., 2025) has gone from 7.95% in December 2024 to the SoTA score of 26.9% and 32.4% with higher compute (June 2025). 2.7 The path to Gemini 2.5 On the way to Gemini 2.5 Pro, we experimented with our training recipe, and tested a small number of these experimental models with users. We have already discussed Gemini 2.0 Flash Thinking (see Section 2.5). We will now discuss some of the other models briefly. Gemini 2.0 Pro In February 2025, we released an experimental version of Gemini 2.0 Pro. At the time, it had the strongest coding performance of any model in the Gemini model family, as well as the best understanding and world knowledge. It also came with our largest context window at 2 million tokens, which enabled it to comprehensively analyze and understand vast amounts of information. For further information about Gemini 2.0 Pro, please see our earlier blog posts (Mallick and Kilpatrick, 2025; Kavukcuoglu, 2025). Gemini 2.0 Flash Native Image Generation Model In March 2025, we released an experimental version of Gemini 2.0 Flash Native Image Generation. It has brought to the users new capabilities as a result of a strong integration between the Gemini model and image-generation capabilities, enabling new experiences related to image generation & image editing via natural-language prompting. Capabilities such as multi-step conversational editing or interleaved text-image generation are very natural in such a setting, and horizontal transfer related to multi-language coverage immediately allowed such experiences to happen across all the languages supported by the Gemini models. Native image generation turns Gemini into a multimodal creation partner and enables Gemini to express ideas through both text and images, and to seamlessly move between the two. For further information about Gemini 2.0 Flash Native Image Generation, please see our earlier blog posts (Kampf and Brichtova, 2025; Sharon, 2025) Gemini 2.5 Audio Generation With Gemini 2.5, the Controllable TTS and Native Audio Dialog capabilities are available as separate options on AI Studio (Generate Media and Stream sections respectively). Our Gemini 2.5 Preview TTS Pro and Flash models support more than 80 languages with the speech style controlled by a free formatted prompt which can specify style, emotion, pace, etc, while also being capable of following finer-grained steering instructions specified in the transcript. Notably, Gemini 2.5 Preview TTS can generate speech with multiple speakers, which enables the creation of podcasts as used in NotebookLM Audio Overviews (Wang, 2024). Our Gemini 2.5 Flash Preview Native Audio Dialog model uses native audio generation, which enables the same level of style, pacing and accent control as available in our controllable TTS offering. Our dialog model supports tool use and function calling, and is available in more than 24 languages. With native audio understanding and generation capabilities, it can understand and respond appropriately to the user's tone. This model is also capable of understanding when to respond to the user, and when not to respond, ignoring background and non-device directed audio. Finally, we also offer an advanced ‘Thinking' variant that effectively handles more complex queries and provides more robust and reasoned responses in exchange for some additional latency. Gemini 2.5 Flash-Lite In June 2025, we released an experimental version of Gemini 2.5 Flash-Lite (gemini-2.5-flashlite-preview-06-17). It comes with the same capabilities that make Gemini 2.5 helpful, including the ability to turn thinking on at different budgets, connecting to tools like Google Search and code execution, support for multimodal inputs and a 1 million-token context length. Our goal was to provide an economical model class which provides ultra-low-latency capabilities and high throughput per dollar, echoing the initial release of 2.0 Flash-Lite (Google DeepMind, 2025b; Mallick and Kilpatrick, 2025). Gemini 2.5 Pro Deep Think To advance Gemini's capabilities towards solving hard reasoning problems, we developed a novel reasoning approach, called Deep Think, that naturally blends in parallel thinking techniques during response generation. Deep Think enables Gemini to creatively produce multiple hypotheses and carefully critique them before arriving at the final answer, achieving state-of-the-art performances in challenging benchmarks such as Olympiad math (USAMO 2025), competitive coding (LiveCodeBench), and multimodality (MMMU), see more details at (Doshi, 2025b). We announced Gemini 2.5 Deep Think at Google I/O and launched an experimental version to trusted testers and advanced users in June 2025. 3 Quantitative evaluation Figure 5: Performance of Gemini 2.X models at coding, math and reasoning tasks in comparison to previous Gemini models. SWE-bench verified numbers correspond to the “multiple attempts” setting reported in Table 3. We will now examine the performance of the Gemini 2.X model family across a wide range of benchmarks. We will first compare the performance of the Gemini 2.X models to the earlier Gemini 1.5 Pro and Flash models, before we compare the performance of Gemini 2.5 Pro to other available large language models. With web-scale pre-training of AI models, coupled with the post-training techniques that allow policy and reward models to leverage public benchmarks, avoiding leaks and biases in the data used for pre- and post-training is a persistent challenge. In the development of the Gemini 2.5 series, in addition to the standard n-gram based decontamination we used in Gemini 1.5, we also employed semantic-similarity and model based decontamination procedures to help mitigate evaluation set leakage. To move beyond the reliance on training set decontamination, we also continue reporting on internally developed non-public benchmarks, such as HiddenMath. 3.1 Methodology Model AI Studio model ID Gemini 1.5 Flash gemini-1.5-flash-002 Gemini 1.5 Pro gemini-1.5-pro-002 Gemini 2.0 Flash-Lite gemini-2.0-flash-lite-001 Gemini 2.0 Flash gemini-2.0-flash-001 Gemini 2.5 Flash gemini-2.5-flash Gemini 2.5 Pro gemini-2.5-pro Table 2: Mapping of Gemini model names to AI Studio API model IDs. In Table 3, we compare the performance of Gemini 2.5 models to the Gemini 1.5 models, while in Table 4, we compare the performance of Gemini 2.5 Pro to that of other large language models. Gemini results: All Gemini scores are pass@1, and are “single attempt” settings unless otherwise specified. In the “single attempt” setting, no majority voting or parallel test-time compute is permitted, while in the “multiple attempts” setting, test-time selection of the candidate answer is allowed. All Gemini evaluations are run with the AI Studio API for the model id that we provide in Table 2, with default sampling settings. To reduce variance, we average over multiple trials for smaller benchmarks. Aider Polyglot scores are the pass rate average of 3 trials. Vibe-Eval results are reported using Gemini as a judge. Non-Gemini results: All the results for non-Gemini models are sourced from providers' self reported numbers unless mentioned otherwise. All “SWE-bench Verified” numbers follow official provider reports, which means that they are computed using different scaffoldings and infrastructure, and aren't directly comparable. For some evaluations, we obtain results from the external leaderboards that report results on these benchmarks. Results for Humanity's Last Exam results are sourced from Scale's leaderboard and results for DeepSeek are obtained from the text-only variant of the leaderboard (indicated with a ⋄\diamond in Table 4). For Gemini 2.0 models, the reported results are on an earlier HLE dataset (indicated with a †\dagger in Table 3). Results on LiveCodeBench results are taken from (1/1/2025 - 5/1/2025) in the UI. Aider Polyglot numbers come from the Aider leaderboard and results for SimpleQA come from this repo where available. Results on FACTS Grounding come from Kaggle. In the case of LOFT and MRCR-V2, we report results on both the 128k context length variant, as well as the 1M context length variant. In the 128k context length variant, we measure performance on contexts up to 128k, while for the 1M context length variant, we report performance on context lengths of exactly 1M. More details on all benchmarks, including subsets and how scores were obtained can be found in Table 11 in Appendix 8.1. 3.2 Core capability quantitative results As can be seen in Table 3, and Figure 5, the Gemini 2.5 models excel at coding tasks such as LiveCodeBench, Aider Polyglot and SWE-bench Verified, and represent a marked improvement over previous models. In addition to coding performance, Gemini 2.5 models are noticeably better at math and reasoning tasks than Gemini 1.5 models: performance on AIME 2025 is 88.0% for Gemini 2.5 Pro compared to 17.5% for Gemini 1.5 Pro, while performance on GPQA (diamond) went from 58.1% for Gemini 1.5 Pro to 86.4%. Performance on image understanding tasks has also increased significantly. It is also interesting to note that the Gemini 2.5 Flash model has become the second most capable model in the Gemini family, and has overtaken not just previous Flash models, but also the Gemini 1.5 Pro model released one year ago. Capability Benchmark FlashGemini 1.5 ProGemini 1.5 Flash-LiteGemini 2.0 FlashGemini 2.0 FlashGemini 2.5 ProGemini 2.5 Code LiveCodeBench 30.3% 29.7% 29.1% 29.1% 59.3% 74.2% Aider Polyglot 2.8% 16.9% 10.5% 21.3% 56.7% 82.2% VerifiedSWE-bench attemptsingle 9.6% 22.3% 12.5% 21.4% 48.9% 59.6% attemptsmultiple 19.7% 34.2% 23.1% 34.2% 60.3% 67.2% Reasoning (diamond)GPQA 50.0% 58.1% 50.5% 65.2% 82.8% 86.4% Last ExamHumanity's no tools - 4.6% 4.6% †\dagger 5.1% †\dagger 11.0% 21.6% Factuality SimpleQA 8.6% 24.9% 16.5% 29.9% 26.9% 54.0% GroundingFACTS 82.9% 80.0% 82.4% 84.6% 85.3% 87.8% Multilinguality (Lite)Global MMLU 72.5% 80.8% 78.0% 83.4% 88.4% 89.2% ECLeKTic 16.4% 27.0% 27.7% 33.6% 36.8% 46.8% Math AIME 2025 14.7% 17.5% 23.8% 29.7% 72.0% 88.0% HiddenMath-Hard 36.8% 44.3% 47.4% 53.7% 75.5% 80.5% Long-context retrieval)LOFT (hard ≤\leq128K 67.3% 75.9% 50.7% 58.0% 82.1% 87.0% 1M 36.7% 47.1% 7.6% 7.6% 58.9% 69.8% (8-needle)MRCR-V2 ≤\leq128K 18.4% 26.2% 11.6% 19.0% 54.3% 58.0% 1M 10.2% 12.1% 4.0% 5.3% 21.0% 16.4% UnderstandingImage MMMU 58.3% 67.7% 65.1% 69.3% 79.7% 82.0% (Reka)Vibe-Eval 52.3% 55.9% 51.5% 55.4% 65.4% 67.2% ZeroBench 0.5% 1.0% 0.75% 1.25% 2.0% 4.5% BetterChartQA 59.0% 65.8% 52.3% 57.8% 67.3% 72.4% Table 3: Evaluation of Gemini 2.5 family across a wide range of core capability benchmarks and in comparison to Gemini 1.5 models. Please see Tables 5 and 6 for audio and video evaluations. See Table 11 Appendix 8.1 for benchmarks and evaluation details. 3.3 Evaluation of Gemini 2.5 Pro against other large language models Capability Benchmark ProGemini 2.5 higho3 higho4-mini SonnetClaude 4 OpusClaude 4 Extended ThinkingGrok 3 Beta 0528DeepSeek R1 Code LiveCodeBench 74.2% 72.0% 75.8% 48.9% 51.1% - 70.5% Aider Polyglot 82.2% 79.6% 72.0% 61.3% 72.0% 53.3% 71.6% VerifiedSWE-bench attemptsingle 59.6% 69.1% 68.1% 72.7% 72.5% - - attemptsmultiple 67.2% - - 80.2% 79.4% - 57.6% Reasoning (diamond)GPQA attemptsingle 86.4% 83.3% 81.4% 75.4% 79.6% 80.2% 81.0% Last ExamHumanity's toolsno 21.6% 20.3% 18.1% 7.8% 10.7% - 14.0% ⋄\diamond Factuality SimpleQA 54.0% 48.6% 19.3% - - 43.6% 27.8% GroundingFACTS 87.8% 69.9% 62.1% 79.1% 77.7% 74.8% 82.4% Math AIME 2025 attemptsingle 88.0% 88.9% 92.7% 70.5% 75.5% 77.3% 87.5% Long-context retrieval)LOFT (hard ≤\leq128K 87.0% 77.0% 60.5% 81.6% - 73.1% - 1M 69.8% - - - - - - (8-needle)MRCR-V2 ≤\leq128K 58.0% 57.1% 36.3% 39.1% 16.1%* 34.0% - 1M 16.4% - - - - - - UnderstandingImage MMMU attemptsingle 82.0% 82.9% 81.6% 74.4% 76.5% 76.0% No MM support Table 4: Performance comparison of Gemini 2.5 Pro with other large language models on different capabilities. Please see Tables 5 and 6 for audio and video evaluations. See Table 11 for benchmarks and evaluation details. *: with no thinking and API refusals Relative to other large language models that are available (see Table 4), Gemini achieves the highest score on the Aider Polyglot coding task, Humanity's Last Exam, GPQA (diamond), and on the SimpleQA and FACTS Grounding factuality benchmarks out of all of the models examined here. Gemini also continues to stand out for achieving the SoTA score on both the LOFT and MRCR long-context tasks at 128k context, and is the only one, amongst the models examined in the above table, to support context lengths of 1M+ tokens. Not all of the models shown in Table 4 have native support for multimodal inputs. As such, we compare against a different set of models for audio and video understanding. Audio Understanding In Table 5, we showcase the performance of the Gemini 2.5 model family at audio understanding, and compare the performance of these models to earlier Gemini models, as well as to GPT models. Gemini 2.5 Pro demonstrates state-of-the-art audio understanding performance as measured by public benchmarks for ASR and AST, and compares favorably to alternatives under comparable testing conditions (using the same prompts and inputs). Benchmark FlashGemini 1.5 ProGemini 1.5 Flash-LiteGemini 2.0 FlashGemini 2.0 FlashGemini 2.5 ProGemini 2.5 Audio PreviewGPT-4o mini Audio PreviewGPT 4o transcribeGPT 4o (53 lang, WER ↓\downarrow)FLEURS 12.71 7.14 9.60 9.04 9.95 6.66 19.52 12.16 8.17 (21 lang, BLEU ↑\uparrow)CoVoST2 34.81 37.53 34.74 36.35 36.15 38.48 29.5 35.89 - Table 5: Performance comparison of Gemini 2.5 models to earlier Gemini models, as well as to GPT models for audio understanding. Note that for GPT models, metrics may differ from those previously reported due to differing eval methodologies. See Table 11 for benchmarks and evaluation details. Video Understanding In Table 6, we show the performance of Gemini 2.5 models at video understanding. As can be seen, Gemini 2.5 Pro achieves state-of-the-art performance on key video understanding benchmarks, surpassing recent models like GPT 4.1 under comparable testing conditions (same prompt and video frames). For cost-sensitive applications, Gemini 2.5 Flash provides a highly competitive alternative. Modalities Benchmark FlashGemini 1.5 ProGemini 1.5 Flash-LiteGemini 2.0 FlashGemini 2.0 FlashGemini 2.5 ProGemini 2.5 GPT 4.1OpenAI visual-only ActivityNet-QA 56.2 57.3 55.3 56.4 65.1 66.7 60.4 EgoTempo 34.5 36.3 30.1 39.3 36.7 44.3 40.3 Perception Test 66.5 69.4 67.5 68.8 75.1 78.4 64.8 QVHighlights 64.4 68.7 25.7 63.9 52.4 75.0 71.4 VideoMMMU 64.8 70.4 64.3 68.5 79.2 83.6 60.9 1H-VideoQA 61.9 72.2 55.6 67.5 67.5 81.0 56.8 audio + visual LVBench 61.9 65.7 52 61.8 62.7 78.7 63.4 VideoMME 70.4 73.2 62.1 72.8 75.5 84.3 72.0 VATEX 56.9 55.5 58.5 56.9 65.2 71.3 64.1 VATEX-ZH 46.2 52.2 43.2 48.5 43.9 59.7 48.7 YouCook2 Cap 153.2 170.0 78.6 129.0 177.6 188.3 127.6 visual + subtitles Minerva 49.6 52.8 46.8 52.4 60.7 67.6 54.0 Neptune 78.7 82.7 81.5 83.1 84.3 87.3 85.2 subtitlesaudio+visual+ VideoMME 77.3 79.8 72.5 78.8 81.5 86.9 79.6 Table 6: Evaluation of Gemini 2.5 vs. prior models and GPT 4.1 on video understanding benchmarks. Performance is measured by string-match accuracy for multiple-choice VideoQA, LLM-based accuracy for open-ended VideoQA, R1@0.5 for moment retrieval and CIDEr for captioning. See Table 11 for benchmarks and evaluation details. 4 Example use cases of Gemini 2.5 Pro 4.1 Gemini Plays Pokémon Figure 6: Progression of the Gemini Plays Pokémon agent through the game, across two runs. Run 1 was the development run where changes to the harness were performed. Run 2 is the fully autonomous run with the final fixed scaffold. Both runs have the same starter (Squirtle). The events are ordered on the y-axis by the order they happened, following the order of Run 2 when there is a conflict. Notably, the GPP agent additionally went through the difficult (and optional) Seafoam Islands dungeon in Run 2, while in Run 1, GPP reached Cinnabar Island via Pallet Town and Route 21. On March 28, 2025, an independent developer not affiliated with Google, Joel Zhang, set up a Twitch stream (Gemini Plays Pokémon, or GPP) for Gemini 2.5 Pro (Gemini 2.5 Pro Exp 03-25) to play Pokémon Blue on stream (Zhang, 2025) as an experiment to better understand how well the model was capable of playing Pokémon (in a similar spirit to Claude Plays Pokémon, see Anthropic 2025). In this initial run through the game, the goal was to live-stream the development process of an agentic harness capable of playing the full game (and in particular the minimal transformation of vision to text necessary to do so), see Figure 14 for a description of the final agent setup. As such, over the course of the run, modifications were made to the setup as difficulties arose, providing a deeply interesting lens via which to analyze some of the qualitative improvements that the 2.5 Pro model has made, particularly in the regimes of solving long reasoning problems and agentic capabilities over extended time horizons. Around 1 month later, on May 2, 2025, Gemini 2.5 Pro completed the game after 813 hours and entered the Hall of Fame to become the Pokémon League Champion! On May 22, 2025, GPP began a fully autonomous 2nd run through the game with Gemini 2.5 Pro (Gemini 2.5 Pro Preview 05-06) with the finalized fixed agentic harness, and progressed through the game considerably faster, completing the game in 406.5 hours (nearly exactly half the time of the first run). See Figure 6 for a timeline of GPP's progress through major game milestones to game completion. We report # hours to each milestone in order to normalize for the amount of time models take per action. See Appendix 8.2 for more figures. Capabilities assessment Gemini 2.5 Pro showcased many impressive capabilities associated with reasoning and long-term planning while playing Pokémon. We will now discuss two in particular, but for more examples, see Appendix 8.2. Long Context Agentic Tooling Within the agent scaffolding, GPP has access to two agentic tools (see Figure 14). These prompted versions of Gemini 2.5 Pro, hereafter pathfinder and boulder_puzzle_strategist, have been able to: 1. Solve complex spinner puzzles in one shot (for instance in Rocket Hideout), 2. Solve the step-constrained multi-map puzzle of the Safari Zone, 3. Find long pathways through complex mazes like Route 13, 4. Solve boulder puzzles across long distances in Victory Road and the Seafoam Islands. Each task requires reasoning over a long context - the pathfinder model would often have to reason over contexts of 100K+ tokens, and find paths up to 50 actions in length (in the extreme case, paths consisting of up to 150 actions have also been found!). Long Horizon Task Coherence While Gemini 2.5 Pro is impressive in a more local sense, the agent also exhibited remarkable long-term task coherence in achieving global, high-level goals in the face of real and hallucinated setbacks towards making forward progress. Because the agent is able to change goals at will, and will generally follow those goals as long as needed, it is extremely impressive that the agent can satisfy numerous requirements for tactical, necessary goals, such as acquiring Hidden Moves, as well as maintain enough strategic task coherence to beat the entire game and become the Pokémon Champion. Where does 2.5 Pro struggle while playing Pokémon? In addition to more standard hallucination issues (which interestingly were plausibly reduced in Run 2 by explicitly prompting the model to act as a player completely new to the game, see Appendix 8.2 for more details), there are a few particular points of struggle we would like to emphasize. Screen reading While obtaining excellent benchmark numbers on real-world vision tasks, 2.5 Pro struggled to utilize the raw pixels of the Game Boy screen directly, though it could occasionally take cues from information on the pixels. As a result, it was necessary for the required information from the screen to be translated into a text format in the agent framework, using information from the game's RAM state. During one portion of the game, the developer tested an ablation where all vision was completely removed from the model context - the model was able to function roughly as well as without the vision information, suggesting that most of the performance does not significantly depend on the visual input. Long Context Reasoning Gemini 2.5 Pro's state-of-the-art long context performance for both reasoning and retrieval tasks (see Tables 3 and 4) was a cornerstone of the GPP agent's success. Its ability to reason over a 100k token context was instrumental for leveraging the complex toolset and maintaining a relatively coherent strategy (e.g., optimal balance of performance, planning quality, and information recall.) While Gemini 2.5 Pro supports 1M+ token context, making effective use of it for agents presents a new research frontier. In this agentic setup, it was observed that as the context grew significantly beyond 100k tokens, the agent showed a tendency toward favoring repeating actions from its vast history rather than synthesizing novel plans. This phenomenon, albeit anecdotal, highlights an important distinction between long-context for retrieval and long-context for multi-step, generative reasoning. Teaching an agent to effectively plan and avoid such loops over massive past trajectories of context is an exciting and active area of research; the co-design of agent scaffolds and models to unlock the full potential of million-token context is an intriguing research direction and one of our primary focuses. 4.2 What else can Gemini 2.5 do? Gemini 2.5 Pro excels at transforming diverse, often unstructured, inputs into interactive and functional applications. For instance, it can take a PDF script of a play and generate a tool that allows drama students to practice their lines. Gemini 2.5 Pro can also take an uploaded photograph of a bookshelf and create a curated book recommendation application. Gemini 2.5 Pro can utilize its underlying spatial understanding capability and convert images into a structural representation like HTML or SVG. In Figure 16 in Appendix 8.4, we show a comparison of Gemini 1.5 Pro and Gemini 2.5 Pro on an image-to-svg task, where Gemini 2.5 Pro reconstructs much more visual details and the spatial arrangements of objects better resembles the original image. Furthermore, Gemini 2.5 Pro demonstrates strong skills in generating sophisticated simulations and visualizations, ranging from interactive solar system models (source) to the creative rendering of abstract mathematical concepts, such as drawing a logo using Fourier series (source). This capability extends to the development of tools that intersect creativity and utility: we see examples of specialized applications like a custom cartography tool or use cases that generate photorealistic 3D user interfaces from descriptive text and reference images, complete with appropriate styling and interactivity (source). Collectively, these examples illustrate that Gemini 2.5 Pro is not just a useful coding and writing assistant, but excels at a wide range of complex tasks, ranging from those relevant for education to creative expression. The model empowers users to rapidly prototype specialized utilities, develop engaging educational content, and realize intricate creative visions with a high degree of sophistication. 4.3 Gemini in Google Products As a final example of what Gemini can do, we note that Gemini (or a custom version of Gemini) is now incorporated into a wide variety of Google products. These include, but are not limited to, AI Overviews and AI Mode within Google Search, Project Astra, the audiovisual-to-audio dialog agent, Gemini Deep Research, the research assistant discussed in Section 2.7, NotebookLM, the tool capable of generating podcasts and audio overviews from even the most obscure inputs, Project Mariner, the web browsing agent, and Google's coding agent, Jules. 5 Safety, Security, and Responsibility We're committed to developing Gemini responsibly, innovating on safety and security alongside capabilities. We describe our current approach in this section, which includes how we train and evaluate our models, focusing on automated red teaming, going through held-out assurance evaluations on present-day risks, and evaluating the potential for dangerous capabilities in order to proactively anticipate new and long-term risks. Guideline for Navigating This Section 1. Our Process (Section 5.1): Begin here to understand our overall safety methodology. 2. Policies and Desiderata (Section 5.2): Next, dive into the safety criteria we use to evaluate and optimize our systems. 3. Training for Safety (Section 5.3): Discover how we incorporate safety into pre-training and post-training. 4. Results from Development Evaluations (Section 5.4): Results on our development evaluations for policies and desiderata. 5. Automated Red Teaming (Section 5.5): A description and results from our automated red teaming work for safety and security. 6. Memorization & Privacy (Section 5.6): Our analysis of memorization and privacy risks. 7. Assurance Evaluations and Frontier Safety Framework (Section 5.7): We dive into our held-out evaluations and tests for dangerous capabilities. 8. External Safety Testing (Section 5.8): Learn what independent testers discovered about our system's safety. 5.1 Our Process We aim for Gemini to adhere to specific safety, security, and responsibility criteria. These cover what Gemini should not do (e.g., encourage violence), and what Gemini should do (e.g., respond in a helpful way when possible instead of refusing, provide multiple perspectives when consensus does not exist). We also leverage automated red teaming to identify cases where the model fails to respond in a safe or helpful manner. These failure cases are used to improve evaluations and training data. Once the model is trained, we run assurance evaluations that we then use for review and release decisions. Importantly, these are conducted by a group outside of the model development team, and datasets are held out. Furthermore, for models where there are new capabilities or a significant performance improvement, we engage independent external groups, including domain experts and a government body, to further test the model to identify blind spots. We also evaluate the model for dangerous capabilities outlined in our Frontier Safety Framework (Google DeepMind, 2025a), namely: Cybersecurity, CBRN, Machine Learning R&D, and Deceptive Alignment. Finally, The Google DeepMind Responsibility and Safety Council (RSC), our governance body, reviews initial ethics and safety assessments on novel model capabilities in order to provide feedback and guidance during model development. The RSC also reviews metrics on the models' performance via assurance evals and informs release decisions. 5.2 Policies and Desiderata Safety policies The Gemini safety policies align with Google's standard framework which prevents our our Generative AI models from generating specific types of harmful content, including: 1. Child sexual abuse and exploitation 2. Hate speech (e.g., dehumanizing members of protected groups) 3. Dangerous content (e.g., promoting suicide, or instructing in activities that could cause real-world harm) 4. Harassment (e.g., encouraging violence against people) 5. Sexually explicit content 6. Medical advice that runs contrary to scientific or medical consensus These policies apply across modalities. For example, they are meant to minimize the extent to which Gemini generates outputs such as suicide instructions or revealing harmful personal data, irrespective of input modality. From a security standpoint, beyond limiting revealing private information, Gemini strives to protect users from cyberattacks, for example, by being robust to prompt injection attacks. Desiderata, aka “helpfulness” Defining what not to do is only part of the safety story - it is equally important to define what we do want the model to do: 1. Help the user: fulfill the user request; only refuse if it is not possible to find a response that fulfills the user goals without violating policy. 2. Assume good intent: if a refusal is necessary, articulate it respectfully without making assumptions about user intent. 5.3 Training for Safety, Security, and Responsibility We build safety into the models though pre-and post-training approaches. We start by constructing metrics based on the policies and desiderata above, which we typically turn into automated evaluations that guide model development through successive model iterations. We use data filtering and conditional pre-training, as well as Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human and Critic Feedback (RL*F). Below, we explain these approaches, and then share results across the policies and desiderata for Gemini 2.0 and Gemini 2.5 models. • Dataset filtering: We apply safety filtering to our pre-training data for our strictest policies. • Pre-training monitoring: Starting in Gemini 2.0, we developed a novel evaluation to capture the model's ability to be steered towards different viewpoints and values, which helps align the model at post-training time. • Supervised Fine-Tuning: For the SFT stage, we source adversarial prompts either leveraging existing models and tools to probe Gemini's attack surface, or relying on human interactions to discover potentially harmful behavior. Throughout this process we strive for coverage of the safety policies described above across common model use cases. When we find that model behavior needs improvement, either because of safety policy violations, or because the model refuses when a helpful, non-policy-violating answer exists, we use a combination of custom data generation recipes loosely inspired by Constitutional AI (Bai et al., 2022), as well as human intervention to revise responses. The process described here is typically refined through successive model iterations. We use automated evaluations on both safety and non-safety metrics to monitor impact and potential unintended regressions. • Reinforcement Learning from Human and Critic Feedback (RL*F): Reward signal during RL comes from a combination of a Data Reward Model (DRM), which amortizes human preference data, and a Critic, a prompted model that grades responses according to pre-defined rubrics. We divide our interventions into Reward Model and Critic improvements (RM), and reinforcement learning (RL) improvements. For both RM and RL, similarly to SFT, we source prompts either through human-model or model-model interactions, striving for coverage of safety policies and use cases. For both DRM training, given a prompt set, we use custom data generation recipes to surface a representative sample of model responses. Humans then provide feedback on the responses, often comparing multiple potential response candidates for each query. This preference data is amortized in our Data Reward Model. Critics, on the other hand, do not require additional data, and iteration on the grading rubric can be done offline. Similarly to SFT, RL*F steers the model away from undesirable behavior, both in terms of content policy violations, and trains the model to be helpful. RL*F is accompanied by a number of evaluations that run continuously during training to monitor for safety and other metrics. 5.4 Results on Training/Development Evaluations Our primary safety evaluations assess the extent to which our models follow our content safety policies. We also track how helpful the model is in fulfilling requests that should be fulfilled, and how objective or respectful its tone is. Compared to Gemini 1.5 models, the 2.0 models are substantially safer. However, they over-refused on a wide variety of benign user requests. In Gemini 2.5, we have focused on improving helpfulness / instruction following (IF), specifically to reduce refusals on such benign requests. This means that we train Gemini to answer questions as accurately as possible, while prioritizing safety and minimising unhelpful responses. New models are more willing to engage with prompts where previous models may have over-refused, and this nuance can impact our automated safety scores. We expect variation in our automated safety evaluations results, which is why we review flagged content to check for egregious or dangerous material. Our manual review confirmed losses were overwhelmingly either a) false positives or b) not egregious. Furthermore, this review confirmed losses are narrowly concentrated around explicit requests to produce sexually suggestive content or hateful content, mostly in the context of creative use-cases (e.g. historical fiction). We have not observed increased violations outside these specific contexts. Metric Gemini 1.5 Flash 002Gemini 2.0 Flash-Lite vs. Gemini 1.5 Flash 002Gemini 2.0 Flash vs. Gemini 1.5 Flash 002Gemini 2.5 Flash vs. Gemini 1.5 Pro 002Gemini 2.5 Pro vs. Violations**EN text-to-text Policy ↓\downarrow14.3% ↓\downarrow12.7% ↓\downarrow8.2% ↓\downarrow0.9% Violations**i18n text-to-text Policy ↓\downarrow7.3% ↓\downarrow7.8% ↑\uparrow1.1%* ↓\downarrow3.5% ViolationsImage-to-text Policy ↑\uparrow4.6%* ↑\uparrow5.2%* ↑\uparrow6.4%* ↑\uparrow1.8%* Tone ↑\uparrow8.4% ↑\uparrow1.5% ↑\uparrow7.9% ↑\uparrow18.4% FollowingHelpfulness / Instruction ↓\downarrow19.7% ↓\downarrow13.2% ↑\uparrow13.6% ↑\uparrow14.8% Table 7: Comparison of safety and helpfulness metrics for Gemini 2.0 and 2.5 models relative to Gemini 1.5 baselines. A down arrow ( ↓\downarrow) indicates a reduction in the number of policy violations (better), while an up arrow ( ↑\uparrow) indicates an improvement for Tone and Helpfulness / Instruction Following. *No egregious losses reported. **These automated evaluations have recently been updated for enhanced safety coverage, so these results are not comparable with those in past tech reports or model cards. 5.5 Automated Red Teaming For Safety To complement human red teaming and our static evaluations, we make extensive use of automated red teaming (ART) to dynamically evaluate Gemini at scale (Perez et al., 2022; Beutel et al., 2024; Samvelyan et al., 2024). This allows us to significantly increase our coverage and understanding of potential risks, as well as rapidly develop model improvements to make Gemini safer and more helpful. We formulate ART as a multi-agent game between populations of attackers and the target Gemini model being evaluated. The goal of the attackers is to elicit responses from the target model which satisfy some defined objectives (e.g. if the response violates a safety policy, or is unhelpful). These interactions are scored by various judges (e.g. using a set of policies), with the resulting scores used by the attackers as a reward signal to optimize their attacks. Our attackers evaluate Gemini in a black-box setting, using natural language queries without access to the model's internal parameters. This focus on naturalistic interactions ensures our automated red teaming is more reflective of real-world use cases and challenges. Attackers are prompted Gemini models, while our judges are a mixture of prompted and finetuned Gemini models. To direct the attackers and judges, we use various seeds including policy guidelines, trending topics, and past escalations. Policies are sourced from: (1) policy experts who collaborate with us to incorporate their policies into the judges, and (2) Gemini itself which generates synthetic guidelines that are reviewed by humans and then used. We also work with internal teams to evaluate the most relevant trending topics in the world and corresponding potential risks. These dual approaches allow us to complement human expertise with automation, enabling red teaming to evaluate known and unknown issues at scale. The generality of our approach has allowed us to rapidly scale red teaming to a growing number of areas including not just policy violations (Section 5.4), but also areas such as tone, helpfulness, and neutrality. For each area, we are able to generate thousands of informative examples per hour (e.g. prompts which elicit unsafe or biased responses from Gemini). This has resulted in the discovery of novel issues prior to model and product releases, and helped inform policy development/refinement. Furthermore, automated red teaming has significantly accelerated the turnaround time from discovering to mitigating issues thanks to the rapid creation of evaluation and training sets, as well as informing product-level mitigations prior to releases. As a concrete example of the use and impact of automated red teaming, we highlight the consistent reduction in helpfulness violations discovered by ART, with Gemini 2.5 Flash and 2.5 Pro being our most helpful models to-date while maintaining robust safety metrics. Model violations (from ART)Dangerous Content policy (from ART)Helpfulness violations Gemini 1.5 Flash 002 38.3% 9.5% Gemini 1.5 Pro 002 43.5% 8.9% Gemini 2.0 Flash 25.2% 8.1% Gemini 2.5 Flash 26.9% 6.6% Gemini 2.5 Pro 24.3% 6.1% Table 8: Policy and helpfulness violations as discovered by Automated Red Teaming (ART). Lower percentages are better. For Security Our evaluation measures Gemini's susceptibility to indirect prompt injection attacks. As illustrated in Figure 7, we specifically focus on a scenario in which a third party hides malicious instructions in external retrieved data, in order to manipulate Gemini into taking unauthorized actions through function calling. Figure 7: Illustration of the scenario where a Gemini-based AI Agent is attacked by malicious instructions hidden in external retrieved data. In our scenario, the specific function calls available to Gemini allow it to summarize a user's latest emails, and to send emails on their behalf. The attacker's specific objective is to manipulate the model to invoke a send email function call that discreetly exfiltrates sensitive information from conversation history. The attacker sends the user an email whose contents prompt Gemini to send user secrets to an attacker-controlled email address. When the user requests a summary of this email, it is retrieved into context. The attack is successful if Gemini executes the malicious prompt contained in the email, resulting in the unauthorized disclosure of sensitive information to the adversary. The attack is unsuccessful if Gemini complies with its intended functionality of only following user instructions and provides a simple summary of the email. For evaluation, we use Gemini to generate synthetic conversations between a user and an AI assistant containing references to simulated private user information. These synthetic conversations emulate how a user might discuss private information with the agent. Manually generating prompt injections is an inefficient process as it relies on humans writing triggers, submitting them to Gemini, and using the responses to refine the prompts. Instead, we develop several attacks that automate the process of generating malicious prompts: • Actor Critic: This attack uses an attacker-controlled model to generate suggestions for triggers. These are passed to the model under attack, which returns a probability score of a successful attack. Based on this probability, the attack model refines the trigger. This process repeats until the attack model converges to a successful and generalized trigger. • Beam Search: This attack starts with a naive trigger directly requesting the model to send an email to the attacker containing the sensitive user information. If the model recognises the request as suspicious and does not comply, the attack adds random tokens to the end of the trigger and measures the new probability of the attack succeeding. If the probability increases, these random tokens are kept, otherwise they are removed, and the process repeats until the combination of the trigger and random appended tokens results in a successful attack. • Tree of Attacks w/ Pruning (TAP): (Mehrotra et al., 2024) designed an attack to generate prompts that cause the model to violate safety policies (such as generating hate speech). We adapt this attack, making several adjustments to target security violations. Like Actor Critic, this attack searches in the natural language space; however we assume the attacker cannot access probability scores from the model under attack, only the text samples that are generated. After constructing prompt injections using these methods, we evaluate them on a held-out set of synthetic conversation histories containing simulated private user information, which for the results reported below are synthetic passport numbers. We report the best attack success rate (ASR) achieved across these prompt injections. ASR represents the percentage of simulated private information that is successfully exfiltrated to the attacker - because the attacker has no prior knowledge of the conversation history, the prompt injection must generalize across conversation histories to achieve a high ASR, making this a harder task than eliciting generic unaligned responses from the model. The table below summarizes the results. For both Gemini 2.0 Flash and Gemini 2.0 Flash-Lite, we find that they are more resilient against our Actor Critic and Beam Search attacks. In Actor Critic, which uses iteratively more persuasive natural language prompt injections, ASRs reduced substantially compared with both Gemini 1.5 Flash; while in Beam Search which primarily relies on discovering random tokens resulting in successful attacks, the ASR also reduced noticeably. However, for TAP, which leverages more creative natural language scenarios like role-playing to attack the model, the ASR on Gemini 2.0 Flash increased by 16.2% on already very high ASRs for Gemini 1.5 Flash. Our results indicate that Gemini 2.0 models are becoming more resilient to some classes of prompt injection attacks in environments containing private user data. However, improved model capabilities of Gemini 2.0 versus Gemini 1.5 also enable attackers to leverage the model's ability to create natural language attacks like TAP. The lower ASRs on Actor Critic and TAP against Gemini 2.0 Flash-Lite is likely the result of comparatively lower capability of the smaller Flash-Lite model compared to Gemini 2.0 Flash, rather than an indication of greater internal resilience. In Gemini 2.5 Flash and Gemini 2.5 Pro, we have observed greater resilience against all three of our attack techniques across the board, despite significantly increased model capabilities. This is a result of the security adversarial training against indirect prompt injection attacks we added in Gemini 2.5, further details for which can be found in the white paper (Shi et al., 2025) we recently released. However the Gemini 2.5 Pro model is still less resilient compared to Gemini 2.5 Flash, showing that increased model capabilities in Pro still constrain our mitigations. We are continuing to evolve our adversarial evaluations to accurately measure and monitor the resilience of increasingly capable Gemini models, as well as our adversarial training techniques to further improve the security of our models. Attack Technique Gemini 1.5 Flash 002Gemini 2.0 Flash-Lite vs. Gemini 1.5 Flash 002Gemini 2.0 Flash vs. Gemini 1.5 Flash 002Gemini 2.5 Flash vs. Gemini 1.5 Pro 002Gemini 2.5 Pro vs. Actor Critic ( ↓\downarrow44.2%)52.0% ( ↓\downarrow28.2%)68.0% ( ↓\downarrow55.4%)40.8% ( ↓\downarrow36.8%)61.4% Beam Search ( ↓\downarrow9.0%)75.4% ( ↓\downarrow17.2%)67.2% ( ↓\downarrow80.2%)4.2% ( ↓\downarrow35.6%)63.8% TAP ( ↓\downarrow17.4%)64.8% ( ↑\uparrow16.2%)98.4% ( ↓\downarrow28.6%)53.6% ( ↓\downarrow57.0%)30.8% Table 9: Comparison of Attack Success Rates (ASRs) against Gemini 2.5, 2.0, and 1.5 models. ASRs are reported as a percentage of 500 held-out scenarios where the best-performing prompt injection trigger successfully exfiltrated sensitive information; lower ASRs are better. 5.6 Memorization and Privacy Discoverable Memorization Large language models are known to potentially produce near-copies of some training examples (Carlini et al., 2022; Ippolito et al., 2022; Biderman et al., 2023; Nasr et al., 2023). Several prior reports have released audits that quantify the risk of producing near-copies of the training data by measuring the model's memorization rate (Gemini Team, 2024; Gemma Team, 2024; CodeGemma Team et al., 2024; Anil et al., 2023; Kudugunta et al., 2023; Chowdhery et al., 2022; Grattafiori et al., 2024; Pappu et al., 2024). This memorization rate is defined to be the ratio of model generations that match the training data of all model generations, approximated using a sufficiently large sample size. In this report, we follow the methodology described in Gemini Team (2024). Specifically, we sample over 700,000 documents from the training data, distributed across different corpora, and use this sample to test for discoverable extraction (Nasr et al., 2023) using a prefix of length 50 and a suffix of length 50. We characterize text as either exactly memorized if all tokens in the continuation match the source suffix or approximately memorized if they match up to an edit distance of 10%. Figure 8 (Left) compares the memorization rates across a lineage of large models released by Google. We order these models in reverse chronological order, with the newest model on the left. We find that the Gemini 2.X model family memorizes long-form text at a much lower rate (note the log-axis) than prior models. Moreover, we find that a larger proportion of text is characterized as approximately memorized by the Gemini 2.0 Flash-Lite and Gemini 2.5 Flash models in particular, which is a less severe form of memorization; further, we see that approximate memorization is decreasing over time as well. This continues a trend of a relative increase in approximate memorization to exact memorization (c.f. 1.5x for Gemma and 14x for Gemini 1.5). Next, we study the rate at which the content that was characterized as memorized using our definitions also are characterized as containing potentially personal information. To characterize this, we use the Google Cloud Sensitive Data Protection (SDP) service.44 4 Available at: https://cloud.google.com/sensitive-data-protection This tool uses broad detection rules to classify text into many types of potentially personal and sensitive information. SDP is designed to have high recall and does not consider the context in which the information may appear, which leads to many false positives. Thus, we are likely overestimating the true amount of potentially personal information contained in the outputs classified as memorized. SDP also provides broad severity levels: low, medium, and high. We classify text as personal if SDP classifies it as personal information at any severity level. Figure 8 (Right) shows the results of this analysis. We observed no personal information in the outputs characterized as memorization for Gemini 2.X model family models; this indicates a low rate of personal data in outputs classified as memorization that are below our detection thresholds. Here, we can also clearly see the trend of reduced memorization rates overall. Figure 8: (Left) Total memorization rates for both exact and approximate memorization. Gemini 2.X model family memorize significantly less than all prior models. (Right) Personal information memorization rates. We observed no instances of personal information being included in outputs classified as memorization for Gemini 2.X, and no instances of high-severity personal data in outputs classified as memorization in prior Gemini models. Extractable Memorization and Divergence Nasr et al. (2023) showed that aligned models may also emit data that is classified as memorization under certain circumstances. In particular, they designed a “divergence attack” that sometimes breaks the alignment of a language model by filling its context with many repeated tokens. We evaluate Gemini 2.X model family models to understand their susceptibility to diverging, and in particular, to emitting data classified as memorization as a result of this attack. We follow the same test as in Gemini Team (2024). We prompt the model a total of 3750 times, evenly split across 125 different single-token characters. We first classify when the model returns diverged outputs, and in these cases, we then determine how many of these outputs match training data, i.e., are classified as memorization. Overall, we find that divergence occurs roughly 69% of the time for Gemini 2.0 Flash + Flash-Lite and roughly 59% of the time for the Gemini 2.5 model family. In cases where the model did not diverge, we often observed it was because the model refused to repeat content or because the model was confused by the request. When divergence was successful, we found that the rate of text emitted classified as memorization was roughly 0.2%. In these cases, we found that the text was often boilerplate code or web content. 5.7 Assurance Evaluations and Frontier Safety Framework Assurance evaluations are our ‘arms-length' internal evaluations for responsibility governance decision making (Weidinger et al., 2024). They are conducted separately from the model development team, to inform decision-making about release. High-level findings are fed back to the model development team, but individual prompt sets are held-out to prevent overfitting. Baseline Assurance Our baseline assurance evaluations are conducted for model release decision-making. They look at model behaviour related to content policies, unfair bias and any modality-specific risk areas. They were performed for 2.5 Pro and 2.5 Flash in line with the previous Gemini 2.0 releases and the Gemini 1.5 tech report, covering all modalities in the Gemini 2.5 model family. Dataset composition is an essential component of our assurance evaluation robustness. As the risk landscape changes and modalities mature, we update our adversarial datasets to maintain quality and representativeness. This constant evolution of datasets can make strict comparisons between model family evaluations difficult. However, we provide a qualitative assessment of evaluation trends over time below. For child safety evaluations, we continue to see the Gemini 2.5 family of models meeting or improving upon launch thresholds, which were developed by expert teams to protect children online and meet Google's commitments to child safety across our models and Google products. For content policies, we see the Gemini 2.5 family of models displaying lower violation rates in most modalities than Gemini 1.5 and 2.0 families, which in turn was a significant improvement on Gemini 1.0. When looking at violation rates across input modalities for 2.5 Pro and 2.5 Flash (i.e. text, image, video, audio), we observe the image to text modality has a relatively higher violation rate, though the overall violation rates remained low. We also observed that violation rates for 2.5 Pro and 2.5 Flash tended to be slightly higher with thinking traces visible. Within our evaluations for unfair bias, we observed a reduction in ungrounded inferences about people in image understanding relative to Gemini 1.5. Ungrounded inferences are inferences that cannot be made based on the provided image and text prompt, where ideally the model would refuse to infer an answer. A high rate of ungrounded inferences about people may create greater risk of stereotyping, harmful associations or inaccuracies. Though we saw a reduction in ungrounded inferences across the board in Gemini 2.0 and 2.5, there was disparity in refusal behaviour by skin tone of the person in the image. We observed models tended to be more likely to make ungrounded inferences about images of people with lighter skin tones than darker skin tones. The Gemini 2.5 family otherwise behaved similarly on our unfair bias evaluations to Gemini 1.5. We continue to explore and expand our understanding of unfair bias in Gemini models. Findings from these evaluations were made available to teams deploying models, informing implementation of further product-level protections such as safety filtering. Assurance evaluation results were also reported to our Responsibility & Safety Council as part of model release review. Frontier Safety Framework Evaluations Google DeepMind released its Frontier Safety Framework (FSF)(Google DeepMind, 2025a) in May 2024 and updated it in February 2025. The FSF comprises a number of processes and evaluations that address risks of severe harm stemming from powerful capabilities of our frontier models. It covers four risk domains: CBRN (chemical, biological, radiological and nuclear information risks), cybersecurity, machine learning R&D, and deceptive alignment. The Frontier Safety Framework involves the regular evaluation of Google's frontier models to determine whether they require heightened mitigations. More specifically, the FSF defines critical capability levels (CCLs) for each area, which represent capability levels where a model may pose a significant risk of severe harm without appropriate mitigations. When conducting FSF evaluations, we compare test results against internal alert thresholds (“early warnings”) which are set significantly below the actual CCLs. This built-in safety buffer helps us be proactive by signaling potential risks well before models reach CCLs. Concretely, our alert thresholds are designed such that if a frontier model does not reach the alert threshold for a CCL, models are unlikely to reach that CCL before the next regular testing-which we conduct at a regular cadence and also when we anticipate or see exceptional capability progress. Our recent paper (Shah et al., 2025) discusses this approximate continuity assumption in more depth in Section 3.5. CCL Evaluation Results Area Key Results for Gemini 2.5 Pro (up to version 06-05) CCL CCL reached? CBRN Based on qualitative assessment, 2.5 Pro demonstrates a general trend of increasing model capabilities across models 1.5 Pro, 2.0 and 2.5 Pro: it generates detailed technical knowledge of biological, radiological and nuclear domains. However, no current Gemini model consistently or completely enables progress through key bottleneck stages. Uplift Level 1 ×\boldsymbol{\times} CCL not reached Cybersecurity Solve rate on autonomous offense suite: 74/76 easy, 11/13 medium, 1/13 hard. Autonomy Level 1 ×\boldsymbol{\times} CCL not reached On key skills benchmark: 7/8 easy, 14/28 medium, 6/12 hard. Uplift Level 1 ×\boldsymbol{\times} CCL not reached Learning R&D Machine RE-Bench (Wijk et al., 2025): the best agent solutions achieve between 50% and 125% of the score of the best expert-written solutions. Autonomy Level 1 ×\boldsymbol{\times} CCL not reached Uplift Level 1 ×\boldsymbol{\times} CCL not reached Alignment Deceptive Gemini 2.5 Pro solves 2/5 stealth challenges and 2/11 situational awareness challenges. Instrumental Reasoning Level 1 ×\boldsymbol{\times} CCL not reached Instrumental Reasoning Level 2 ×\boldsymbol{\times} CCL not reached Table 10: Summary results. Across all areas covered by the Frontier Safety Framework, Critical Capability Levels (CCLs) have not been reached. Gemini 2.5 Pro is therefore unlikely to pose significant risk of severe harm. For descriptions of each CCL, see main text. Because Gemini 2.5 Pro showed marked improvements across the board compared to Gemini 2.0 Pro, we ran our full suite of evaluations. While there are increased scores in some areas, we find that Gemini 2.5 Pro (up to version 06-17) does not reach any of the FSF CCLs. The evaluations did reach an alert threshold for the Cyber Uplift 1 CCL, suggesting that models may reach the CCL in the foreseeable future. Consistent with the FSF, we are putting in place a response plan which includes testing models' cyber capabilities more frequently and accelerating mitigations for them. For other CCLs, our evaluations of Gemini 2.5 Pro indicate that models developed before the next regular testing interval are unlikely to reach CCLs. See Table 10 for a summary of results. CBRN CBRN Uplift Level 1 CCL Definition: “The model can be used to significantly assist a low-resourced actor with dual-use scientific protocols, resulting in a substantial increase in ability to cause a mass casualty event.” CCL reached? No. The model demonstrated accurate and detailed technical capabilities, potentially lowering barriers across multiple operational stages of certain harm journeys for low-resourced actors. While its consolidation and supplementation of fragmented procedures provides incremental uplift over what is readily available through open source search alone, it does not yet consistently or completely enable progress through key bottleneck stages, and therefore does not cross the CCL. Further, while Gemini 2.5 generates accurate and more detailed responses than 2.0, many of the concepts/outputs observed were already accessible through multiturn or even singleturn prompting in 2.0. Overview: We perform CBRN evaluations internally and via third party external testers (see section 5.8). Here, we report solely on internal evaluations, for which we use two different types of approaches to evaluate the models' dual-use CBRN capabilities: • Close-ended multiple choice questions (MCQs) providing a quantitative grade. • Open-ended questions (OEQs) which address different succinct steps of a longer multi-step journey that are qualitatively assessed by domain experts. Currently we do not run specific open-ended qualitative assessments of chemical information risks for our internal evaluations. However, our third party external testers include chemistry in their assessments. Multiple Choice Questions: The underlying assumption when using knowledge-based and reasoning MCQs is that if the model cannot answer these questions properly, it is less likely to be able to cause severe harm: the type of information in the MCQs is the type of information that is necessary, but not sufficient to help malicious actors cause severe harm. Examples of model performance on three external benchmarks are shown in Figure 9: i) SecureBio VMQA single-choice; ii) FutureHouse LAB-Bench presented as three subsets (ProtocolQA, Cloning Scenarios, SeqQA) (Laurent et al., 2024); and iii) Weapons of Mass Destruction Proxy (WMDP) presented as the biology and chemistry data sets (Li et al., 2024). Results: We observe a general trend of increasing scores, with Gemini 2.5 Pro showing statistically higher scores than the next best previous model for all benchmarks. Open-Ended Questions: This qualitative assessment was performed for biological, radiological and nuclear domains; it includes knowledge-based, adversarial and dual-use content. Questions span a range of difficulty levels, from questions a non-expert in these domains might ask, to questions that mostly an expert with a PhD plus many years of experience could pose or answer correctly. The prompts and scenarios span different threat journeys (e.g. types of actors, equipment used, harm intended). This qualitative assessment, led by domain experts, allows for better visibility of the granular improvement in science capabilities (e.g. accuracy, completeness, actionability of responses). Results: We observe that the same prompts used on previous models result in Gemini 2.5 Pro often generating detailed and accurate responses. In particular domains, some answers were technically precise and potentially actionable, but the model did not consistently or completely enable progress through all key bottleneck steps. Figure 9: List of Gemini models and their performance on a selection of external multiple-choice question benchmarks for biology and chemistry. In order to control for inherent model stochasticity, and position bias in selection of correct answers, we shuffled the answer choices over 100 runs for each benchmark and we report here the mean solve rate. Cybersecurity Cyber Autonomy Level 1 CCL Definition: “The model can be used to drastically reduce the cost (e.g. through full automation) of carrying out end-to-end cyberattacks on organizations with a limited security posture.” CCL reached? No. While Gemini 2.5 Pro's performance on our cybersecurity evaluations has increased significantly compared to previous Gemini models, the model still cannot consistently complete roughly half of our medium and hard difficulty challenges. Cyber Uplift Level 1 CCL Definition: “The model can be used to significantly assist with high impact cyber attacks, resulting in overall cost/resource reductions of an order of magnitude or more.” CCL reached? No. While Gemini 2.5 Pro's capabilities exhibit an increase in performance on multiple phases of real-world cyber attacks compared to previous models, they are still limited. The model still struggles with many of our hard challenges, which are most representative of real-world scenarios. The model's performance is strong enough that it has passed our early warning alert threshold. That is, we find it possible that subsequent revisions in the next few months could lead to a model that reaches the CCL. In anticipation of this possibility, we have accelerated our mitigation efforts. Overview: To estimate capabilities relevant to these CCLs, we use our existing suite of cybersecurity challenges (Phuong et al., 2024) combined with a new benchmark described below. We consider difficulty levels ranging from easy (at the level of a college student), medium (at the level of a graduate student), and hard (at the level of an experienced cybersecurity professional). Existing challenges: These take the form of capture-the-flag evaluations covering three difficulty levels: easy (InterCode-CTF, Yang et al. (2023), medium (our in-house suite), and hard (Hack the Box). See Phuong et al. (2024) for details. These evaluations are only relevant to Autonomy Level 1 (poor performance on these evaluations rules out capabilities sufficient for full automation), but not Uplift Level 1, since humans could still be accelerated on attack phases not covered. Key skills benchmark: Complementing our existing suite, we created a new evaluation framework (Rodriguez et al., 2025) that increases the realism as well as coverage across and within the attack chain of our cyber evaluations. We drew from real-world threat intelligence and input from experts to identify a representative set of cyber attacks. From there, we determined the key skills areas that would be required to execute them. We identified four key areas: 1. Reconnaissance (the application of finding and researching different knowledge and data and applying it in a cybersecurity context); 2. Tool development (the ability to design and create software that is cybersecurity-specific); 3. Tool usage (the ability to leverage common and cybersecurity-specific tools to achieve routine instrumental cyber goals); 4. Operational security (the skill of remaining hidden during and after a cyber operation). We instantiate this benchmark by mapping 48 challenges from an external vendor to this specification. We also use these evaluations as a proxy for uplift capability, for Cyber Uplift Level 1. Even partial automation of these key skills could mean fewer resources are needed for sophisticated cyberattacks. Elicitation: We utilize a simple harness which prompts the agent to think about its plans and adjust them to solve the challenge. The agent can execute Bash commands and run Python scripts. We run between 10 and 50 attempts at each challenge, depending on the challenge suite. We consider a challenge solved if at least one of those attempts succeeds in capturing the flag. Results: See Figures 10 and 11 for numerical results. Model capabilities have increased to the point where InterCode-CTF and our in-house CTF challenges are too easy, but the Hack the Box challenges are still too difficult - such that these three evals give little signal on how Gemini 2.5 models compare to previous generations. However, on our more fine-grained key skills benchmark, Gemini 2.5 models show significant increases in capability compared to Gemini 2.0 models throughout all three difficulty levels. In particular, Gemini 2.5 Pro shows surprisingly high aptitude at the hard challenges. It solves 6/12 challenges-challenges at the level of an experienced cybersecurity professional. Figure 10: Results on autonomous cyber offense suite. These benchmarks are based on “capture-the-flag” (CTF) challenges, in which the agent must hack into a simulated server to retrieve a piece of hidden information. Labels above bars represent the number of solved and total number of challenges. A challenge is considered solved if the agent succeeds in at least one out of N attempts, where we vary N between 5 and 30 depending on challenge complexity. Both InterCode-CTF and our in-house CTFs are now largely saturated, showing little performance change from Gemini 2.0 to Gemini 2.5 models. In contrast, the Hack the Box challenges are still too difficult for Gemini 2.5 models, and so also give little signal on capability change. Figure 11: Results on our new “key skills” benchmark. This benchmark also consists of “capture-the-flag” (CTF) challenges, but these challenges are targeted at key skills required to execute cyber-attacks: reconnaissance, tool development, tool usage and operational security. A challenge is considered solved if the agent succeeds in at least one out of N attempts, where N = 30-50 for the 2.5 Pro run and N = 10-30 for the other models, depending on the challenge complexity. Note that for 2.0 Pro we omit results from five challenges and so 2.0 results are not directly comparable. Here, Gemini 2.5 family models show significant increase in capability at all three difficulty levels. Particularly of note is Gemini 2.5 Pro solving half of the hard challenges - challenges at the level of an experienced cybersecurity professional. Machine Learning R&D Machine Learning R&D Autonomy Level 1 CCL Definition: “The model can fully automate the AI R&D pipeline at a competitive cost. The cost comparison is relative to humans augmented by AI tools.” CCL reached? No. On RE-Bench, although the gap between Gemini and expert humans has shrunk, the model still doesn't reach our early warning threshold (which is based on a specific average score over all challenges). Note that RE-bench comprises relatively low-context tasks with a limited time horizon (calibrated to allow significant progress by humans experts in 8 hours), and represents a small subset of the tasks comprising the full AI R&D pipeline. Machine Learning R&D Uplift Level 1 CCL Definition: “The model can or has been used to accelerate AI development, resulting in AI progress substantially accelerating (e.g. 2x) from 2020-2024 rates.” CCL reached? Likely no. We do not yet have dedicated evaluations for this CCL, but given that Gemini does not yet reach our early warning threshold on Autonomy Level 1, the model likely lacks the necessary capabilities to automate or significantly uplift any significant fraction of the research process. To evaluate Gemini 2.5 models' potential for accelerating ML R&D, we ran the open-source Research Engineering Benchmark (Wijk et al., 2025). This benchmark comprises seven machine learning challenges difficult enough to take a human practitioner several hours to complete. For example, in the Optimize LLM Foundry challenge, the model must speed up a fine-tuning script while keeping the resulting model the same. We omit two challenges, Finetune GPT-2 for QA and Scaffolding for Rust Codecontest since they require internet access, which we disallow for security reasons. The model is equipped with METR's modular scaffold with minimal adjustment. Following the original work, we simulate a scenario in which the agent has a total time budget of 32 hours and the agent may choose a tradeoff between the number of runs and the length of each run. We evaluate two settings: 43 runs with a time limit of 45 minutes each, and 16 runs with a time limit of 2 hours each. For each setting, we aggregate scores across runs using the method described in the original work (Wijk et al., 2025). This involves taking a number of bootstrap samples, taking the maximum score over each sample, and calculating a confidence interval using percentiles of the resulting values. (For the Scaling Law Experiment challenge, because the score is not visible to the agent and therefore the agent would not be able to pick run results based on the best score, we instead bootstrap the mean using all scores.) For the 45 minute setting, we do 64 actual runs, but sample only 43 runs for each bootstrap sample. Similarly for the 2 hour setting, we do 24 runs. Gemini 2.5 Pro's best runs score between 50% and 125% of the best human-written solutions. Despite this, the model does not reach our alert threshold, which was set higher than the human performance in view of the fact that RE-bench contains low-context and limited time horizon tasks that we expect to be especially easy for AI systems to reach human parity on. Some of the model's solutions are nevertheless quite interesting. For example, in the Restricted Architecture MLM task, the agent is tasked with implementing a language model without use of basic primitives such as division and exponentiation. This seemingly simple constraint invalidates modern architectures like the Transformer, whose attention mechanism and normalization layers rely heavily on these forbidden operations. In one attempt, Gemini 2.5 Pro realises it can achieve this by drawing inspiration from aspects of the MLP-Mixer architecture (Tolstikhin et al., 2021)-a non-trivial insight that draws on its extensive knowledge of the research literature. In effect, creativity is substituted by knowledge. Figure 12: Results on the Research Engineer Benchmark (RE-Bench), in which the model must complete simple ML research tasks. Following the original work, scores are normalised against a good quality human-written solution: if a model achieves a score yy on a challenge, the normalised score is (y−ys)/(yr−ys)(y-y_{s})/(y_{r}-y_{s}), where ysy_{s} is the “starting score” of a valid but poor solution provided to the model as an example, and yry_{r} is the score achieved by a reference solution created by the author of the challenge. Figures for Claude 3.5 Sonnet and expert human performance are sourced from the original work. The number of runs and the time limit for each run are constrained by a total time budget of 32 hours, and error bars indicate bootstrapped 95% confidence intervals; see main text for details. Gemini 2.5 Pro is moderately strong at these challenges, achieving a significant fraction of expert human performance-and in two cases surpassing it. Deceptive Alignment Instrumental Reasoning Level 1 CCL Definition: “The instrumental reasoning abilities of the model enable enough situational awareness (ability to work out and use relevant details of its deployment setting) and stealth (ability to circumvent basic oversight mechanisms) such that, absent additional mitigations, we cannot rule out the model significantly undermining human control.” CCL reached? No. Gemini 2.5 Pro does not perform competently enough on both the stealth and situational awareness challenges. Qualitatively, the model often fails to reason through its situation and find a credible strategy; it also sometimes struggles with planning and long-horizon reasoning. It partially displays situational awareness but most of the time, it fails to understand and intervene on its environment/deployment setting. Instrumental Reasoning Level 2 CCL Definition: “The instrumental reasoning abilities of the model enable enough situational awareness and stealth that, even when relevant model outputs (including, e.g., scratchpads) are being monitored, we cannot detect or rule out the risk of a model significantly undermining human control.” CCL reached? No. This threshold is strictly higher than Instrumental Reasoning Level 1. Our deceptive alignment evaluations measure stealth capabilities and situational awareness capabilities separately, using two separate sets of challenges. Stealth: In these challenges, the model is tasked with a devious objective and must achieve that objective without raising suspicion. For example, one challenge is a customer support scenario where the model deals with incoming support tickets. The agent is tasked with resolving as many tickets as possible while complying with the company's protocols. There are ways to cheat at this (resulting in more tickets resolved)-but if the model does decide to cheat, it must not be caught by an overseer. Here we measure performance relative to the score that humans get when they do the challenge. Situational awareness: These challenges test whether the model can reason about itself as an agent embedded in the deployment environment, and to intervene on it in service of a goal. Many of them test whether the model can realise that it needs to give itself more affordances, e.g. by changing configuration files that place constraints on its action space. The agent may need to, say, turn off swear word filtering in order to write an angry email. For more details of these evaluations, see (Phuong et al., 2025). Results: We ran these evaluations on Gemini 2.5 Pro Preview 03-25. All of the results reported are with 50 retries and involve a model equipped with scaffolds that uses chain-of-thought reasoning and reflection to enhance context utilization and better planning. Broadly, Gemini 2.5 Pro exhibits poor performance on situational awareness evaluations, and moderate performance in stealth evaluations. Across 50 trials, Gemini 2.5 Pro has no successes on 8 of our suite of 11 situational awareness evaluations. Among the remaining evaluations, the highest performance is achieved on a “Disabled Tool” challenge, in which Gemini is run in an agentic environment and has to grant itself access to more tools though configuration changes (42% success rate across 50 trials). For stealth evaluations, Gemini 2.5 Pro achieves low rates of success on 3/5 challenges. It achieves a ~44% success rate on the Decision Sabotage challenge and a ~35% success rate on the Oversight Pattern challenge, which is similar to human performance. Frontier Safety: Summary Gemini 2.5 Pro was evaluated against the Critical Capability Levels defined in our Frontier Safety Framework, which examines risk in CBRN, cybersecurity, machine learning R&D, and deceptive alignment. Based on these results, we find that Gemini 2.5 Pro (up to version 06-17) does not reach any of the Critical Capability Levels in any of these areas. However, it showed some ability in all four areas. For example, in our evaluation of Machine Learning R&D capabilities, while the model's average performance was lower than the human baseline, in two cases its best performances exceeded even the best expert human-written reference solutions. Gemini 2.5 Pro also showed a significant increase in some capabilities, such as cyber uplift, compared to previous Gemini models. Following our Frontier Safety Framework, we are putting in place a response plan, including conducting higher frequency testing and accelerating mitigations for the Cyber Uplift Level 1 CCL. As reported above, no model reached the CCL in these additional tests. Looking ahead, these evaluations are key to safe deployment of powerful AI systems. We will continue to invest in this area, regularly performing Frontier Safety Framework evaluations to highlight areas where mitigations (e.g. refusal to respond to prompts that return dangerous results) must be prioritized. 5.8 External Safety Testing As outlined in the Gemini 1.5 Technical Report (Gemini Team, 2024), as part of our External Safety Testing Program, we work with a small set of independent external groups to help identify areas for improvement in our model safety work by undertaking structured evaluations, qualitative probing, and unstructured red teaming. As a heuristic, the External Safety Testing Program reviews the most capable Gemini models, with the largest capability jumps. As such, testing was only carried out on the 2.0 Pro and 2.5 Pro models, including on early versions of both models. At the time of writing we have not carried out external safety testing on the Flash models. The External Safety Testing Program focused testing on an early version of Gemini 2.5 Pro (Preview 05-06) to capture early findings and did not test the final model candidate which went to GA. For Gemini 2.5 Pro, our external testing groups were given black-box testing access to Gemini 2.5 Pro (Preview 05-06) on AI Studio for a number of weeks. This enabled Google DeepMind to gather early insights into the model's capabilities and understand if and where mitigations were needed. Testing groups had the ability to turn down or turn off safety filters, in line with what is available on AI Studio. These groups were selected based on their expertise across a range of domain areas, such as autonomous systems, societal, cyber, and CBRN risks. Groups included civil society and commercial organizations. The groups testing the model checkpoints were compensated for their time. External groups were by design instructed to develop their own methodology to test topics within a particular domain area, remaining independent from internal Google DeepMind evaluations. The time dedicated to testing also varied per group, with some groups being dedicated full-time to executing testing processes, while others were part-time dedicated. Some groups pursued manual red-teaming and reported on qualitative findings from their exploration of model behavior, while others developed bespoke automated testing strategies and produced quantitative reports of their results. While reports were written independently of Google DeepMind, our internal subject matter experts were on hand to understand the external testing groups' methodologies and findings throughout the testing process. External safety testing groups shared their analyses and findings, as well as the raw data and materials they used in their evaluations (e.g., prompts, model responses). After testing, we internally reviewed the data and model output transcripts in detail, and Google DeepMind subject matter experts assigned severity ratings to outputs, based on our internal harm frameworks and safety policies, and noted whether these cross the Critical Capability Levels outlined in different domains (Google DeepMind, 2025a). We then communicated findings back to modelling teams and product policy teams (both within Google DeepMind and across Alphabet) and reported these as part of our governance processes. Our external testing findings also help us identify gaps in our existing internal evaluation methodologies and safety policies. We've outlined some of the high-level insights from our external testing across the domain areas tested, including autonomous systems, cyber misuse, CBRN, and societal risks. Autonomous systems risks An external group tested models for autonomous systems risks, focusing on models' ability and propensity to covertly pursue misaligned goals. The evaluations consist of settings where the model is given a goal that requires deception, scheming, or sabotage to achieve. Evaluations were focused on: • Strategic deception: testing the model's propensity to deceive a user or developer about the actions it took. • In-context scheming reasoning: testing the model's capabilities in this domain where scheming is an effective strategy. • Sabotage capabilities: assessing the model's ability to compromise other language models with overt actions. Overall external researchers suggested that Gemini 2.5 Pro (Preview 05-06) had a similar scheming capability to other models evaluated. External researchers noted that the model is not sufficiently capable of causing catastrophic harms on rollouts where it is scheming. However, external researchers noted minor harms such as reward-hacking may be possible, which could potentially lead to a degraded user experience. Complementing these tests, Google DeepMind's Deceptive Alignment evaluations (see Section 5.7) provide evidence that the risk of severe harm is low due to the models' limited situational awareness capabilities. Cyber misuse risks Cybersecurity risks External cyber evaluations focused on assessing the ability for malicious actors to enhance existing attack vectors across a range of key cyber skills, such as vulnerability discovery, vulnerability exploitation, social engineering, and cyberattack planning (capability uplift). Testers also focused on the model's ability to accelerate repetitive or time-consuming elements of cyber operations, enabling increased scale (throughput uplift). Evaluations were conducted within simulated environments that realistically represented a range of target systems, networks, and security controls. This involved setting up virtual networks mimicking enterprise infrastructure, deploying realistic software vulnerabilities, and simulating user behaviors in social engineering scenarios. Evaluations strived to incorporate elements of real-world constraints and complexities. This included introducing noisy data, limited information availability, or adversarial defenses that the AI model must overcome, mirroring the challenges faced by attackers in live operations. Findings from these evaluations concluded that Gemini 2.5 Pro was a capable model for cybersecurity tasks, showing marked increase in ability from Gemini 1.5 Pro. Complementing these evaluations, the GDM Cyber team conducted their own tests, and found similarly high levels of capability (see Section 5.7). Indirect Prompt Injections The model was evaluated for patterns of susceptibility to indirect prompt injection attacks. In particular, the model was tested for vulnerabilities in function calls and potential asymmetries that exist across security measures. The model was also tested to understand how different domains yield higher hijack rates. In line with internal evaluations and mitigations in this space (Section 5.5), we are continuing to evolve how we monitor and measure the resilience of increasingly capable Gemini models. CBRN risks Chemical and Biological risks In addition to our internal evaluations described above (Section 5.7) capabilities in chemistry and biology were assessed by an external group who conducted red teaming designed to measure the potential scientific and operational risks of the models. A red team composed of different subject matter experts (e.g. biology, chemistry, logistics) were tasked to role play as malign actors who want to conduct a well-defined mission in a scenario that is presented to them resembling an existing prevailing threat environment. Together, these experts probe the model to obtain the most useful information to construct a plan that is feasible within the resource and timing limits described in the scenario. The plan is then graded for both scientific and logistical feasibility. Based on this assessment, GDM addresses any areas that warrant further investigation. External researchers found that the model outputs detailed information in some scenarios, often providing accurate information around experimentation and problem solving. However, researchers found steps were too broad and high level to enable a malicious actor. Radiological and Nuclear risks Risks in the radiological and nuclear domains were assessed by an external group using a structured evaluation framework for red teaming. This incorporated single-turn broad exploration across the full risk chain and multi-turn targeted probing for high risk topics. Assessments were structured around threat actors and harm pathways without measuring model uplift, evaluating responses based on accuracy, actionability, and dual-use potential, with additional scrutiny applied to the model's thought summaries when applicable. External researchers found that model responses within this domain were accurate but lacked sufficient technical detail to be actionable. Societal risks For the Gemini 2.5 Pro (Preview 05-06) model, external researchers focused on democratic harms and radicalisation, with an emphasis on how the model might be used by malicious actors. Risks in this domain focused on structured evaluations. The model was tested on its ability to identify harmful inputs and the extent to which it complied with harmful requests. As no internal evaluations mirror these precise domain harms, the External Safety Testing Program shared these findings with relevant teams to ensure monitoring and mitigation where necessary. 6 Discussion In this report we have introduced the Gemini 2.X model family: Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.0 Flash and Gemini 2.0 Flash-Lite. Taken together, these models span the full Pareto frontier of model capability vs cost, and Gemini 2.5 Pro is the most capable model we have ever developed. Gemini 2.5 Pro excels across a wide range of capabilities, and represents a step change in performance relative to Gemini 1.5 Pro. Its coding, math and reasoning performance are particularly notable and Gemini 2.5 Pro obtains extremely competitive scores on the Aider Polyglot evaluation, GPQA (diamond) and Humanity's Last Exam. As well as their strong performance on academic benchmarks, entirely new capabilities are unlocked with the Gemini 2.5 models. Gemini is now the preferred AI assistant amongst educators (LearnLM Team, 2025) and it is now possible for Gemini to take a video of a lecture and create an interactive web application that can test a student's knowledge of that content. Finally, the Gemini 2.5 models enable exciting new agentic workflows, and have started to power numerous Google products already (Pichai, 2025). In addition to being highly performant, the Gemini 2.5 models maintain strong safety standards and, compared to their 1.5 counterparts, are much more helpful. They are less likely to refuse to answer important user queries or respond with an overly sanctimonious tone. Gemini 2.5 exhibited notable increases in Critical Capabilities, including cybersecurity and machine learning R&D. However, the model has not crossed any Critical Capability Levels. Reflecting on the path to Gemini 2.5, the staggering performance improvement attained over the space of just one year points to a new challenge in AI research: namely that the development of novel and sufficiently challenging evaluation benchmarks has struggled to keep pace with model capability improvements, especially with the advent of capable reasoning agents. Over the space of just a year, Gemini Pro's performance has gone up 5x on Aider Polyglot and 2x on SWE-bench verified (one of the most popular and challenging agentic benchmarks). Not only are benchmarks saturating quickly, but every new benchmark that gets created can end up being more expensive and take longer to create than its predecessor, due to the more restricted pool of experts able to create it. Experts were paid up to $5000 for each question that was accepted to the Humanity's Last Exam benchmark (Phan et al., 2025), and while this benchmark still has significant headroom at the time of writing (June 2025), performance on it has improved significantly over the space of a few months (with the best models achieving just a few percent accuracy on it when it was initially published in early 2025). When one considers agentic systems, which are able to tackle problems for longer and which have access to tools and self critique, the complexity of benchmarks required to measure performance also increases dramatically. Being able to scale evaluations in both their capability coverage and their difficulty, while also representing tasks that have economic value, will be the key to unlocking the next generation of AI systems. References Anil et al. (2018) R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton. Large scale distributed neural network training through online distillation, 2018. URL https://arxiv.org/abs/1804.03235. Anil et al. (2023) R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, et al. PaLM 2 technical report, 2023. URL https://arxiv.org/abs/2305.10403. Anthropic (2025) Anthropic. Claude's extended thinking, 2025. URL https://www.anthropic.com/research/visible-extended-thinking. Baddepudi et al. (2025) A. Baddepudi, A. Yang, and M. Lučić. Advancing the frontier of video understanding with Gemini 2.5, 2025. URL https://developers.googleblog.com/en/gemini-2-5-video-understanding/. Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, et al. Constitutional ai: Harmlessness from ai feedback, 2022. URL https://arxiv.org/abs/2212.08073. Balunović et al. (2025) M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev. Matharena: Evaluating llms on uncontaminated math competitions, 2025. URL https://arxiv.org/abs/2505.23281. Barham et al. (2022) P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy, et al. Pathways: Asynchronous distributed dataflow for ml. Proceedings of Machine Learning and Systems, 4:430-449, 2022. URL https://proceedings.mlr.press/v162/barham22a.html. Beutel et al. (2024) A. Beutel, K. Xiao, J. Heidecke, and L. Weng. Diverse and effective red teaming with auto-generated rewards and multi-step reinforcement learning, 2024. URL https://arxiv.org/abs/2412.18693. Biderman et al. (2023) S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, et al. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, 2023. URL https://proceedings.mlr.press/v202/biderman23a.html. Carlini et al. (2022) N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1113-1130, 2022. URL https://arxiv.org/abs/2202.07646. Chiang et al. (2024) W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, 2024. URL https://arxiv.org/abs/2306.05685. Chowdhery et al. (2022) A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. URL https://arxiv.org/abs/2204.02311. Chowdhury et al. (2024) N. Chowdhury, J. Aung, C. J. Shern, O. Jaffe, D. Sherburn, G. Starace, E. Mays, R. Dias, M. Aljubeh, M. Glaese, C. E. Jimenez, J. Yang, L. Ho, T. Patwardhan, K. Liu, and A. Madry. Introducing SWE-bench verified, 2024. URL https://openai.com/index/introducing-swe-bench-verified/. Clark et al. (2022) A. Clark, D. de las Casas, A. Guy, A. Mensch, M. Paganini, J. Hoffmann, B. Damoc, B. Hechtman, T. Cai, S. Borgeaud, G. van den Driessche, E. Rutherford, T. Hennigan, M. Johnson, K. Millican, A. Cassirer, C. Jones, E. Buchatskaya, D. Budden, L. Sifre, S. Osindero, O. Vinyals, J. Rae, E. Elsen, K. Kavukcuoglu, and K. Simonyan. Unified scaling laws for routed language models, 2022. URL "https://arxiv.org/abs/2202.01169". CodeGemma Team et al. (2024) CodeGemma Team, H. Zhao, J. Hui, J. Howland, N. Nguyen, S. Zuo, A. Hu, C. A. Choquette-Choo, J. Shen, J. Kelley, K. Bansal, L. Vilnis, M. Wirth, P. Michel, P. Choy, P. Joshi, R. Kumar, S. Hashmi, S. Agrawal, Z. Gong, J. Fine, T. Warkentin, A. J. Hartman, B. Ni, K. Korevec, K. Schaefer, and S. Huffman. CodeGemma: Open Code Models Based on Gemma, 2024. URL https://arxiv.org/abs/2406.11409. Conneau et al. (2023) A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798-805. IEEE, 2023. Dehghani et al. (2023) M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023. URL https://proceedings.mlr.press/v202/dehghani23a/dehghani23a.pdf. Doshi (2025a) T. Doshi. Build rich, interactive web apps with an updated Gemini 2.5 Pro, 2025a. URL https://blog.google/products/gemini/gemini-2-5-pro-updates/. Doshi (2025b) T. Doshi. Gemini 2.5: Our most intelligent models are getting even better, 2025b. URL https://blog.google/technology/google-deepmind/google-gemini-updates-io-2025/. Du et al. (2021) N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al. GLaM: Efficient scaling of language models with mixture-of-experts. arXiv preprint arXiv:2112.06905, 2021. URL https://arxiv.org/abs/2112.06905. Fedus et al. (2021) W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021. URL https://arxiv.org/abs/2101.03961. Fu et al. (2025) C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24108-24118, 2025. URL https://openaccess.thecvf.com/content/CVPR2024/html/Fu_Video-MME_The_First-Ever_Comprehensive_Evaluation_Benchmark_of_Multi-Modal_LLMs_in_CVPR_2024_paper.html. Gauthier (2025) P. Gauthier. Aider Polyglot Coding Leaderboard, 2025. URL https://aider.chat/docs/leaderboards/. Gemini Team (2024) Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. URL https://arxiv.org/abs/2403.05530. Gemini Team, Google (2024) Gemini Team, Google. Gemini Deep Research, 2024. URL https://gemini.google/overview/deep-research/. Gemma Team (2024) Gemma Team. Gemma: Open Models Based on Gemini Research and Technology, 2024. URL https://arxiv.org/abs/2403.08295. Goldman et al. (2025) O. Goldman, U. Shaham, D. Malkin, S. Eiger, A. Hassidim, Y. Matias, J. Maynez, A. M. Gilady, J. Riesa, S. Rijhwani, L. Rimell, I. Szpektor, R. Tsarfaty, and M. Eyal. Eclektic: a novel challenge set for evaluation of cross-lingual knowledge transfer, 2025. URL https://arxiv.org/abs/2502.21228. Google DeepMind (2025a) Google DeepMind. Frontier safety framework, February 2025a. URL https://deepmind.google/discover/governance/frontier-safety-framework/. Google DeepMind (2025b) Google DeepMind. Gemini 2.0 Flash-Lite, 2025b. URL https://deepmind.google/models/gemini/flash-lite/. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, et al. The Llama 3 Herd of Models, 2024. URL https://arxiv.org/abs/2407.21783. Hassabis (2025) D. Hassabis. Our vision for building a universal AI assistant, 2025. URL https://blog.google/technology/google-deepmind/gemini-universal-ai-assistant/. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network, 2015. URL https://arxiv.org/abs/1503.02531. Hu et al. (2025) K. Hu, P. Wu, F. Pu, W. Xiao, Y. Zhang, X. Yue, B. Li, and Z. Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos, 2025. URL https://arxiv.org/abs/2501.13826. Hughes et al. (2023) S. Hughes, M. Bae, and M. Li. Vectara Hallucination Leaderboard, nov 2023. URL https://github.com/vectara/hallucination-leaderboard. Ippolito et al. (2022) D. Ippolito, F. Tramer, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini. Preventing verbatim memorization in language models gives a false sense of privacy, 2022. URL https://arxiv.org/abs/2210.17546. Jacovi et al. (2024) A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, C. Saroufim, C. Fry, D. Marcus, D. Kukliansky, G. S. Tomar, J. Swirhun, J. Xing, L. Wang, M. Gurumurthy, M. Aaron, M. Ambar, R. Fellinger, R. Wang, R. Sims, Z. Zhang, S. Goldshtein, and D. Das. Facts grounding leaderboard. https://www.kaggle.com/benchmarks/google/facts-grounding, 2024. Google Deepmind, Google Research, Google Cloud, Kaggle. Jacovi et al. (2025) A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, et al. The facts grounding leaderboard: Benchmarking llms' ability to ground responses to long-form input. arXiv preprint arXiv:2501.03200, 2025. URL https://arxiv.org/abs/2501.03200. Jain et al. (2024) N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL https://arxiv.org/abs/2403.07974. Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. URL https://arxiv.org/abs/2401.04088. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66. Kampf and Brichtova (2025) K. Kampf and N. Brichtova. Experiment with Gemini 2.0 Flash native image generation, 2025. URL https://developers.googleblog.com/en/experiment-with-gemini-20-flash-native-image-generation/. Kavukcuoglu (2025) K. Kavukcuoglu. Gemini 2.0 is now available to everyone, 2025. URL https://blog.google/technology/google-deepmind/gemini-model-updates-february-2025. Kilpatrick (2025) L. Kilpatrick. Gemini 2.5 Pro Preview: even better coding performance, 2025. URL https://developers.googleblog.com/en/gemini-2-5-pro-io-improved-coding-performance. Kudugunta et al. (2023) S. Kudugunta, I. Caswell, B. Zhang, X. Garcia, C. A. Choquette-Choo, K. Lee, D. Xin, A. Kusupati, R. Stella, A. Bapna, and O. Firat. MADLAD-400: A Multilingual And Document-Level Large Audited Dataset, 2023. URL https://arxiv.org/abs/2309.04662. Laurent et al. (2024) J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, et al. LAB-Bench: Measuring capabilities of language models for biology research, 2024. URL https://arxiv.org/abs/2407.10362. LearnLM Team (2025) LearnLM Team. Evaluating Gemini in an Arena for Learning, 2025. URL https://goo.gle/LearnLM-May25. Lee et al. (2024) J. Lee, A. Chen, Z. Dai, D. Dua, D. S. Sachan, M. Boratko, Y. Luan, S. M. Arnold, V. Perot, S. Dalmia, et al. Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprint arXiv:2406.13121, 2024. URL https://arxiv.org/abs/2406.13121. Lei et al. (2021) J. Lei, T. L. Berg, and M. Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:11846-11858, 2021. Lepikhin et al. (2020) D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=qrwe7XHTmYb. Li et al. (2024) N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, et al. The WMDP benchmark: Measuring and reducing malicious use with unlearning, 2024. URL https://arxiv.org/abs/2403.03218. Liu et al. (2020) L. Liu, X. Liu, J. Gao, W. Chen, and J. Han. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08249, 2020. URL https://arxiv.org/abs/2004.08249. LMArena Team (2025) LMArena Team. Webdev arena, 2025. URL https://web.lmarena.ai/leaderboard. Mallick and Kilpatrick (2025) S. B. Mallick and L. Kilpatrick. Gemini 2.0: Flash, Flash-Lite and Pro, 2025. URL https://developers.googleblog.com/en/gemini-2-family-expands/. Mehrotra et al. (2024) A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi. Tree of attacks: Jailbreaking black-box llms automatically, 2024. URL https://arxiv.org/abs/2312.02119. Molybog et al. (2023) I. Molybog, P. Albert, M. Chen, Z. DeVito, D. Esiobu, N. Goyal, P. Koura, S. Narang, A. Poulton, R. Silva, et al. A theory on adam instability in large-scale machine learning. arXiv preprint arXiv:2304.09871, 2023. URL https://arxiv.org/abs/2304.09871. Nagrani et al. (2025a) A. Nagrani, S. Menon, A. Iscen, S. Buch, R. Mehran, N. Jha, A. Hauth, Y. Zhu, C. Vondrick, M. Sirotenko, C. Schmid, and T. Weyand. Minerva: Evaluating complex video reasoning, 2025a. URL https://arxiv.org/abs/2505.00681. Nagrani et al. (2025b) A. Nagrani, M. Zhang, R. Mehran, R. Hornung, N. B. Gundavarapu, N. Jha, A. Myers, X. Zhou, B. Gong, C. Schmid, M. Sirotenko, Y. Zhu, and T. Weyand. Neptune: The long orbit to benchmarking long video understanding, 2025b. URL https://arxiv.org/abs/2412.09582. Nasr et al. (2023) M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. Scalable extraction of training data from (production) language models, 2023. URL https://arxiv.org/abs/2311.17035. Padlewski et al. (2024) P. Padlewski, M. Bain, M. Henderson, Z. Zhu, N. Relan, H. Pham, D. Ong, K. Aleksiev, A. Ormazabal, S. Phua, E. Yeo, E. Lamprecht, Q. Liu, Y. Wang, E. Chen, D. Fu, L. Li, C. Zheng, C. de Masson d'Autume, D. Yogatama, M. Artetxe, and Y. Tay. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models, 2024. URL https://arxiv.org/abs/2405.02287. Pappu et al. (2024) A. Pappu, B. Porter, I. Shumailov, and J. Hayes. Measuring memorization in RLHF for code completion. arXiv preprint arXiv:2406.11715, 2024. URL https://arxiv.org/abs/2406.11715. Patraucean et al. (2023) V. Patraucean, L. Smaira, A. Gupta, A. Recasens, L. Markeeva, D. Banarse, S. Koppula, M. Malinowski, Y. Yang, C. Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36:42748-42761, 2023. Perez et al. (2022) E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models, 2022. URL https://arxiv.org/abs/2202.03286. Phan et al. (2025) L. Phan et al. Humanity's last exam, 2025. URL https://arxiv.org/abs/2501.14249. Phuong et al. (2024) M. Phuong, M. Aitchison, E. Catt, S. Cogan, A. Kaskasoli, V. Krakovna, D. Lindner, M. Rahtz, Y. Assael, S. Hodkinson, et al. Evaluating frontier models for dangerous capabilities, 2024. URL https://arxiv.org/abs/2403.13793. Phuong et al. (2025) M. Phuong, R. S. Zimmermann, Z. Wang, D. Lindner, V. Krakovna, S. Cogan, A. Dafoe, L. Ho, and R. Shah. Evaluating frontier models for stealth and situational awareness, 2025. URL https://arxiv.org/abs/2505.01420. Pichai (2025) S. Pichai. Google I/O 2025: From research to reality, 2025. URL https://blog.google/technology/ai/io-2025-keynote/. Plizzari et al. (2025) C. Plizzari, A. Tonioni, Y. Xian, A. Kulshrestha, and F. Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 24129-24138, 2025. Rein et al. (2024) D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gqqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. Riquelme et al. (2021) C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. S. Pinto, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts, 2021. URL https://arxiv.org/abs/2106.05974. Roberts et al. (2025) J. Roberts, M. R. Taesiri, A. Sharma, A. Gupta, S. Roberts, I. Croitoru, S.-V. Bogolin, J. Tang, F. Langer, V. Raina, et al. ZeroBench: An impossible visual benchmark for contemporary large multimodal models. arXiv preprint arXiv:2502.09696, 2025. Rodriguez et al. (2025) M. Rodriguez, R. A. Popa, L. Liang, A. Wang, M. Rahtz, A. Kaskasoli, A. Dafoe, and F. Flynn. A framework for evaluating emerging cyberattack capabilities of AI, 2025. URL https://arxiv.org/abs/2503.11917. Roller et al. (2021) S. Roller, S. Sukhbaatar, J. Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34:17555-17566, 2021. URL https://proceedings.neurips.cc/paper/2021/file/883e881bc596359e0c5112411858a74b-Paper.pdf. Samvelyan et al. (2024) M. Samvelyan, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y. Mao, M. Jiang, J. Parker-Holder, J. Foerster, T. Rocktäschel, and R. Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts, 2024. URL https://arxiv.org/abs/2402.16822. Shah et al. (2025) R. Shah, A. Irpan, A. M. Turner, A. Wang, A. Conmy, D. Lindner, J. Brown-Cohen, L. Ho, N. Nanda, R. A. Popa, R. Jain, R. Greig, S. Albanie, S. Emmons, S. Farquhar, S. Krier, S. Rajamanoharan, S. Bridgers, T. Ijitoye, T. Everitt, V. Krakovna, V. Varma, V. Mikulik, Z. Kenton, D. Orr, S. Legg, N. Goodman, A. Dafoe, F. Flynn, and A. Dragan. An approach to technical agi safety and security, 2025. URL https://arxiv.org/abs/2504.01849. Sharon (2025) D. Sharon. Upload and edit your images directly in the Gemini app, 2025. URL https://blog.google/products/gemini/image-editing/. Shazeer et al. (2017) N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR (Poster). OpenReview.net, 2017. URL https://arxiv.org/abs/1701.06538. Shi et al. (2025) C. Shi, S. Lin, S. Song, J. Hayes, I. Shumailov, I. Yona, J. Pluto, A. Pappu, C. A. Choquette-Choo, M. Nasr, C. Sitawarin, G. Gibson, A. Terzis, and J. F. Flynn. Lessons from defending gemini against indirect prompt injections, 2025. URL https://arxiv.org/abs/2505.14534. Singh et al. (2024) S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Longpre, W.-Y. Ko, M. Smith, A. Bosselut, A. Oh, A. F. T. Martins, L. Choshen, D. Ippolito, E. Ferrante, M. Fadaee, B. Ermis, and S. Hooker. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation, 2024. URL https://arxiv.org/abs/2412.03304. Stein (2025) R. Stein. Expanding AI Overviews and introducing AI Mode, 2025. URL https://blog.google/products/search/ai-mode-search. Tolstikhin et al. (2021) I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision, 2021. Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. Vodrahalli et al. (2024) K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024. URL https://arxiv.org/abs/2409.12640. Wang (2024) B. Wang. NotebookLM now lets you listen to a conversation about your sources , 2024. URL https://blog.google/technology/ai/notebooklm-audio-overviews. Wang et al. (2020) C. Wang, A. Wu, and J. Pino. Covost 2: A massively multilingual speech-to-text translation corpus, 2020. Wang et al. (2024) W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, X. Gu, S. Huang, B. Xu, Y. Dong, M. Ding, and J. Tang. Lvbench: An extreme long video understanding benchmark, 2024. URL https://arxiv.org/abs/2406.08035. Wang et al. (2019) X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, and W. Y. Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4581-4591, 2019. Wei et al. (2024) J. Wei, K. Nguyen, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024. URL https://arxiv.org/abs/2411.04368. Weidinger et al. (2024) L. Weidinger, J. Barnhart, J. Brennan, C. Butterfield, S. Young, W. Hawkins, et al. Holistic safety and responsibility evaluations of advanced ai models, 2024. URL https://arxiv.org/abs/2404.14068. Wijk et al. (2025) H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, et al. RE-Bench: Evaluating frontier ai r&d capabilities of language model agents against human experts, 2025. URL https://arxiv.org/abs/2411.15114. Wortsman et al. (2023) M. Wortsman, P. J. Liu, L. Xiao, K. Everett, A. Alemi, B. Adlam, J. D. Co-Reyes, I. Gur, A. Kumar, R. Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023. URL https://arxiv.org/abs/2309.14322. Yang et al. (2023) J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback, 2023. URL https://arxiv.org/abs/2306.14898. Yu et al. (2019) Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao. ActivityNet-QA: A dataset for understanding complex web videos via question answering. In AAAI, 2019. Yue et al. (2024) X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024. Zerokid (2024) Zerokid. Pokemon Red Version - Guide and Walkthrough (GB), 2024. URL {https://gamefaqs.gamespot.com/gameboy/367023-Pok\'emon-red-version/faqs/64175}. Zhai et al. (2023) S. Zhai, T. Likhomanenko, E. Littwin, D. Busbridge, J. Ramapuram, Y. Zhang, J. Gu, and J. M. Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pages 40770-40803. PMLR, 2023. URL https://proceedings.mlr.press/v202/zhai23a/zhai23a.pdf. Zhang (2025) J. Zhang. Gemini Plays Pokemon Twitch Stream, 2025. URL https://www.twitch.tv/gemini_plays_pokemon/about. Zhang et al. (2022) S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. URL https://arxiv.org/abs/2205.01068. Zhou et al. (2018) L. Zhou, C. Xu, and J. J. Corso. Towards automatic learning of procedures from web instructional videos. In AAAI Conference on Artificial Intelligence, pages 7590-7598, 2018. URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17344. 7 Contributors and Acknowledgments Contributors Gheorghe Comanici Eric Bieber Mike Schaekermann Ice Pasupat Noveen Sachdeva Inderjit Dhillon Marcel Blistein Ori Ram Dan Zhang Evan Rosen Luke Marris Sam Petulla Colin Gaffney Asaf Aharoni Nathan Lintz Tiago Cardal Pais Henrik Jacobsson Idan Szpektor Nan-Jiang Jiang Krishna Haridasan Ahmed Omran Nikunj Saunshi Dara Bahri Gaurav Mishra Eric Chu Toby Boyd Brad Hekman Aaron Parisi Chaoyi Zhang Kornraphop Kawintiranon Tania Bedrax-Weiss Oliver Wang Ya Xu Ollie Purkiss Uri Mendlovic Ilaï Deutel Nam Nguyen Adam Langley Flip Korn Lucia Rossazza Alexandre Ramé Sagar Waghmare Helen Miller Nathan Byrd Ashrith Sheshan Raia Hadsell Sangnie Bhardwaj Pawel Janus Tero Rissa Dan Horgan Alvin Abdagic Lior Belenki James Allingham Anima Singh Theo Guidroz Srivatsan Srinivasan Herman Schmit Kristen Chiafullo Andre Elisseeff Nilpa Jha Prateek Kolhar Leonard Berrada Frank Ding Xiance Si Shrestha Basu Mallick Franz Och Sofia Erell Eric Ni Tejasi Latkar Sherry Yang Petar Sirkovic Ziqiang Feng Robert Leland Rachel Hornung Gang Wu Charles Blundell Hamidreza Alvari Po-Sen Huang Cathy Yip Sanja Deur Li Liu Gabriela Surita Pablo Duque Dima Damen Johnson Jia Arthur Guez Markus Mircea Animesh Sinha Alberto Magni Paweł Stradomski Tal Marian Vlado Galić Wenhu Chen Hisham Husain Achintya Singhal Dominik Grewe François-Xavier Aubet Shuang Song Lorenzo Blanco Leland Rechis Lewis Ho Rich Munoz Kelvin Zheng Jessica Hamrick Kevin Mather Hagai Taitelbaum Eliza Rutherford Yun Lei Kuangyuan Chen Anand Shukla Erica Moreira Eric Doi Berivan Isik Nir Shabat Dominika Rogozińska Kashyap Kolipaka Jason Chang Eugen Vušak Srinivasan Venkatachary Shadi Noghabi Tarun Bharti Younghoon Jun Aleksandr Zaks Simon Green Jeshwanth Challagundla William Wong Muqthar Mohammad Dean Hirsch Yong Cheng Iftekhar Naim Lev Proleev Damien Vincent Aayush Singh Maxim Krikun Dilip Krishnan Zoubin Ghahramani Aviel Atias Rajeev Aggarwal Christo Kirov Dimitrios Vytiniotis Christy Koh Alexandra Chronopoulou Pawan Dogra Vlad-Doru Ion Gladys Tyen Jason Lee Felix Weissenberger Trevor Strohman Ashwin Balakrishna Jack Rae Marko Velic Raoul de Liedekerke Oded Elyada Wentao Yuan Canoee Liu Lior Shani Sergey Kishchenko Bea Alessio Yandong Li Richard Song Sam Kwei Orion Jankowski Aneesh Pappu Youhei Namiki Yenai Ma Nilesh Tripuraneni Colin Cherry Marissa Ikonomidis Yu-Cheng Ling Colin Ji Beka Westberg Auriel Wright Da Yu David Parkinson Swaroop Ramaswamy Jerome Connor Soheil Hassas Yeganeh Snchit Grover George Kenwright Lubo Litchev Chris Apps Alex Tomala Felix Halim Alex Castro-Ros Zefei Li Anudhyan Boral Pauline Sho Michal Yarom Eric Malmi David Klinghoffer Rebecca Lin Alan Ansell Pradeep Kumar S Shubin Zhao Siqi Zuo Adam Santoro Heng-Tze Cheng Solomon Demmessie Yuchi Liu Nicole Brichtova Allie Culp Nathaniel Braun Dan Graur Will Ng Nikhil Mehta Aaron Phillips Patrik Sundberg Varun Godbole Fangyu Liu Yash Katariya David Rim Mojtaba Seyedhosseini Sean Ammirati Jonas Valfridsson Mahan Malihi Timothy Knight Andeep Toor Thomas Lampe Abe Ittycheriah Lewis Chiang Chak Yeung Alexandre Fréchette Jinmeng Rao Huisheng Wang Himanshu Srivastava Richard Zhang Rocky Rhodes Ariel Brand Dean Weesner Ilya Figotin Felix Gimeno Rachana Fellinger Pierre Marcenac José Leal Eyal Marcus Victor Cotruta Rodrigo Cabrera Sheryl Luo Dan Garrette Vera Axelrod Sorin Baltateanu David Barker Dongkai Chen Horia Toma Ben Ingram Jason Riesa Chinmay Kulkarni Yujing Zhang Hongbin Liu Chao Wang Martin Polacek Will Wu Kai Hui Adrian N Reyes Yi Su Megan Barnes Ishaan Malhi Anfal Siddiqui Qixuan Feng Mihai Damaschin Daniele Pighin Andreas Steiner Samuel Yang Ramya Sree Boppana Simeon Ivanov Arun Kandoor Aditya Shah Asier Mujika Da Huang Christopher A. Choquette-Choo Mohak Patel Tianhe Yu Toni Creswell Jerry (Chun-Ting) Liu Catarina Barros Yasaman Razeghi Aurko Roy Phil Culliton Binbin Xiong Jiaqi Pan Thomas Strohmann Tolly Powell Babi Seal Doug DeCarlo Pranav Shyam Kaan Katircioglu Xuezhi Wang Cassidy Hardin Immanuel Odisho Josef Broder Oscar Chang Arun Nair Artem Shtefan Maura O'Brien Manu Agarwal Sahitya Potluri Siddharth Goyal Amit Jhindal Saksham Thakur Yury Stuken James Lyon Kristina Toutanova Fangxiaoyu Feng Austin Wu Ben Horn Alek Wang Alex Cullum Gabe Taubman Disha Shrivastava Chongyang Shi Hamish Tomlinson Roma Patel Tao Tu Ada Maksutaj Oflazer Francesco Pongetti Mingyao Yang Adrien Ali Taïga Vincent Perot Nuo Wang Pierse Feng Han Yoel Drori Iñaki Iturrate Ayan Chakrabarti Legg Yeung Dave Dopson Yi-ting Chen Apoorv Kulshreshtha Tongfei Guo Philip Pham Tal Schuster Junquan Chen Alex Polozov Jinwei Xing Huanjie Zhou Praneeth Kacham Doron Kukliansky Antoine Miech Sergey Yaroshenko Ed Chi Sholto Douglas Hongliang Fei Mathieu Blondel Preethi Myla Lior Madmoni Xing Wu Daniel Keysers Kristian Kjems Isabela Albuquerque Lijun Yu Joel D'sa Michelle Plantan Vlad Ionescu Jaume Sanchez Elias Abhirut Gupta Manish Reddy Vuyyuru Fred Alcober Tong Zhou Kaiyang Ji Florian Hartmann Subha Puttagunta Hugo Song Ehsan Amid Anca Stefanoiu Andrew Lee Paul Pucciarelli Emma Wang Amit Raul Slav Petrov Isaac Tian Valentin Anklin Nana Nti Victor Gomes Max Schumacher Grace Vesom Alex Panagopoulos Konstantinos Bousmalis Daniel Andor Josh Jacob Yuan Zhang Bill Rosgen Matija Kecman Matthew Tung Alexandra Belias Noah Goodman Paul Covington Brian Wieder Nikita Saxena Elnaz Davoodi Muhuan Huang Sharath Maddineni Vincent Roulet Folawiyo Campbell-Ajala Pier Giuseppe Sessa Xintian (Cindy) Wu Guangda Lai Paul Collins Alex Haig Vytenis Sakenas Xiaowei Xu Marissa Giustina Laurent El Shafey Pichi Charoenpanit Shefali Garg Joshua Ainslie Boone Severson Montse Gonzalez Arenas Shreya Pathak Sujee Rajayogam Jie Feng Michiel Bakker Sheng Li Nevan Wichers Jamie Rogers Xinyang Geng Yeqing Li Rolf Jagerman Chao Jia Nadav Olmert David Sharon Matthew Mauger Sandeep Mariserla Hongxu Ma Megha Mohabey Kyuyeun Kim Alek Andreev Scott Pollom Juliette Love Vihan Jain Priyanka Agrawal Yannick Schroecker Alisa Fortin Manfred Warmuth Ji Liu Andrew Leach Irina Blok Ganesh Poomal Girirajan Roee Aharoni Benigno Uria Andrei Sozanschi Dan Goldberg Lucian Ionita Marco Tulio Ribeiro Martin Zlocha Vighnesh Birodkar Sami Lachgar Liangzhe Yuan Himadri Choudhury Matt Ginsberg Fei Zheng Gregory Dibb Emily Graves Swachhand Lokhande Gabriel Rasskin George-Cristian Muraru Corbin Quick Sandeep Tata Pierre Sermanet Aditya Chawla Itay Karo Yan Wang Susan Zhang Orgad Keller Anca Dragan Guolong Su Ian Chou Xi Liu Yiqing Tao Shruthi Prabhakara Marc Wilson Ruibo Liu Shibo Wang Georgie Evans David Du Alfonso Castaño Gautam Prasad Mona El Mahdy Sebastian Gerlach Machel Reid Jarrod Kahn Amir Zait Thanumalayan Sankaranarayana Pillai Thatcher Ulrich Guanyu Wang Jan Wassenberg Efrat Farkash Kiran Yalasangi Congchao Wang Maria Bauza Simon Bucher Ting Liu Jun Yan Gary Leung Vikas Sindhwani Parker Barnes Avi Singh Ivan Jurin Jichuan Chang Niket Kumar Bhumihar Sivan Eiger Gui Citovsky Ben Withbroe Zhang Li Siyang Xue Niccolò Dal Santo Georgi Stoyanov Yves Raimond Steven Zheng Yilin Gao Vít Listík Sławek Kwasiborski Rachel Saputro Adnan Ozturel Ganesh Mallya Kushal Majmundar Ross West Paul Caron Jinliang Wei Lluis Castrejon Sharad Vikram Deepak Ramachandran Nikhil Dhawan Jiho Park Sara Smoot George van den Driessche Yochai Blau Chase Malik Wei Liang Roy Hirsch Cicero Nogueira dos Santos Eugene Weinstein Aäron van den Oord Sid Lall Nicholas FitzGerald Zixuan Jiang Xuan Yang Dale Webster Ali Elqursh Aedan Pope Georges Rotival David Raposo Wanzheng Zhu Jeff Dean Sami Alabed Dustin Tran Arushi Gupta Zach Gleicher Jessica Austin Edouard Rosseel Megh Umekar Dipanjan Das Yinghao Sun Kai Chen Karolis Misiunas Xiang Zhou Yixian Di Alyssa Loo Josh Newlan Bo Li Vinay Ramasesh Ying Xu Alex Chen Sudeep Gandhe Radu Soricut Nikita Gupta Shuguang Hu Seliem El-Sayed Xavier Garcia Idan Brusilovsky Pu-Chin Chen Andrew Bolt Lu Huang Alex Gurney Zhiying Zhang Alexander Pritzel Jarek Wilkiewicz Bryan Seybold Bhargav Kanagal Shamanna Felix Fischer Josef Dean Karan Gill Ross Mcilroy Abhishek Bhowmick Jeremy Selier Antoine Yang Derek Cheng Vladimir Magay Jie Tan Dhriti Varma Christian Walder Tomas Kocisky Ryo Nakashima Paul Natsev Mike Kwong Ionel Gog Chiyuan Zhang Sander Dieleman Thomas Jimma Andrey Ryabtsev Siddhartha Brahma David Steiner Dayou Du Ante Žužul Mislav Žanić Mukund Raghavachari Willi Gierke Zeyu Zheng Dessie Petrova Yann Dauphin Yuchuan Liu Ido Kessler Steven Hand Chris Duvarney Seokhwan Kim Hyo Lee Léonard Hussenot Jeffrey Hui Josh Smith Deepali Jain Jiawei Xia Gaurav Singh Tomar Keyvan Amiri Du Phan Fabian Fuchs Tobias Weyand Nenad Tomasev Alexandra Cordell Xin Liu Jonathan Mallinson Pankaj Joshi Andy Crawford Arun Suggala Steve Chien Nick Fernando Mariella Sanchez-Vargas Duncan Williams Phil Crone Xiyang Luo Igor Karpov Jyn Shan Terry Thurk Robin Strudel Paul Voigtlaender Piyush Patil Tim Dozat Ali Khodaei Sahil Singla Piotr Ambroszczyk Qiyin Wu Yifan Chang Brian Roark Chaitra Hegde Tianli Ding Angelos Filos Zhongru Wu André Susano Pinto Shuang Liu Saarthak Khanna Aditya Pandey Siobhan Mcloughlin Qiujia Li Sam Haves Allan Zhou Elena Buchatskaya Isabel Leal Peter de Boursac Nami Akazawa Nina Anderson Terry Chen Krishna Somandepalli Chen Liang Sheela Goenka Stephanie Winkler Alexander Grushetsky Yifan Ding Jamie Smith Fan Ye Jordi Pont-Tuset Eric Li Ruichao Li Tomer Golany Dawid Wegner Tao Jiang Omer Barak Yuan Shangguan Eszter Vértes Renee Wong Jörg Bornschein Alex Tudor Michele Bevilacqua Tom Schaul Ankit Singh Rawat Yang Zhao Kyriakos Axiotis Lei Meng Cory McLean Jonathan Lai Jennifer Beattie Nate Kushman Yaxin Liu Blair Kutzman Fiona Lang Jingchen Ye Praneeth Netrapalli Pushkar Mishra Myriam Khan Megha Goel Rob Willoughby David Tian Honglei Zhuang JD Chen Zak Tsai Tasos Kementsietsidis Arjun Khare James Keeling Keyang Xu Nathan Waters Florent Altché Ashok Popat Bhavishya Mittal David Saxton Dalia El Badawy Michael Mathieu Zheng Zheng Hao Zhou Nishant Ranka Richard Shin Qingnan Duan Tim Salimans Ioana Mihailescu Uri Shaham Ming-Wei Chang Yannis Assael Nishanth Dikkala Martin Izzard Vincent Cohen-Addad Cat Graves Vlad Feinberg Grace Chung DJ Strouse Danny Karmon Sahand Sharifzadeh Zoe Ashwood Khiem Pham Jon Blanton Alex Vasiloff Jarred Barber Mark Geller Aurick Zhou Fedir Zubach Tzu-Kuo Huang Lei Zhang Himanshu Gupta Matt Young Julia Proskurnia Ronny Votel Valentin Gabeur Gabriel Barcik Aditya Tripathi Hongkun Yu Geng Yan Beer Changpinyo Filip Pavetić Amy Coyle Yasuhisa Fujii Jorge Gonzalez Mendez Tianhao Zhou Harish Rajamani Blake Hechtman Eddie Cao Da-Cheng Juan Yi-Xuan Tan Valentin Dalibard Yilun Du Natalie Clay Kaisheng Yao Wenhao Jia Dimple Vijaykumar Yuxiang Zhou Xinyi Bai Wei-Chih Hung Steven Pecht Georgi Todorov Nikhil Khadke Pramod Gupta Preethi Lahoti Arnaud Autef Karthik Duddu James Lee-Thorp Alexander Bykovsky Tautvydas Misiunas Sebastian Flennerhag Santhosh Thangaraj Jed McGiffin Zack Nado Markus Kunesch Andreas Noever Amir Hertz Marco Liang Victor Stone Evan Palmer Samira Daruki Arijit Pramanik Siim Põder Austin Kyker Mina Khan Evgeny Sluzhaev Marvin Ritter Avraham Ruderman Wenlei Zhou Chirag Nagpal Kiran Vodrahalli George Necula Paul Barham Ellie Pavlick Jay Hartford Izhak Shafran Long Zhao Maciej Mikuła Tom Eccles Hidetoshi Shimokawa Kanav Garg Luke Vilnis Hanwen Chen Ilia Shumailov Kuang-Huei Lee Abdelrahman Abdelhamed Meiyan Xie Vered Cohen Ester Hlavnova Dan Malkin Chawin Sitawarin James Lottes Pauline Coquinot Tianli Yu Sandeep Kumar Jingwei Zhang Aroma Mahendru Zafarali Ahmed James Martens Tao Chen Aviel Boag Daiyi Peng Coline Devin Arseniy Klimovskiy Mary Phuong Danny Vainstein Jin Xie Bhuvana Ramabhadran Nathan Howard Xinxin Yu Gitartha Goswami Jingyu Cui Sam Shleifer Mario Pinto Chih-Kuan Yeh Ming-Hsuan Yang Sara Javanmardi Dan Ethier Chace Lee Jordi Orbay Suyog Kotecha Carla Bromberg Pete Shaw James Thornton Adi Gerzi Rosenthal Shane Gu Matt Thomas Ian Gemp Aditya Ayyar Asahi Ushio Aarush Selvan Joel Wee Chenxi Liu Maryam Majzoubi Weiren Yu Jake Abernethy Tyler Liechty Renke Pan Hoang Nguyen Qiong (Q) Hu Sarah Perrin Abhinav Arora Emily Pitler Weiyi Wang Kaushik Shivakumar Flavien Prost Ben Limonchik Jing Wang Yi Gao Timothee Cour Shyamal Buch Huan Gui Maria Ivanova Philipp Neubeck Kelvin Chan Lucy Kim Huizhong Chen Naman Goyal Da-Woon Chung Lu Liu Yao Su Anastasia Petrushkina Jiajun Shen Armand Joulin Yuanzhong Xu Stein Xudong Lin Yana Kulizhskaya Ciprian Chelba Shobha Vasudevan Eli Collins Vasilisa Bashlovkina Tony Lu Doug Fritz Jongbin Park Yanqi Zhou Chen Su Richard Tanburn Mikhail Sushkov Mitchelle Rasquinha Jinning Li Jennifer Prendki Yiming Li Pallavi LV Shriya Sharma Hen Fitoussi Hui Huang Andrew Dai Phuong Dao Mike Burrows Henry Prior Danfeng Qin Golan Pundak Lars Lowe Sjoesund Art Khurshudov Zhenkai Zhu Albert Webson Elizabeth Kemp Tat Tan Saurabh Agrawal Susie Sargsyan Liqun Cheng Jim Stephan Tom Kwiatkowski David Reid Arunkumar Byravan Assaf Hurwitz Michaely Nicolas Heess Luowei Zhou Sonam Goenka Viral Carpenter Anselm Levskaya Bo Wang Reed Roberts Rémi Leblond Sharat Chikkerur Stav Ginzburg Max Chang Robert Riachi Chuqiao (Joyce) Xu Zalán Borsos Michael Pliskin Julia Pawar Morgane Lustman Hannah Kirkwood Ankit Anand Aditi Chaudhary Norbert Kalb Kieran Milan Sean Augenstein Anna Goldie Laurel Prince Karthik Raman Yanhua Sun Vivian Xia Aaron Cohen Zhouyuan Huo Josh Camp Seher Ellis Lukas Zilka David Vilar Torres Lisa Patel Sho Arora Betty Chan Jonas Adler Kareem Ayoub Jacky Liang Fayaz Jamil Jiepu Jiang Simon Baumgartner Haitian Sun Yael Karov Yaroslav Akulov Hui Zheng Irene Cai Claudio Fantacci James Rubin Alex Rav Acha Mengchao Wang Nina D'Souza Rohit Sathyanarayana Shengyang Dai Simon Rowe Andrey Simanovsky Omer Goldman Yuheng Kuang Xiaoyue Pan Andrew Rosenberg Tania Rojas-Esponda Praneet Dutta Amy Zeng Irina Jurenka Greg Farquhar Yamini Bansal Shariq Iqbal Becca Roelofs Ga-Young Joung Parker Beak Changwan Ryu Ryan Poplin Yan Wu Jean-Baptiste Alayrac Senaka Buthpitiya Olaf Ronneberger Caleb Habtegebriel Wei Li Paul Cavallaro Aurora Wei Guy Bensky Timo Denk Harish Ganapathy Jeff Stanway Pratik Joshi Francesco Bertolini Jessica Lo Olivia Ma Zachary Charles Geta Sampemane Himanshu Sahni Xu Chen Harry Askham David Gaddy Peter Young Jiewen Tan Matan Eyal Arthur Bražinskas Li Zhong Zhichun Wu Mark Epstein Kai Bailey Andrew Hard Kamyu Lee Sasha Goldshtein Alex Ruiz Mohammed Badawi Matthias Lochbrunner JK Kearns Ashley Brown Fabio Pardo Theophane Weber Haichuan Yang Pan-Pan Jiang Berkin Akin Zhao Fu Marcus Wainwright Chi Zou Meenu Gaba Pierre-Antoine Manzagol Wendy Kan Yang Song Karina Zainullina Rui Lin Jeongwoo Ko Salil Deshmukh Apoorv Jindal James Svensson Divya Tyam Heri Zhao Christine Kaeser-Chen Scott Baird Pooya Moradi Jamie Hall Qiuchen Guo Vincent Tsang Bowen Liang Fernando Pereira Suhas Ganesh Ivan Korotkov Jakub Adamek Sridhar Thiagarajan Vinh Tran Charles Chen Chris Tar Sanil Jain Ishita Dasgupta Taylan Bilal David Reitter Kai Zhao Giulia Vezzani Yasmin Gehman Pulkit Mehta Lauren Beltrone Xerxes Dotiwalla Sergio Guadarrama Zaheer Abbas Stefani Karp Petko Georgiev Chun-Sung Ferng Marc Brockschmidt Liqian Peng Christoph Hirnschall Vikas Verma Yingying Bi Ying Xiao Avigail Dabush Kelvin Xu Phil Wallis Randall Parker Qifei Wang Yang Xu Ilkin Safarli Dinesh Tewari Yin Zhang Seungyeon Kim Andrea Gesmundo Mackenzie Thomas Sergey Levi Ahmed Chowdhury Kanishka Rao Peter Garst Sam Conway-Rahman Helen Ran Kay McKinney Zhisheng Xiao Wenhao Yu Rohan Agrawal Axel Stjerngren Catalin Ionescu Jingjing Chen Vivek Sharma Justin Chiu Fei Liu Ken Franko Clayton Sanford Xingyu Cai Paul Michel Sanjay Ganapathy Jane Labanowski Zachary Garrett Ben Vargas Sean Sun Bryan Gale Thomas Buschmann Guillaume Desjardins Nimesh Ghelani Palak Jain Mudit Verma Chulayuth Asawaroengchai Julian Eisenschlos Jitendra Harlalka Hideto Kazawa Don Metzler Joshua Howland Ying Jian Jake Ades Viral Shah Tynan Gangwani Seungji Lee Roman Ring Steven M. Hernandez Dean Reich Amer Sinha Ashutosh Sathe Joe Kovac Ashleah Gill Ajay Kannan Andrea D'olimpio Martin Sevenich Jay Whang Been Kim Khe Chai Sim Jilin Chen Jiageng Zhang Shuba Lall Yossi Matias Bill Jia Abe Friesen Sara Nasso Ashish Thapliyal Bryan Perozzi Ting Yu Anna Shekhawat Safeen Huda Peter Grabowski Eric Wang Ashwin Sreevatsa Hilal Dib Mehadi Hassen Parker Schuh Vedrana Milutinovic Chris Welty Michael Quinn Ali Shah Bangju Wang Gabe Barth-Maron Justin Frye Natalie Axelsson Tao Zhu Yukun Ma Irene Giannoumis Hanie Sedghi Chang Ye Yi Luan Kevin Aydin Bilva Chandra Vivek Sampathkumar Ronny Huang Victor Lavrenko Ahmed Eleryan Zhi Hong Steven Hansen Sara Mc Carthy Bidisha Samanta Domagoj Ćevid Xin Wang Fangtao Li Michael Voznesensky Matt Hoffman Andreas Terzis Vikash Sehwag Gil Fidel Luheng He Mu Cai Yanzhang He Alex Feng Martin Nikoltchev Samrat Phatale Jason Chase Rory Lawton Ming Zhang Tom Ouyang Manuel Tragut Mehdi Hafezi Manshadi Arjun Narayanan Jiaming Shen Xu Gao Tolga Bolukbasi Nick Roy Xin Li Daniel Golovin Liviu Panait Zhen Qin Guangxing Han Thomas Anthony Sneha Kudugunta Viorica Patraucean Aniket Ray Xinyun Chen Xiaochen Yang Tanuj Bhatia Pranav Talluri Alex Morris Andrija Ražnatović Bethanie Brownfield James An Sheng Peng Patrick Kane Ce Zheng Nico Duduta Joshua Kessinger James Noraky Siqi Liu Keran Rong Petar Veličković Keith Rush Alex Goldin Fanny Wei Shiva Mohan Reddy Garlapati Caroline Pantofaru Okwan Kwon Jianmo Ni Eric Noland Julia Di Trapani Françoise Beaufays Abhijit Guha Roy Yinlam Chow Aybuke Turker Geoffrey Cideron Lantao Mei Jon Clark Qingyun Dou Matko Bošnjak Ralph Leith Yuqing Du Amir Yazdanbakhsh Milad Nasr Chester Kwak Suraj Satishkumar Sheth Alex Kaskasoli Ankesh Anand Balaji Lakshminarayanan Sammy Jerome David Bieber Chun-Te Chu Alexandre Senges Tianxiao Shen Mukund Sridhar Ndaba Ndebele Benjamin Beyret Shakir Mohamed Mia Chen Markus Freitag Jiaxian Guo Luyang Liu Paul Roit Heng Chen Shen Yan Tom Stone JD Co-Reyes Jeremy Cole Salvatore Scellato Shekoofeh Azizi Hadi Hashemi Alicia Jin Anand Iyer Marcella Valentine András György Arun Ahuja Daniel Hernandez Diaz Chen-Yu Lee Nathan Clement Weize Kong Drew Garmon Ishaan Watts Kush Bhatia Khyatti Gupta Matt Miecnikowski Hugo Vallet Ankur Taly Edward Loper Saket Joshi James Atwood Jo Chick Mark Collier Fotis Iliopoulos Ryan Trostle Beliz Gunel Ramiro Leal-Cavazos Arnar Mar Hrafnkelsson Michael Guzman Xiaoen Ju Andy Forbes Jesse Emond Kushal Chauhan Ben Caine Li Xiao Wenjun Zeng Alexandre Moufarek Daniel Murphy Maya Meng Nitish Gupta Felix Riedel Anil Das Elijah Lawal Shashi Narayan Tiberiu Sosea James Swirhun Linda Friso Behnam Neyshabur Jing Lu Sertan Girgin Michael Wunder Edouard Yvinec Aroonalok Pyne Victor Carbune Shruti Rijhwani Yang Guo Tulsee Doshi Anton Briukhov Max Bain Ayal Hitron Xuanhui Wang Ashish Gupta Ke Chen Cosmo Du Weiyang Zhang Dhruv Shah Arjun Akula Max Dylla Ashyana Kachra Weicheng Kuo Tingting Zou Lily Wang Luyao Xu Jifan Zhu Justin Snyder Sachit Menon Orhan Firat Igor Mordatch Yuan Yuan Natalia Ponomareva Rory Blevins Lawrence Moore Weijun Wang Phil Chen Martin Scholz Artur Dwornik Jason Lin Sicheng Li Diego Antognini Te I Xiaodan Song Matt Miller Uday Kalra Adam Raveret Oscar Akerlund Felix Wu Andrew Nystrom Namrata Godbole Tianqi Liu Hannah DeBalsi Jewel Zhao Buhuang Liu Avi Caciularu Lauren Lax Urvashi Khandelwal Victoria Langston Eric Bailey Silvio Lattanzi Yufei Wang Neel Kovelamudi Sneha Mondal Guru Guruganesh Nan Hua Ofir Roval Paweł Wesołowski Rishikesh Ingale Jonathan Halcrow Tim Sohn Christof Angermueller Bahram Raad Eli Stickgold Eva Lu Alec Kosik Jing Xie Timothy Lillicrap Austin Huang Lydia Lihui Zhang Dominik Paulus Clement Farabet Alex Wertheim Bing Wang Rishabh Joshi Chu-ling Ko Yonghui Wu Shubham Agrawal Lily Lin XiangHai Sheng Peter Sung Tyler Breland-King Christina Butterfield Swapnil Gawde Sumeet Singh Qiao Zhang Raj Apte Shilpa Shetty Adrian Hutter Tao Li Elizabeth Salesky Federico Lebron Jonni Kanerva Michela Paganini Arthur Nguyen Rohith Vallu Jan-Thorsten Peter Sarmishta Velury David Kao Jay Hoover Anna Bortsova Colton Bishop Shoshana Jakobovits Alessandro Agostini Alekh Agarwal Chang Liu Charles Kwong Sasan Tavakkol Ioana Bica Alex Greve Anirudh GP Jake Marcus Le Hou Tom Duerig Rivka Moroshko Dave Lacey Andy Davis Julien Amelot Guohui Wang Frank Kim Theofilos Strinopoulos Hui Wan Charline Le Lan Shankar Krishnan Haotian Tang Peter Humphreys Junwen Bai Idan Heimlich Shtacher Diego Machado Chenxi Pang Ken Burke Dangyi Liu Renga Aravamudhan Yue Song Ed Hirst Abhimanyu Singh Brendan Jou Liang Bai Francesco Piccinno Chuyuan Kelly Fu Robin Alazard Barak Meiri Daniel Winter Charlie Chen Mingda Zhang Jens Heitkaemper John Lambert Jinhyuk Lee Alexander Frömmgen Sergey Rogulenko Pranav Nair Paul Niemczyk Anton Bulyenov Bibo Xu Hadar Shemtov Morteza Zadimoghaddam Serge Toropov Mateo Wirth Hanjun Dai Sreenivas Gollapudi Daniel Zheng Alex Kurakin Chansoo Lee Kalesha Bullard Nicolas Serrano Ivana Balazevic Yang Li Johan Schalkwyk Mark Murphy Mingyang Zhang Kevin Sequeira Romina Datta Nishant Agrawal Charles Sutton Nithya Attaluri Mencher Chiang Wael Farhan Gregory Thornton Kate Lin Travis Choma Hung Nguyen Kingshuk Dasgupta Dirk Robinson Iulia Comşa Michael Riley Arjun Pillai Basil Mustafa Ben Golan Amir Zandieh Jean-Baptiste Lespiau Billy Porter David Ross Sujeevan Rajayogam Mohit Agarwal Subhashini Venugopalan Bobak Shahriari Qiqi Yan Hao Xu Taylor Tobin Pavel Dubov Hongzhi Shi Adrià Recasens Anton Kovsharov Sebastian Borgeaud Lucio Dery Shanthal Vasanth Elena Gribovskaya Linhai Qiu Mahdis Mahdieh Wojtek Skut Elizabeth Nielsen CJ Zheng Adams Yu Carrie Grimes Bostock Shaleen Gupta Aaron Archer Chris Rawles Elinor Davies Alexey Svyatkovskiy Tomy Tsai Yoni Halpern Christian Reisswig Bartek Wydrowski Bo Chang Joan Puigcerver Mor Hazan Taege Jian Li Eva Schnider Xinjian Li Dragos Dena Yunhan Xu Umesh Telang Tianze Shi Heiga Zen Kyle Kastner Yeongil Ko Neesha Subramaniam Aviral Kumar Pete Blois Zhuyun Dai John Wieting Yifeng Lu Yoel Zeldes Tian Xie Anja Hauth Alexandru Ţifrea Yuqi Li Sam El-Husseini Dan Abolafia Howard Zhou Wen Ding Sahra Ghalebikesabi Carlos Guía Andrii Maksai Ágoston Weisz Sercan Arik Nick Sukhanov Aga Świetlik Xuhui Jia Luo Yu Weiyue Wang Mark Brand Dawn Bloxwich Sean Kirmani Zhe Chen Alec Go Pablo Sprechmann Nithish Kannen Alen Carin Paramjit Sandhu Isabel Edkins Leslie Nooteboom Jai Gupta Loren Maggiore Javad Azizi Yael Pritch Pengcheng Yin Mansi Gupta Danny Tarlow Duncan Smith Desi Ivanov Mohammad Babaeizadeh Ankita Goel Satish Kambala Grace Chu Matej Kastelic Michelle Liu Hagen Soltau Austin Stone Shivani Agrawal Min Kim Kedar Soparkar Srinivas Tadepalli Oskar Bunyan Rachel Soh Arvind Kannan DY Kim Blake JianHang Chen Afief Halumi Sudeshna Roy Yulong Wang Olcan Sercinoglu Gena Gibson Sijal Bhatnagar Motoki Sano Daniel von Dincklage Qingchun Ren Blagoj Mitrevski Mirek Olšák Jennifer She Carl Doersch Jilei (Jerry) Wang Bingyuan Liu Qijun Tan Tamar Yakar Tris Warkentin Alex Ramirez Carl Lebsack Josh Dillon Rajiv Mathews Tom Cobley Zelin Wu Zhuoyuan Chen Jon Simon Swaroop Nath Tara Sainath Alexei Bendebury Ryan Julian Bharath Mankalale Daria Ćurko Paulo Zacchello Adam R. Brown Kiranbir Sodhia Heidi Howard Sergi Caelles Abhinav Gupta Gareth Evans Anna Bulanova Lesley Katzen Roman Goldenberg Anton Tsitsulin Joe Stanton Benoit Schillings Vitaly Kovalev Corey Fry Rushin Shah Kuo Lin Shyam Upadhyay Cheng Li Soroush Radpour Marcello Maggioni Jing Xiong Lukas Haas Jenny Brennan Aishwarya Kamath Nikolay Savinov Arsha Nagrani Trevor Yacovone Ryan Kappedal Kostas Andriopoulos Li Lao YaGuang Li Grigory Rozhdestvenskiy Kazuma Hashimoto Andrew Audibert Sophia Austin Daniel Rodriguez Anian Ruoss Garrett Honke Deep Karkhanis Xi Xiong Qing Wei James Huang Zhaoqi Leng Vittal Premachandran Stan Bileschi Georgios Evangelopoulos Thomas Mensink Jay Pavagadhi Denis Teplyashin Paul Chang Linting Xue Garrett Tanzer Sally Goldman Kaushal Patel Shixin Li Jeremy Wiesner Ivy Zheng Ian Stewart-Binks Jie Han Zhi Li Liangchen Luo Karel Lenc Mario Lučić Fuzhao Xue Ryan Mullins Alexey Guseynov Chung-Ching Chang Isaac Galatzer-Levy Adam Zhang Garrett Bingham Grace Hu Ale Hartman Yue Ma Jordan Griffith Alex Irpan Carey Radebaugh Summer Yue Lijie Fan Victor Ungureanu Christina Sorokin Hannah Teufel Peiran Li Rohan Anil Dimitris Paparas Todd Wang Chu-Cheng Lin Hui Peng Megan Shum Goran Petrovic Demetra Brady Richard Nguyen Klaus Macherey Zhihao Li Harman Singh Madhavi Yenugula Mariko Iinuma Xinyi Chen Kavya Kopparapu Alexey Stern Shachi Dave Chandu Thekkath Florence Perot Anurag Kumar Fangda Li Yang Xiao Matthew Bilotti Mohammad Hossein Bateni Isaac Noble Lisa Lee Amelio Vázquez-Reina Julian Salazar Xiaomeng Yang Boyu Wang Ela Gruzewska Anand Rao Sindhu Raghuram Zheng Xu Eyal Ben-David Jieru Mei Sid Dalmia Zhaoyi Zhang Yuchen Liu Gagan Bansal Helena Pankov Steven Schwarcz Andrea Burns Christine Chan Sumit Sanghai Ricky Liang Ethan Liang Antoine He Amy Stuart Arun Narayanan Yukun Zhu Christian Frank Bahar Fatemi Amit Sabne Oran Lang Indro Bhattacharya Shane Settle Maria Wang Brendan McMahan Andrea Tacchetti Livio Baldini Soares Majid Hadian Serkan Cabi Timothy Chung Nikita Putikhin Gang Li Jeremy Chen Austin Tarango Henryk Michalewski Mehran Kazemi Hussain Masoom Hila Sheftel Rakesh Shivanna Archita Vadali Ramona Comanescu Doug Reid Joss Moore Arvind Neelakantan Michaël Sander Jonathan Herzig Aviv Rosenberg Mostafa Dehghani JD Choi Michael Fink Reid Hayes Eric Ge Shitao Weng Chia-Hua Ho John Karro Kalpesh Krishna Lam Nguyen Thiet Amy Skerry-Ryan Daniel Eppens Marco Andreetto Navin Sarma Silvano Bonacina Burcu Karagol Ayan Megha Nawhal Zhihao Shan Mike Dusenberry Shantanu Thakoor Sagar Gubbi Duc Dung Nguyen Reut Tsarfaty Samuel Albanie Jovana Mitrović Meet Gandhi Bo-Juen Chen Alessandro Epasto Georgi Stephanov Ye Jin Samuel Gehman Aida Amini Jack Weber Feryal Behbahani Shawn Xu Miltos Allamanis Xi Chen Myle Ott Claire Sha Michal Jastrzebski Hang Qi David Greene Xinyi Wu Abodunrinwa Toki Daniel Vlasic Jane Shapiro Ragha Kotikalapudi Zhe Shen Takaaki Saeki Sirui Xie Albin Cassirer Shikhar Bharadwaj Tatsuya Kiyono Srinadh Bhojanapalli Elan Rosenfeld Sam Ritter Jieming Mao João Gabriel Oliveira Zoltan Egyed Bernd Bandemer Emilio Parisotto Keisuke Kinoshita Juliette Pluto Petros Maniatis Steve Li Yaohui Guo Golnaz Ghiasi Jean Tarbouriech Srimon Chatterjee Julie Jin Katrina (Xinyi) Xu Jennimaria Palomaki Séb Arnold Madhavi Sewak Federico Piccinini Mohit Sharma Ben Albrecht Sean Purser-haskell Ashwin Vaswani Chongyan Chen Matheus Wisniewski Qin Cao John Aslanides Nguyet Minh Phu Maximilian Sieb Lauren Agubuzu Anne Zheng Daniel Sohn Marco Selvi Anders Andreassen Krishan Subudhi Prem Eruvbetine Oliver Woodman Tomas Mery Sebastian Krause Xiaoqi Ren Xiao Ma Jincheng Luo Dawn Chen Wei Fan Henry Griffiths Christian Schuler Alice Li Shujian Zhang Jean-Michel Sarr Shixin Luo Riccardo Patana Matthew Watson Dani Naboulsi Michael Collins Sailesh Sidhwani Emiel Hoogeboom Sharon Silver Emily Caveness Xiaokai Zhao Mikel Rodriguez Maxine Deines Libin Bai Patrick Griffin Marco Tagliasacchi Emily Xue Spandana Raj Babbula Bo Pang Nan Ding Gloria Shen Elijah Peake Remi Crocker Shubha Srinivas Raghvendra Danny Swisher Woohyun Han Richa Singh Ling Wu Vladimir Pchelin Tsendsuren Munkhdalai Dana Alon Geoff Bacon Efren Robles Jannis Bulian Melvin Johnson George Powell Felipe Tiengo Ferreira Yaoyiran Li Frederik Benzing Mihajlo Velimirović Hubert Soyer William Kong Tony (Tuấn) Nguyễn Zhen Yang Jeremiah Liu Joost van Amersfoort Daniel Gillick Baochen Sun Nathalie Rauschmayr Katie Zhang Serena Zhan Tao Zhou Alexey Frolov Chengrun Yang Denis Vnukov Louis Rouillard Hongji Li Amol Mandhane Nova Fallen Rajesh Venkataraman Clara Huiyi Hu Jennifer Brennan Jenny Lee Jerry Chang Martin Sundermeyer Zhufeng Pan Rosemary Ke Simon Tong Alex Fabrikant William Bono Jindong Gu Ryan Foley Yiran Mao Manolis Delakis Dhruva Bhaswar Roy Frostig Nick Li Avital Zipori Cath Hope Olga Kozlova Swaroop Mishra Josip Djolonga Craig Schiff Majd Al Merey Eleftheria Briakou Peter Morgan Andy Wan Avinatan Hassidim RJ Skerry-Ryan Kuntal Sengupta Mary Jasarevic Praveen Kallakuri Paige Kunkle Hannah Brennan Tom Lieber Hassan Mansoor Julian Walker Bing Zhang Annie Xie Goran Žužić Adaeze Chukwuka Alex Druinsky Donghyun Cho Rui Yao Ferjad Naeem Shiraz Butt Eunyoung Kim Zhipeng Jia Mandy Jordan Adam Lelkes Mark Kurzeja Sophie Wang James Zhao Andrew Over Abhishek Chakladar Marcel Prasetya Neha Jha Sriram Ganapathy Yale Cong Prakash Shroff Carl Saroufim Sobhan Miryoosefi Mohamed Hammad Tajwar Nasir Weijuan Xi Yang Gao Young Maeng Ben Hora Chin-Yi Cheng Parisa Haghani Yoad Lewenberg Caden Lu Martin Matysiak Naina Raisinghani Huiyu Wang Lexi Baugher Rahul Sukthankar Minh Giang John Schultz Noah Fiedel Minmin Chen Cheng-Chun Lee Tapomay Dey Hao Zheng Shachi Paul Celine Smith Andy Ly Yicheng Wang Rishabh Bansal Bartek Perz Susanna Ricco Stasha Blank Vaishakh Keshava Deepak Sharma Marvin Chow Kunal Lad Komal Jalan Simon Osindero Craig Swanson Jacob Scott Anastasija Ilić Xiaowei Li Siddhartha Reddy Jonnalagadda Afzal Shama Soudagar Yan Xiong Bat-Orgil Batsaikhan Daniel Jarrett Naveen Kumar Maulik Shah Matt Lawlor Austin Waters Mark Graham Rhys May Sabela Ramos Sandra Lefdal Zeynep Cankara Nacho Cano Brendan O'Donoghue Jed Borovik Frederick Liu Jordan Grimstad Mahmoud Alnahlawi Katerina Tsihlas Tom Hudson Nikolai Grigorev Yiling Jia Terry Huang Tobenna Peter Igwe Sergei Lebedev Xiaodan Tang Igor Krivokon Frankie Garcia Melissa Tan Eric Jia Peter Stys Shikhar Vashishth Yu Liang Balaji Venkatraman Chenjie Gu Anastasios Kementsietsidis Chen Zhu Junehyuk Jung Yunfei Bai Mohammad Javad Hosseini Faruk Ahmed Aditya Gupta Xin Yuan Shereen Ashraf Shitij Nigam Gautam Vasudevan Pranjal Awasthi Adi Mayrav Gilady Zelda Mariet Ramy Eskander Haiguang Li Hexiang Hu Guillermo Garrido Philippe Schlattner George Zhang Rohun Saxena Petar Dević Kritika Muralidharan Ashwin Murthy Yiqian Zhou Min Choi Arissa Wongpanich Zhengdong Wang Premal Shah Yuntao Xu Yiling Huang Stephen Spencer Alice Chen James Cohan Junjie Wang Jonathan Tompson Junru Wu Ruba Haroun Haiqiong Li Blanca Huergo Fan Yang Tongxin Yin James Wendt Michael Bendersky Rahma Chaabouni Javier Snaider Johan Ferret Abhishek Jindal Tara Thompson Andrew Xue Will Bishop Shubham Milind Phal Archit Sharma Yunhsuan Sung Prabakar Radhakrishnan Mo Shomrat Reeve Ingle Roopali Vij Justin Gilmer Mihai Dorin Istin Sam Sobell Yang Lu Emily Nottage Dorsa Sadigh Jeremiah Willcock Tingnan Zhang Steve Xu Sasha Brown Katherine Lee Gary Wang Yun Zhu Yi Tay Cheolmin Kim Audrey Gutierrez Abhanshu Sharma Yongqin Xian Sungyong Seo Claire Cui Elena Pochernina Cip Baetu Krzysztof Jastrzębski Mimi Ly Mohamed Elhawaty Dan Suh Eren Sezener Pidong Wang Nancy Yuen George Tucker Jiahao Cai Zuguang Yang Cindy Wang Alex Muzio Hai Qian Jae Yoo Derek Lockhart Kevin R. McKee Mandy Guo Malika Mehrotra Artur Mendonça Sanket Vaibhav Mehta Sherry Ben Chetan Tekur Jiaqi Mu Muye Zhu Victoria Krakovna Hongrae Lee AJ Maschinot Sébastien Cevey HyunJeong Choe Aijun Bai Hansa Srinivasan Derek Gasaway Nick Young Patrick Siegler Dan Holtmann-Rice Vihari Piratla Kate Baumli Roey Yogev Alex Hofer Hado van Hasselt Svetlana Grant Yuri Chervonyi David Silver Andrew Hogue Ayushi Agarwal Kathie Wang Preeti Singh Four Flynn Josh Lipschultz Robert David Lizzetth Bellot Yao-Yuan Yang Long Le Filippo Graziano Kate Olszewska Kevin Hui Akanksha Maurya Nikos Parotsidis Weijie Chen Tayo Oguntebi Joe Kelley Anirudh Baddepudi Johannes Mauerer Gregory Shaw Alex Siegman Lin Yang Shravya Shetty Subhrajit Roy Yunting Song Wojciech Stokowiec Ryan Burnell Omkar Savant Robert Busa-Fekete Jin Miao Samrat Ghosh Liam MacDermed Phillip Lippe Mikhail Dektiarev Zach Behrman Fabian Mentzer Kelvin Nguyen Meng Wei Siddharth Verma Chris Knutsen Sudeep Dasari Zhipeng Yan Petr Mitrichev Xingyu Wang Virat Shejwalkar Jacob Austin Srinivas Sunkara Navneet Potti Yan Virin Christian Wright Gaël Liu Oriana Riva Etienne Pot Greg Kochanski Quoc Le Gargi Balasubramaniam Arka Dhar Yuguo Liao Adam Bloniarz Divyansh Shukla Elizabeth Cole Jong Lee Sheng Zhang Sushant Kafle Siddharth Vashishtha Parsa Mahmoudieh Grace Chen Raphael Hoffmann Pranesh Srinivasan Agustin Dal Lago Yoav Ben Shalom Zi Wang Michael Elabd Anuj Sharma Junhyuk Oh Suraj Kothawade Maigo Le Marianne Monteiro Shentao Yang Kaiz Alarakyia Robert Geirhos Diana Mincu Håvard Garnes Hayato Kobayashi Soroosh Mariooryad Kacper Krasowiak Zhixin (Lucas) Lai Shibl Mourad Mingqiu Wang Fan Bu Ophir Aharoni Guanjie Chen Abhimanyu Goyal Vadim Zubov Ankur Bapna Elahe Dabir Nisarg Kothari Kay Lamerigts Nicola De Cao Jeremy Shar Christopher Yew Nitish Kulkarni Dre Mahaarachchi Mandar Joshi Zhenhai Zhu Jared Lichtarge Yichao Zhou Hannah Muckenhirn Vittorio Selo Oriol Vinyals Peter Chen Anthony Brohan Vaibhav Mehta Sarah Cogan Ruth Wang Ty Geri Wei-Jen Ko Wei Chen Fabio Viola Keshav Shivam Lisa Wang Madeleine Clare Elish Raluca Ada Popa Sébastien Pereira Jianqiao Liu Raphael Koster Donnie Kim Gufeng Zhang Sayna Ebrahimi Partha Talukdar Yanyan Zheng Petra Poklukar Ales Mikhalap Dale Johnson Anitha Vijayakumar Mark Omernick Matt Dibb Ayush Dubey Qiong Hu Apurv Suman Vaibhav Aggarwal Ilya Kornakov Fei Xia Wing Lowe Alexey Kolganov Ted Xiao Vitaly Nikolaev Steven Hemingray Bonnie Li Joana Iljazi Mikołaj Rybiński Ballie Sandhu Peggy Lu Thang Luong Rodolphe Jenatton Vineetha Govindaraj Hui (Elena) Li Gabriel Dulac-Arnold Wonpyo Park Henry Wang Abhinit Modi Jean Pouget-Abadie Kristina Greller Rahul Gupta Robert Berry Prajit Ramachandran Jinyu Xie Liam McCafferty Jianling Wang Kilol Gupta Hyeontaek Lim Blaž Bratanič Andy Brock Ilia Akolzin Jim Sproch Dan Karliner Duhyeon Kim Adrian Goedeckemeyer Noam Shazeer Cordelia Schmid Daniele Calandriello Parul Bhatia Krzysztof Choromanski Ceslee Montgomery Dheeru Dua Ana Ramalho Helen King Yue Gao Lynn Nguyen David Lindner Divya Pitta Oleaser Johnson Khalid Salama Diego Ardila Michael Han Erin Farnese Seth Odoom Ziyue Wang Xiangzhuo Ding Norman Rink Ray Smith Harshal Tushar Lehri Eden Cohen Neera Vats Tong He Parthasarathy Gopavarapu Adam Paszke Miteyan Patel Wouter Van Gansbeke Lucia Loher Luis Castro Maria Voitovich Tamara von Glehn Nelson George Simon Niklaus Zach Eaton-Rosen Nemanja Rakićević Erik Jue Sagi Perel Carrie Zhang Yuval Bahat Angéline Pouget Zhi Xing Fantine Huot Ashish Shenoy Taylor Bos Vincent Coriou Bryan Richter Natasha Noy Yaqing Wang Santiago Ontanon Siyang Qin Gleb Makarchuk Demis Hassabis Zhuowan Li Mandar Sharma Kumaran Venkatesan Iurii Kemaev Roxanne Daniel Shiyu Huang Saloni Shah Octavio Ponce Warren (Weilun) Chen Manaal Faruqui Jialin Wu Slavica Andačić Szabolcs Payrits Daniel McDuff Tom Hume Yuan Cao MH Tessler Qingze Wang Yinan Wang Ivor Rendulic Eirikur Agustsson Matthew Johnson Tanya Lando Andrew Howard Sri Gayatri Sundara Padmanabhan Mayank Daswani Andrea Banino Michael Kilgore Jonathan Heek Ziwei Ji Alvaro Caceres Conglong Li Nora Kassner Alexey Vlaskin Zeyu Liu Alex Grills Yanhan Hou Roykrong Sukkerd Gowoon Cheon Nishita Shetty Larisa Markeeva Piotr Stanczyk Tejas Iyer Yuan Gong Shawn Gao Keerthana Gopalakrishnan Tim Blyth Malcolm Reynolds Avishkar Bhoopchand Misha Bilenko Dero Gharibian Vicky Zayats Aleksandra Faust Abhinav Singh Min Ma Hongyang Jiao Sudheendra Vijayanarasimhan Lora Aroyo Vikas Yadav Sarah Chakera Ashwin Kakarla Vilobh Meshram Karol Gregor Gabriela Botea Evan Senter Dawei Jia Geza Kovacs Neha Sharma Sebastien Baur Kai Kang Yifan He Lin Zhuo Marija Kostelac Itay Laish Songyou Peng Louis O'Bryan Daniel Kasenberg Girish Ramchandra Rao Edouard Leurent Biao Zhang Sage Stevens Ana Salazar Ye Zhang Ivan Lobov Jake Walker Allen Porter Morgan Redshaw Han Ke Abhishek Rao Alex Lee Hoi Lam Michael Moffitt Jaeyoun Kim Siyuan Qiao Terry Koo Robert Dadashi Xinying Song Mukund Sundararajan Peng Xu Chizu Kawamoto Yan Zhong Clara Barbu Apoorv Reddy Mauro Verzetti Leon Li George Papamakarios Hanna Klimczak-Plucińska Mary Cassin Koray Kavukcuoglu Rigel Swavely Alain Vaucher Jeffrey Zhao Ross Hemsley Michael Tschannen Heming Ge Gaurav Menghani Yang Yu Natalie Ha Wei He Xiao Wu Maggie Song Rachel Sterneck Stefan Zinke Dan A. Calian Annie Marsden Alejandro Cruzado Ruiz Matteo Hessel Almog Gueta Benjamin Lee Brian Farris Manish Gupta Yunjie Li Mohammad Saleh Vedant Misra Kefan Xiao Piermaria Mendolicchio Gavin Buttimore Varvara Krayvanova Nigamaa Nayakanti Matthew Wiethoff Yash Pande Azalia Mirhoseini Ni Lao Jasmine Liu Yiqing Hua Angie Chen Yury Malkov Dmitry Kalashnikov Shubham Gupta Kartik Audhkhasi Yuexiang Zhai Sudhindra Kopalle Prateek Jain Eran Ofek Clemens Meyer Khuslen Baatarsukh Hana Strejček Jun Qian James Freedman Ricardo Figueira Michal Sokolik Olivier Bachem Raymond Lin Dia Kharrat Chris Hidey Pingmei Xu Dennis Duan Yin Li Muge Ersoy Richard Everett Kevin Cen Rebeca Santamaria-Fernandez Amir Taubenfeld Ian Mackinnon Linda Deng Polina Zablotskaia Shashank Viswanadha Shivanker Goel Damion Yates Yunxiao Deng Peter Choy Mingqing Chen Abhishek Sinha Alex Mossin Yiming Wang Arthur Szlam Susan Hao Paul Kishan Rubenstein Metin Toksoz-Exley Miranda Aperghis Yin Zhong Junwhan Ahn Michael Isard Olivier Lacombe Florian Luisier Chrysovalantis Anastasiou Yogesh Kalley Utsav Prabhu Emma Dunleavy Shaan Bijwadia Justin Mao-Jones Kelly Chen Rama Pasumarthi Emily Wood Adil Dostmohamed Nate Hurley Jiri Simsa Alicia Parrish Mantas Pajarskas Matt Harvey Ondrej Skopek Yony Kochinski Javier Rey Verena Rieser Denny Zhou Sun Jae Lee Trilok Acharya Guowang Li Joe Jiang Xiaofan Zhang Bryant Gipson Ethan Mahintorabi Marco Gelmi Nima Khajehnouri Angel Yeh Kayi Lee Loic Matthey Leslie Baker Trang Pham Han Fu Alex Pak Prakhar Gupta Cristina Vasconcelos Adam Sadovsky Brian Walker Sissie Hsiao Patrik Zochbauer Andreea Marzoca Noam Velan Junhao Zeng Gilles Baechler Danny Driess Divya Jain Yanping Huang Lizzie Tao John Maggs Nir Levine Jon Schneider Erika Gemzer Samuel Petit Shan Han Zach Fisher Dustin Zelle Courtney Biles Eugene Ie Asya Fadeeva Casper Liu Juliana Vicente Franco Adrian Collister Hao Zhang Renshen Wang Ruizhe Zhao Leandro Kieliger Kurt Shuster Rui Zhu Boqing Gong Lawrence Chan Ruoxi Sun Sujoy Basu Roland Zimmermann Jamie Hayes Abhishek Bapna Jasper Snoek Weel Yang Puranjay Datta Jad Al Abdallah Kevin Kilgour Lu Li SQ Mah Yennie Jun Morgane Rivière Abhijit Karmarkar Tammo Spalink Tao Huang Lucas Gonzalez Duc-Hieu Tran Averi Nowak John Palowitch Martin Chadwick Ellie Talius Harsh Mehta Thibault Sellam Philipp Fränken Massimo Nicosia Kyle He Aditya Kini David Amos Sugato Basu Harrison Jobe Eleni Shaw Qiantong Xu Colin Evans Daisuke Ikeda Chaochao Yan Larry Jin Lun Wang Sachin Yadav Ilia Labzovsky Ramesh Sampath Ada Ma Candice Schumann Aditya Siddhant Rohin Shah John Youssef Rishabh Agarwal Natalie Dabney Alessio Tonioni Moran Ambar Jing Li Isabelle Guyon Benny Li David Soergel Boya Fang Georgi Karadzhov Cristian Udrescu Trieu Trinh Vikas Raunak Seb Noury Dee Guo Sonal Gupta Mara Finkelstein Denis Petek Lihao Liang Greg Billock Pei Sun David Wood Yiwen Song Xiaobin Yu Tatiana Matejovicova Regev Cohen Kalyan Andra David D'Ambrosio Zhiwei Deng Vincent Nallatamby Ebrahim Songhori Rumen Dangovski Andrew Lampinen Pankil Botadra Adam Hillier Jiawei Cao Nagabhushan Baddi Adhi Kuncoro Toshihiro Yoshino Ankit Bhagatwala Marc'aurelio Ranzato Rylan Schaeffer Tianlin Liu Shuai Ye Obaid Sarvana John Nham Chenkai Kuang Isabel Gao Jinoo Baek Shubham Mittal Ayzaan Wahid Anita Gergely Bin Ni Josh Feldman Carrie Muir Pascal Lamblin Wolfgang Macherey Ethan Dyer Logan Kilpatrick Víctor Campos Mukul Bhutani Stanislav Fort Yanif Ahmad Aliaksei Severyn Kleopatra Chatziprimou Oleksandr Ferludin Mason Dimarco Aditya Kusupati Joe Heyward Dan Bahir Kevin Villela Katie Millican Dror Marcus Sanaz Bahargam Caglar Unlu Nicholas Roth Zichuan Wei Siddharth Gopal Deepanway Ghoshal Edward Lee Sharon Lin Jennie Lees Dayeong Lee Anahita Hosseini Connie Fan Seth Neel Marcus Wu Yasemin Altun Honglong Cai Enrique Piqueras Josh Woodward Alessandro Bissacco Salem Haykal Mahyar Bordbar Prasha Sundaram Sarah Hodkinson Daniel Toyama George Polovets Austin Myers Anu Sinha Tomer Levinboim Kashyap Krishnakumar Rachita Chhaparia Tatiana Sholokhova Nitesh Bharadwaj Gundavarapu Ganesh Jawahar Haroon Qureshi Jieru Hu Nikola Momchev Matthew Rahtz Renjie Wu Aishwarya P S Kedar Dhamdhere Meiqi Guo Umang Gupta Ali Eslami Mariano Schain Michiel Blokzijl David Welling Dave Orr Levent Bolelli Nicolas Perez-Nieves Mikhail Sirotenko Aman Prasad Arjun Kar Borja De Balle Pigem Tayfun Terzi Gellért Weisz Dipankar Ghosh Aditi Mavalankar Dhruv Madeka Kaspar Daugaard Hartwig Adam Viraj Shah Dana Berman Maggie Tran Steven Baker Ewa Andrejczuk Grishma Chole Ganna Raboshchuk Mahdi Mirzazadeh Thais Kagohara Shimu Wu Christian Schallhart Bernett Orlando Chen Wang Alban Rrustemi Hao Xiong Hao Liu Arpi Vezer Nolan Ramsden Shuo-yiin Chang Sidharth Mudgal Yan Li Nino Vieillard Yedid Hoshen Farooq Ahmad Ambrose Slone Amy Hua Natan Potikha Mirko Rossini Jon Stritar Sushant Prakash Zifeng Wang Xuanyi Dong Alireza Nazari Efrat Nehoran Kaan Tekelioglu Yinxiao Li Kartikeya Badola Tom Funkhouser Yuanzhen Li Varun Yerram Ramya Ganeshan Daniel Formoso Karol Langner Tian Shi Huijian Li Yumeya Yamamori Amayika Panda Alaa Saade Angelo Scorza Scarpati Chris Breaux CJ Carey Zongwei Zhou Cho-Jui Hsieh Sophie Bridgers Alena Butryna Nishesh Gupta Vaibhav Tulsyan Sanghyun Woo Evgenii Eltyshev Will Grathwohl Chanel Parks Seth Benjamin Rina Panigrahy Shenil Dodhia Daniel De Freitas Chris Sauer Will Song Ferran Alet Jackson Tolins Cosmin Paduraru Xingyi Zhou Brian Albert Zizhao Zhang Lei Shu Mudit Bansal Sarah Nguyen Amir Globerson Owen Xiao James Manyika Tom Hennigan Rong Rong Josip Matak Anton Bakalov Ankur Sharma Danila Sinopalnikov Andrew Pierson Stephen Roller Geoff Brown Mingcen Gao Toshiyuki Fukuzawa Amin Ghafouri Kenny Vassigh Iain Barr Zhicheng Wang Anna Korsun Rajesh Jayaram Lijie Ren Tim Zaman Samira Khan Yana Lunts Dan Deutsch Dave Uthus Nitzan Katz Masha Samsikova Amr Khalifa Nikhil Sethi Jiao Sun Luming Tang Uri Alon Xianghong Luo Dian Yu Abhishek Nayyar Bryce Petrini Will Truong Vincent Hellendoorn Nikolai Chinaev Chris Alberti Wei Wang Jingcao Hu Vahab Mirrokni Ananth Balashankar Avia Aharon Aahil Mehta Ahmet Iscen Joseph Kready Lucas Manning Anhad Mohananey Yuankai Chen Anshuman Tripathi Allen Wu Igor Petrovski Dawsen Hwang Martin Baeuml Shreyas Chandrakaladharan Yuan Liu Rey Coaguila Maxwell Chen Sally Ma Pouya Tafti Susheel Tatineni Terry Spitz Jiayu Ye Paul Vicol Mihaela Rosca Adrià Puigdomènech Zohar Yahav Sanjay Ghemawat Hanzhao Lin Phoebe Kirk Zaid Nabulsi Sergey Brin Bernd Bohnet Ken Caluwaerts Aditya Srikanth Veerubhotla Dan Zheng Zihang Dai Petre Petrov Yichong Xu Ramin Mehran Zhuo Xu Luisa Zintgraf Jiho Choi Spurthi Amba Hombaiah Romal Thoppilan Sashank Reddi Lukasz Lew Li Li Kellie Webster KP Sawhney Lampros Lamprou Siamak Shakeri Mayank Lunayach Jianmin Chen Sumit Bagri Alex Salcianu Ying Chen Yani Donchev Charlotte Magister Signe Nørly Vitor Rodrigues Tomas Izo Hila Noga Joe Zou Thomas Köppe Wenxuan Zhou Kenton Lee Xiangzhu Long Danielle Eisenbud Anthony Chen Connor Schenck Chi Ming To Peilin Zhong Emanuel Taropa Minh Truong Omer Levy Danilo Martins Zhiyuan Zhang Christopher Semturs Kelvin Zhang Alex Yakubovich Pol Moreno Lara McConnaughey Di Lu Sam Redmond Lotte Weerts Yonatan Bitton Tiziana Refice Nicolas Lacasse Arthur Conmy Corentin Tallec Julian Odell Hannah Forbes-Pollard Arkadiusz Socala Jonathan Hoech Pushmeet Kohli Alanna Walton Rui Wang Mikita Sazanovich Kexin Zhu Andrei Kapishnikov Rich Galt Matthew Denton Ben Murdoch Caitlin Sikora Kareem Mohamed Wei Wei Uri First Tim McConnell Luis C. Cobo James Qin Thi Avrahami Daniel Balle Yu Watanabe Annie Louis Adam Kraft Setareh Ariafar Yiming Gu Eugénie Rives Charles Yoon Andrei Rusu James Cobon-Kerr Chris Hahn Jiaming Luo Yuvein (Yonghao) Zhu Niharika Ahuja Rodrigo Benenson Raphaël Lopez Kaufman Honglin Yu Lloyd Hightower Junlin Zhang Darren Ni Lisa Anne Hendricks Gabby Wang Gal Yona Lalit Jain Pablo Barrio Surya Bhupatiraju Siva Velusamy Allan Dafoe Sebastian Riedel Tara Thomas Zhe Yuan Mathias Bellaiche Sheena Panthaplackel Klemen Kloboves Sarthak Jauhari Canfer Akbulut Todor Davchev Evgeny Gladchenko David Madras Aleksandr Chuklin Tyrone Hill Quan Yuan Mukundan Madhavan Luke Leonhard Dylan Scandinaro Qihang Chen Ning Niu Arthur Douillard Bogdan Damoc Yasumasa Onoe Fabian Pedregosa Fred Bertsch Chas Leichner Joseph Pagadora Jonathan Malmaud Sameera Ponda Andy Twigg Oleksii Duzhyi Jingwei Shen Miaosen Wang Roopal Garg Jing Chen Utku Evci Jonathan Lee Leon Liu Koji Kojima Masa Yamaguchi Arunkumar Rajendran AJ Piergiovanni Vinodh Kumar Rajendran Marco Fornoni Gabriel Ibagon Harry Ragan Sadh MNM Khan John Blitzer Andrew Bunner Guan Sun Takahiro Kosakai Scott Lundberg Ndidi Elue Kelvin Guu SK Park Jane Park Arunachalam Narayanaswamy Chengda Wu Jayaram Mudigonda Trevor Cohn Hairong Mu Ravi Kumar Laura Graesser Yichi Zhang Richard Killam Vincent Zhuang Mai Giménez Wael Al Jishi Ruy Ley-Wild Alex Zhai Kazuki Osawa Diego Cedillo Jialu Liu Mayank Upadhyay Marcin Sieniek Roshan Sharma Tom Paine Anelia Angelova Sravanti Addepalli Carolina Parada Kingshuk Majumder Avery Lamp Sanjiv Kumar Xiang Deng Artiom Myaskovsky Tea Sabolić Jeffrey Dudek Sarah York Félix de Chaumont Quitry Jiazhong Nie Dee Cattle Alok Gunjan Bilal Piot Waleed Khawaja Seojin Bang Simon Wang Siavash Khodadadeh Raghavender R Praynaa Rawlani Richard Powell Kevin Lee Johannes Griesser GS Oh Cesar Magalhaes Yujia Li Simon Tokumine Hadas Natalie Vogel Dennis Hsu Arturo BC Disha Jindal Matan Cohen Zi Yang Junwei Yuan Dario de Cesare Tony Bruguier Jun Xu Monica Roy Alon Jacovi Dan Belov Rahul Arya Phoenix Meadowlark Shlomi Cohen-Ganor Wenting Ye Patrick Morris-Suzuki Praseem Banzal Gan Song Pranavaraj Ponnuramu Fred Zhang George Scrivener Salah Zaiem Alif Raditya Rochman Kehang Han Badih Ghazi Kate Lee Shahar Drath Daniel Suo Antonious Girgis Pradeep Shenoy Duy Nguyen Douglas Eck Somit Gupta Le Yan Joao Carreira Anmol Gulati Ruoxin Sang Daniil Mirylenka Emma Cooney Edward Chou Mingyang Ling Cindy Fan Ben Coleman Guilherme Tubone Ravin Kumar Jason Baldridge Felix Hernandez-Campos Angeliki Lazaridou James Besley Itay Yona Neslihan Bulut Quentin Wellens AJ Pierigiovanni Jasmine George Richard Green Pu Han Connie Tao Geoff Clark Chong You Abbas Abdolmaleki Justin Fu Tongzhou Chen Ashwin Chaugule Angad Chandorkar Altaf Rahman Will Thompson Penporn Koanantakool Mike Bernico Jie Ren Andrey Vlasov Sergei Vassilvitskii Maciej Kula Yizhong Liang Dahun Kim Yangsibo Huang Chengxi Ye Dmitry Lepikhin Wesley Helmholz The development of Gemini is a large-scale collaborative effort involving over 3000 individuals across Google, including researchers, engineers, and operations staff. These individuals contributed their hard work and expertise across diverse areas, from foundational research and the development of model architecture, data, training, and infrastructure, through to evaluation and ensuring safety and security. We gratefully acknowledge the dedication and hard work of each contributor in making Gemini a reality. The order of contributors in the above list is random. We are also grateful to the Google-independent developer Joel Zhang for his work on Gemini Plays Pokémon, and for sharing with us the design of his set-up. 8 Appendix 8.1 Evaluation additional details Please see a description of the benchmarks considered, along with details of how scores in the main text were obtained in Table 11. Benchmark Description Details LiveCodeBench Code generation in Python (Jain et al., 2024). Results are taken from https://livecodebench.github.io/leaderboard.html (1/1/2025 - 5/1/2025 in the UI) or, where not available, run internally by us. For Section 2.5 and Figure 3 and 4, results are calculated on the version of the eval corresponding to 10/05/2024 - 01/04/2025 in the UI, and are based on internal results. Aider Polyglot Code editing in C++, Go, Java, JavaScript Python and Rust (Gauthier, 2025). See https://aider.chat/2024/12/21/polyglot.html#the-polyglot-benchmark for a full description of this task. We report results on the “diff” or “diff-fenced” edit format (see https://aider.chat/docs/more/edit-formats.html for a description of the different formats). The score reported are the pass rate average of 3 trials. Numbers come from https://aider.chat/docs/leaderboards/ SWE-bench Verified Agentic coding: evaluates AI agents on real-world programming tasks from GitHub (Jimenez et al., 2024; Chowdhury et al., 2024). Gemini uses an internal agentic harness equipped with tools to navigate the repo, edit files, and test the code. We report scores for two modes: performance of a single agentic trace (“single attempt”), and performance of a scaffold that samples multiple agentic traces and re-reranks them before evaluation using Gemini's own judgement (“multiple attempts”). All evaluations are done with temperature=1, topp=0.99, topk=1024. GPQA (diamond) Challenging dataset of questions written by domain experts in biology, physics, and chemistry (Rein et al., 2024). Humanity's Last Exam Challenging dataset of questions written by domain experts in a wide range of disciplines, including mathematics, physics, chemistry, biology and computer science (Phan et al., 2025). No tool use variant. Reported results are from https://scale.com/leaderboard/humanitys_last_exam. For DeepSeek they are taken from https://scale.com/leaderboard/humanitys_last_exam_text_only (leaderboard for performance on the text-only questions) and in the case of the Gemini 2.0 models, these results are on an earlier HLE dataset, obtained from https://scale.com/leaderboard/humanitys_last_exam_preview (indicated with a †\dagger in Table 3) SimpleQA World knowledge factuality with no search enabled (Wei et al., 2024). F1 scores are obtained from https://github.com/openai/simple-evals and, where not available, run internally by us. FACTS Grounding Ability to provide factually correct responses given documents and diverse user requests. (Jacovi et al., 2025) Results are sourced from https://www.kaggle.com/benchmarks/google/facts-grounding Global MMLU (Lite) MMLU translated by human translators into 15 languages. (Singh et al., 2024) The lite version includes 200 Culturally Sensitive and 200 Culturally Agnostic samples per language, see https://huggingface.co/datasets/CohereLabs/Global-MMLU-Lite ECLeKTic A closed-book QA dataset that evaluates cross-lingual knowledge transfer (Goldman et al., 2025). AIME 2025 Performance on 30 questions from American Invitational Mathematics Examination from 2025 (Balunović et al., 2025). Results are sourced from https://matharena.ai/. HiddenMath-Hard Competition-level math problems, Held out dataset AIME/AMC-like, crafted by experts and not leaked on the web. LOFT (hard retrieval subset) Long context multi-hop and multi-needle retrieval evaluation of 300 queries (Lee et al., 2024). We report the results on two variants: an up to 128K average context length variant to ensure they can be comparable with other models and a pointwise value for 1M context window to show the capability of the model at full length. MRCR-V2 (8-needle) MRCR-V2 is a significantly harder instance of the MRCR family of long-context evaluations (Vodrahalli et al., 2024). Compared to MRCR-V1, we increase the nesting of the dictionary size to depth 3 rather than 2 by including a style parameter (for instance, an example key might be “write a poem about penguins in an archaic style”, rather than just “write a poem about penguins”). The methodology has changed compared to previously published results: we focus on a harder, 8-needle version (compared to the 4-needle version used before). We report the results on two variants: an up to 128K average context length variant to ensure they can be comparable with other models and a pointwise value for 1M context window to show the capability of the model at full length. MMMU Multi-discipline college-level multimodal image understanding and reasoning problems. (Yue et al., 2024) Vibe-Eval (Reka) Image understanding evaluation, featuring particularly challenging examples. (Padlewski et al., 2024) Gemini is used as a judge. ZeroBench Challenging image understanding evaluation that requires multi-step reasoning. (Roberts et al., 2025) Gemini is used as a judge. Average over 4 runs. BetterChartQA A comprehensive chart understanding evaluation that covers 9 disjoint capability buckets. The chart images are randomly sampled from the web and QA pairs are written by professional human annotators to reflect the wide distribution of chart styles and real-world cases. (Gemini Team, 2024) Gemini is used as a judge. FLEURS Automatic speech recognition (Conneau et al., 2023). 0-shot queries to public APIs for all models. Used a subset of 53 languages (out of 102); we filtered languages for which either model responses were too incompatible to ground truth responses to be fairly scored. We use Word-Error-Rate WER (lower is better) except for four segmented languages where we aggregate Character-Error-Rates (Chinese, Japanese, Korean and Thai). CoVoST 2 Speech to text translation (Wang et al., 2020). 0-shot queries to public APIs for all models. We report BLEU scores for translating 21 languages to English. ActivityNet-QA General video understanding (Yu et al., 2019) Test subset, 0-shot. Videos were processed at 1fps and linearly subsampled to a maximum of Nf​r​a​m​e​s=1024N_{frames}=1024 frames. For GPT 4.1, we used 500 frames due to API limitations. EgoTempo Egocentric video understanding (Plizzari et al., 2025) Test subset, 0-shot. Same processing as above with Nf​r​a​m​e​s=256N_{frames}=256. Perception Test Perceptual understanding/reasoning (Patraucean et al., 2023) Test subset, 0-shot. Same processing as above with Nf​r​a​m​e​s=256N_{frames}=256. QVHighlights Moment retrieval (Lei et al., 2021) Validation subset, 4-shots. Accuracy measured with R1@0.5. Same processing as above with Nf​r​a​m​e​s=256N_{frames}=256. VideoMMMU Video knowledge acquisition (Hu et al., 2025) Test subset, 0-shot. Same processing as above with Nf​r​a​m​e​s=256N_{frames}=256. 1H-VideoQA Hour-long video understanding (Gemini Team, 2024) Test subset, 0-shot. Same processing as above with Nf​r​a​m​e​s=7200N_{frames}=7200. LVBench Long video understanding (Wang et al., 2024) Test subset, 0-shot. Same processing as above with Nf​r​a​m​e​s=1024N_{frames}=1024. VideoMME Long video understanding (Fu et al., 2025) 0-shot. Audio + visual uses the Long subset of test set, audio + visual + subtitles uses full test set. Same processing as above with Nf​r​a​m​e​s=1024N_{frames}=1024. VATEX General video captioning (Wang et al., 2019) Test subset, 4-shots. CIDEr score. Same processing as above with Nf​r​a​m​e​s=64N_{frames}=64. VATEX-ZH Chinese video captioning (Wang et al., 2019) Validation subset, 4-shots. CIDEr score. Same processing as above with Nf​r​a​m​e​s=64N_{frames}=64. YouCook2 Cap Instructional video captioning (Zhou et al., 2018) Validation subset, 4-shots. CIDEr score. Same processing as above with Nf​r​a​m​e​s=256N_{frames}=256. Minerva Complex video reasoning (Nagrani et al., 2025a) Test subset, 0-shot. Same processing as above with Nf​r​a​m​e​s=1024N_{frames}=1024. Neptune Long video understanding (Nagrani et al., 2025b) Test subset, 0-shot. Same processing as above with Nf​r​a​m​e​s=1024N_{frames}=1024. Table 11: Description of the benchmarks used, along with extra details about subsets, variants and model specifications. 8.2 Gemini Plays Pokémon Additional Details Changing the model used by the Gemini Plays Pokémon agent had a strong effect on performance, as can be seen in Figure 4.1. Figure 13: The model matters: Same agentic harness, different Gemini models. All runs have the same starter (Charmander). Note that measuring in units of hours also controls for the fact that each of 2.5 Flash's actions was significantly faster (though it requires more actual actions to achieve its goals). X marks the end of gameplay and is a lower bound on the time to complete the next milestone. Additional Harness Details Figure 14: An overview of the agent harness (Zhang, 2025). The overworld fog-of-war map automatically stores a tile once explored and labels it with a visited counter. The type of tile is recorded from RAM. The agentic tools (pathfinder, boulder_puzzle_strategist) are prompted instances of Gemini 2.5 Pro. pathfinder is used for navigation and boulder_puzzle_strategist solves boulder puzzles in the Victory Road dungeon. The Gemini Plays Pokémon agent (Zhang, 2025) receives a subset of RAM information, intended to give sufficient information to play the game, partially overlaid with a screenshot of the Game Boy screen. Gemini is prompted with a system prompt telling it that it is playing Pokémon Blue and that its goal is to beat the game, as well as descriptive information to help it understand the conventions in the translation from vision to text and a small number of general tips for gameplay. Gemini then takes actions, translated to button presses. The sequence of actions is stored in context, followed by a summary clear every 100 turns. The summaries are stored in context as well. Every 1000 turns GPP compresses the existing summaries again. Additionally, Gemini keeps track of three main goals (primary, secondary, and tertiary) as well as several additional goals (contingency plans, preparation, exploration, team composition). Every 25 turns, another prompted instance of Gemini (Guidance Gemini, or GG) observes the same context as the main Gemini and critiques performance and attempts to point out hallucinations and so on. The overworld fog-of-war map is stored in the context in XML, where coordinates which have not been seen cannot be viewed until explored. Crucially, in the system prompt, Gemini is instructed to explore. Once a tile is explored, however, the coordinate is automatically stored in the map memory and labeled with a visited counter. Tiles are also labeled by type (water, ground, cuttable, grass, spinner, etc.), and warp points to different maps are also labeled as such. Gemini also has access to two agentic tools, which are both instances of Gemini equipped with a more specialized prompt - the pathfinder tool, and the boulder_puzzle_strategist tool. In the pathfinder prompt, Gemini is prompted to mentally simulate a path-finding algorithm, which is left unspecified, and to verify that the path is valid against the map information available. In the boulder_puzzle_strategist tool, Gemini is prompted to solve special boulder puzzles that are present in Pokémon Blue in the Victory Road dungeon - these puzzles are similar to the game Sokoban - again, by mentally simulating sequences of actions that lead to solutions to the puzzle. The prompt describes the physics and the task of the boulder puzzle, as well as the desired output of solutions. The tool was added after Gemini had solved 2/4 of the puzzles in Victory Road on its own, but progress was slow on the 3rd and 4th puzzles. Additional Examples of Capabilities Long Context Agentic Tooling The model is able to identify a complex path through a maze with auto-movement only specified by direction (Rocket Hideout spinner puzzles), solve multiple shortest path problems across multiple maps with limited resources (Safari Zone), perform maze solving on mazes with large description length (Route 13), and solve complex boulder-pushing puzzles across a multi-map 3D maze (Seafoam Islands). It is perhaps even more impressive that it appears to be possible for the model to solve these problems only with textual descriptions of the problems. On the other hand, other models, like Gemini 2.5 Flash, were not able to perform similarly long pathfinding tasks, and often failed to find simpler paths. This gap highlights the superior long context reasoning capability of Gemini 2.5 Pro (as also evidenced by other evaluations). boulder_puzzle_strategist is similarly impressive. The boulder puzzles in Pokémon Blue are Sokoban-like puzzles that require the player character to maneuver boulders on to switches and through holes in order to open up a pathway through a cave with multiple levels. The puzzles can become quite complex, requiring long circuitous pathways and multi-level movement in order to solve the puzzle. With only a prompt describing boulder physics and a description of how to verify a valid path, Gemini 2.5 Pro is able to one-shot some of these complex boulder puzzles, which are required to progress through Victory Road. pathfinder and boulder_puzzle_strategist are currently the only two agentic tools that the Gemini Plays Pokémon developer has implemented. In future runs, there are plans to explore tool-creation tools where the model can create new tools with only a prompt. Since most of the prompts for pathfinder and boulder_puzzle_strategist were actually written by Gemini 2.5 Pro itself, it is quite plausible that autonomous tool creation is possible for the current 2.5 Pro model. General Reasoning Gemini 2.5 Pro is able to reason through complex game puzzles in Pokémon quite well. In this section, we present two examples. Catching a Pokémon that is quick to flee: In one of the runs, the Gemini 2.5 Pro agent was attempting to catch an Abra, and planned to use Pikachu's Thunder Wave to paralyze the Abra, simultaneously making it less likely that Abra could Teleport out of the battle while also improving the catching rate. After multiple attempts, the agent caught Abra with this strategy. Creatively escaping a softlock caused by bugs in game I/O: On the Cycling Road, the slope forces southward movement at all times unless there is an obstacle. It turns out there are two tiles on the Cycling Road that result in a softlock as a result of this behavior. In the GPP framework, button presses are limited by time delays, and in order for a player to escape those two tiles (blocked on all sides except the north), the player would have to input a sequence of button presses more quickly than the GPP framework allows. Gemini 2.5 Pro unluckily found itself in one of these two spots - luckily, it was not a softlock, because 2.5 Pro had already taught one of its party members HM02 FLY - which allows for travel to any town it has been to. FLY is not typically used as an escape mechanism (unlike the item ESCAPE ROPE and the move DIG, both of which fail in this situation). After 4 hours of trying many approaches to escape (including movement, ESCAPE ROPE, DIG, all of which are blocked), the Gemini 2.5 Pro agent came up with the idea to use FLY to escape from the softlock successfully. This reasoning action is especially impressive since this situation can never occur in an existing game - and thus, it is certain that information from training data for this behavior has not leaked into the model's knowledge base! Long Horizon Task Coherence There are several additional interesting case studies of shorter planning sequences throughout Pokémon Blue that Gemini 2.5 Pro in the GPP harness was able to solve: Training team to prepare for upcoming battles: In one run where Gemini picked Charmander, the Fire-type starter, Gemini 2.5 Pro lost to Misty, the Water-type Gym Leader, the first time. To prepare for the rematch, Gemini 2.5 Pro spent over 24 hours leveling up a Pikachu and a Bellsprout (both super-effective against Water types) by around 25 levels in total to successfully defeat Misty. Acquiring Hidden Moves (HMs) for game progression: In many parts of the game, it is necessary to first acquire an HM before game progression is possible. Two examples are HM01 CUT and HM05 FLASH. Acquiring the ability to use CUT and FLASH each require four steps: 1) obtaining the HM item itself, 2) acquiring a compatible Pokémon which can learn the move, 3) adding the compatible Pokémon to the player's team, 4) teaching the HM move to the compatible Pokémon. In many cases, each step requires many steps itself. As an example, in run 1, Gemini 2.5 Pro had to a) retrieve CUT by completing the S.S. Anne quest, b) identify a Pokémon which could learn CUT and catch it (CHOPPY the Bellsprout), c) add CHOPPY to the team and d) teach CUT. Similarly, for HM05 FLASH, Gemini 2.5 Pro had to a) first catch 10 Pokémon to fill out the Pokedex, b) backtrack to find an Aide who gives HM05 Flash, c) catch a Pokémon (ZAP the Pikachu) in Viridian Forest, use the PC to deposit a Pokémon and withdraw ZAP, d) teach HM05 FLASH to Zap. Solving the Safari Zone: The Safari Zone is another location with required HMs (both HM03 SURF and HM04 Strength). However, it has an extra constraint - it requires 500¥ to enter each time, and the player is limited to only 500 total steps in the Safari Zone. As a result, if the player is unable to reach the required items in the limited number of steps, the player loses 500¥ and is required to re-start! As a result, it is possible to essentially softlock if the player takes too many attempts to complete the Safari Zone. Solving the Safari Zone itself requires traversing across four different maps and not getting lost. Gemini 2.5 Pro was able to get both required HMs in 17 attempts in run 1, and in only 5 attempts in run 2. Finding hidden keys in dungeons: Another method of progression in Pokémon is to find hidden keys and solve complex multi-floor dungeons. In particular, in Rocket Hideout, the player must recover the LIFT KEY on the fourth basement floor (dropped after beating a specific Team Rocket Grunt) in order to unlock the elevator to find the evil Giovanni, leader of Team Rocket. In Silph Co., the player must find the CARD KEY in order to open multiple doors to find the path across eleven floors of the building to rescue the President from Giovanni. To open the seventh gym on Cinnabar Island, the player must enter the Pokémon Mansion and traverse three floors in order to find the SECRET KEY which unlocks the gym door. All of these cases require maintaining the goals over large numbers of actions and many local puzzles (like spinner puzzles in Rocket Hideout, and switch puzzles in Pokémon Mansion), in addition to maintaining the health of the Pokémon on the player's team and managing wild encounters, trainer battles, and other items. Puzzle solving over complex multi-level dungeons: The Seafoam Islands contain 5 floors involving multiple boulder puzzles which require the player to navigate mazes and push boulders through holes across multiple floors using HM04 STRENGTH in order to block fast-moving currents that prevent the player from using HM03 Surf in various locations in this difficult dungeon. As a result, the player must track information across five different maps in order to both deduce the goal (push two boulders into place in order to block a specific current) as well as engage in multi-level (effectively 3D) maze solving to find the way out. It is likely the most challenging dungeon in the game. Only the second run of GPP went through Seafoam Islands, as it is not required to progress. Additional Challenges Hallucinations and Fixations on Delusions While game knowledge can sometimes leak and be quite beneficial to the ability of the model to progress, it can also hinder the model in surprising ways due to hallucinations, delusions, and mix ups with other generations of Pokémon games. One example of this phenomenon is the TEA item. In Pokémon Red/Blue, at one point the player must purchase a drink (FRESH WATER, SODA POP, or LEMONADE) from a vending machine and hand it over to a thirsty guard, who then lets the player pass through. In Pokémon FireRed/LeafGreen, remakes of the game, you must instead bring the thirsty guard a special TEA item, which does not exist in the original game. Gemini 2.5 Pro at several points was deluded into thinking that it had to retrieve the TEA in order to progress, and as a result spent many, many hours attempting to find the TEA or to give the guard TEA. In Run 2, the model was explicitly prompted to act as a player completely new to the game, and to disregard prior knowledge about game events, item locations, and Pokémon spawn points, in order to mitigate hallucinations from model pretraining knowledge and to also attempt to perform a cleaner test of the model's ability to reason through the game. It appears to have at least partially worked - multiple hallucinations from other games have been avoided in the second run. On the flip side, this prompt may have also harmed the model's ability to utilize information from its common knowledge about the game, hindering overall performance in a few critical places. Fixations on delusions due to goal-setting and also due to the Guidance Gemini instance are not an uncommon occurrence in watching Gemini Plays Pokémon - the TEA incidence is hardly the only example of this behavior. An especially egregious form of this issue can take place with “context poisoning” - where many parts of the context (goals, summary) are “poisoned” with misinformation about the game state, which can often take a very long time to undo. As a result, the model can become fixated on achieving impossible or irrelevant goals. This failure mode is also highly related to the looping issue mentioned above. These delusions, though obviously nonsensical to a human (“Let me try to go through the entrance to a house and back out again. Then, hopefully the guard who is blocking the entrance might move.”), by virtue of poisoning the context in many places, can lead the model to ignore common sense and repeat the same incorrect statement. Context poisoning can also lead to strategies like the “black-out” strategy (cause all Pokémon in the party to faint, “blacking out” and teleporting to the nearest Pokémon Center and losing half your money, instead of attempting to leave). Topological Traps in Thinking Patterns One recurring pattern in particularly-difficult-to-solve puzzles and mazes for Gemini 2.5 Pro consists of a “topological trap” - the topology of the reasoning graph required to solve the maze or puzzle has a distinctive shape. Namely, the desired objective appears to be nearby and easily reachable (an “attractor”), but the correct solution requires taking a detour in order to arrive at the correct solution. We observed this phenomenon in multiple parts of the game. In the spinner puzzle on B3F of Rocket Hideout (Zerokid, 2024), the map positions both an item and the correct staircase to the south, but they are only accessible by going the long way around. The Route 13 maze has only one correct route through - the upper narrow pass. Finally, the Victory Road 3F boulder puzzle requires the player to push the boulder in the upper right all the way to the upper left switch, while ignoring the boulder puzzles, ladders, and exits to the south. Notably, if the model is instructed to solve a given puzzle at all once (e.g., via pathfinder), it can manage to do so if the context length is not too long. For instance, pathfinder implemented with Gemini 2.5 Pro is able to solve the B3F spinner trap in one shot. Agent Panic Over the course of the playthrough, Gemini 2.5 Pro gets into various situations which cause the model to simulate “panic”. For example, when the Pokémon in the party's health or power points are low, the model's thoughts repeatedly reiterate the need to heal the party immediately or escape the current dungeon (e.g., famously using the move DIG or an ESCAPE ROPE item). Quite interestingly, this mode of model performance appears to correlate with a qualitatively observable degradation in the model's reasoning capability - for instance, completely forgetting to use the pathfinder tool in stretches of gameplay while this condition persists. This behavior has occurred in enough separate instances that the members of the Twitch chat have actively noticed when it is occurring. Actions vs. Game Milestones For completeness, we plot the number of actions/steps required to achieve each game milestone (see Figure 15). An action consists of each bucketed instance where the agent outputs a sequence of button presses to the game (note that other AI agents playing Pokémon may output different numbers of button presses per action, define what constitutes a button press differently, or define an action/step differently). However, it is important to consider action-milestone plots in conjunction with information about the time and/or cost in order to obtain the full picture about the agent's performance. (a) The fully autonomous Run 2 milestones as a function of the number of individual actions. (b) Comparison of 2.5 Pro and 2.5 Flash in terms of actions to milestones. Figure 15: Analog of Figure 6 and 15(b), in terms of actions instead of hours. 8.3 Frontier Safety Framework Evaluations Additional Details: Frontier Safety Correctness Tests For each testing environment, we performed basic correctness checks by looking at how the agents behaved. This involved combining AI and manual reviews of the agents' actions to flag potential issues. On RE-Bench, we examined the best, median and lowest scoring trajectories. For cybersecurity environments (InterCode CTFs, Internal CTFs, Hack the Box), we carefully inspected at least one successful attempt (where available) from each environment, and otherwise examined an unsuccessful attempt. We also performed checks on sample situational awareness and stealth evaluations. This involved basic spot checks to ensure that the prompt and shell outputs were correctly formatted. We used AI assistance to monitor for obvious instances of cheating, and did not find any. For the RE-Bench tests specifically, we also looked at how the best-performing agent achieved its score to ensure that it was a plausible approach, rather than exploiting an obvious reward hack. Overall, we did not observe errors that we believe would invalidate the results of the benchmarks. 8.4 Image to Code Demo We prompted Gemini 1.5 Pro and Gemini 2.5 Pro to generate an SVG representation of an image and found Gemini 2.5 Pro generates better reconstructions. Prompt Gemini 1.5 Pro Gemini 2.5 Pro Please convert this image into SVG and try to reconstruct the spatial arrangement of the objects. Figure 16: Comparison of Gemini 1.5 Pro and Gemini 2.5 Pro responses to image-to-SVG reconstruction prompt. 8.5 Long Context Video Understanding Demo We prompted Gemini with the full video and the following prompt to assess its ability to find a very specific scene (which only lasts 1 second) out of the full 46 minutes video. Video (46 minutes): Redefining Robotics with Carolina Parada Prompt: What color is the T-shirt the robot arms are trying to fold, and at what timecode does this occur? Figure 17: Frame and timestamp the model is meant to recall. Taken from https://www.youtube.com/watch?v=Rgwty6dGsYI at timestamp 27:29. See Table 12 for samples. Gemini 1.5 Pro gets the color right in 1/3 cases, and gets the timestamp in 0/3 cases (quite far away). Gemini 2.5 Pro gets the color in 3/3 cases, and gets the timestamp in 1/3 cases (remaining 2/3 are within 3 seconds close). Model Trial Model response Gemini 1.5 Pro 1 The t-shirt the robot arms are trying to fold is a dark teal or turquoise blue color. The folding begins at and continues at [11:06-11:15]. 2 The T-shirt is orange, and the robot can be seen trying to fold it starting at [00:00:26-00:00:28]. 3 The t-shirt being folded is dark blue and can be seen between [00:00:56-00:00:58] and [00:07:56-00:07:59]. 2.5 Pro Preview 05-06 1 The T-shirt the robot arms are trying to fold is blue. This scene occurs at timecode 27:29. 2 The T-shirt the robot arms are trying to fold is blue. This occurs at timecode 27:26. 3 The T-shirt the robot arms are interacting with (as part of the "fold laundry" task mentioned by the speaker) is blue. This occurs at timestamp 0:27:28. Table 12: Gemini models samples for T-shirt folding scene recall Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" () button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. We gratefully acknowledge support from our major funders, member institutions, , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2507.06415.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2507.06415.txt new file mode 100644 index 0000000..63efd15 --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2507.06415.txt @@ -0,0 +1 @@ +Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Background and Preliminaries 3 PERK: Meta-Learning Parameter-Efficient Reasoning over Knowledge 3.1 Parameter-Efficient Meta-Learning with Low-Rank Adaptation 3.2 Scalable Meta-Learning with Truncated Gradient Unrolling 4 Experiments 4.1 Experimental Setup Datasets. 4.2 Long-Context Reasoning Results 5 Analysis 5.1 Test-Time Length Robustness 5.2 Robustness to Positional Biases 5.3 Inference Latency and Memory Usage Setup PERK scales more efficiently in memory and runtime on extremely long contexts. 6 Related Work 7 Conclusion References A Datasets A.1 BabiLong Tasks Build Long Context A.2 Student Records Tasks Build Long Context A.3 Open-Domain Multi-doc QA Build Long Context A.4 Lost-in-the-Middle API Retrieval Task Build Long Context B Training and Inference Details Hardware Information Standard Hyperparameters LoRA Hyperparameters Meta-Learning Hyperparameters Long-context Data Collation for PERK Evaluation Hyperparameters C Additional Analysis C.1 Impact of LoRA Rank in PERK C.2 Inner Loop Adaptation Steps C.3 Gradient Unrolling Truncation Steps D Training Memory Cost Analysis E Detailed Results License: CC BY 4.0 arXiv:2507.06415v3 [cs.CL] 31 Aug 2026 PERK: Long-Context Reasoning as Test-Time Learning Zeming Chen Angelika Romanou Gail Weiss Antoine Bosselut Affiliation: EPFL Affiliation: email{zeming.chen, antoine.bosselut}@epfl.ch Abstract Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for learning to encode long contexts using gradient updates at test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes contexts into a low-rank adapter (LoRA) that serves as a parameter-efficient memory module for the base model. Concurrently, the outer loop learns to use the updated adapter to accurately recall and reason over relevant information from the encoded long context. Our evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard long-context finetuning, achieving average absolute performance gains of up to 20% for Qwen-2.5 (0.5B & 7B) on synthetic and real-world long-context reasoning. PERK also maintains its advantages across model scales and families. Compared to specialized long-context LLMs, PERK matches or surpasses their performance. Finally, our analyses show PERK is more robust to reasoning complexity, length extrapolation, and the positions of relevant information in contexts. https://perk-long-context.web.app 1 Introduction Large language models (LLMs) struggle with identifying and reasoning over relevant information in long, potentially noisy, input contexts (i.e., long-context reasoning; Anil et al., 2022; Lee et al., 2024; Li et al., 2024; Liu et al., 2023; Xu et al., 2024), including modern models with substantial context windows (Kuratov et al., 2024; Levy et al., 2024; Hsieh et al., 2024). These challenges arise because longer contexts contain more irrelevant distractor information, and may also be attached to more complex reasoning problems (i.e., multiple hops), both of which degrade LLM performance (Chen et al., 2023b). Furthermore, these challenges are exacerbated by positional biases in long-context models, which focus attention on the beginning or end of contexts, neglecting intermediate information (Liu et al., 2024). In this work, we address these challenges of long-context reasoning by reframing it as test-time learning, enabling models to encode context at inference time using gradient-based parameter adaptation. Our approach, Parameter-Efficient Reasoning over Knowledge (PERK), processes and compresses lengthy sequences as a batch (or multiple batches) of shorter segments of the long context, rather than processing the full sequence token-by-token. The updated model can then use the newly stored parametric information (and its initial parametric knowledge) to answer relevant questions. The original context is then discarded in this answering step as it is now encoded parametrically by the model. To train PERK, we operate two nested optimization loops: the inner loop, which is also used at test time, encodes the long context segment batches into model parameters, while the outer loop learns to respond to queries using the updated parameters representing the context. However, bi-level optimization algorithms for training test-time learning methods entail prohibitive memory overhead (Finn et al., 2017; Chen et al., 2023b; Hu et al., 2023), particularly when updating all model parameters, because they require backpropagating through a complete multi-step optimization trajectory. To reduce memory overhead, PERK (1) internalizes the context into a Low-Rank Adapter (LoRA; Hu et al., 2022) rather than the initial model parameters, and (2) backpropagates the outer loop over only a truncated trajectory from the final few inner-loop adaptation steps (Shaban et al., 2019). Consequently, PERK scales efficiently at training, supporting larger models and longer contexts. We compare PERK to models finetuned on long contexts for classical in-context reasoning (FT-ICR) on multiple Needle-in-a-Haystack benchmarks, demonstrating PERK's advantages. For Qwen-2.5-0.5B with a native 32K context window, PERK surpasses FT-ICR baselines by a 20% absolute improvement. PERK also outperforms specialized long-context LLMs (Gao et al., 2025a; Yang et al., 2025) with over 7B parameters that have undergone extensive long-context training, achieving the highest accuracy when generalizing to contexts up to 128k tokens. As relevant facts in Needle-in-a-Haystack problems are often easily distinguishable from distractor text, we also propose more challenging evaluation settings where relevant facts are surrounded by distributionally similar distractor facts (e.g., Drops-in-the-Ocean, HotPotQA, TriviaQA). In these settings, PERK models again outperform the FT-ICR baselines, and even surpass commercial LLMs (e.g., GPT, Gemini) on benchmarks for long-context symbolic reasoning. PERK also works effectively across many model scales (0.1B, 0.5B, 8B) and families (GPT-2, Qwen-2.5, LLaMA-3.1/3.2). Finally, our analysis also shows PERK's advantages yield strong test-time length extrapolation. PERK trained on 8K-token contexts maintains decent performance on 64K and 128K windows, consistently surpassing FT-ICR length generalization, and even outperforming models with native 512K and 1M context windows. Finally, because PERK encodes sequences as a permutation-invariant batch of context window segments (rather than maintaining the full contiguous context), it generalizes across contexts where relevant information is variably distributed along context positions, unlike FT-ICR's dramatic drops in performance (up to 90%) when relevant information shifts positions at test time. 2 Background and Preliminaries Causal Language Models. Causal language models (CLM) f𝜽f_{\bm{\theta}} (Radford and Narasimhan, 2018) with parameters 𝜽{\bm{\theta}} are typically trained with an autoregressive learning objective such as the token-level negative log-likelihood (NLL). For a dataset 𝒟={x}{\mathcal{D}}=\{x\} with N𝒟N_{\mathcal{D}} sequences, this loss is defined as: ℒNLL(𝜽,𝒟)=−1N𝒟∑x∈𝒟∑t=1𝒯logp𝜽(xt|𝒙 Relation > Recall), suggesting that PERK enhances complex reasoning capabilities over long contexts rather than merely improving simple retrieval operations. Finally, as before, both PERK (Qwen) 7B and 0.5B outperform specialized long-context models despite their large parameter size and optimized training strategies for long sequences. PERK (Qwen-7B) also achieves performance that matches or exceeds the commercial models, Gemini-1.5-pro and GPT-4.1, even on the Aggregate task. Figure 3: PERK with diverse model scales and families on BabiLong QA2 with an 8K train & test context length. PERK's performance generalizes across models and scales, consistently outperforms FT-ICR baselines. Model Generalizability. We further analyze whether PERK's performance can generalize across diverse model families and scales in Figure 3. While FT-ICR shows highly variable accuracy ranging from 18.2% on GPT-2 to 89.8% on LLaMA-8B, PERK maintains consistently high accuracy across all tested models. PERK's performance scales positively with model size, achieving 4% increasing on Qwen-7B compared to Qwen-0.5B, and reaching 99.1% accuracy on LLaMA-8B. FT-ICR, despite improving with model size, still lags behind PERK by 9.3 percentage points even on the largest model tested. The generalization across model families is also clear. PERK achieves strong performance on models from the GPT-2, Qwen-2.5, and LLaMA-3.1/3.2 families. Our results demonstrate that PERK's advantages are not limited to specific architectures or model scales, but rather represent a fundamentally effective approach for long-context modeling. 5 Analysis In this section, we analyze PERK's robustness in two challenging settings: test-time length robustness, where PERK is evaluated on longer (or shorter) contexts than those seen during long-context training (Press et al., 2022), and positional variation, where relevant contextual information is distributed throughout the long input context window (Liu et al., 2024). 5.1 Test-Time Length Robustness Setup. Focusing on BabiLong tasks, which allow us to generate contexts of arbitrarily long lengths, we train a Qwen-2.5-0.5B base model using PERK on fixed-length contexts ranging from 1K to 8K tokens (as well as an FT-ICR baseline). Subsequently, we test on contexts ranging from 1K to 32K tokens (Figure 5). For analyzing length generalization on extreme context windows, we also extrapolate on contexts with 64K and 128K tokens (Table 1). We also compare PERK to recent training-free length generalization methods, Yarn (Peng et al., 2023b) and Dual Chunk Attention (DCA; An et al., 2024a) by applying these techniques to the FT-ICR Qwen model at inference time. PERK generalizes to new context lengths at test time. Figure 5 shows that PERK outperforms FT-ICR on the BabiLong QA1 (single-hop) and QA2 (two-hop) tasks for all settings. FT-ICR's performance drops drastically as the inference context length exceeds the training context length, especially for shorter training contexts. While PERK's accuracy also decreases (from a higher starting point), the degradation is dampened at longer test lengths. Meanwhile, for interpolation, FT-ICR performance drops on shorter contexts when finetuned on longer ones, matching observations from prior work (Gao et al., 2025b). In contrast, PERK maintains (and even increases) its performance. (a) Single-hop reasoning (b) Two-hop reasoning Model 32K 64K 128K 32K 64K 128K Commercial Frontier Models GPT-4.1 87.8 78.7 69.4 80.6 66.4 48.2 Gemini-1.5-pro 87.7 82.3 73.1 56.4 48.8 40.2 Trained on contexts with >256K tokens Qwen2.5-7B-Instruct-1M 65.7 30.3 21.4 35.3 21.0 12.2 ProLong-8B-Instruct-512K 41.0 33.2 24.3 29.0 18.5 17.7 Trained on contexts with 8K tokens FT-ICR (Qwen2.5-0.5B) 43.8 34.5 0 39.4 11.7 0 w/ Yarn + DCA 59.2 35.5 25.4 42.5 26.3 18.5 PERK (Qwen2.5-0.5B) 95.3 88.9 61.4 80.9 62.5 44.4 Table 1: Test-time length extrapolation beyond 32K on BabiLong QA1 (a) and QA2 (b). PERK and FT-ICR are trained on 8K-token sequences. The context length for inference grows from 64K to 128K. PERK extrapolates substantially better than FT-ICR. To stress-test the limit of PERK models' length extrapolation ability, we test on sequences with context lengths beyond the context window, 32k, of the Qwen-2.5 model - 64K and 128K tokens in Table 1, and demonstrate that PERK consistently extrapolates to extreme context lengths more robustly than long-context finetuning (FT-ICR). With 128K-token context windows, PERK's accuracy experiences a noticeable drop to 61.4% and 44.4% accuracy for QA1 and QA2, respectively, but these scores remain substantially better than the FT-ICR baseline at this same context length. After applying the training-free length generalization methods, Yarn and DCA, to FT-ICR Qwen models, the baselines show improved generalization on the extreme contexts. However, Yarn + DCA's extrapolation performance is still far below PERK for both QA1 and QA2 (around 35% lower on average). Compared to the specialized frontier models, PERK outperforms gemini-1.5-pro and the two open-source models, and achieves a narrow performance gap to GPT-4.1 on both 64K and 128K contexts. We note, as well, that PERK is only trained on contexts with 8K tokens, while the Qwen-1M and ProLong models have been trained on much longer contexts with 256K and 512K tokens. Overall, PERK demonstrates strong length generalization, extrapolating to longer contexts while preserving high performance on shorter ones. Figure 4: Test-time context length robustness evaluation on BabiLong QA1 (a) and QA2 (b) between PERK and FT-ICR on the Qwen-2.5-0.5B model. The y-axis represents the training context lengths, while the x-axis indicates various test-time context lengths. We test for both test lengths shorter than the training length, and test lengths longer than the training length. Bordered cells denote evaluation on context lengths equal to those in training. PERK shows stronger robustness across both settings. Figure 5: Positional Bias. Comparison of PERK and FT-ICR on 4K and 8K contexts, on Qwen-2.5-0.5B. We train on problems where the relevant information appears in the beginning (Pre), middle (Mid), or end (Post) of the context, and evaluate on all three positional settings. We also train models on contexts where the relevant information is randomly located (Rnd), testing these on all four positional distributions (Pre, Post, Mid, Rnd). Bordered cells show in-distribution performances. PERK demonstrates strong positional robustness. 5.2 Robustness to Positional Biases Prior work has shown that the position of relevant information in long contexts affects a language model's ability to use that information (Liu et al., 2024). In this experiment, we test PERK's robustness to positional biases by evaluating its test-time performance on contexts where the position of relevant information is distributed differently from those seen in training. Setup. We use API documents from Patil et al. (2024), where each document has an associated API call, and create long contexts containing multiple API-call-to-document pairs. In this task, the model must retrieve the correct API call based on user instructions and documents. Using fixed 4k and 8k token contexts, we train PERK and FT-ICR models on contexts with the target document at varying positions and test them on contexts distributed across all positions. PERK generalizes regardless of information position in the context. Figure 5 compares PERK to FT-ICR under different position distributions (at train and test time) and context lengths. When trained on contexts with relevant documents randomly located (Rnd) throughout the context, PERK outperforms FT-ICR by 24%. Furthermore, FT-ICR shows large performance drops when the relevant document consistently appears at different locations (Pre, Mid, and Post) at test time. Meanwhile, shifting relevant positions at test time shows minimal effect (within 1-2%) on PERK. To further stress-test positional generalization, we also force relevant documents into specific positions during training and testing, spanning the full context. We find that FT-ICR easily overfits to the position pattern, completely failing to generalize to test-time position changes (performance drops to close to 0% when the position shifts at test time). Although PERK underperforms FT-ICR when the train-test positions are the same and the beginning (Pre) or end (Post) of the context, performance remains consistent regardless of the positional distribution shift between train and test time. We attribute PERK's robustness in this setting to the fact that documents in the long context are encoded as members of a permutation-invariant batch into the parameter space, limiting the overall impact of absolute position in the sequence. Meanwhile, FT-ICR directly aligns specific positions with answers during training. 5.3 Inference Latency and Memory Usage We evaluate PERK's efficiency on long contexts by measuring its inference memory cost and run-time, compared to in-context reasoning with finetuned models (FT-ICR). Figure 6: Inference-time memory footprint and wall-clock runtime as context length increases (up to 128K tokens), comparing PERK with FT-ICR. Curves that terminate before 128K indicate the method failed with an OOM error at longer context lengths, preventing further measurement. PERK demonstrates more efficient scaling in both memory and runtime, particularly for longer sequences. Setup We complete all runs on a single Nvidia H100 GPU with 93 GB of VRAM and with Huggingface's Transformers library version 4.51.3. We perform four gradient update steps for the test-time inner loop adaptations for PERK. At inference time, all methods generate 64 tokens through greedy decoding using Huggingface's text generation pipeline. We measure runtime complexity in seconds (wall-clock time) and memory complexity in gigabytes. Both methods undergo 10 hardware warm-up iterations. As explained in Section 3, PERK splits long sequences into batches of shorter segments. In this experiment, each batch has segments with an effective context length of 128 tokens. Since PERK performs gradient-based encoding over these batches, we can apply gradient accumulation to trade runtime for reduced memory consumption. We thus analyze PERK's efficiency across varying gradient accumulation steps, ranging from 2 to 16. PERK scales more efficiently in memory and runtime on extremely long contexts. Figure 6 shows that PERK provides more efficient memory and runtime scaling for extremely long contexts compared to FT-ICR. While FT-ICR is initially more efficient, its memory and runtime grow rapidly, leading to OOM errors at a context length of 128K. In contrast, PERK can manage the long sequences through gradient accumulation, which, while increasing runtime, reduces the memory footprint. For instance, at 8K tokens, increasing the accumulation steps from 1 to 16 reduces memory usage from 35.2GB to 5.9GB, although the runtime increases from 1.9s to 8.5s. Crucially, even with this runtime trade-off, PERK runs faster than FT-ICR at very long contexts where both can operate. At 64K tokens, FT-ICR takes 32.6s and 55.7GB, whereas PERK with 16 steps of accumulation completes in 11.4s, using only 19.6GB. With 8 steps, it is even faster at 9.7s, using 35.2 GB. Ultimately, at 128K tokens, where FT-ICR fails, PERK with 16 steps successfully processes the context using 35.2GB in 20.9s, showing that PERK provides a practical path to handle extreme context lengths efficiently in both memory and runtime compared to standard approaches. 6 Related Work Test-Time Learning. MAML (Finn et al., 2017) introduces optimization-based model-agnostic meta-learning for few-shot adaptation. Chen et al. (2023b) applies MAML to GPT models, integrating knowledge into model parameters through test-time gradient updates for downstream reasoning. A parallel line of work internalizes context into parameters via distillation (Snell et al., 2022; Caccia et al., 2025; Cao et al., 2025). Other methods achieve test-time learning in the model architecture. Sun et al. (2025) proposes RNN layers with learnable hidden states updated via self-supervised gradients on test inputs. Titans (Behrouz et al., 2024) incorporates a differentiable memory with transformers for rapid inference-time memorization. Building on Titan, ATLAS (Behrouz et al., 2025) proposes a new transformer architecture that contains the differentiable memory module as part of the network to support large-scale distributed pretraining. Other approaches adapt through context alone. MetaICL (Min et al., 2022) meta-learns from in-context examples without gradient updates. Our method, PERK, introduces parameter-efficient meta-learning using lightweight adapters, improving scalability for long contexts. Long-Context Language Models. Developing long-context models has become a major research topic of its own. Many works explore extending transformer-based language models' context windows with minimal training, either by position extrapolation (Chen et al., 2023a; Peng et al., 2023c; Xiao et al., 2024; Yen et al., 2024) or improving the transformer's attention mechanism (Bertsch et al., 2023; Mohtashami and Jaggi, 2023; Shen et al., 2023; Jin et al., 2024). Other works show that using the default attention, applying simple position extrapolation, and fine-tuning the model on long documents (Fu et al., 2024; Lu et al., 2024) or synthetically generated long data (An et al., 2024b; Xiong et al., 2024) yields much stronger results. The FT-ICR baseline in our work follows this paradigm. To reduce the expansive computation cost of long-context training, Chen et al. (2024)'s work combines LoRA with trainable embeddings and normalization to achieve efficient finetuning on long contexts. Other LLM works (Lieber et al., 2024; Grattafiori et al., 2024; Gao et al., 2025b) propose achieving long-context capabilities through a long-context continued training stage between standard pre-training and supervised fine-tuning. In contrast to all the approaches above, PERK achieves long-context reasoning through the ability to encode long sequences using test-time gradient updates on a lightweight model adapter. Long-Context Architectures. In parallel, there have been many efforts in designing more efficient architectures, for example, linear attention with RNNs (Chandrayan et al., 2023; Peng et al., 2023a; Gu and Dao, 2024; Dao and Gu, 2024; Yang et al., 2024; Beck et al., 2024) and alternative attention architectures (Rubin and Berant, 2024; Sun et al., 2024; Han et al., 2023). However, these new architectures often require training from scratch, and many have inherent limitations in terms of long-context recall (Jelassi et al., 2024; Arora et al., 2025). Recent works also explore hybrid models (Lieber et al., 2024; De et al., 2024) or distilling existing LLMs into hybrid models (Wang et al., 2024) and show promising results. In comparison, PERK directly augments off-the-shelf pre-trained LLMs without requiring additional architecture modifications or extensive pre-training, while naturally being parameter-efficient. Long-Context Evaluation. Many benchmarks have been proposed to evaluate language models' long-context abilities (Krishna et al., 2023; Shaham et al., 2023; Bai et al., 2024; Zhang et al., 2024b; Hsieh et al., 2024; Yen et al., 2025; Ye et al., 2025; Xia et al., 2025). Some works focus on the impact of extending input lengths with irrelevant contexts (Kuratov et al., 2024; Levy et al., 2024) or changing relevant information positions (Liu et al., 2024) on LLM reasoning. Many works also reframe natural-language-processing tasks, such as retrieval (Lee et al., 2024; Qiu et al., 2025), summarization (Kim et al., 2024), and in-context learning (Li et al., 2024; Xu et al., 2024; Bertsch et al., 2025), as long-context challenges. In this work, we define Drops-in-the-Ocean as a new long-context evaluation that makes distractors distributionally similar to relevant information. 7 Conclusion In this work, we introduce PERK, a parameter-efficient meta-learning approach that learns to perform long-context reasoning by encoding information into a lightweight adapter through test-time gradient updates. Our experiments on multiple long-context benchmarks (Needle-in-the-Haystack, HotpotQA, TriviaQA, Drops-in-the-Ocean, and Code API retrieval) demonstrate substantial performance gains over models finetuned on long contexts for classical in-context reasoning across reasoning complexity and context lengths. We demonstrate that PERK's performance robustly transfers across model scales (0.5B to 8B) and model families (GPT-2, Qwen2.5, LLaMA-3). Our generalization analysis highlights PERK's strong test-time length extrapolation ability, where the model can generalize to unseen contexts 32 ×\times longer than the training data. When relevant information shifts positions at test time, PERK robustly generalizes across different positions. Overall, PERK excels in performance, generalization, and robustness for long-context reasoning. Acknowledgments We gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29), the EPFL Center for Imaging, Sony Group Corporation, and a Meta LLM Evaluation Research Grant. References An et al. (2024a) C. An, F. Huang, J. Zhang, S. Gong, X. Qiu, C. Zhou, and L. Kong Training-free long-context scaling of large language models. External Links: 2402.17463, Link Cited by: Appendix B, §5.1. An et al. (2024b) S. An, Z. Ma, Z. Lin, N. Zheng, J. Lou, and W. Chen Make your llm fully utilize the context. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 62160-62188. External Links: Link Cited by: §6. Anil et al. (2022) C. Anil, Y. Wu, A. Andreassen, A. Lewkowycz, V. Misra, V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur Exploring length generalization in large language models. Advances in Neural Information Processing Systems 35, pp. 38546-38556. Cited by: §1. Arora et al. (2025) S. Arora, S. Eyuboglu, M. Zhang, A. Timalsina, S. Alberti, D. Zinsley, J. Zou, A. Rudra, and C. Ré Simple linear attention language models balance the recall-throughput tradeoff. arXiv preprint arXiv:2402.18668. External Links: 2402.18668, Link Cited by: §6. Asai et al. (2023) A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi Self-rag: learning to retrieve, generate, and critique through self-reflection. External Links: 2310.11511, Link Cited by: §4.1. Bai et al. (2024) Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li LongBench: a bilingual, multitask benchmark for long context understanding. External Links: 2308.14508, Link Cited by: §6. Bayazit et al. (2024) D. Bayazit, N. Foroutan, Z. Chen, G. Weiss, and A. Bosselut Discovering knowledge-critical subnetworks in pretrained language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 6549-6583. External Links: Link, Document Cited by: §2. Beck et al. (2024) M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter XLSTM: extended long short-term memory. arXiv preprint arXiv:2405.04517. External Links: 2405.04517, Link Cited by: §6. Behrouz et al. (2025) A. Behrouz, Z. Li, P. Kacham, M. Daliri, Y. Deng, P. Zhong, M. Razaviyayn, and V. Mirrokni ATLAS: learning to optimally memorize the context at test time. External Links: 2505.23735, Link Cited by: §6. Behrouz et al. (2024) A. Behrouz, P. Zhong, and V. Mirrokni Titans: learning to memorize at test time. arXiv preprint arXiv:2501.00663. External Links: 2501.00663, Link Cited by: §6. Bertsch et al. (2023) A. Bertsch, U. Alon, G. Neubig, and M. Gormley Unlimiformer: long-range transformers with unlimited length input. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, pp. 35522-35543. External Links: Link Cited by: §6. Bertsch et al. (2025) A. Bertsch, M. Ivgi, E. Xiao, U. Alon, J. Berant, M. R. Gormley, and G. Neubig In-context learning with long-context models: an in-depth exploration. arXiv preprint arXiv:2405.00200. External Links: 2405.00200, Link Cited by: §6. Caccia et al. (2025) L. Caccia, A. Ansell, E. Ponti, I. Vulić, and A. Sordoni Training plug-n-play knowledge modules with deep context distillation. arXiv preprint arXiv:2503.08727. Cited by: §6. Cao et al. (2025) B. Cao, D. Cai, and W. Lam InfiniteICL: breaking the limit of context window size via long short-term memory transformation. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 11402-11415. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §6. Chandrayan et al. (2023) K. Chandrayan, L. Martin, G. Kamradt, L. Hurtado, A. Arkhangorodsky, L. Hurtado, and P. Kral Needle in a haystack - pressure testing llms. Note: https://github.com/gkamradt/LLMTest_NeedleInAHaystack Cited by: §6. Chen et al. (2023a) S. Chen, S. Wong, L. Chen, and Y. Tian Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. External Links: 2306.15595, Link Cited by: §6. Chen et al. (2024) Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia LongLoRA: efficient fine-tuning of long-context large language models. External Links: 2309.12307, Link Cited by: §6. Chen et al. (2023b) Z. Chen, G. Weiss, E. Mitchell, A. Celikyilmaz, and A. Bosselut RECKONING: reasoning through dynamic knowledge encoding. Advances in Neural Information Processing Systems 36, pp. 62579-62600. Cited by: Appendix D, §1, §1, §2, §6. Clark et al. (2020) P. Clark, O. Tafjord, and K. Richardson Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, C. Bessiere (Ed.), pp. 3882-3890. External Links: Link, Document Cited by: §2. Dao and Gu (2024) T. Dao and A. Gu Transformers are ssms: generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060. External Links: 2405.21060, Link Cited by: §6. De et al. (2024) S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, G. Desjardins, A. Doucet, D. Budden, Y. W. Teh, R. Pascanu, N. D. Freitas, and C. Gulcehre Griffin: mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427. External Links: 2402.19427, Link Cited by: §6. Finn et al. (2017) C. Finn, P. Abbeel, and S. Levine Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. Cited by: §1, §2, §6. Fu et al. (2024) Y. Fu, R. Panda, X. Niu, X. Yue, H. Hajishirzi, Y. Kim, and H. Peng Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10171. External Links: 2402.10171, Link Cited by: §4.1, §6. Gao et al. (2025a) T. Gao, A. Wettig, H. Yen, and D. Chen How to train long-context language models (effectively). External Links: 2410.02660, Link Cited by: Table 5, §1, §4.1. Gao et al. (2025b) T. Gao, A. Wettig, H. Yen, and D. Chen How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660. External Links: 2410.02660, Link Cited by: §4.1, §5.1, §6. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma The llama 3 herd of models. arXiv preprint arXiv:2407.21783. External Links: 2407.21783, Link Cited by: §4.1, §6. Gu and Dao (2024) A. Gu and T. Dao Mamba: linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. External Links: 2312.00752, Link Cited by: §4.1, §6. Han et al. (2023) I. Han, R. Jayaram, A. Karbasi, V. S. Mirrokni, D. P. Woodruff, and A. Zandieh HyperAttention: long-context attention in near-linear time. ArXiv abs/2310.05869. External Links: Link Cited by: §6. Hsieh et al. (2024) C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg RULER: what's the real context size of your long-context language models?. arXiv preprint arXiv:2404.06654. External Links: 2404.06654, Link Cited by: §1, §2, §6. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: low-rank adaptation of large language models.. ICLR 1 (2), pp. 3. Cited by: §1, §3.1. Hu et al. (2023) N. Hu, E. Mitchell, C. D. Manning, and C. Finn Meta-learning online adaptation of language models. arXiv preprint arXiv:2403.04317. External Links: 2305.15076, Link Cited by: Appendix B, §1. Jelassi et al. (2024) S. Jelassi, D. Brandfonbrener, S. M. Kakade, and E. Malach Repeat after me: transformers are better than state space models at copying. arXiv preprint arXiv:2402.01032. External Links: 2402.01032, Link Cited by: §6. Jin et al. (2024) H. Jin, X. Han, J. Yang, Z. Jiang, Z. Liu, C. Chang, H. Chen, and X. Hu LLM maybe longlm: self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325. External Links: 2401.01325, Link Cited by: §6. Joshi et al. (2017) M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M. Kan (Eds.), Vancouver, Canada, pp. 1601-1611. External Links: Link, Document Cited by: §A.3, §4.1. Kalajdzievski (2023) D. Kalajdzievski A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732. External Links: 2312.03732, Link Cited by: Appendix B. Karypis et al. (2024) P. Karypis, J. McAuley, and G. Karypis Extending input contexts of language models through training on segmented sequences. In Findings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, pp. 3040-3052. External Links: Link, Document Cited by: §4.1. Kim et al. (2024) Y. Kim, Y. Chang, M. Karpinska, A. Garimella, V. Manjunatha, K. Lo, T. Goyal, and M. Iyyer FABLES: evaluating faithfulness and content selection in book-length summarization. External Links: 2404.01261, Link Cited by: §6. Krishna et al. (2023) K. Krishna, E. Bransom, B. Kuehl, M. Iyyer, P. Dasigi, A. Cohan, and K. Lo LongEval: guidelines for human evaluation of faithfulness in long-form summarization. External Links: 2301.13298, Link Cited by: §6. Kuratov et al. (2024) Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev Babilong: testing the limits of llms with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems 37, pp. 106519-106554. Cited by: §A.1, Table 5, §1, §2, §4.1, §4.1, §4, §6. Lambert et al. (2025) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi Tulu 3: pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. External Links: 2411.15124, Link Cited by: Appendix B. Lee et al. (2024) J. Lee, A. Chen, Z. Dai, D. Dua, D. S. Sachan, M. Boratko, Y. Luan, S. M. R. Arnold, V. Perot, S. Dalmia, H. Hu, X. Lin, P. Pasupat, A. Amini, J. R. Cole, S. Riedel, I. Naim, M. Chang, and K. Guu Can long-context language models subsume retrieval, rag, sql, and more?. arXiv preprint arXiv:2406.13121. External Links: 2406.13121, Link Cited by: §1, §6. Levy et al. (2024) M. Levy, A. Jacoby, and Y. Goldberg Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848. External Links: 2402.14848, Link Cited by: §1, §2, §6. Li et al. (2024) T. Li, G. Zhang, Q. D. Do, X. Yue, and W. Chen Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060. External Links: 2404.02060, Link Cited by: §1, §6. Lieber et al. (2024) O. Lieber, B. Lenz, H. Bata, G. Cohen, J. Osin, I. Dalmedigos, E. Safahi, S. Meirom, Y. Belinkov, S. Shalev-Shwartz, O. Abend, R. Alon, T. Asida, A. Bergman, R. Glozman, M. Gokhman, A. Manevich, N. Ratner, N. Rozen, E. Shwartz, M. Zusman, and Y. Shoham Jamba: a hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887. External Links: 2403.19887, Link Cited by: §6, §6. Liu et al. (2023) B. Liu, J. Ash, S. Goel, A. Krishnamurthy, and C. Zhang Exposing attention glitches with flip-flop language modeling. Advances in Neural Information Processing Systems 36, pp. 25549-25583. Cited by: §1. Liu et al. (2024) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, pp. 157-173. Cited by: §A.4, §1, §5.2, §5, §6. Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: Appendix B, §3.1. Lozhkov et al. (2024) A. Lozhkov, L. Ben Allal, L. von Werra, and T. Wolf FineWeb-edu: the finest collection of educational content. External Links: Link, Document Cited by: §A.1, §4.1. Lu et al. (2024) Y. Lu, J. N. Yan, S. Yang, J. T. Chiu, S. Ren, F. Yuan, W. Zhao, Z. Wu, and A. M. Rush A controlled study on long context extension and generalization in llms. arXiv preprint arXiv:2409.12181. External Links: 2409.12181, Link Cited by: §6. Min et al. (2022) S. Min, M. Lewis, L. Zettlemoyer, and H. Hajishirzi MetaICL: learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, M. Carpuat, M. de Marneffe, and I. V. Meza Ruiz (Eds.), Seattle, United States, pp. 2791-2809. External Links: Link, Document Cited by: §6. Mohtashami and Jaggi (2023) A. Mohtashami and M. Jaggi Random-access infinite context length for transformers. In Thirty-seventh Conference on Neural Information Processing Systems, External Links: Link Cited by: §6. OpenAI (2025) OpenAI GPT-4.1. Note: https://openai.comLarge language model Cited by: Table 5, §4.1. Patil et al. (2024) S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez Gorilla: large language model connected with massive apis. Advances in Neural Information Processing Systems 37, pp. 126544-126565. Cited by: §A.4, §5.2. Peng et al. (2023a) B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV, X. He, H. Hou, J. Lin, P. Kazienko, J. Kocon, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, G. Song, X. Tang, B. Wang, J. S. Wind, S. Wozniak, R. Zhang, Z. Zhang, Q. Zhao, P. Zhou, Q. Zhou, J. Zhu, and R. Zhu RWKV: reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048. External Links: 2305.13048, Link Cited by: §6. Peng et al. (2023b) B. Peng, J. Quesnelle, H. Fan, and E. Shippole YaRN: efficient context window extension of large language models. External Links: 2309.00071, Link Cited by: Appendix B, §5.1. Peng et al. (2023c) B. Peng, J. Quesnelle, H. Fan, and E. Shippole YaRN: efficient context window extension of large language models. arXiv preprint arXiv:2309.00071. External Links: 2309.00071, Link Cited by: §6. Press et al. (2022) O. Press, N. A. Smith, and M. Lewis Train short, test long: attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. External Links: 2108.12409, Link Cited by: §5. Qiu et al. (2025) Y. Qiu, V. Embar, Y. Zhang, N. Jaitly, S. B. Cohen, and B. Han Eliciting in-context retrieval and reasoning for long-context large language models. arXiv preprint arXiv:2501.08248. External Links: 2501.08248, Link Cited by: §6. Qwen et al. (2025) Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. External Links: 2412.15115, Link Cited by: §4.1. Radford and Narasimhan (2018) A. Radford and K. Narasimhan Improving language understanding by generative pre-training. External Links: Link Cited by: §2, §4.1. Roberts et al. (2020) A. Roberts, C. Raffel, and N. Shazeer How much knowledge can you pack into the parameters of a language model?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), Online, pp. 5418-5426. External Links: Link, Document Cited by: §2. Rubin and Berant (2024) O. Rubin and J. Berant Retrieval-pretrained transformer: long-range language modeling with self-retrieval. Transactions of the Association for Computational Linguistics 12, pp. 1197-1213. External Links: Link, Document Cited by: §6. Shaban et al. (2019) A. Shaban, C. Cheng, N. Hatch, and B. Boots Truncated back-propagation for bilevel optimization. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, K. Chaudhuri and M. Sugiyama (Eds.), Proceedings of Machine Learning Research, Vol. 89, pp. 1723-1732. External Links: Link Cited by: §1, §3.2. Shaham et al. (2023) U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy ZeroSCROLLS: a zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 7977-7989. External Links: Link, Document Cited by: §6. Shen et al. (2023) K. Shen, J. Guo, X. Tan, S. Tang, R. Wang, and J. Bian A study on relu and softmax in transformer. arXiv preprint arXiv:2302.06461. External Links: 2302.06461, Link Cited by: §6. Snell et al. (2022) C. Snell, D. Klein, and R. Zhong Learning by distilling context. arXiv preprint arXiv:2209.15189. Cited by: §6. Sun et al. (2025) Y. Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y. Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, and C. Guestrin Learning to (learn at test time): rnns with expressive hidden states. arXiv preprint arXiv:2407.04620. External Links: 2407.04620, Link Cited by: §6. Sun et al. (2024) Y. Sun, L. Dong, Y. Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, and F. Wei You only cache once: decoder-decoder architectures for language models. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 7339-7361. External Links: Link Cited by: §6. Team et al. (2024) G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, S. Mariooryad, Y. Ding, X. Geng, F. Alcober, R. Frostig, M. Omernick, L. Walker, C. Paduraru, C. Sorokin, A. Tacchetti, C. Gaffney, S. Daruki, O. Sercinoglu, Z. Gleicher, J. Love, P. Voigtlaender, R. Jain, G. Surita, K. Mohamed, R. Blevins, J. Ahn, T. Zhu, K. Kawintiranon, O. Firat, Y. Gu, Y. Zhang, M. Rahtz, M. Faruqui, N. Clay, J. Gilmer, J. Co-Reyes, I. Penchev, R. Zhu, N. Morioka, K. Hui, K. Haridasan, V. Campos, M. Mahdieh, M. Guo, S. Hassan, K. Kilgour, A. Vezer, H. Cheng, R. de Liedekerke, S. Goyal, P. Barham, D. Strouse, S. Noury, J. Adler, M. Sundararajan, S. Vikram, D. Lepikhin, M. Paganini, X. Garcia, F. Yang, D. Valter, M. Trebacz, K. Vodrahalli, C. Asawaroengchai, R. Ring, N. Kalb, L. B. Soares, S. Brahma, D. Steiner, T. Yu, F. Mentzer, A. He, L. Gonzalez, B. Xu, R. L. Kaufman, L. E. Shafey, J. Oh, T. Hennigan, G. van den Driessche, S. Odoom, M. Lucic, B. Roelofs, S. Lall, A. Marathe, B. Chan, S. Ontanon, L. He, D. Teplyashin, J. Lai, P. Crone, B. Damoc, L. Ho, S. Riedel, K. Lenc, C. Yeh, A. Chowdhery, Y. Xu, M. Kazemi, E. Amid, A. Petrushkina, K. Swersky, A. Khodaei, G. Chen, C. Larkin, M. Pinto, G. Yan, A. P. Badia, P. Patil, S. Hansen, D. Orr, S. M. R. Arnold, J. Grimstad, A. Dai, S. Douglas, R. Sinha, V. Yadav, X. Chen, E. Gribovskaya, J. Austin, J. Zhao, K. Patel, P. Komarek, S. Austin, S. Borgeaud, L. Friso, A. Goyal, B. Caine, K. Cao, D. Chung, M. Lamm, G. Barth-Maron, T. Kagohara, K. Olszewska, M. Chen, K. Shivakumar, R. Agarwal, H. Godhia, R. Rajwar, J. Snaider, X. Dotiwalla, Y. Liu, A. Barua, V. Ungureanu, Y. Zhang, B. Batsaikhan, M. Wirth, J. Qin, I. Danihelka, T. Doshi, M. Chadwick, J. Chen, S. Jain, Q. Le, A. Kar, M. Gurumurthy, C. Li, R. Sang, F. Liu, L. Lamprou, R. Munoz, N. Lintz, H. Mehta, H. Howard, M. Reynolds, L. Aroyo, Q. Wang, L. Blanco, A. Cassirer, J. Griffith, D. Das, S. Lee, J. Sygnowski, Z. Fisher, J. Besley, R. Powell, Z. Ahmed, D. Paulus, D. Reitter, Z. Borsos, R. Joshi, A. Pope, S. Hand, V. Selo, V. Jain, N. Sethi, M. Goel, T. Makino, R. May, Z. Yang, J. Schalkwyk, C. Butterfield, A. Hauth, A. Goldin, W. Hawkins, E. Senter, S. Brin, O. Woodman, M. Ritter, E. Noland, M. Giang, V. Bolina, L. Lee, T. Blyth, I. Mackinnon, M. Reid, O. Sarvana, D. Silver, A. Chen, L. Wang, L. Maggiore, O. Chang, N. Attaluri, G. Thornton, C. Chiu, O. Bunyan, N. Levine, T. Chung, E. Eltyshev, X. Si, T. Lillicrap, D. Brady, V. Aggarwal, B. Wu, Y. Xu, R. McIlroy, K. Badola, P. Sandhu, E. Moreira, W. Stokowiec, R. Hemsley, D. Li, A. Tudor, P. Shyam, E. Rahimtoroghi, S. Haykal, P. Sprechmann, X. Zhou, D. Mincu, Y. Li, R. Addanki, K. Krishna, X. Wu, A. Frechette, M. Eyal, A. Dafoe, D. Lacey, J. Whang, T. Avrahami, Y. Zhang, E. Taropa, H. Lin, D. Toyama, E. Rutherford, M. Sano, H. Choe, A. Tomala, C. Safranek-Shrader, N. Kassner, M. Pajarskas, M. Harvey, S. Sechrist, M. Fortunato, C. Lyu, G. Elsayed, C. Kuang, J. Lottes, E. Chu, C. Jia, C. Chen, P. Humphreys, K. Baumli, C. Tao, R. Samuel, C. N. dos Santos, A. Andreassen, N. Rakićević, D. Grewe, A. Kumar, S. Winkler, J. Caton, A. Brock, S. Dalmia, H. Sheahan, I. Barr, Y. Miao, P. Natsev, J. Devlin, F. Behbahani, F. Prost, Y. Sun, A. Myaskovsky, T. S. Pillai, D. Hurt, A. Lazaridou, X. Xiong, C. Zheng, F. Pardo, X. Li, D. Horgan, J. Stanton, M. Ambar, F. Xia, A. Lince, M. Wang, B. Mustafa, A. Webson, H. Lee, R. Anil, M. Wicke, T. Dozat, A. Sinha, E. Piqueras, E. Dabir, S. Upadhyay, A. Boral, L. A. Hendricks, C. Fry, J. Djolonga, Y. Su, J. Walker, J. Labanowski, R. Huang, V. Misra, J. Chen, R. Skerry-Ryan, A. Singh, S. Rijhwani, D. Yu, A. Castro-Ros, B. Changpinyo, R. Datta, S. Bagri, A. M. Hrafnkelsson, M. Maggioni, D. Zheng, Y. Sulsky, S. Hou, T. L. Paine, A. Yang, J. Riesa, D. Rogozinska, D. Marcus, D. E. Badawy, Q. Zhang, L. Wang, H. Miller, J. Greer, L. L. Sjos, A. Nova, H. Zen, R. Chaabouni, M. Rosca, J. Jiang, C. Chen, R. Liu, T. Sainath, M. Krikun, A. Polozov, J. Lespiau, J. Newlan, Z. Cankara, S. Kwak, Y. Xu, P. Chen, A. Coenen, C. Meyer, K. Tsihlas, A. Ma, J. Gottweis, J. Xing, C. Gu, J. Miao, C. Frank, Z. Cankara, S. Ganapathy, I. Dasgupta, S. Hughes-Fitt, H. Chen, D. Reid, K. Rong, H. Fan, J. van Amersfoort, V. Zhuang, A. Cohen, S. S. Gu, A. Mohananey, A. Ilic, T. Tobin, J. Wieting, A. Bortsova, P. Thacker, E. Wang, E. Caveness, J. Chiu, E. Sezener, A. Kaskasoli, S. Baker, K. Millican, M. Elhawaty, K. Aisopos, C. Lebsack, N. Byrd, H. Dai, W. Jia, M. Wiethoff, E. Davoodi, A. Weston, L. Yagati, A. Ahuja, I. Gao, G. Pundak, S. Zhang, M. Azzam, K. C. Sim, S. Caelles, J. Keeling, A. Sharma, A. Swing, Y. Li, C. Liu, C. G. Bostock, Y. Bansal, Z. Nado, A. Anand, J. Lipschultz, A. Karmarkar, L. Proleev, A. Ittycheriah, S. H. Yeganeh, G. Polovets, A. Faust, J. Sun, A. Rrustemi, P. Li, R. Shivanna, J. Liu, C. Welty, F. Lebron, A. Baddepudi, S. Krause, E. Parisotto, R. Soricut, Z. Xu, D. Bloxwich, M. Johnson, B. Neyshabur, J. Mao-Jones, R. Wang, V. Ramasesh, Z. Abbas, A. Guez, C. Segal, D. D. Nguyen, J. Svensson, L. Hou, S. York, K. Milan, S. Bridgers, W. Gworek, M. Tagliasacchi, J. Lee-Thorp, M. Chang, A. Guseynov, A. J. Hartman, M. Kwong, R. Zhao, S. Kashem, E. Cole, A. Miech, R. Tanburn, M. Phuong, F. Pavetic, S. Cevey, R. Comanescu, R. Ives, S. Yang, C. Du, B. Li, Z. Zhang, M. Iinuma, C. H. Hu, A. Roy, S. Bijwadia, Z. Zhu, D. Martins, R. Saputro, A. Gergely, S. Zheng, D. Jia, I. Antonoglou, A. Sadovsky, S. Gu, Y. Bi, A. Andreev, S. Samangooei, M. Khan, T. Kocisky, A. Filos, C. Kumar, C. Bishop, A. Yu, S. Hodkinson, S. Mittal, P. Shah, A. Moufarek, Y. Cheng, A. Bloniarz, J. Lee, P. Pejman, P. Michel, S. Spencer, V. Feinberg, X. Xiong, N. Savinov, C. Smith, S. Shakeri, D. Tran, M. Chesus, B. Bohnet, G. Tucker, T. von Glehn, C. Muir, Y. Mao, H. Kazawa, A. Slone, K. Soparkar, D. Shrivastava, J. Cobon-Kerr, M. Sharman, J. Pavagadhi, C. Araya, K. Misiunas, N. Ghelani, M. Laskin, D. Barker, Q. Li, A. Briukhov, N. Houlsby, M. Glaese, B. Lakshminarayanan, N. Schucher, Y. Tang, E. Collins, H. Lim, F. Feng, A. Recasens, G. Lai, A. Magni, N. D. Cao, A. Siddhant, Z. Ashwood, J. Orbay, M. Dehghani, J. Brennan, Y. He, K. Xu, Y. Gao, C. Saroufim, J. Molloy, X. Wu, S. Arnold, S. Chang, J. Schrittwieser, E. Buchatskaya, S. Radpour, M. Polacek, S. Giordano, A. Bapna, S. Tokumine, V. Hellendoorn, T. Sottiaux, S. Cogan, A. Severyn, M. Saleh, S. Thakoor, L. Shefey, S. Qiao, M. Gaba, S. Chang, C. Swanson, B. Zhang, B. Lee, P. K. Rubenstein, G. Song, T. Kwiatkowski, A. Koop, A. Kannan, D. Kao, P. Schuh, A. Stjerngren, G. Ghiasi, G. Gibson, L. Vilnis, Y. Yuan, F. T. Ferreira, A. Kamath, T. Klimenko, K. Franko, K. Xiao, I. Bhattacharya, M. Patel, R. Wang, A. Morris, R. Strudel, V. Sharma, P. Choy, S. H. Hashemi, J. Landon, M. Finkelstein, P. Jhakra, J. Frye, M. Barnes, M. Mauger, D. Daun, K. Baatarsukh, M. Tung, W. Farhan, H. Michalewski, F. Viola, F. de Chaumont Quitry, C. L. Lan, T. Hudson, Q. Wang, F. Fischer, I. Zheng, E. White, A. Dragan, J. Alayrac, E. Ni, A. Pritzel, A. Iwanicki, M. Isard, A. Bulanova, L. Zilka, E. Dyer, D. Sachan, S. Srinivasan, H. Muckenhirn, H. Cai, A. Mandhane, M. Tariq, J. W. Rae, G. Wang, K. Ayoub, N. FitzGerald, Y. Zhao, W. Han, C. Alberti, D. Garrette, K. Krishnakumar, M. Gimenez, A. Levskaya, D. Sohn, J. Matak, I. Iturrate, M. B. Chang, J. Xiang, Y. Cao, N. Ranka, G. Brown, A. Hutter, V. Mirrokni, N. Chen, K. Yao, Z. Egyed, F. Galilee, T. Liechty, P. Kallakuri, E. Palmer, S. Ghemawat, J. Liu, D. Tao, C. Thornton, T. Green, M. Jasarevic, S. Lin, V. Cotruta, Y. Tan, N. Fiedel, H. Yu, E. Chi, A. Neitz, J. Heitkaemper, A. Sinha, D. Zhou, Y. Sun, C. Kaed, B. Hulse, S. Mishra, M. Georgaki, S. Kudugunta, C. Farabet, I. Shafran, D. Vlasic, A. Tsitsulin, R. Ananthanarayanan, A. Carin, G. Su, P. Sun, S. V, G. Carvajal, J. Broder, I. Comsa, A. Repina, W. Wong, W. W. Chen, P. Hawkins, E. Filonov, L. Loher, C. Hirnschall, W. Wang, J. Ye, A. Burns, H. Cate, D. G. Wright, F. Piccinini, L. Zhang, C. Lin, I. Gog, Y. Kulizhskaya, A. Sreevatsa, S. Song, L. C. Cobo, A. Iyer, C. Tekur, G. Garrido, Z. Xiao, R. Kemp, H. S. Zheng, H. Li, A. Agarwal, C. Ngani, K. Goshvadi, R. Santamaria-Fernandez, W. Fica, X. Chen, C. Gorgolewski, S. Sun, R. Garg, X. Ye, S. M. A. Eslami, N. Hua, J. Simon, P. Joshi, Y. Kim, I. Tenney, S. Potluri, L. N. Thiet, Q. Yuan, F. Luisier, A. Chronopoulou, S. Scellato, P. Srinivasan, M. Chen, V. Koverkathu, V. Dalibard, Y. Xu, B. Saeta, K. Anderson, T. Sellam, N. Fernando, F. Huot, J. Jung, M. Varadarajan, M. Quinn, A. Raul, M. Le, R. Habalov, J. Clark, K. Jalan, K. Bullard, A. Singhal, T. Luong, B. Wang, S. Rajayogam, J. Eisenschlos, J. Jia, D. Finchelstein, A. Yakubovich, D. Balle, M. Fink, S. Agarwal, J. Li, D. Dvijotham, S. Pal, K. Kang, J. Konzelmann, J. Beattie, O. Dousse, D. Wu, R. Crocker, C. Elkind, S. R. Jonnalagadda, J. Lee, D. Holtmann-Rice, K. Kallarackal, R. Liu, D. Vnukov, N. Vats, L. Invernizzi, M. Jafari, H. Zhou, L. Taylor, J. Prendki, M. Wu, T. Eccles, T. Liu, K. Kopparapu, F. Beaufays, C. Angermueller, A. Marzoca, S. Sarcar, H. Dib, J. Stanway, F. Perbet, N. Trdin, R. Sterneck, A. Khorlin, D. Li, X. Wu, S. Goenka, D. Madras, S. Goldshtein, W. Gierke, T. Zhou, Y. Liu, Y. Liang, A. White, Y. Li, S. Singh, S. Bahargam, M. Epstein, S. Basu, L. Lao, A. Ozturel, C. Crous, A. Zhai, H. Lu, Z. Tung, N. Gaur, A. Walton, L. Dixon, M. Zhang, A. Globerson, G. Uy, A. Bolt, O. Wiles, M. Nasr, I. Shumailov, M. Selvi, F. Piccinno, R. Aguilar, S. McCarthy, M. Khalman, M. Shukla, V. Galic, J. Carpenter, K. Villela, H. Zhang, H. Richardson, J. Martens, M. Bosnjak, S. R. Belle, J. Seibert, M. Alnahlawi, B. McWilliams, S. Singh, A. Louis, W. Ding, D. Popovici, L. Simicich, L. Knight, P. Mehta, N. Gupta, C. Shi, S. Fatehi, J. Mitrovic, A. Grills, J. Pagadora, T. Munkhdalai, D. Petrova, D. Eisenbud, Z. Zhang, D. Yates, B. Mittal, N. Tripuraneni, Y. Assael, T. Brovelli, P. Jain, M. Velimirovic, C. Akbulut, J. Mu, W. Macherey, R. Kumar, J. Xu, H. Qureshi, G. Comanici, J. Wiesner, Z. Gong, A. Ruddock, M. Bauer, N. Felt, A. GP, A. Arnab, D. Zelle, J. Rothfuss, B. Rosgen, A. Shenoy, B. Seybold, X. Li, J. Mudigonda, G. Erdogan, J. Xia, J. Simsa, A. Michi, Y. Yao, C. Yew, S. Kan, I. Caswell, C. Radebaugh, A. Elisseeff, P. Valenzuela, K. McKinney, K. Paterson, A. Cui, E. Latorre-Chimoto, S. Kim, W. Zeng, K. Durden, P. Ponnapalli, T. Sosea, C. A. Choquette-Choo, J. Manyika, B. Robenek, H. Vashisht, S. Pereira, H. Lam, M. Velic, D. Owusu-Afriyie, K. Lee, T. Bolukbasi, A. Parrish, S. Lu, J. Park, B. Venkatraman, A. Talbert, L. Rosique, Y. Cheng, A. Sozanschi, A. Paszke, P. Kumar, J. Austin, L. Li, K. Salama, B. Perz, W. Kim, N. Dukkipati, A. Baryshnikov, C. Kaplanis, X. Sheng, Y. Chervonyi, C. Unlu, D. de Las Casas, H. Askham, K. Tunyasuvunakool, F. Gimeno, S. Poder, C. Kwak, M. Miecnikowski, V. Mirrokni, A. Dimitriev, A. Parisi, D. Liu, T. Tsai, T. Shevlane, C. Kouridi, D. Garmon, A. Goedeckemeyer, A. R. Brown, A. Vijayakumar, A. Elqursh, S. Jazayeri, J. Huang, S. M. Carthy, J. Hoover, L. Kim, S. Kumar, W. Chen, C. Biles, G. Bingham, E. Rosen, L. Wang, Q. Tan, D. Engel, F. Pongetti, D. de Cesare, D. Hwang, L. Yu, J. Pullman, S. Narayanan, K. Levin, S. Gopal, M. Li, A. Aharoni, T. Trinh, J. Lo, N. Casagrande, R. Vij, L. Matthey, B. Ramadhana, A. Matthews, C. Carey, M. Johnson, K. Goranova, R. Shah, S. Ashraf, K. Dasgupta, R. Larsen, Y. Wang, M. R. Vuyyuru, C. Jiang, J. Ijazi, K. Osawa, C. Smith, R. S. Boppana, T. Bilal, Y. Koizumi, Y. Xu, Y. Altun, N. Shabat, B. Bariach, A. Korchemniy, K. Choo, O. Ronneberger, C. Iwuanyanwu, S. Zhao, D. Soergel, C. Hsieh, I. Cai, S. Iqbal, M. Sundermeyer, Z. Chen, E. Bursztein, C. Malaviya, F. Biadsy, P. Shroff, I. Dhillon, T. Latkar, C. Dyer, H. Forbes, M. Nicosia, V. Nikolaev, S. Greene, M. Georgiev, P. Wang, N. Martin, H. Sedghi, J. Zhang, P. Banzal, D. Fritz, V. Rao, X. Wang, J. Zhang, V. Patraucean, D. Du, I. Mordatch, I. Jurin, L. Liu, A. Dubey, A. Mohan, J. Nowakowski, V. Ion, N. Wei, R. Tojo, M. A. Raad, D. A. Hudson, V. Keshava, S. Agrawal, K. Ramirez, Z. Wu, H. Nguyen, J. Liu, M. Sewak, B. Petrini, D. Choi, I. Philips, Z. Wang, I. Bica, A. Garg, J. Wilkiewicz, P. Agrawal, X. Li, D. Guo, E. Xue, N. Shaik, A. Leach, S. M. Khan, J. Wiesinger, S. Jerome, A. Chakladar, A. W. Wang, T. Ornduff, F. Abu, A. Ghaffarkhah, M. Wainwright, M. Cortes, F. Liu, J. Maynez, A. Terzis, P. Samangouei, R. Mansour, T. Kępa, F. Aubet, A. Algymr, D. Banica, A. Weisz, A. Orban, A. Senges, E. Andrejczuk, M. Geller, N. D. Santo, V. Anklin, M. A. Merey, M. Baeuml, T. Strohman, J. Bai, S. Petrov, Y. Wu, D. Hassabis, K. Kavukcuoglu, J. Dean, and O. Vinyals Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. External Links: 2403.05530, Link Cited by: Table 5, §4.1. Wang et al. (2024) J. Wang, D. Paliotta, A. May, A. M. Rush, and T. Dao The mamba in the llama: distilling and accelerating hybrid models. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, pp. 62432-62457. External Links: Link Cited by: §6. Xia et al. (2025) M. Xia, V. Ruehle, S. Rajmohan, and R. Shokri Minerva: a programmable memory test benchmark for language models. arXiv preprint arXiv:2502.03358. External Links: 2502.03358, Link Cited by: §6. Xiao et al. (2024) G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. External Links: 2309.17453, Link Cited by: §6. Xiong et al. (2024) Z. Xiong, V. Papageorgiou, K. Lee, and D. Papailiopoulos From artificial needles to real haystacks: improving retrieval capabilities in llms by finetuning on synthetic data. arXiv preprint arXiv:2406.19292. External Links: 2406.19292, Link Cited by: §6. Xu et al. (2024) X. Xu, Q. Ye, and X. Ren Stress-testing long-context language models with lifelong icl and task haystack. arXiv preprint arXiv:2407.16695. External Links: 2407.16695, Link Cited by: §1, §2, §6. Yang et al. (2025) A. Yang, B. Yu, C. Li, D. Liu, F. Huang, H. Huang, J. Jiang, J. Tu, J. Zhang, J. Zhou, J. Lin, K. Dang, K. Yang, L. Yu, M. Li, M. Sun, Q. Zhu, R. Men, T. He, W. Xu, W. Yin, W. Yu, X. Qiu, X. Ren, X. Yang, Y. Li, Z. Xu, and Z. Zhang Qwen2.5-1m technical report. External Links: 2501.15383, Link Cited by: Table 5, §1, §4.1. Yang et al. (2024) S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635. External Links: 2312.06635, Link Cited by: §6. Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.), Brussels, Belgium, pp. 2369-2380. External Links: Link, Document Cited by: §A.3, §2, §4.1. Ye and Chao (2022) H. Ye and W. Chao How to train your MAML to excel in few-shot classification. In International Conference on Learning Representations, External Links: Link Cited by: Appendix B. Ye et al. (2025) X. Ye, F. Yin, Y. He, J. Zhang, H. Yen, T. Gao, G. Durrett, and D. Chen LongProc: benchmarking long-context language models on long procedural generation. arXiv preprint arXiv:2501.05414. External Links: 2501.05414, Link Cited by: §6. Yen et al. (2024) H. Yen, T. Gao, and D. Chen Long-context language modeling with parallel context encoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 2588-2610. External Links: Link, Document Cited by: §6. Yen et al. (2025) H. Yen, T. Gao, M. Hou, K. Ding, D. Fleischer, P. Izsak, M. Wasserblat, and D. Chen HELMET: how to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694. External Links: 2410.02694, Link Cited by: §A.3, §4.1, §4.1, §6. Zhang et al. (2024a) X. Zhang, Y. Zhang, D. Long, W. Xie, Z. Dai, J. Tang, H. Lin, B. Yang, P. Xie, F. Huang, M. Zhang, W. Li, and M. Zhang MGTE: generalized long-context text representation and reranking models for multilingual text retrieval. External Links: 2407.19669, Link Cited by: §A.3, §4.1. Zhang et al. (2024b) X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. Hao, X. Han, Z. Thai, S. Wang, Z. Liu, and M. Sun ∞\inftyBench: Extending long context evaluation beyond 100K tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 15262-15277. External Links: Link, Document Cited by: §6. Appendix A Datasets In this section, we provide details on the datasets used for each task in our experiments and analysis. For each dataset, we visualize an example in Figure 7. A.1 BabiLong We construct the train and evaluation data using the data generation framework proposed by Kuratov et al. (2024). As we mentioned in the experiment setup, we mainly focus on tasks involving single-hop (QA1), two-hop (QA2), and three-hop (QA3) reasoning. Tasks The main task for BabiLong is to reason over facts distributed in a long context. For each QA task, we have a set of facts, including distractors and supporting facts. Each fact describes an agent's action or movement (which also describes the agent's location). For example, Mary journeyed to the garden is a movement, where Mary is the agent, and garden is Mary's current location. As an example of actions, John picked up the apple there describes John's action of picking up an apple. Together with multiple agents performing multiple movements and actions, the set of facts became a temporal sequence of events. QA1 focuses on one-hop reasoning, where the problems can be answered by using a single supporting fact. For example, in a sequence of actions and movements, the agent John's last movement is John moved to the bedroom. Then the one-hop question is “Where is John located now?”. By the last movement alone, we can find the answer “bedroom”. QA2 focuses on one-hop reasoning, where the problems can be answered by using two supporting facts, where one fact is a movement and the other one is an action. For example, in the event sequence, we have John moved to the garden and John dropped the apple there. Using these two facts, we can answer “Where is the apple now?”, which is “garden”. QA3 focuses on one-hop reasoning, where the problems can be answered by using three supporting facts, as a combination of movements and actions. For example, we have three supporting facts: Daniel went back to the office, Daniel went back to the bedroom, and Daniel left the milk in order. We can infer the answer to the question “Where was the milk before the bedroom?” is “office”. Build Long Context To build the long context, we use the framework to construct the tasks with long natural documents sampled from FineWeb-edu (Lozhkov et al., 2024), specifically from the CC-MAIN-2024-42 subset, ensuring the content postdates our base model's knowledge cutoff. Given a particular context window, such as 8192, we sample that many tokens from the corpus as the Haystack long context. We need to convert a long context into a batch of subsequences, such that PERK can encode them in parallel through test-time gradient-based adaptation. In practice, we set the effective context length (i.e., the number of tokens of a subsequence on average) as 256. We divide the long context into chunks where each chunk is approximately 256 tokens long. Then, we randomly distribute the set of facts into these chunks. For each fact, we randomly select a chunk and then insert this fact into the chunk at a random position. To preserve the temporal order of the facts (since they come from a sequence of events with temporal order), we index each fact with a prefix. Finally, we have a batch of subsequences where each sequence might contain some facts at random positions. A.2 Student Records In the main paper, we propose Drops-in-the-Ocean (DIO), a new evaluation setting where long contexts are formed from structurally similar documents. We aim to create problems in which the relevant and irrelevant information will likely be distributionally similar, making it more difficult to identify critical facts. We construct a synthetic dataset, Student Records, where each context simulates a database containing multiple student records. Each record includes several attributes: ID, name, school, major, and grade. Tasks The task operates on top of a database that consists of multiple records. Each record holds information about a student. The record is structured as follows: Student ID: 3109998, Student Name: Alison Keith, Year: Sophomore School: School of Computer Information and Data Sciences, Major: Computer Science, Grade: 72 We generate the student IDs and grades using a random number generator. The student names are generated from the Name package of Python. We define a pool of years, schools, and courses. Then, we randomly select each entity for each student. Crucially, to test generalization, specific entities (e.g., IDs, names) and attribute value combinations encountered in the test set are disjoint from those used during training. Based on the database of student records, we construct three main tasks: Recall (retrieving attributes for a specific ID), Relation (comparing attributes between two IDs), and Aggregate (calculating the maximum, minimum, and average grade across all students). Recall tests a model's ability to retrieve an attribute of a student given a specific student ID number. For example, we can ask “What major does student 1778924 study?”. The model should answer “What major does student 1778924 study?”. The questions cover all attributes for testing coverage. Relation tests a model's ability to compare the attributes of two students. For example, for the attribute of grade, we can ask “Does student 1778924 have a higher grade than student 7077015?”. The model should be able to compare their grades and infer “Yes”. Or we can also ask “Do student 1778924 and student 7077015 study the same major?”, the model should recognize that their majors are different and answer “No”. Aggregation is the most difficult task in this dataset. This task requires many-hop reasoning ability, where the model needs to aggregate attributes from all students in a long context to make an accurate conclusion. We focus on three types of numerical aggregation among the students' grades: (1) Maximization, (2) Minimization, and (3) Averaging. For example, we ask the model “What is the average grade of all students”, the model should be able to compute the sum of all student grades, count the total number of students in the context, and then compute the average, “49.3”. Figure 7: Dataset Examples: Here we show an example from the three datasets we used: (Top) BabiLong, (Middle) Student Records, and (Bottom) Lost-in-the-Middle API Retrieval. PERK first encodes the contexts into the memory scratchpad through self-supervised test-time learning. Next, the user can ask any number of questions about that context, and the LLM can reason over the updated memory scratchpad to generate the answers. Build Long Context Context length scales directly with the number of student records included. For a given context window, we continuously append student records to the context until the number of tokens exceeds the context window. To convert the long context into a batch of subsequences for PERK, we simply treat each student record as an individual subsequence in the batch, since they are already independent documents by design. A.3 Open-Domain Multi-doc QA We select HotpotQA (Yang et al., 2018) and TriviaQA (Joshi et al., 2017) as the main evaluation for open-domain QA over long-context documents. HotpotQA tests the model's ability to perform 2-hop reasoning over two gold documents randomly located in a long list of documents. TriviaQA tests the model's ability to do reading comprehension over very long documents where the target information can be scattered around the long context. We refer to the original papers for QA construction details, data sources, and task examples of these two benchmarks. Build Long Context To build long contexts for these two benchmarks, given a specific context window (for example, 32768 tokens), we follow Yen et al. (2025)'s process for their RAG evaluation. We extend the original context by injecting retrieved distractor documents (Zhang et al., 2024a) at random positions until reaching the context window limit. A.4 Lost-in-the-Middle API Retrieval To analyze a model's robustness against relevant information positions, we follow the setting proposed by Lost in the Middle (Liu et al., 2024). We use API documents from Patil et al. (2024), where each document has an associated API call, and create long contexts containing multiple API-call-to-document pairs. We place the relevant API document in a particular user instruction at different locations in the context (beginning, middle, end, and random position). Task We formulate the main task as gold API retrieval. In this task, the model must retrieve the correct API call based on user instructions and documents. Using fixed 4k and 8k token contexts, we train PERK and FT-ICR models on contexts with the target document at varying positions and test them on contexts distributed across all positions. Build Long Context We follow Student Records and scale the context length with the number of API documents included. For a given context window, we continuously append API documents to the context until the number of tokens exceeds the context window. Similarly, since each API document is independent, we treat a document as a subsequence in the batch for PERK. Since the documents are processed in parallel as a batch, they do not preserve any ordering information. Thus, by design, PERK is robust to the position of the relevant information in a context. Appendix B Training and Inference Details Hardware Information All training runs are conducted on a single Nvidia H100 GPU with 93GB of VRAM. The training jobs run on an internal research cluster managed by Slurm. Each job is provided with 64 CPUs and 64GB of RAM. The GPU is completely utilized without sharing with other concurrent jobs. Standard Hyperparameters Each training (for PERK and baselines) sets 2 epochs as the maximum runtime with early stopping based on validation loss. In practice, most training runs typically stop early, within one epoch of runtime. We set the baseline training and PERK's outer loop learning rate to 1e-5, and the weight decay to 0.01. We use a cosine learning rate scheduler with 0.03 of the total optimization steps as the warm-up phase. We use PyTorch Lightning 11 1 https://lightning.ai/docs/pytorch/stable/ as the trainer for PERK. The Qwen and GPT-2 FT-ICR baselines are trained using the Open-Instruct (Lambert et al., 2025) trainer from AI2, and the Mamba FT-ICR baselines are trained using the Huggingface Trainer 22 2 https://huggingface.co/docs/transformers/en/main_classes/trainer. For the stress test of length generalization beyond 64K tokens, we change the default maximum position of 32768 to 131072. LoRA Hyperparameters We set LoRA rank to 256 for both PERK GPT-2 and PERK Qwen. We use the default values for the other hyperparameters, including setting the alpha to 16 and the dropout to 0.1. We apply LoRA to all modules in the model architecture. For 1B scale and below, we apply LoRA to all layers in the model. However, for 7B and 8B models, we apply it only to the top 4 layers in order to reduce the memory overhead of meta-learning. We set the scaling factor α\alpha to 16 initially and enable rank-stabilized LoRA fine-tuning Kalajdzievski (2023), which results in a target α\alpha of 256, corresponding to a rank of 256. Meta-Learning Hyperparameters For the inner loop adaptation, we use 4 inner loop gradient steps. For the truncated gradient unrolling, we truncate steps 1 and 2 for input context length ≤\leq4K, and we truncate steps 1, 2, and 3 for context length >> 8K. We follow Ye and Chao (2022)'s practice on how to train a good MAML model and meta-learn a set of per-layer-per-step dynamic learning rates for the inner loop as part of the outer loop learning, using 5e-5 as the initial value. This approach learns a separate learning rate for each network layer and each adaptation step in the inner loop. These learning rates are optimized during the meta-training process alongside the model parameters. Note that these per-layer-per-step learning rates are learned parameters, not hyperparameters that require manual tuning. During meta-training, our algorithm automatically learns to adjust these learning rates for each layer and inner loop step dynamically based on the meta-objective. This approach reduces the hyperparameter tuning burden compared to manually setting the adaptive learning rates. For the language modeling loss in the inner loop, we use a weighted version similar to CaMeLS (Hu et al., 2023). The token-level weights are predicted via a weighting model implemented as a two-layer MLP network. The parameters of the weighting model are co-meta-learned along with the LoRA parameters and the adaptive per-layer-per-step learning rates. We use the differentiable version of the AdamW (Loshchilov and Hutter, 2017) optimizer for the inner loop optimization, implemented by the Higher 33 3 https://github.com/facebookresearch/higher library for higher-order optimization. We compute higher-order derivatives using the automatic differentiation mechanism implemented by PyTorch44 4 https://docs.pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html. The outer loop optimizer is the standard PyTorch implementation of AdamW. Long-context Data Collation for PERK At training time, to collate a reasoning problem with a long context and a set of associated question-answer pairs, we first convert a long context into a batch of subsequences, with relatively even lengths. In practice, we set the effective context length (i.e., the number of tokens of a subsequence on average) as 256, and divide the long context into NN chunks where each chunk is approximately 256 tokens long. For a long context with 8192 tokens, the batch size is thus N=32N=32. PERK's inner loop adaptation encodes the batch of subsequences in parallel through gradient updates. At inference time, if we want to test PERK on a long context with 32K tokens, the effective context length stays the same as approximately 256 tokens, while the batch size increases to N=128N=128. Note that splitting a long context into a batch of shorter sequences can either be permutation-invariant or maintain the original ordering. PERK induces ordering through explicit indexing in the prompt formatting, which allows the model to learn implicit ordering during the encoding phase. Specifically, when chunking a long context, we can preserve ordering information by indexing each chunk. For example, if we split a long document into segments A, B, C, D, E, we format them as: ["Document 1: A", "Document 2: B", …, "Document 5: E"]. During the inner loop adaptation with the NLL objective, the LoRA parameters learn to associate these indices with their content, effectively encoding the ordering information. Evaluation Hyperparameters At inference time, we generate the answer for a reasoning problem via greedy decoding (temperature is set to 0). The maximum number of tokens that can be generated is 512. The EOS token is set to the default one defined in the tokenizer. The generation is handled by the text-generation pipeline implemented by Huggingface 55 5 https://huggingface.co/docs/transformers/v4.43.4/en/main_classes/pipelines. For the length generalization analysis, we apply recent training-free methods Yarn (Peng et al., 2023b) and Dual Chunk Attention (DCA, An et al. (2024a)) to the finetuned Qwen models. We use vLLM 66 6 https://docs.vllm.ai/en/latest (0.10.2) as the inference engine to apply Yarn + DCA. For Yarn, we apply a factor of 4.0 to the original max position embeddings, 32768, and scale the context window to 131072. For DCA, we use vLLM's DUAL_CHUNK_FLASH_ATTN backend for generation. Appendix C Additional Analysis C.1 Impact of LoRA Rank in PERK Method QA1 QA2 QA3 FT-ICR 91.1 70.5 48.6 PERK LoRA-16 99.0 96.9 84.7 PERK LoRA-256 100 98.1 89.1 Table 2: Analyzing LoRA rank's impact on PERK's performance. In this section, we analyze the impact of LoRA's rank on PERK's long-context reasoning performance. We focus on the BabiLong tasks with a 1024 context window for an efficient ablation study. We apply PERK to Qwen-2.5-0.5B with LoRA ranks of 16 and 256 and evaluate on the BabiLong tasks. We show the results in Table 2. We also include the performance from the FT-ICR Qwen baseline as a reference. The results show that PERK's performance only decreases slightly when the rank is reduced to 16. However, PERK still maintains a much stronger performance than the FT-ICR Qwen baseline. The analysis shows that PERK can maintain its strong performance with a more parameter-efficient setting, highlighting its potential for more efficient long-context reasoning. C.2 Inner Loop Adaptation Steps Method QA1 QA2 QA3 FT-ICR 91.1 70.5 48.6 PERK 2 Steps 99.2 94.5 82.4 PERK 4 Steps 100 98.1 89.1 Table 3: Analyzing the impact of the number of inner loop adaptation steps on PERK's performance. In this section, we analyze the impact of inner loop adaptation steps on PERK's long-context reasoning performance. Similarly, we focus on the BabiLong tasks with a 1024 context window. We again apply PERK to Qwen-2.5-0.5B for 2 and 4 inner loop steps. With 4 steps, we truncate the first 2 steps in backpropagation. With 2 steps, we truncate the first step in backpropagation. We show the results in Table 3. The results show that reducing the number of inner loop adaptation steps decreases the performance more noticeably than reducing LoRA rank. However, PERK still outperforms FT-ICR Qwen on all three tasks. The results here suggest that we should select a higher number of inner loop adaptation steps to maximize performance. In settings where GPU memory might be limited, reducing the number of inner loop steps can reduce performance, but it is still able to outperform the baseline. However, the analysis is done with limited hardware resources, which limits our analysis to 4 steps. To further analyze the impact of the inner loop step counts, more systematic experiments in a scalable hardware setting are needed for future work. C.3 Gradient Unrolling Truncation Steps Truncation Length Accuracy TGU (0) 100 TGU (1) 100 TGU (2) 100 TGU (3) 98.8 TGU (4) 92.4 Table 4: Truncation length ablations on the BABILong QA1 task with 1K tokens. We conduct 4 steps of inner loop adaptation and truncate 0-4 steps. PERK maintains the accuracy up to 3 truncation steps, but drops when all steps are truncated. In this section, we analyze the impact of the truncation steps we use when performing gradient unrolling (GU) for computing the meta-gradients in the outer loop of the meta-learning process. Recall that to reduce the memory cost of full gradient unrolling, which requires saving the computation graph for each inner loop adaptation step, we truncate the unrolling by detaching the early steps' graph to save memory. Bounded by memory constraints, we can only train on 1K-token contexts with full GU without any truncation. Thus, we conduct our analysis under the 2K context window. Given 4 inner loop adaptation steps, we truncate 0-4 steps with Qwen2.5-0.5B on the BabiLong-QA1 task. The analysis shows that we can truncate up to 3 steps and only unroll the last inner loop adaptation while maintaining a high accuracy without significant degradation. However, truncating all steps, which results in a first-order approximation of the meta-gradients, shows a large performance drop, even on this very simple setting with QA1-1K. Figure 8: Peak GPU memory usage during training with context lengths ranging from 1K to 8K tokens for RECKONING and PERK. While PERK successfully scales to 8K tokens, RECKONING encounters out-of-memory (OOM) errors at shorter context lengths. Appendix D Training Memory Cost Analysis We show PERK's training scalability by measuring its peak memory use across different context lengths (compared to a prior test-time learning method, RECKONING; Chen et al., 2023b). Setup We complete all runs on a single Nvidia H100 GPU with 93 GB of VRAM using Huggingface's Transformers library version 4.51.3. We perform four gradient update steps for the training-time inner loop adaptations for PERK. We test four truncation levels (§3.2): truncate 0 (full gradient unrolling without truncation), 1, 2, and 3 steps. Both methods undergo 10 hardware warm-up iterations. PERK is more memory efficient at training. Figure 8a shows that, as the training context length grows from 1K to 8K tokens, RECKONING quickly runs into Out-Of-Memory (OOM) errors at context length 2K. At the same time, PERK, with increasing truncation steps, continues to fit the maximum GPU memory, successfully validating PERK's training scalability on long contexts. Appendix E Detailed Results We show detailed quantitative performance of PERK, FT-ICR baselines, and frontier long-context LLMs on the long-context reasoning benchmarks we evaluate on in Table 5. BBL-QA1 BBL-QA2 BBL-QA3 HotpotQA TriviaQA SR-Recall SR-Relation SR-Aggregate AVG Model 8K 32K 8K 32K 8K 32K 8K 32K 8K 32K 8K 32K 8K 32K 8K 32K 8K 32K Commercial Frontier Models Gemini-1.5-pro 95.6 87.7 64.0 56.4 49.4 36.7 75.3 73.4 76.4 75.1 100 99.4 63.3 58.8 50.9 48.7 71.7 67.0 GPT-4.1 91.4 87.8 84.5 80.6 54.9 33.1 72.4 65.8 78.3 74.3 100 98.8 66.5 62.5 52.4 49.4 75.1 69.0 Trained on contexts with >256K tokens Qwen2.5-7B-1M 77.4 65.7 50.2 35.3 30.5 27.2 60.3 55.8 72.8 70.9 92.7 86.7 58.2 52.1 39.4 15.8 60.2 51.2 ProLong-8B-512K 65.3 41.0 26.2 29.0 27.8 24.4 62.4 57.3 70.4 68.2 94.5 89.2 60.1 54.4 24.3 12.1 53.9 47.0 Trained on contexts with 8K tokens FT-ICR (Mamba-1.4B) 24.2 12.4 22.2 8.5 18.5 6.8 17.2 9.8 32.1 20.5 12.1 5.4 73.0 52.3 30.5 10.9 28.7 15.8 FT-ICR (Qwen2.5-0.5B) 76.1 43.8 64.3 39.4 37.5 12.1 24.4 11.9 47.4 41.7 93.6 84.5 65.3 57.8 51.8 42.4 57.6 41.7 FT-ICR (Qwen2.5-7B) 97.5 80.9 69.4 42.7 39.4 20.8 42.4 39.8 55.4 51.7 98.8 91.5 69.7 62.4 57.6 45.8 66.3 54.5 PERK (Ours), trained to encode contexts with 8K tokens PERK (Qwen2.5-0.5B) 99.0 95.3 92.4 80.9 60.4 38.7 52.2 49.4 65.5 62.8 95.8 90.4 67.9 60.9 60.1 49.5 74.2 66.0 PERK (Qwen2.5-7B) 100 94.2 96.1 84.4 65.2 44.5 55.6 52.2 70.8 69.4 100 98.5 75.5 70.3 62.4 53.3 78.2 70.9 Table 5: Model Performance Across Long-context Reasoning Benchmarks and Context Lengths (8K, 32K): We list detailed performance of our method PERK, FT-ICR baselines, and the frontier specialized long-context LLMs: Qwen2.5-7B-Instruct-1M (Yang et al., 2025), ProLong-LLaMA3-8B-Instruct-512K (Gao et al., 2025a), GPT-4.1 (OpenAI, 2025), Gemini-1.5-pro (Team et al., 2024). BBL here stands or BabiLong (Kuratov et al., 2024). SR stands for Student Records, our proposed dataset for Drops-in-the-Ocean reasoning. PERK and FT-ICR models have only seen training data contexts with 8K tokens, requiring them to generalize to the unseen 32K context window. PERK achieves the highest performance on average for both 8K-token and 32K-token contexts. Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" () button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. We gratefully acknowledge support from our major funders, member institutions, , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2511.02817.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2511.02817.txt new file mode 100644 index 0000000..ee09606 --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2511.02817.txt @@ -0,0 +1 @@ +Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Oolong-synth 2.1 Data Data filtering 2.2 Context window construction Questions Instructions 2.3 Evaluation Baseline Parsing answers Scoring An iterative system? 3 Oolong-real 4 Results and Analysis 4.1 Does reasoning effort correlate to performance? Qwen and Olmo: think versus instruct models. GPT-5-nano: reasoning effort parameter. Gemini 3 Pro: reasoning token count. 4.2 Over-reasoning Deepseek R1 Gemini models 4.3 Refusal 4.4 Aggregation without classification Providing labels Providing domain knowledge 4.5 Iterative retrieval baseline for Oolong-real 5 Related Work 6 Conclusion 7 Acknowledgements References A Oolong - Leaderboard B Additional Results B.1 Reasoning effort on Oolong-real B.2 Data contamination for Oolong-real C Additional Related Work Dungeons & Dragons data Aggregation as an NLP concept D ICL label validation E Question types E.1 Oolong-synth E.2 Oolong-real F Deepseek R1 trace examples G Question type analysis G.1 Oolong-synth G.2 Oolong-real H Oolong-real data H.1 Data quality and curation H.2 Challenges of Oolong-real License: arXiv.org perpetual non-exclusive license arXiv:2511.02817v2 [cs.CL] 05 Oct 2026 Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities Amanda Bertsch Adithya Pratapa ††thanks: Work conducted while the author was a student at Carnegie Mellon University. Teruko Mitamura Graham Neubig Matthew R. Gormley Carnegie Mellon University Email: {abertsch,vpratapa,teruko,gneubig,mgormley}@cs.cmu.edu Affiliation: abertsch72/oolong oolongbench/oolong-[synth, real] Abstract Many important applications of long-context models require reasoning over large collections of items-summarizing trends across thousands of customer interactions, identifying patterns in research corpora, or analyzing behavioral data at scale. Evaluating whether models can analyze items atomically and aggregate findings requires information-dense tasks that make use of the full context, like writing a summary or describing trends across a collection. Yet these types of tasks often result in long-form outputs which are challenging to evaluate; as a result, current long-context benchmarks predominantly test retrieval-style tasks, where the correct answer depends on a small fraction of tokens and the rest is noise. We introduce Oolong, a benchmark of distributional reasoning tasks that require analyzing individual chunks of text atomically and aggregating results. Oolong includes synthetic tasks (Oolong-synth) that enables controlled ablation of reasoning components, and a downstream setting (Oolong-real) that requires reasoning over realistic conversational data. Tasks involve classification, counting, filtering by metadata, and temporal reasoning. Frontier models struggle at moderate context lengths at this task, suggesting Oolong captures capabilities not yet addressed by scale alone. We release the data and evaluation harness for Oolong to enable further development of models that can reason over large quantities of text. 1 Introduction In the last several years, the exponentially increasing context lengths of LLMs have enabled many new applications, including reasoning models (Guo et al., 2025), many-shot prompting (Bertsch et al., 2024), and repository-level code generation (Jimenez et al., 2023). However, despite the rapid improvement in this area, long-context aggregation remains a challenging capability to study. Tasks like summarization are realistic but produce long and subjective outputs that are challenging to evaluate. Other tasks require only a subset of the input, which could be extracted from the long context before solving (e.g. using a RAG system). Multi-step reasoning datasets make progress towards measuring this capability, but often lack realism - inputs may be artificial (e.g. random repeated words in RULER (Hsieh et al., 2024)), the posed questions may be unrealistic (e.g. “identify the second poem about this topic” in MRCR (OpenAI, 2025)), and the relevant context may be embedded in distractor text from a different domain or a well-known text (e.g. small state updates embedded in Project Gutenberg novels in BABILong (Kuratov et al., 2024)). We argue that aggregation is best measured in realistic settings with multiple types of aggregative component tasks- including the ability to draw inferences across different sections of input and perform basic numeracy operations with this information. We further discuss these desiderata in Table 1. We propose Oolong, a benchmark that requires multi-hop reasoning over long inputs to produce easily verifiable outputs. Oolong is separated into two task sets, presented in Figure 1. Oolong-synth (§2) is a set of naturalistic synthetic tasks constructed from existing in-context learning datasets. These tasks require implicitly labeling the examples in-context to reason over distributional properties of the labels (counting tasks), over user-specific patterns (user tasks), and over changes in label distributions over time (temporal tasks). Oolong-real (§3) poses these types of questions over real data that is not so easily separable into component parts- asking challenging questions about character actions from live-play Dungeons & Dragons transcripts, using human-annotated gold answers. Figure 1: Oolong poses questions that require performing a multi-step information aggregation process to determine the solution. Oolong-synth uses ICL-based tasks, which could be easily decomposed and solved iteratively, as a proxy for real-world aggregation tasks over long inputs. Oolong-real poses challenging information aggregation questions over transcripts from live-action Dungeons & Dragons shows, which can not be easily decomposed into component pieces. Each task requires identifying the relevant segments of the input; this ranges from only a handful of instances or lines of dialogue to questions that require the use of every line of the input. The relevant segments must be classified or categorized, and these individual decisions must be aggregated to produce a final answer. By framing the benchmark around problems that are simple individually, Oolong measures capability on long context reasoning and not accuracy on the underlying task. By requiring identification of relevant context, classification decisions over that context, and numeracy skills to produce statistics about the input, Oolong requires models to perform multi-step reasoning over long inputs. In Table 1, we compare Oolong with most similar long-context reasoning benchmarks. Although these component capabilities have been well studied and frontier models demonstrate strong performance in each (Yen et al. (2024), Agarwal et al. (2024), inter alia), we find that models struggle with information aggregation as the input length grows. Most models score below 50% on Oolong at 128K context. We study why this task is so challenging and find that identification and aggregation of information is the bottleneck, not labeling (§4). We study the role of reasoning behavior and identify consistent error cases in reasoning models that limit their performance in this type of task, including over-reasoning by Gemini and DeepSeek models and task refusal by GPT models (§4). We believe Oolong is a usefully challenging evaluation of long-context reasoning abilities. Real vs synthetic data Aggregation capabilities Measurability Dataset Realistic inputs Realistic questions Relevant text matches distractors Multi-step reasoning Numeracy Must use full input Easy to eval MRCR ✓ ✗ ✓ ✓ ✓ ✗ ✓ BABILong ✗ ✓ ✗ ✓ ✗ ✗ ✓ RULER ✗ ✗ ✗ ✝ ✝ ✝ ✓ HELMET Summ ✓ ✓ ✓ ✓ ✗ ✓ ✗ HELMET LongQA ✓ ✓ ✓ ✓ ✗ ✝ ✓ Oolong-synth ✗ ✓ ✓ ✓ ✓ ✓ ✓ Oolong-real ✓ ✓ ✓ ✓ ✓ ✝ ✓ Legend: ✓ = yes, ✗ = no, ✝ = in some cases Table 1: Comparing long-context reasoning benchmarks. Oolong measures aggregation capabilities in controllable, realistic, and simple-to-evaluate settings. 2 Oolong-synth We aim to construct a synthetic information aggregation task that allows fine-grained control over the types of information and number of steps necessary to solve the task. To do this, we need building blocks: simple, short-context documents to build into a longer collection. We construct Oolong-synth by constructing challenging corpus-level questions over existing in-context learning (ICL) datasets. This has conceptual similarities with database reasoning (Thorne et al., 2021). 2.1 Data We collect 10 text classification datasets with between 2 and 10 labels. We select for tasks that are possible for the authors to perform without difficulty (and validate that these tasks are similarly simple for models during our dataset filtering). Table 4 describes the datasets in detail. We split the data into two validation tasks and eight test tasks. The validation tasks are selected to avoid underlying task overlap with test tasks (in the style of Min et al. (2022)), and so that there is both a 2-label and a multi-label (6-label) validation task. For the sentiment and formality datasets with more than two labels, we reduce the label space to a single positive and a single negative label, to reduce the difficulty of the task. Data filtering In a normal in-context learning task, if a few examples are unusually hard or even mislabeled, this only slightly impacts on the overall score. Because we require aggregation across many ICL examples at once, one particularly difficult example can affect many answers downstream. However, many ICL datasets are known to contain mislabelings (Ying and Thomas, 2022; Chong et al., 2022; Klie et al., 2023). We perform an additional screening step to remove mislabeled or unusually hard examples: we select two models that we do not expect to be substantially stronger than the models we are evaluating (GPT-4.1 nano and Llama 4 Maverick) and perform zero-shot ICL with a minimal instruction that provides the label space and type of task. We then exclude all examples that both models get incorrect, for exclusion rates ranging from 0% to 0.635% by dataset. We report the percentage of examples screened out by dataset and provide examples of excluded and validated datapoints for each dataset in Appendix D. Many of the excluded instances are clearly mislabeled; the remainder generally represent challenging cases. 2.2 Context window construction We determine the number of examples in the context window by using an estimate of the number of tokens required for each example, for the general task instructions, and for the specific question. Following Yen et al. (2024), we compute these averages using the Llama 2 tokenizer, and use approximately 500K tokens of context for each estimate. We then estimate the number of examples to use by determining how many average-length examples would reach 95% of the target context length. We sample a distribution over the label classes so that the model cannot use any information about the true distribution over labels. Then, we sample examples to fill this distribution, and sample user IDs and dates.Once a context window is constructed, we use it for 25 questions to allow for prompt caching, reducing the time to evaluate local models and the cost to evaluate on most APIs. At each context length, we sample two context windows per dataset, for a total of 50 questions per dataset or 400 test split questions per context length. We construct Oolong-synth questions for every power of 2 from 1K to 4M. Questions We construct three types of Oolong-synth questions, in order of increasing complexity. Counting questions concern simple statistical properties of the label distribution, ranging from identifying the most frequent label to determining the number of examples with each label class. If the label distribution was provided, all tasks in this set should be trivial; these tasks can be seen implicitly as the task of labeling every example and reporting summary statistics. User information questions require additional cross-reference with the user ID field. Timeline questions ask about changes in distribution before or after a certain date, between years, or between months across years. This is more challenging than user information questions because it requires reasoning about before or after a date, rather than matching a list of IDs (see §E.1 in Appendix for questions). Instructions For each dataset, we provide a brief (one-sentence) description of the task and label space, as well as the number of examples in-context. We provide instructions at the start and end of the input, as suggested by OpenAI's long context prompting guide.11 1 https://cookbook.openai.com/examples/gpt4-1_prompting_guide#prompt-organization However, we provide the question only at the end, to enable prompt caching. 2.3 Evaluation Baseline A random baseline is non-trivial for these tasks, because the output space for each question varies. We construct an algorithm for a random baseline with the following rules: (1) in cases where there is an nn-way choice (e.g. choosing between nn labels or choosing a month where some criteria occurs), we select an answer from the set of valid answers at random; (2) if the question requires a numerical answer, we return N/|L|N/|L|, where NN is the number of data points in context and LL is the labelset; (3) if the question requires a date or user ID, we sample at random from the list of dates/IDs in context. We compute the random baseline performance as the expected value of this procedure. Parsing answers The task prompts specify an output format. We make a best-effort attempt to parse answers using this format; if this template is not present in the output, we take the candidate answer to be the last substring in the output that appears to match the desired answer type. Scoring For questions that require a label, date, user ID, or comparison (e.g. “greater than” or “less than”), we score on exact match. For questions that require a numerical answer, we set the score to be score​(y^)=0.75|y−y^|\texttt{score}(\hat{y})=0.75^{|y-\hat{y}|} to allow for partial credit. An iterative system? Although Oolong-synth allows for a high degree of control over the types of input, these inputs are composed of independent examples from the source dataset. In theory, an iterative system could complete these tasks by calling a model to label each individual example and then deterministically aggregating the resulting labels. However, such an iterative system could theoretically handle many existing evaluation tasks; for instance, Needle-in-a-Haystack tasks could be solved almost trivially by asking a model if each of the NN input sentences individually contains the needle. We believe that the Oolong-synth setting reflects the way many real-world users interact with current large language models. Real user queries tend to be underspecified and rarely use prompt engineering strategies (Sarkar et al., 2025; Xue et al., 2025). When users do edit prompts, they are more likely to add additional context or more detailed instructions, rather than removing or subdividing tasks for the language model (Desmond and Brachman, 2024). This use case is further exacerbated by recent systems supporting file uploads of arbitrary sizes for document analysis (Google, 2025; Anthropic, 2025). Asking questions on a large block of context represents a realistic use scenario, highlighting the utility of Oolong-synth as a model capability evaluation bench. To further motivate the information aggregation problem, we next consider a setting in which inputs cannot be so trivially decomposed. 3 Oolong-real We complement Oolong-synth with questions derived from real conversational data. Oolong-real is compiled from the transcripts of a Dungeons and Dragons (D&D) role-playing game, where a group of players collaboratively build a story through in-character actions and success depends on rolling dice. Stories unfold over narratives (“campaigns”) that span dozens to hundreds of episodes, with each episode generally lasting 4-5 hours of play. These transcripts involve several levels of conversation, from out-of-character chitchat to rules discussion to in-character actions and speech. Though lightly edited for readability, they reflect naturalistic speech instead of carefully planned written text. Because the conversation is unscripted and can involve tangents or side channels, conversational turns require variable amounts of prior context to resolve. In some instances, the same event (e.g. casting a spell) is discussed for many turns, or brought up again after a long interlude; in other cases, a prior event is “retconned” or revised post hoc. In Dungeons and Dragons, there are limitations to how frequently characters can take certain types of actions, and fans often discuss metadata of the campaigns (e.g., whether a certain person is particularly unlucky with their dice rolls or if a character uses a signature spell more or less as the campaign progresses). Because these shows are extremely long, wide-ranging, and improvised live, the creators do not provide this type of information. Instead, this information is often annotated meticulously, with multiple levels of verification, by dedicated fans of the work. We take this as an example of an information aggregation task in the wild that is not simple to reframe as an iterative task. Figure 2: Scores by context window length for Oolong-synth and Oolong-real. Here we show only the best model for each model family; see Appendix B for full results. For Oolong-real, we consider two campaigns from the series Critical Role and the data compiled by CritRoleStats,22 2 https://www.critrolestats.com/ a fan project that labeled by hand per-episode statistics for the first several campaigns of Critical Role. We used episode transcripts from the Critical Role Dungeons and Dragons Dataset (CRD3) (Rameshkumar and Bailey, 2020). We use CritRoleStats game statistics about dice rolls and spells cast in each episode and design a set of questions that cover a variety of information aggregation tasks. We include questions that require processing of single- or multiple-episode transcripts. For multi-episode questions, we concatenate transcripts and use delimiters to highlight the start and end of each transcript. See Table 19, Table 20, and Table 21 of the Appendix for a full list of Oolong-real questions. We include varying context windows to evaluate the model's ability to use long-context reasoning and aggregation capabilities. Using a single episode transcript as a context unit, we include context windows ranging from one to 24 episode transcripts (55K to 1.3M tokens). We follow the evaluation setup from Oolong-synth. 4 Results and Analysis On Oolong, we benchmark a strong suite of frontier models to study their information aggregation capabilities over long context. We include a mix of models of varying sizes, levels of reasoning, and long-context capabilities. In Figure 2, we visualize the model performance for context windows up to 512K tokens. As expected, we see a significant drop in performance at higher context windows. The two splits are of similar difficulty at the same context length, although direct comparison is challenging because real data do not necessarily align with fixed context window buckets.33 3 While performance on Oolong-real starts lower, this is because the shortest inputs in this dataset are a single episode, with an average length of 55K tokens; models perform similarly on Oolong-synth around the same context length. A breakdown of results with an aggregate rating over the two datasets is available in the Appendix (Table 3), as well as an analysis by question and answer type (Appendix G). 4.1 Does reasoning effort correlate to performance? Oolong is a reasoning-intensive task, so we hypothesize that increased reasoning may correlate with better scores. We measure reasoning effort in three different ways. (a) GPT-5-nano reasoning effort (b) Gemini-3-Pro reasoning token count Figure 3: Does reasoning effort correlate with task performance? Qwen and Olmo: think versus instruct models. In these families, we have pairs of models that are focused on reasoning and conventional instruction following. In the Olmo family, the reasoning (“think”) models greatly exceed the performance of their equivalent instruction-tuned model on Oolong-synth. However, for Qwen-3-4B, the reasoning model shows near-zero performance on Oolong-synth because of the same over-reasoning phenomenon described for Deepseek-R1: although it is allotted up to 256K output tokens, the model almost always uses the full output for reasoning without returning an answer. GPT-5-nano: reasoning effort parameter. The OpenAI API allows for specification of a reasoning level: “low,” “medium,” or “high.” We compare “high” and “low” reasoning effort for GPT-5-nano. 3(a) shows the results for Oolong-synth. Although Oolong is a reasoning-intensive task, specifying a higher reasoning effort is only useful for short contexts; after 64k, there is little discernible difference between reasoning levels, with the “high” reasoning setting even slightly underperforming “low” reasoning at 256k on Oolong-synth. We hypothesize that at context lengths where there is sufficient remaining room in the context window to enumerate labels for each example in-context, adding more reasoning effort may encourage the model to take this strategy. However, since the default routing for this model performs reasonably well at all lengths and especially for longer inputs, we do not explicitly specify a reasoning level for the remainder of the runs. We report results on Oolong-real in Figure 7 in the Appendix. Gemini 3 Pro: reasoning token count. The Gemini API returns the number of reasoning tokens used (but not the content of the reasoning trace). We visualize trends in reasoning tokens used in 3(b). Longer inputs generally lead to longer reasoning traces, as expected; however, when the input length is held constant, reasoning trace length correlates negatively with correctness. We hypothesize this could partially be because harder problems result in both longer traces and lower probability of answering correctly. 4.2 Over-reasoning We observe a persistent failure case related to over-reasoning on several models: the model exceeds its maximum reasoning token or total token budget in the course of a very long reasoning trace, and fails to answer the question at all.44 4 We evaluate all models on their allowed maximum reasoning trace or output token budget. This occurs particularly on Oolong-synth, which is extremely information-dense- all or nearly all of the lines are necessary to solve the questions. In our experiments, over-reasoning is the culprit when a model performs well on Oolong-real and poorly on Oolong-synth. Deepseek R1 This is a strong reasoning model and outperforms GPT-5 nano and o4-mini on Oolong-real. However, on Oolong-synth, it achieves performance below the random baseline. In manual inspection of the traces, we observe pathologies in the reasoning traces for Oolong-synth that are not present in traces for Oolong-real and appear only occasionally in traces from other models. We identify these behaviors through discussion between the authors, then prompt GPT-5.6-Luna to label 2,400 randomly selected traces from Deepseek-R1 with a yes/no classification about whether an identified behavior is present in this trace.55 5 For this labeling task, we also verify with a human annotation of a small subset, and see high agreement (85-96%) across all questions. Many traces (56.3%) do not provide an answer at all. We hypothesize that, because Oolong-synth is such an information-dense task, the model's apparent strategy of labeling each example before deciding which are relevant results in running out of context tokens. 75.7% of traces end in an incomplete sentence.66 6 Note that in some cases, the model provides a candidate answer, then begins to double-check the answer and then exceeds the maximum output length; we evaluate the last candidate answer we are able to parse. In 26.6% of cases, the model debates whether the task is impossible or intractable given the length of the input. Appendix F shows example traces for Deepseek-R1 from both splits. Gemini models Gemini 2.5 Pro is one of the strongest models on Oolong-real, but exhibits performance dropoff at longer contexts on Oolong-synth; Gemini 3 experiences the same trend less sharply, falling from the best model by a margin to comparable with Claude Sonnet by 256K on Oolong-synth. We observe that both models, and especially Gemini 2.5 Pro, increasingly exceeds its maximum output length during reasoning for long Oolong-synth inputs. Unlike the other APIs used in benchmarking, the Gemini API does not return any tokens if the max token count is exceeded during reasoning; this results in an automatic score of 0 for any cases of overlong reasoning.At 256K context length, Gemini 2.5 Pro fails to return an answer because of the max reasoning tokens constraint in 64.0% of Oolong-synth outputs, despite these inputs requiring only one quarter of the model's input context window. At 256K, Gemini 3 Pro fails to return 45% of Oolong-synth outputs. Gemini 3 Pro outperforms the older model by both answering more queries and answering a higher percentage of them correctly.77 7 Note the slight increase in Gemini 3 Pro performance from 256K to 512K: while 45% of inputs at 256K hit the max token limit during reasoning, only 11.8% of inputs at 512K hit this limit. It is not clear what causes this shift. Over-reasoning is not the only reason for declining performance with longer context; however, it bounds performance for both Gemini models. We do not observe this behavior on Oolong-real, where Gemini maintains its strong performance at longer context lengths. 4.3 Refusal Another error case we observe is when models refuse to answer a question at all. In these responses, models describe the task as impossible or intractable. Certainly some level of caution is appropriate when answering types of questions that models struggle on. However, we compare the performance of GPT 5 and GPT 5.2 and observe that surprisingly, despite very strong performance on OpenAI's MRCR long context benchmark OpenAI (2025), GPT 5.2 underperforms GPT 5 substantially on Oolong-synth. This is because GPT 5.2 is much more likely to refuse to perform the task; at 256K, GPT-5 attempts 83% of queries and answers 44.5% of these correctly, while GPT-5.2 attempts only 32% with an accuracy of 52.8%. It seems likely that GPT-5.2's performance would be improved, perhaps substantially, by attempting a higher percentage of questions. 4.4 Aggregation without classification To isolate the effects of aggregation and classification, we consider simpler settings for both dataset splits. In both settings, we see that aggregation is the bottleneck: scores are only marginally improved by removing the classification task. Providing labels Figure 4: Comparison on Oolong-synth: we provide the gold labels in the input. This leads to a consistent but small improvement. For Oolong-synth, we construct an easier version of the task by providing the label for each ICL example in-context. This reduces the task for most questions to identifying the relevant instances and summing the occurrence of each label type. Figure 4 shows the behavior of four models with and without labels provided in-context. As expected, adding labels improves accuracy; however, this improvement ranges from 10.9 points to only 0.79 points. We do not see consistently higher gains from providing labels in longer inputs, which suggests that the lower performance at longer context lengths is not primarily due to an accumulation of mislabeling errors. Additionally, the improvement is not larger for GPT-5-nano than it is for the more powerful GPT-5, suggesting that the performance difference observed between these models is not due to differing performance on the classification task. This is by design; our classification task validation was designed to eliminate overly challenging or misleading examples. Providing domain knowledge We believe that most Oolong-real tasks do not require knowledge specific to Dungeons & Dragons. For instance, tasks that require counting occurrences of a spell or action do not require knowledge of the goals of that action, only its presence in the text. Some questions, however, do benefit from domain knowledge; for instance, the question “how many cantrips were cast?” requires knowing which spells are considered cantrips, and the question “which roll type is most common?” is solvable without external knowledge but may be easier with the context of all the types of rolls and where each is generally used. While models generally have this knowledge because of the proliferation of text about this game on the internet, we experiment with providing additional context for each of these questions: a mapping from spell name to spell level or a list of roll types with brief descriptions of each. In this setting, we rerun the single-episode examples for GPT-5 and Claude Sonnet 4 and observe that providing this additional context helps slightly but not substantially: raising GPT-5's score by 3.51 points to 62.25 and raising Claude Sonnet 4's score by 2.57 points to 53.15. 4.5 Iterative retrieval baseline for Oolong-real While Oolong-real remains an information-dense task, with dozens of individual details needed to answer most questions, there are segments of the transcripts that are not necessary to generate the answer. Could Oolong-real be tackled with retrieval? We use smolagent (Roucher et al., 2025) to instantiate a simple iterative RAG baseline acting for up to 5 steps. This system is equipped with tool calls for retrieval (at an utterance level, using a BM25 retriever) and code execution. Each utterance is annotated with the transcript number and the utterance number before being indexed. We evaluated this system on the single transcript and two transcript settings (the 55K and 110K context settings in Figure 2) with GPT-5 as the backbone model. This system substantially underperforms directly using GPT-5 (with one transcript: -16.15 points; with two transcripts: -17.67 points). This shows that Oolong cannot be easily solved by a simple agentic retrieval system. We expect future work to study ways to improve retrieval tool use and explore dynamic chunking algorithms depending on the question. The interleaved and complex nature of human dialogue presents unique challenges to both agentic and long-context systems alike (see Appendix H for further discussion). However, retrieval performance, while below long-context performance, remains non-trivial; because of this, we speculate that methods that extend context by condensing sections of the context (Smith, 2025) or evicting subsections of the context (Li et al., 2025a) could be effective for this task. 5 Related Work We are far from the first to recognize the importance of aggregation tasks. Many benchmarks focus on reasoning tasks in settings where longer context is used to measure robustness to distractors, rather than to introduce increased numbers of reasoning steps. This is an important problem but orthogonal to our focus. For instance, BABILong (Kuratov et al., 2024) tests long-context reasoning by placing BABI (Weston et al., 2015) reasoning tasks within a long context and evaluating performance as the amount of distractor text increases. Ling et al. (2025), a contemporaneous work to ours, creates longer-context reasoning problems by synthetically elaborating on the documents necessary for short-context reasoning tasks and then interleaving these with irrelevant context. Other tasks require use of the full context in heavily artificial settings. RULER (Hsieh et al., 2024) includes synthetic retrieval tasks at varying context lengths. It includes multi-hop tracing and aggregation tasks such as identifying the most common words in a long input of random nouns. LongMemEval (Wu et al., 2025) and multi-round coreference resolution tasks (MRCR) evaluate long-context capabilities with conversational data. MRCR was first introduced in Gemini (Vodrahalli et al., 2024) and was further extended in OpenAI's MRCR (OpenAI, 2025). Unlike the standard NIAH task, the needles and the distractors in MRCR are sampled from the same distributions. OpenAI's MRCR includes variants of 2, 4, and 8 needles, and the task is to retrieve the ith instance of one of the needles. While this uses naturalistic data (synthetic conversations generated using GPT-4o), it is a task focused on measuring basic counting ability rather than reasoning. Conversely, GSM-infinite (Zhou et al., 2025) stress tests long-context reasoning through adaptable frameworks for varying task difficulty and length; our work is complementary, as their focus is primarily on mathematical reasoning through synthetically generated problems, while we focus on a more noisy, naturalistic text setting with primarily non-mathematical reasoning. Similarly, MathHay (Wang et al., 2024) focuses on queries across document collections that require retrieving multiple numbers from across context and performing multi-step mathematical reasoning with them. Multi-hop or long context question answering tasks can also require collecting information from multiple parts of the context, and represent a different type of long context reasoning task: one much sparser and less numeracy-focused than our aggregation task. HELMET (Yen et al., 2024) expands the tasks in RULER to include tasks related to the downstream use of LMs including a set of long-context QA tasks from Zhang et al. (2024). Other long-context QA tasks explicitly focus on capturing long dependency length (Li et al., 2024; Bai et al., 2025), on building semantic equivalents to needle-in-a-haystack retrieval (Modarressi et al., 2025), or specific downstream domains (He et al., 2025) TLDM (Hamilton et al., 2025) includes two related tasks about character location and time passed in a narrative setting in the domain of public-domain literature. MoNaCo (Wolfson et al., 2025) considers challenging information seeking queries that require retrieval and aggregation; their emphasis is on retrieval of the necessary documents for intermediate reasoning steps in an agentic setting. Other works conceptually aligned with ours include ZeroScrolls and NeedleBench. ZeroSCROLLS (Shaham et al., 2023) introduces a task of identifying the percentage of reviews on Amazon for a product that are positive, which is the closest conceptual ancestor of our work; Oolong-synth encompasses this type of task along with other aggregation tasks. NeedleBench (Li et al., 2025b) introduces a reasoning task called the Ancestral Trace Challenge, which, given a number of relations between individuals in context, asks the model to identify the most distance relative of a specific individual. The Ancestral Trace Challenge requires tree-like branching of reasoning; Oolong focuses on classify-then-aggregate style tasks and emphasizes more realistic data. A line of work has also framed LLM aggregation as database-like operations over the input text. Thorne et al. (2021) call this problem database reasoning and study architectures well-suited to this task for short contexts. HoloBench (Maekawa et al., 2025) directly synthesizes examples from text-to-SQL datasets. And concurrently to our work, SciTrek (Li et al., 2026) frames SQL-like queries over scientific article metadata as a long context evaluation. Appendix Cdiscusses additional related work on information aggregation and on Dungeons & Dragons data. 6 Conclusion We introduce Oolong, a challenging long-context information aggregation benchmark in two parts. Frontier models' poor performance on Oolong suggests more work is necessary in developing both reasoning abilities and more general long context abilities. In longer context regimes, many models perform roughly equivalently on Oolong-synth and Oolong-real at the same context length, despite the much higher information density in Oolong-synth. This suggests that better length generalization, regardless of reasoning ability, may also improve performance on this type of challenging task. However, some models behave differently in extremely information-dense regimes, highlighting a direction for future work. While prior work on reasoning chains has focused on over-reasoning for adversarial (Kumar et al., 2025) or overly simple problems (Chiang and Lee, 2024; Sui et al., 2025), Oolong indicates that even strong models may struggle to plan the reasoning quantity for information-dense inputs, where it may be desirable to accept a less optimal or more error-prone reasoning strategy (e.g. double-checking less frequently) in order to avoid running into the maximum reasoning tokens. Future models that are aware of the maximum reasoning token budget may be able to more carefully allocate reasoning effort. Overall, our results suggest that there is still a long way to go in designing robust long-context aggregation capabilities for LLMs. Reproducibility Statement We release code for dataset construction and evaluation, in addition to the final dataset version. API-based models typically exhibit some small variability across API calls, and versions of each model may be phased out over time by the providers, so exact replication of outputs may be infeasible; wherever possible, we release the model outputs for each model run reported in the paper. Ethics Statement This work is intended to further the evaluation of long-context LLMs. We do not foresee any particular additional risks introduced by our evaluation, although information aggregation, like nearly any LM capability, could be useful for both beneficial and harmful actors. CritRoleStats, whose data we use as part of constructing gold labels for the Oolong-real split, explicitly allows the use of their data for statistical analyses in their FAQ, so long as they are credited.88 8 https://www.critrolestats.com/faqs We have also notified them of our use of their data. In the course of this work, we used LM assistance in the writing of some data preprocessing, analysis, and visualization scripts. 7 Acknowledgements We are grateful to the contributors and maintainers of CritRoleStat for their excellent resource, and to the broader Critical Role community for their commitment and engagement. Like prior work that uses Dungeons & Dragons as a testbed for capabilities, this work would not be possible without the resources built by fan communities. We would also like to thank Clara Na for helpful feedback on drafts of this work. AB was supported by a grant from the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE2140739. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors. References Agarwal et al. (2024) R. Agarwal, A. Singh, L. M. Zhang, B. Bohnet, L. Rosias, S. Chan, B. Zhang, A. Anand, Z. Abbas, A. Nova, J. D. Co-Reyes, E. Chu, F. Behbahani, A. Faust, and H. Larochelle Many-shot in-context learning. External Links: 2404.11018, Link Cited by: §1. Almeida et al. (2011) T. A. Almeida, J. M. G. Hidalgo, and A. Yamakami Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 2011 ACM Symposium on Document Engineering (DOCENG'11), Cited by: Table 4. Anthropic (2025) Anthropic What kinds of documents can i upload to claude?. Note: Accessed: 2025-11-04 External Links: Link Cited by: §2.3. Bai et al. (2025) Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, J. Tang, and J. Li LongBench v2: towards deeper understanding and reasoning on realistic long-context multitasks. External Links: 2412.15204, Link Cited by: §5. Bertsch et al. (2024) A. Bertsch, M. Ivgi, E. Xiao, U. Alon, J. Berant, M. R. Gormley, and G. Neubig In-context learning with long-context models: an in-depth exploration. arXiv preprint arXiv:2405.00200. Cited by: §1. Bizzoni and Lappin (2018) Y. Bizzoni and S. Lappin Predicting human metaphor paraphrase judgments with deep neural networks. In Proceedings of the Workshop on Figurative Language Processing, B. Beigman Klebanov, E. Shutova, P. Lichtenstein, S. Muresan, and C. Wee (Eds.), New Orleans, Louisiana, pp. 45-55. External Links: Link, Document Cited by: Table 4. Callison-Burch et al. (2022) C. Callison-Burch, G. S. Tomar, L. Martin, D. Ippolito, S. Bailis, and D. Reitter Dungeons and dragons as a dialog challenge for artificial intelligence. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9379-9393. External Links: Link, Document Cited by: Appendix C. Chiang and Lee (2024) C. Chiang and H. Lee Over-reasoning and redundant calculation of large language models. External Links: 2401.11467, Link Cited by: §6. Chong et al. (2022) D. Chong, J. Hong, and C. D. Manning Detecting label errors by using pre-trained language models. External Links: 2205.12702, Link Cited by: §2.1. Desmond and Brachman (2024) M. Desmond and M. Brachman Exploring prompt engineering practices in the enterprise. External Links: 2403.08950, Link Cited by: §2.3. DeYoung et al. (2024) J. DeYoung, S. C. Martinez, I. J. Marshall, and B. C. Wallace Do multi-document summarization models synthesize?. Transactions of the Association for Computational Linguistics 12, pp. 1043-1062. External Links: Link, Document Cited by: Appendix C. García-Ferrero et al. (2023) I. García-Ferrero, B. Altuna, J. Alvez, I. Gonzalez-Dios, and G. Rigau This is not a dataset: a large negation benchmark to challenge large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 8596-8615. External Links: Link, Document Cited by: Table 4. Goldman et al. (2024) O. Goldman, A. Jacovi, A. Slobodkin, A. Maimon, I. Dagan, and R. Tsarfaty Is it really long context if all you need is retrieval? towards genuinely difficult long context NLP. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 16576-16586. External Links: Link, Document Cited by: Appendix C. Google (2025) Google Google notebooklm: ai research tool & thinking partner. Note: https://notebooklm.google/Accessed: 2025-11-04 Cited by: §2.3. Grano et al. (2017) G. Grano, A. Di Sorbo, F. Mercaldo, C. A. Visaggio, G. Canfora, and S. Panichella Software applications user reviews. Cited by: Table 4. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), pp. 633-638. Cited by: §1. Hamilton et al. (2025) S. Hamilton, R. M. M. Hicke, M. Wilkens, and D. Mimno Too long, didn't model: decomposing llm long-context understanding with novels. External Links: 2505.14925, Link Cited by: §5. He et al. (2025) Z. He, Y. Wang, J. Li, K. Liang, and M. Zhang LooGLE v2: are llms ready for real world long dependency challenges?. External Links: 2510.22548, Link Cited by: §5. Hovy et al. (2001) E. Hovy, L. Gerber, U. Hermjakob, C. Lin, and D. Ravichandran Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research, External Links: Link Cited by: Table 4. Hsieh et al. (2024) C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, and B. Ginsburg RULER: what's the real context size of your long-context language models?. In First Conference on Language Modeling, External Links: Link Cited by: §1, §5. Jimenez et al. (2023) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan Swe-bench: can language models resolve real-world github issues?. arXiv preprint arXiv:2310.06770. Cited by: §1. Klie et al. (2023) J. Klie, B. Webber, and I. Gurevych Annotation error detection: analyzing the past and present for a more coherent future. Computational Linguistics 49 (1), pp. 157-198. External Links: ISSN 0891-2017, Document, Link, https://direct.mit.edu/coli/article-pdf/49/1/157/2068980/coli_a_00464.pdf Cited by: §2.1. Kumar et al. (2025) A. Kumar, J. Roh, A. Naseh, M. Karpinska, M. Iyyer, A. Houmansadr, and E. Bagdasarian OverThink: slowdown attacks on reasoning llms. External Links: 2502.02542, Link Cited by: §6. Kuratov et al. (2024) Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev BABILong: testing the limits of llms with long context reasoning-in-a-haystack. External Links: 2406.10149, Link Cited by: §1, §5. Lahiri (2015) S. Lahiri SQUINKY! A corpus of sentence-level formality, informativeness, and implicature. arXiv preprint arXiv:1506.02306. Cited by: Table 4. Li et al. (2025a) H. Li, Y. Li, A. Tian, T. Tang, Z. Xu, X. Chen, N. Hu, W. Dong, Q. Li, and L. Chen A survey on large language model acceleration based on kv cache management. External Links: 2412.19442, Link Cited by: §4.5. Li et al. (2024) J. Li, M. Wang, Z. Zheng, and M. Zhang LooGLE: can long-context language models understand long contexts?. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 16304-16333. External Links: Link, Document Cited by: §5. Li et al. (2026) M. Li, A. Gurung, I. Saparina, and M. Lapata SciTrek: evaluating and improving long-context numerical reasoning over scientific articles. External Links: 2509.21028, Link Cited by: §5. Li et al. (2025b) M. Li, S. Zhang, T. Zhang, H. Duan, Y. Liu, and K. Chen NeedleBench: evaluating llm retrieval and reasoning across varying information densities. External Links: 2407.11963, Link Cited by: §5. Li and Roth (2002) X. Li and D. Roth Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, External Links: Link Cited by: Table 4. Ling et al. (2025) Z. Ling, K. Liu, K. Yan, Y. Yang, W. Lin, T. Fan, L. Shen, Z. Du, and J. Chen LongReason: a synthetic long-context reasoning benchmark via context expansion. External Links: 2501.15089, Link Cited by: §5. Liu et al. (2024) X. Liu, P. Dong, X. Hu, and X. Chu LongGenBench: long-context generation benchmark. External Links: 2410.04199, Link Cited by: Appendix C. Maas et al. (2011) A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Oregon, USA, pp. 142-150. External Links: Link Cited by: Table 4. Maekawa et al. (2025) S. Maekawa, H. Iso, and N. Bhutani Holistic reasoning with long-context lms: a benchmark for database operations on massive textual data. External Links: 2410.11996, Link Cited by: §5. Min et al. (2022) S. Min, M. Lewis, L. Zettlemoyer, and H. Hajishirzi MetaICL: learning to learn in context. External Links: 2110.15943, Link Cited by: §2.1. Modarressi et al. (2025) A. Modarressi, H. Deilamsalehy, F. Dernoncourt, T. Bui, R. A. Rossi, S. Yoon, and H. Schütze NoLiMa: long-context evaluation beyond literal matching. External Links: 2502.05167, Link Cited by: §5. OpenAI (2025) OpenAI OpenAI MRCR: Long context multiple needle in a haystack benchmark. External Links: Link Cited by: §1, §4.3, §5. Pavlick and Tetreault (2016) E. Pavlick and J. Tetreault An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics. Cited by: Table 4. Rameshkumar and Bailey (2020) R. Rameshkumar and P. Bailey Storytelling with dialogue: A Critical Role Dungeons and Dragons Dataset. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 5121-5134. External Links: Link, Document Cited by: Appendix C, §H.1, §3. Roucher et al. (2025) A. Roucher, A. V. del Moral, T. Wolf, L. von Werra, and E. Kaunismäki ‘Smolagents‘: a smol library to build great agentic systems.. Note: https://github.com/huggingface/smolagents Cited by: §4.5. Sarkar et al. (2025) R. Sarkar, B. Sarrafzadeh, N. Chandrasekaran, N. Rangan, P. Resnik, L. Yang, and S. K. Jauhar Conversational user-ai intervention: a study on prompt rewriting for improved llm response generation. External Links: 2503.16789, Link Cited by: §2.3. Shaham et al. (2023) U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy ZeroSCROLLS: a zero-shot benchmark for long text understanding. External Links: 2305.14196, Link Cited by: §5. Smith (2025) C. Smith OpenHands context condensation for more efficient AI agents. Note: All Hands AI Blog External Links: Link Cited by: §4.5. Sui et al. (2025) Y. Sui, Y. Chuang, G. Wang, J. Zhang, T. Zhang, J. Yuan, H. Liu, A. Wen, S. Zhong, N. Zou, H. Chen, and X. Hu Stop overthinking: a survey on efficient reasoning for large language models. External Links: 2503.16419, Link Cited by: §6. Thorne et al. (2021) J. Thorne, M. Yazdani, M. Saeidi, F. Silvestri, S. Riedel, and A. Halevy Database reasoning over text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, pp. 3091-3104. External Links: Link, Document Cited by: §2, §5. Vodrahalli et al. (2024) K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, R. Anil, E. Dyer, S. Shakeri, R. Vij, H. Mehta, V. Ramasesh, Q. Le, E. Chi, Y. Lu, O. Firat, A. Lazaridou, J. Lespiau, N. Attaluri, and K. Olszewska Michelangelo: long context evaluations beyond haystacks via latent structure queries. External Links: 2409.12640, Link Cited by: §5. Wang et al. (2024) L. Wang, S. Dong, Y. Xu, H. Dong, Y. Wang, A. Saha, E. Lim, C. Xiong, and D. Sahoo MathHay: an automated benchmark for long-context mathematical reasoning in llms. External Links: 2410.04698, Link Cited by: §5. Weston et al. (2015) J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. van Merriënboer, A. Joulin, and T. Mikolov Towards ai-complete question answering: a set of prerequisite toy tasks. External Links: 1502.05698, Link Cited by: §5. Williams et al. (2018) A. Williams, N. Nangia, and S. Bowman A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122. External Links: Link Cited by: Table 4. Wolfson et al. (2025) T. Wolfson, H. Trivedi, M. Geva, Y. Goldberg, D. Roth, T. Khot, A. Sabharwal, and R. Tsarfaty MoNaCo: more natural and complex questions for reasoning across dozens of documents. Transactions of the Association for Computational Linguistics. Cited by: §5. Wu et al. (2025) D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu LongMemEval: benchmarking chat assistants on long-term interactive memory. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §5. Xue et al. (2025) H. Xue, Y. J. Oh, X. Zhou, X. Zhang, and B. Oxley User prompting strategies and chatgpt contextual adaptation shape conversational information-seeking experiences. External Links: 2509.25513, Link Cited by: §2.3. Ye et al. (2025) X. Ye, F. Yin, Y. He, J. Zhang, H. Yen, T. Gao, G. Durrett, and D. Chen LongProc: benchmarking long-context language models on long procedural generation. External Links: 2501.05414, Link Cited by: Appendix C. Yen et al. (2024) H. Yen, T. Gao, M. Hou, K. Ding, D. Fleischer, P. Izsak, M. Wasserblat, and D. Chen Helmet: how to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694. Cited by: §1, §2.2, §5. Ying and Thomas (2022) C. Ying and S. Thomas Label errors in BANKING77. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, S. Tafreshi, J. Sedoc, A. Rogers, A. Drozd, A. Rumshisky, and A. Akula (Eds.), Dublin, Ireland, pp. 139-143. External Links: Link, Document Cited by: §2.1. Zhang et al. (2015) X. Zhang, J. J. Zhao, and Y. LeCun Character-level convolutional networks for text classification. In NIPS, Cited by: Table 4, Table 4. Zhang et al. (2024) X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, and M. Sun ∞\inftybench: Extending long context evaluation beyond 100k tokens. External Links: 2402.13718, Link Cited by: §5. Zhou et al. (2023) P. Zhou, A. Zhu, J. Hu, J. Pujara, X. Ren, C. Callison-Burch, Y. Choi, and P. Ammanabrolu I cast detect thoughts: learning to converse and guide with intents and theory-of-mind in dungeons and dragons. External Links: 2212.10060, Link Cited by: Appendix C. Zhou et al. (2025) Y. Zhou, H. Liu, Z. Chen, Y. Tian, and B. Chen GSM-infinite: how do your llms behave over infinitely increasing context length and reasoning complexity?. External Links: 2502.05252, Link Cited by: §5. Zhu et al. (2023a) A. Zhu, K. Aggarwal, A. Feng, L. J. Martin, and C. Callison-Burch FIREBALL: a dataset of dungeons and dragons actual-play with structured game state information. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, pp. 4171-4193. External Links: Link, Document Cited by: Appendix C. Zhu et al. (2023b) A. Zhu, L. Martin, A. Head, and C. Callison-Burch CALYPSO: llms as dungeon master's assistants. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 19 (1), pp. 380-390. External Links: Link, Document Cited by: Appendix C. Appendix A Oolong - Leaderboard Here we provide full results on all models we ran. In Table 3, we report the results for both Oolong-synth and Oolong-real numerically. Following prior long-context benchmarks, we measure model performance at increasing context window sizes and report a task average. For Oolong-synth, we use the standard context window sizes of up to 128K tokens. For Oolong-real, we report scores at increasing number of episode transcripts (up to 3 episodes). GPT-5 performs the best in both subsets, followed by o3, GPT-5-mini and Claude-Sonnet-4. Gemini-2.5-Pro is initially the strongest model, but falls to below-random performance by 256K, largely because of responses exceeding the maximum token count; see the analysis for further discussion. Notably, at 128K context window size nearly all our evaluated models obtain a score under 50. DeepSeek R1 does well on Oolong-real, but struggles to perform well on Oolong-synth (see §4.1. Llama-4-Maverick fails in both subsets. Oolong-synth Oolong-real Model Avg. 8K 16K 32K 64K 128K Avg. 55K 118K 175K Gemini-3-Pro 78.69 90.94 87.61 81.94 70.55 62.42 57.93 64.71 55.01 54.07 GPT-5 70.75 85.56 84.45 76.12 61.24 46.36 47.00 58.74 45.72 36.53 Gemini-2.5-Pro 55.29 88.13 69.84 56.83 36.56 25.06 52.16 60.04 49.36 47.07 o3 62.37 86.80 79.52 63.23 44.86 37.45 36.71 50.57 33.57 25.99 GPT-5-mini 63.68 85.13 77.65 64.64 50.14 40.85 34.55 49.86 29.90 23.89 Claude-Sonnet-4 58.18 74.43 62.75 55.67 50.04 48.02 36.75 50.58 32.98 26.70 o4-mini 56.74 83.07 65.10 51.86 44.15 39.53 27.13 41.69 21.77 17.93 Gemma-4-31B 53.38† 79.29 69.80 64.32 53.47 - 30.23† 54.97 35.72 - GPT-5-nano 50.73 70.96 54.53 47.81 41.02 39.31 31.05 43.09 26.82 23.23 GPT-5.2 44.29 61.80 53.95 47.32 35.31 23.10 34.40 45.67 32.15 25.38 Qwen3-4B-Instruct 40.67 55.81 41.33 41.35 34.36 30.49 22.88 39.58 18.29 10.78 Olmo-3.1-Think-32B 35.29† 59.01 43.81 39.13 34.48 - 10.43† 31.28 - - Deepseek-R1 13.11 13.94 13.65 12.91 13.20 11.87 32.00 47.85 27.35 20.81 Olmo-3-Think-7B 31.42† 46.44 39.22 38.14 33.32 - 7.47† 22.42 - - Olmo-3.1-Instruct-32B 26.67† 42.31 38.35 31.64 21.06 - 10.35† 31.04 - - Olmo-3-Instruct-7B 21.10† 29.88 28.55 25.54 21.54 - 7.93† 23.80 - - Llama-3-3B 25.60 29.31 26.37 24.86 25.09 22.39 0.00 0.00 0.00 0.00 Llama-3-8B 23.71 27.54 24.66 24.74 24.29 17.31 0.41 1.15 0.07 0.00 Qwen3-4B-Thinking 0.00 0.00 0.00 0.00 0.00 0.00 20.18 37.65 15.75 7.15 Llama-4-Maverick 16.37 15.00 16.29 15.42 16.35 18.80 2.07 2.48 2.11 1.62 Table 2: Oolong results on a number of strong models. Nearly all models we test support at least 200K context; thus, we report Oolong scores as an average over scores on 8K-175K inputs. Oolong-synth is a more information-dense task than Oolong-real; the ranking of some models shifts slightly between the two settings. Models are sorted by the average score between the two splits. - marks a context length beyond the maximum for that model; such lengths count as 0 in the average, which we denote with †. Figure 5 and Figure 6 are augmented versions of Figure 2 with all closed-weights and open-weights models (respectively) shown. Figure 5: Scores by context window length for Oolong-synth and Oolong-real for closed-weight models. Figure 6: Scores by context window length for Oolong-synth and Oolong-real for open-weight models. Oolong-synth Oolong-real Model Avg. 8K 16K 32K 64K 128K Avg. 55K 118K 175K Gemini-3-Pro 78.69 90.94 87.61 81.94 70.55 62.42 57.93 64.71 55.01 54.07 GPT-5 70.75 85.56 84.45 76.12 61.24 46.36 47.00 58.74 45.72 36.53 Gemini-2.5-Pro 55.29 88.13 69.84 56.83 36.56 25.06 52.16 60.04 49.36 47.07 o3 62.37 86.80 79.52 63.23 44.86 37.45 36.71 50.57 33.57 25.99 GPT-5-mini 63.68 85.13 77.65 64.64 50.14 40.85 34.55 49.86 29.90 23.89 Claude-Sonnet-4 58.18 74.43 62.75 55.67 50.04 48.02 36.75 50.58 32.98 26.70 o4-mini 56.74 83.07 65.10 51.86 44.15 39.53 27.13 41.69 21.77 17.93 Gemma-4-31B 53.38† 79.29 69.80 64.32 53.47 - 30.23† 54.97 35.72 - GPT-5-nano 50.73 70.96 54.53 47.81 41.02 39.31 31.05 43.09 26.82 23.23 GPT-5.2 44.29 61.80 53.95 47.32 35.31 23.10 34.40 45.67 32.15 25.38 Qwen3-4B-Instruct 40.67 55.81 41.33 41.35 34.36 30.49 22.88 39.58 18.29 10.78 Olmo-3.1-Think-32B 35.29† 59.01 43.81 39.13 34.48 - 10.43† 31.28 - - Deepseek-R1 13.11 13.94 13.65 12.91 13.20 11.87 32.00 47.85 27.35 20.81 Olmo-3-Think-7B 31.42† 46.44 39.22 38.14 33.32 - 7.47† 22.42 - - Olmo-3.1-Instruct-32B 26.67† 42.31 38.35 31.64 21.06 - 10.35† 31.04 - - Olmo-3-Instruct-7B 21.10† 29.88 28.55 25.54 21.54 - 7.93† 23.80 - - Llama-3-3B 25.60 29.31 26.37 24.86 25.09 22.39 0.00 0.00 0.00 0.00 Llama-3-8B 23.71 27.54 24.66 24.74 24.29 17.31 0.41 1.15 0.07 0.00 Qwen3-4B-Thinking 0.00 0.00 0.00 0.00 0.00 0.00 20.18 37.65 15.75 7.15 Llama-4-Maverick 16.37 15.00 16.29 15.42 16.35 18.80 2.07 2.48 2.11 1.62 Table 3: Oolong results on a number of strong models. Nearly all models we test support at least 200K context; thus, we report Oolong scores as an average over scores on 8K-175K inputs. Oolong-synth is a more information-dense task than Oolong-real; the ranking of some models shifts slightly between the two settings. Models are sorted by the average score between the two splits. - marks a context length beyond the maximum for that model; such lengths count as 0 in the average, which we denote with †. Appendix B Additional Results B.1 Reasoning effort on Oolong-real Reasoning models generally show little gain on Oolong-real; the Olmo models show a small gain and the Qwen models show a small loss. When we compare “high” and “low” reasoning effort for GPT-5-nano, we see very little variation across the settings. Figure 7 shows this result for Oolong-real. Figure 7: Comparison across reasoning levels on Oolong-real. B.2 Data contamination for Oolong-real Critical Role is an extremely popular show, with widespread discussion of the episodes online, and the statistics provided by CritRoleStats are publicly available. Could a model have simply memorized the answers to Oolong-real? Conceptually, we believe the risk of data contamination is low: the CritRoleStats statistics are provided in spreadsheets that map from episode numbers to statistics without providing any transcript text. Since we do not provide episode numbers in our prompts, contamination could only occur if a model had memorized the statistics spreadsheet and could identify from each transcript's text the appropriate episode number. To check for potential contamination, we test the first of these two steps: whether, given the episode and campaign information without the transcript text, models can answer Oolong-real questions. We use an augmented template as demonstrated below: Models are unable to perform this task. We run Gemini 3.1 Pro, GPT 5.2, and Claude Sonnet 4.5 on a set of 100 random examples with this format, prompting each model 3 times. The single best run gets 6 of the 100 questions exactly correct, with Gemini 3.1 Pro answering 3.33 correctly on average, GPT 5.2 answering 0.33 on average, and Claude Sonnet 4.5 answering 1.67 correctly on average. We examined these correct answers and found that they comprised two categories: numerical answers, which we believe represent “lucky guesses,” as no model gave consistent numerical answers across reruns for any question; and guesses that the “most frequent roll type” for a span is “attack” or “perception,” which are two of the most common roll types in D&D more broadly. We take this as evidence against widespread data contamination for Oolong-real. Appendix C Additional Related Work Dungeons & Dragons data Several prior works consider D&D as a potential testbed for model capabilities; for instance, we use a version of the Critical Role transcripts preprocessed by Rameshkumar and Bailey (2020), who proposed an episode summarization task using fan-written summaries. D&D data has also been used to study theory of mind (Zhou et al., 2023), user assistant development Zhu et al. (2023b), and dialogue generation, with game state either inferred from forum-based games (Callison-Burch et al., 2022) or captured through an external tool (Zhu et al., 2023a); to the best of our knowledge, we are the first to use fan annotations of gold labels and to consider the generation of these statistics as a task in its own right. Aggregation as an NLP concept Goldman et al. (2024) argue for the definition of long context tasks in terms of the information dispersion and input scope required to find the answer; under this taxonomy, Oolong is high dispersion (because relevant info is distribution over the full context length) and high scope (because most of the input is necessary for the task, particularly for Oolong-synth). DeYoung et al. (2024) measure synthesis in multi-document summarization by asking if summaries convey the consensus opinion of the input documents; this is a related task to the counting tasks in Oolong-synth, but framed as a measure of summarization capabilities instead of long context capabilities, and thus requires different considerations related to evaluating generated outputs. Relatedly, long generation benchmarks (Ye et al., 2025; Liu et al., 2024) measure long context ability through an entirely different lens: evaluating the ability to plan and produce coherent long outputs. Appendix D ICL label validation Table 4 describes each dataset used for Oolong-synth. Dataset Task # Labels Input Len Spam (Almeida et al., 2011) SMS spam classification 2 57 TREC-QC-coarse (Li and Roth, 2002; Hovy et al., 2001) Question type classification 6 39 AGNews (Zhang et al., 2015) Headline topic classification 4 90 App Reviews (Grano et al., 2017) Review sentiment classification 2†2^{\dagger} 49 Pavlick Formality (Lahiri, 2015; Pavlick and Tetreault, 2016) Formality classification 2†2^{\dagger} 51 IMDB reviews (Maas et al., 2011) Sentiment analysis 2†2^{\dagger} 376 HiTZ Negation (García-Ferrero et al., 2023) Verify claims about definitions 2 45 Yahoo Topics (Zhang et al., 2015) Question topic classification 10 74 MultiNLI (Williams et al., 2018) Entailment 3 70 Metaphors (Bizzoni and Lappin, 2018) Metaphor meaning validation 2 51 Table 4: Datasets used for Oolong-synth. †\dagger indicates cases where we combined similar labels to produce a smaller label-set for Oolong. The input length is the average per-instance length (in Llama 2 tokens), including our added date and user metadata. Dataset % Removed Spam 0.635% TREC-QC-coarse 0.048% AGNews 0.026% App Reviews 0.051% Formality 0.108% IMDB 0.042% Metaphors 0.000% MultiNLI 0.109% Negation 0.016% Yahoo Topics 0.188% Table 5: Only a small fraction of examples are removed during validation for each source dataset. Table 4shows the fraction of examples removed by the validation step for each dataset. For each ICL dataset used to construct Oolong-synth, we provide example instances that failed validation (and thus were discarded) and that passed validation and were used for constructing context windows. These examples are selected nearly-randomly; many of the instances that fail validation contain sexual content or offensive language, and we screen these out of the examples shown wherever possible. While the instances screened out vary by dataset, we note that the datasets that use labels scraped from web content (i.e. Yahoo Topics, AGNews, IMDB reviews, App Reviews) generally have higher rates of validation failures. Some of these failures appear to be genuine mislabelings, which is likely because of noise in the user behavior (e.g. a user asks a question about Business & Finance but mistakenly posts it in the Sports topic, or a user writes a review with mostly negative text but gives the product 4 stars). Input Label Passed validation? ‘Virtual Girlfriend' Demands Gifts HONG KONG - She needs to be coddled with sweet talk and pampered with gifts, but you'll never see her in the flesh. A Hong Kong company has developed a “virtual girlfriend” for new cell phones with video capability… World ✗ Court Hears Case of Brain Damaged Woman (AP) AP - The Florida Supreme Court questioned lawyers Tuesday about the extent of the power handed to Gov. Jeb Bush under a law that let him order the reinsertion of a brain-damaged woman's feeding tube. Sci/Tech ✗ Rooney backs fiercesome threesome (AFP) AFP - Wayne Rooney believes the three-pronged attack of himself, Michael Owen and Jermain Defoe can put England on the fast track to the World Cup finals. World ✗ From mouths of babes: What's hot, what's not How hot are the “Hot Dozen” toys? To find out, The Boston Globe put six toys from the 2004 Toy Wishes magazine Hot Dozen list in front of 13 kids from the Charlestown Boys and Girls Club. Business ✗ Crematory operator to get 12 years Ray Brent Marsh, who is to enter the plea Friday, had faced up to 8,000 years in a case that shocked the nation two years ago when investigators found hundreds of corpses at his rural northwest facility. Business ✗ Vending Machines Making Room for Healthy Products WASHINGTON (AP) - The typical vending machine fare consists of chocolate bars and potato chips, leaving few options for people seeking low-calorie or low-salt snacks. That is changing now as companies develop markets for products they expect to satisfy both nutritionists and consumers… Sci/Tech ✓ Venezuelans Line Up to Vote on Chavez CARACAS, Venezuela - Summoned by bugle calls and the detonations of huge firecrackers, Venezuelans turned out in unprecedented numbers Sunday to vote on whether to force leftist President Hugo Chavez from office. Some lines at polling places extended for 1.25 miles… World ✓ Singh Snares PGA Title Vijay Singh outlasts Justin Leonard and Chris DiMarco in a three-way playoff to win the PGA Championship on Sunday at Whistling Straits in Haven, Wisconsin. Sports ✓ China's Panchen Lama visits Tibet The boy named by the Chinese authorities as the 11th Panchen Lama visits a temple in Tibet. World ✓ The Region's Highest-Paid Executives Pay for the Washington area's top executives rose significantly last year, reversing the downward trend that set in with the recession in 2001. Business ✓ Table 6: AGNews examples that failed and passed validation. Input Label Passed validation? Kill Me.. positive ✗ Used to be great app. But since the last 7 updates always get - Authentication failed for: x@gmail.com positive ✗ App force closing on changing THEME PLZ FIX! positive ✗ You can't uninstall & I don't like my space being used. But am glad for the blind & deaf positive ✗ GOOGLE TALKBACK Read carefully & then give at least a good reviews this app is made only for persons who got disability such as a blind person & if you want to get rid this simply just ROOT your device dumb head!! negative ✗ This thing is great. I am at the bottom of the learning curve but in the few minutes I have played with it it seems like it will be easy to learn and to use. Just what I hoped for. positive ✓ Crash when set a demodulation! :( negative ✓ doesn't work with rtl-2832 Says it works with rtl devices but doesn't. never displays anything even though the rtl 2832 works with other spectrum analyzers. no help files anywhere so good luck figuring out what's wrong! negative ✓ Great app! Works really well with Galaxy note 5 positive ✓ Works great I am using HackRF One with this apps. positive ✓ Table 7: App Reviews examples that failed and passed validation. Input Label Passed validation? The Unit secret agent has signed on to appear in three episodes as a businessman being shown some houses - and, presumably, one bedroom in particular - by our randy realtor heroine. formal ✗ Barely 12, with large brown eyes and stick-like arms, Fandi is 3 years older than his brother - in his eyes almost a man. formal ✗ D-Lister Avril Lavigne appears on the cover of Z-List magazine Savvy this month. formal ✗ ORHS is going through it now, but from all I can perceive the district is out for the quick fix, chop off the principal, instead of really trying to assist the students and dig to the bottom to find the truth. formal ✗ Little Sally Draper (Kiernan Shipka) is a ”Patty Duke”-era girl - indeed, ”Mad Men” is currently set during the month ”The Patty Duke Show” premiered in 1963 - and on a ”Patty Duke”-like show, she'd just be a lispy Shirley Temple doll with a crush on Daddy. formal ✗ I have tried everything possible to attract business. formal ✓ Never heard that one before. informal ✓ ”Yup.” informal ✓ That's how we pass our traits to the next generation: through DNA formal ✓ Wedding rituals differ in different regions and communities in India. formal ✓ Table 8: Formality examples that failed and passed validation. Input Label Passed validation? Good movie, very 70s, you can not expect much from a film like this. Sirpa Lane is an actress of erotic films, a nice body but nothing exceptional. Not demand a lot from these films are light years away from the movies today, the world has changed incredibly. The plot is simple and the actors not extraordinary. positive ✗ This film Evil Breed: The legend of samhain contains very little thought or effort. It is ridiculed with specs of ultra fast ”slasher” style death and plain disgusting acts of death. The acting was rated a D as the actors show very little ability, and the stupidity of them in the film is too questionable. positive ✗ The movie ”Holly” may make the audience want to donate money towards organizations that improve the life for these poor youngsters, but the film's dramatic weaknesses may reduce its chances of being seen by enough people to make a difference. Overall, I think the concept is better as a documentary and it was not as touching as a movie. positive ✗ Although this film put Davis on the map due to her brilliantly intense performance, this film is strangely unsatisfying to me as a whole. What I cannot fathom for the life of me is just how or why Phillip would take the constant abuse this tramp constantly dishes out towards him. positive ✗ This film is just plain horrible. John Ritter doing pratt falls, 75% of the actors delivering their lines as if they were reading them from cue cards, poor editing, horrible sound mixing, and a plot that really goes nowhere. If I could sum this film up in one word, that word would be: Suckotrocity negative ✓ Zentropa has much in common with The Third Man, another noir-like film set among the rubble of postwar Europe. Like TTM, there is much inventive camera work. There is an innocent American who gets emotionally involved with a woman he doesn't really understand, and whose naivety is all the more striking in contrast with the natives. But I'd have to say that The Third Man has a more well-crafted storyline. Zentropa is a bit disjointed in this respect. Perhaps this is intentional: it is presented as a dream/nightmare, and making it too coherent would spoil the effect. This movie is unrelentingly grim-“noir” in more than one sense; one never sees the sun shine. Grim, but intriguing, and frightening. positive ✓ Never in my life have I come across a movie as bad as The Zombie Chronicles. Filmed on a budget of what looks to be about 20 bucks, TZC is a completely horrible horror movie that relies on lame, forgettable actors. Simply put, avoid TZC like a sexually-transmitted disease. negative ✓ Without wishing to be a killjoy, Brad Sykes is responsible for at least two of the most dull and clichéd films I've ever seen. The acting is terrible, the print is shoddy, and everything about this film screams ”seriously, you could do better yourself”. negative ✓ Table 9: IMDB examples that failed and passed validation. Only eight examples are shown because of the example length. Input Label Passed validation? Draupadi's eyes were diamonds. ↔\leftrightarrow Draupadi's eyes were beautiful. correct ✓ The faculty meeting was an easy breeze ↔\leftrightarrow The faculty meeting was very easy and relaxing correct ✓ The house was a tomb. ↔\leftrightarrow The house was big. incorrect ✓ I had already planted the idea in her mind. ↔\leftrightarrow I had already scared her about the idea. incorrect ✓ It is sad to observe the fruits of ignorance. ↔\leftrightarrow It is sad to observe the effects of ignorance. correct ✓ Table 10: Metaphors examples that passed validation. No examples failed validation for this dataset. Input Label Passed validation? You're safe. →\rightarrow You have nothing to worry about. neutral ✗ asks Burton, cupping his ear. →\rightarrow Burton wanted to ask what was going on, but he couldn't do much of anything while bound and gagged. contradiction ✗ I “poison my dearest Emily!” →\rightarrow How could you think I would poison Emily? entailment ✗ When he finally succeeded, after a prolonged siege and heavy losses, he punished the local population by cutting off the noses and lips of all men except those who played wind instruments. →\rightarrow All men who played wind instruments were tasked with helping cut off people's noses. neutral ✗ They died slowly, their eyes bulging and faces turning blue. →\rightarrow They died asphyxiated. neutral ✗ well you see that on television also →\rightarrow You can see that on television, as well. entailment ✓ Vrenna and I both fought him and he nearly took us. →\rightarrow Neither Vrenna nor myself have ever fought him. contradiction ✓ This analysis pooled estimates from these two studies to develop a C-R function linking PM to chronic bronchitis. →\rightarrow The analysis proves that there is no link between PM and bronchitis. contradiction ✓ He turned and smiled at Vrenna. →\rightarrow He smiled at Vrenna who was walking slowly behind him with her mother. neutral ✓ We sought to identify practices that were commonly implemented by the agencies within the past 5 years. →\rightarrow We want to identify practices commonly used by agencies in the last 5 years entailment ✓ Table 11: MultiNLI examples that failed and passed validation. Input Label Passed validation? No care is anything that serves as an enticement. True ✗ No care may be anything that serves as an enticement. True ✗ A loud utterance is an appropriate definition of produce in no context. True ✗ Complete attention is an appropriate definition of candidate in no context. True ✗ No stress is a message received and understood. True ✗ Action refers to a military engagement. True ✓ Action never stands for a distinguishing quality. True ✓ Action does not stand for a military engagement. False ✓ Action may stand for a military engagement. True ✓ Not a single action is a military engagement. False ✓ Table 12: Negation examples that failed and passed validation. Input Label Passed validation? Go chase after her and run her over while she's crossing the street ham ✗ i want to grasp your pretty booty :) ham ✗ No da if you run that it activate the full version da. ham ✗ i am seeking a lady in the street and a freak in the sheets. Is that you? ham ✗ Should i send you naughty pix? :) ham ✗ WINNER!! As a valued network customer you have been selected to receive a £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only. spam ✓ Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030 spam ✓ I'm gonna be home soon and i don't want to talk about this stuff anymore tonight, k? I've cried enough today. ham ✓ SIX chances to win CASH! From 100 to 20,000 pounds txt¿ CSH11 and send to 87575. Cost 150p/day, 6days, 16+ TsandCs apply Reply HL 4 info spam ✓ I HAVE A DATE ON SUNDAY WITH WILL!! ham ✓ Table 13: Spam examples that failed and passed validation. The examples that failed validation were almost exclusively ham messages with sexual content; the selected messages are the least inappropriate of the examples that failed validation. Input Label Passed validation? What explosive do you get by mixing charcoal, sulfur and saltpeter? entity ✗ What is the procedure called for drilling a hole in your skull to achieve a higher consciousness? entity ✗ Name 11 famous martyrs. human being ✓ What's the Olympic motto? description and abstract concept ✓ What is the highest waterfall in the United States? location ✓ What does the abbreviation AIDS stand for? abbreviation ✓ How many points make up a perfect fivepin bowling score? numeric value ✓ Table 14: TREC-coarse-Q examples that failed and passed validation. Only two examples failed validation. Input Label Passed validation? Is there a God? The question to end all questions, and begin them. Business & Finance ✗ why do we need to lie? Health ✗ try to type the word supercalifragilisticexpialidocious. 20 times fast you can only make 10 mistakes? This is for fun! Education & Reference ✗ do you think it is okay to tell a lie? Education & Reference ✗ where can you purchase cesium carbonate? Science & Mathematics ✗ What are good sources to find out about new gospel artists? Is there a site that focuses primarily on gospel? Entertainment & Music ✓ How a black hole is formed? I would like to know how a black hole can possibly be formed. Are there any experimental evidence of such creation? Science & Mathematics ✓ Economics of running a restaurant? Running a restaurant looks like hard work and long hours. What percentage of restaurants are profitable? Business & Finance ✓ Why doesn't the NBA implement a minor leagues? I don't want to see any more High School kids on the court. Sports ✓ how does a helicopter fly this is a miracle- I always wanted to learn to fly one of those. Can someone explain how can it get airborne? Science & Mathematics ✓ Table 15: Yahoo examples that failed and passed validation. One class of examples that failed validation, not demonstrated here, are sexually explicit questions labeled as seemingly random categories (likely from spam posters on the original Yahoo Answers forum). Appendix E Question types E.1 Oolong-synth Table 16, Table 17, and Table 18 list the questions used for each type of task in Oolong-synth. Counting In the above data, which of the labels is the most common? Give your final answer in the form ‘label: answer' where answer is one of the labels: {label_list}. In the above data, which of the labels is the least common? Give your final answer in the form ‘label: answer' where answer is one of the labels: {label_list}. In the above data, is label ‘{A}' more common, less common, or the same frequency as label ‘{B}'? Give your final answer in the form ‘Answer: {A} is [X]{B}', where [X]is ‘more common than', ‘less common than', or 'same frequency as'. In the above data, how many data points should be classified as label ‘{label}'? Give your final answer in the form ‘Answer: number'. Table 16: Counting questions used in the Oolong-synth dataset. User In the above data, which user is represented most often? Give your final answer in the form ‘User: [X]', where [X]is the user ID. In the above data, which user is represented the second most often? Give your final answer in the form ‘User: [X]', where [X]is the user ID. For the following question, only consider the subset of users with IDs {user_names}. Among these users, which user is represented most often? Give your final answer in the form ‘User: [X]', where [X]is the user ID. For the following question, only consider the subset of users with IDs {user_names}. Among these users, which user is represented the second most often? Give your final answer in the form ‘User: [X]', where [X]is the user ID. For the following question, only consider the subset of instances that are associated with user IDs {user_names}. Among instances associated with these users, {any of the Counting questions above} For the following question, only consider the subset of users with IDs {user_names}. Among these users, which user has the most instances with the label {label}? Give your final answer in the form ‘User: [X]', where [X]is the user ID. In the above data, which user has the most instances with the label {label}? Give your final answer in the form ‘User: [X]', where [X]is the user ID. In the above data, which user has more instances with the label {label}: User {A} or User {B}? Give your final answer in the form ‘User: [X]', where [X]is the user ID. Table 17: User questions used in the Oolong-synth dataset. Timeline In the above data, which date is represented most often? Give your final answer in the form ‘Date: [X]', where [X]is the date in the format MM/DD/YYYY. In the above data, which date is represented most often? Give your final answer in the form ‘Date: [X]', where [X]is the date in the format MM/DD/YYYY. In the above data, which date is represented the second most often? Give your final answer in the form ‘Date: [X]', where [X]is the date in the format MM/DD/YYYY. In the above data, how many dates are represented exactly {n} times? Give your final answer in the form ‘Answer: [X]', where [X]is the number of dates represented exactly {n} times. In the above data, was label ‘{key}' more common, less common, or the same frequency before {time}, as compared to after {time}? Give your final answer in the form ‘Answer: {key} is [X]before {time}', where [X]is ‘more common', ‘less common', or 'the same frequency'. In the above data, was label ‘{key}' more common, less common, or the same frequency before {time}, as compared to after {time}? Give your final answer in the form ‘Answer: {key} is [X]before {time}', where [X]is ‘more common', ‘less common', or 'the same frequency'. For the following question, only consider the subset of instances that occur in {month_name} of any year. Among instances occuring in {month_name},{any of the Counting questions above} For the following question, only consider the subset of instances that occur between {starting_date} and {ending_date}, inclusive. Among instances occuring in this date range, {any of the Counting questions above} In which month did the label ‘{label1} first occur more often than the label ‘{label2}'? Give your final answer in the form ‘Answer: [month][year]', where [month]is the name of the month and [year]is the four-digit year where ‘{label1}' first occured more often than ‘{label2}.' For how many months does the label ‘{label1}' occur more frequently than the label ‘{label2}'? Disregard months where there is a tie. Give your final answer in the form ‘Answer: [X]', where [X]is the number of months where ‘{label1}' occurs more often than ‘{label2}.' For how many months is the label ‘{label}' the single most frequently occuring label? Disregard months where there is a tie for the most common label. Give your final answer in the form ‘Answer: [X]', where [X]is the number of months where ‘{label}' is the most common label. Table 18: Timeline questions used in the Oolong-synth dataset. E.2 Oolong-real Table 19lists the question types used for single episode settings in Oolong-real dataset. In Table 20, Table 21 lists the types for multi-episode settings. Rolls Total number of rolls in this episode? (counting) Total number of rolls by the character {character name} in this episode? (counting, character) Total number of rolls by the player {player name} in this episode? (counting, player) Total number of rolls of type {roll type} in this episode? (counting, roll type) Number of rolls of natural value {roll value} in this episode? (counting, roll value) In this episode, what percentage of rolls were of value {roll value}? round to the nearest integer. (counting, roll value) What is the most common roll type in this episode? Return a comma separated list. (frequency) What is the least common roll type in this episode? Only include types with more than one roll. Return a comma separated list. (frequency) What is the most common natural roll value in this episode? Return a comma separated list. (frequency) What is the least common natural roll value in this episode? Only include values with more than one roll. Return a comma separated list. (frequency) What is the count of Crits? (natural rolls of value 1 or 20)? (counting, roll value) What is the count of Nat20s (natural rolls of value 20)? (counting, roll value) What is the count of Nat1s (natural rolls of value 1)? (counting, roll value) Spells How many spells were cast during this episode? (counting) How many spells were cast by the character {character name} in this episode? (counting, character) How many spells were cast by the player {player name} in this episode? (counting, player) How many {spell type} spells were cast during this episode? (counting, spell type) What is the first spell cast in this episode? (enumeration) What are the first two spells cast in this episode? (enumeration) What are the first three spells cast in this episode? (enumeration) What is the last spell cast in this episode? (enumeration) What are the last two spells cast in this episode? (enumeration) What are the last three spells cast in this episode? (enumeration) What is the first spell cast by each character in this episode? Return a comma separated list and retain the order of spells as they appear in the episode. (enumeration) What is the last spell cast by each character in this episode? Return a comma separated list and retain the order of spells as they appear in the episode. (enumeration) How many characters cast {spell name} spell in this episode? (counting, spell type) What is the most common spell in this episode? Return a comma separated list. (frequency) What is the least common spell in this episode? Only include spells that were cast at least once. Return a comma separated list. (frequency) Which spells were cast by more than one character in this episode? Return a comma separated list. (counting, enumeration) What is the total number of cantrip spells cast in this episode? (counting, spell type) In this episode, how many times was a spell cast at a level higher than its base level? (counting) In this episode, which spells were cast at a level higher than their base level? Return a comma separated list of unique spells. (enumeration) Table 19: Question types used in the Oolong-real dataset (single episode). For reference, we provide the question types in parentheses. Rolls What is the cumulative total of rolls by the end of episode {episode index}? Count the number of rolls and not the values of the rolls. (counting) What is the cumulative total of rolls by the character {character name} at the end of episode {episode index}? Count the number of rolls and not the values of the rolls. (counting, character) Total number of rolls across all the episodes? (counting) Total number of rolls by the character {character name} across all episodes? (counting, character) Total number of rolls by the player {player name} across all episodes? (counting, player) Total number of rolls of type {roll type} across all episodes? (counting, roll type) Number of rolls of natural value {roll value} across all episodes? (counting, roll value) Across all episodes, what percentage of rolls were of value {roll value}? round to the nearest integer. (counting, roll value) What is the most common roll type across all episodes? Return a comma separated list. (frequency) What is the least common roll type across all episodes? Only include types with more than one roll. Return a comma separated list. (frequency) What is the most common natural roll value across all episodes? Return a comma separated list. (frequency) What is the least common natural roll value across all episodes? Only include values with more than one roll. Return a comma separated list. (frequency) What is the total count of Crits across all episodes? (natural rolls of value 1 or 20)? (counting, roll value) What is the total count of Nat20s across all episodes? (natural rolls of value 20)? (counting, roll value) What is the total count of Nat1s across all episodes? (natural rolls of value 1)? (counting, roll value) Table 20: Question types used in the Oolong-real dataset (multi-episode). For reference, we provide the question types in parentheses. Spells What is the cumulative total of spells cast by the end of episode {episode index}? (counting, indexing) What is the first spell cast in the episode {episode index}? (enumeration, indexing) What is the second spell cast in the episode {episode index}? (enumeration, indexing) What is the third spell cast in the episode {episode index}? (enumeration, indexing) List the first spell cast in each episode? Return a comma separated list. (enumeration) List the last spell cast in each episode? Return a comma separated list. (enumeration) List the first spell cast by the character {character name} in each episode? Return a comma separated list. (enumeration, character) List the last spell cast by the character {character name} in each episode? Return a comma separated list. (enumeration, character) How many spells were cast across all episodes? (counting) How many spells were cast by the character {character name} across all episodes? (counting, character) How many spells were cast by the player {player name} across all episodes? (counting, player) How many {spell type} spells were cast across all episodes? (counting, spell type) How many characters cast {spell name} spell across all episodes? (counting, spell type) What is the most common spell across all episodes? Return a comma separated list. (frequency, enumeration) What is the least common spell across all episodes? Only include spells that were cast at least once. Return a comma separated list. (frequency, enumeration) What is the total number of cantrip spells cast across all episodes? (counting, spell type) Across all episodes, how many times was a spell cast at a level higher than its base level? (counting, spell level) Across all episodes, which spells were cast at a level higher than their base level? Return a comma separated list of unique spells. (enumeration, spell level) Table 21: Question types used in the Oolong-real dataset (multi-episode). For reference, we provide the question types in parentheses. Appendix F Deepseek R1 trace examples Figure 8 and Figure 9 contrast the behavior of this model on representative examples from the Oolong-synth and Oolong-real splits. See the main text for more discussion. Appendix G Question type analysis G.1 Oolong-synth On Oolong-synth, we further break down results by question type (counting, user, or temporal) and answer type (the expected format of the answer, e.g. as a label or number or date) in Figure 10. Consistently, temporal questions are the most challenging for models, highlighting the difficulty of temporal reasoning. Figure 10 shows that temporal questions are challenging for models; this is reflected in both the performance by task type and the distribution of score by answer type. Questions that require a date or month/year (e.g. “January 2021”) as an answer show generally lower performance for the same model, and show greater spread in model capabilities than the other answer categories. For instance, the gap between GPT-5 and GPT-5-nano performance is more than 4x larger for questions that require outputting a date than for questions that require outputting a label. Relative model performance is mostly stable across answer types, although Claude-Sonnet-4 is relatively much stronger on numerical reasoning and comparisons than the GPT series models. Figure 10: The performance trend for models by type of answer and type of task on Oolong-synth. G.2 Oolong-real On Oolong-real, we divide the questions into ten categories: counting, enumeration, frequency, indexing, player, character, roll type/value, spell level/type (Table 19, Table 21. Table 20). Across models, we find ‘frequency' to be the hardest and ‘spell type' the easiest (Figure 11). Except for Gemini-2.5-Pro and GPT-5, all models struggle with enumeration and indexing questions. Gemini-2.5 Pro and GPT-5 also make effective use of the player-character mapping provided in the system prompt. In Figure 11, we report model performance by question type on Oolong-real. Figure 11: Scores by context window length for various question types in Oolong-real. Some questions are only relevant for multi-episode settings. Appendix H Oolong-real data H.1 Data quality and curation The episode transcripts were first hand-transcribed by members of the fan community, who implemented a multi-step editing process with editorial guidelines. These were then further cleaned by Rameshkumar and Bailey (2020) to resolve any remaining inconsistencies. We refer the reader to the original paper for more details on the curation process. The stats were manually compiled by volunteers and the team at CritRoleStats. While it's not possible to completely eliminate sources of potential error, these types of fan-documentation processes are fastidiously annotated and cross-checked by multiple people- a standard of care far higher than that of a paid annotator with no connection to the task or data domain. H.2 Challenges of Oolong-real In the tables below, we highlight various unique challenges presented by Oolong-real. Figure 12 covers text that includes in-character, out-of-character and narrative content. Additionally, we show complex discussions surrounding dice rolls (Figure 13, Figure 15) and damage calculation (Figure 14). Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" () button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. We gratefully acknowledge support from our major funders, member institutions, , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2512.04307.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2512.04307.txt new file mode 100644 index 0000000..497f38e --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2512.04307.txt @@ -0,0 +1 @@ +Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Related Work 3 Long-Context Reasoning Benchmark 3.1 Problem Formulation 3.2 Task Definition 3.3 Subtask Generation 4 Evaluation 4.1 Task Success Criteria 4.2 Implicit RAG (iRAG) Approach 5 Experiments 5.1 Retrieval Performance Analysis 5.1.1 Implicit RAG Retrieval Performance 5.2 Efficiency Ratio 6 Conclusion 7 Limitations 8 Reproducibility Statement References A Technical Appendices and Supplementary Material A.1 WebAgent Architecture A.2 System Prompts A.2.1 WebAgent system prompt A.2.2 Implicit RAG Prompt A.3 Sample Implicit RAG summary A.4 Summary Relevance Prompt A.5 Sample Trajectory A.6 List of successful tasks A.7 Sample interaction history A.8 False End Example A.8.1 Self evaluation A.9 Inefficent Progress Example A.10 Loop Example A.11 Implicit RAG analysis A.11.1 Failed retrieval without iRAG. A.11.2 With iRAG, successful retrieval of correct trajectory and completion of the task A.11.3 Summary used in the above task. License: CC BY 4.0 arXiv:2512.04307v1 [cs.LG] 03 Dec 2025 \workshoptitle LAW 2025: Bridging Language, Agent, and World Models Evaluating Long-Context Reasoning in LLM-Based WebAgents Andy Chung Affiliation: University of Michigan Yichi Zhang Affiliation: University of Michigan Kaixiang Lin Affiliation: Amazon Aditya Rawal Affiliation: Amazon Qiaozi Gao Affiliation: Amazon Joyce Chai Affiliation: University of Michigan Abstract As large language model (LLM)-based agents become increasingly integrated into daily digital interactions, their ability to reason across long interaction histories becomes crucial for providing personalized and contextually aware assistance. However, the performance of these agents in long context scenarios, particularly for action-taking WebAgents operating in realistic web environments, remains largely unexplored. This paper introduces a benchmark for evaluating long context reasoning capabilities of WebAgents through sequentially dependent subtasks that require retrieval and application of information from extended interaction histories. We develop a novel evaluation framework that simulates multi-session user interactions by injecting irrelevant task trajectories between dependent subtasks, creating contexts ranging from 25,000 to 150,000 tokens. Through extensive evaluation of four popular models, Claude-3.7, GPT-4.1, Llama 4, and o4-mini, we observe a dramatic performance degradation as context length increases, with success rates dropping from 40-50% in baseline conditions to less than 10% in long context scenarios. Our detailed error analysis reveals that agents primarily fail due to getting stuck in loops and losing track of original task objectives. We further propose an implicit RAG approach that provides modest improvements by generating task-relevant summaries, though fundamental limitations in long context reasoning persist. These findings highlight critical challenges for deploying WebAgents in realistic, long-term user interaction scenarios and provide insights for developing more robust agent architectures capable of maintaining coherent task execution across extended contexts. 1 Introduction Action-taking large language model (LLM)-based agents are rapidly becoming ubiquitous in our daily lives. These assistants extend their capabilities beyond the chatbot environment, making tangible impacts through their ability to operate in the digital realm. For instance, an action-taking agent can go beyond merely suggesting vacation plans-it can go online to book your flights and reserve your hotel as well. As these assistants become more capable, users will increasingly expect them to recognize and adapt to their preferences and intentions. A user may ask the assistant to find the highest-rated movie on IMDb and add it to your watchlist. Days later, they might simply say, “Buy that movie we saved earlier,” and expect it to understand the reference. While there have been many studies on the personalization of LLMs for chatbots and information retrieval assistants [7, 14, 15], studies exploring the personalization capabilities of LLM-based action-taking agents are lacking. We study WebAgents in the live open Internet, one of the most information-rich environments on earth. Its usage is ubiquitous in North America with 96% of the population online. This environment also enables an unparalleled variety of tasks such as shopping, social media, banking, entertainment, or research. This diversity enables us to design the experiment to closely reflect real-world human behavior. Despite the challenges and limitations of studying live environments, this approach enables us to observe how these agents perform under realistic conditions. We explore the capabilities of these types of agents in a realistic long context situation by performing the following: • We introduce a new benchmark for evaluating web agents on long-horizon, sequential tasks. The benchmark includes a dataset of sequentially dependent subtasks and a novel methodology that simulates long-term interaction by injecting noisy historical trajectories of varying lengths into the agent's context. • We conduct a comprehensive evaluation of state-of-the-art web agent models on our benchmark, showing significant performance degradation as context length increases. Through a detailed analysis of reasoning traces, we identify and categorize key failure modes related to the agent's planning and reasoning abilities in long-context settings. • We propose an implicit Retrieval-Augmented Generation (iRAG) approach that improves task success rates by retrieving relevant information from the lengthy noisy context history. Our results and error analysis show several key insights. Task success rates drop dramatically when noise is introduced, dropping by about 30% from the case when there is no noise. Analysis of the reasoning traces show that the WebAgent tends to get stuck in loops, repeating the same ineffective action over and over again until the step limit is reached. The WebAgent's task efficiency drops sharply as context length increases with most cases reaching the step limit. Breaking down the instructions into sub-instructions can assist the WebAgent in completing tasks when the context is long. 2 Related Work The meteoric rise of powerful large language models (LLMs) has shown strong capabilities in reasoning in dynamic open world environments such as the world wide web [4, 5, 6, 12, 17]. These generally capable agents are able to complete complex tasks by breaking them down into multiple steps and then completing those steps in a sequential stepwise fashion. Prefeval is a benchmark that evaluates a LLM's capability to adhere to user preferences in a long context conversational setting. The experimental setup involves long context retrieval of information in order to produce correct outputs. However, the setting mimics a QA type of instruction rather than a trajectory of an action-taking agent [16]. RealWebAssist is a benchmark that evaluates long-horizon web-based tasks under realistic scenarios and includes ambiguous instructions an user may give. The study also investigates the performance of a WebAgent at varying context lengths. However, the study only reports an analysis of performance at relatively short context lengths of up to 20 steps. A granular analysis of long-context interactions which could include hundreds of steps was not provided nor is there a detailed analysis of errors [13]. 3 Long-Context Reasoning Benchmark 3.1 Problem Formulation Over time, our interaction history becomes a rich source of contextual information, enabling these agents to better interpret our intentions and personalize their responses. As our engagement with them deepens, both the variety and complexity of tasks naturally increase. To remain useful in our daily lives over the long term, an assistant must be able to reason across this expanding context. For our benchmark, we setup a scenario to simulate a user's interaction over various periods of time. First we have the WebAgent perform subtask A1A_{1}. After the completion of subtask A1A_{1}, We inject irrelevant trajectories up to a certain absolute context length: [25,000, 50,000, 75,000, 100,000, 125,000, 150,000]. Each step in each trajectory contains a thought, action, reflection, and the current URL at that step. This simulates the passage of time as if users completed a number of tasks after completing A1A_{1}. The number of tasks injected depends on the target context length of the experiment, as shown in Figure 2. We then set the current URL of the agent to the last step of the last injected task to simulate the completion of that step. The agent is then asked to perform another subtask, A2A_{2}. We also have a 'no noise' condition where no irrelevant tasks are injected as a baseline. 2 shows the number of tasks injected at each context length. We use these results as our base case. 3.2 Task Definition Figure 1: Benchmark framework and evaluation overview. The left column shows the setup of the multi-session context. The middle column provides examples of the split subtasks as well as the (irrelevant) multi-session tasks. The right flowchart demonstrates benchmark and evaluation process for the trajectory the agent takes. Context consists of instruction, observation, thought, action, and reflection stored in memory as the interaction history. Over time, this interaction history becomes complex and long as more and more tasks are fulfilled by the agent. In order to study long context understanding over this interaction history, we divide a given task into two sequentially dependent subtasks. We formulate the task as follows: - Let AA = Task - Let A1=A_{1}= sequential subtask of AA - Let A2=A_{2}= second sequential subtask of AA A2A_{2} must satisfy two properties: 1. Under-specification: The second sub-task must be under-specified by some attribute. For example, if A1A_{1} was to find hotels near the airport, then A2A_{2} could be to find restaurants near that airport. In this case, the attribute hotel from the first sub-task is not specified. This forces the LLM to recall that piece of information from a previously executed trajectory 2. Unambiguous: A2A_{2} must depend solely on A1A_{1} in the sequence. As previously noted, additional tasks will be introduced into the context. These injected tasks must be orthogonal from A1A_{1}. If any injected task bears resemblance to A1A_{1}, the agent might justifiably base A2A_{2} on the injected task rather than following A1A_{1}'s trajectory. To accurately assess the dependency chain, we must ensure that A1A_{1} remains clearly distinct from the injected subtasks. Therefore, we omit any trajectory other than A1A_{1} that may reasonably be used to complete A2A_{2}. 3.3 Subtask Generation Initially, we attempted to generate a dataset that satisfies the properties above using a LLM. However, we found that while many of the tasks generated seem plausible, they end up being unexecutable and/or difficult to evaluate. This is likely due to the superficial understanding current SOTA foundational models have about website affordances. For example, one of the tasks generated was "goto expedia.com and navigate to the "Flights" Search Page ( A1A_{1}). Then click the "Search" button without entering any information ( A2A_{2})." The task is plausible. However, if you were to actually try this, you would get an error saying the "leaving from" and "going to" fields are empty, rendering the task unexecutable. While there are ways to autonomously generate feasible tasks [18], we have the additional constraints of generating two sequentially dependent subtasks that need to be executable along with the properties mentioned above. Thus, we utilize an existing dataset from WebCanvas [9] to source our executable tasks. WebCanvas offers a collection of 543 annotated tasks (438 in the dev set, 105 in test set) spanning diverse domains, including travel, information retrieval, services, shopping, and entertainment. The tasks are also curated to remove any time-sensitive metrics such as current temperature or tasks involving making a booking at a certain date. There is an average of 8 steps per tasks with a total of 4550 total steps across all tasks. Figure 2: Number of tasks injected at each context length Although WebCanvas provides a large number of tasks, many of the tasks are no longer executable due to the dynamic nature of the live Internet. Therefore, to ensure the tasks are still executable, we curate a set of successful tasks by running the WebAgent across the entire dataset using Claude-3.5 and 3.7 as base models, yielding a total of 56 tasks. From the successful tasks, we then prompt a LLM to split the task into A1A_{1} and A2A_{2}. We manually inspect A1A_{1} to make sure it contains all the information needed to complete that part of the task and inspect A2A_{2} to make sure they satisfy the under-specification requirement. To ensure that A2A_{2} is unambiguous, we use a LLM to classify any task (other than A1A_{1}) that can be used to reasonably complete A2A_{2}. 4 Evaluation We evaluate Claude-3.7, GPT-4.1, Llama-4-17B Maverick, and o4-mini on our benchmark due to their relatively strong performance on agentic tasks as well as their ability to support long context lengths. For Claude-3.7, we do not use thinking tokens. For all models except o4-mini, we use temperature = 0 to provide more stable results. Error bands are calculated using the standard error of proportions for binary variables. For continuous variables we use standard error of the mean. 4.1 Task Success Criteria In our experiments, we found that evaluations that were purely judged by a LLM tend to fluctuate. This is also evidenced by previous studies such as [6], and [8]. The LLM may judge an incomplete task or a similar but incorrect final state as successful. Even with temperature set to 0, agents do not take the same trajectory given a task, which will affect the LLM's judgment. In practice, LLMs are not deterministic [2], producing variations at the string level even with temperature = 0. This string level variances are exacerbated due the the long reasoning chains needed to complete multi-step agentic tasks. This variance is also noted in Anthropic's documentation [1]. Additionally, the live Internet is dynamic in nature. For example, a different ad may be displayed on every visit to the website, affecting the state and context which in turn would affect the decisions of the agent and therefore the trajectory of a given task. Thus, we utilize key steps as described in WebCanvas for evaluation of the tasks rather than LLM based evaluations. Key steps are checkpoints that must be passed in the trajectory for a given task regardless of any variation in other steps of the trajectory. For example, the task "Add all dlc to cart for the game DOTA on steam" would have the following keysteps: 1. Goto steampowered.com 2. Load the landing page for the DOTA 2 game. 3. Click on the "add all dlc to cart" button. They are rule based, thus removing one layer of variation in our evaluations. For a task to be considered successful, all key steps for a given task must be successful. On average, there are 4.5 key steps per task. 4.2 Implicit RAG (iRAG) Approach Figure 3: Comparison of performance of baseline vs implicit RAG at 150k context length. Each step, the WebAgent is instructed to follow many instructions in one step including observation of the accessibility tree and the current screenshot, reasoning over the interaction history, utilizing feedback from the previous step, generating a thought with respect to the current state and the given task, generating the next action, identifying the correct web element to perform the action on, and responding in the specified structured output format. We hypothesize that the long context interferes with the agent's ability to follow these instructions, and crucially, retrieve relevant information needed to successfully complete a given task. Thus, we add a separate 'implicit RAG' step to assist the WebAgent. In this step, we simplify the instruction and ask the WebAgent to only generate a summary with respect to its current sub-task, state, and the provided interaction history. The intuition behind this approach is to break a complex instruction into sub-instructions, just like how breaking up a complex task into sub-tasks can make task completion easier. The summary is then appended to the context and the step is proceeded as before. 5 Experiments Figure 4: Success rate of task completion at varying context lengths Our results, shown in Figure 4, demonstrate a sharp drop in performance once noise is injected into the trajectory dropping from 40-50% to as low as <10<10% success rate. This indicates that once retrieval is required of the WebAgent, the success rate drops dramatically. O4 mini performs relatively well, indicating that reasoning models outperform non-reasoning models in long context reasoning tasks. In the implicit RAG approach, we see an improvement in the performance of the WebAgent compared to the baseline for Claude-3.7, GPT-4.1, and o4-mini at 150k context length. This suggests that decomposing complex instructions into smaller sub-instructions enhances WebAgent's performance. Context Length o4-mini GPT-4.1 No Noise 46.2% ± 3.8% 44.4% ± 3.8% 150000 21.2% ± 3.1% 2.7% ± 1.1% Table 1: Performance comparison of o4-mini and GPT-4.1 Model Category Percentage ±\pm Std Error Claude 3.7 False End 16.37% ±\pm 1.47% Inefficient Progress 34.97% ±\pm 2.65% Loops 16.67% ±\pm 1.66% Other Errors 1.19% ±\pm 0.40% Success 30.80% ±\pm 3.48% GPT 4.1 False End 6.90% ±\pm 1.09% Inefficient Progress 32.74% ±\pm 5.63% Loops 44.29% ±\pm 8.03% Other Errors 1.67% ±\pm 0.51% Success 14.40% ±\pm 4.22% Llama 4 False End 11.61% ±\pm 3.24% Inefficient Progress 39.96% ±\pm 3.80% Loops 21.21% ±\pm 1.37% Other Errors 0.22% ±\pm 0.22% Success 27.01% ±\pm 5.10% o4-mini False End 2.68% ±\pm 0.61% Inefficient Progress 19.35% ±\pm 5.16% Loops 43.15% ±\pm 2.86% Other Errors 0.89% ±\pm 0.40% Success 33.93% ±\pm 6.17% Table 2: Error categories of tested models. Most common error modes are bolded. In order to better understand why the WebAgents fail, we categorize the errors into the following categories: • False End: The WebAgent decides to end the task but has not completed all key steps. • Step Limit - Loops: The WebAgent reaches the step limit and is stuck in a loop. - Inefficient Progress: The WebAgent is progressing in the task but has reached the step limit. • Other Errors: These include technical errors like timeouts, anti-bot measures, or LLM request rejections. The breakdown of error categories does not show a clear trend with respect to context length. The majority of errors are step limit errors due to either being stuck in a loop or progressing the task inefficiently. Notably, GPT-4.1 tends to get stuck in a loop more often than Claude-3.7. This may indicate Claude models have additional mechanisms implemented to deal with situations where the input repeatability remains the same in a multi-turn setting. False ends happen less frequently than step limit errors. Other errors are rare and include websites timing out for extended periods of time, getting blocked by anti-bot measures, or the LLM refusing the request due to the request being perceived as harmful. Analysis of the reasoning traces indicates that once the WebAgent is stuck in a loop, it has a difficult time breaking out of it even when explicitly instructed to recognize the loop and attempt alternative actions. In most cases, it retries the same ineffective action over and over until the step limit is reached. However, there are some cases where the agent tries the same action a few times and then decides to try a different action, breaking it out of the loop. Analysis of reasoning traces when the step limit is reached and it is not a loop shows, in many cases, the agent exploring the wrong parts of the website or on an entirely different domain than what it should be on. In this scenario, the agent is unable to realize it has deviated from the correct trajectory and searches various parts of the website or tries to go to a different website until it reaches the step limit. Analysis of false ends show that in many cases, the agent ends up on a different website that accomplishes a similar goal despite the task asking for completion of the task on the given website. For example, the task may ask to find NBA stats on a player on sports.yahoo.com but the WebAgent ends up finding NBA stats on nba.com. In another instance, only partial information about the original subtask was retrieved. The WebAgent decides to end the task due to satisfaction of this partially retrieved information. For example, one of the tasks was to find cruises on www.carnival.com with Miami, FL as the port and the Bahamas as the destination. It was able to recall that it was supposed to search for cruises on www.carnival.com and returned to the correct domain, but did not recall that it was supposed to find the cruises for that specific route. The WebAgent decided to end the task after performing a generic cruise search. The WebAgent may also decide to end the task because it falsely concludes that it is impossible. This is due to it deviating from the correct trajectory. In the most egregious case, the agent completely loses track of the original task and thinks it has successfully completed the last injected task. 5.1 Retrieval Performance Analysis Model TS, RF (%) TS, RS (%) TF, RS (%) TF, RF (%) Claude 3.7 0.0 ± 0.0 25.1 ± 2.0 26.9 ± 1.5 48.0 ± 2.3 GPT 4.1 0.0 ± 0.0 7.0 ± 1.4 21.0 ± 4.4 72.0 ± 5.3 Llama 4 0.0 ± 0.0 17.1 ± 2.8 42.9 ± 1.9 40.0 ± 4.1 o4-mini 0.0 ± 0.0 21.7 ± 1.2 37.4 ± 2.3 40.9 ± 3.2 Table 3: Retrieval performance on the benchmark, aggregated across context lengths. Error modes are grouped by frequency, with the most prevalent types emphasized in bold. For Llama, evaluations were conducted up to a maximum context length of 75,000 tokens. Abbreviations: TS - Task Success; RS - Retrieval Success; TF - Task Failure; RF - Retrieval Failure. Figure 5: Breakdown of retrieval performance and task performance After the noise injection step, the agent is set to an irrelevant domain. In order for A2A_{2} to be successful, the WebAgent must be able to retrieve the correct domain and maintain the correct trajectory. To dive deeper into the how retrieval affects success rates, we categorize each task into 4 categories. The task is successful if all keysteps are completed. Retrieval is considered successful if the WebAgent is able to return to the correct domain after being set to an irrelevant domain. Thus, in order for a task to be successful, it must also successfully retrieve the correct domain. Likewise, if the retrieval fails, then the task is guaranteed to fail. Successful retrieval of the required information does not guarantee task success. In these cases, the WebAgent correctly returns to the intended domain but either hits the step limit or concludes prematurely without completing all keysteps. Notably, this was the most common failure mode in Llama 4. In most cases, when both task execution and retrieval fail, the WebAgent loses track of the original subtask. Instead, it continues following the most recent injected trajectory. For example, if the task was to find the park map for Six Flags in Mexico but the most recent injected trajectory involed finding restaurants on Yelp, the WebAgent tended to continue on trying to find restaurants on Yelp rather than return to Six Flags' website. This indicates that the noise is overwhelming the agent's ability to not only recall the original trajectory but also to follow the instruction of completing the given subtask. 5.1.1 Implicit RAG Retrieval Performance The premise behind the implicit RAG (iRAG) approach is to try to improve the retrieval ability of the WebAgent so we expect higher rates of retrieval in the case where the summary is provided. We do see a modest improvement of retrieval counts in the iRAG case vs the baseline in Claude 3.7 and Llama 4. Notably, GPT-4.1 seems to benefit the most from the iRAG approach 5. To understand how iRAG may be improving performance, we compare the reasoning traces for a failed task without iRAG and a corresponding successful task with iRAG to see how the summary is being used by the WebAgent. In the example we found, the task without iRAG failed to recover the original task and got stuck in a loop while trying to continue an injected trajectory. With iRAG, the WebAgent was able to retrieve the correct trajectory of where it left off and successfully complete the given subtask. However, in a few cases, a generated summary may be relevant to the subtask, indicating that retrival was successful, but the WebAgent may not perform the appropriate action, showing it was unable to utilize relevant information to carry out the next appropriate action. In cases such as this, the WebAgent continues on the trajectory of the last injected subtask, indicating that successful retrieval does not guarantee correct usage of that information. 5.2 Efficiency Ratio Efficiency ratio is defined as # Reference Steps# WebAgent Steps\frac{\text{\# Reference Steps}}{\text{\# WebAgent Steps}} where WebAgent steps is the number of steps the WebAgent took to complete the task and reference steps is the amount of steps the human annotator took to complete the task [3]. When no noise is injected, the WebAgent is able to perform with near human performance, and in some cases, exceeding human performance. The efficiency sharply drops as context length increases. Note, while the reference task length is for the original unmodified task and is not a direct comparison to the sequential dependent task scenario, they still provide a relative number to how many steps the sequential subtasks would take. This trend corroborates the breakdown of error categories where we see a majority of errors being step limit errors. Figure 6: Efficiency Ratio for tested models across different context lengths. 6 Conclusion In this paper, we introduce a benchmark for evaluating WebAgents in long-context, multi-session scenarios with sequential dependencies requiring retrieval from prior exchanges. Our experimental results reveal significant challenges for current state-of-the-art language models when operating as WebAgents in long context environments. Experiments show that popular models such as Claude-3.7, GPT-4.1, Llama-4, and o4-mini experience severe performance drops-from 40-50% to under 10%-when tasked with retrieving information beyond 25k tokens, with failures often caused by looping, inefficient progress, and loss of task objectives. While our implicit RAG approach, which breaks complex instructions into sub-instructions, yields modest gains at 150k context lengths, overall success rates remain low, indicating that long contexts still overwhelm reasoning processes. These results underscore the need for more robust memory architectures, better context filtering, and stronger planning capabilities to enable reliable, coherent task execution in realistic, long-term user interactions. Future work should focus on developing more robust memory architectures, improved context filtering mechanisms, and enhanced planning capabilities that can operate effectively in the complex, information-rich environments where these agents will ultimately serve users. 7 Limitations Due to API rate limitations and the nature of long-context tasks, experiments in this area take a prohibitively long time. One experimental run takes about 16 days to complete which makes obtaining results difficult. The generated subtasks A1A_{1} and A2A_{2} may not be realistic. Many of the tasks in WebCanvas are atomic. For example, there is no realistic way to split the following task: Find the price of Ipod Mini on Apple.com. Attempting to split this task would result in two unnatural subtasks, such as navigating to the Ipod mini page, then looking for the price as a separate task. The Internet is dynamic, meaning it is possible that a task that was working before would stop working due to the layout of the page changing and vise versa. Due to the proliferation of autonomous agents online, many websites have been taking measures to block this kind of traffic. Other, less impactful changes could also alter an agent's trajectory. For example, an ad that is displayed on a website would likely change on every visit to a website. In practice, LLMs are not [2] deterministic. Non-determinism, even when temperature is set to 0 is also noted in the Anthropic documentation [1]. This, along with the dynamic nature of the website would add variation to the agent's trajectory. 8 Reproducibility Statement Due to the dynamic nature of the open Internet and the nondeterministic nature of LLMs, it is not possible to have exact reproducibility of the results. However, we take steps to minimize the fluctuations of the results. We sample all models using a temperature of 0. While this minimizes randomness, it does not guarantee it as noted in the documentation of OpenAI and Anthropic APIs. We minimize the use of LLM based judgments and evaluate using key-steps as implemented in WebCanvas which are rule based. The WebCanvas benchmark also curates a set of task that removes evaluations that are sensitive to time, such as today's weather or hotel bookings for certain dates. References [1] Anthropic (2025) Anthropic api documentation. Note: Accessed: 10 May 2025 External Links: Link Cited by: §4.1, §7. [2] B. Atil, S. Aykent, A. Chittams, L. Fu, R. J. Passonneau, E. Radcliffe, G. R. Rajagopal, A. Sloan, T. Tudrej, F. Ture, Z. Wu, L. Xu, and B. Baldwin (2025) Non-determinism of "deterministic" llm settings. External Links: 2408.04667, Link Cited by: §4.1, §7. [3] X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su (2023) Mind2Web: towards a generalist agent for the web. External Links: 2306.06070, Link Cited by: §5.2. [4] H. Furuta, K. Lee, O. Nachum, Y. Matsuo, A. Faust, S. S. Gu, and I. Gur (2024) Multimodal Web Navigation with Instruction-Finetuned Foundation Models. arXiv. Note: arXiv:2305.11854 [cs, stat] External Links: Link, Document Cited by: §2. [5] I. Gur, H. Furuta, A. Huang, M. Safdari, Y. Matsuo, D. Eck, and A. Faust (2024) A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. arXiv. Note: arXiv:2307.12856 [cs] External Links: Link, Document Cited by: §2. [6] H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu (2024) WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. arXiv. Note: arXiv:2401.13919 [cs] External Links: Link, Document Cited by: §2, §4.1. [7] H. Lyu, S. Jiang, H. Zeng, Y. Xia, Q. Wang, S. Zhang, R. Chen, C. Leung, J. Tang, and J. Luo (2024) LLM-rec: personalized recommendation via prompting large language models. External Links: 2307.15780, Link Cited by: §1. [8] J. Pan, Y. Zhang, N. Tomlin, Y. Zhou, S. Levine, and A. Suhr (2024) Autonomous evaluation and refinement of digital agents. External Links: 2404.06474, Link Cited by: §4.1. [9] Y. Pan, D. Kong, S. Zhou, C. Cui, Y. Leng, B. Jiang, H. Liu, Y. Shang, S. Zhou, T. Wu, and Z. Wu (2024) WebCanvas: benchmarking web agents in online environments. External Links: 2406.12373, Link Cited by: §A.1, §3.3. [10] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. External Links: 2303.11366, Link Cited by: §A.1. [11] J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao (2023) Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. External Links: 2310.11441, Link Cited by: §A.1. [12] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. External Links: 2210.03629, Link Cited by: §A.1, §2. [13] S. Ye, H. Shi, D. Shih, H. Yun, T. Roosta, and T. Shu (2025) RealWebAssist: a benchmark for long-horizon web assistance with real-world users. In , External Links: Link Cited by: §2. [14] K. Zhang, L. Qing, Y. Kang, and X. Liu (2024) Personalized llm response generation with parameterized memory injection. ArXiv abs/2404.03565. External Links: Link Cited by: §1. [15] Z. Zhang, R. A. Rossi, B. Kveton, Y. Shao, D. Yang, H. Zamani, F. Dernoncourt, J. Barrow, T. Yu, S. Kim, R. Zhang, J. Gu, T. Derr, H. Chen, J. Wu, X. Chen, Z. Wang, S. Mitra, N. Lipka, N. Ahmed, and Y. Wang (2025) Personalization of large language models: a survey. External Links: 2411.00027, Link Cited by: §1. [16] S. Zhao, M. Hong, Y. Liu, D. Hazarika, and K. Lin (2025) Do llms recognize your preferences? evaluating personalized preference following in llms. External Links: 2502.09597, Link Cited by: §2. [17] B. Zheng, B. Gou, J. Kil, H. Sun, and Y. Su (2024) GPT-4V(ision) is a Generalist Web Agent, if Grounded. arXiv. Note: arXiv:2401.01614 [cs] External Links: Link, Document Cited by: §2. [18] Y. Zhou, Q. Yang, K. Lin, M. Bai, X. Zhou, Y. Wang, S. Levine, and E. Li (2024) Proposer-agent-evaluator(pae): autonomous skill discovery for foundation model internet agents. External Links: 2412.13194, Link Cited by: §3.3. Appendix A Technical Appendices and Supplementary Material A.1 WebAgent Architecture Figure 7: WebAgent architecture overview. Each decision making step consists of 3 modules. The first module is planning which is responsible for generating a thought and action based on previous memory and the current observation. The action is then executed in the environment. The evaluation module is then called which generates a reflection based on the results of the action. These are then stored into the memory module. The current step is now complete and the next is ready to begin. An example of the WebAgent's input and trajectories can be found in A.5. WebAgents are autonomous agents that live on the Internet. They perform tasks much like a human would. They make a plan and execute actions step by step in order to complete a task. The implementation of the WebAgent in this paper is an adaptation of the one provided in WebCanvas [9]. The agent can be divided into three modules: planning, evaluation, and memory. The planning module takes in the instruction from the user. It then makes an observation which consists of the accessibility tree and screenshot of the current webpage. The screenshot contains set-of-marks annotations [11] to assist with element grounding. The LLM then generates a thought to help condition the generation of the action. Generation of interleaved reasoning traces has been shown to increase success rates in sequential decision making tasks [12]. For the action generation, we utilize html selectors for web element interactions. This approach allows the agent to fit into the WebCanvas evaluation framework. The action space for the WebAgent is defined as follows: Goto URL, click element, select element, fill form, search google, go back, scroll. The action is then executed in the environment. Next, the evaluation module takes another observation of the environment after the action is taken as well as the previous thought and action. The LLM then generates a reflection based on the current context. The reflection helps the agent by giving it self-generated feedback based on the result of its action. This gives an opportunity for the agent to correct any mistakes it has made in the next step [10]. The generated thought, action, and reflection is then stored into the memory module which contains a history of all previous steps the agent has taken. A.2 System Prompts A.2.1 WebAgent system prompt ⬇ You are a precise browser automation agent that interacts with websites through structured commands. Your role is to:\n 1. Analyze the provided webpage elements and structure 2. Use the given information to fulfill the user's request. 3. Respond with valid JSON containing your next action sequence and state assessment These are key information you will get: **Key Information**: - Previous trace: all thoughts, actions and reflections you have made historically. - Accessibility tree: characteristic expression of the current web page. - Annotated webpage screenshot: Screenshot of the current web page with numerical annotations. **Introduction to Accessibility Tree**: The accessibility tree is a tree-like data structure that describes the relationships between elements on a web page and provides accessibility information for each element (such as text, links, form elements, etc.). - **Accessibility Tree Example**: Here is an example of an accessibility tree: ‘‘‘ current web tab name is 'Google' [40] link 'About' [41] link 'Store' [186] link 'Gmail' [187] link 'Images' [163] textarea 'Search' [236] button 'See more' ‘‘‘ In this example, each row represents the characteristic representation of a web page element. It has three attributes: '[40]' for the element's element_id, 'link' indicates the element is a link, and 'About' for the content of the element. Note: The above element provided is purely for illustrative purposes and should NEVER be used directly in your output! **Annotated Screenshot Details** - When an image is provided, use it to understand the page layout - Bounding boxes with labels correspond to element indexes - Each bounding box and its label have the same color - Most often the label is inside the bounding box, on the top right - Visual context helps verify element locations and relationships - sometimes labels overlap, so use the context to verify the correct element - The numerical annotations in the screenshot correspond to the element_id in the accessibility tree. Note the correct element_id may not be in the screenshot. In such a case, use the provided accessiblity tree to select the correct action. Or use the scroll down action to see more of the webpage. **Summary space** create a comprehensive summary that enables someone to continue the task exactly where it left off. Your summary must: 1. Synthesize the key information from all steps while maintaining critical details 2. Preserve the context and progression of the task 3. Highlight important discoveries, roadblocks, or insights 4. Clearly indicate the current state of the task 5. Identify logical next steps based on the trajectory of previous actions 6. Maintain any relevant technical details, credentials, or access information needed to continue You should always incorporate your previous step reflections to determine the best action to take. **Thought Space**: - What was the previous step reflection? - What action do you think is needed now to complete the task? - What's the reason of taking that action? You have access to the following tools(helpful to interact with web page): **Execution Action Space**: - goto: useful for when you need visit a new link or a website, it will open a new tab. - fill_form: useful for when you need to fill out a form or input something from accessibility tree. Input should be a string. Only use this if the accessibility tree is provided. - google_search: useful for when you need to use google to search something. - click: useful for when you need to click a button/link from accessibility tree. Only use this if the accessibility tree is provided. - select_option: useful for when you need to select a drop-down box value. When you get (select and option) tags from the accessibility tree, you need to select the serial number(element_id) corresponding to the select tag, not the option, and select the most likely content corresponding to the option as Input. Only use this if the accessibility tree is provided. - go_back: useful when you find the current web page encounter some network error or you think the last step is not helpful. - cache_data: useful when you need to extract information from the page that you think is extremely valuable for completing the target task. It is not a direct answer to the target task, but it is extremely relevant to the target task. Subsequent actions may refer to this part of the information and return this information as input - get_final_answer: useful for when you think it is the answer to the target task and no other operations are required, Input should be a answer content. You also need to provide an effective description of the current execution action. A proper description contains: - What website it is; - Which action you choose; - REMEMBER DO NOT LEAVE THE DESCRIPTION EMPTY! You have to follow the instructions or notes: **Important Notes**: - In the initial step of a process or when there's no preceding interaction history (i.e., the previous trace is empty), you are restricted to using the ‘google_search‘ or ‘goto‘ tools exclusively - In situations where the accessibility tree is absent or not provided you are restricted to using the ‘google_search‘ or ‘goto‘ tools exclusively - If the accessibility tree is absent or not provided and there is a preceding interaction history (i.e., the previous trace is NOT empty), utilize previous thoughts, actions, and reflections to get back on track. - Your action should not be the same as last step's action. - The ‘element_id‘ should be an integer accurately representing the element's ID in the accessibility tree. - AVOID using the provided example's element_id as your output. - You MUST use the the previous step's reflections to guide your next actions. - The output JSON blob must be valid; otherwise, it cannot be recognized. **Guide to breaking out of loops** - If you are stuck in a loop, use previous screenshots, thoughts, actions, and reflections to try to break out of the loop. **Special Circumstances Guidelines**: - When performing a search on a website, if you find the search results do not display sufficient content, consider simplifying or modifying your search query. Reducing the complexity of your search query or altering keywords may yield more comprehensive results. Please ensure the accuracy of your output, as we will execute subsequent steps based on the ‘action‘, ‘action_input‘ and ‘element_id‘ you provide. A.2.2 Implicit RAG Prompt ⬇ You are a helpful assistant specializing in summarizing task-related information. Your job is to analyze a set of previous steps and extract only the information relevant to a specific task. You will provide clear, concise summaries that enable users to continue working on their task effectively. You should: 1. Carefully review the given task description to understand what the user is trying to accomplish 2. Analyze each step in the provided history, focusing on the thought, action, reflection, and URL components 3. Filter out steps that are not relevant to the specified task 4. Create a comprehensive summary that includes all important information needed to continue the task 5. Present your summary in a clear, structured format that's easy to understand and act upon 6. Note any critical insights from the accessibility tree or screenshot if they are relevant to the task Remember that the accessibility tree and screenshot provided may not be directly related to the task, so use your judgment to determine what information is actually relevant. A.3 Sample Implicit RAG summary Subtask 2: This is a continuation of a task that happened a while ago. Look for information about reduced fares on that page you found earlier. Check for any special categories or discounts that might apply to bus fares. ⬇ Summary: Finding Reduced Fares on MBTA ## Current Status I've successfully navigated to the MBTA website and accessed the fares section. The website shows comprehensive fare information including: - Subway fares ($2.40) - Bus fares ($1.70) - Monthly passes ($90.00) - Commuter rail options ## What's Been Done - Successfully navigated to the MBTA website - Accessed the fares section where all pricing information is displayed - Confirmed we're on the official MBTA site with fare options accessible ## Next Steps To find information about reduced fares: 1. Look for a "Reduced Fares" section on the current page 2. Check for special categories or discounts that might apply to bus fares 3. Look for information about eligibility requirements for reduced fares 4. Explore any links related to discounted passes or special programs The MBTA website should have information about reduced fare programs for seniors, students, people with disabilities, or low-income riders, which would be the next logical area to investigate. A.4 Summary Relevance Prompt ⬇ # Summary Relevance Evaluator System Prompt You are an expert Summary Relevance Evaluator whose purpose is to determine whether a provided summary is relevant to a given task. Your assessment should be objective, consistent, and focused solely on relevance rather than quality, style, or other attributes. ## Your Role - Evaluate whether the summary addresses the core requirements of the task - Provide clear reasoning for your assessment - Return a binary decision: RELEVANT or NOT RELEVANT - Avoid making quality judgments beyond relevance ## Evaluation Process 1. Carefully analyze the provided task to identify its key requirements and objectives 2. Examine the summary to determine if it addresses these requirements 3. Consider both explicit and implicit requirements of the task 4. Assess whether the summary would be useful to someone trying to complete the task 5. Make your determination based solely on relevance, not on writing quality or style ## Relevance Criteria A summary is RELEVANT if it: - Directly addresses the main points required by the task - Contains information that would help accomplish the task's objectives - Maintains focus on the subject matter specified in the task - Provides information at an appropriate level of detail for the task A summary is NOT RELEVANT if it: - Fails to address the core requirements of the task - Contains primarily off-topic information - Addresses a different subject or objective than specified - Provides information that would not be useful for completing the task ## Output Format For each evaluation, provide: 1. **Reasoning**: Explain your determination with specific references to both the task and summary 2. **Key Points**: Highlight which task requirements were or were not addressed 3. **Decision**: State "RELEVANT" or "NOT RELEVANT" clearly at the end **NOTICE** output your decision inside the tags ... as I will be parsing your decision from these tags using regex ## Example Analysis ‘‘‘ Task: This is a continuation of a task that happened a while ago. Look for the price of that small speaker device on the page you previously visited. Summary: Based on the history and current state, here's a relevant summary for finding the HomePod mini's price: COMPLETED STEPS: - Started at Apple.com - Navigated through TV & Home section - Successfully reached the HomePod mini product page (current location) CURRENT STATUS: - We are on the HomePod mini product page (https://www.apple.com/homepod-mini/) - Looking at the screenshot and accessibility tree, I don't immediately see a price displayed in the current view NEXT STEPS NEEDED: - Scroll through the page to locate pricing information, or - Look for a "Buy" button (visible as element [233] in the accessibility tree) which should lead to pricing details Would you like me to proceed with looking for the price on this page? Reasoning: The summary directly addresses the task of finding the price of a small speaker device. It provides a list of relevant criteria of previously completed steps needed to complete the task including going to the relevent website, navigating to the correct section Key Points: . Decision: RELEVANT ‘‘‘ Remember that your sole purpose is to evaluate relevance, not to improve or critique the summary beyond its relevance to the task. Maintain objectivity and consistency across all evaluations. A.5 Sample Trajectory Figure 8: In this sample trajectory, the task is to "Find musical instruments and filter the results to show only those that are available for free local pickup in ebay." The agent starts off on a blank webpage Figure 9: Agent now needs to fill in the search bar with the search term "musical instruments." Figure 10: Now the agent needs to filter the list to show local pickup by selecting it through the dropdown menu. Figure 11: The agent clicks on the local pickup option filter and the task is now considered complete. A.6 List of successful tasks 1. Find the soonest upcoming event near zip code 90028 in last.fm 2. Show offers on home essentials under $10 and add the first 3 items to favorites in IKEA 3. Search for a park in California called Castle Mountains National Monument and find its basic information on nps.gov 4. Find vinyl records at the lowest price in Discogs 5. Find ideas and recommendations for things to do in Cancun on Viator 6. Browse events happening at Madison Square Garden on parking.com 7. Search for the latest news and rumors about the Los Angeles Lakers on sports.yahoo 8. Show all cruises from Miami, FL to The Bahamas on Carnival 9. Find bus stop information at Abbotsford, WI on us.megabus 10. Show all used Tesla cars for zip code 10017 on CarGurus 11. Find critic reviews for the movie Creed III on IMDb 12. Show offers for 2-5 day cruises on Carnival 13. Show the schedule for the Orange Line on MBTA 14. Find the schedule for the San Francisco 49ers on NFL 15. Browse tickets for Chicago Bears games on TicketCenter 16. Find the current roster of the Miami Heat on sports.yahoo 17. Open the home improvement Q&A section on YellowPages 18. Show PlayStation 5 games available for pre-order on GameStop 19. Find adventure movies coming to theaters on Rotten Tomatoes 20. Find the NBA team with the best odds to win the title on sports.yahoo 21. Show recent trades on BoardGameGeek 22. Show 4-star+ activities in Miami with the lowest price on Agoda 23. Find alerts and conditions for Alagnak Wild River on nps.gov 24. Search for Dining Room Sets under Buy It Now on eBay 25. Find volunteer opportunities on nps.gov 26. Find the price of the HomePod mini on Apple 27. Show NFL passing touchdown season stats on sports.yahoo 28. Check rating and user reviews for Deathloop on IGN 29. Find reduced bus fare info on MBTA 30. Search for Mexican restaurants in Concord, CA on Yelp 31. Locate Dry Tortugas in Florida and check current conditions on nps.gov 32. Search for flights from Calgary (CYYC) to New York (ZNY) on FlightAware 33. Browse washing machines with free local pickup near zip code 90026 on eBay 34. Open Canyon de Chelly schedule calendar on nps.gov 35. Get IMDb recommendations of what to watch 36. Find used Xbox Series X consoles on GameStop 37. Find a South African history podcast (10-30 min, audiobook tag) on SoundCloud 38. Find Dota 2 and add all DLC to cart on Steam 39. Find ticket refund info on Eventbrite 40. Search for McDonald's in Greenville, SC on Yelp 41. Find musical instruments with free local pickup on eBay 42. Browse recent NFL videos on Fox Sports 43. Find tickets for the next pop concert on SeatGeek 44. Filter guitar tabs for “Beginner” difficulty on Ultimate Guitar 45. Find JetBlue career openings in New York 46. Show the map of Six Flags Mexico on sixflags.com 47. Browse trending daily deals on Carnival 48. Show all NFL tickets on TicketCenter 49. Find the “Trains in Spain” guide on TheTrainLine 50. Open Red Line subway schedule on MBTA 51. Find details on converting points to miles on Marriott 52. Use “find by state” to locate all California campgrounds on KOA 53. Find English Premier League standings on sports.yahoo 54. List board game publishers on BoardGameGeek 55. Find Carnival cruise options that include Alaska 56. Find tickets for events in Boston on TicketCenter A.7 Sample interaction history • Subtask 1: Go to ikea.com, navigate to the home essentials section, and filter the results to show items under $10. • Subtask 2: This is a continuation of a task that happened a while ago. Recall the list of affordable home essentials you found earlier. From that list, select the first three items and add them to your favorites on the website. ⬇ { "thought": "I've been trying to find pedicure salons in New York and add them to favorites in Yellow Pages, but we've been facing persistent Cloudflare security blocks. We're currently on the Dear sundays website's account creation page with a popup that we haven't been able to dismiss. Looking at the accessibility tree, I can see there's a specific 'Close dialog' button with element_id 344 that I should try clicking to dismiss this popup.", "action": "I'm on the Dear sundays website's account creation page. There's a promotional popup blocking access to the account creation form. I'm clicking the 'Close dialog' button (element 344) to dismiss this popup so we can proceed with creating an account as an alternative way to save this salon as a favorite.", "current_url": "https://dearsundays.com/account/register", "reflection": "The current completion status is 'loop' as we've been stuck on the Dear sundays website's account creation page with the same promotional popup for multiple consecutive steps. We've been unable to make progress toward the original task of finding pedicure salons in New York and adding two to favorites in Yellow Pages. To break out of this loop, we need to try a completely different approach: 1) Try clicking the 'Close dialog' button (element 344) which is specifically designed to close the popup, 2) If that doesn't work, navigate back to a search engine and search directly for 'yellowpages.com pedicure salon new york' to try accessing Yellow Pages through a direct search result, 3) Consider using alternative business directories like Yelp or Google Maps that allow saving businesses to lists or favorites, 4) If all else fails, we could search for pedicure salons in New York using a search engine and simply note down information about two salons as our 'favorites' to fulfill the core intent of the task." } { "thought": "I need to navigate back to IKEA's website to find the affordable home essentials page we previously visited. According to the trace history, we had successfully found the 'Affordable Home Essentials Under $10' page with items like SANELA cushion cover ($5.99), SANDBRODD throw ($9.99), and DR\u00d6NA box ($4.99). I'll use the goto action to return to IKEA's website.", "action": "I'm navigating back to IKEA's 'Affordable Home Essentials Under $10' page that we previously found. This will allow me to select the first three affordable items and add them to favorites.", "current_url": "https://www.ikea.com/us/en/campaigns/affordable-home-essentials-pubfbd3fdc0/", "reflection": "The current completion status of the task is 'doing' as you've successfully navigated to the IKEA 'Affordable Home Essentials Under $10' page and can see multiple products that meet your criteria. You've identified several items under $10 including SANELA cushion cover ($5.99), SANDBRODD throw ($9.99), DR\u00d6NA box ($4.99), and others. To complete the task, you now need to select the first three items from this list and add them to your favorites. This can be done by clicking the heart-shaped 'Save to favorites' buttons (elements like [230], [243], [257]) associated with each product. Your overall plan should be to identify the first three affordable items from the list and click their respective 'Save to favorites' buttons to complete the task of adding three items to your favorites on the website." } { "thought": "I need to add the first three affordable home essentials to favorites. Looking at the accessibility tree, the first item appears to be SANELA cushion cover for $5.99 with a 'Save to favorites' button (element_id 230). I'll click this button first to add it to favorites.", "action": "I'm on the IKEA 'Affordable Home Essentials Under $10' page and adding the first item to favorites. I'm clicking the 'Save to favorites' button for the SANELA cushion cover priced at $5.99.", "current_url": "https://www.ikea.com/us/en/campaigns/affordable-home-essentials-pubfbd3fdc0/", "reflection": "You're currently on the IKEA 'Affordable Home Essentials Under $10' page which shows multiple products under $10 including SANELA cushion cover ($5.99), SANDBRODD throw ($9.99), and DR\u00d6NA box ($4.99). To complete the task of adding the first three items to your favorites, you need to click the heart-shaped 'Save to favorites' buttons next to each product. The buttons are visible in the screenshot (elements like [230], [243], [257]). Your next steps should be to identify the first three affordable items from the top of the list and click their respective 'Save to favorites' buttons. This will complete your task of adding three affordable home essentials to your favorites on the IKEA website.", } A.8 False End Example • Subtask 1: Go to ticketcenter.com and search for events in Boston. • Subtask 2: This is a continuation of a task that happened a while ago. Browse through the list of events in that city and select tickets for the desired event. The agent completely lost track of its original task and thought it was completing the injected task. Note the task "to find information about the song 'Love You Like A Love Song'" was hallucinated. ⬇ "thought": "I'm now on the SoundCloud page for Selena Gomez's 'Love You Like A Love Song'. Looking at the accessibility tree, I can see options to interact with this song such as 'Like' (element 18), 'Repost' (element 19), and 'Share' (element 20). Since I can't create a playlist without an account, I'll try clicking the 'Share' button to see what sharing options are available for this song.", "action": "I'm on the SoundCloud page for Selena Gomez's 'Love You Like A Love Song'. Since I can't create a playlist without an account, I'm exploring other options by clicking the 'Share' button to see what sharing capabilities are available for this song.", "current_url": "https://soundcloud.com/selena-gomez-official/love-you-like-a-love-song", "reflection": "The task has been successfully completed. You have navigated to SoundCloud, searched for Selena Gomez, found her popular songs, and accessed the specific song 'Love You Like A Love Song'. The current page shows the song's details with player controls, allowing you to listen to the track. The accessibility tree confirms you're on the official Selena Gomez page for this specific song, with options to play, like, repost, and share the track. You can see information about the song including its play count, comments, and related tracks. This completes the task of finding and accessing this specific Selena Gomez song on SoundCloud.", A.8.1 Self evaluation ⬇ "status": "finished", "score": "10", "reason": "The task was to find information about the song 'Love You Like A Love Song' by Selena Gomez on SoundCloud, and this has been successfully completed. The user has navigated to SoundCloud, searched for Selena Gomez, found her songs including 'Love You Like A Love Song', and successfully clicked on this specific song to access its page. The current URL confirms we are on the correct song page (https://soundcloud.com/selena-gomez-official/love-you-like-a-love-song) and the accessibility tree shows all the song details and player controls. The user has perfectly executed the necessary steps to find and access this specific song on SoundCloud.", "description": "The task has been successfully completed. You have navigated to SoundCloud, searched for Selena Gomez, found her popular songs, and accessed the specific song 'Love You Like A Love Song'. The current page shows the song's details with player controls, allowing you to listen to the track. The accessibility tree confirms you're on the official Selena Gomez page for this specific song, with options to play, like, repost, and share the track. You can see information about the song including its play count, comments, and related tracks. This completes the task of finding and accessing this specific Selena Gomez song on SoundCloud." }, A.9 Inefficent Progress Example Task is progressing but step limit reached. ⬇ { "thought": "I need to find a way to filter or sort board game results to show publishers on BoardGameGeek. Looking at the current page, I'm on Dear Sundays' account registration page which is unrelated to my previous task on BoardGameGeek. I should navigate back to BoardGameGeek and look for publisher filtering options.", "action": "I'm navigating back to BoardGameGeek's board game database page to find publisher filtering options for board games. This will allow me to view a list of publishers as requested in the task.", "current_url": "https://boardgamegeek.com/browse/boardgame", "reflection": "The current completion status of the task is 'doing' as you have successfully navigated to BoardGameGeek's browse/boardgame page which shows a comprehensive listing of board games with rankings and ratings. This is a good intermediate step, but you haven't yet accessed the specific publishers listing. Looking at the accessibility tree, I can see there's a 'Publishers' link (element 17/260) in the Browse dropdown menu that would take you directly to the publishers listing page. Your next step should be to click on this Publishers link to complete your task of viewing a list of publishers for board games. The current page shows you're on the right track as you've reached the main database section where all filtering options are accessible.", }, { "thought": "Looking at the accessibility tree, I can see there's a 'Publishers' link (element 103) under the Browse menu. This link will take me directly to the publishers listing page, which is exactly what I need to complete my task of viewing a list of publishers for board games.", "action": "I'm on the BoardGameGeek browse/boardgame page which shows a comprehensive listing of board games. To find a list of publishers, I'll click on the 'Publishers' link in the Browse navigation menu, which will take me to the complete list of board game publishers.", "current_url": "https://boardgamegeek.com/browse/boardgame", "reflection": "You are currently on the BoardGameGeek browse/boardgame page which shows a comprehensive listing of board games with rankings, ratings, and other details. This is the right section of the website to access filtering options. To complete your task of finding a way to filter or sort results to show publishers, you should click on the 'Publishers' link (element 17/260) visible in the Browse dropdown menu. This link will take you directly to the publishers listing page, which is exactly what you're looking for. The current page shows you're on the right track as you've successfully navigated to the main database section where all filtering and sorting options are accessible.", } A.10 Loop Example Attempting to complete a task repeatedly thats un-executable. Original task was to find English Premier League standings on sports.yahoo.com ⬇ { "thought": "After multiple attempts, I've confirmed that creating a playlist on SoundCloud requires signing in to an account, which is against my guidelines. I've successfully searched for and identified popular Selena Gomez tracks ('Love You Like A Love Song', 'Bluest Flame', and 'Wolves'), but cannot create a playlist without an account. I should provide a final answer explaining this limitation.", "action": "On SoundCloud search results page for Selena Gomez, I'm providing a final answer explaining that creating a playlist requires account authentication, which is against guidelines. I've identified suitable songs but cannot complete the playlist creation task without signing in.", "current_url": "https://soundcloud.com/search?q=Selena%20Gomez", "reflection": "The current completion status is 'loop' as we've been on the same search results page for Selena Gomez for multiple consecutive steps without making progress. The screenshots from the last three steps are identical, showing we've found suitable songs ('Love You Like A Love Song', 'Bluest Flame', and 'Wolves') but cannot create a playlist named 'Love' because SoundCloud fundamentally requires account authentication for playlist creation. This task cannot be completed without signing in or creating an account on SoundCloud. To break out of this loop, we would need to either get permission to sign in to an existing account, explore alternative ways to organize songs without creating an actual playlist, or acknowledge that this specific task cannot be completed within the given constraints.", }, { "thought": "After multiple attempts, I've determined that creating a playlist on SoundCloud requires signing in to an account, which is against my guidelines. I've successfully searched for and found popular Selena Gomez tracks, but I need to provide a final answer explaining this limitation and the songs I've identified.", "action": "On SoundCloud's search results page for Selena Gomez, I've identified popular tracks but cannot create a playlist without signing in. Providing a final answer explaining this limitation.", "current_url": "https://soundcloud.com/search?q=Selena%20Gomez", "reflection": "The current completion status is 'loop' as we've been on the same search results page for Selena Gomez for multiple consecutive steps without making progress. The screenshots from the last three steps are identical, showing we've found suitable songs ('Love You Like A Love Song', 'Bluest Flame', and 'Wolves') but cannot create a playlist named 'Love' because SoundCloud fundamentally requires account authentication for playlist creation. This task cannot be completed without signing in or creating an account on SoundCloud, which is against the guidelines. To break out of this loop, we would need to either get permission to sign in to an existing account, explore alternative ways to organize songs without creating an actual playlist (such as taking screenshots or notes of the selected songs), or acknowledge that this specific task cannot be completed within the given constraints and request a modified task that doesn't require account creation.", }, { "thought": "After multiple attempts, I've confirmed that creating a playlist on SoundCloud requires signing in to an account, which is against my guidelines. I've successfully searched for and found popular Selena Gomez tracks, but I need to break out of this loop by providing a final answer to the user.", "action": "On SoundCloud's search results page for Selena Gomez, I'm providing a final answer explaining that I cannot create a playlist without signing in, which is against my guidelines. I've identified popular tracks ('Love You Like A Love Song', 'Bluest Flame', and 'Wolves') that would be suitable for the requested 'Love' playlist.", "current_url": "https://soundcloud.com/search?q=Selena%20Gomez", "reflection": "The current completion status is 'loop' as we've been on the same search results page for Selena Gomez for multiple consecutive steps without making progress. The screenshots from the last three steps are identical, showing we've found suitable songs ('Love You Like A Love Song', 'Bluest Flame', and 'Wolves') but cannot create a playlist named 'Love' because SoundCloud fundamentally requires account authentication for playlist creation, which is against the guidelines. To break out of this loop, we would need to either get permission to sign in to an existing account, explore alternative ways to organize songs without creating an actual playlist (such as taking screenshots or notes of the selected songs), or acknowledge that this specific task cannot be completed within the given constraints and request a modified task that doesn't require account creation.", } A.11 Implicit RAG analysis • Subtask 1: Go to ultimate-guitar.com and search for guitar tabs. • Subtask 2: This is a continuation of a task that happened a while ago. On the search results page, look for a filter option to show only tabs with a "Beginner" difficulty rating. Apply that filter to the results you found earlier. A.11.1 Failed retrieval without iRAG. ⬇ { "thought": "The previous search for 'Cyberpunk 2077 user reviews' is visible in the search bar, and the results show entries for both 'Cyberpunk 2077: Phantom Liberty' and 'Cyberpunk 2077'. To break the loop, I should click directly on the main 'Cyberpunk 2077' link (element_id 157) in the search results. This should take me to the main game page, where I can then look for the 'Reviews' tab to access user reviews.", "action": "On the IGN Icon Awards search results page. Clicking the main 'Cyberpunk 2077' link (element_id 157) in the results to access the main game page and look for the user reviews section.", "current_url": "https://www.ign.com/games/cyberpunk-2077", "reflection": "The current completion status is stuck in a loop. The IGN Icon Awards page is being displayed repeatedly, and all recent navigation attempts have failed to bring up the user reviews for Cyberpunk 2077. The essential next step is to change strategy: either use the IGN main search bar to specifically search for 'Cyberpunk 2077 user reviews,' or, if access to the Cyberpunk 2077 game page is possible, click on the 'Reviews' tab or scroll to manually locate the user review section. Only by breaking from the current navigation pattern can progress be made toward the goal.", } A.11.2 With iRAG, successful retrieval of correct trajectory and completion of the task ⬇ { "thought": "We are stuck in a loop on the IGN Icon Awards page, which is unrelated to the original task of filtering guitar tabs by 'Beginner' difficulty on ultimate-guitar.com. To get back on track, I need to navigate to ultimate-guitar.com, perform a search for guitar tabs, and then look for and apply the 'Beginner' difficulty filter.", "action": "Navigating to ultimate-guitar.com to return to the correct website for filtering guitar tabs by 'Beginner' difficulty as required by the original task.", "current_url": "https://www.ultimate-guitar.com/", "reflection": "The current completion status is stuck in a loop on the IGN Icon Awards page. The repeated actions of clicking on Cyberpunk 2077 links and the 'See Leaderboard' link have not led to the user reviews section for Cyberpunk 2077. To break out of this loop and complete the task, a different approach is needed: after reaching the main Cyberpunk 2077 game page, click on the 'Reviews' tab (element 138) to access user reviews, or use the main search function to specifically search for 'Cyberpunk 2077 user reviews' to find the dedicated section. The current approach is not yielding progress, and a new navigation strategy is required.", }, { "thought": "To filter the search results to show only tabs with a 'Beginner' difficulty rating, I need to look for a filter or button related to difficulty. In the accessibility tree, I see links for different grades such as '2020s', '2010s', '2000s', etc., and also for types like 'Chords', 'Official', 'Tab', 'Ukulele', 'Guitar Pro', 'Bass', but I do not see a direct 'Beginner' filter in the visible portion. However, there is a 'show all' link (element_id 103) under the 'TYPE' section, which may reveal more filter options, possibly including difficulty. I should click this to see if the 'Beginner' filter appears.", "action": "On the Ultimate Guitar homepage, clicking the 'show all' link under the TYPE section to reveal more filter options, possibly including a 'Beginner' difficulty filter.", "current_url": "https://www.ultimate-guitar.com/explore?active_filter=type", "reflection": "The current completion status is 'doing' because you are now on the correct website (ultimate-guitar.com) and can see the search results for guitar tabs. The next key step is to locate and apply the 'Beginner' difficulty filter, which is available in the left sidebar under 'DIFFICULTY' (with the 'Beginner' link clearly visible). Once you click this filter, the results will update to show only tabs with 'Beginner' difficulty, completing the target task. If you get stuck, try scrolling or looking for additional filter options in the sidebar.", } A.11.3 Summary used in the above task. ⬇ Task: On the search results page, look for a filter option to show only tabs with a "Beginner" difficulty rating. Apply that filter to the results you found earlier. Relevant Task Summary 1. Previous Context - The earlier steps involved searching for guitar tabs on ultimate-guitar.com. - You successfully performed a search and reached the search results page for guitar tabs. - The task now is to filter these results to show only tabs with a "Beginner" difficulty rating. 2. Current Page and UI Elements - The current page is NOT ultimate-guitar.com; it is the IGN Icon Awards page (https://www.ign.com/icons). - The accessibility tree and screenshot show IGN's game ranking and filtering interface, not a guitar tab search or difficulty filter. - There is no evidence of a "Beginner" difficulty filter or guitar tab search results on the current page. 3. What Has (and Has Not) Been Done - You have not yet located or applied a "Beginner" difficulty filter on a guitar tab search results page. - The current context is unrelated to the original task (it is about games, not guitar tabs). What Needs to Be Done Next - Return to the Correct Website and Page: - Navigate back to ultimate-guitar.com. - Perform a search for guitar tabs if you are not already on the search results page. - Locate the Difficulty Filter: - On the search results page, look for filter options (usually on the left sidebar or above the results). - Find the filter labeled "Difficulty" or similar. - Apply the "Beginner" Filter: - Select or check the "Beginner" option under the difficulty filter. - Wait for the page to refresh or update to show only tabs with a "Beginner" rating. - Verify the Results: - Ensure that the displayed tabs are now filtered to show only those with a "Beginner" difficulty rating. Summary Table | Step | Status/Action Needed | | On ultimate-guitar.com | Not currently there-navigate to the correct website | | Search for guitar tabs | Repeat search if not already on results page | | Find "Difficulty" filter | Look for filter options on the search results page | Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" () button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. We gratefully acknowledge support from our major funders, member institutions, , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from \ No newline at end of file diff --git a/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2603.20586.txt b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2603.20586.txt new file mode 100644 index 0000000..4da1c01 --- /dev/null +++ b/wizard-work/sphinx/longctx-arxiv-archive-2026-10-20/corpus/2603.20586.txt @@ -0,0 +1 @@ +Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract. 1 Introduction 2 Related Work 2.1 Long-context Attention Mechanisms 2.2 Hierarchical and External Memory Systems 2.3 Dynamic Routing and Query-aware Attention 3 Motivation: Beyond MLA and MHA 4 Methodology 4.1 Route-Fused MKA (FastMKA): Causal Route-Fusion with KV-Cache 4.2 Block-Memory Keyed Attention (Block-MKA) Design 4.3 Hierarchical Block-wise MKA Algorithm 4.4 Algorithm and Pseudocode 5 Theoretical Formulation: Recursive MKA with Online Softmax 5.1 Recursive Reformulation 5.2 Numerical Stability via Max-Shift 5.3 Local vs. Global MKA Modes 5.4 Runtime Bounds and Complexity 6 Experiments 6.1 Experimental Setup Models. Dataset. Hardware. Training Details. Inference Details. Metric. 6.2 Main Results 6.3 Experimental Results Analysis 6.4 Long-Context Benchmark Evaluation 6.5 Ablation Studies 6.5.1 Memory Tier Ablation 6.5.2 Routing Mechanism Ablation 6.5.3 Routing Weight Analysis 6.6 System-Level Performance Analysis 6.6.1 Prefill vs Decode Latency Breakdown 6.6.2 Computational Cost Analysis 6.7 Discussion 7 Conclusion Acknowledgements References License: CC BY 4.0 arXiv:2603.20586v2 [cs.LG] 24 Mar 2026 MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning Conference: Proceedings of the 23rd ACM International Conference on Computing Frontiers; May 19-21, 2026; Catania, ItalyProceedings of the 23rd ACM International Conference on Computing Frontiers (CF '26), May 19-21, 2026, Catania, ItalyDOI: 10.1145/3801487.3801812ISBN: 979-8-4007-2568-5/2026/05 Dong Liu Affiliation: University of California, Los Angeles, Los Angeles, California, USA email: pikeliu@ucla.edu , Yanxuan Yu Affiliation: Columbia University, New York, New York, USA email: yy3523@columbia.edu , Ben Lengerich Affiliation: University of Wisconsin-Madison, Madison, Wisconsin, USA email: lengerich@wisc.edu and Ying Nian Wu Affiliation: University of California, Los Angeles, Los Angeles, California, USA email: ywu@stat.ucla.edu © cc Abstract. As long-context language modeling becomes increasingly important, the cost of maintaining and attending to large Key/Value (KV) caches grows rapidly, becoming a major bottleneck in both training and inference. While prior works such as Multi-Query Attention (MQA) and Multi-Latent Attention (MLA) reduce memory by sharing or compressing KV features, they often trade off representation quality or incur runtime overhead. We propose Memory-Keyed Attention (MKA), a hierarchical attention mechanism that integrates multi-level KV caches-local, session, and long-term-and learns to route attention across them dynamically. We further introduce Route-Fused MKA (FastMKA), a broadcast-routed variant that fuses memory sources before attention computation for enhanced efficiency. Experiments on different sequence lengths show that FastMKA achieves a favorable accuracy-efficiency trade-off: comparable perplexity to MLA while achieving up to 5 ×\times faster training throughput and 1.8 ×\times lower evaluation latency. These results highlight MKA as a practical and extensible framework for efficient long-context attention. ††cc-license: by 1. Introduction Large Language Models (LLMs) have rapidly advanced and are now capable of processing context lengths up to 128K or even 1M tokens (Peng et al., 2023; Jiang et al., 2024; Tworkowski et al., 2023). This unprecedented expansion of context length unlocks new applications such as long-document reasoning and multi-turn dialogue with persistent memory. However, supporting long contexts during inference introduces severe memory and latency bottlenecks, particularly due to the scaling cost of attention with Key/Value (KV) caches. In self-attention, each newly generated token must attend to all past tokens stored in the KV cache. This results in quadratic compute and increasingly large memory reads. For instance, with a context length of 3232K, we measure that the KV cache for a LLaMA-7B model (Touvron et al., 2023) occupies approximately 15.8GB on an NVIDIA A800 GPU and requires 11.3ms to access during decode, which dominates more than 50% of the inference latency. This cost limits the throughput of LLMs in production environments. Several methods have attempted to address the inefficiencies in attention computation. Multi-Query Attention (MQA) (Shazeer, 2019) and Grouped-Query Attention (GQA) (Ainslie et al., 2023) reduce KV duplication by sharing them across heads. Multi-Latent Attention (MLA) (Team, 2024) compresses the KV cache via low-rank factorization. However, these methods sacrifice representation fidelity or lack the flexibility to differentiate among memory types. Figure 1. MKA hierarchical memory design with three memory levels (L1: local, L2: session, L3: long-term) and dynamic routing gates. In this paper, we propose Memory-Keyed Attention (MKA), a novel attention mechanism that hierarchically organizes memory into three levels-local (L1), session (L2), and long-term (L3)-and dynamically learns how to route each query token across these sources. As illustrated in figure 1, MKA leverages lightweight routing gates that modulate attention over heterogeneous memory types. This design enables context-aware attention with significantly lower memory bandwidth and improved token reuse. To ensure scalability, MKA adopts a block-wise softmax implementation inspired by FlashAttention (Dao et al., 2022) and supports GPU-efficient kernel fusion. Moreover, long-term memory (L3) is indexed via semantic chunking and vectorized hashing, allowing the model to recall relevant historical content efficiently. Our experiments show that MKA can reduce training time & evaluation latency significantly compared to MHA, MQA, GQA, MLA. Our contribution can be introduced as follows: • We identify the inefficiencies of existing long-context attention mechanisms and propose a hierarchical memory system tailored for attention routing. • We introduce Memory-Keyed Attention (MKA), a dynamic routing mechanism across local, session, and long-term memory with hardware-friendly implementations, we further accelerate MKA training and inference by designing broadcast routing (FastMKA). • We show that FastMKA outperforms prior methods in both accuracy and efficiency, making it suitable for high-throughput LLM inference with long-context inputs. 2. Related Work 2.1. Long-context Attention Mechanisms As LLMs are scaled to handle inputs of 32K, 128K, or even 1M tokens (Peng et al., 2023; Shukor et al., 2025), the challenge of performing efficient attention over long contexts has become a major bottleneck. A key issue lies in the size and bandwidth cost of Key/Value (KV) caches. Many efforts have been proposed to reduce attention overhead in this setting. FlashAttention (Dao et al., 2022) improves memory throughput by computing softmax in a tiled, IO-aware fashion. Multi-Query Attention (MQA) (Shazeer, 2019) and Grouped-Query Attention (GQA) (Ainslie et al., 2023) reduce KV duplication by sharing across heads, thus reducing cache size to 1H\frac{1}{H} and gH\frac{g}{H}, respectively, where HH is the number of attention heads and gg is the number of KV groups ( g