Report GitHub Issue × Title: Content selection saved. Describe the issue below: Description: Submit without GitHub Submit in GitHub arXiv is now an independent nonprofit! Learn more × Back to arXiv Why HTML? Report Issue Back to Abstract Download PDF Abstract 1 Introduction 2 Oolong-synth 2.1 Data Data filtering 2.2 Context window construction Questions Instructions 2.3 Evaluation Baseline Parsing answers Scoring An iterative system? 3 Oolong-real 4 Results and Analysis 4.1 Does reasoning effort correlate to performance? Qwen and Olmo: think versus instruct models. GPT-5-nano: reasoning effort parameter. Gemini 3 Pro: reasoning token count. 4.2 Over-reasoning Deepseek R1 Gemini models 4.3 Refusal 4.4 Aggregation without classification Providing labels Providing domain knowledge 4.5 Iterative retrieval baseline for Oolong-real 5 Related Work 6 Conclusion 7 Acknowledgements References A Oolong - Leaderboard B Additional Results B.1 Reasoning effort on Oolong-real B.2 Data contamination for Oolong-real C Additional Related Work Dungeons & Dragons data Aggregation as an NLP concept D ICL label validation E Question types E.1 Oolong-synth E.2 Oolong-real F Deepseek R1 trace examples G Question type analysis G.1 Oolong-synth G.2 Oolong-real H Oolong-real data H.1 Data quality and curation H.2 Challenges of Oolong-real License: arXiv.org perpetual non-exclusive license arXiv:2511.02817v2 [cs.CL] 05 Oct 2026 Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities Amanda Bertsch Adithya Pratapa ††thanks: Work conducted while the author was a student at Carnegie Mellon University. Teruko Mitamura Graham Neubig Matthew R. Gormley Carnegie Mellon University Email: {abertsch,vpratapa,teruko,gneubig,mgormley}@cs.cmu.edu Affiliation: abertsch72/oolong oolongbench/oolong-[synth, real] Abstract Many important applications of long-context models require reasoning over large collections of items-summarizing trends across thousands of customer interactions, identifying patterns in research corpora, or analyzing behavioral data at scale. Evaluating whether models can analyze items atomically and aggregate findings requires information-dense tasks that make use of the full context, like writing a summary or describing trends across a collection. Yet these types of tasks often result in long-form outputs which are challenging to evaluate; as a result, current long-context benchmarks predominantly test retrieval-style tasks, where the correct answer depends on a small fraction of tokens and the rest is noise. We introduce Oolong, a benchmark of distributional reasoning tasks that require analyzing individual chunks of text atomically and aggregating results. Oolong includes synthetic tasks (Oolong-synth) that enables controlled ablation of reasoning components, and a downstream setting (Oolong-real) that requires reasoning over realistic conversational data. Tasks involve classification, counting, filtering by metadata, and temporal reasoning. Frontier models struggle at moderate context lengths at this task, suggesting Oolong captures capabilities not yet addressed by scale alone. We release the data and evaluation harness for Oolong to enable further development of models that can reason over large quantities of text. 1 Introduction In the last several years, the exponentially increasing context lengths of LLMs have enabled many new applications, including reasoning models (Guo et al., 2025), many-shot prompting (Bertsch et al., 2024), and repository-level code generation (Jimenez et al., 2023). However, despite the rapid improvement in this area, long-context aggregation remains a challenging capability to study. Tasks like summarization are realistic but produce long and subjective outputs that are challenging to evaluate. Other tasks require only a subset of the input, which could be extracted from the long context before solving (e.g. using a RAG system). Multi-step reasoning datasets make progress towards measuring this capability, but often lack realism - inputs may be artificial (e.g. random repeated words in RULER (Hsieh et al., 2024)), the posed questions may be unrealistic (e.g. “identify the second poem about this topic” in MRCR (OpenAI, 2025)), and the relevant context may be embedded in distractor text from a different domain or a well-known text (e.g. small state updates embedded in Project Gutenberg novels in BABILong (Kuratov et al., 2024)). We argue that aggregation is best measured in realistic settings with multiple types of aggregative component tasks- including the ability to draw inferences across different sections of input and perform basic numeracy operations with this information. We further discuss these desiderata in Table 1. We propose Oolong, a benchmark that requires multi-hop reasoning over long inputs to produce easily verifiable outputs. Oolong is separated into two task sets, presented in Figure 1. Oolong-synth (§2) is a set of naturalistic synthetic tasks constructed from existing in-context learning datasets. These tasks require implicitly labeling the examples in-context to reason over distributional properties of the labels (counting tasks), over user-specific patterns (user tasks), and over changes in label distributions over time (temporal tasks). Oolong-real (§3) poses these types of questions over real data that is not so easily separable into component parts- asking challenging questions about character actions from live-play Dungeons & Dragons transcripts, using human-annotated gold answers. Figure 1: Oolong poses questions that require performing a multi-step information aggregation process to determine the solution. Oolong-synth uses ICL-based tasks, which could be easily decomposed and solved iteratively, as a proxy for real-world aggregation tasks over long inputs. Oolong-real poses challenging information aggregation questions over transcripts from live-action Dungeons & Dragons shows, which can not be easily decomposed into component pieces. Each task requires identifying the relevant segments of the input; this ranges from only a handful of instances or lines of dialogue to questions that require the use of every line of the input. The relevant segments must be classified or categorized, and these individual decisions must be aggregated to produce a final answer. By framing the benchmark around problems that are simple individually, Oolong measures capability on long context reasoning and not accuracy on the underlying task. By requiring identification of relevant context, classification decisions over that context, and numeracy skills to produce statistics about the input, Oolong requires models to perform multi-step reasoning over long inputs. In Table 1, we compare Oolong with most similar long-context reasoning benchmarks. Although these component capabilities have been well studied and frontier models demonstrate strong performance in each (Yen et al. (2024), Agarwal et al. (2024), inter alia), we find that models struggle with information aggregation as the input length grows. Most models score below 50% on Oolong at 128K context. We study why this task is so challenging and find that identification and aggregation of information is the bottleneck, not labeling (§4). We study the role of reasoning behavior and identify consistent error cases in reasoning models that limit their performance in this type of task, including over-reasoning by Gemini and DeepSeek models and task refusal by GPT models (§4). We believe Oolong is a usefully challenging evaluation of long-context reasoning abilities. Real vs synthetic data Aggregation capabilities Measurability Dataset Realistic inputs Realistic questions Relevant text matches distractors Multi-step reasoning Numeracy Must use full input Easy to eval MRCR ✓ ✗ ✓ ✓ ✓ ✗ ✓ BABILong ✗ ✓ ✗ ✓ ✗ ✗ ✓ RULER ✗ ✗ ✗ ✝ ✝ ✝ ✓ HELMET Summ ✓ ✓ ✓ ✓ ✗ ✓ ✗ HELMET LongQA ✓ ✓ ✓ ✓ ✗ ✝ ✓ Oolong-synth ✗ ✓ ✓ ✓ ✓ ✓ ✓ Oolong-real ✓ ✓ ✓ ✓ ✓ ✝ ✓ Legend: ✓ = yes, ✗ = no, ✝ = in some cases Table 1: Comparing long-context reasoning benchmarks. Oolong measures aggregation capabilities in controllable, realistic, and simple-to-evaluate settings. 2 Oolong-synth We aim to construct a synthetic information aggregation task that allows fine-grained control over the types of information and number of steps necessary to solve the task. To do this, we need building blocks: simple, short-context documents to build into a longer collection. We construct Oolong-synth by constructing challenging corpus-level questions over existing in-context learning (ICL) datasets. This has conceptual similarities with database reasoning (Thorne et al., 2021). 2.1 Data We collect 10 text classification datasets with between 2 and 10 labels. We select for tasks that are possible for the authors to perform without difficulty (and validate that these tasks are similarly simple for models during our dataset filtering). Table 4 describes the datasets in detail. We split the data into two validation tasks and eight test tasks. The validation tasks are selected to avoid underlying task overlap with test tasks (in the style of Min et al. (2022)), and so that there is both a 2-label and a multi-label (6-label) validation task. For the sentiment and formality datasets with more than two labels, we reduce the label space to a single positive and a single negative label, to reduce the difficulty of the task. Data filtering In a normal in-context learning task, if a few examples are unusually hard or even mislabeled, this only slightly impacts on the overall score. Because we require aggregation across many ICL examples at once, one particularly difficult example can affect many answers downstream. However, many ICL datasets are known to contain mislabelings (Ying and Thomas, 2022; Chong et al., 2022; Klie et al., 2023). We perform an additional screening step to remove mislabeled or unusually hard examples: we select two models that we do not expect to be substantially stronger than the models we are evaluating (GPT-4.1 nano and Llama 4 Maverick) and perform zero-shot ICL with a minimal instruction that provides the label space and type of task. We then exclude all examples that both models get incorrect, for exclusion rates ranging from 0% to 0.635% by dataset. We report the percentage of examples screened out by dataset and provide examples of excluded and validated datapoints for each dataset in Appendix D. Many of the excluded instances are clearly mislabeled; the remainder generally represent challenging cases. 2.2 Context window construction We determine the number of examples in the context window by using an estimate of the number of tokens required for each example, for the general task instructions, and for the specific question. Following Yen et al. (2024), we compute these averages using the Llama 2 tokenizer, and use approximately 500K tokens of context for each estimate. We then estimate the number of examples to use by determining how many average-length examples would reach 95% of the target context length. We sample a distribution over the label classes so that the model cannot use any information about the true distribution over labels. Then, we sample examples to fill this distribution, and sample user IDs and dates.Once a context window is constructed, we use it for 25 questions to allow for prompt caching, reducing the time to evaluate local models and the cost to evaluate on most APIs. At each context length, we sample two context windows per dataset, for a total of 50 questions per dataset or 400 test split questions per context length. We construct Oolong-synth questions for every power of 2 from 1K to 4M. Questions We construct three types of Oolong-synth questions, in order of increasing complexity. Counting questions concern simple statistical properties of the label distribution, ranging from identifying the most frequent label to determining the number of examples with each label class. If the label distribution was provided, all tasks in this set should be trivial; these tasks can be seen implicitly as the task of labeling every example and reporting summary statistics. User information questions require additional cross-reference with the user ID field. Timeline questions ask about changes in distribution before or after a certain date, between years, or between months across years. This is more challenging than user information questions because it requires reasoning about before or after a date, rather than matching a list of IDs (see §E.1 in Appendix for questions). Instructions For each dataset, we provide a brief (one-sentence) description of the task and label space, as well as the number of examples in-context. We provide instructions at the start and end of the input, as suggested by OpenAI's long context prompting guide.11 1 https://cookbook.openai.com/examples/gpt4-1_prompting_guide#prompt-organization However, we provide the question only at the end, to enable prompt caching. 2.3 Evaluation Baseline A random baseline is non-trivial for these tasks, because the output space for each question varies. We construct an algorithm for a random baseline with the following rules: (1) in cases where there is an nn-way choice (e.g. choosing between nn labels or choosing a month where some criteria occurs), we select an answer from the set of valid answers at random; (2) if the question requires a numerical answer, we return N/|L|N/|L|, where NN is the number of data points in context and LL is the labelset; (3) if the question requires a date or user ID, we sample at random from the list of dates/IDs in context. We compute the random baseline performance as the expected value of this procedure. Parsing answers The task prompts specify an output format. We make a best-effort attempt to parse answers using this format; if this template is not present in the output, we take the candidate answer to be the last substring in the output that appears to match the desired answer type. Scoring For questions that require a label, date, user ID, or comparison (e.g. “greater than” or “less than”), we score on exact match. For questions that require a numerical answer, we set the score to be score​(y^)=0.75|y−y^|\texttt{score}(\hat{y})=0.75^{|y-\hat{y}|} to allow for partial credit. An iterative system? Although Oolong-synth allows for a high degree of control over the types of input, these inputs are composed of independent examples from the source dataset. In theory, an iterative system could complete these tasks by calling a model to label each individual example and then deterministically aggregating the resulting labels. However, such an iterative system could theoretically handle many existing evaluation tasks; for instance, Needle-in-a-Haystack tasks could be solved almost trivially by asking a model if each of the NN input sentences individually contains the needle. We believe that the Oolong-synth setting reflects the way many real-world users interact with current large language models. Real user queries tend to be underspecified and rarely use prompt engineering strategies (Sarkar et al., 2025; Xue et al., 2025). When users do edit prompts, they are more likely to add additional context or more detailed instructions, rather than removing or subdividing tasks for the language model (Desmond and Brachman, 2024). This use case is further exacerbated by recent systems supporting file uploads of arbitrary sizes for document analysis (Google, 2025; Anthropic, 2025). Asking questions on a large block of context represents a realistic use scenario, highlighting the utility of Oolong-synth as a model capability evaluation bench. To further motivate the information aggregation problem, we next consider a setting in which inputs cannot be so trivially decomposed. 3 Oolong-real We complement Oolong-synth with questions derived from real conversational data. Oolong-real is compiled from the transcripts of a Dungeons and Dragons (D&D) role-playing game, where a group of players collaboratively build a story through in-character actions and success depends on rolling dice. Stories unfold over narratives (“campaigns”) that span dozens to hundreds of episodes, with each episode generally lasting 4-5 hours of play. These transcripts involve several levels of conversation, from out-of-character chitchat to rules discussion to in-character actions and speech. Though lightly edited for readability, they reflect naturalistic speech instead of carefully planned written text. Because the conversation is unscripted and can involve tangents or side channels, conversational turns require variable amounts of prior context to resolve. In some instances, the same event (e.g. casting a spell) is discussed for many turns, or brought up again after a long interlude; in other cases, a prior event is “retconned” or revised post hoc. In Dungeons and Dragons, there are limitations to how frequently characters can take certain types of actions, and fans often discuss metadata of the campaigns (e.g., whether a certain person is particularly unlucky with their dice rolls or if a character uses a signature spell more or less as the campaign progresses). Because these shows are extremely long, wide-ranging, and improvised live, the creators do not provide this type of information. Instead, this information is often annotated meticulously, with multiple levels of verification, by dedicated fans of the work. We take this as an example of an information aggregation task in the wild that is not simple to reframe as an iterative task. Figure 2: Scores by context window length for Oolong-synth and Oolong-real. Here we show only the best model for each model family; see Appendix B for full results. For Oolong-real, we consider two campaigns from the series Critical Role and the data compiled by CritRoleStats,22 2 https://www.critrolestats.com/ a fan project that labeled by hand per-episode statistics for the first several campaigns of Critical Role. We used episode transcripts from the Critical Role Dungeons and Dragons Dataset (CRD3) (Rameshkumar and Bailey, 2020). We use CritRoleStats game statistics about dice rolls and spells cast in each episode and design a set of questions that cover a variety of information aggregation tasks. We include questions that require processing of single- or multiple-episode transcripts. For multi-episode questions, we concatenate transcripts and use delimiters to highlight the start and end of each transcript. See Table 19, Table 20, and Table 21 of the Appendix for a full list of Oolong-real questions. We include varying context windows to evaluate the model's ability to use long-context reasoning and aggregation capabilities. Using a single episode transcript as a context unit, we include context windows ranging from one to 24 episode transcripts (55K to 1.3M tokens). We follow the evaluation setup from Oolong-synth. 4 Results and Analysis On Oolong, we benchmark a strong suite of frontier models to study their information aggregation capabilities over long context. We include a mix of models of varying sizes, levels of reasoning, and long-context capabilities. In Figure 2, we visualize the model performance for context windows up to 512K tokens. As expected, we see a significant drop in performance at higher context windows. The two splits are of similar difficulty at the same context length, although direct comparison is challenging because real data do not necessarily align with fixed context window buckets.33 3 While performance on Oolong-real starts lower, this is because the shortest inputs in this dataset are a single episode, with an average length of 55K tokens; models perform similarly on Oolong-synth around the same context length. A breakdown of results with an aggregate rating over the two datasets is available in the Appendix (Table 3), as well as an analysis by question and answer type (Appendix G). 4.1 Does reasoning effort correlate to performance? Oolong is a reasoning-intensive task, so we hypothesize that increased reasoning may correlate with better scores. We measure reasoning effort in three different ways. (a) GPT-5-nano reasoning effort (b) Gemini-3-Pro reasoning token count Figure 3: Does reasoning effort correlate with task performance? Qwen and Olmo: think versus instruct models. In these families, we have pairs of models that are focused on reasoning and conventional instruction following. In the Olmo family, the reasoning (“think”) models greatly exceed the performance of their equivalent instruction-tuned model on Oolong-synth. However, for Qwen-3-4B, the reasoning model shows near-zero performance on Oolong-synth because of the same over-reasoning phenomenon described for Deepseek-R1: although it is allotted up to 256K output tokens, the model almost always uses the full output for reasoning without returning an answer. GPT-5-nano: reasoning effort parameter. The OpenAI API allows for specification of a reasoning level: “low,” “medium,” or “high.” We compare “high” and “low” reasoning effort for GPT-5-nano. 3(a) shows the results for Oolong-synth. Although Oolong is a reasoning-intensive task, specifying a higher reasoning effort is only useful for short contexts; after 64k, there is little discernible difference between reasoning levels, with the “high” reasoning setting even slightly underperforming “low” reasoning at 256k on Oolong-synth. We hypothesize that at context lengths where there is sufficient remaining room in the context window to enumerate labels for each example in-context, adding more reasoning effort may encourage the model to take this strategy. However, since the default routing for this model performs reasonably well at all lengths and especially for longer inputs, we do not explicitly specify a reasoning level for the remainder of the runs. We report results on Oolong-real in Figure 7 in the Appendix. Gemini 3 Pro: reasoning token count. The Gemini API returns the number of reasoning tokens used (but not the content of the reasoning trace). We visualize trends in reasoning tokens used in 3(b). Longer inputs generally lead to longer reasoning traces, as expected; however, when the input length is held constant, reasoning trace length correlates negatively with correctness. We hypothesize this could partially be because harder problems result in both longer traces and lower probability of answering correctly. 4.2 Over-reasoning We observe a persistent failure case related to over-reasoning on several models: the model exceeds its maximum reasoning token or total token budget in the course of a very long reasoning trace, and fails to answer the question at all.44 4 We evaluate all models on their allowed maximum reasoning trace or output token budget. This occurs particularly on Oolong-synth, which is extremely information-dense- all or nearly all of the lines are necessary to solve the questions. In our experiments, over-reasoning is the culprit when a model performs well on Oolong-real and poorly on Oolong-synth. Deepseek R1 This is a strong reasoning model and outperforms GPT-5 nano and o4-mini on Oolong-real. However, on Oolong-synth, it achieves performance below the random baseline. In manual inspection of the traces, we observe pathologies in the reasoning traces for Oolong-synth that are not present in traces for Oolong-real and appear only occasionally in traces from other models. We identify these behaviors through discussion between the authors, then prompt GPT-5.6-Luna to label 2,400 randomly selected traces from Deepseek-R1 with a yes/no classification about whether an identified behavior is present in this trace.55 5 For this labeling task, we also verify with a human annotation of a small subset, and see high agreement (85-96%) across all questions. Many traces (56.3%) do not provide an answer at all. We hypothesize that, because Oolong-synth is such an information-dense task, the model's apparent strategy of labeling each example before deciding which are relevant results in running out of context tokens. 75.7% of traces end in an incomplete sentence.66 6 Note that in some cases, the model provides a candidate answer, then begins to double-check the answer and then exceeds the maximum output length; we evaluate the last candidate answer we are able to parse. In 26.6% of cases, the model debates whether the task is impossible or intractable given the length of the input. Appendix F shows example traces for Deepseek-R1 from both splits. Gemini models Gemini 2.5 Pro is one of the strongest models on Oolong-real, but exhibits performance dropoff at longer contexts on Oolong-synth; Gemini 3 experiences the same trend less sharply, falling from the best model by a margin to comparable with Claude Sonnet by 256K on Oolong-synth. We observe that both models, and especially Gemini 2.5 Pro, increasingly exceeds its maximum output length during reasoning for long Oolong-synth inputs. Unlike the other APIs used in benchmarking, the Gemini API does not return any tokens if the max token count is exceeded during reasoning; this results in an automatic score of 0 for any cases of overlong reasoning.At 256K context length, Gemini 2.5 Pro fails to return an answer because of the max reasoning tokens constraint in 64.0% of Oolong-synth outputs, despite these inputs requiring only one quarter of the model's input context window. At 256K, Gemini 3 Pro fails to return 45% of Oolong-synth outputs. Gemini 3 Pro outperforms the older model by both answering more queries and answering a higher percentage of them correctly.77 7 Note the slight increase in Gemini 3 Pro performance from 256K to 512K: while 45% of inputs at 256K hit the max token limit during reasoning, only 11.8% of inputs at 512K hit this limit. It is not clear what causes this shift. Over-reasoning is not the only reason for declining performance with longer context; however, it bounds performance for both Gemini models. We do not observe this behavior on Oolong-real, where Gemini maintains its strong performance at longer context lengths. 4.3 Refusal Another error case we observe is when models refuse to answer a question at all. In these responses, models describe the task as impossible or intractable. Certainly some level of caution is appropriate when answering types of questions that models struggle on. However, we compare the performance of GPT 5 and GPT 5.2 and observe that surprisingly, despite very strong performance on OpenAI's MRCR long context benchmark OpenAI (2025), GPT 5.2 underperforms GPT 5 substantially on Oolong-synth. This is because GPT 5.2 is much more likely to refuse to perform the task; at 256K, GPT-5 attempts 83% of queries and answers 44.5% of these correctly, while GPT-5.2 attempts only 32% with an accuracy of 52.8%. It seems likely that GPT-5.2's performance would be improved, perhaps substantially, by attempting a higher percentage of questions. 4.4 Aggregation without classification To isolate the effects of aggregation and classification, we consider simpler settings for both dataset splits. In both settings, we see that aggregation is the bottleneck: scores are only marginally improved by removing the classification task. Providing labels Figure 4: Comparison on Oolong-synth: we provide the gold labels in the input. This leads to a consistent but small improvement. For Oolong-synth, we construct an easier version of the task by providing the label for each ICL example in-context. This reduces the task for most questions to identifying the relevant instances and summing the occurrence of each label type. Figure 4 shows the behavior of four models with and without labels provided in-context. As expected, adding labels improves accuracy; however, this improvement ranges from 10.9 points to only 0.79 points. We do not see consistently higher gains from providing labels in longer inputs, which suggests that the lower performance at longer context lengths is not primarily due to an accumulation of mislabeling errors. Additionally, the improvement is not larger for GPT-5-nano than it is for the more powerful GPT-5, suggesting that the performance difference observed between these models is not due to differing performance on the classification task. This is by design; our classification task validation was designed to eliminate overly challenging or misleading examples. Providing domain knowledge We believe that most Oolong-real tasks do not require knowledge specific to Dungeons & Dragons. For instance, tasks that require counting occurrences of a spell or action do not require knowledge of the goals of that action, only its presence in the text. Some questions, however, do benefit from domain knowledge; for instance, the question “how many cantrips were cast?” requires knowing which spells are considered cantrips, and the question “which roll type is most common?” is solvable without external knowledge but may be easier with the context of all the types of rolls and where each is generally used. While models generally have this knowledge because of the proliferation of text about this game on the internet, we experiment with providing additional context for each of these questions: a mapping from spell name to spell level or a list of roll types with brief descriptions of each. In this setting, we rerun the single-episode examples for GPT-5 and Claude Sonnet 4 and observe that providing this additional context helps slightly but not substantially: raising GPT-5's score by 3.51 points to 62.25 and raising Claude Sonnet 4's score by 2.57 points to 53.15. 4.5 Iterative retrieval baseline for Oolong-real While Oolong-real remains an information-dense task, with dozens of individual details needed to answer most questions, there are segments of the transcripts that are not necessary to generate the answer. Could Oolong-real be tackled with retrieval? We use smolagent (Roucher et al., 2025) to instantiate a simple iterative RAG baseline acting for up to 5 steps. This system is equipped with tool calls for retrieval (at an utterance level, using a BM25 retriever) and code execution. Each utterance is annotated with the transcript number and the utterance number before being indexed. We evaluated this system on the single transcript and two transcript settings (the 55K and 110K context settings in Figure 2) with GPT-5 as the backbone model. This system substantially underperforms directly using GPT-5 (with one transcript: -16.15 points; with two transcripts: -17.67 points). This shows that Oolong cannot be easily solved by a simple agentic retrieval system. We expect future work to study ways to improve retrieval tool use and explore dynamic chunking algorithms depending on the question. The interleaved and complex nature of human dialogue presents unique challenges to both agentic and long-context systems alike (see Appendix H for further discussion). However, retrieval performance, while below long-context performance, remains non-trivial; because of this, we speculate that methods that extend context by condensing sections of the context (Smith, 2025) or evicting subsections of the context (Li et al., 2025a) could be effective for this task. 5 Related Work We are far from the first to recognize the importance of aggregation tasks. Many benchmarks focus on reasoning tasks in settings where longer context is used to measure robustness to distractors, rather than to introduce increased numbers of reasoning steps. This is an important problem but orthogonal to our focus. For instance, BABILong (Kuratov et al., 2024) tests long-context reasoning by placing BABI (Weston et al., 2015) reasoning tasks within a long context and evaluating performance as the amount of distractor text increases. Ling et al. (2025), a contemporaneous work to ours, creates longer-context reasoning problems by synthetically elaborating on the documents necessary for short-context reasoning tasks and then interleaving these with irrelevant context. Other tasks require use of the full context in heavily artificial settings. RULER (Hsieh et al., 2024) includes synthetic retrieval tasks at varying context lengths. It includes multi-hop tracing and aggregation tasks such as identifying the most common words in a long input of random nouns. LongMemEval (Wu et al., 2025) and multi-round coreference resolution tasks (MRCR) evaluate long-context capabilities with conversational data. MRCR was first introduced in Gemini (Vodrahalli et al., 2024) and was further extended in OpenAI's MRCR (OpenAI, 2025). Unlike the standard NIAH task, the needles and the distractors in MRCR are sampled from the same distributions. OpenAI's MRCR includes variants of 2, 4, and 8 needles, and the task is to retrieve the ith instance of one of the needles. While this uses naturalistic data (synthetic conversations generated using GPT-4o), it is a task focused on measuring basic counting ability rather than reasoning. Conversely, GSM-infinite (Zhou et al., 2025) stress tests long-context reasoning through adaptable frameworks for varying task difficulty and length; our work is complementary, as their focus is primarily on mathematical reasoning through synthetically generated problems, while we focus on a more noisy, naturalistic text setting with primarily non-mathematical reasoning. Similarly, MathHay (Wang et al., 2024) focuses on queries across document collections that require retrieving multiple numbers from across context and performing multi-step mathematical reasoning with them. Multi-hop or long context question answering tasks can also require collecting information from multiple parts of the context, and represent a different type of long context reasoning task: one much sparser and less numeracy-focused than our aggregation task. HELMET (Yen et al., 2024) expands the tasks in RULER to include tasks related to the downstream use of LMs including a set of long-context QA tasks from Zhang et al. (2024). Other long-context QA tasks explicitly focus on capturing long dependency length (Li et al., 2024; Bai et al., 2025), on building semantic equivalents to needle-in-a-haystack retrieval (Modarressi et al., 2025), or specific downstream domains (He et al., 2025) TLDM (Hamilton et al., 2025) includes two related tasks about character location and time passed in a narrative setting in the domain of public-domain literature. MoNaCo (Wolfson et al., 2025) considers challenging information seeking queries that require retrieval and aggregation; their emphasis is on retrieval of the necessary documents for intermediate reasoning steps in an agentic setting. Other works conceptually aligned with ours include ZeroScrolls and NeedleBench. ZeroSCROLLS (Shaham et al., 2023) introduces a task of identifying the percentage of reviews on Amazon for a product that are positive, which is the closest conceptual ancestor of our work; Oolong-synth encompasses this type of task along with other aggregation tasks. NeedleBench (Li et al., 2025b) introduces a reasoning task called the Ancestral Trace Challenge, which, given a number of relations between individuals in context, asks the model to identify the most distance relative of a specific individual. The Ancestral Trace Challenge requires tree-like branching of reasoning; Oolong focuses on classify-then-aggregate style tasks and emphasizes more realistic data. A line of work has also framed LLM aggregation as database-like operations over the input text. Thorne et al. (2021) call this problem database reasoning and study architectures well-suited to this task for short contexts. HoloBench (Maekawa et al., 2025) directly synthesizes examples from text-to-SQL datasets. And concurrently to our work, SciTrek (Li et al., 2026) frames SQL-like queries over scientific article metadata as a long context evaluation. Appendix Cdiscusses additional related work on information aggregation and on Dungeons & Dragons data. 6 Conclusion We introduce Oolong, a challenging long-context information aggregation benchmark in two parts. Frontier models' poor performance on Oolong suggests more work is necessary in developing both reasoning abilities and more general long context abilities. In longer context regimes, many models perform roughly equivalently on Oolong-synth and Oolong-real at the same context length, despite the much higher information density in Oolong-synth. This suggests that better length generalization, regardless of reasoning ability, may also improve performance on this type of challenging task. However, some models behave differently in extremely information-dense regimes, highlighting a direction for future work. While prior work on reasoning chains has focused on over-reasoning for adversarial (Kumar et al., 2025) or overly simple problems (Chiang and Lee, 2024; Sui et al., 2025), Oolong indicates that even strong models may struggle to plan the reasoning quantity for information-dense inputs, where it may be desirable to accept a less optimal or more error-prone reasoning strategy (e.g. double-checking less frequently) in order to avoid running into the maximum reasoning tokens. Future models that are aware of the maximum reasoning token budget may be able to more carefully allocate reasoning effort. Overall, our results suggest that there is still a long way to go in designing robust long-context aggregation capabilities for LLMs. Reproducibility Statement We release code for dataset construction and evaluation, in addition to the final dataset version. API-based models typically exhibit some small variability across API calls, and versions of each model may be phased out over time by the providers, so exact replication of outputs may be infeasible; wherever possible, we release the model outputs for each model run reported in the paper. Ethics Statement This work is intended to further the evaluation of long-context LLMs. We do not foresee any particular additional risks introduced by our evaluation, although information aggregation, like nearly any LM capability, could be useful for both beneficial and harmful actors. CritRoleStats, whose data we use as part of constructing gold labels for the Oolong-real split, explicitly allows the use of their data for statistical analyses in their FAQ, so long as they are credited.88 8 https://www.critrolestats.com/faqs We have also notified them of our use of their data. In the course of this work, we used LM assistance in the writing of some data preprocessing, analysis, and visualization scripts. 7 Acknowledgements We are grateful to the contributors and maintainers of CritRoleStat for their excellent resource, and to the broader Critical Role community for their commitment and engagement. Like prior work that uses Dungeons & Dragons as a testbed for capabilities, this work would not be possible without the resources built by fan communities. We would also like to thank Clara Na for helpful feedback on drafts of this work. AB was supported by a grant from the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE2140739. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors. References Agarwal et al. (2024) R. Agarwal, A. Singh, L. M. Zhang, B. Bohnet, L. Rosias, S. Chan, B. Zhang, A. Anand, Z. Abbas, A. Nova, J. D. Co-Reyes, E. Chu, F. Behbahani, A. Faust, and H. Larochelle Many-shot in-context learning. External Links: 2404.11018, Link Cited by: §1. Almeida et al. (2011) T. A. Almeida, J. M. G. Hidalgo, and A. Yamakami Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 2011 ACM Symposium on Document Engineering (DOCENG'11), Cited by: Table 4. Anthropic (2025) Anthropic What kinds of documents can i upload to claude?. Note: Accessed: 2025-11-04 External Links: Link Cited by: §2.3. Bai et al. (2025) Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, J. Tang, and J. Li LongBench v2: towards deeper understanding and reasoning on realistic long-context multitasks. External Links: 2412.15204, Link Cited by: §5. Bertsch et al. (2024) A. Bertsch, M. Ivgi, E. Xiao, U. Alon, J. Berant, M. R. Gormley, and G. Neubig In-context learning with long-context models: an in-depth exploration. arXiv preprint arXiv:2405.00200. Cited by: §1. Bizzoni and Lappin (2018) Y. Bizzoni and S. Lappin Predicting human metaphor paraphrase judgments with deep neural networks. In Proceedings of the Workshop on Figurative Language Processing, B. Beigman Klebanov, E. Shutova, P. Lichtenstein, S. Muresan, and C. Wee (Eds.), New Orleans, Louisiana, pp. 45-55. External Links: Link, Document Cited by: Table 4. Callison-Burch et al. (2022) C. Callison-Burch, G. S. Tomar, L. Martin, D. Ippolito, S. Bailis, and D. Reitter Dungeons and dragons as a dialog challenge for artificial intelligence. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9379-9393. External Links: Link, Document Cited by: Appendix C. Chiang and Lee (2024) C. Chiang and H. Lee Over-reasoning and redundant calculation of large language models. External Links: 2401.11467, Link Cited by: §6. Chong et al. (2022) D. Chong, J. Hong, and C. D. Manning Detecting label errors by using pre-trained language models. External Links: 2205.12702, Link Cited by: §2.1. Desmond and Brachman (2024) M. Desmond and M. Brachman Exploring prompt engineering practices in the enterprise. External Links: 2403.08950, Link Cited by: §2.3. DeYoung et al. (2024) J. DeYoung, S. C. Martinez, I. J. Marshall, and B. C. Wallace Do multi-document summarization models synthesize?. Transactions of the Association for Computational Linguistics 12, pp. 1043-1062. External Links: Link, Document Cited by: Appendix C. García-Ferrero et al. (2023) I. García-Ferrero, B. Altuna, J. Alvez, I. Gonzalez-Dios, and G. Rigau This is not a dataset: a large negation benchmark to challenge large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 8596-8615. External Links: Link, Document Cited by: Table 4. Goldman et al. (2024) O. Goldman, A. Jacovi, A. Slobodkin, A. Maimon, I. Dagan, and R. Tsarfaty Is it really long context if all you need is retrieval? towards genuinely difficult long context NLP. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 16576-16586. External Links: Link, Document Cited by: Appendix C. Google (2025) Google Google notebooklm: ai research tool & thinking partner. Note: https://notebooklm.google/Accessed: 2025-11-04 Cited by: §2.3. Grano et al. (2017) G. Grano, A. Di Sorbo, F. Mercaldo, C. A. Visaggio, G. Canfora, and S. Panichella Software applications user reviews. Cited by: Table 4. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), pp. 633-638. Cited by: §1. Hamilton et al. (2025) S. Hamilton, R. M. M. Hicke, M. Wilkens, and D. Mimno Too long, didn't model: decomposing llm long-context understanding with novels. External Links: 2505.14925, Link Cited by: §5. He et al. (2025) Z. He, Y. Wang, J. Li, K. Liang, and M. Zhang LooGLE v2: are llms ready for real world long dependency challenges?. External Links: 2510.22548, Link Cited by: §5. Hovy et al. (2001) E. Hovy, L. Gerber, U. Hermjakob, C. Lin, and D. Ravichandran Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Language Technology Research, External Links: Link Cited by: Table 4. Hsieh et al. (2024) C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, and B. Ginsburg RULER: what's the real context size of your long-context language models?. In First Conference on Language Modeling, External Links: Link Cited by: §1, §5. Jimenez et al. (2023) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan Swe-bench: can language models resolve real-world github issues?. arXiv preprint arXiv:2310.06770. Cited by: §1. Klie et al. (2023) J. Klie, B. Webber, and I. Gurevych Annotation error detection: analyzing the past and present for a more coherent future. Computational Linguistics 49 (1), pp. 157-198. External Links: ISSN 0891-2017, Document, Link, https://direct.mit.edu/coli/article-pdf/49/1/157/2068980/coli_a_00464.pdf Cited by: §2.1. Kumar et al. (2025) A. Kumar, J. Roh, A. Naseh, M. Karpinska, M. Iyyer, A. Houmansadr, and E. Bagdasarian OverThink: slowdown attacks on reasoning llms. External Links: 2502.02542, Link Cited by: §6. Kuratov et al. (2024) Y. Kuratov, A. Bulatov, P. Anokhin, I. Rodkin, D. Sorokin, A. Sorokin, and M. Burtsev BABILong: testing the limits of llms with long context reasoning-in-a-haystack. External Links: 2406.10149, Link Cited by: §1, §5. Lahiri (2015) S. Lahiri SQUINKY! A corpus of sentence-level formality, informativeness, and implicature. arXiv preprint arXiv:1506.02306. Cited by: Table 4. Li et al. (2025a) H. Li, Y. Li, A. Tian, T. Tang, Z. Xu, X. Chen, N. Hu, W. Dong, Q. Li, and L. Chen A survey on large language model acceleration based on kv cache management. External Links: 2412.19442, Link Cited by: §4.5. Li et al. (2024) J. Li, M. Wang, Z. Zheng, and M. Zhang LooGLE: can long-context language models understand long contexts?. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 16304-16333. External Links: Link, Document Cited by: §5. Li et al. (2026) M. Li, A. Gurung, I. Saparina, and M. Lapata SciTrek: evaluating and improving long-context numerical reasoning over scientific articles. External Links: 2509.21028, Link Cited by: §5. Li et al. (2025b) M. Li, S. Zhang, T. Zhang, H. Duan, Y. Liu, and K. Chen NeedleBench: evaluating llm retrieval and reasoning across varying information densities. External Links: 2407.11963, Link Cited by: §5. Li and Roth (2002) X. Li and D. Roth Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, External Links: Link Cited by: Table 4. Ling et al. (2025) Z. Ling, K. Liu, K. Yan, Y. Yang, W. Lin, T. Fan, L. Shen, Z. Du, and J. Chen LongReason: a synthetic long-context reasoning benchmark via context expansion. External Links: 2501.15089, Link Cited by: §5. Liu et al. (2024) X. Liu, P. Dong, X. Hu, and X. Chu LongGenBench: long-context generation benchmark. External Links: 2410.04199, Link Cited by: Appendix C. Maas et al. (2011) A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Oregon, USA, pp. 142-150. External Links: Link Cited by: Table 4. Maekawa et al. (2025) S. Maekawa, H. Iso, and N. Bhutani Holistic reasoning with long-context lms: a benchmark for database operations on massive textual data. External Links: 2410.11996, Link Cited by: §5. Min et al. (2022) S. Min, M. Lewis, L. Zettlemoyer, and H. Hajishirzi MetaICL: learning to learn in context. External Links: 2110.15943, Link Cited by: §2.1. Modarressi et al. (2025) A. Modarressi, H. Deilamsalehy, F. Dernoncourt, T. Bui, R. A. Rossi, S. Yoon, and H. Schütze NoLiMa: long-context evaluation beyond literal matching. External Links: 2502.05167, Link Cited by: §5. OpenAI (2025) OpenAI OpenAI MRCR: Long context multiple needle in a haystack benchmark. External Links: Link Cited by: §1, §4.3, §5. Pavlick and Tetreault (2016) E. Pavlick and J. Tetreault An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics. Cited by: Table 4. Rameshkumar and Bailey (2020) R. Rameshkumar and P. Bailey Storytelling with dialogue: A Critical Role Dungeons and Dragons Dataset. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 5121-5134. External Links: Link, Document Cited by: Appendix C, §H.1, §3. Roucher et al. (2025) A. Roucher, A. V. del Moral, T. Wolf, L. von Werra, and E. Kaunismäki ‘Smolagents‘: a smol library to build great agentic systems.. Note: https://github.com/huggingface/smolagents Cited by: §4.5. Sarkar et al. (2025) R. Sarkar, B. Sarrafzadeh, N. Chandrasekaran, N. Rangan, P. Resnik, L. Yang, and S. K. Jauhar Conversational user-ai intervention: a study on prompt rewriting for improved llm response generation. External Links: 2503.16789, Link Cited by: §2.3. Shaham et al. (2023) U. Shaham, M. Ivgi, A. Efrat, J. Berant, and O. Levy ZeroSCROLLS: a zero-shot benchmark for long text understanding. External Links: 2305.14196, Link Cited by: §5. Smith (2025) C. Smith OpenHands context condensation for more efficient AI agents. Note: All Hands AI Blog External Links: Link Cited by: §4.5. Sui et al. (2025) Y. Sui, Y. Chuang, G. Wang, J. Zhang, T. Zhang, J. Yuan, H. Liu, A. Wen, S. Zhong, N. Zou, H. Chen, and X. Hu Stop overthinking: a survey on efficient reasoning for large language models. External Links: 2503.16419, Link Cited by: §6. Thorne et al. (2021) J. Thorne, M. Yazdani, M. Saeidi, F. Silvestri, S. Riedel, and A. Halevy Database reasoning over text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, pp. 3091-3104. External Links: Link, Document Cited by: §2, §5. Vodrahalli et al. (2024) K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, R. Anil, E. Dyer, S. Shakeri, R. Vij, H. Mehta, V. Ramasesh, Q. Le, E. Chi, Y. Lu, O. Firat, A. Lazaridou, J. Lespiau, N. Attaluri, and K. Olszewska Michelangelo: long context evaluations beyond haystacks via latent structure queries. External Links: 2409.12640, Link Cited by: §5. Wang et al. (2024) L. Wang, S. Dong, Y. Xu, H. Dong, Y. Wang, A. Saha, E. Lim, C. Xiong, and D. Sahoo MathHay: an automated benchmark for long-context mathematical reasoning in llms. External Links: 2410.04698, Link Cited by: §5. Weston et al. (2015) J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. van Merriënboer, A. Joulin, and T. Mikolov Towards ai-complete question answering: a set of prerequisite toy tasks. External Links: 1502.05698, Link Cited by: §5. Williams et al. (2018) A. Williams, N. Nangia, and S. Bowman A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122. External Links: Link Cited by: Table 4. Wolfson et al. (2025) T. Wolfson, H. Trivedi, M. Geva, Y. Goldberg, D. Roth, T. Khot, A. Sabharwal, and R. Tsarfaty MoNaCo: more natural and complex questions for reasoning across dozens of documents. Transactions of the Association for Computational Linguistics. Cited by: §5. Wu et al. (2025) D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu LongMemEval: benchmarking chat assistants on long-term interactive memory. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §5. Xue et al. (2025) H. Xue, Y. J. Oh, X. Zhou, X. Zhang, and B. Oxley User prompting strategies and chatgpt contextual adaptation shape conversational information-seeking experiences. External Links: 2509.25513, Link Cited by: §2.3. Ye et al. (2025) X. Ye, F. Yin, Y. He, J. Zhang, H. Yen, T. Gao, G. Durrett, and D. Chen LongProc: benchmarking long-context language models on long procedural generation. External Links: 2501.05414, Link Cited by: Appendix C. Yen et al. (2024) H. Yen, T. Gao, M. Hou, K. Ding, D. Fleischer, P. Izsak, M. Wasserblat, and D. Chen Helmet: how to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694. Cited by: §1, §2.2, §5. Ying and Thomas (2022) C. Ying and S. Thomas Label errors in BANKING77. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, S. Tafreshi, J. Sedoc, A. Rogers, A. Drozd, A. Rumshisky, and A. Akula (Eds.), Dublin, Ireland, pp. 139-143. External Links: Link, Document Cited by: §2.1. Zhang et al. (2015) X. Zhang, J. J. Zhao, and Y. LeCun Character-level convolutional networks for text classification. In NIPS, Cited by: Table 4, Table 4. Zhang et al. (2024) X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, and M. Sun ∞\inftybench: Extending long context evaluation beyond 100k tokens. External Links: 2402.13718, Link Cited by: §5. Zhou et al. (2023) P. Zhou, A. Zhu, J. Hu, J. Pujara, X. Ren, C. Callison-Burch, Y. Choi, and P. Ammanabrolu I cast detect thoughts: learning to converse and guide with intents and theory-of-mind in dungeons and dragons. External Links: 2212.10060, Link Cited by: Appendix C. Zhou et al. (2025) Y. Zhou, H. Liu, Z. Chen, Y. Tian, and B. Chen GSM-infinite: how do your llms behave over infinitely increasing context length and reasoning complexity?. External Links: 2502.05252, Link Cited by: §5. Zhu et al. (2023a) A. Zhu, K. Aggarwal, A. Feng, L. J. Martin, and C. Callison-Burch FIREBALL: a dataset of dungeons and dragons actual-play with structured game state information. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, pp. 4171-4193. External Links: Link, Document Cited by: Appendix C. Zhu et al. (2023b) A. Zhu, L. Martin, A. Head, and C. Callison-Burch CALYPSO: llms as dungeon master's assistants. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment 19 (1), pp. 380-390. External Links: Link, Document Cited by: Appendix C. Appendix A Oolong - Leaderboard Here we provide full results on all models we ran. In Table 3, we report the results for both Oolong-synth and Oolong-real numerically. Following prior long-context benchmarks, we measure model performance at increasing context window sizes and report a task average. For Oolong-synth, we use the standard context window sizes of up to 128K tokens. For Oolong-real, we report scores at increasing number of episode transcripts (up to 3 episodes). GPT-5 performs the best in both subsets, followed by o3, GPT-5-mini and Claude-Sonnet-4. Gemini-2.5-Pro is initially the strongest model, but falls to below-random performance by 256K, largely because of responses exceeding the maximum token count; see the analysis for further discussion. Notably, at 128K context window size nearly all our evaluated models obtain a score under 50. DeepSeek R1 does well on Oolong-real, but struggles to perform well on Oolong-synth (see §4.1. Llama-4-Maverick fails in both subsets. Oolong-synth Oolong-real Model Avg. 8K 16K 32K 64K 128K Avg. 55K 118K 175K Gemini-3-Pro 78.69 90.94 87.61 81.94 70.55 62.42 57.93 64.71 55.01 54.07 GPT-5 70.75 85.56 84.45 76.12 61.24 46.36 47.00 58.74 45.72 36.53 Gemini-2.5-Pro 55.29 88.13 69.84 56.83 36.56 25.06 52.16 60.04 49.36 47.07 o3 62.37 86.80 79.52 63.23 44.86 37.45 36.71 50.57 33.57 25.99 GPT-5-mini 63.68 85.13 77.65 64.64 50.14 40.85 34.55 49.86 29.90 23.89 Claude-Sonnet-4 58.18 74.43 62.75 55.67 50.04 48.02 36.75 50.58 32.98 26.70 o4-mini 56.74 83.07 65.10 51.86 44.15 39.53 27.13 41.69 21.77 17.93 Gemma-4-31B 53.38† 79.29 69.80 64.32 53.47 - 30.23† 54.97 35.72 - GPT-5-nano 50.73 70.96 54.53 47.81 41.02 39.31 31.05 43.09 26.82 23.23 GPT-5.2 44.29 61.80 53.95 47.32 35.31 23.10 34.40 45.67 32.15 25.38 Qwen3-4B-Instruct 40.67 55.81 41.33 41.35 34.36 30.49 22.88 39.58 18.29 10.78 Olmo-3.1-Think-32B 35.29† 59.01 43.81 39.13 34.48 - 10.43† 31.28 - - Deepseek-R1 13.11 13.94 13.65 12.91 13.20 11.87 32.00 47.85 27.35 20.81 Olmo-3-Think-7B 31.42† 46.44 39.22 38.14 33.32 - 7.47† 22.42 - - Olmo-3.1-Instruct-32B 26.67† 42.31 38.35 31.64 21.06 - 10.35† 31.04 - - Olmo-3-Instruct-7B 21.10† 29.88 28.55 25.54 21.54 - 7.93† 23.80 - - Llama-3-3B 25.60 29.31 26.37 24.86 25.09 22.39 0.00 0.00 0.00 0.00 Llama-3-8B 23.71 27.54 24.66 24.74 24.29 17.31 0.41 1.15 0.07 0.00 Qwen3-4B-Thinking 0.00 0.00 0.00 0.00 0.00 0.00 20.18 37.65 15.75 7.15 Llama-4-Maverick 16.37 15.00 16.29 15.42 16.35 18.80 2.07 2.48 2.11 1.62 Table 2: Oolong results on a number of strong models. Nearly all models we test support at least 200K context; thus, we report Oolong scores as an average over scores on 8K-175K inputs. Oolong-synth is a more information-dense task than Oolong-real; the ranking of some models shifts slightly between the two settings. Models are sorted by the average score between the two splits. - marks a context length beyond the maximum for that model; such lengths count as 0 in the average, which we denote with †. Figure 5 and Figure 6 are augmented versions of Figure 2 with all closed-weights and open-weights models (respectively) shown. Figure 5: Scores by context window length for Oolong-synth and Oolong-real for closed-weight models. Figure 6: Scores by context window length for Oolong-synth and Oolong-real for open-weight models. Oolong-synth Oolong-real Model Avg. 8K 16K 32K 64K 128K Avg. 55K 118K 175K Gemini-3-Pro 78.69 90.94 87.61 81.94 70.55 62.42 57.93 64.71 55.01 54.07 GPT-5 70.75 85.56 84.45 76.12 61.24 46.36 47.00 58.74 45.72 36.53 Gemini-2.5-Pro 55.29 88.13 69.84 56.83 36.56 25.06 52.16 60.04 49.36 47.07 o3 62.37 86.80 79.52 63.23 44.86 37.45 36.71 50.57 33.57 25.99 GPT-5-mini 63.68 85.13 77.65 64.64 50.14 40.85 34.55 49.86 29.90 23.89 Claude-Sonnet-4 58.18 74.43 62.75 55.67 50.04 48.02 36.75 50.58 32.98 26.70 o4-mini 56.74 83.07 65.10 51.86 44.15 39.53 27.13 41.69 21.77 17.93 Gemma-4-31B 53.38† 79.29 69.80 64.32 53.47 - 30.23† 54.97 35.72 - GPT-5-nano 50.73 70.96 54.53 47.81 41.02 39.31 31.05 43.09 26.82 23.23 GPT-5.2 44.29 61.80 53.95 47.32 35.31 23.10 34.40 45.67 32.15 25.38 Qwen3-4B-Instruct 40.67 55.81 41.33 41.35 34.36 30.49 22.88 39.58 18.29 10.78 Olmo-3.1-Think-32B 35.29† 59.01 43.81 39.13 34.48 - 10.43† 31.28 - - Deepseek-R1 13.11 13.94 13.65 12.91 13.20 11.87 32.00 47.85 27.35 20.81 Olmo-3-Think-7B 31.42† 46.44 39.22 38.14 33.32 - 7.47† 22.42 - - Olmo-3.1-Instruct-32B 26.67† 42.31 38.35 31.64 21.06 - 10.35† 31.04 - - Olmo-3-Instruct-7B 21.10† 29.88 28.55 25.54 21.54 - 7.93† 23.80 - - Llama-3-3B 25.60 29.31 26.37 24.86 25.09 22.39 0.00 0.00 0.00 0.00 Llama-3-8B 23.71 27.54 24.66 24.74 24.29 17.31 0.41 1.15 0.07 0.00 Qwen3-4B-Thinking 0.00 0.00 0.00 0.00 0.00 0.00 20.18 37.65 15.75 7.15 Llama-4-Maverick 16.37 15.00 16.29 15.42 16.35 18.80 2.07 2.48 2.11 1.62 Table 3: Oolong results on a number of strong models. Nearly all models we test support at least 200K context; thus, we report Oolong scores as an average over scores on 8K-175K inputs. Oolong-synth is a more information-dense task than Oolong-real; the ranking of some models shifts slightly between the two settings. Models are sorted by the average score between the two splits. - marks a context length beyond the maximum for that model; such lengths count as 0 in the average, which we denote with †. Appendix B Additional Results B.1 Reasoning effort on Oolong-real Reasoning models generally show little gain on Oolong-real; the Olmo models show a small gain and the Qwen models show a small loss. When we compare “high” and “low” reasoning effort for GPT-5-nano, we see very little variation across the settings. Figure 7 shows this result for Oolong-real. Figure 7: Comparison across reasoning levels on Oolong-real. B.2 Data contamination for Oolong-real Critical Role is an extremely popular show, with widespread discussion of the episodes online, and the statistics provided by CritRoleStats are publicly available. Could a model have simply memorized the answers to Oolong-real? Conceptually, we believe the risk of data contamination is low: the CritRoleStats statistics are provided in spreadsheets that map from episode numbers to statistics without providing any transcript text. Since we do not provide episode numbers in our prompts, contamination could only occur if a model had memorized the statistics spreadsheet and could identify from each transcript's text the appropriate episode number. To check for potential contamination, we test the first of these two steps: whether, given the episode and campaign information without the transcript text, models can answer Oolong-real questions. We use an augmented template as demonstrated below: Models are unable to perform this task. We run Gemini 3.1 Pro, GPT 5.2, and Claude Sonnet 4.5 on a set of 100 random examples with this format, prompting each model 3 times. The single best run gets 6 of the 100 questions exactly correct, with Gemini 3.1 Pro answering 3.33 correctly on average, GPT 5.2 answering 0.33 on average, and Claude Sonnet 4.5 answering 1.67 correctly on average. We examined these correct answers and found that they comprised two categories: numerical answers, which we believe represent “lucky guesses,” as no model gave consistent numerical answers across reruns for any question; and guesses that the “most frequent roll type” for a span is “attack” or “perception,” which are two of the most common roll types in D&D more broadly. We take this as evidence against widespread data contamination for Oolong-real. Appendix C Additional Related Work Dungeons & Dragons data Several prior works consider D&D as a potential testbed for model capabilities; for instance, we use a version of the Critical Role transcripts preprocessed by Rameshkumar and Bailey (2020), who proposed an episode summarization task using fan-written summaries. D&D data has also been used to study theory of mind (Zhou et al., 2023), user assistant development Zhu et al. (2023b), and dialogue generation, with game state either inferred from forum-based games (Callison-Burch et al., 2022) or captured through an external tool (Zhu et al., 2023a); to the best of our knowledge, we are the first to use fan annotations of gold labels and to consider the generation of these statistics as a task in its own right. Aggregation as an NLP concept Goldman et al. (2024) argue for the definition of long context tasks in terms of the information dispersion and input scope required to find the answer; under this taxonomy, Oolong is high dispersion (because relevant info is distribution over the full context length) and high scope (because most of the input is necessary for the task, particularly for Oolong-synth). DeYoung et al. (2024) measure synthesis in multi-document summarization by asking if summaries convey the consensus opinion of the input documents; this is a related task to the counting tasks in Oolong-synth, but framed as a measure of summarization capabilities instead of long context capabilities, and thus requires different considerations related to evaluating generated outputs. Relatedly, long generation benchmarks (Ye et al., 2025; Liu et al., 2024) measure long context ability through an entirely different lens: evaluating the ability to plan and produce coherent long outputs. Appendix D ICL label validation Table 4 describes each dataset used for Oolong-synth. Dataset Task # Labels Input Len Spam (Almeida et al., 2011) SMS spam classification 2 57 TREC-QC-coarse (Li and Roth, 2002; Hovy et al., 2001) Question type classification 6 39 AGNews (Zhang et al., 2015) Headline topic classification 4 90 App Reviews (Grano et al., 2017) Review sentiment classification 2†2^{\dagger} 49 Pavlick Formality (Lahiri, 2015; Pavlick and Tetreault, 2016) Formality classification 2†2^{\dagger} 51 IMDB reviews (Maas et al., 2011) Sentiment analysis 2†2^{\dagger} 376 HiTZ Negation (García-Ferrero et al., 2023) Verify claims about definitions 2 45 Yahoo Topics (Zhang et al., 2015) Question topic classification 10 74 MultiNLI (Williams et al., 2018) Entailment 3 70 Metaphors (Bizzoni and Lappin, 2018) Metaphor meaning validation 2 51 Table 4: Datasets used for Oolong-synth. †\dagger indicates cases where we combined similar labels to produce a smaller label-set for Oolong. The input length is the average per-instance length (in Llama 2 tokens), including our added date and user metadata. Dataset % Removed Spam 0.635% TREC-QC-coarse 0.048% AGNews 0.026% App Reviews 0.051% Formality 0.108% IMDB 0.042% Metaphors 0.000% MultiNLI 0.109% Negation 0.016% Yahoo Topics 0.188% Table 5: Only a small fraction of examples are removed during validation for each source dataset. Table 4shows the fraction of examples removed by the validation step for each dataset. For each ICL dataset used to construct Oolong-synth, we provide example instances that failed validation (and thus were discarded) and that passed validation and were used for constructing context windows. These examples are selected nearly-randomly; many of the instances that fail validation contain sexual content or offensive language, and we screen these out of the examples shown wherever possible. While the instances screened out vary by dataset, we note that the datasets that use labels scraped from web content (i.e. Yahoo Topics, AGNews, IMDB reviews, App Reviews) generally have higher rates of validation failures. Some of these failures appear to be genuine mislabelings, which is likely because of noise in the user behavior (e.g. a user asks a question about Business & Finance but mistakenly posts it in the Sports topic, or a user writes a review with mostly negative text but gives the product 4 stars). Input Label Passed validation? ‘Virtual Girlfriend' Demands Gifts HONG KONG - She needs to be coddled with sweet talk and pampered with gifts, but you'll never see her in the flesh. A Hong Kong company has developed a “virtual girlfriend” for new cell phones with video capability… World ✗ Court Hears Case of Brain Damaged Woman (AP) AP - The Florida Supreme Court questioned lawyers Tuesday about the extent of the power handed to Gov. Jeb Bush under a law that let him order the reinsertion of a brain-damaged woman's feeding tube. Sci/Tech ✗ Rooney backs fiercesome threesome (AFP) AFP - Wayne Rooney believes the three-pronged attack of himself, Michael Owen and Jermain Defoe can put England on the fast track to the World Cup finals. World ✗ From mouths of babes: What's hot, what's not How hot are the “Hot Dozen” toys? To find out, The Boston Globe put six toys from the 2004 Toy Wishes magazine Hot Dozen list in front of 13 kids from the Charlestown Boys and Girls Club. Business ✗ Crematory operator to get 12 years Ray Brent Marsh, who is to enter the plea Friday, had faced up to 8,000 years in a case that shocked the nation two years ago when investigators found hundreds of corpses at his rural northwest facility. Business ✗ Vending Machines Making Room for Healthy Products WASHINGTON (AP) - The typical vending machine fare consists of chocolate bars and potato chips, leaving few options for people seeking low-calorie or low-salt snacks. That is changing now as companies develop markets for products they expect to satisfy both nutritionists and consumers… Sci/Tech ✓ Venezuelans Line Up to Vote on Chavez CARACAS, Venezuela - Summoned by bugle calls and the detonations of huge firecrackers, Venezuelans turned out in unprecedented numbers Sunday to vote on whether to force leftist President Hugo Chavez from office. Some lines at polling places extended for 1.25 miles… World ✓ Singh Snares PGA Title Vijay Singh outlasts Justin Leonard and Chris DiMarco in a three-way playoff to win the PGA Championship on Sunday at Whistling Straits in Haven, Wisconsin. Sports ✓ China's Panchen Lama visits Tibet The boy named by the Chinese authorities as the 11th Panchen Lama visits a temple in Tibet. World ✓ The Region's Highest-Paid Executives Pay for the Washington area's top executives rose significantly last year, reversing the downward trend that set in with the recession in 2001. Business ✓ Table 6: AGNews examples that failed and passed validation. Input Label Passed validation? Kill Me.. positive ✗ Used to be great app. But since the last 7 updates always get - Authentication failed for: x@gmail.com positive ✗ App force closing on changing THEME PLZ FIX! positive ✗ You can't uninstall & I don't like my space being used. But am glad for the blind & deaf positive ✗ GOOGLE TALKBACK Read carefully & then give at least a good reviews this app is made only for persons who got disability such as a blind person & if you want to get rid this simply just ROOT your device dumb head!! negative ✗ This thing is great. I am at the bottom of the learning curve but in the few minutes I have played with it it seems like it will be easy to learn and to use. Just what I hoped for. positive ✓ Crash when set a demodulation! :( negative ✓ doesn't work with rtl-2832 Says it works with rtl devices but doesn't. never displays anything even though the rtl 2832 works with other spectrum analyzers. no help files anywhere so good luck figuring out what's wrong! negative ✓ Great app! Works really well with Galaxy note 5 positive ✓ Works great I am using HackRF One with this apps. positive ✓ Table 7: App Reviews examples that failed and passed validation. Input Label Passed validation? The Unit secret agent has signed on to appear in three episodes as a businessman being shown some houses - and, presumably, one bedroom in particular - by our randy realtor heroine. formal ✗ Barely 12, with large brown eyes and stick-like arms, Fandi is 3 years older than his brother - in his eyes almost a man. formal ✗ D-Lister Avril Lavigne appears on the cover of Z-List magazine Savvy this month. formal ✗ ORHS is going through it now, but from all I can perceive the district is out for the quick fix, chop off the principal, instead of really trying to assist the students and dig to the bottom to find the truth. formal ✗ Little Sally Draper (Kiernan Shipka) is a ”Patty Duke”-era girl - indeed, ”Mad Men” is currently set during the month ”The Patty Duke Show” premiered in 1963 - and on a ”Patty Duke”-like show, she'd just be a lispy Shirley Temple doll with a crush on Daddy. formal ✗ I have tried everything possible to attract business. formal ✓ Never heard that one before. informal ✓ ”Yup.” informal ✓ That's how we pass our traits to the next generation: through DNA formal ✓ Wedding rituals differ in different regions and communities in India. formal ✓ Table 8: Formality examples that failed and passed validation. Input Label Passed validation? Good movie, very 70s, you can not expect much from a film like this. Sirpa Lane is an actress of erotic films, a nice body but nothing exceptional. Not demand a lot from these films are light years away from the movies today, the world has changed incredibly. The plot is simple and the actors not extraordinary. positive ✗ This film Evil Breed: The legend of samhain contains very little thought or effort. It is ridiculed with specs of ultra fast ”slasher” style death and plain disgusting acts of death. The acting was rated a D as the actors show very little ability, and the stupidity of them in the film is too questionable. positive ✗ The movie ”Holly” may make the audience want to donate money towards organizations that improve the life for these poor youngsters, but the film's dramatic weaknesses may reduce its chances of being seen by enough people to make a difference. Overall, I think the concept is better as a documentary and it was not as touching as a movie. positive ✗ Although this film put Davis on the map due to her brilliantly intense performance, this film is strangely unsatisfying to me as a whole. What I cannot fathom for the life of me is just how or why Phillip would take the constant abuse this tramp constantly dishes out towards him. positive ✗ This film is just plain horrible. John Ritter doing pratt falls, 75% of the actors delivering their lines as if they were reading them from cue cards, poor editing, horrible sound mixing, and a plot that really goes nowhere. If I could sum this film up in one word, that word would be: Suckotrocity negative ✓ Zentropa has much in common with The Third Man, another noir-like film set among the rubble of postwar Europe. Like TTM, there is much inventive camera work. There is an innocent American who gets emotionally involved with a woman he doesn't really understand, and whose naivety is all the more striking in contrast with the natives. But I'd have to say that The Third Man has a more well-crafted storyline. Zentropa is a bit disjointed in this respect. Perhaps this is intentional: it is presented as a dream/nightmare, and making it too coherent would spoil the effect. This movie is unrelentingly grim-“noir” in more than one sense; one never sees the sun shine. Grim, but intriguing, and frightening. positive ✓ Never in my life have I come across a movie as bad as The Zombie Chronicles. Filmed on a budget of what looks to be about 20 bucks, TZC is a completely horrible horror movie that relies on lame, forgettable actors. Simply put, avoid TZC like a sexually-transmitted disease. negative ✓ Without wishing to be a killjoy, Brad Sykes is responsible for at least two of the most dull and clichéd films I've ever seen. The acting is terrible, the print is shoddy, and everything about this film screams ”seriously, you could do better yourself”. negative ✓ Table 9: IMDB examples that failed and passed validation. Only eight examples are shown because of the example length. Input Label Passed validation? Draupadi's eyes were diamonds. ↔\leftrightarrow Draupadi's eyes were beautiful. correct ✓ The faculty meeting was an easy breeze ↔\leftrightarrow The faculty meeting was very easy and relaxing correct ✓ The house was a tomb. ↔\leftrightarrow The house was big. incorrect ✓ I had already planted the idea in her mind. ↔\leftrightarrow I had already scared her about the idea. incorrect ✓ It is sad to observe the fruits of ignorance. ↔\leftrightarrow It is sad to observe the effects of ignorance. correct ✓ Table 10: Metaphors examples that passed validation. No examples failed validation for this dataset. Input Label Passed validation? You're safe. →\rightarrow You have nothing to worry about. neutral ✗ asks Burton, cupping his ear. →\rightarrow Burton wanted to ask what was going on, but he couldn't do much of anything while bound and gagged. contradiction ✗ I “poison my dearest Emily!” →\rightarrow How could you think I would poison Emily? entailment ✗ When he finally succeeded, after a prolonged siege and heavy losses, he punished the local population by cutting off the noses and lips of all men except those who played wind instruments. →\rightarrow All men who played wind instruments were tasked with helping cut off people's noses. neutral ✗ They died slowly, their eyes bulging and faces turning blue. →\rightarrow They died asphyxiated. neutral ✗ well you see that on television also →\rightarrow You can see that on television, as well. entailment ✓ Vrenna and I both fought him and he nearly took us. →\rightarrow Neither Vrenna nor myself have ever fought him. contradiction ✓ This analysis pooled estimates from these two studies to develop a C-R function linking PM to chronic bronchitis. →\rightarrow The analysis proves that there is no link between PM and bronchitis. contradiction ✓ He turned and smiled at Vrenna. →\rightarrow He smiled at Vrenna who was walking slowly behind him with her mother. neutral ✓ We sought to identify practices that were commonly implemented by the agencies within the past 5 years. →\rightarrow We want to identify practices commonly used by agencies in the last 5 years entailment ✓ Table 11: MultiNLI examples that failed and passed validation. Input Label Passed validation? No care is anything that serves as an enticement. True ✗ No care may be anything that serves as an enticement. True ✗ A loud utterance is an appropriate definition of produce in no context. True ✗ Complete attention is an appropriate definition of candidate in no context. True ✗ No stress is a message received and understood. True ✗ Action refers to a military engagement. True ✓ Action never stands for a distinguishing quality. True ✓ Action does not stand for a military engagement. False ✓ Action may stand for a military engagement. True ✓ Not a single action is a military engagement. False ✓ Table 12: Negation examples that failed and passed validation. Input Label Passed validation? Go chase after her and run her over while she's crossing the street ham ✗ i want to grasp your pretty booty :) ham ✗ No da if you run that it activate the full version da. ham ✗ i am seeking a lady in the street and a freak in the sheets. Is that you? ham ✗ Should i send you naughty pix? :) ham ✗ WINNER!! As a valued network customer you have been selected to receive a £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only. spam ✓ Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030 spam ✓ I'm gonna be home soon and i don't want to talk about this stuff anymore tonight, k? I've cried enough today. ham ✓ SIX chances to win CASH! From 100 to 20,000 pounds txt¿ CSH11 and send to 87575. Cost 150p/day, 6days, 16+ TsandCs apply Reply HL 4 info spam ✓ I HAVE A DATE ON SUNDAY WITH WILL!! ham ✓ Table 13: Spam examples that failed and passed validation. The examples that failed validation were almost exclusively ham messages with sexual content; the selected messages are the least inappropriate of the examples that failed validation. Input Label Passed validation? What explosive do you get by mixing charcoal, sulfur and saltpeter? entity ✗ What is the procedure called for drilling a hole in your skull to achieve a higher consciousness? entity ✗ Name 11 famous martyrs. human being ✓ What's the Olympic motto? description and abstract concept ✓ What is the highest waterfall in the United States? location ✓ What does the abbreviation AIDS stand for? abbreviation ✓ How many points make up a perfect fivepin bowling score? numeric value ✓ Table 14: TREC-coarse-Q examples that failed and passed validation. Only two examples failed validation. Input Label Passed validation? Is there a God? The question to end all questions, and begin them. Business & Finance ✗ why do we need to lie? Health ✗ try to type the word supercalifragilisticexpialidocious. 20 times fast you can only make 10 mistakes? This is for fun! Education & Reference ✗ do you think it is okay to tell a lie? Education & Reference ✗ where can you purchase cesium carbonate? Science & Mathematics ✗ What are good sources to find out about new gospel artists? Is there a site that focuses primarily on gospel? Entertainment & Music ✓ How a black hole is formed? I would like to know how a black hole can possibly be formed. Are there any experimental evidence of such creation? Science & Mathematics ✓ Economics of running a restaurant? Running a restaurant looks like hard work and long hours. What percentage of restaurants are profitable? Business & Finance ✓ Why doesn't the NBA implement a minor leagues? I don't want to see any more High School kids on the court. Sports ✓ how does a helicopter fly this is a miracle- I always wanted to learn to fly one of those. Can someone explain how can it get airborne? Science & Mathematics ✓ Table 15: Yahoo examples that failed and passed validation. One class of examples that failed validation, not demonstrated here, are sexually explicit questions labeled as seemingly random categories (likely from spam posters on the original Yahoo Answers forum). Appendix E Question types E.1 Oolong-synth Table 16, Table 17, and Table 18 list the questions used for each type of task in Oolong-synth. Counting In the above data, which of the labels is the most common? Give your final answer in the form ‘label: answer' where answer is one of the labels: {label_list}. In the above data, which of the labels is the least common? Give your final answer in the form ‘label: answer' where answer is one of the labels: {label_list}. In the above data, is label ‘{A}' more common, less common, or the same frequency as label ‘{B}'? Give your final answer in the form ‘Answer: {A} is [X]{B}', where [X]is ‘more common than', ‘less common than', or 'same frequency as'. In the above data, how many data points should be classified as label ‘{label}'? Give your final answer in the form ‘Answer: number'. Table 16: Counting questions used in the Oolong-synth dataset. User In the above data, which user is represented most often? Give your final answer in the form ‘User: [X]', where [X]is the user ID. In the above data, which user is represented the second most often? Give your final answer in the form ‘User: [X]', where [X]is the user ID. For the following question, only consider the subset of users with IDs {user_names}. Among these users, which user is represented most often? Give your final answer in the form ‘User: [X]', where [X]is the user ID. For the following question, only consider the subset of users with IDs {user_names}. Among these users, which user is represented the second most often? Give your final answer in the form ‘User: [X]', where [X]is the user ID. For the following question, only consider the subset of instances that are associated with user IDs {user_names}. Among instances associated with these users, {any of the Counting questions above} For the following question, only consider the subset of users with IDs {user_names}. Among these users, which user has the most instances with the label {label}? Give your final answer in the form ‘User: [X]', where [X]is the user ID. In the above data, which user has the most instances with the label {label}? Give your final answer in the form ‘User: [X]', where [X]is the user ID. In the above data, which user has more instances with the label {label}: User {A} or User {B}? Give your final answer in the form ‘User: [X]', where [X]is the user ID. Table 17: User questions used in the Oolong-synth dataset. Timeline In the above data, which date is represented most often? Give your final answer in the form ‘Date: [X]', where [X]is the date in the format MM/DD/YYYY. In the above data, which date is represented most often? Give your final answer in the form ‘Date: [X]', where [X]is the date in the format MM/DD/YYYY. In the above data, which date is represented the second most often? Give your final answer in the form ‘Date: [X]', where [X]is the date in the format MM/DD/YYYY. In the above data, how many dates are represented exactly {n} times? Give your final answer in the form ‘Answer: [X]', where [X]is the number of dates represented exactly {n} times. In the above data, was label ‘{key}' more common, less common, or the same frequency before {time}, as compared to after {time}? Give your final answer in the form ‘Answer: {key} is [X]before {time}', where [X]is ‘more common', ‘less common', or 'the same frequency'. In the above data, was label ‘{key}' more common, less common, or the same frequency before {time}, as compared to after {time}? Give your final answer in the form ‘Answer: {key} is [X]before {time}', where [X]is ‘more common', ‘less common', or 'the same frequency'. For the following question, only consider the subset of instances that occur in {month_name} of any year. Among instances occuring in {month_name},{any of the Counting questions above} For the following question, only consider the subset of instances that occur between {starting_date} and {ending_date}, inclusive. Among instances occuring in this date range, {any of the Counting questions above} In which month did the label ‘{label1} first occur more often than the label ‘{label2}'? Give your final answer in the form ‘Answer: [month][year]', where [month]is the name of the month and [year]is the four-digit year where ‘{label1}' first occured more often than ‘{label2}.' For how many months does the label ‘{label1}' occur more frequently than the label ‘{label2}'? Disregard months where there is a tie. Give your final answer in the form ‘Answer: [X]', where [X]is the number of months where ‘{label1}' occurs more often than ‘{label2}.' For how many months is the label ‘{label}' the single most frequently occuring label? Disregard months where there is a tie for the most common label. Give your final answer in the form ‘Answer: [X]', where [X]is the number of months where ‘{label}' is the most common label. Table 18: Timeline questions used in the Oolong-synth dataset. E.2 Oolong-real Table 19lists the question types used for single episode settings in Oolong-real dataset. In Table 20, Table 21 lists the types for multi-episode settings. Rolls Total number of rolls in this episode? (counting) Total number of rolls by the character {character name} in this episode? (counting, character) Total number of rolls by the player {player name} in this episode? (counting, player) Total number of rolls of type {roll type} in this episode? (counting, roll type) Number of rolls of natural value {roll value} in this episode? (counting, roll value) In this episode, what percentage of rolls were of value {roll value}? round to the nearest integer. (counting, roll value) What is the most common roll type in this episode? Return a comma separated list. (frequency) What is the least common roll type in this episode? Only include types with more than one roll. Return a comma separated list. (frequency) What is the most common natural roll value in this episode? Return a comma separated list. (frequency) What is the least common natural roll value in this episode? Only include values with more than one roll. Return a comma separated list. (frequency) What is the count of Crits? (natural rolls of value 1 or 20)? (counting, roll value) What is the count of Nat20s (natural rolls of value 20)? (counting, roll value) What is the count of Nat1s (natural rolls of value 1)? (counting, roll value) Spells How many spells were cast during this episode? (counting) How many spells were cast by the character {character name} in this episode? (counting, character) How many spells were cast by the player {player name} in this episode? (counting, player) How many {spell type} spells were cast during this episode? (counting, spell type) What is the first spell cast in this episode? (enumeration) What are the first two spells cast in this episode? (enumeration) What are the first three spells cast in this episode? (enumeration) What is the last spell cast in this episode? (enumeration) What are the last two spells cast in this episode? (enumeration) What are the last three spells cast in this episode? (enumeration) What is the first spell cast by each character in this episode? Return a comma separated list and retain the order of spells as they appear in the episode. (enumeration) What is the last spell cast by each character in this episode? Return a comma separated list and retain the order of spells as they appear in the episode. (enumeration) How many characters cast {spell name} spell in this episode? (counting, spell type) What is the most common spell in this episode? Return a comma separated list. (frequency) What is the least common spell in this episode? Only include spells that were cast at least once. Return a comma separated list. (frequency) Which spells were cast by more than one character in this episode? Return a comma separated list. (counting, enumeration) What is the total number of cantrip spells cast in this episode? (counting, spell type) In this episode, how many times was a spell cast at a level higher than its base level? (counting) In this episode, which spells were cast at a level higher than their base level? Return a comma separated list of unique spells. (enumeration) Table 19: Question types used in the Oolong-real dataset (single episode). For reference, we provide the question types in parentheses. Rolls What is the cumulative total of rolls by the end of episode {episode index}? Count the number of rolls and not the values of the rolls. (counting) What is the cumulative total of rolls by the character {character name} at the end of episode {episode index}? Count the number of rolls and not the values of the rolls. (counting, character) Total number of rolls across all the episodes? (counting) Total number of rolls by the character {character name} across all episodes? (counting, character) Total number of rolls by the player {player name} across all episodes? (counting, player) Total number of rolls of type {roll type} across all episodes? (counting, roll type) Number of rolls of natural value {roll value} across all episodes? (counting, roll value) Across all episodes, what percentage of rolls were of value {roll value}? round to the nearest integer. (counting, roll value) What is the most common roll type across all episodes? Return a comma separated list. (frequency) What is the least common roll type across all episodes? Only include types with more than one roll. Return a comma separated list. (frequency) What is the most common natural roll value across all episodes? Return a comma separated list. (frequency) What is the least common natural roll value across all episodes? Only include values with more than one roll. Return a comma separated list. (frequency) What is the total count of Crits across all episodes? (natural rolls of value 1 or 20)? (counting, roll value) What is the total count of Nat20s across all episodes? (natural rolls of value 20)? (counting, roll value) What is the total count of Nat1s across all episodes? (natural rolls of value 1)? (counting, roll value) Table 20: Question types used in the Oolong-real dataset (multi-episode). For reference, we provide the question types in parentheses. Spells What is the cumulative total of spells cast by the end of episode {episode index}? (counting, indexing) What is the first spell cast in the episode {episode index}? (enumeration, indexing) What is the second spell cast in the episode {episode index}? (enumeration, indexing) What is the third spell cast in the episode {episode index}? (enumeration, indexing) List the first spell cast in each episode? Return a comma separated list. (enumeration) List the last spell cast in each episode? Return a comma separated list. (enumeration) List the first spell cast by the character {character name} in each episode? Return a comma separated list. (enumeration, character) List the last spell cast by the character {character name} in each episode? Return a comma separated list. (enumeration, character) How many spells were cast across all episodes? (counting) How many spells were cast by the character {character name} across all episodes? (counting, character) How many spells were cast by the player {player name} across all episodes? (counting, player) How many {spell type} spells were cast across all episodes? (counting, spell type) How many characters cast {spell name} spell across all episodes? (counting, spell type) What is the most common spell across all episodes? Return a comma separated list. (frequency, enumeration) What is the least common spell across all episodes? Only include spells that were cast at least once. Return a comma separated list. (frequency, enumeration) What is the total number of cantrip spells cast across all episodes? (counting, spell type) Across all episodes, how many times was a spell cast at a level higher than its base level? (counting, spell level) Across all episodes, which spells were cast at a level higher than their base level? Return a comma separated list of unique spells. (enumeration, spell level) Table 21: Question types used in the Oolong-real dataset (multi-episode). For reference, we provide the question types in parentheses. Appendix F Deepseek R1 trace examples Figure 8 and Figure 9 contrast the behavior of this model on representative examples from the Oolong-synth and Oolong-real splits. See the main text for more discussion. Appendix G Question type analysis G.1 Oolong-synth On Oolong-synth, we further break down results by question type (counting, user, or temporal) and answer type (the expected format of the answer, e.g. as a label or number or date) in Figure 10. Consistently, temporal questions are the most challenging for models, highlighting the difficulty of temporal reasoning. Figure 10 shows that temporal questions are challenging for models; this is reflected in both the performance by task type and the distribution of score by answer type. Questions that require a date or month/year (e.g. “January 2021”) as an answer show generally lower performance for the same model, and show greater spread in model capabilities than the other answer categories. For instance, the gap between GPT-5 and GPT-5-nano performance is more than 4x larger for questions that require outputting a date than for questions that require outputting a label. Relative model performance is mostly stable across answer types, although Claude-Sonnet-4 is relatively much stronger on numerical reasoning and comparisons than the GPT series models. Figure 10: The performance trend for models by type of answer and type of task on Oolong-synth. G.2 Oolong-real On Oolong-real, we divide the questions into ten categories: counting, enumeration, frequency, indexing, player, character, roll type/value, spell level/type (Table 19, Table 21. Table 20). Across models, we find ‘frequency' to be the hardest and ‘spell type' the easiest (Figure 11). Except for Gemini-2.5-Pro and GPT-5, all models struggle with enumeration and indexing questions. Gemini-2.5 Pro and GPT-5 also make effective use of the player-character mapping provided in the system prompt. In Figure 11, we report model performance by question type on Oolong-real. Figure 11: Scores by context window length for various question types in Oolong-real. Some questions are only relevant for multi-episode settings. Appendix H Oolong-real data H.1 Data quality and curation The episode transcripts were first hand-transcribed by members of the fan community, who implemented a multi-step editing process with editorial guidelines. These were then further cleaned by Rameshkumar and Bailey (2020) to resolve any remaining inconsistencies. We refer the reader to the original paper for more details on the curation process. The stats were manually compiled by volunteers and the team at CritRoleStats. While it's not possible to completely eliminate sources of potential error, these types of fan-documentation processes are fastidiously annotated and cross-checked by multiple people- a standard of care far higher than that of a paid annotator with no connection to the task or data domain. H.2 Challenges of Oolong-real In the tables below, we highlight various unique challenges presented by Oolong-real. Figure 12 covers text that includes in-character, out-of-character and narrative content. Additionally, we show complex discussions surrounding dice rolls (Figure 13, Figure 15) and damage calculation (Figure 14). Experimental support, please view the build logs for errors. Generated by L A T E xml . Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" () button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion, and welcome developer contributions. We gratefully acknowledge support from our major funders, member institutions, , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from