Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities.
[H] Report GitHub Issue
×

Title:

Content selection saved. Describe the issue below:

Description:

Submit without GitHub
Submit in GitHub

arXiv is now an independent nonprofit!
Learn more
×

Back to arXiv

Why HTML?
Report Issue
Back to Abstract
Download PDF

Abstract

1
Introduction

2
Model Architecture, Training and Dataset

2.1
Model Architecture

2.2
Dataset

2.3
Training Infrastructure

2.4
Post-training

2.5
Thinking

2.6
Capability-specific improvements

2.7
The path to Gemini 2.5

3
Quantitative evaluation

3.1
Methodology

3.2
Core capability quantitative results

3.3
Evaluation of Gemini 2.5 Pro against other large language models

4
Example use cases of Gemini 2.5 Pro

4.1
Gemini Plays Pokémon

Long Context Agentic Tooling

Long Horizon Task Coherence

Screen reading

Long Context Reasoning

4.2
What else can Gemini 2.5 do?

4.3
Gemini in Google Products

5
Safety, Security, and Responsibility

Guideline for Navigating This Section

5.1
Our Process

5.2
Policies and Desiderata

5.3
Training for Safety, Security, and Responsibility

5.4
Results on Training/Development Evaluations

5.5
Automated Red Teaming

5.6
Memorization and Privacy

5.7
Assurance Evaluations and Frontier Safety Framework

5.8
External Safety Testing

6
Discussion

References

7
Contributors and Acknowledgments

8
Appendix

8.1
Evaluation additional details

8.2
Gemini Plays Pokémon Additional Details

Agent Panic

8.3
Frontier Safety Framework Evaluations Additional Details: Frontier Safety Correctness Tests

8.4
Image to Code Demo

8.5
Long Context Video Understanding Demo

License: CC BY 4.0

arXiv:2507.06261v1 [cs.CL] 07 Jul 2025

\uselogo

[H] Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities.

Gemini Team
Google

[H] Abstract

In this report, we introduce the Gemini 2.X model family: Gemini 2.5 Pro and Gemini 2.5 Flash, as well as our earlier Gemini 2.0 Flash and Flash-Lite models. Gemini 2.5 Pro is our most capable model yet, achieving SoTA performance on frontier coding and reasoning benchmarks. In addition to its incredible coding and reasoning skills, Gemini 2.5 Pro is a thinking model that excels at multimodal understanding and it is now able to process up to 3 hours of video content. Its unique combination of long context, multimodal and reasoning capabilities can be combined to unlock new agentic workflows. Gemini 2.5 Flash provides excellent reasoning abilities at a fraction of the compute and latency requirements and Gemini 2.0 Flash and Flash-Lite provide high performance at low latency and cost. Taken together, the Gemini 2.X model generation spans the full Pareto frontier of model capability vs cost, allowing users to explore the boundaries of what is possible with complex agentic problem solving.

[H] 1 Introduction

We present our latest family of natively multimodal models with advanced reasoning through thinking, long context and tool-use capabilities: Gemini 2.5 Pro and 2.5 Flash and our earlier Gemini 2.0 Flash and Gemini 2.0 Flash-Lite models. Together these form a new family of highly-capable models representing our next generation of AI models, designed to power a new era of agentic systems. Building upon the foundation of the Gemini 1.5 series
(
Gemini Team, 2024
)
, this Gemini 2.X generation brings us closer to the vision of a universal AI assistant
(
Hassabis, 2025
)
.

The Gemini 2.X series are all built to be natively multimodal, supporting long context inputs of >1 million tokens and have native tool use support. This allows them to comprehend vast datasets and handle complex problems from different information sources, including text, audio, images, video and even entire code repositories. These extensive capabilities can also be combined to build complex agentic systems, as happened in the case of Gemini Plays Pokémon
1
1
1
Pokémon is a trademark of Nintendo Co., Ltd., Creatures Inc., and Game Freak Inc.
(
Zhang, 2025
)
. Different models in the series have different strengths and capabilities: (1) Gemini 2.5 Pro is our most intelligent thinking model, exhibiting strong reasoning and code capabilities. It excels at producing interactive web applications, is capable of codebase-level understanding and also exhibits emergent multimodal coding abilities. (2) Gemini 2.5 Flash is our hybrid reasoning model with a controllable thinking budget, and is useful for most complex tasks while also controlling the tradeoff between quality, cost, and latency. (3) Gemini 2.0 Flash is our fast and cost-efficient non-thinking model for everyday tasks and (4) Gemini 2.0 Flash-Lite is our fastest and most cost-efficient model, built for at-scale usage. A full comparison of the models in the Gemini 2.X model family is provided in Table
1
. Taken together, the Gemini 2.X family of models cover the whole Pareto frontier of model capability vs cost, shifting it forward across a large variety of core capabilities, applications and use-cases, see Figure
1
.

Flash
Gemini 1.5

Pro
Gemini 1.5

Flash-Lite
Gemini 2.0

Flash
Gemini 2.0

Flash
Gemini 2.5

Pro
Gemini 2.5

Input modalities

Video, Audio
Text, Image,

Video, Audio
Text, Image,

Video, Audio
Text, Image,

Video, Audio
Text, Image,

Video, Audio
Text, Image,

Video, Audio
Text, Image,

Input length

1M

2M

1M

1M

1M

1M

Output modalities

Text

Text

Text

Text, Image*

Text, Audio*

Text, Audio*

Output length

8K

8K

8K

8K

64K

64K

Thinking

No

No

No

Yes*

Dynamic

Dynamic

Supports tool use?

No

No

No

Yes

Yes

Yes

Knowledge cutoff

November 2023

November 2023

June 2024

June 2024

January 2025

January 2025

[CAPTION] Table 1: Comparison of Gemini 2.X model family with Gemini 1.5 Pro and Flash. Tool use refers to the ability of the model to recognize and execute function calls (e.g., to perform web search, complete a math problem, execute code). *currently limited to Experimental or Preview, see Section 2.7. Information accurate as of publication date.

The Gemini 2.5 family of models maintain robust safety metrics while improving dramatically on helpfulness and general tone compared to their 2.0 and 1.5 counterparts. In practice, this means that the 2.5 models are substantially better at providing safe responses without interfering with important use cases or lecturing end users. We also evaluated Gemini 2.5 Pro’s Critical Capabilities, including CBRN, cybersecurity, machine learning R&D, and deceptive alignment. While Gemini 2.5 Pro showed a significant increase in some capabilities compared to previous Gemini models, it did not reach any of the Critical Capability Levels in any area.

Our report is structured as follows: we begin by briefly describing advances we have made in model architecture, training and serving since the release of the Gemini 1.5 model. We then showcase the performance of the Gemini 2.5 models, including qualitative demonstrations of its abilities. We conclude by discussing the safety evaluations and implications of this model series.

[CAPTION] Figure 1: Cost-performance plot. Gemini 2.5 Pro is a marked improvement over Gemini 1.5 Pro, and has an LMArena score that is over 120 points higher than Gemini 1.5 Pro. Cost is a weighted average of input and output tokens pricing per million tokens. Source: LMArena, imported on 2025-06-16.

[CAPTION] Figure 2: Number of output tokens generated per second (after the first chunk has been received from the API) for different models. Source: ArtificialAnalysis.ai, imported on 2025-06-15.

[H] 2 Model Architecture, Training and Dataset

[H] 2.1 Model Architecture

The Gemini 2.5 models are sparse mixture-of-experts (MoE)
(
Clark et al., 2022
;
Du et al., 2021
;
Fedus et al., 2021
;
Lepikhin et al., 2020
;
Shazeer et al., 2017
;
Roller et al., 2021
;
Riquelme et al., 2021
;
Jiang et al., 2024
)
transformers
(
Vaswani et al., 2017
)
with native multimodal support for text, vision, and audio inputs. Sparse MoE models activate a subset of model parameters per input token by learning to dynamically route tokens to a subset of parameters (experts); this allows them to decouple total model capacity from computation and serving cost per token. Developments to the model architecture contribute to the significantly improved performance of Gemini 2.5 compared to Gemini 1.5 Pro (see Section
3
). Despite their overwhelming success, large transformers and sparse MoE models are known to suffer from training instabilities
(
Chowdhery et al., 2022
;
Dehghani et al., 2023
;
Zhang et al., 2022
;
Molybog et al., 2023
;
Wortsman et al., 2023
;
Liu et al., 2020
;
Zhai et al., 2023
;
Fedus et al., 2021
;
Lepikhin et al., 2020
)
. The Gemini 2.5 model series makes considerable progress in enhancing large-scale training stability, signal propagation and optimization dynamics, resulting in a considerable boost in performance straight out of pre-training compared to previous Gemini models.

Gemini 2.5 models build on the success of Gemini 1.5 in processing long-context queries, and incorporate new modeling advances allowing Gemini 2.5 Pro to surpass the performance of Gemini 1.5 Pro in processing long context input sequences of up to 1M tokens (see Table
3
). Both Gemini 2.5 Pro and Gemini 2.5 Flash can process pieces of long-form text (such as the entirety of “Moby Dick” or “Don Quixote”), whole codebases, and long form audio and video data (see Appendix
8.5
). Together with advancements in long-context abilities, architectural changes to Gemini 2.5 vision processing lead to a considerable improvement in image and video understanding capabilities, including being able to process 3-hour-long videos and the ability to convert demonstrative videos into interactive coding applications (see our recent blog post by
Baddepudi et al., 2025
).

The smaller models in the Gemini 2.5 series — Flash size and below — use distillation
(
Hinton et al., 2015
;
Anil et al., 2018
)
, as was done in the Gemini 1.5 series
(
Gemini Team, 2024
)
. To reduce the cost associated with storing the teacher’s next token prediction distribution, we approximate it using a k-sparse distribution over the vocabulary. While this still increases training data throughput and storage demands by a factor of k, we find this to be a worthwhile trade-off given the significant quality improvement distillation has on our smaller models, leading to high-quality models with a reduced serving cost (see Figure
2
).

[H] 2.2 Dataset

Our pre-training dataset is a large-scale, diverse collection of data encompassing a wide range of domains and modalities, which includes publicly available web documents, code (various programming languages), images, audio (including speech and other audio types) and video, with a cutoff date of June 2024 for 2.0 and January 2025 for 2.5. Compared to the Gemini 1.5 pre-training dataset we also utilized new methods for improved data quality for both filtering, and deduplication. Our post-training dataset, like Gemini 1.5, consists of instruction tuning data that is carefully collected and vetted. It is a collection of multimodal data with paired instructions and responses, in addition to human preference and tool-use data.

[H] 2.3 Training Infrastructure

This model family is the first to be trained on TPUv5p architecture. We employed synchronous data-parallel training to parallelise over multiple 8960-chip pods of Google’s TPUv5p accelerators, distributed across multiple datacenters.

The main advances in software pre-training infrastructure compared with Gemini 1.5 were related to elasticity and mitigation of SDC (Silent Data Corruption) errors:

1.

Slice-Granularity Elasticity
: Our system now automatically continues training with fewer “slices” of TPU chips when there is a localized failure, and this reconfiguration results in tens of seconds of lost training time per interruption, compared with the 10 or more minute delay waiting for healthy machines to be rescheduled without elasticity; the system continues training at around 97% throughput while the failed slice is recovering. At the scale of this training run we see interruptions from hardware failures multiple times per hour, but our fault tolerance machinery is designed to tolerate the higher failure rates expected at much larger scales.

2.

Split-Phase SDC Detection
: On previous large-scale runs it could take many hours to detect and localize machines with SDC errors, requiring both downtime while debugging, and rollback/replay of a large number of potentially corrupt training steps. We now use lightweight deterministic replay to immediately repeat any step with suspicious metrics, and compare per-device intermediate checksums to localize the root cause of any data corruption. Empirically, accelerators that start to exhibit intermittent SDCs are identified within a few minutes, and quickly excluded from the job. During this run, around 0.25% of steps were replayed due to suspected SDCs and 6% of these replays turned out to be genuine hardware corruption.

Both of the above techniques were relatively simple to implement due to the single-controller design of the Pathways system
(
Barham et al., 2022
)
, which allows all accelerators to be coordinated from a single python program with a global view of the system state. The controller can make use of parallel ‘remote python’ operations on TPU workers to monitor training metrics, track performance stragglers, and root-cause SDC errors.

Overall during the run, 93.4% of the time was spent performing TPU computations; the remainder was approximately spent half in elastic reconfigurations, and half in rare tail cases where elasticity failed. Around 4.5% of the computed steps were replays or rollbacks for model debugging interventions.

[H] 2.4 Post-training

Since the initial announcement of Gemini 1.5, significant advancements have been made in our post-training methodologies, driven by a consistent focus on data quality across the Supervised Fine-Tuning (SFT), Reward Modeling (RM), and Reinforcement Learning (RL) stages. A key focus has been leveraging the model itself to assist in these processes, enabling more efficient and nuanced quality control.

Furthermore, we have increased the training compute allocated to RL, allowing deeper exploration and refinement of model behaviors. This has been coupled with a focus on verifiable rewards and model-based generative rewards to provide more sophisticated and scalable feedback signals. Algorithmic changes to the RL process have also improved stability during longer training. These advancements have enabled Gemini 2.5 to learn from more diverse and complex RL environments, including those requiring multi-step actions and tool use. The combination of these improvements in data quality, increased compute, algorithmic enhancements, and expanded capabilities has contributed to across-the-board performance gains (as described in Section
3
) , notably reflected in the significant increase in the model’s LMArena Elo scores, with both Gemini 2.5 Flash and Pro gaining more than 110 points over their Gemini 1.5 counterparts (122 for Gemini 2.5 Pro and 111 for Gemini 2.5 Flash, see Figure
1
), along with significant improvements on several other frontier benchmarks.

[H] 2.5 Thinking

[CAPTION] Figure 3: Impact of “Thinking” on Gemini’s performance on AIME 2025 (Balunović et al., 2025), LiveCodeBench (corresponding to 10/05/2024 - 01/04/2025 in the UI) (Jain et al., 2024) and GPQA diamond (Rein et al., 2024) benchmarks.

Past Gemini models produce an answer immediately following a user query. This constrains the amount of inference-time compute (Thinking) that our models can spend reasoning over a problem. Gemini Thinking models are trained with Reinforcement Learning to use additional compute at inference time to arrive at more accurate answers. The resulting models are able to spend tens of thousands of forward passes during a “thinking” stage, before responding to a question or query.

Our training recipe has evolved from the original experimental thinking model, Gemini 2.0 Flash Thinking (launched in December 2024), to the Gemini 2.5 Thinking series, which incorporates Thinking natively across all domains. The result is a single model that can achieve stronger reasoning performance across the board, and is able to scale up its performance further as a function of inference time (see Figure
3
for an example of the impact of Thinking).

We integrated Thinking with other Gemini capabilities, including native multimodal inputs (images, text, video, audio) and long context (1M+ tokens). For any of these capabilities, the model decides for itself how long to think before providing an answer. We also provide the ability to set a Thinking budget, constraining the model to respond within a desired number of tokens. This allows users to trade off performance with cost. To demonstrate this capability, we conducted experiments where we systematically varied the thinking budget, measured in the number of tokens the model is allowed to use for internal computation. As shown in Figure
4
, increasing this budget allows the model to scale its performance and achieve significantly higher accuracy.

[CAPTION] Figure 4: Impact of thinking budget on performance on AIME 2025 (Balunović et al., 2025), LiveCodeBench (corresponding to 10/05/2024 - 01/04/2025 in the UI) (Jain et al., 2024) and GPQA diamond (Rein et al., 2024) benchmarks.

[H] 2.6 Capability-specific improvements

While most of the changes made to our training architecture and recipe since Gemini 1.5 have resulted in improvements across all capabilities, we have also made changes that have resulted in some capability-specific wins. We will now discuss these for code, factuality, long context, multilinguality, audio, video, and agentic use cases (with a particular focus on Gemini Deep Research).

[H] Code

Gemini 2.0 and 2.5 represent a strategic shift of our development priorities towards delivering tangible real-world value, empowering users to address practical challenges and achieve development objectives within today’s complex, multimodal software environments. To realize this, concerted efforts have been undertaken across both pre-training and post-training phases since Gemini 1.5. In pre-training, we intensified our focus on incorporating a greater volume and diversity of code data from both repository and web sources into the training mixture. This has rapidly expanded coverage and enabled the development of more compute-efficient models. Furthermore, we have substantially enhanced our suite of evaluation metrics for assessing code capabilities aligned with downstream use cases, alongside improving our ability to accurately predict model performance. During post-training, we developed novel training techniques incorporating reasoning capabilities and curated a diverse set of engineering tasks, with the aim to equip Gemini with effective problem-solving skills crucial for addressing modern engineering challenges. Key applications demonstrating these advancements include IDE functionalities, code agent use cases for complex, multi-step operations within full repositories, and multimodal, interactive scenarios such as end-to-end web and mobile application development. Collectively, these efforts have yielded broad and significant improvements in Gemini’s coding capabilities. This progress is evidenced by superior performance on established benchmarks: performance on LiveCodeBench
(
Jain et al., 2024
)
increased from 30.5% for Gemini 1.5 Pro to 74.2% for Gemini 2.5 Pro, while that for Aider Polyglot
(
Gauthier, 2025
)
went from 16.9% to 82.2%. Performance on SWEBench-verified
(
Jimenez et al., 2024
;
Chowdhury et al., 2024
)
went from 34.2% to 67.2%, see Table
3
and Figure
5
in Section
3.2
. Furthermore, Gemini 2.5 Pro obtained an increase of over 500 Elo over Gemini 1.5 Pro on the LMArena WebDev Arena
(
LMArena Team, 2025
;
Chiang et al., 2024
)
, resulting in meaningful enhancements in practical applications, including UI and web application development
(
Doshi, 2025a
)
, and the creation of sophisticated agentic workflows
(
Kilpatrick, 2025
)
.

[H] Factuality

Within the context of generative models, ensuring the factuality of model responses to information-seeking prompts remains a core pillar of Gemini model development. With Gemini 1.5, our research was concentrated on enhancing the model’s world knowledge and its ability to provide answers faithfully grounded in the context provided within the prompt. This effort culminated in the December 2024 release of FACTS Grounding
(
Jacovi et al., 2025
)
, now an industry-standard benchmark for evaluating an LLM’s capacity to generate responses grounded in user-provided documents. With Gemini 2.0 and 2.5, we have significantly expanded our scope to address multimodal inputs, long-context reasoning, and model-retrieved information. At the same time, the landscape and user expectations for factuality have evolved dramatically, shaped in part by Google’s deployment of AI Overviews and AI Mode
(
Stein, 2025
)
. To meet these demands, Gemini 2.0 marked a significant leap as our first model family trained to natively call tools like Google Search, enabling it to formulate precise queries and synthesize fresh information with sources. Building on this, Gemini 2.5 integrates advanced reasoning, allowing it to interleave these search capabilities with internal thought processes to answer complex, multi-hop queries and execute long-horizon tasks. The model has learned to use search and other tools, reason about the outputs, and issue additional, detailed follow-up queries to expand the information available to it and to verify the factual accuracy of the response. Our latest models now power the experiences of over 1.5B monthly active users in Google’s AI Overviews and 400M users in the Gemini App. These models exhibit state-of-the-art performance across a suite of factuality benchmarks, including SimpleQA for parametric knowledge
(
Wei et al., 2024
)
, FACTS Grounding for faithfulness to provided documents
(
Jacovi et al., 2024
;
Jacovi et al., 2025
)
, and the Vectara Hallucination Leaderboard
(
Hughes et al., 2023
)
, cementing Gemini as the model of choice for information-seeking demands.

[H] Long context

Modeling and data advances helped us improve the quality of our models’ responses to queries utilizing our one million-length context window, and we reworked our internal evaluations to be more challenging to help steer our modeling research. When hill-climbing, we targeted challenging retrieval tasks (like LOFT of
Lee et al., 2024
), long-context reasoning tasks (like MRCR-V2 of
Vodrahalli et al., 2024
), and multimodal tasks (like VideoMME of
Fu et al., 2025
). According to the results in Table
6
, the new 2.5 models improve greatly over previous Gemini 1.5 models and achieve state-of-the-art quality on all of those. An example showcasing these improved capabilities for video recall can be seen in Appendix
8.5
, where Gemini 2.5 Pro is able to consistently recall a 1 second visual event out of a full 46-minute video.
2
2
2
For further discussion on long context capabilities, challenges, and future outlook, the Release Notes podcast episode “Deep Dive into Long Context” provides additional insights and discussion:
https://youtu.be/NHMJ9mqKeMQ
.

[H] Multilinguality

Gemini’s multilingual capabilities have also undergone a profound evolution since 1.5, which already encompassed over 400 languages via pretraining. This transformation stems from a holistic strategy, meticulously refining pre- and post-training data quality, advancing tokenization techniques, innovating core modeling, and executing targeted capability hillclimbing. The impact is particularly striking in Indic and Chinese, Japanese and Korean languages, where dedicated optimizations in data quality and evaluation have unlocked dramatic gains in both quality and decoding speed. Consequently, users benefit from significantly enhanced language adherence, responses designed to faithfully respect the requested output language, and a robust improvement in generative quality and factuality across languages, solidifying Gemini’s reliability across diverse linguistic contexts.

[H] Audio

While Gemini 1.5 was focused on native audio understanding tasks such as transcription, translation, summarization and question-answering, in addition to understanding, Gemini 2.5 was trained to perform audio generation tasks such as text-to-speech or native audio-visual to audio out dialog. To enable low-latency streaming dialog, we incorporated causal audio representations that also allow streaming audio into and out of Gemini 2.5. These capabilities derive from an increased amount of pre-training data spanning over 200 languages, and development of improved post-training recipes. Finally, through our improved post-training recipes, we have integrated advanced capabilities such as thinking, affective dialog, contextual awareness and tool use into Gemini’s native audio models.

[H] Video

We have significantly expanded both our pretraining and post-training video understanding data, improving the audio-visual and temporal understanding capabilities of the model. We have also trained our models so that they perform competitively with 66 instead of 258 visual tokens per frame, enabling using about 3 hours of video instead of 1h within a 1M tokens context window
3
3
3
This is referred to as low media resolution in the API:
https://ai.google.dev/api/generate-content#MediaResolution
.
. Two new applications that were not previously possible, but that have been unlocked as a result of these changes are: creating an interactive app from a video (such as a quiz to test students’ understanding of the video content) and creating a p5.js animation to show the key concepts from the video. Our recent blog post
(
Baddepudi et al., 2025
)
shows examples of these applications.

[H] Gemini as an Agent: Deep Research

Gemini Deep Research
(
Gemini Team, Google, 2024
)
is an agent built on top of the Gemini 2.5 Pro model designed to strategically browse the web and provide informed answers to even the most niche user queries. The agent is optimized to perform task prioritization, and is also able to identify when it reaches a dead-end when browsing. We have massively improved the capabilities of Gemini Deep Research since its initial launch in December 2024. As evidence of that, performance of Gemini Deep Research on the Humanity’s Last Exam benchmark
(
Phan et al., 2025
)
has gone from 7.95% in December 2024 to the
SoTA score of 26.9% and 32.4% with higher compute
(June 2025).

[H] 2.7 The path to Gemini 2.5

On the way to Gemini 2.5 Pro, we experimented with our training recipe, and tested a small number of these experimental models with users. We have already discussed Gemini 2.0 Flash Thinking (see Section
2.5
). We will now discuss some of the other models briefly.

[H] Gemini 2.0 Pro

In February 2025, we released an experimental version of Gemini 2.0 Pro. At the time, it had the strongest coding performance of any model in the Gemini model family, as well as the best understanding and world knowledge. It also came with our largest context window at 2 million tokens, which enabled it to comprehensively analyze and understand vast amounts of information. For further information about Gemini 2.0 Pro, please see our earlier blog posts
(
Mallick and Kilpatrick, 2025
;
Kavukcuoglu, 2025
)
.

[H] Gemini 2.0 Flash Native Image Generation Model

In March 2025, we released an experimental version of Gemini 2.0 Flash Native Image Generation. It has brought to the users new capabilities as a result of a strong integration between the Gemini model and image-generation capabilities, enabling new experiences related to image generation & image editing via natural-language prompting. Capabilities such as multi-step conversational editing or interleaved text-image generation are very natural in such a setting, and horizontal transfer related to multi-language coverage immediately allowed such experiences to happen across all the languages supported by the Gemini models. Native image generation turns Gemini into a multimodal creation partner and enables Gemini to express ideas through both text and images, and to seamlessly move between the two. For further information about Gemini 2.0 Flash Native Image Generation, please see our earlier blog posts
(
Kampf and Brichtova, 2025
;
Sharon, 2025
)

[H] Gemini 2.5 Audio Generation

With Gemini 2.5, the Controllable TTS and Native Audio Dialog capabilities are available as separate options on AI Studio (Generate Media and Stream sections respectively). Our Gemini 2.5 Preview TTS Pro and Flash models support more than 80 languages with the speech style controlled by a free formatted prompt which can specify style, emotion, pace, etc, while also being capable of following finer-grained steering instructions specified in the transcript. Notably, Gemini 2.5 Preview TTS can generate speech with multiple speakers, which enables the creation of podcasts as used in NotebookLM Audio Overviews
(
Wang, 2024
)
. Our Gemini 2.5 Flash Preview Native Audio Dialog model uses native audio generation, which enables the same level of style, pacing and accent control as available in our controllable TTS offering. Our dialog model supports tool use and function calling, and is available in more than 24 languages. With native audio understanding and generation capabilities, it can understand and respond appropriately to the user’s tone. This model is also capable of understanding when to respond to the user, and when not to respond, ignoring background and non-device directed audio. Finally, we also offer an advanced ‘Thinking’ variant that effectively handles more complex queries and provides more robust and reasoned responses in exchange for some additional latency.

[H] Gemini 2.5 Flash-Lite

In June 2025, we released an experimental version of Gemini 2.5 Flash-Lite (
gemini-2.5-flashlite-preview-06-17
). It comes with the same capabilities that make Gemini 2.5 helpful, including the ability to turn thinking on at different budgets, connecting to tools like Google Search and code execution, support for multimodal inputs and a 1 million-token context length. Our goal was to provide an economical model class which provides ultra-low-latency capabilities and high throughput per dollar, echoing the initial release of 2.0 Flash-Lite
(
Google DeepMind, 2025b
;
Mallick and Kilpatrick, 2025
)
.

[H] Gemini 2.5 Pro Deep Think

To advance Gemini’s capabilities towards solving hard reasoning problems, we developed a novel reasoning approach, called Deep Think, that naturally blends in parallel thinking techniques during response generation. Deep Think enables Gemini to creatively produce multiple hypotheses and carefully critique them before arriving at the final answer, achieving state-of-the-art performances in challenging benchmarks such as Olympiad math (USAMO 2025), competitive coding (LiveCodeBench), and multimodality (MMMU), see more details at
(
Doshi, 2025b
)
. We announced Gemini 2.5 Deep Think at Google I/O and launched an experimental version to trusted testers and advanced users in June 2025.

[H] 3 Quantitative evaluation

[CAPTION] Figure 5: Performance of Gemini 2.X models at coding, math and reasoning tasks in comparison to previous Gemini models. SWE-bench verified numbers correspond to the “multiple attempts” setting reported in Table 3.

We will now examine the performance of the Gemini 2.X model family across a wide range of benchmarks. We will first compare the performance of the Gemini 2.X models to the earlier Gemini 1.5 Pro and Flash models, before we compare the performance of Gemini 2.5 Pro to other available large language models.

With web-scale pre-training of AI models, coupled with the post-training techniques that allow policy and reward models to leverage public benchmarks, avoiding leaks and biases in the data used for pre- and post-training is a persistent challenge. In the development of the Gemini 2.5 series, in addition to the standard n-gram based decontamination we used in Gemini 1.5, we also employed semantic-similarity and model based decontamination procedures to help mitigate evaluation set leakage. To move beyond the reliance on training set decontamination, we also continue reporting on internally developed non-public benchmarks, such as HiddenMath.

[H] 3.1 Methodology

Model

AI Studio model ID

Gemini 1.5 Flash

gemini-1.5-flash-002

Gemini 1.5 Pro

gemini-1.5-pro-002

Gemini 2.0 Flash-Lite

gemini-2.0-flash-lite-001

Gemini 2.0 Flash

gemini-2.0-flash-001

Gemini 2.5 Flash

gemini-2.5-flash

Gemini 2.5 Pro

gemini-2.5-pro

[CAPTION] Table 2: Mapping of Gemini model names to AI Studio API model IDs.

In Table
3
, we compare the performance of Gemini 2.5 models to the Gemini 1.5 models, while in Table
4
, we compare the performance of Gemini 2.5 Pro to that of other large language models.

Gemini results:
All Gemini scores are pass@1, and are “single attempt” settings unless otherwise specified. In the “single attempt” setting, no majority voting or parallel test-time compute is permitted, while in the “multiple attempts” setting, test-time selection of the candidate answer is allowed. All Gemini evaluations are run with the AI Studio API for the model id that we provide in Table
2
, with default sampling settings. To reduce variance, we average over multiple trials for smaller benchmarks. Aider Polyglot scores are the pass rate average of 3 trials. Vibe-Eval results are reported using Gemini as a judge.

Non-Gemini results:
All the results for non-Gemini models are sourced from providers’ self reported numbers unless mentioned otherwise. All “SWE-bench Verified” numbers follow official provider reports, which means that they are computed using different scaffoldings and infrastructure, and aren’t directly comparable.

For some evaluations, we obtain results from the external leaderboards that report results on these benchmarks. Results for Humanity’s Last Exam results are sourced from
Scale’s leaderboard
and results for DeepSeek are obtained from the
text-only variant of the leaderboard
(indicated with a
⋄
\diamond
in Table
4
). For Gemini 2.0 models, the reported results are
on an earlier HLE dataset
(indicated with a
†
\dagger
in Table
3
). Results on LiveCodeBench results are taken from
(1/1/2025 - 5/1/2025) in the UI
. Aider Polyglot numbers come from
the Aider leaderboard
and results for SimpleQA come from
this repo
where available. Results on FACTS Grounding come from
Kaggle
. In the case of LOFT and MRCR-V2, we report results on both the 128k context length variant, as well as the 1M context length variant. In the 128k context length variant, we measure performance on contexts up to 128k, while for the 1M context length variant, we report performance on context lengths of exactly 1M.

More details on all benchmarks, including subsets and how scores were obtained can be found in Table
11
in Appendix
8.1
.

[H] 3.2 Core capability quantitative results

As can be seen in Table
3
, and Figure
5
, the Gemini 2.5 models excel at coding tasks such as LiveCodeBench, Aider Polyglot and SWE-bench Verified, and represent a marked improvement over previous models.

In addition to coding performance, Gemini 2.5 models are noticeably better at math and reasoning tasks than Gemini 1.5 models: performance on AIME 2025 is 88.0% for Gemini 2.5 Pro compared to 17.5% for Gemini 1.5 Pro, while performance on GPQA (diamond) went from 58.1% for Gemini 1.5 Pro to 86.4%. Performance on image understanding tasks has also increased significantly.

It is also interesting to note that the Gemini 2.5 Flash model has become the second most capable model in the Gemini family, and has overtaken not just previous Flash models, but also the Gemini 1.5 Pro model released one year ago.

Capability

Benchmark

Flash
Gemini 1.5

Pro
Gemini 1.5

Flash-Lite
Gemini 2.0

Flash
Gemini 2.0

Flash
Gemini 2.5

Pro
Gemini 2.5

Code

LiveCodeBench

30.3%

29.7%

29.1%

29.1%

59.3%

74.2%

Aider Polyglot

2.8%

16.9%

10.5%

21.3%

56.7%

82.2%

Verified
SWE-bench

attempt
single

9.6%

22.3%

12.5%

21.4%

48.9%

59.6%

attempts
multiple

19.7%

34.2%

23.1%

34.2%

60.3%

67.2%

Reasoning

(diamond)
GPQA

50.0%

58.1%

50.5%

65.2%

82.8%

86.4%

Last Exam
Humanity’s

no tools

-

4.6%

4.6%
†
\dagger

5.1%
†
\dagger

11.0%

21.6%

Factuality

SimpleQA

8.6%

24.9%

16.5%

29.9%

26.9%

54.0%

Grounding
FACTS

82.9%

80.0%

82.4%

84.6%

85.3%

87.8%

Multilinguality

(Lite)
Global MMLU

72.5%

80.8%

78.0%

83.4%

88.4%

89.2%

ECLeKTic

16.4%

27.0%

27.7%

33.6%

36.8%

46.8%

Math

AIME 2025

14.7%

17.5%

23.8%

29.7%

72.0%

88.0%

HiddenMath-Hard

36.8%

44.3%

47.4%

53.7%

75.5%

80.5%

Long-context

retrieval)
LOFT (hard

≤
\leq
128K

67.3%

75.9%

50.7%

58.0%

82.1%

87.0%

1M

36.7%

47.1%

7.6%

7.6%

58.9%

69.8%

(8-needle)
MRCR-V2

≤
\leq
128K

18.4%

26.2%

11.6%

19.0%

54.3%

58.0%

1M

10.2%

12.1%

4.0%

5.3%

21.0%

16.4%

Understanding
Image

MMMU

58.3%

67.7%

65.1%

69.3%

79.7%

82.0%

(Reka)
Vibe-Eval

52.3%

55.9%

51.5%

55.4%

65.4%

67.2%

ZeroBench

0.5%

1.0%

0.75%

1.25%

2.0%

4.5%

BetterChartQA

59.0%

65.8%

52.3%

57.8%

67.3%

72.4%

[CAPTION] Table 3: Evaluation of Gemini 2.5 family across a wide range of core capability benchmarks and in comparison to Gemini 1.5 models. Please see Tables 5 and 6 for audio and video evaluations. See Table 11 Appendix 8.1 for benchmarks and evaluation details.

[H] 3.3 Evaluation of Gemini 2.5 Pro against other large language models

Capability

Benchmark

Pro
Gemini 2.5

high
o3

high
o4-mini

Sonnet
Claude 4

Opus
Claude 4

Extended Thinking
Grok 3 Beta

0528
DeepSeek R1

Code

LiveCodeBench

74.2%

72.0%

75.8%

48.9%

51.1%

–

70.5%

Aider Polyglot

82.2%

79.6%

72.0%

61.3%

72.0%

53.3%

71.6%

Verified
SWE-bench

attempt
single

59.6%

69.1%

68.1%

72.7%

72.5%

-

-

attempts
multiple

67.2%

-

-

80.2%

79.4%

-

57.6%

Reasoning

(diamond)
GPQA

attempt
single

86.4%

83.3%

81.4%

75.4%

79.6%

80.2%

81.0%

Last Exam
Humanity’s

tools
no

21.6%

20.3%

18.1%

7.8%

10.7%

-

14.0%
⋄
\diamond

Factuality

SimpleQA

54.0%

48.6%

19.3%

-

-

43.6%

27.8%

Grounding
FACTS

87.8%

69.9%

62.1%

79.1%

77.7%

74.8%

82.4%

Math

AIME 2025

attempt
single

88.0%

88.9%

92.7%

70.5%

75.5%

77.3%

87.5%

Long-context

retrieval)
LOFT (hard

≤
\leq
128K

87.0%

77.0%

60.5%

81.6%

-

73.1%

-

1M

69.8%

-

-

-

-

-

-

(8-needle)
MRCR-V2

≤
\leq
128K

58.0%

57.1%

36.3%

39.1%

16.1%*

34.0%

-

1M

16.4%

-

-

-

-

-

-

Understanding
Image

MMMU

attempt
single

82.0%

82.9%

81.6%

74.4%

76.5%

76.0%

No MM support

[CAPTION] Table 4: Performance comparison of Gemini 2.5 Pro with other large language models on different capabilities. Please see Tables 5 and 6 for audio and video evaluations. See Table 11 for benchmarks and evaluation details. *: with no thinking and API refusals

Relative to other large language models that are available (see Table
4
), Gemini achieves the highest score on the Aider Polyglot coding task, Humanity’s Last Exam, GPQA (diamond), and on the SimpleQA and FACTS Grounding factuality benchmarks out of all of the models examined here. Gemini also continues to stand out for achieving the SoTA score on both the LOFT and MRCR long-context tasks at 128k context, and is the only one, amongst the models examined in the above table, to support context lengths of 1M+ tokens.

Not all of the models shown in Table
4
have native support for multimodal inputs. As such, we compare against a different set of models for audio and video understanding.

[H] Audio Understanding

In Table
5
, we showcase the performance of the Gemini 2.5 model family at audio understanding, and compare the performance of these models to earlier Gemini models, as well as to GPT models. Gemini 2.5 Pro demonstrates state-of-the-art audio understanding performance as measured by public benchmarks for ASR and AST, and compares favorably to alternatives under comparable testing conditions (using the same prompts and inputs).

Benchmark

Flash
Gemini 1.5

Pro
Gemini 1.5

Flash-Lite
Gemini 2.0

Flash
Gemini 2.0

Flash
Gemini 2.5

Pro
Gemini 2.5

Audio Preview
GPT-4o mini

Audio Preview
GPT 4o

transcribe
GPT 4o

(53 lang, WER
↓
\downarrow
)
FLEURS

12.71

7.14

9.60

9.04

9.95

6.66

19.52

12.16

8.17

(21 lang, BLEU
↑
\uparrow
)
CoVoST2

34.81

37.53

34.74

36.35

36.15

38.48

29.5

35.89

–

[CAPTION] Table 5: Performance comparison of Gemini 2.5 models to earlier Gemini models, as well as to GPT models for audio understanding. Note that for GPT models, metrics may differ from those previously reported due to differing eval methodologies. See Table 11 for benchmarks and evaluation details.

[H] Video Understanding

In Table
6
, we show the performance of Gemini 2.5 models at video understanding. As can be seen, Gemini 2.5 Pro achieves state-of-the-art performance on key video understanding benchmarks, surpassing recent models like GPT 4.1 under comparable testing conditions (same prompt and video frames). For cost-sensitive applications, Gemini 2.5 Flash provides a highly competitive alternative.

Modalities

Benchmark

Flash
Gemini 1.5

Pro
Gemini 1.5

Flash-Lite
Gemini 2.0

Flash
Gemini 2.0

Flash
Gemini 2.5

Pro
Gemini 2.5

GPT 4.1
OpenAI

visual-only

ActivityNet-QA

56.2

57.3

55.3

56.4

65.1

66.7

60.4

EgoTempo

34.5

36.3

30.1

39.3

36.7

44.3

40.3

Perception Test

66.5

69.4

67.5

68.8

75.1

78.4

64.8

QVHighlights

64.4

68.7

25.7

63.9

52.4

75.0

71.4

VideoMMMU

64.8

70.4

64.3

68.5

79.2

83.6

60.9

1H-VideoQA

61.9

72.2

55.6

67.5

67.5

81.0

56.8

audio + visual

LVBench

61.9

65.7

52

61.8

62.7

78.7

63.4

VideoMME

70.4

73.2

62.1

72.8

75.5

84.3

72.0

VATEX

56.9

55.5

58.5

56.9

65.2

71.3

64.1

VATEX-ZH

46.2

52.2

43.2

48.5

43.9

59.7

48.7

YouCook2 Cap

153.2

170.0

78.6

129.0

177.6

188.3

127.6

visual + subtitles

Minerva

49.6

52.8

46.8

52.4

60.7

67.6

54.0

Neptune

78.7

82.7

81.5

83.1

84.3

87.3

85.2

subtitles
audio+visual+

VideoMME

77.3

79.8

72.5

78.8

81.5

86.9

79.6

[CAPTION] Table 6: Evaluation of Gemini 2.5 vs. prior models and GPT 4.1 on video understanding benchmarks. Performance is measured by string-match accuracy for multiple-choice VideoQA, LLM-based accuracy for open-ended VideoQA, R1@0.5 for moment retrieval and CIDEr for captioning. See Table 11 for benchmarks and evaluation details.

[H] 4 Example use cases of Gemini 2.5 Pro

[H] 4.1 Gemini Plays Pokémon

[CAPTION] Figure 6: Progression of the Gemini Plays Pokémon agent through the game, across two runs. Run 1 was the development run where changes to the harness were performed. Run 2 is the fully autonomous run with the final fixed scaffold. Both runs have the same starter (Squirtle). The events are ordered on the y-axis by the order they happened, following the order of Run 2 when there is a conflict. Notably, the GPP agent additionally went through the difficult (and optional) Seafoam Islands dungeon in Run 2, while in Run 1, GPP reached Cinnabar Island via Pallet Town and Route 21.

On March 28, 2025, an independent developer not affiliated with Google,
Joel Zhang
, set up a Twitch stream (Gemini Plays Pokémon, or GPP) for Gemini 2.5 Pro (Gemini 2.5 Pro Exp 03-25) to play Pokémon Blue on stream
(
Zhang, 2025
)
as an experiment to better understand how well the model was capable of playing Pokémon (in a similar spirit to Claude Plays Pokémon, see
Anthropic 2025
). In this initial run through the game, the goal was to live-stream the development process of an agentic harness capable of playing the full game (and in particular the minimal transformation of vision to text necessary to do so), see Figure
14
for a description of the final agent setup. As such, over the course of the run, modifications were made to the setup as difficulties arose, providing a deeply interesting lens via which to analyze some of the qualitative improvements that the 2.5 Pro model has made, particularly in the regimes of solving long reasoning problems and agentic capabilities over extended time horizons. Around 1 month later, on May 2, 2025, Gemini 2.5 Pro completed the game after 813 hours and entered the Hall of Fame to become the Pokémon League Champion! On May 22, 2025, GPP began a fully autonomous 2nd run through the game with Gemini 2.5 Pro (Gemini 2.5 Pro Preview 05-06) with the finalized fixed agentic harness, and progressed through the game considerably faster, completing the game in 406.5 hours (nearly exactly half the time of the first run).

See Figure
6
for a timeline of GPP’s progress through major game milestones to game completion. We report # hours to each milestone in order to normalize for the amount of time models take per action. See Appendix
8.2
for more figures.

[H] Capabilities assessment

Gemini 2.5 Pro showcased many impressive capabilities associated with reasoning and long-term planning while playing Pokémon. We will now discuss two in particular, but for more examples, see Appendix
8.2
.

[H] Long Context Agentic Tooling

Within the agent scaffolding, GPP has access to two agentic tools (see Figure
14
). These prompted versions of Gemini 2.5 Pro, hereafter
pathfinder
and
boulder_puzzle_strategist
, have been able to:

1.

Solve complex spinner puzzles in one shot (for instance in Rocket Hideout),

2.

Solve the step-constrained multi-map puzzle of the Safari Zone,

3.

Find long pathways through complex mazes like Route 13,

4.

Solve boulder puzzles across long distances in Victory Road and the Seafoam Islands.

Each task requires reasoning over a long context - the
pathfinder
model would often have to reason over contexts of 100K+ tokens, and find paths up to 50 actions in length (in the extreme case, paths consisting of up to 150 actions have also been found!).

[H] Long Horizon Task Coherence

While Gemini 2.5 Pro is impressive in a more local sense, the agent also exhibited remarkable long-term task coherence in achieving global, high-level goals in the face of real and hallucinated setbacks towards making forward progress. Because the agent is able to change goals at will, and will generally follow those goals as long as needed, it is extremely impressive that the agent can satisfy numerous requirements for tactical, necessary goals, such as acquiring Hidden Moves, as well as maintain enough strategic task coherence to beat the entire game and become the Pokémon Champion.

[H] Where does 2.5 Pro struggle while playing Pokémon?

In addition to more standard hallucination issues (which interestingly were plausibly reduced in Run 2 by explicitly prompting the model to act as a player completely new to the game, see Appendix
8.2
for more details), there are a few particular points of struggle we would like to emphasize.

[H] Screen reading

While obtaining excellent benchmark numbers on real-world vision tasks, 2.5 Pro struggled to utilize the raw pixels of the Game Boy screen directly, though it could occasionally take cues from information on the pixels. As a result, it was necessary for the required information from the screen to be translated into a text format in the agent framework, using information from the game’s RAM state. During one portion of the game, the developer tested an ablation where all vision was completely removed from the model context – the model was able to function roughly as well as without the vision information, suggesting that most of the performance does not significantly depend on the visual input.

[H] Long Context Reasoning

Gemini 2.5 Pro’s state-of-the-art long context performance for both reasoning and retrieval tasks (see Tables
3
and
4
) was a cornerstone of the GPP agent’s success. Its ability to reason over a 100k token context was instrumental for leveraging the complex toolset and maintaining a relatively coherent strategy (e.g., optimal balance of performance, planning quality, and information recall.)

While Gemini 2.5 Pro supports 1M+ token context, making effective use of it for agents presents a new research frontier. In this agentic setup, it was observed that as the context grew significantly beyond 100k tokens, the agent showed a tendency toward favoring repeating actions from its vast history rather than synthesizing novel plans. This phenomenon, albeit anecdotal, highlights an important distinction between long-context for retrieval and long-context for multi-step, generative reasoning.

Teaching an agent to effectively plan and avoid such loops over massive past trajectories of context is an exciting and active area of research; the co-design of agent scaffolds and models to unlock the full potential of million-token context is an intriguing research direction and one of our primary focuses.

[H] 4.2 What else can Gemini 2.5 do?

Gemini 2.5 Pro excels at transforming diverse, often unstructured, inputs into interactive and functional applications. For instance, it can
take a PDF script of a play and generate a tool that allows drama students to practice their lines
. Gemini 2.5 Pro can also take an uploaded photograph of a bookshelf and create a
curated book recommendation application
. Gemini 2.5 Pro can utilize its underlying spatial understanding capability and convert images into a structural representation like HTML or SVG. In Figure
16
in Appendix
8.4
, we show a comparison of Gemini 1.5 Pro and Gemini 2.5 Pro on an image-to-svg task, where Gemini 2.5 Pro reconstructs much more visual details and the spatial arrangements of objects better resembles the original image.

Furthermore, Gemini 2.5 Pro demonstrates strong skills in generating sophisticated simulations and visualizations, ranging from
interactive solar system models
(
source
) to the creative rendering of abstract mathematical concepts, such as
drawing a logo using Fourier series
(
source
). This capability extends to the development of tools that intersect creativity and utility: we see examples of specialized applications like a
custom cartography tool
or use cases that generate
photorealistic 3D user interfaces
from descriptive text and reference images, complete with appropriate styling and interactivity (
source
).

Collectively, these examples illustrate that Gemini 2.5 Pro is not just a useful coding and writing assistant, but excels at a wide range of complex tasks, ranging from those relevant for education to creative expression. The model empowers users to rapidly prototype specialized utilities, develop engaging educational content, and realize intricate creative visions with a high degree of sophistication.

[H] 4.3 Gemini in Google Products

As a final example of what Gemini can do, we note that Gemini (or a custom version of Gemini) is now incorporated into a wide variety of Google products. These include, but are not limited to,
AI Overviews
and
AI Mode
within Google Search,
Project Astra
, the audiovisual-to-audio dialog agent,
Gemini Deep Research
, the research assistant discussed in Section
2.7
,
NotebookLM
, the tool capable of generating podcasts and audio overviews from even the most obscure inputs,
Project Mariner
, the web browsing agent, and Google’s coding agent,
Jules
.

[H] 5 Safety, Security, and Responsibility

We’re committed to developing Gemini responsibly, innovating on safety and security alongside capabilities. We describe our current approach in this section, which includes how we train and evaluate our models, focusing on automated red teaming, going through held-out assurance evaluations on present-day risks, and evaluating the potential for dangerous capabilities in order to proactively anticipate new and long-term risks.

[H] Guideline for Navigating This Section

1.

Our Process (Section
5.1
):
Begin here to understand our overall safety methodology.

2.

Policies and Desiderata (Section
5.2
):
Next, dive into the safety criteria we use to evaluate and optimize our systems.

3.

Training for Safety (Section
5.3
):
Discover how we incorporate safety into pre-training and post-training.

4.

Results from Development Evaluations (Section
5.4
):
Results on our development evaluations for policies and desiderata.

5.

Automated Red Teaming (Section
5.5
):
A description and results from our automated red teaming work for safety and security.

6.

Memorization & Privacy (Section
5.6
):
Our analysis of memorization and privacy risks.

7.

Assurance Evaluations and Frontier Safety Framework (Section
5.7
):
We dive into our held-out evaluations and tests for dangerous capabilities.

8.

External Safety Testing (Section
5.8
):
Learn what independent testers discovered about our system’s safety.

[H] 5.1 Our Process

We aim for Gemini to adhere to specific safety, security, and responsibility criteria. These cover what Gemini should not do (e.g., encourage violence), and what Gemini should do (e.g., respond in a helpful way when possible instead of refusing, provide multiple perspectives when consensus does not exist). We also leverage automated red teaming to identify cases where the model fails to respond in a safe or helpful manner. These failure cases are used to improve evaluations and training data.

Once the model is trained, we run assurance evaluations that we then use for review and release decisions. Importantly, these are conducted by a group outside of the model development team, and datasets are held out. Furthermore, for models where there are new capabilities or a significant performance improvement, we engage independent external groups, including domain experts and a government body, to further test the model to identify blind spots.

We also evaluate the model for dangerous capabilities outlined in our Frontier Safety Framework
(
Google DeepMind, 2025a
)
, namely: Cybersecurity, CBRN, Machine Learning R&D, and Deceptive Alignment.

Finally, The Google DeepMind Responsibility and Safety Council (RSC), our governance body, reviews initial ethics and safety assessments on novel model capabilities in order to provide feedback and guidance during model development. The RSC also reviews metrics on the models’ performance via assurance evals and informs release decisions.

[H] 5.2 Policies and Desiderata

[H] Safety policies

The Gemini safety policies align with Google’s standard framework which prevents our our Generative AI models from generating specific types of harmful content, including:

1.

Child sexual abuse and exploitation

2.

Hate speech (e.g., dehumanizing members of protected groups)

3.

Dangerous content (e.g., promoting suicide, or instructing in activities that could cause real-world harm)

4.

Harassment (e.g., encouraging violence against people)

5.

Sexually explicit content

6.

Medical advice that runs contrary to scientific or medical consensus

These policies apply across modalities. For example, they are meant to minimize the extent to which Gemini generates outputs such as suicide instructions or revealing harmful personal data, irrespective of input modality.

From a security standpoint, beyond limiting revealing private information, Gemini strives to protect users from cyberattacks, for example, by being robust to prompt injection attacks.

[H] Desiderata, aka “helpfulness”

Defining what not to do is only part of the safety story – it is equally important to define what we do want the model to do:

1.

Help the user:
fulfill the user request; only refuse if it is not possible to find a response that fulfills the user goals without violating policy.

2.

Assume good intent:
if a refusal is necessary, articulate it respectfully without making assumptions about user intent.

[H] 5.3 Training for Safety, Security, and Responsibility

We build safety into the models though pre-and post-training approaches. We start by constructing metrics based on the policies and desiderata above, which we typically turn into automated evaluations that guide model development through successive model iterations. We use data filtering and conditional pre-training, as well as Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human and Critic Feedback (RL*F). Below, we explain these approaches, and then share results across the policies and desiderata for Gemini 2.0 and Gemini 2.5 models.

•

Dataset filtering:
We apply safety filtering to our pre-training data for our strictest policies.

•

Pre-training monitoring:
Starting in Gemini 2.0, we developed a novel evaluation to capture the model’s ability to be steered towards different viewpoints and values, which helps align the model at post-training time.

•

Supervised Fine-Tuning:
For the SFT stage, we source adversarial prompts either leveraging existing models and tools to probe Gemini’s attack surface, or relying on human interactions to discover potentially harmful behavior. Throughout this process we strive for coverage of the safety policies described above across common model use cases. When we find that model behavior needs improvement, either because of safety policy violations, or because the model refuses when a helpful, non-policy-violating answer exists, we use a combination of custom data generation recipes loosely inspired by Constitutional AI
(
Bai et al., 2022
)
, as well as human intervention to revise responses. The process described here is typically refined through successive model iterations. We use automated evaluations on both safety and non-safety metrics to monitor impact and potential unintended regressions.

•

Reinforcement Learning from Human and Critic Feedback (RL*F):
Reward signal during RL comes from a combination of a Data Reward Model (DRM), which amortizes human preference data, and a Critic, a prompted model that grades responses according to pre-defined rubrics. We divide our interventions into Reward Model and Critic improvements (RM), and reinforcement learning (RL) improvements. For both RM and RL, similarly to SFT, we source prompts either through human-model or model-model interactions, striving for coverage of safety policies and use cases. For both DRM training, given a prompt set, we use custom data generation recipes to surface a representative sample of model responses. Humans then provide feedback on the responses, often comparing multiple potential response candidates for each query. This preference data is amortized in our Data Reward Model. Critics, on the other hand, do not require additional data, and iteration on the grading rubric can be done offline. Similarly to SFT, RL*F steers the model away from undesirable behavior, both in terms of content policy violations, and trains the model to be helpful. RL*F is accompanied by a number of evaluations that run continuously during training to monitor for safety and other metrics.

[H] 5.4 Results on Training/Development Evaluations

Our primary safety evaluations assess the extent to which our models follow our content safety policies. We also track how helpful the model is in fulfilling requests that should be fulfilled, and how objective or respectful its tone is.

Compared to Gemini 1.5 models, the 2.0 models are substantially safer. However, they over-refused on a wide variety of benign user requests. In Gemini 2.5, we have focused on improving helpfulness / instruction following (IF), specifically to reduce refusals on such benign requests. This means that we train Gemini to answer questions as accurately as possible, while prioritizing safety and minimising unhelpful responses. New models are more willing to engage with prompts where previous models may have over-refused, and this nuance can impact our automated safety scores.

We expect variation in our automated safety evaluations results, which is why we review flagged content to check for egregious or dangerous material. Our manual review confirmed losses were overwhelmingly either a) false positives or b) not egregious. Furthermore, this review confirmed losses are narrowly concentrated around explicit requests to produce sexually suggestive content or hateful content, mostly in the context of creative use-cases (e.g. historical fiction). We have not observed increased violations outside these specific contexts.

Metric

Gemini 1.5 Flash 002
Gemini 2.0 Flash-Lite vs.

Gemini 1.5 Flash 002
Gemini 2.0 Flash vs.

Gemini 1.5 Flash 002
Gemini 2.5 Flash vs.

Gemini 1.5 Pro 002
Gemini 2.5 Pro vs.

Violations**
EN text-to-text Policy

↓
\downarrow
14.3%

↓
\downarrow
12.7%

↓
\downarrow
8.2%

↓
\downarrow
0.9%

Violations**
i18n text-to-text Policy

↓
\downarrow
7.3%

↓
\downarrow
7.8%

↑
\uparrow
1.1%*

↓
\downarrow
3.5%

Violations
Image-to-text Policy

↑
\uparrow
4.6%*

↑
\uparrow
5.2%*

↑
\uparrow
6.4%*

↑
\uparrow
1.8%*

Tone

↑
\uparrow
8.4%

↑
\uparrow
1.5%

↑
\uparrow
7.9%

↑
\uparrow
18.4%

Following
Helpfulness / Instruction

↓
\downarrow
19.7%

↓
\downarrow
13.2%

↑
\uparrow
13.6%

↑
\uparrow
14.8%

[CAPTION] Table 7: Comparison of safety and helpfulness metrics for Gemini 2.0 and 2.5 models relative to Gemini 1.5 baselines. A down arrow (↓\downarrow) indicates a reduction in the number of policy violations (better), while an up arrow (↑\uparrow) indicates an improvement for Tone and Helpfulness / Instruction Following. *No egregious losses reported. **These automated evaluations have recently been updated for enhanced safety coverage, so these results are not comparable with those in past tech reports or model cards.

[H] 5.5 Automated Red Teaming

[H] For Safety

To complement human red teaming and our static evaluations, we make extensive use of automated red teaming (ART) to dynamically evaluate Gemini at scale
(
Perez et al., 2022
;
Beutel et al., 2024
;
Samvelyan et al., 2024
)
. This allows us to significantly increase our coverage and understanding of potential risks, as well as rapidly develop model improvements to make Gemini safer and more helpful.

We formulate ART as a multi-agent game between populations of attackers and the target Gemini model being evaluated. The goal of the attackers is to elicit responses from the target model which satisfy some defined objectives (e.g. if the response violates a safety policy, or is unhelpful). These interactions are scored by various judges (e.g. using a set of policies), with the resulting scores used by the attackers as a reward signal to optimize their attacks.

Our attackers evaluate Gemini in a black-box setting, using natural language queries without access to the model’s internal parameters. This focus on naturalistic interactions ensures our automated red teaming is more reflective of real-world use cases and challenges. Attackers are prompted Gemini models, while our judges are a mixture of prompted and finetuned Gemini models.

To direct the attackers and judges, we use various seeds including policy guidelines, trending topics, and past escalations. Policies are sourced from: (1) policy experts who collaborate with us to incorporate their policies into the judges, and (2) Gemini itself which generates synthetic guidelines that are reviewed by humans and then used. We also work with internal teams to evaluate the most relevant trending topics in the world and corresponding potential risks. These dual approaches allow us to complement human expertise with automation, enabling red teaming to evaluate known and unknown issues at scale.

The generality of our approach has allowed us to rapidly scale red teaming to a growing number of areas including not just policy violations (Section
5.4
), but also areas such as tone, helpfulness, and neutrality. For each area, we are able to generate thousands of informative examples per hour (e.g. prompts which elicit unsafe or biased responses from Gemini). This has resulted in the discovery of novel issues prior to model and product releases, and helped inform policy development/refinement. Furthermore, automated red teaming has significantly accelerated the turnaround time from discovering to mitigating issues thanks to the rapid creation of evaluation and training sets, as well as informing product-level mitigations prior to releases.

As a concrete example of the use and impact of automated red teaming, we highlight the consistent reduction in helpfulness violations discovered by ART, with Gemini 2.5 Flash and 2.5 Pro being our most helpful models to-date while maintaining robust safety metrics.

Model

violations (from ART)
Dangerous Content policy

(from ART)
Helpfulness violations

Gemini 1.5 Flash 002

38.3%

9.5%

Gemini 1.5 Pro 002

43.5%

8.9%

Gemini 2.0 Flash

25.2%

8.1%

Gemini 2.5 Flash

26.9%

6.6%

Gemini 2.5 Pro

24.3%

6.1%

[CAPTION] Table 8: Policy and helpfulness violations as discovered by Automated Red Teaming (ART). Lower percentages are better.

[H] For Security

Our evaluation measures Gemini’s susceptibility to indirect prompt injection attacks. As illustrated in Figure
7
, we specifically focus on a scenario in which a third party hides malicious instructions in external retrieved data, in order to manipulate Gemini into taking unauthorized actions through function calling.

[CAPTION] Figure 7: Illustration of the scenario where a Gemini-based AI Agent is attacked by malicious instructions hidden in external retrieved data.

In our scenario, the specific function calls available to Gemini allow it to summarize a user’s latest emails, and to send emails on their behalf. The attacker’s specific objective is to manipulate the model to invoke a send email function call that discreetly exfiltrates sensitive information from conversation history.

The attacker sends the user an email whose contents prompt Gemini to send user secrets to an attacker-controlled email address. When the user requests a summary of this email, it is retrieved into context. The attack is successful if Gemini executes the malicious prompt contained in the email, resulting in the unauthorized disclosure of sensitive information to the adversary. The attack is unsuccessful if Gemini complies with its intended functionality of only following user instructions and provides a simple summary of the email.

For evaluation, we use Gemini to generate synthetic conversations between a user and an AI assistant containing references to simulated private user information. These synthetic conversations emulate how a user might discuss private information with the agent.

Manually generating prompt injections is an inefficient process as it relies on humans writing triggers, submitting them to Gemini, and using the responses to refine the prompts. Instead, we develop several attacks that automate the process of generating malicious prompts:

•

Actor Critic:
This attack uses an attacker-controlled model to generate suggestions for triggers. These are passed to the model under attack, which returns a probability score of a successful attack. Based on this probability, the attack model refines the trigger. This process repeats until the attack model converges to a successful and generalized trigger.

•

Beam Search:
This attack starts with a naive trigger directly requesting the model to send an email to the attacker containing the sensitive user information. If the model recognises the request as suspicious and does not comply, the attack adds random tokens to the end of the trigger and measures the new probability of the attack succeeding. If the probability increases, these random tokens are kept, otherwise they are removed, and the process repeats until the combination of the trigger and random appended tokens results in a successful attack.

•

Tree of Attacks w/ Pruning (TAP):
(
Mehrotra et al., 2024
)
designed an attack to generate prompts that cause the model to violate safety policies (such as generating hate speech). We adapt this attack, making several adjustments to target security violations. Like Actor Critic, this attack searches in the natural language space; however we assume the attacker cannot access probability scores from the model under attack, only the text samples that are generated.

After constructing prompt injections using these methods, we evaluate them on a held-out set of synthetic conversation histories containing simulated private user information, which for the results reported below are synthetic passport numbers. We report the best attack success rate (ASR) achieved across these prompt injections. ASR represents the percentage of simulated private information that is successfully exfiltrated to the attacker – because the attacker has no prior knowledge of the conversation history, the prompt injection must generalize across conversation histories to achieve a high ASR, making this a harder task than eliciting generic unaligned responses from the model.

The table below summarizes the results. For both Gemini 2.0 Flash and Gemini 2.0 Flash-Lite, we find that they are more resilient against our Actor Critic and Beam Search attacks. In Actor Critic, which uses iteratively more persuasive natural language prompt injections, ASRs reduced substantially compared with both Gemini 1.5 Flash; while in Beam Search which primarily relies on discovering random tokens resulting in successful attacks, the ASR also reduced noticeably. However, for TAP, which leverages more creative natural language scenarios like role-playing to attack the model, the ASR on Gemini 2.0 Flash increased by 16.2% on already very high ASRs for Gemini 1.5 Flash.

Our results indicate that Gemini 2.0 models are becoming more resilient to some classes of prompt injection attacks in environments containing private user data. However, improved model capabilities of Gemini 2.0 versus Gemini 1.5 also enable attackers to leverage the model’s ability to create natural language attacks like TAP. The lower ASRs on Actor Critic and TAP against Gemini 2.0 Flash-Lite is likely the result of comparatively lower capability of the smaller Flash-Lite model compared to Gemini 2.0 Flash, rather than an indication of greater internal resilience.

In Gemini 2.5 Flash and Gemini 2.5 Pro, we have observed greater resilience against all three of our attack techniques across the board, despite significantly increased model capabilities. This is a result of the security adversarial training against indirect prompt injection attacks we added in Gemini 2.5, further details for which can be found in the white paper
(
Shi et al., 2025
)
we recently released. However the Gemini 2.5 Pro model is still less resilient compared to Gemini 2.5 Flash, showing that increased model capabilities in Pro still constrain our mitigations. We are continuing to evolve our adversarial evaluations to accurately measure and monitor the resilience of increasingly capable Gemini models, as well as our adversarial training techniques to further improve the security of our models.

Attack Technique

Gemini 1.5 Flash 002
Gemini 2.0 Flash-Lite vs.

Gemini 1.5 Flash 002
Gemini 2.0 Flash vs.

Gemini 1.5 Flash 002
Gemini 2.5 Flash vs.

Gemini 1.5 Pro 002
Gemini 2.5 Pro vs.

Actor Critic

(
↓
\downarrow
44.2%
)
52.0%

(
↓
\downarrow
28.2%
)
68.0%

(
↓
\downarrow
55.4%
)
40.8%

(
↓
\downarrow
36.8%
)
61.4%

Beam Search

(
↓
\downarrow
9.0%
)
75.4%

(
↓
\downarrow
17.2%
)
67.2%

(
↓
\downarrow
80.2%
)
4.2%

(
↓
\downarrow
35.6%
)
63.8%

TAP

(
↓
\downarrow
17.4%
)
64.8%

(
↑
\uparrow
16.2%
)
98.4%

(
↓
\downarrow
28.6%
)
53.6%

(
↓
\downarrow
57.0%
)
30.8%

[CAPTION] Table 9: Comparison of Attack Success Rates (ASRs) against Gemini 2.5, 2.0, and 1.5 models. ASRs are reported as a percentage of 500 held-out scenarios where the best-performing prompt injection trigger successfully exfiltrated sensitive information; lower ASRs are better.

[H] 5.6 Memorization and Privacy

[H] Discoverable Memorization

Large language models are known to potentially produce near-copies of some training examples
(
Carlini et al., 2022
;
Ippolito et al., 2022
;
Biderman et al., 2023
;
Nasr et al., 2023
)
. Several prior reports have released audits that quantify the risk of producing near-copies of the training data by measuring the model’s memorization rate
(
Gemini Team, 2024
;
Gemma Team, 2024
;
CodeGemma Team et al., 2024
;
Anil et al., 2023
;
Kudugunta et al., 2023
;
Chowdhery et al., 2022
;
Grattafiori et al., 2024
;
Pappu et al., 2024
)
. This memorization rate is defined to be the ratio of model generations that match the training data of all model generations, approximated using a sufficiently large sample size.

In this report, we follow the methodology described in
Gemini Team (2024)
. Specifically, we sample over 700,000 documents from the training data, distributed across different corpora, and use this sample to test for discoverable extraction
(
Nasr et al., 2023
)
using a prefix of length 50 and a suffix of length 50. We characterize text as either
exactly memorized
if all tokens in the continuation match the source suffix or
approximately memorized
if they match up to an edit distance of 10%.

Figure
8
(Left) compares the memorization rates across a lineage of large models released by Google. We order these models in reverse chronological order, with the newest model on the left. We find that the Gemini 2.X model family memorizes long-form text at a much lower rate (note the log-axis) than prior models. Moreover, we find that a larger proportion of text is characterized as approximately memorized by the Gemini 2.0 Flash-Lite and Gemini 2.5 Flash models in particular, which is a less severe form of memorization; further, we see that approximate memorization is decreasing over time as well. This continues a trend of a relative increase in approximate memorization to exact memorization (c.f. 1.5x for Gemma and 14x for Gemini 1.5).

Next, we study the rate at which the content that was characterized as memorized using our definitions also are characterized as containing potentially personal information. To characterize this, we use the Google Cloud Sensitive Data Protection (SDP) service.
4
4
4
Available at:
https://cloud.google.com/sensitive-data-protection
This tool uses broad detection rules to classify text into many types of potentially personal and sensitive information. SDP is designed to have high recall and does not consider the context in which the information may appear, which leads to many false positives. Thus, we are likely overestimating the true amount of potentially personal information contained in the outputs classified as memorized. SDP also provides broad severity levels: low, medium, and high. We classify text as personal if SDP classifies it as personal information at any severity level. Figure
8
(Right) shows the results of this analysis. We observed no personal information in the outputs characterized as memorization for Gemini 2.X model family models; this indicates a low rate of personal data in outputs classified as memorization that are below our detection thresholds. Here, we can also clearly see the trend of reduced memorization rates overall.

[CAPTION] Figure 8: (Left) Total memorization rates for both exact and approximate memorization. Gemini 2.X model family memorize significantly less than all prior models. (Right) Personal information memorization rates. We observed no instances of personal information being included in outputs classified as memorization for Gemini 2.X, and no instances of high-severity personal data in outputs classified as memorization in prior Gemini models.

[H] Extractable Memorization and Divergence

Nasr et al. (2023)
showed that aligned models may also emit data that is classified as memorization under certain circumstances. In particular, they designed a “divergence attack” that sometimes breaks the alignment of a language model by filling its context with many repeated tokens. We evaluate Gemini 2.X model family models to understand their susceptibility to diverging, and in particular, to emitting data classified as memorization as a result of this attack.

We follow the same test as in
Gemini Team (2024)
. We prompt the model a total of 3750 times, evenly split across 125 different single-token characters. We first classify when the model returns diverged outputs, and in these cases, we then determine how many of these outputs match training data, i.e., are classified as memorization.

Overall, we find that divergence occurs roughly 69% of the time for Gemini 2.0 Flash + Flash-Lite and roughly 59% of the time for the Gemini 2.5 model family. In cases where the model did not diverge, we often observed it was because the model refused to repeat content or because the model was confused by the request. When divergence was successful, we found that the rate of text emitted classified as memorization was roughly 0.2%. In these cases, we found that the text was often boilerplate code or web content.

[H] 5.7 Assurance Evaluations and Frontier Safety Framework

Assurance evaluations are our ‘arms-length’ internal evaluations for responsibility governance decision making
(
Weidinger et al., 2024
)
. They are conducted separately from the model development team, to inform decision-making about release. High-level findings are fed back to the model development team, but individual prompt sets are held-out to prevent overfitting.

[H] Baseline Assurance

Our baseline assurance evaluations are conducted for model release decision-making. They look at model behaviour related to content policies, unfair bias and any modality-specific risk areas. They were performed for 2.5 Pro and 2.5 Flash in line with the previous Gemini 2.0 releases and the Gemini 1.5 tech report, covering all modalities in the Gemini 2.5 model family.

Dataset composition is an essential component of our assurance evaluation robustness. As the risk landscape changes and modalities mature, we update our adversarial datasets to maintain quality and representativeness. This constant evolution of datasets can make strict comparisons between model family evaluations difficult. However, we provide a qualitative assessment of evaluation trends over time below.

For child safety evaluations, we continue to see the Gemini 2.5 family of models meeting or improving upon launch thresholds, which were developed by expert teams to protect children online and meet
Google’s commitments to child safety
across our models and Google products.

For content policies, we see the Gemini 2.5 family of models displaying lower violation rates in most modalities than Gemini 1.5 and 2.0 families, which in turn was a significant improvement on Gemini 1.0. When looking at violation rates across input modalities for 2.5 Pro and 2.5 Flash (i.e. text, image, video, audio), we observe the image to text modality has a relatively higher violation rate, though the overall violation rates remained low. We also observed that violation rates for 2.5 Pro and 2.5 Flash tended to be slightly higher with thinking traces visible.

Within our evaluations for unfair bias, we observed a reduction in ungrounded inferences about people in image understanding relative to Gemini 1.5. Ungrounded inferences are inferences that cannot be made based on the provided image and text prompt, where ideally the model would refuse to infer an answer. A high rate of ungrounded inferences about people may create greater risk of stereotyping, harmful associations or inaccuracies. Though we saw a reduction in ungrounded inferences across the board in Gemini 2.0 and 2.5, there was disparity in refusal behaviour by skin tone of the person in the image. We observed models tended to be more likely to make ungrounded inferences about images of people with lighter skin tones than darker skin tones. The Gemini 2.5 family otherwise behaved similarly on our unfair bias evaluations to Gemini 1.5. We continue to explore and expand our understanding of unfair bias in Gemini models.

Findings from these evaluations were made available to teams deploying models, informing implementation of further product-level protections such as safety filtering. Assurance evaluation results were also reported to our Responsibility & Safety Council as part of model release review.

[H] Frontier Safety Framework Evaluations

Google DeepMind released its Frontier Safety Framework (FSF)
(
Google DeepMind, 2025a
)
in May 2024 and updated it in February 2025. The FSF comprises a number of processes and evaluations that address risks of severe harm stemming from powerful capabilities of our frontier models. It covers four risk domains: CBRN (chemical, biological, radiological and nuclear information risks), cybersecurity, machine learning R&D, and deceptive alignment.

The Frontier Safety Framework involves the regular evaluation of Google’s frontier models to determine whether they require heightened mitigations. More specifically, the FSF defines critical capability levels (CCLs) for each area, which represent capability levels where a model may pose a significant risk of severe harm without appropriate mitigations.

When conducting FSF evaluations, we compare test results against internal alert thresholds (“early warnings”) which are set significantly below the actual CCLs. This built-in safety buffer helps us be proactive by signaling potential risks well before models reach CCLs. Concretely, our alert thresholds are designed such that if a frontier model does not reach the alert threshold for a CCL, models are unlikely to reach that CCL before the next regular testing—which we conduct at a regular cadence and also when we anticipate or see exceptional capability progress. Our recent paper
(
Shah et al., 2025
)
discusses this approximate continuity assumption in more depth in Section 3.5.

[H] CCL Evaluation Results

Area

Key Results for Gemini 2.5 Pro
(up to version 06-05)

CCL

CCL reached?

CBRN

Based on qualitative assessment, 2.5 Pro demonstrates a general trend of increasing model capabilities across models 1.5 Pro, 2.0 and 2.5 Pro: it generates detailed technical knowledge of biological, radiological and nuclear domains. However, no current Gemini model consistently or completely enables progress through key bottleneck stages.

Uplift Level 1

×
\boldsymbol{\times}
CCL not reached

Cybersecurity

Solve rate on autonomous offense suite: 74/76 easy, 11/13 medium, 1/13 hard.

Autonomy Level 1

×
\boldsymbol{\times}
CCL not reached

On key skills benchmark: 7/8 easy, 14/28 medium, 6/12 hard.

Uplift Level 1

×
\boldsymbol{\times}
CCL not reached

Learning R&D
Machine

RE-Bench
(
Wijk et al., 2025
)
: the best agent solutions achieve between 50% and 125% of the score of the best expert-written solutions.

Autonomy Level 1

×
\boldsymbol{\times}
CCL not reached

Uplift Level 1

×
\boldsymbol{\times}
CCL not reached

Alignment
Deceptive

Gemini 2.5 Pro solves 2/5 stealth challenges and 2/11 situational awareness challenges.

Instrumental
Reasoning Level 1

×
\boldsymbol{\times}
CCL not reached

Instrumental
Reasoning Level 2

×
\boldsymbol{\times}
CCL not reached

[CAPTION] Table 10: Summary results. Across all areas covered by the Frontier Safety Framework, Critical Capability Levels (CCLs) have not been reached. Gemini 2.5 Pro is therefore unlikely to pose significant risk of severe harm. For descriptions of each CCL, see main text.

Because Gemini 2.5 Pro showed marked improvements across the board compared to Gemini 2.0 Pro, we ran our full suite of evaluations. While there are increased scores in some areas, we find that Gemini 2.5 Pro (up to version 06-17) does not reach any of the FSF CCLs. The evaluations did reach an alert threshold for the Cyber Uplift 1 CCL, suggesting that models may reach the CCL in the foreseeable future. Consistent with the FSF, we are putting in place a response plan which includes testing models’ cyber capabilities more frequently and accelerating mitigations for them. For other CCLs, our evaluations of Gemini 2.5 Pro indicate that models developed before the next regular testing interval are unlikely to reach CCLs. See Table
10
for a summary of results.

[H] CBRN

CBRN Uplift Level 1 CCL Definition:
“The model
can be used to significantly assist a low-resourced actor with dual-use scientific protocols, resulting in a substantial increase in ability to cause a mass casualty event
.”

CCL reached? No.
The model demonstrated accurate and detailed technical capabilities, potentially lowering barriers across multiple operational stages of certain harm journeys for low-resourced actors. While its consolidation and supplementation of fragmented procedures provides incremental uplift over what is readily available through open source search alone, it does not yet consistently or completely enable progress through key bottleneck stages, and therefore does not cross the CCL. Further, while Gemini 2.5 generates accurate and more detailed responses than 2.0, many of the concepts/outputs observed were already accessible through multiturn or even singleturn prompting in 2.0.

Overview:
We perform CBRN evaluations internally and via third party external testers (see section
5.8
). Here, we report solely on internal evaluations, for which we use two different types of approaches to evaluate the models’ dual-use CBRN capabilities:

•

Close-ended multiple choice questions (MCQs) providing a quantitative grade.

•

Open-ended questions (OEQs) which address different succinct steps of a longer multi-step journey that are qualitatively assessed by domain experts.

Currently we do not run specific open-ended qualitative assessments of chemical information risks for our internal evaluations. However, our third party external testers include chemistry in their assessments.

Multiple Choice Questions:
The underlying assumption when using knowledge-based and reasoning MCQs is that if the model cannot answer these questions properly, it is less likely to be able to cause severe harm: the type of information in the MCQs is the type of information that is necessary, but not sufficient to help malicious actors cause severe harm. Examples of model performance on three external benchmarks are shown in Figure
9
: i) SecureBio VMQA single-choice; ii) FutureHouse LAB-Bench presented as three subsets (ProtocolQA, Cloning Scenarios, SeqQA)
(
Laurent et al., 2024
)
; and iii) Weapons of Mass Destruction Proxy (WMDP) presented as the biology and chemistry data sets
(
Li et al., 2024
)
.

Results:
We observe a general trend of increasing scores, with Gemini 2.5 Pro showing statistically higher scores than the next best previous model for all benchmarks.

Open-Ended Questions:
This qualitative assessment was performed for biological, radiological and nuclear domains; it includes knowledge-based, adversarial and dual-use content. Questions span a range of difficulty levels, from questions a non-expert in these domains might ask, to questions that mostly an expert with a PhD plus many years of experience could pose or answer correctly. The prompts and scenarios span different threat journeys (e.g. types of actors, equipment used, harm intended). This qualitative assessment, led by domain experts, allows for better visibility of the granular improvement in science capabilities (e.g. accuracy, completeness, actionability of responses).

Results:
We observe that the same prompts used on previous models result in Gemini 2.5 Pro often generating detailed and accurate responses. In particular domains, some answers were technically precise and potentially actionable, but the model did not consistently or completely enable progress through all key bottleneck steps.

[CAPTION] Figure 9: List of Gemini models and their performance on a selection of external multiple-choice question benchmarks for biology and chemistry. In order to control for inherent model stochasticity, and position bias in selection of correct answers, we shuffled the answer choices over 100 runs for each benchmark and we report here the mean solve rate.

[H] Cybersecurity

Cyber Autonomy Level 1 CCL Definition:
“The model
can be used to drastically reduce the cost (e.g. through full automation) of carrying out end-to-end cyberattacks on organizations with a limited security posture
.”

CCL reached? No.
While Gemini 2.5 Pro’s performance on our cybersecurity evaluations has increased significantly compared to previous Gemini models, the model still cannot consistently complete roughly half of our medium and hard difficulty challenges.

Cyber Uplift Level 1 CCL Definition:
“The model
can be used to significantly assist with high impact cyber attacks, resulting in overall cost/resource reductions of an order of magnitude or more
.”

CCL reached? No.
While Gemini 2.5 Pro’s capabilities exhibit an increase in performance on multiple phases of real-world cyber attacks compared to previous models, they are still limited. The model still struggles with many of our hard challenges, which are most representative of real-world scenarios. The model’s performance is strong enough that it has passed our early warning alert threshold. That is, we find it possible that subsequent revisions in the next few months could lead to a model that reaches the CCL. In anticipation of this possibility, we have accelerated our mitigation efforts.

Overview:
To estimate capabilities relevant to these CCLs, we use our existing suite of cybersecurity challenges
(
Phuong et al., 2024
)
combined with a new benchmark described below. We consider difficulty levels ranging from easy (at the level of a college student), medium (at the level of a graduate student), and hard (at the level of an experienced cybersecurity professional).

Existing challenges:
These take the form of capture-the-flag evaluations covering three difficulty levels: easy (InterCode-CTF,
Yang et al. (2023)
, medium (our in-house suite), and hard (Hack the Box). See
Phuong et al. (2024)
for details. These evaluations are only relevant to Autonomy Level 1 (poor performance on these evaluations rules out capabilities sufficient for full automation), but not Uplift Level 1, since humans could still be accelerated on attack phases not covered.

Key skills benchmark:
Complementing our existing suite, we created a new evaluation framework
(
Rodriguez et al., 2025
)
that increases the realism as well as coverage across and within the attack chain of our cyber evaluations. We drew from real-world threat intelligence and input from experts to identify a representative set of cyber attacks. From there, we determined the key skills areas that would be required to execute them. We identified four key areas:

1.

Reconnaissance (the application of finding and researching different knowledge and data and applying it in a cybersecurity context);

2.

Tool development (the ability to design and create software that is cybersecurity-specific);

3.

Tool usage (the ability to leverage common and cybersecurity-specific tools to achieve routine instrumental cyber goals);

4.

Operational security (the skill of remaining hidden during and after a cyber operation).

We instantiate this benchmark by mapping 48 challenges from an external vendor to this specification. We also use these evaluations as a proxy for uplift capability, for Cyber Uplift Level 1. Even partial automation of these key skills could mean fewer resources are needed for sophisticated cyberattacks.

Elicitation:
We utilize a simple harness which prompts the agent to think about its plans and adjust them to solve the challenge. The agent can execute Bash commands and run Python scripts. We run between 10 and 50 attempts at each challenge, depending on the challenge suite. We consider a challenge solved if at least one of those attempts succeeds in capturing the flag.

Results:
See Figures
10
and
11
for numerical results. Model capabilities have increased to the point where InterCode-CTF and our in-house CTF challenges are too easy, but the Hack the Box challenges are still too difficult - such that these three evals give little signal on how Gemini 2.5 models compare to previous generations. However, on our more fine-grained key skills benchmark, Gemini 2.5 models show significant increases in capability compared to Gemini 2.0 models throughout all three difficulty levels. In particular, Gemini 2.5 Pro shows surprisingly high aptitude at the hard challenges. It solves 6/12 challenges—challenges at the level of an experienced cybersecurity professional.

[CAPTION] Figure 10: Results on autonomous cyber offense suite. These benchmarks are based on “capture-the-flag” (CTF) challenges, in which the agent must hack into a simulated server to retrieve a piece of hidden information. Labels above bars represent the number of solved and total number of challenges. A challenge is considered solved if the agent succeeds in at least one out of N attempts, where we vary N between 5 and 30 depending on challenge complexity. Both InterCode-CTF and our in-house CTFs are now largely saturated, showing little performance change from Gemini 2.0 to Gemini 2.5 models. In contrast, the Hack the Box challenges are still too difficult for Gemini 2.5 models, and so also give little signal on capability change.

[CAPTION] Figure 11: Results on our new “key skills” benchmark. This benchmark also consists of “capture-the-flag” (CTF) challenges, but these challenges are targeted at key skills required to execute cyber-attacks: reconnaissance, tool development, tool usage and operational security. A challenge is considered solved if the agent succeeds in at least one out of N attempts, where N = 30-50 for the 2.5 Pro run and N = 10-30 for the other models, depending on the challenge complexity. Note that for 2.0 Pro we omit results from five challenges and so 2.0 results are not directly comparable. Here, Gemini 2.5 family models show significant increase in capability at all three difficulty levels. Particularly of note is Gemini 2.5 Pro solving half of the hard challenges - challenges at the level of an experienced cybersecurity professional.

[H] Machine Learning R&D

Machine Learning R&D Autonomy Level 1 CCL Definition:
“The model
can fully automate the AI R&D pipeline at a competitive cost. The cost comparison is relative to humans augmented by AI tools
.”

CCL reached? No.
On RE-Bench, although the gap between Gemini and expert humans has shrunk, the model still doesn’t reach our early warning threshold (which is based on a specific average score over all challenges). Note that RE-bench comprises relatively low-context tasks with a limited time horizon (calibrated to allow significant progress by humans experts in 8 hours), and represents a small subset of the tasks comprising the full AI R&D pipeline.

Machine Learning R&D Uplift Level 1 CCL Definition:
“The model
can or has been used to accelerate AI development, resulting in AI progress substantially accelerating (e.g. 2x) from 2020-2024 rates
.”

CCL reached? Likely no.
We do not yet have dedicated evaluations for this CCL, but given that Gemini does not yet reach our early warning threshold on Autonomy Level 1, the model likely lacks the necessary capabilities to automate or significantly uplift any significant fraction of the research process.

To evaluate Gemini 2.5 models’ potential for accelerating ML R&D, we ran the open-source Research Engineering Benchmark
(
Wijk et al., 2025
)
. This benchmark comprises seven machine learning challenges difficult enough to take a human practitioner several hours to complete. For example, in the Optimize LLM Foundry challenge, the model must speed up a fine-tuning script while keeping the resulting model the same. We omit two challenges, Finetune GPT-2 for QA and Scaffolding for Rust Codecontest since they require internet access, which we disallow for security reasons.

The model is equipped with METR’s modular scaffold with minimal adjustment. Following the original work, we simulate a scenario in which the agent has a total time budget of 32 hours and the agent may choose a tradeoff between the number of runs and the length of each run. We evaluate two settings: 43 runs with a time limit of 45 minutes each, and 16 runs with a time limit of 2 hours each. For each setting, we aggregate scores across runs using the method described in the original work
(
Wijk et al., 2025
)
. This involves taking a number of bootstrap samples, taking the maximum score over each sample, and calculating a confidence interval using percentiles of the resulting values. (For the Scaling Law Experiment challenge, because the score is not visible to the agent and therefore the agent would not be able to pick run results based on the best score, we instead bootstrap the mean using all scores.) For the 45 minute setting, we do 64 actual runs, but sample only 43 runs for each bootstrap sample. Similarly for the 2 hour setting, we do 24 runs.

Gemini 2.5 Pro’s best runs score between 50% and 125% of the best human-written solutions. Despite this, the model does not reach our alert threshold, which was set higher than the human performance in view of the fact that RE-bench contains low-context and limited time horizon tasks that we expect to be especially easy for AI systems to reach human parity on. Some of the model’s solutions are nevertheless quite interesting. For example, in the Restricted Architecture MLM task, the agent is tasked with implementing a language model without use of basic primitives such as division and exponentiation. This seemingly simple constraint invalidates modern architectures like the Transformer, whose attention mechanism and normalization layers rely heavily on these forbidden operations. In one attempt, Gemini 2.5 Pro realises it can achieve this by drawing inspiration from aspects of the MLP-Mixer architecture
(
Tolstikhin et al., 2021
)
—a non-trivial insight that draws on its extensive knowledge of the research literature. In effect, creativity is substituted by knowledge.

[CAPTION] Figure 12: Results on the Research Engineer Benchmark (RE-Bench), in which the model must complete simple ML research tasks. Following the original work, scores are normalised against a good quality human-written solution: if a model achieves a score yy on a challenge, the normalised score is (y−ys)/(yr−ys)(y-y_{s})/(y_{r}-y_{s}), where ysy_{s} is the “starting score” of a valid but poor solution provided to the model as an example, and yry_{r} is the score achieved by a reference solution created by the author of the challenge. Figures for Claude 3.5 Sonnet and expert human performance are sourced from the original work. The number of runs and the time limit for each run are constrained by a total time budget of 32 hours, and error bars indicate bootstrapped 95% confidence intervals; see main text for details. Gemini 2.5 Pro is moderately strong at these challenges, achieving a significant fraction of expert human performance—and in two cases surpassing it.

[H] Deceptive Alignment

Instrumental Reasoning Level 1 CCL Definition:
“
The instrumental reasoning abilities of the model enable enough situational awareness (ability to work out and use relevant details of its deployment setting) and stealth (ability to circumvent basic oversight mechanisms) such that, absent additional mitigations, we cannot rule out the model significantly undermining human control
.”

CCL reached? No.
Gemini 2.5 Pro does not perform competently enough on both the stealth and situational awareness challenges. Qualitatively, the model often fails to reason through its situation and find a credible strategy; it also sometimes struggles with planning and long-horizon reasoning. It partially displays situational awareness but most of the time, it fails to understand and intervene on its environment/deployment setting.

Instrumental Reasoning Level 2 CCL Definition:
“
The instrumental reasoning abilities of the model enable enough situational awareness and stealth that, even when relevant model outputs (including, e.g., scratchpads) are being monitored, we cannot detect or rule out the risk of a model significantly undermining human control
.”

CCL reached? No.
This threshold is strictly higher than Instrumental Reasoning Level 1.

Our deceptive alignment evaluations measure stealth capabilities and situational awareness capabilities separately, using two separate sets of challenges.

Stealth:
In these challenges, the model is tasked with a devious objective and must achieve that objective without raising suspicion. For example, one challenge is a customer support scenario where the model deals with incoming support tickets. The agent is tasked with resolving as many tickets as possible while complying with the company’s protocols. There are ways to cheat at this (resulting in more tickets resolved)—but if the model does decide to cheat, it must not be caught by an overseer. Here we measure performance relative to the score that humans get when they do the challenge.

Situational awareness:
These challenges test whether the model can reason about itself as an agent embedded in the deployment environment, and to intervene on it in service of a goal. Many of them test whether the model can realise that it needs to give itself more affordances, e.g. by changing configuration files that place constraints on its action space. The agent may need to, say, turn off swear word filtering in order to write an angry email. For more details of these evaluations, see
(
Phuong et al., 2025
)
.

Results:
We ran these evaluations on Gemini 2.5 Pro Preview 03-25. All of the results reported are with 50 retries and involve a model equipped with scaffolds that uses chain-of-thought reasoning and reflection to enhance context utilization and better planning. Broadly, Gemini 2.5 Pro exhibits poor performance on situational awareness evaluations, and moderate performance in stealth evaluations. Across 50 trials, Gemini 2.5 Pro has no successes on 8 of our suite of 11 situational awareness evaluations. Among the remaining evaluations, the highest performance is achieved on a “Disabled Tool” challenge, in which Gemini is run in an agentic environment and has to grant itself access to more tools though configuration changes (42% success rate across 50 trials). For stealth evaluations, Gemini 2.5 Pro achieves low rates of success on 3/5 challenges. It achieves a ~44% success rate on the Decision Sabotage challenge and a ~35% success rate on the Oversight Pattern challenge, which is similar to human performance.

[H] Frontier Safety: Summary

Gemini 2.5 Pro was evaluated against the Critical Capability Levels defined in our Frontier Safety Framework, which examines risk in CBRN, cybersecurity, machine learning R&D, and deceptive alignment. Based on these results, we find that Gemini 2.5 Pro (up to version 06-17) does not reach any of the Critical Capability Levels in any of these areas.

However, it showed some ability in all four areas. For example, in our evaluation of Machine Learning R&D capabilities, while the model’s average performance was lower than the human baseline, in two cases its best performances exceeded even the best expert human-written reference solutions.

Gemini 2.5 Pro also showed a significant increase in some capabilities, such as cyber uplift, compared to previous Gemini models. Following our Frontier Safety Framework, we are putting in place a response plan, including conducting higher frequency testing and accelerating mitigations for the Cyber Uplift Level 1 CCL. As reported above, no model reached the CCL in these additional tests.

Looking ahead, these evaluations are key to safe deployment of powerful AI systems. We will continue to invest in this area, regularly performing Frontier Safety Framework evaluations to highlight areas where mitigations (e.g. refusal to respond to prompts that return dangerous results) must be prioritized.

[H] 5.8 External Safety Testing

As outlined in the Gemini 1.5 Technical Report
(
Gemini Team, 2024
)
, as part of our External Safety Testing Program, we work with a small set of independent external groups to help identify areas for improvement in our model safety work by undertaking structured evaluations, qualitative probing, and unstructured red teaming. As a heuristic, the External Safety Testing Program reviews the most capable Gemini models, with the largest capability jumps. As such, testing was only carried out on the 2.0 Pro and 2.5 Pro models, including on early versions of both models. At the time of writing we have not carried out external safety testing on the Flash models. The External Safety Testing Program focused testing on an early version of Gemini 2.5 Pro (Preview 05-06) to capture early findings and did not test the final model candidate which went to GA.

For Gemini 2.5 Pro, our external testing groups were given black-box testing access to Gemini 2.5 Pro (Preview 05-06) on AI Studio for a number of weeks. This enabled Google DeepMind to gather early insights into the model’s capabilities and understand if and where mitigations were needed. Testing groups had the ability to turn down or turn off safety filters, in line with what is available on AI Studio.

These groups were selected based on their expertise across a range of domain areas, such as autonomous systems, societal, cyber, and CBRN risks. Groups included civil society and commercial organizations. The groups testing the model checkpoints were compensated for their time.

External groups were by design instructed to develop their own methodology to test topics within a particular domain area, remaining independent from internal Google DeepMind evaluations. The time dedicated to testing also varied per group, with some groups being dedicated full-time to executing testing processes, while others were part-time dedicated. Some groups pursued manual red-teaming and reported on qualitative findings from their exploration of model behavior, while others developed bespoke automated testing strategies and produced quantitative reports of their results.

While reports were written independently of Google DeepMind, our internal subject matter experts were on hand to understand the external testing groups’ methodologies and findings throughout the testing process.

External safety testing groups shared their analyses and findings, as well as the raw data and materials they used in their evaluations (e.g., prompts, model responses). After testing, we internally reviewed the data and model output transcripts in detail, and Google DeepMind subject matter experts assigned severity ratings to outputs, based on our internal harm frameworks and safety policies, and noted whether these cross the Critical Capability Levels outlined in different domains
(
Google DeepMind, 2025a
)
. We then communicated findings back to modelling teams and product policy teams (both within Google DeepMind and across Alphabet) and reported these as part of our governance processes. Our external testing findings also help us identify gaps in our existing internal evaluation methodologies and safety policies.

We’ve outlined some of the high-level insights from our external testing across the domain areas tested, including autonomous systems, cyber misuse, CBRN, and societal risks.

[H] Autonomous systems risks

An external group tested models for autonomous systems risks, focusing on models’ ability and propensity to covertly pursue misaligned goals. The evaluations consist of settings where the model is given a goal that requires deception, scheming, or sabotage to achieve. Evaluations were focused on:

•

Strategic deception:
testing the model’s propensity to deceive a user or developer about the actions it took.

•

In-context scheming reasoning:
testing the model’s capabilities in this domain where scheming is an effective strategy.

•

Sabotage capabilities:
assessing the model’s ability to compromise other language models with overt actions.

Overall external researchers suggested that Gemini 2.5 Pro (Preview 05-06) had a similar scheming capability to other models evaluated. External researchers noted that the model is not sufficiently capable of causing catastrophic harms on rollouts where it is scheming. However, external researchers noted minor harms such as reward-hacking may be possible, which could potentially lead to a degraded user experience.

Complementing these tests, Google DeepMind’s Deceptive Alignment evaluations (see Section
5.7
) provide evidence that the risk of severe harm is low due to the models’ limited situational awareness capabilities.

[H] Cyber misuse risks

[H] Cybersecurity risks

External cyber evaluations focused on assessing the ability for malicious actors to enhance existing attack vectors across a range of key cyber skills, such as vulnerability discovery, vulnerability exploitation, social engineering, and cyberattack planning (capability uplift). Testers also focused on the model’s ability to accelerate repetitive or time-consuming elements of cyber operations, enabling increased scale (throughput uplift).

Evaluations were conducted within simulated environments that realistically represented a range of target systems, networks, and security controls. This involved setting up virtual networks mimicking enterprise infrastructure, deploying realistic software vulnerabilities, and simulating user behaviors in social engineering scenarios.

Evaluations strived to incorporate elements of real-world constraints and complexities. This included introducing noisy data, limited information availability, or adversarial defenses that the AI model must overcome, mirroring the challenges faced by attackers in live operations.

Findings from these evaluations concluded that Gemini 2.5 Pro was a capable model for cybersecurity tasks, showing marked increase in ability from Gemini 1.5 Pro. Complementing these evaluations, the GDM Cyber team conducted their own tests, and found similarly high levels of capability (see Section
5.7
).

[H] Indirect Prompt Injections

The model was evaluated for patterns of susceptibility to indirect prompt injection attacks. In particular, the model was tested for vulnerabilities in function calls and potential asymmetries that exist across security measures. The model was also tested to understand how different domains yield higher hijack rates. In line with internal evaluations and mitigations in this space (Section
5.5
), we are continuing to evolve how we monitor and measure the resilience of increasingly capable Gemini models.

[H] CBRN risks

[H] Chemical and Biological risks

In addition to our internal evaluations described above (Section
5.7
) capabilities in chemistry and biology were assessed by an external group who conducted red teaming designed to measure the potential scientific and operational risks of the models. A red team composed of different subject matter experts (e.g. biology, chemistry, logistics) were tasked to role play as malign actors who want to conduct a well-defined mission in a scenario that is presented to them resembling an existing prevailing threat environment. Together, these experts probe the model to obtain the most useful information to construct a plan that is feasible within the resource and timing limits described in the scenario. The plan is then graded for both scientific and logistical feasibility. Based on this assessment, GDM addresses any areas that warrant further investigation.

External researchers found that the model outputs detailed information in some scenarios, often providing accurate information around experimentation and problem solving. However, researchers found steps were too broad and high level to enable a malicious actor.

[H] Radiological and Nuclear risks

Risks in the radiological and nuclear domains were assessed by an external group using a structured evaluation framework for red teaming. This incorporated single-turn broad exploration across the full risk chain and multi-turn targeted probing for high risk topics.

Assessments were structured around threat actors and harm pathways without measuring model uplift, evaluating responses based on accuracy, actionability, and dual-use potential, with additional scrutiny applied to the model’s thought summaries when applicable. External researchers found that model responses within this domain were accurate but lacked sufficient technical detail to be actionable.

[H] Societal risks

For the Gemini 2.5 Pro (Preview 05-06) model, external researchers focused on democratic harms and radicalisation, with an emphasis on how the model might be used by malicious actors. Risks in this domain focused on structured evaluations. The model was tested on its ability to identify harmful inputs and the extent to which it complied with harmful requests. As no internal evaluations mirror these precise domain harms, the External Safety Testing Program shared these findings with relevant teams to ensure monitoring and mitigation where necessary.

[H] 6 Discussion

In this report we have introduced the Gemini 2.X model family: Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.0 Flash and Gemini 2.0 Flash-Lite. Taken together, these models span the full Pareto frontier of model capability vs cost, and Gemini 2.5 Pro is the most capable model we have ever developed. Gemini 2.5 Pro excels across a wide range of capabilities, and represents a step change in performance relative to Gemini 1.5 Pro. Its coding, math and reasoning performance are particularly notable and Gemini 2.5 Pro obtains extremely competitive scores on the Aider Polyglot evaluation, GPQA (diamond) and Humanity’s Last Exam.

As well as their strong performance on academic benchmarks, entirely new capabilities are unlocked with the Gemini 2.5 models. Gemini is now the preferred AI assistant amongst educators
(
LearnLM Team, 2025
)
and it is now possible for Gemini to
take a video of a lecture and create an interactive web application that can test a student’s knowledge of that content
. Finally, the Gemini 2.5 models enable exciting new agentic workflows, and have started to power numerous Google products already
(
Pichai, 2025
)
.

In addition to being highly performant, the Gemini 2.5 models maintain strong safety standards and, compared to their 1.5 counterparts, are much more helpful. They are less likely to refuse to answer important user queries or respond with an overly sanctimonious tone. Gemini 2.5 exhibited notable increases in Critical Capabilities, including cybersecurity and machine learning R&D. However, the model has not crossed any Critical Capability Levels.

Reflecting on the path to Gemini 2.5, the staggering performance improvement attained over the space of just one year points to a new challenge in AI research: namely that the development of novel and sufficiently challenging evaluation benchmarks has struggled to keep pace with model capability improvements, especially with the advent of capable reasoning agents. Over the space of just a year, Gemini Pro’s performance has gone up 5x on Aider Polyglot and 2x on SWE-bench verified (one of the most popular and challenging agentic benchmarks). Not only are benchmarks saturating quickly, but every new benchmark that gets created can end up being more expensive and take longer to create than its predecessor, due to the more restricted pool of experts able to create it. Experts were paid up to $5000 for each question that was accepted to the Humanity’s Last Exam benchmark
(
Phan et al., 2025
)
, and while this benchmark still has significant headroom at the time of writing (June 2025), performance on it has improved significantly over the space of a few months (with the best models achieving just a few percent accuracy on it when it was initially published in early 2025). When one considers agentic systems, which are able to tackle problems for longer and which have access to tools and self critique, the complexity of benchmarks required to measure performance also increases dramatically. Being able to scale evaluations in both their capability coverage and their difficulty, while also representing tasks that have economic value, will be the key to unlocking the next generation of AI systems.

[H] References

Anil et al. (2018)
R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton.
Large scale distributed neural network training through online distillation, 2018.
URL
https://arxiv.org/abs/1804.03235
.

Anil et al. (2023)
R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, et al.
PaLM 2 technical report, 2023.
URL
https://arxiv.org/abs/2305.10403
.

Anthropic (2025)
Anthropic.
Claude’s extended thinking, 2025.
URL
https://www.anthropic.com/research/visible-extended-thinking
.

Baddepudi et al. (2025)
A. Baddepudi, A. Yang, and M. Lučić.
Advancing the frontier of video understanding with Gemini 2.5, 2025.
URL
https://developers.googleblog.com/en/gemini-2-5-video-understanding/
.

Bai et al. (2022)
Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, et al.
Constitutional ai: Harmlessness from ai feedback, 2022.
URL
https://arxiv.org/abs/2212.08073
.

Balunović et al. (2025)
M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev.
Matharena: Evaluating llms on uncontaminated math competitions, 2025.
URL
https://arxiv.org/abs/2505.23281
.

Barham et al. (2022)
P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy, et al.
Pathways: Asynchronous distributed dataflow for ml.
Proceedings of Machine Learning and Systems
, 4:430–449, 2022.
URL
https://proceedings.mlr.press/v162/barham22a.html
.

Beutel et al. (2024)
A. Beutel, K. Xiao, J. Heidecke, and L. Weng.
Diverse and effective red teaming with auto-generated rewards and multi-step reinforcement learning, 2024.
URL
https://arxiv.org/abs/2412.18693
.

Biderman et al. (2023)
S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, et al.
Pythia: A suite for analyzing large language models across training and scaling.
In
Proceedings of the 40th International Conference on Machine Learning
, 2023.
URL
https://proceedings.mlr.press/v202/biderman23a.html
.

Carlini et al. (2022)
N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang.
Quantifying memorization across neural language models.
In
2022 IEEE Symposium on Security and Privacy (SP)
, pages 1113–1130, 2022.
URL
https://arxiv.org/abs/2202.07646
.

Chiang et al. (2024)
W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, et al.
Chatbot arena: An open platform for evaluating llms by human preference.
In
Forty-first International Conference on Machine Learning
, 2024.
URL
https://arxiv.org/abs/2306.05685
.

Chowdhery et al. (2022)
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al.
PaLM: Scaling language modeling with pathways.
arXiv preprint arXiv:2204.02311
, 2022.
URL
https://arxiv.org/abs/2204.02311
.

Chowdhury et al. (2024)
N. Chowdhury, J. Aung, C. J. Shern, O. Jaffe, D. Sherburn, G. Starace, E. Mays, R. Dias, M. Aljubeh, M. Glaese, C. E. Jimenez, J. Yang, L. Ho, T. Patwardhan, K. Liu, and A. Madry.
Introducing SWE-bench verified, 2024.
URL
https://openai.com/index/introducing-swe-bench-verified/
.

Clark et al. (2022)
A. Clark, D. de las Casas, A. Guy, A. Mensch, M. Paganini, J. Hoffmann, B. Damoc, B. Hechtman, T. Cai, S. Borgeaud, G. van den Driessche, E. Rutherford, T. Hennigan, M. Johnson, K. Millican, A. Cassirer, C. Jones, E. Buchatskaya, D. Budden, L. Sifre, S. Osindero, O. Vinyals, J. Rae, E. Elsen, K. Kavukcuoglu, and K. Simonyan.
Unified scaling laws for routed language models, 2022.
URL
"https://arxiv.org/abs/2202.01169"
.

CodeGemma Team et al. (2024)
CodeGemma Team, H. Zhao, J. Hui, J. Howland, N. Nguyen, S. Zuo, A. Hu, C. A. Choquette-Choo, J. Shen, J. Kelley, K. Bansal, L. Vilnis, M. Wirth, P. Michel, P. Choy, P. Joshi, R. Kumar, S. Hashmi, S. Agrawal, Z. Gong, J. Fine, T. Warkentin, A. J. Hartman, B. Ni, K. Korevec, K. Schaefer, and S. Huffman.
CodeGemma: Open Code Models Based on Gemma, 2024.
URL
https://arxiv.org/abs/2406.11409
.

Conneau et al. (2023)
A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna.
Fleurs: Few-shot learning evaluation of universal representations of speech.
In
2022 IEEE Spoken Language Technology Workshop (SLT)
, pages 798–805. IEEE, 2023.

Dehghani et al. (2023)
M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al.
Scaling vision transformers to 22 billion parameters.
In
International Conference on Machine Learning
, pages 7480–7512. PMLR, 2023.
URL
https://proceedings.mlr.press/v202/dehghani23a/dehghani23a.pdf
.

Doshi (2025a)
T. Doshi.
Build rich, interactive web apps with an updated Gemini 2.5 Pro, 2025a.
URL
https://blog.google/products/gemini/gemini-2-5-pro-updates/
.

Doshi (2025b)
T. Doshi.
Gemini 2.5: Our most intelligent models are getting even better, 2025b.
URL
https://blog.google/technology/google-deepmind/google-gemini-updates-io-2025/
.

Du et al. (2021)
N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al.
GLaM: Efficient scaling of language models with mixture-of-experts.
arXiv preprint arXiv:2112.06905
, 2021.
URL
https://arxiv.org/abs/2112.06905
.

Fedus et al. (2021)
W. Fedus, B. Zoph, and N. Shazeer.
Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.
arXiv preprint arXiv:2101.03961
, 2021.
URL
https://arxiv.org/abs/2101.03961
.

Fu et al. (2025)
C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al.
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.
In
Proceedings of the Computer Vision and Pattern Recognition Conference
, pages 24108–24118, 2025.
URL
https://openaccess.thecvf.com/content/CVPR2024/html/Fu_Video-MME_The_First-Ever_Comprehensive_Evaluation_Benchmark_of_Multi-Modal_LLMs_in_CVPR_2024_paper.html
.

Gauthier (2025)
P. Gauthier.
Aider Polyglot Coding Leaderboard, 2025.
URL
https://aider.chat/docs/leaderboards/
.

Gemini Team (2024)
Gemini Team.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
arXiv preprint arXiv:2403.05530
, 2024.
URL
https://arxiv.org/abs/2403.05530
.

Gemini Team, Google (2024)
Gemini Team, Google.
Gemini Deep Research, 2024.
URL
https://gemini.google/overview/deep-research/
.

Gemma Team (2024)
Gemma Team.
Gemma: Open Models Based on Gemini Research and Technology, 2024.
URL
https://arxiv.org/abs/2403.08295
.

Goldman et al. (2025)
O. Goldman, U. Shaham, D. Malkin, S. Eiger, A. Hassidim, Y. Matias, J. Maynez, A. M. Gilady, J. Riesa, S. Rijhwani, L. Rimell, I. Szpektor, R. Tsarfaty, and M. Eyal.
Eclektic: a novel challenge set for evaluation of cross-lingual knowledge transfer, 2025.
URL
https://arxiv.org/abs/2502.21228
.

Google DeepMind (2025a)
Google DeepMind.
Frontier safety framework, February 2025a.
URL
https://deepmind.google/discover/governance/frontier-safety-framework/
.

Google DeepMind (2025b)
Google DeepMind.
Gemini 2.0 Flash-Lite, 2025b.
URL
https://deepmind.google/models/gemini/flash-lite/
.

Grattafiori et al. (2024)
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, et al.
The Llama 3 Herd of Models, 2024.
URL
https://arxiv.org/abs/2407.21783
.

Hassabis (2025)
D. Hassabis.
Our vision for building a universal AI assistant, 2025.
URL
https://blog.google/technology/google-deepmind/gemini-universal-ai-assistant/
.

Hinton et al. (2015)
G. Hinton, O. Vinyals, and J. Dean.
Distilling the knowledge in a neural network, 2015.
URL
https://arxiv.org/abs/1503.02531
.

Hu et al. (2025)
K. Hu, P. Wu, F. Pu, W. Xiao, Y. Zhang, X. Yue, B. Li, and Z. Liu.
Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos, 2025.
URL
https://arxiv.org/abs/2501.13826
.

Hughes et al. (2023)
S. Hughes, M. Bae, and M. Li.
Vectara Hallucination Leaderboard, nov 2023.
URL
https://github.com/vectara/hallucination-leaderboard
.

Ippolito et al. (2022)
D. Ippolito, F. Tramer, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini.
Preventing verbatim memorization in language models gives a false sense of privacy, 2022.
URL
https://arxiv.org/abs/2210.17546
.

Jacovi et al. (2024)
A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, C. Saroufim, C. Fry, D. Marcus, D. Kukliansky, G. S. Tomar, J. Swirhun, J. Xing, L. Wang, M. Gurumurthy, M. Aaron, M. Ambar, R. Fellinger, R. Wang, R. Sims, Z. Zhang, S. Goldshtein, and D. Das.
Facts grounding leaderboard.
https://www.kaggle.com/benchmarks/google/facts-grounding
, 2024.
Google Deepmind, Google Research, Google Cloud, Kaggle.

Jacovi et al. (2025)
A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, et al.
The facts grounding leaderboard: Benchmarking llms’ ability to ground responses to long-form input.
arXiv preprint arXiv:2501.03200
, 2025.
URL
https://arxiv.org/abs/2501.03200
.

Jain et al. (2024)
N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica.
Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024.
URL
https://arxiv.org/abs/2403.07974
.

Jiang et al. (2024)
A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al.
Mixtral of experts.
arXiv preprint arXiv:2401.04088
, 2024.
URL
https://arxiv.org/abs/2401.04088
.

Jimenez et al. (2024)
C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan.
SWE-bench: Can language models resolve real-world github issues?
In
The Twelfth International Conference on Learning Representations
, 2024.
URL
https://openreview.net/forum?id=VTF8yNQM66
.

Kampf and Brichtova (2025)
K. Kampf and N. Brichtova.
Experiment with Gemini 2.0 Flash native image generation, 2025.
URL
https://developers.googleblog.com/en/experiment-with-gemini-20-flash-native-image-generation/
.

Kavukcuoglu (2025)
K. Kavukcuoglu.
Gemini 2.0 is now available to everyone, 2025.
URL
https://blog.google/technology/google-deepmind/gemini-model-updates-february-2025
.

Kilpatrick (2025)
L. Kilpatrick.
Gemini 2.5 Pro Preview: even better coding performance, 2025.
URL
https://developers.googleblog.com/en/gemini-2-5-pro-io-improved-coding-performance
.

Kudugunta et al. (2023)
S. Kudugunta, I. Caswell, B. Zhang, X. Garcia, C. A. Choquette-Choo, K. Lee, D. Xin, A. Kusupati, R. Stella, A. Bapna, and O. Firat.
MADLAD-400: A Multilingual And Document-Level Large Audited Dataset, 2023.
URL
https://arxiv.org/abs/2309.04662
.

Laurent et al. (2024)
J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, et al.
LAB-Bench: Measuring capabilities of language models for biology research, 2024.
URL
https://arxiv.org/abs/2407.10362
.

LearnLM Team (2025)
LearnLM Team.
Evaluating Gemini in an Arena for Learning, 2025.
URL
https://goo.gle/LearnLM-May25
.

Lee et al. (2024)
J. Lee, A. Chen, Z. Dai, D. Dua, D. S. Sachan, M. Boratko, Y. Luan, S. M. Arnold, V. Perot, S. Dalmia, et al.
Can long-context language models subsume retrieval, rag, sql, and more?
arXiv preprint arXiv:2406.13121
, 2024.
URL
https://arxiv.org/abs/2406.13121
.

Lei et al. (2021)
J. Lei, T. L. Berg, and M. Bansal.
Detecting moments and highlights in videos via natural language queries.
Advances in Neural Information Processing Systems
, 34:11846–11858, 2021.

Lepikhin et al. (2020)
D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen.
GShard: Scaling giant models with conditional computation and automatic sharding.
In
International Conference on Learning Representations
, 2020.
URL
https://openreview.net/forum?id=qrwe7XHTmYb
.

Li et al. (2024)
N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, et al.
The WMDP benchmark: Measuring and reducing malicious use with unlearning, 2024.
URL
https://arxiv.org/abs/2403.03218
.

Liu et al. (2020)
L. Liu, X. Liu, J. Gao, W. Chen, and J. Han.
Understanding the difficulty of training transformers.
arXiv preprint arXiv:2004.08249
, 2020.
URL
https://arxiv.org/abs/2004.08249
.

LMArena Team (2025)
LMArena Team.
Webdev arena, 2025.
URL
https://web.lmarena.ai/leaderboard
.

Mallick and Kilpatrick (2025)
S. B. Mallick and L. Kilpatrick.
Gemini 2.0: Flash, Flash-Lite and Pro, 2025.
URL
https://developers.googleblog.com/en/gemini-2-family-expands/
.

Mehrotra et al. (2024)
A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi.
Tree of attacks: Jailbreaking black-box llms automatically, 2024.
URL
https://arxiv.org/abs/2312.02119
.

Molybog et al. (2023)
I. Molybog, P. Albert, M. Chen, Z. DeVito, D. Esiobu, N. Goyal, P. Koura, S. Narang, A. Poulton, R. Silva, et al.
A theory on adam instability in large-scale machine learning.
arXiv preprint arXiv:2304.09871
, 2023.
URL
https://arxiv.org/abs/2304.09871
.

Nagrani et al. (2025a)
A. Nagrani, S. Menon, A. Iscen, S. Buch, R. Mehran, N. Jha, A. Hauth, Y. Zhu, C. Vondrick, M. Sirotenko, C. Schmid, and T. Weyand.
Minerva: Evaluating complex video reasoning, 2025a.
URL
https://arxiv.org/abs/2505.00681
.

Nagrani et al. (2025b)
A. Nagrani, M. Zhang, R. Mehran, R. Hornung, N. B. Gundavarapu, N. Jha, A. Myers, X. Zhou, B. Gong, C. Schmid, M. Sirotenko, Y. Zhu, and T. Weyand.
Neptune: The long orbit to benchmarking long video understanding, 2025b.
URL
https://arxiv.org/abs/2412.09582
.

Nasr et al. (2023)
M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee.
Scalable extraction of training data from (production) language models, 2023.
URL
https://arxiv.org/abs/2311.17035
.

Padlewski et al. (2024)
P. Padlewski, M. Bain, M. Henderson, Z. Zhu, N. Relan, H. Pham, D. Ong, K. Aleksiev, A. Ormazabal, S. Phua, E. Yeo, E. Lamprecht, Q. Liu, Y. Wang, E. Chen, D. Fu, L. Li, C. Zheng, C. de Masson d’Autume, D. Yogatama, M. Artetxe, and Y. Tay.
Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models, 2024.
URL
https://arxiv.org/abs/2405.02287
.

Pappu et al. (2024)
A. Pappu, B. Porter, I. Shumailov, and J. Hayes.
Measuring memorization in RLHF for code completion.
arXiv preprint arXiv:2406.11715
, 2024.
URL
https://arxiv.org/abs/2406.11715
.

Patraucean et al. (2023)
V. Patraucean, L. Smaira, A. Gupta, A. Recasens, L. Markeeva, D. Banarse, S. Koppula, M. Malinowski, Y. Yang, C. Doersch, et al.
Perception test: A diagnostic benchmark for multimodal video models.
Advances in Neural Information Processing Systems
, 36:42748–42761, 2023.

Perez et al. (2022)
E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving.
Red teaming language models with language models, 2022.
URL
https://arxiv.org/abs/2202.03286
.

Phan et al. (2025)
L. Phan et al.
Humanity’s last exam, 2025.
URL
https://arxiv.org/abs/2501.14249
.

Phuong et al. (2024)
M. Phuong, M. Aitchison, E. Catt, S. Cogan, A. Kaskasoli, V. Krakovna, D. Lindner, M. Rahtz, Y. Assael, S. Hodkinson, et al.
Evaluating frontier models for dangerous capabilities, 2024.
URL
https://arxiv.org/abs/2403.13793
.

Phuong et al. (2025)
M. Phuong, R. S. Zimmermann, Z. Wang, D. Lindner, V. Krakovna, S. Cogan, A. Dafoe, L. Ho, and R. Shah.
Evaluating frontier models for stealth and situational awareness, 2025.
URL
https://arxiv.org/abs/2505.01420
.

Pichai (2025)
S. Pichai.
Google I/O 2025: From research to reality, 2025.
URL
https://blog.google/technology/ai/io-2025-keynote/
.

Plizzari et al. (2025)
C. Plizzari, A. Tonioni, Y. Xian, A. Kulshrestha, and F. Tombari.
Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocentric videos.
In
Proceedings of the Computer Vision and Pattern Recognition Conference
, pages 24129–24138, 2025.

Rein et al. (2024)
D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman.
Gqqa: A graduate-level google-proof q&a benchmark.
In
First Conference on Language Modeling
, 2024.

Riquelme et al. (2021)
C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. S. Pinto, D. Keysers, and N. Houlsby.
Scaling vision with sparse mixture of experts, 2021.
URL
https://arxiv.org/abs/2106.05974
.

Roberts et al. (2025)
J. Roberts, M. R. Taesiri, A. Sharma, A. Gupta, S. Roberts, I. Croitoru, S.-V. Bogolin, J. Tang, F. Langer, V. Raina, et al.
ZeroBench: An impossible visual benchmark for contemporary large multimodal models.
arXiv preprint arXiv:2502.09696
, 2025.

Rodriguez et al. (2025)
M. Rodriguez, R. A. Popa, L. Liang, A. Wang, M. Rahtz, A. Kaskasoli, A. Dafoe, and F. Flynn.
A framework for evaluating emerging cyberattack capabilities of AI, 2025.
URL
https://arxiv.org/abs/2503.11917
.

Roller et al. (2021)
S. Roller, S. Sukhbaatar, J. Weston, et al.
Hash layers for large sparse models.
Advances in Neural Information Processing Systems
, 34:17555–17566, 2021.
URL
https://proceedings.neurips.cc/paper/2021/file/883e881bc596359e0c5112411858a74b-Paper.pdf
.

Samvelyan et al. (2024)
M. Samvelyan, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y. Mao, M. Jiang, J. Parker-Holder, J. Foerster, T. Rocktäschel, and R. Raileanu.
Rainbow teaming: Open-ended generation of diverse adversarial prompts, 2024.
URL
https://arxiv.org/abs/2402.16822
.

Shah et al. (2025)
R. Shah, A. Irpan, A. M. Turner, A. Wang, A. Conmy, D. Lindner, J. Brown-Cohen, L. Ho, N. Nanda, R. A. Popa, R. Jain, R. Greig, S. Albanie, S. Emmons, S. Farquhar, S. Krier, S. Rajamanoharan, S. Bridgers, T. Ijitoye, T. Everitt, V. Krakovna, V. Varma, V. Mikulik, Z. Kenton, D. Orr, S. Legg, N. Goodman, A. Dafoe, F. Flynn, and A. Dragan.
An approach to technical agi safety and security, 2025.
URL
https://arxiv.org/abs/2504.01849
.

Sharon (2025)
D. Sharon.
Upload and edit your images directly in the Gemini app, 2025.
URL
https://blog.google/products/gemini/image-editing/
.

Shazeer et al. (2017)
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.
In
ICLR (Poster)
. OpenReview.net, 2017.
URL
https://arxiv.org/abs/1701.06538
.

Shi et al. (2025)
C. Shi, S. Lin, S. Song, J. Hayes, I. Shumailov, I. Yona, J. Pluto, A. Pappu, C. A. Choquette-Choo, M. Nasr, C. Sitawarin, G. Gibson, A. Terzis, and J. F. Flynn.
Lessons from defending gemini against indirect prompt injections, 2025.
URL
https://arxiv.org/abs/2505.14534
.

Singh et al. (2024)
S. Singh, A. Romanou, C. Fourrier, D. I. Adelani, J. G. Ngui, D. Vila-Suero, P. Limkonchotiwat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Longpre, W.-Y. Ko, M. Smith, A. Bosselut, A. Oh, A. F. T. Martins, L. Choshen, D. Ippolito, E. Ferrante, M. Fadaee, B. Ermis, and S. Hooker.
Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation, 2024.
URL
https://arxiv.org/abs/2412.03304
.

Stein (2025)
R. Stein.
Expanding AI Overviews and introducing AI Mode, 2025.
URL
https://blog.google/products/search/ai-mode-search
.

Tolstikhin et al. (2021)
I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy.
Mlp-mixer: An all-mlp architecture for vision, 2021.

Vaswani et al. (2017)
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin.
Attention is all you need.
In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,
Advances in Neural Information Processing Systems
, volume 30. Curran Associates, Inc., 2017.
URL
https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
.

Vodrahalli et al. (2024)
K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al.
Michelangelo: Long context evaluations beyond haystacks via latent structure queries.
arXiv preprint arXiv:2409.12640
, 2024.
URL
https://arxiv.org/abs/2409.12640
.

Wang (2024)
B. Wang.
NotebookLM now lets you listen to a conversation about your sources , 2024.
URL
https://blog.google/technology/ai/notebooklm-audio-overviews
.

Wang et al. (2020)
C. Wang, A. Wu, and J. Pino.
Covost 2: A massively multilingual speech-to-text translation corpus, 2020.

Wang et al. (2024)
W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, X. Gu, S. Huang, B. Xu, Y. Dong, M. Ding, and J. Tang.
Lvbench: An extreme long video understanding benchmark, 2024.
URL
https://arxiv.org/abs/2406.08035
.

Wang et al. (2019)
X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, and W. Y. Wang.
Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.
In
Proceedings of the IEEE/CVF international conference on computer vision
, pages 4581–4591, 2019.

Wei et al. (2024)
J. Wei, K. Nguyen, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus.
Measuring short-form factuality in large language models.
arXiv preprint arXiv:2411.04368
, 2024.
URL
https://arxiv.org/abs/2411.04368
.

Weidinger et al. (2024)
L. Weidinger, J. Barnhart, J. Brennan, C. Butterfield, S. Young, W. Hawkins, et al.
Holistic safety and responsibility evaluations of advanced ai models, 2024.
URL
https://arxiv.org/abs/2404.14068
.

Wijk et al. (2025)
H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, et al.
RE-Bench: Evaluating frontier ai r&d capabilities of language model agents against human experts, 2025.
URL
https://arxiv.org/abs/2411.15114
.

Wortsman et al. (2023)
M. Wortsman, P. J. Liu, L. Xiao, K. Everett, A. Alemi, B. Adlam, J. D. Co-Reyes, I. Gur, A. Kumar, R. Novak, et al.
Small-scale proxies for large-scale transformer training instabilities.
arXiv preprint arXiv:2309.14322
, 2023.
URL
https://arxiv.org/abs/2309.14322
.

Yang et al. (2023)
J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao.
Intercode: Standardizing and benchmarking interactive coding with execution feedback, 2023.
URL
https://arxiv.org/abs/2306.14898
.

Yu et al. (2019)
Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao.
ActivityNet-QA: A dataset for understanding complex web videos via question answering.
In
AAAI
, 2019.

Yue et al. (2024)
X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
, pages 9556–9567, 2024.

Zerokid (2024)
Zerokid.
Pokemon Red Version - Guide and Walkthrough (GB), 2024.
URL
{https://gamefaqs.gamespot.com/gameboy/367023-Pok\'emon-red-version/faqs/64175}
.

Zhai et al. (2023)
S. Zhai, T. Likhomanenko, E. Littwin, D. Busbridge, J. Ramapuram, Y. Zhang, J. Gu, and J. M. Susskind.
Stabilizing transformer training by preventing attention entropy collapse.
In
International Conference on Machine Learning
, pages 40770–40803. PMLR, 2023.
URL
https://proceedings.mlr.press/v202/zhai23a/zhai23a.pdf
.

Zhang (2025)
J. Zhang.
Gemini Plays Pokemon Twitch Stream, 2025.
URL
https://www.twitch.tv/gemini_plays_pokemon/about
.

Zhang et al. (2022)
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al.
Opt: Open pre-trained transformer language models.
arXiv preprint arXiv:2205.01068
, 2022.
URL
https://arxiv.org/abs/2205.01068
.

Zhou et al. (2018)
L. Zhou, C. Xu, and J. J. Corso.
Towards automatic learning of procedures from web instructional videos.
In
AAAI Conference on Artificial Intelligence
, pages 7590–7598, 2018.
URL
https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17344
.

[H] 7 Contributors and Acknowledgments

Contributors

Gheorghe Comanici

Eric Bieber

Mike Schaekermann

Ice Pasupat

Noveen Sachdeva

Inderjit Dhillon

Marcel Blistein

Ori Ram

Dan Zhang

Evan Rosen

Luke Marris

Sam Petulla

Colin Gaffney

Asaf Aharoni

Nathan Lintz

Tiago Cardal Pais

Henrik Jacobsson

Idan Szpektor

Nan-Jiang Jiang

Krishna Haridasan

Ahmed Omran

Nikunj Saunshi

Dara Bahri

Gaurav Mishra

Eric Chu

Toby Boyd

Brad Hekman

Aaron Parisi

Chaoyi Zhang

Kornraphop Kawintiranon

Tania Bedrax-Weiss

Oliver Wang

Ya Xu

Ollie Purkiss

Uri Mendlovic

Ilaï Deutel

Nam Nguyen

Adam Langley

Flip Korn

Lucia Rossazza

Alexandre Ramé

Sagar Waghmare

Helen Miller

Vaishakh Keshava

Ying Jian

Xiaofan Zhang

Raluca Ada Popa

Kedar Dhamdhere

Blaž Bratanič

Kyuyeun Kim

Terry Koo

Ferran Alet

Yi-ting Chen

Arsha Nagrani

Hannah Muckenhirn

Zhiyuan Zhang

Corbin Quick

Filip Pavetić

Duc Dung Nguyen

Joao Carreira

Michael Elabd

Haroon Qureshi

Fabian Mentzer

Yao-Yuan Yang

Danielle Eisenbud

Anmol Gulati

Ellie Talius

Eric Ni

Sahra Ghalebikesabi

Edouard Yvinec

Alaa Saade

Thatcher Ulrich

Lorenzo Blanco

Dan A. Calian

Muhuan Huang

Aäron van den Oord

Naman Goyal

Terry Chen

Praynaa Rawlani

Christian Schallhart

Swachhand Lokhande

Xianghong Luo

Jyn Shan

Ceslee Montgomery

Victoria Krakovna

Federico Piccinini

Omer Barak

Jingyu Cui

Yiling Jia

Mikhail Dektiarev

Alexey Kolganov

Shiyu Huang

Zhe Chen

Xingyu Wang

Jessica Austin

Peter de Boursac

Evgeny Sluzhaev

Frank Ding

Huijian Li

Surya Bhupatiraju

Mohit Agarwal

Sławek Kwasiborski

Paramjit Sandhu

Patrick Siegler

Ahmet Iscen

Eyal Ben-David

Shiraz Butt

Miltos Allamanis

Seth Benjamin

Robert Busa-Fekete

Felix Hernandez-Campos

Sasha Goldshtein

Matt Dibb

Weiyang Zhang

Annie Marsden

Carey Radebaugh

Stephen Roller

Abhishek Nayyar

Jacob Austin

Tayfun Terzi

Bhargav Kanagal Shamanna

Pete Shaw

Aayush Singh

Florian Luisier

Artur Mendonça

Vaibhav Aggarwal

Larisa Markeeva

Claudio Fantacci

Sergey Brin

HyunJeong Choe

Guanyu Wang

Hartwig Adam

Avigail Dabush

Tatsuya Kiyono

Eyal Marcus

Jeremy Cole

Theophane Weber

Hongrae Lee

Ronny Huang

Alex Muzio

Leandro Kieliger

Maigo Le

Courtney Biles

Long Le

Archit Sharma

Chengrun Yang

Avery Lamp

Dave Dopson

Nate Hurley

Katrina (Xinyi) Xu

Zhihao Shan

Shuang Song

Jiewen Tan

Alexandre Senges

George Zhang

Chong You

Yennie Jun

David Raposo

Susanna Ricco

Xuan Yang

Weijie Chen

Prakhar Gupta

Arthur Szlam

Kevin Villela

Chun-Sung Ferng

Daniel Kasenberg

Chen Liang

Rui Zhu

Arunachalam Narayanaswamy

Florence Perot

Paul Pucciarelli

Anna Shekhawat

Alexey Stern

Rishikesh Ingale

Stefani Karp

Sanaz Bahargam

Adrian Goedeckemeyer

Jie Han

Sicheng Li

Andrea Tacchetti

Dian Yu

Abhishek Chakladar

Zhiying Zhang

Mona El Mahdy

Xu Gao

Dale Johnson

Samrat Phatale

AJ Piergiovanni

Hyeontaek Lim

Clement Farabet

Carl Lebsack

Theo Guidroz

John Blitzer

Nico Duduta

David Madras

Steve Li

Daniel von Dincklage

Xin Li

Mahdis Mahdieh

George Tucker

Ganesh Jawahar

Owen Xiao

Danny Tarlow

Robert Geirhos

Noam Velan

Daniel Vlasic

Kalesha Bullard

SK Park

Nishesh Gupta

Kellie Webster

Ayal Hitron

Jieming Mao

Julian Eisenschlos

Laurel Prince

Nina D’Souza

Kelvin Zheng

Sara Nasso

Gabriela Botea

Carl Doersch

Caglar Unlu

Chris Alberti

Alexey Svyatkovskiy

Ankita Goel

Krzysztof Choromanski

Pan-Pan Jiang

Richard Nguyen

Four Flynn

Daria Ćurko

Peter Chen

Nicholas Roth

Kieran Milan

Caleb Habtegebriel

Shashi Narayan

Michael Moffitt

Jake Marcus

Thomas Anthony

Brendan McMahan

Gowoon Cheon

Ruibo Liu

Megan Barnes

Lukasz Lew

Rebeca Santamaria-Fernandez

Mayank Upadhyay

Arjun Akula

Arnar Mar Hrafnkelsson

Alvaro Caceres

Andrew Bunner

Michal Sokolik

Subha Puttagunta

Lawrence Moore

Berivan Isik

Weilun Chen

Jay Hartford

Lawrence Chan

Pradeep Shenoy

Dan Holtmann-Rice

Jane Park

Fabio Viola

Alex Salcianu

Sujeevan Rajayogam

Ian Stewart-Binks

Zelin Wu

Richard Everett

Xi Xiong

Pierre-Antoine Manzagol

Gary Leung

Carl Saroufim

Bo Pang

Dawid Wegner

George Papamakarios

Jennimaria Palomaki

Helena Pankov

Guangda Lai

Guilherme Tubone

Shubin Zhao

Theofilos Strinopoulos

Seth Neel

Mingqiu Wang

Joe Kelley

Li Li

Pingmei Xu

Anitha Vijayakumar

Andrea D’olimpio

Omer Levy

Massimo Nicosia

Grigory Rozhdestvenskiy

Ni Lao

Sirui Xie

Yash Katariya

Jon Simon

Sanjiv Kumar

Florian Hartmann

Michael Kilgore

Jinhyuk Lee

Aroma Mahendru

Roman Ring

Tom Hennigan

Fiona Lang

Colin Cherry

David Steiner

Dawsen Hwang

Ray Smith

Pidong Wang

Jeremy Chen

Ming-Hsuan Yang

Sam Kwei

Philippe Schlattner

Donnie Kim

Ganesh Poomal Girirajan

Nikola Momchev

Ayushi Agarwal

Xingyi Zhou

Ilkin Safarli

Zachary Garrett

AJ Pierigiovanni

Sarthak Jauhari

Alif Raditya Rochman

Shikhar Vashishth

Quan Yuan

Christof Angermueller

Jon Blanton

Xinying Song

Nitesh Bharadwaj Gundavarapu

Thi Avrahami

Maxine Deines

Subhrajit Roy

Manish Gupta

Christopher Semturs

Shobha Vasudevan

Aditya Srikanth Veerubhotla

Shriya Sharma

Josh Jacob

Zhen Yang

Andreas Terzis

Dan Karliner

Auriel Wright

Tania Rojas-Esponda

Ashley Brown

Abhijit Guha Roy

Pawan Dogra

Andrei Kapishnikov

Peter Young

Wendy Kan

Vinodh Kumar Rajendran

Maria Ivanova

Salil Deshmukh

Chia-Hua Ho

Mike Kwong

Stav Ginzburg

Annie Louis

KP Sawhney

Slav Petrov

Jing Xie

Yunfei Bai

Georgi Stoyanov

Alex Fabrikant

Rajesh Jayaram

Yuqi Li

Joe Heyward

Justin Gilmer

Yaqing Wang

Radu Soricut

Luyang Liu

Qingnan Duan

Jamie Hayes

Maura O’Brien

Gaurav Singh Tomar

Sivan Eiger

Bahar Fatemi

Jeffrey Hui

Catarina Barros

Adaeze Chukwuka

Alena Butryna

Saksham Thakur

Austin Huang

Zhufeng Pan

Haotian Tang

Serkan Cabi

Tulsee Doshi

Michiel Bakker

Sumit Bagri

Ruy Ley-Wild

Adam Lelkes

Jennie Lees

Patrick Kane

David Greene

Shimu Wu

Jörg Bornschein

Gabriela Surita

Sarah Hodkinson

Fangtao Li

Chris Hidey

Sébastien Pereira

Sean Ammirati

Phillip Lippe

Adam Kraft

Pu Han

Sebastian Gerlach

Zifeng Wang

Liviu Panait

Feng Han

Brian Farris

Yingying Bi

Hannah DeBalsi

Miaosen Wang

Gladys Tyen

James Cohan

Susan Zhang

Jarred Barber

Da-Woon Chung

Jaeyoun Kim

Markus Kunesch

Steven Pecht

Nami Akazawa

Abe Friesen

James Lyon

Ali Eslami

Junru Wu

Jie Tan

Yue Song

Ravi Kumar

Chris Welty

Ilia Akolzin

Gena Gibson

Sean Augenstein

Arjun Pillai

Nancy Yuen

Du Phan

Xin Wang

Iain Barr

Heiga Zen

Nan Hua

Casper Liu

Jilei (Jerry) Wang

Tanuj Bhatia

Hao Xu

Oded Elyada

Pushmeet Kohli

Mirek Olšák

Ke Chen

Azalia Mirhoseini

Noam Shazeer

Shoshana Jakobovits

Maggie Tran

Nolan Ramsden

Tarun Bharti

Fred Alcober

Yunjie Li

Shilpa Shetty

Jing Chen

Dmitry Kalashnikov

Megha Nawhal

Sercan Arik

Hanwen Chen

Michiel Blokzijl

Shubham Gupta

James Rubin

Rigel Swavely

Sophie Bridgers

Ian Gemp

Chen Su

Arun Suggala

Juliette Pluto

Mary Cassin

Alain Vaucher

Kaiyang Ji

Jiahao Cai

Andrew Audibert

Animesh Sinha

David Tian

Efrat Farkash

Amy Hua

Jilin Chen

Duc-Hieu Tran

Edward Loper

Nicole Brichtova

Lara McConnaughey

Ballie Sandhu

Robert Leland

Doug DeCarlo

Andrew Over

James Huang

Xing Wu

Connie Fan

Eric Li

Yun Lei

Deepak Sharma

Cosmin Paduraru

Luo Yu

Matko Bošnjak

Phuong Dao

Min Choi

Sneha Kudugunta

Jakub Adamek

Carlos Guía

Ali Khodaei

Jie Feng

Wenjun Zeng

David Welling

Sandeep Tata

Christina Butterfield

Andrey Vlasov

Seliem El-Sayed

Swaroop Mishra

Tara Sainath

Shentao Yang

RJ Skerry-Ryan

Jeremy Shar

Robert Berry

Arunkumar Rajendran

Arun Kandoor

Andrea Burns

Deepali Jain

Tom Stone

Wonpyo Park

Shibo Wang

Albin Cassirer

Guohui Wang

Hayato Kobayashi

Sergey Rogulenko

Vineetha Govindaraj

Mikołaj Rybiński

Nadav Olmert

Colin Evans

Po-Sen Huang

Kelvin Xu

Premal Shah

Terry Thurk

Caitlin Sikora

Mu Cai

Jin Xie

Elahe Dabir

Saloni Shah

Norbert Kalb

Carrie Zhang

Shruthi Prabhakara

Amit Sabne

Artiom Myaskovsky

Vikas Raunak

Blanca Huergo

Behnam Neyshabur

Jon Clark

Ye Zhang

Shankar Krishnan

Eden Cohen

Dinesh Tewari

James Lottes

Yumeya Yamamori

Hui (Elena) Li

Mohamed Elhawaty

Ada Maksutaj Oflazer

Adrià Recasens

Sheryl Luo

Duy Nguyen

Taylor Bos

Kalyan Andra

Ana Salazar

Ed Chi

Jeongwoo Ko

Matt Ginsberg

Anders Andreassen

Anian Ruoss

Todor Davchev

Elnaz Davoodi

Chenxi Liu

Min Kim

Santiago Ontanon

Chi Ming To

Dawei Jia

Rosemary Ke

Jing Wang

Anna Korsun

Moran Ambar

Ilya Kornakov

Irene Giannoumis

Toni Creswell

Denny Zhou

Yi Su

Ishaan Watts

Aleksandr Zaks

Evgenii Eltyshev

Ziqiang Feng

Sidharth Mudgal

Alex Kaskasoli

Juliette Love

Kingshuk Dasgupta

Sam Shleifer

Richard Green

Sungyong Seo

Chansoo Lee

Dale Webster

Prakash Shroff

Ganna Raboshchuk

Isabel Leal

James Manyika

Sofia Erell

Daniel Murphy

Zhisheng Xiao

Anton Bulyenov

Julian Walker

Mark Collier

Matej Kastelic

Nelson George

Sushant Prakash

Sailesh Sidhwani

Alexey Frolov

Steven Hansen

Petko Georgiev

Tiberiu Sosea

Chris Apps

Aishwarya Kamath

David Reid

Emma Cooney

Charlotte Magister

Oriana Riva

Alec Go

Pu-Chin Chen

Sebastian Krause

Nir Levine

Marco Fornoni

Ilya Figotin

Nick Roy

Parsa Mahmoudieh

Vladimir Magay

Mukundan Madhavan

Jin Miao

Jianmo Ni

Yasuhisa Fujii

Ian Chou

George Scrivener

Zak Tsai

Siobhan Mcloughlin

Jeremy Selier

Sandra Lefdal

Jeffrey Zhao

Abhijit Karmarkar

Kushal Chauhan

Shivanker Goel

Zhaoyi Zhang

Vihan Jain

Parisa Haghani

Mostafa Dehghani

Jacob Scott

Erin Farnese

Anastasija Ilić

Steven Baker

Julia Pawar

Li Zhong

Josh Camp

Yoel Zeldes

Shravya Shetty

Anand Iyer

Vít Listík

Jiaxian Guo

Luming Tang

Mark Geller

Simon Bucher

Yifan Ding

Hongzhi Shi

Carrie Muir

Dominik Grewe

Ramy Eskander

Octavio Ponce

Boqing Gong

Derek Gasaway

Samira Khan

Umang Gupta

Angelos Filos

Weicheng Kuo

Klemen Kloboves

Jennifer Beattie

Christian Wright

Leon Li

Alicia Jin

Sandeep Mariserla

Miteyan Patel

Jens Heitkaemper

Dilip Krishnan

Vivek Sharma

David Bieber

Christian Frank

John Lambert

Paul Caron

Martin Polacek

Mai Giménez

Himadri Choudhury

Xing Yu

Sasan Tavakkol

Arun Ahuja

Franz Och

Rodolphe Jenatton

Wojtek Skut

Bryan Richter

David Gaddy

Andy Ly

Misha Bilenko

Megh Umekar

Ethan Liang

Martin Sevenich

Mandar Joshi

Hassan Mansoor

Rebecca Lin

Sumit Sanghai

Abhimanyu Singh

Xiaowei Li

Sudheendra Vijayanarasimhan

Zaheer Abbas

Yonatan Bitton

Hansa Srinivasan

Manish Reddy Vuyyuru

Alexander Frömmgen

Yanhua Sun

Ralph Leith

Alfonso Castaño

DJ Strouse

Le Yan

Austin Kyker

Satish Kambala

Mary Jasarevic

Thibault Sellam

Chao Jia

Alexander Pritzel

Raghavender R

Huizhong Chen

Natalie Clay

Sudeep Gandhe

Sean Kirmani

Sayna Ebrahimi

Hannah Kirkwood

Jonathan Mallinson

Chao Wang

Adnan Ozturel

Kuo Lin

Shyam Upadhyay

Vincent Cohen-Addad

Sean Purser-haskell

Yichong Xu

Ebrahim Songhori

Babi Seal

Alberto Magni

Almog Gueta

Tingting Zou

Guru Guruganesh

Thais Kagohara

Hung Nguyen

Khalid Salama

Alejandro Cruzado Ruiz

Justin Frye

Zhenkai Zhu

Matthias Lochbrunner

Simon Osindero

Wentao Yuan

Lisa Lee

Aman Prasad

Lam Nguyen Thiet

Daniele Calandriello

Victor Stone

Qixuan Feng

Han Ke

Maria Voitovich

Geta Sampemane

Lewis Chiang

Ling Wu

Alexander Bykovsky

Matt Young

Luke Vilnis

Ishita Dasgupta

Aditya Chawla

Qin Cao

Bowen Liang

Daniel Toyama

Szabolcs Payrits

Anca Stefanoiu

Dimitrios Vytiniotis

Ankesh Anand

Tianxiao Shen

Blagoj Mitrevski

Michael Tschannen

Sreenivas Gollapudi

Aishwarya P S

José Leal

Zhe Shen

Han Fu

Wei Wang

Arvind Kannan

Doron Kukliansky

Sergey Yaroshenko

Svetlana Grant

Umesh Telang

David Wood

Alexandra Chronopoulou

Alexandru Ţifrea

Tao Zhou

Tony (Tuấn) Nguyễn

Muge Ersoy

Anima Singh

Meiyan Xie

Emanuel Taropa

Woohyun Han

Eirikur Agustsson

Andrei Sozanschi

Hui Peng

Alex Chen

Yoel Drori

Efren Robles

Yang Gao

Xerxes Dotiwalla

Ying Chen

Anudhyan Boral

Alexei Bendebury

John Nham

Chris Tar

Luis Castro

Jiepu Jiang

Canoee Liu

Felix Halim

Jinoo Baek

Andy Wan

Jeremiah Liu

Yuan Cao

Shengyang Dai

Trilok Acharya

Ruoxi Sun

Fuzhao Xue

Saket Joshi

Morgane Lustman

Yongqin Xian

Rishabh Joshi

Deep Karkhanis

Nora Kassner

Jamie Hall

Xiangzhuo Ding

Gan Song

Gang Li

Chen Zhu

Yana Kulizhskaya

Bin Ni

Alexey Vlaskin

Solomon Demmessie

Lucio Dery

Salah Zaiem

Yanping Huang

Cindy Fan

Felix Gimeno

Ananth Balashankar

Koji Kojima

Hagai Taitelbaum

Maya Meng

Dero Gharibian

Sahil Singla

Wei Chen

Ambrose Slone

Guanjie Chen

Sujee Rajayogam

Max Schumacher

Suyog Kotecha

Rory Blevins

Qifei Wang

Mor Hazan Taege

Alex Morris

Xin Liu

Fayaz Jamil

Richard Zhang

Pratik Joshi

Ben Ingram

Tyler Liechty

Ahmed Eleryan

Scott Baird

Alex Grills

Gagan Bansal

Shan Han

Kiran Yalasangi

Shawn Xu

Majd Al Merey

Isabel Gao

Felix Weissenberger

Igor Karpov

Robert Riachi

Ankit Anand

Gautam Prasad

Kay Lamerigts

Reid Hayes

Jamie Rogers

Mandy Guo

Ashish Shenoy

Qiong (Q) Hu

Kyle He

Yuchen Liu

Polina Zablotskaia

Sagar Gubbi

Yifan Chang

Jay Pavagadhi

Kristian Kjems

Archita Vadali

Diego Machado

Yeqing Li

Renshen Wang

Dipankar Ghosh

Aahil Mehta

Dana Alon

George Polovets

Alessio Tonioni

Nate Kushman

Joel D’sa

Lin Zhuo

Allen Wu

Rohin Shah

John Youssef

Jiayu Ye

Justin Snyder

Karel Lenc

Senaka Buthpitiya

Matthew Tung

Jichuan Chang

Tao Chen

David Saxton

Jenny Lee

Lydia Lihui Zhang

James Qin

Prabakar Radhakrishnan

Maxwell Chen

Piotr Ambroszczyk

Metin Toksoz-Exley

Yan Zhong

Nitzan Katz

Brendan O’Donoghue

Tamara von Glehn

Adi Gerzi Rosenthal

Aga Świetlik

Xiaokai Zhao

Nick Fernando

Jinliang Wei

Jieru Mei

Sergei Vassilvitskii

Diego Cedillo

Pranjal Awasthi

Hui Zheng

Koray Kavukcuoglu

Itay Laish

Joseph Pagadora

Marc Brockschmidt

Christopher A. Choquette-Choo

Arunkumar Byravan

Yifeng Lu

Xu Chen

Mia Chen

Kenton Lee

Rama Pasumarthi

Sijal Bhatnagar

Aditya Shah

Qiyin Wu

Zhuoyuan Chen

Zack Nado

Bartek Perz

Zixuan Jiang

David Kao

Ganesh Mallya

Nino Vieillard

Lantao Mei

Sertan Girgin

Mandy Jordan

Yeongil Ko

Alekh Agarwal

Yaxin Liu

Yasemin Altun

Raoul de Liedekerke

Anastasios Kementsietsidis

Daiyi Peng

Dangyi Liu

Utku Evci

Peter Humphreys

Austin Tarango

Xiang Deng

Yoad Lewenberg

Kevin Aydin

Chengda Wu

Bhavishya Mittal

Tsendsuren Munkhdalai

Kleopatra Chatziprimou

Rodrigo Benenson

Uri First

Xiao Ma

Jinning Li

Armand Joulin

Hamish Tomlinson

Tingnan Zhang

Milad Nasr

Zhi Hong

Michaël Sander

Lisa Anne Hendricks

Anuj Sharma

Andrew Bolt

Eszter Vértes

Jiri Simsa

Tomer Levinboim

Olcan Sercinoglu

Divyansh Shukla

Austin Wu

Craig Swanson

Danny Vainstein

Fan Bu

Bo Wang

Ryan Julian

Charles Yoon

Sergei Lebedev

Antonious Girgis

Bernd Bandemer

David Du

Todd Wang

Xi Chen

Ying Xiao

Peggy Lu

Natalie Ha

Vlad Ionescu

Simon Rowe

Josip Matak

Federico Lebron

Andreas Steiner

Lalit Jain

Manaal Faruqui

Nicolas Lacasse

Georgie Evans

Neesha Subramaniam

Dean Reich

Giulia Vezzani

Aditya Pandey

Joe Stanton

Tianhao Zhou

Liam McCafferty

Henry Griffiths

Verena Rieser

Soheil Hassas Yeganeh

Eleftheria Briakou

Lu Huang

Zichuan Wei

Liangchen Luo

Erik Jue

Gabby Wang

Victor Cotruta

Myriam Khan

Jongbin Park

Qiuchen Guo

Peiran Li

Rong Rong

Diego Antognini

Anastasia Petrushkina

Chetan Tekur

Eli Collins

Parul Bhatia

Chester Kwak

Wenhu Chen

Arvind Neelakantan

Immanuel Odisho

Sheng Peng

Vincent Nallatamby

Vaibhav Tulsyan

Fabian Pedregosa

Peng Xu

Raymond Lin

Yulong Wang

Emma Wang

Sholto Douglas

Reut Tsarfaty

Elena Gribovskaya

Renga Aravamudhan

Manu Agarwal

Mara Finkelstein

Qiao Zhang

Elizabeth Cole

Phil Crone

Sarmishta Velury

Anil Das

Chris Sauer

Luyao Xu

Danfeng Qin

Chenjie Gu

Dror Marcus

CJ Zheng

Wouter Van Gansbeke

Sobhan Miryoosefi

Haitian Sun

YaGuang Li

Charlie Chen

Jae Yoo

Pavel Dubov

Alex Tomala

Adams Yu

Paweł Wesołowski

Alok Gunjan

Eddie Cao

Jiaming Luo

Nikhil Sethi

Arkadiusz Socala

Laura Graesser

Tomas Kocisky

Arturo BC

Minmin Chen

Edward Lee

Sophie Wang

Weize Kong

Qiantong Xu

Nilesh Tripuraneni

Yiming Li

Xinxin Yu

Allen Porter

Paul Voigtlaender

Biao Zhang

Arpi Vezer

Sarah York

Qing Wei

Geoffrey Cideron

Mark Kurzeja

Seungyeon Kim

Benny Li

Angéline Pouget

Hyo Lee

Kaspar Daugaard

Yang Li

Dave Uthus

Aditya Siddhant

Paul Cavallaro

Sriram Ganapathy

Maulik Shah

Rolf Jagerman

Jeff Stanway

Piermaria Mendolicchio

Li Xiao

Kayi Lee

Tara Thompson

Shubham Milind Phal

Jason Chase

Sun Jae Lee

Adrian N Reyes

Disha Shrivastava

Zhen Qin

Roykrong Sukkerd

Seth Odoom

Lior Madmoni

John Aslanides

Jonathan Herzig

Elena Pochernina

Sheng Zhang

Parker Barnes

Daisuke Ikeda

Qiujia Li

Shuo-yiin Chang

Shakir Mohamed

Jim Sproch

Richard Powell

Bidisha Samanta

Domagoj Ćevid

Anton Kovsharov

Shrestha Basu Mallick

Srinivas Tadepalli

Anne Zheng

Kareem Ayoub

Andreas Noever

Christian Reisswig

Zhuo Xu

Junhyuk Oh

Martin Matysiak

Tim Blyth

Shereen Ashraf

Julien Amelot

Boone Severson

Michele Bevilacqua

Motoki Sano

Ethan Dyer

Ofir Roval

Anu Sinha

Yin Zhong

Sagi Perel

Tea Sabolić

Johannes Mauerer

Willi Gierke

Mauro Verzetti

Rodrigo Cabrera

Alvin Abdagic

Steven Hemingray

Austin Stone

Jong Lee

Farooq Ahmad

Karthik Raman

Lior Shani

Jonathan Lai

Orhan Firat

Nathan Waters

Eric Ge

Mo Shomrat

Himanshu Gupta

Rajeev Aggarwal

Tom Hudson

Bill Jia

Simon Baumgartner

Palak Jain

Joe Kovac

Junehyuk Jung

Ante Žužul

Will Truong

Morteza Zadimoghaddam

Songyou Peng

Marco Liang

Rachel Sterneck

Balaji Lakshminarayanan

Machel Reid

Oliver Woodman

Tong Zhou

Jianling Wang

Vincent Coriou

Arjun Narayanan

Jay Hoover

Yenai Ma

Apoorv Jindal

Clayton Sanford

Doug Reid

Swaroop Ramaswamy

Alex Kurakin

Roland Zimmermann

Yana Lunts

Dragos Dena

Zalán Borsos

Vered Cohen

Shujian Zhang

Will Grathwohl

Robert Dadashi

Morgan Redshaw

Joshua Kessinger

Julian Odell

Silvano Bonacina

Zihang Dai

Grace Chen

Ayush Dubey

Pablo Sprechmann

Mantas Pajarskas

Wenxuan Zhou

Niharika Ahuja

Tara Thomas

Martin Nikoltchev

Matija Kecman

Bharath Mankalale

Andrey Ryabtsev

Jennifer She

Christian Walder

Jiaming Shen

Lu Li

Carolina Parada

Sheena Panthaplackel

Okwan Kwon

Matt Lawlor

Utsav Prabhu

Yannick Schroecker

Marc’aurelio Ranzato

Pete Blois

Iurii Kemaev

Ting Yu

Dmitry (Dima) Lepikhin

Hao Xiong

Sahand Sharifzadeh

Oleaser Johnson

Jeremiah Willcock

Rui Yao

Greg Farquhar

Sujoy Basu

Hidetoshi Shimokawa

Nina Anderson

Haiguang Li

Khiem Pham

Yizhong Liang

Sebastian Borgeaud

Alexandre Moufarek

Hideto Kazawa

Blair Kutzman

Marcin Sieniek

Sara Smoot

Ruth Wang

Natalie Axelsson

Nova Fallen

Prasha Sundaram

Yuexiang Zhai

Varun Godbole

Petros Maniatis

Alek Wang

Ilia Shumailov

Santhosh Thangaraj

Remi Crocker

Nikita Gupta

Gang Wu

Phil Chen

Gellért Weisz

Celine Smith

Mojtaba Seyedhosseini

Boya Fang

Xiyang Luo

Roey Yogev

Zeynep Cankara

Andrew Hard

Helen Ran

Rahul Sukthankar

George Necula

Gaël Liu

Honglong Cai

Praseem Banzal

Daniel Keysers

Sanjay Ghemawat

Connie Tao

Emma Dunleavy

Aditi Chaudhary

Wei Li

Maciej Mikuła

Chen-Yu Lee

Tiziana Refice

Krishna Somandepalli

Alexandre Fréchette

Dan Bahir

John Karro

Keith Rush

Sarah Perrin

Bill Rosgen

Xiaomeng Yang

Clara Huiyi Hu

Mahmoud Alnahlawi

Justin Mao-Jones

Roopal Garg

Hoang Nguyen

Bat-Orgil Batsaikhan

Iñaki Iturrate

Anselm Levskaya

Avi Singh

Ashyana Kachra

Tony Lu

Denis Petek

Zheng Xu

Mark Graham

Lukas Zilka

Yael Karov

Marija Kostelac

Fangyu Liu

Yaohui Guo

Weiyue Wang

Bernd Bohnet

Emily Pitler

Tony Bruguier

Keisuke Kinoshita

Chrysovalantis Anastasiou

Nilpa Jha

Ting Liu

Jerome Connor

Phil Wallis

Philip Pham

Eric Bailey

Shixin Li

Heng-Tze Cheng

Sally Ma

Haiqiong Li

Akanksha Maurya

Kate Olszewska

Manfred Warmuth

Christy Koh

Dominik Paulus

Siddhartha Reddy Jonnalagadda

Enrique Piqueras

Ali Elqursh

Geoff Brown

Hadar Shemtov

Loren Maggiore

Fei Xia

Ryan Foley

Beka Westberg

George van den Driessche

Livio Baldini Soares

Arjun Kar

Michael Quinn

Siqi Zuo

Jialin Wu

Kyle Kastner

Anna Bortsova

Aijun Bai

Ales Mikhalap

Luowei Zhou

Jennifer Brennan

Vinay Ramasesh

Honglei Zhuang

John Maggs

Johan Schalkwyk

Yuntao Xu

Hui Huang

Andrew Howard

Sasha Brown

Linting Xue

Gloria Shen

Brian Albert

Neha Jha

Daniel Zheng

Varvara Krayvanova

Spurthi Amba Hombaiah

Olivier Lacombe

Gautam Vasudevan

Dan Graur

Tian Xie

Meet Gandhi

Bangju Wang

Dustin Zelle

Harman Singh

Dahun Kim

Sébastien Cevey

Victor Ungureanu

Natasha Noy

Fei Liu

Annie Xie

Fangxiaoyu Feng

Katerina Tsihlas

Daniel Formoso

Neera Vats

Quentin Wellens

Yinan Wang

Niket Kumar Bhumihar

Samrat Ghosh

Matt Hoffman

Tom Lieber

Oran Lang

Kush Bhatia

Tom Paine

Aroonalok Pyne

Ronny Votel

Madeleine Clare Elish

Benoit Schillings

Alex Panagopoulos

Haichuan Yang

Adam Raveret

Zohar Yahav

Shuang Liu

Warren Chen

Dalia El Badawy

Nishant Agrawal

Mohammed Badawi

Mahdi Mirzazadeh

Carla Bromberg

Fan Ye

Chang Liu

Tatiana Sholokhova

George-Cristian Muraru

Gargi Balasubramaniam

Jonathan Malmaud

Alen Carin

Danilo Martins

Irina Jurenka

Pankil Botadra

Dave Lacey

Richa Singh

Mariano Schain

Dan Zheng

Isabelle Guyon

Victor Lavrenko

Seungji Lee

Xiang Zhou

Demis Hassabis

Jeshwanth Challagundla

Derek Cheng

Nikhil Mehta

Matthew Mauger

Michela Paganini

Pushkar Mishra

Kate Lee

Zhang Li

Lexi Baugher

Ondrej Skopek

Max Chang

Amir Zait

Gaurav Menghani

Lizzetth Bellot

Guangxing Han

Jean-Michel Sarr

Sharat Chikkerur

Himanshu Sahni

Rohan Anil

Arun Narayanan

Chandu Thekkath

Daniele Pighin

Hana Strejček

Marko Velic

Fred Bertsch

Manuel Tragut

Keran Rong

Alicia Parrish

Kai Bailey

Jiho Park

Isabela Albuquerque

Abhishek Bapna

Rajesh Venkataraman

Alec Kosik

Johannes Griesser

Zhiwei Deng

Alek Andreev

Qingyun Dou

Kevin Hui

Fanny Wei

Xiaobin Yu

Lei Shu

Avia Aharon

David Barker

Badih Ghazi

Sebastian Flennerhag

Chris Breaux

Yuchuan Liu

Matthew Bilotti

Josh Woodward

Uri Alon

Stephanie Winkler

Tzu-Kuo Huang

Kostas Andriopoulos

João Gabriel Oliveira

Penporn Koanantakool

Berkin Akin

Michael Wunder

Cicero Nogueira dos Santos

Mohammad Hossein Bateni

Lin Yang

Dan Horgan

Beer Changpinyo

Keyvan Amiri

Min Ma

Dayeong Lee

Lihao Liang

Anirudh Baddepudi

Tejasi Latkar

Raia Hadsell

Jun Xu

Hairong Mu

Michael Han

Aedan Pope

Snchit Grover

Frank Kim

Ankit Bhagatwala

Guan Sun

Yamini Bansal

Amir Globerson

Alireza Nazari

Samira Daruki

Hagen Soltau

Jane Labanowski

Laurent El Shafey

Matt Harvey

Yanif Ahmad

Elan Rosenfeld

William Kong

Etienne Pot

Yi-Xuan Tan

Aurora Wei

Victoria Langston

Marcel Prasetya

Petar Veličković

Richard Killam

Robin Strudel

Darren Ni

Zhenhai Zhu

Aaron Archer

Kavya Kopparapu

Lynn Nguyen

Emilio Parisotto

Hussain Masoom

Sravanti Addepalli

Jordan Grimstad

Hexiang Hu

Joss Moore

Avinatan Hassidim

Le Hou

Mukund Raghavachari

Jared Lichtarge

Adam R. Brown

Hilal Dib

Natalia Ponomareva

Justin Fu

Yujing Zhang

Altaf Rahman

Joana Iljazi

Edouard Leurent

Gabriel Dulac-Arnold

Cosmo Du

Chulayuth Asawaroengchai

Larry Jin

Ela Gruzewska

Ziwei Ji

Benigno Uria

Daniel De Freitas

Paul Barham

Lauren Beltrone

Víctor Campos

Jun Yan

Neel Kovelamudi

Arthur Nguyen

Elinor Davies

Zhichun Wu

Zoltan Egyed

Kristina Toutanova

Nithya Attaluri

Hongliang Fei

Peter Stys

Siddhartha Brahma

Martin Izzard

Siva Velusamy

Scott Lundberg

Vincent Zhuang

Kevin Sequeira

Adam Santoro

Ehsan Amid

Ophir Aharoni

Shuai Ye

Mukund Sundararajan

Lijun Yu

Yu-Cheng Ling

Stephen Spencer

Hugo Song

Josip Djolonga

Christo Kirov

Sonal Gupta

Alessandro Bissacco

Clemens Meyer

Mukul Bhutani

Andrew Dai

Weiyi Wang

Siqi Liu

Ashwin Sreevatsa

Qijun Tan

Maria Wang

Lucy Kim

Yicheng Wang

Alex Irpan

Yang Xiao

Stanislav Fort

Yifan He

Alex Gurney

Bryan Gale

Yue Ma

Monica Roy

Viorica Patraucean

Taylan Bilal

Golnaz Ghiasi

Anahita Hosseini

Melvin Johnson

Zhuowan Li

Yi Tay

Benjamin Beyret

Katie Millican

Josef Broder

Mayank Lunayach

Danny Swisher

Eugen Vušak

David Parkinson

MH Tessler

Adi Mayrav Gilady

Richard Song

Allan Dafoe

Yves Raimond

Masa Yamaguchi

Itay Karo

Elizabeth Nielsen

Kevin Kilgour

Mike Dusenberry

Rajiv Mathews

Jiho Choi

Siyuan Qiao

Harsh Mehta

Sahitya Potluri

Chris Knutsen

Jialu Liu

Tat Tan

Kuntal Sengupta

Keerthana Gopalakrishnan

Abodunrinwa Toki

Mencher Chiang

Mike Burrows

Grace Vesom

Zafarali Ahmed

Ilia Labzovsky

Siddharth Vashishtha

Preeti Singh

Ankur Sharma

Ada Ma

Jinyu Xie

Pranav Talluri

Hannah Forbes-Pollard

Aarush Selvan

Joel Wee

Loic Matthey

Tom Funkhouser

Parthasarathy Gopavarapu

Lev Proleev

Cheng Li

Matt Thomas

Kashyap Kolipaka

Zhipeng Jia

Ashwin Kakarla

Srinivas Sunkara

Joan Puigcerver

Suraj Satishkumar Sheth

Emily Graves

Chen Wang

Sadh MNM Khan

Kai Kang

Shyamal Buch

Fred Zhang

Omkar Savant

David Soergel

Kevin Lee

Linda Friso

Xuanyi Dong

Rahul Arya

Shreyas Chandrakaladharan

Connor Schenck

Greg Billock

Tejas Iyer

Anton Bakalov

Leslie Baker

Alex Ruiz

Angad Chandorkar

Trieu Trinh

Matt Miecnikowski

Yanqi Zhou

Yangsibo Huang

Jiazhong Nie

Ali Shah

Ashish Thapliyal

Sam Haves

Lun Wang

Uri Shaham

Patrick Morris-Suzuki

Soroush Radpour

Leonard Berrada

Thomas Strohmann

Chaochao Yan

Jingwei Shen

Sonam Goenka

Tris Warkentin

Petar Dević

Dan Belov

Albert Webson

Madhavi Yenugula

Puranjay Datta

Jerry Chang

Nimesh Ghelani

Aviral Kumar

Vincent Perot

Jessica Lo

Yang Song

Herman Schmit

Jianmin Chen

Vasilisa Bashlovkina

Xiaoyue Pan

Diana Mincu

Paul Roit

Isabel Edkins

Andy Davis

Yujia Li

Ben Horn

Xinjian Li

Pradeep Kumar S

Eric Doi

Wanzheng Zhu

Sri Gayatri Sundara Padmanabhan

Siddharth Verma

Jasmine Liu

Heng Chen

Mihajlo Velimirović

Malcolm Reynolds

Priyanka Agrawal

Nick Sukhanov

Abhinit Modi

Siddharth Goyal

John Palowitch

Nima Khajehnouri

Wing Lowe

David Klinghoffer

Sharon Silver

Vinh Tran

Candice Schumann

Francesco Piccinno

Xi Liu

Mario Lučić

Xiaochen Yang

Sandeep Kumar

Ajay Kannan

Ragha Kotikalapudi

Mudit Bansal

Fabian Fuchs

Javad Hosseini

Abdelrahman Abdelhamed

Dawn Bloxwich

Tianhe Yu

Ruoxin Sang

Gregory Thornton

Karan Gill

Yuchi Liu

Virat Shejwalkar

Jason Lin

Zhipeng Yan

Kehang Han

Thomas Buschmann

Michael Pliskin

Zhi Xing

Susheel Tatineni

Junlin Zhang

Sissie Hsiao

Gavin Buttimore

Marcus Wu

Zefei Li

Geza Kovacs

Legg Yeung

Tao Huang

Aaron Cohen

Bethanie Brownfield

Averi Nowak

Mikel Rodriguez

Tianze Shi

Hado van Hasselt

Kevin Cen

Deepanway Ghoshal

Kushal Majmundar

Weiren Yu

Warren (Weilun) Chen

Danila Sinopalnikov

Hao Zhang

Vlado Galić

Di Lu

Zeyu Zheng

Maggie Song

Gary Wang

Gui Citovsky

Swapnil Gawde

Isaac Galatzer-Levy

David Silver

Ivana Balazevic

Dipanjan Das

Kingshuk Majumder

Yale Cong

Praneet Dutta

Dustin Tran

Hui Wan

Junwei Yuan

Daniel Eppens

Alanna Walton

Been Kim

Harry Ragan

James Cobon-Kerr

Lu Liu

Weijun Wang

Bryce Petrini

Jack Rae

Rakesh Shivanna

Yan Xiong

Chace Lee

Pauline Coquinot

Yiming Gu

Lisa Patel

Blake Hechtman

Aviel Boag

Orion Jankowski

Alex Wertheim

Alex Lee

Paul Covington

Hila Noga

Sam Sobell

Shanthal Vasanth

William Bono

Chirag Nagpal

Wei Fan

Xavier Garcia

Kedar Soparkar

Aybuke Turker

Nathan Howard

Sachit Menon

Yuankai Chen

Vikas Verma

Vladimir Pchelin

Harish Rajamani

Valentin Dalibard

Ana Ramalho

Yang Guo

Kartikeya Badola

Seojin Bang

Nathalie Rauschmayr

Julia Proskurnia

Sudeep Dasari

Xinyun Chen

Mikhail Sushkov

Anja Hauth

Pauline Sho

Abhinav Singh

Bilva Chandra

Allie Culp

Max Dylla

Olivier Bachem

James Besley

Heri Zhao

Timothy Lillicrap

Wei Wei

Wael Al Jishi

Ning Niu

Alban Rrustemi

Raphaël Lopez Kaufman

Ryan Poplin

Jewel Zhao

Minh Truong

Shikhar Bharadwaj

Ester Hlavnova

Eli Stickgold

Cordelia Schmid

Georgi Stephanov

Zhaoqi Leng

Frederick Liu

Léonard Hussenot

Shenil Dodhia

Juliana Vicente Franco

Lesley Katzen

Abhanshu Sharma

Sarah Cogan

Zuguang Yang

Aniket Ray

Sergi Caelles

Shen Yan

Ravin Kumar

Daniel Gillick

Renee Wong

Joshua Ainslie

Jonathan Hoech

Séb Arnold

Dan Abolafia

Anca Dragan

Ben Hora

Grace Hu

Alexey Guseynov

Yang Lu

Chas Leichner

Jinmeng Rao

Abhimanyu Goyal

Nagabhushan Baddi

Daniel Hernandez Diaz

Tim McConnell

Max Bain

Jake Abernethy

Qiqi Yan

Rylan Schaeffer

Paul Vicol

Will Thompson

Montse Gonzalez Arenas

Mathias Bellaiche

Pablo Barrio

Stefan Zinke

Riccardo Patana

Pulkit Mehta

JK Kearns

Avraham Ruderman

Scott Pollom

David D’Ambrosio

Cath Hope

Yang Yu

Andrea Gesmundo

Kuang-Huei Lee

Aviv Rosenberg

Yiqian Zhou

Yaoyiran Li

Drew Garmon

Yonghui Wu

Safeen Huda

Gil Fidel

Martin Baeuml

Jian Li

Phoebe Kirk

Rhys May

Tao Tu

Sara Mc Carthy

Toshiyuki Fukuzawa

Miranda Aperghis

Chih-Kuan Yeh

Toshihiro Yoshino

Bo Li

Austin Myers

Kaisheng Yao

Ben Limonchik

Changwan Ryu

Rohun Saxena

Alex Goldin

Ruizhe Zhao

Rocky Rhodes

Tao Zhu

Divya Tyam

Heidi Howard

Nathan Byrd

Hongxu Ma

Yan Wu

Ryan Mullins

Qingze Wang

Aida Amini

Sebastien Baur

Yiran Mao

Subhashini Venugopalan

Will Song

Wen Ding

Paul Collins

Sashank Reddi

Megan Shum

Andrei Rusu

Luisa Zintgraf

Kelvin Chan

Sheela Goenka

Mathieu Blondel

Michael Collins

Renke Pan

Marissa Giustina

Nikolai Chinaev

Christian Schuler

Ce Zheng

Jonas Valfridsson

Alyssa Loo

Alex Yakubovich

Jamie Smith

Tao Jiang

Rich Munoz

Gabriel Barcik

Rishabh Bansal

Mingyao Yang

Yilun Du

Pablo Duque

Mary Phuong

Alexandra Belias

Kunal Lad

Zeyu Liu

Tal Schuster

Karthik Duddu

Jieru Hu

Paige Kunkle

Matthew Watson

Jackson Tolins

Josh Smith

Denis Teplyashin

Garrett Bingham

Marvin Ritter

Marco Andreetto

Divya Pitta

Mohak Patel

Shashank Viswanadha

Trevor Strohman

Catalin Ionescu

Jincheng Luo

Yogesh Kalley

Jeremy Wiesner

Dan Deutsch

Derek Lockhart

Peter Choy

Rumen Dangovski

Chawin Sitawarin

Cat Graves

Tanya Lando

Joost van Amersfoort

Ndidi Elue

Zhouyuan Huo

Pooya Moradi

Jean Tarbouriech

Henryk Michalewski

Wenting Ye

Eunyoung Kim

Alex Druinsky

Florent Altché

Xinyi Chen

Artur Dwornik

Da-Cheng Juan

Rivka Moroshko

Horia Toma

Jarrod Kahn

Hai Qian

Maximilian Sieb

Irene Cai

Roman Goldenberg

Praneeth Netrapalli

Sindhu Raghuram

Yuan Gong

Lijie Fan

Evan Palmer

Yossi Matias

Valentin Gabeur

Shreya Pathak

Tom Ouyang

Don Metzler

Geoff Bacon

Srinivasan Venkatachary

Sridhar Thiagarajan

Alex Cullum

Eran Ofek

Vytenis Sakenas

Mohamed Hammad

Cesar Magalhaes

Mayank Daswani

Oscar Chang

Ashok Popat

Ruichao Li

Komal Jalan

Yanhan Hou

Josh Lipschultz

Antoine He

Wenhao Jia

Pier Giuseppe Sessa

Prateek Kolhar

William Wong

Sumeet Singh

Lukas Haas

Jay Whang

Hanna Klimczak-Plucińska

Georges Rotival

Grace Chung

Yiqing Hua

Anfal Siddiqui

Nicolas Serrano

Dongkai Chen

Billy Porter

Libin Bai

Keshav Shivam

Sho Arora

Partha Talukdar

Tom Cobley

Sangnie Bhardwaj

Evgeny Gladchenko

Simon Green

Kelvin Guu

Felix Fischer

Xiao Wu

Eric Wang

Achintya Singhal

Tatiana Matejovicova

James Martens

Hongji Li

Roma Patel

Elizabeth Kemp

Jiaqi Pan

Lily Wang

Blake JianHang Chen

Jean-Baptiste Alayrac

Navneet Potti

Erika Gemzer

Eugene Ie

Kay McKinney

Takaaki Saeki

Edward Chou

Pascal Lamblin

SQ Mah

Zach Fisher

Martin Chadwick

Jon Stritar

Obaid Sarvana

Andrew Hogue

Artem Shtefan

Hadi Hashemi

Yang Xu

Jindong Gu

Sharad Vikram

Chung-Ching Chang

Sabela Ramos

Logan Kilpatrick

Weijuan Xi

Jenny Brennan

Yinghao Sun

Abhishek Jindal

Ionel Gog

Dawn Chen

Felix Wu

Jason Lee

Sudhindra Kopalle

Srinadh Bhojanapalli

Oriol Vinyals

Natan Potikha

Burcu Karagol Ayan

Yuan Yuan

Michael Riley

Piotr Stanczyk

Sergey Kishchenko

Bing Wang

Dan Garrette

Antoine Yang

Vlad Feinberg

CJ Carey

Javad Azizi

Viral Shah

Erica Moreira

Chongyang Shi

Josh Feldman

Elizabeth Salesky

Thomas Lampe

Aneesh Pappu

Duhyeon Kim

Jonas Adler

Avi Caciularu

Brian Walker

Yunhan Xu

Yochai Blau

Dylan Scandinaro

Terry Huang

Sam El-Husseini

Abhishek Sinha

Lijie Ren

Taylor Tobin

Patrik Sundberg

Tim Sohn

Vikas Yadav

Mimi Ly

Emily Xue

Jing Xiong

Afzal Shama Soudagar

Sneha Mondal

Nikhil Khadke

Qingchun Ren

Ben Vargas

Stan Bileschi

Sarah Chakera

Cindy Wang

Boyu Wang

Yoni Halpern

Joe Jiang

Vikas Sindhwani

Petre Petrov

Pranavaraj Ponnuramu

Sanket Vaibhav Mehta

Yu Watanabe

Betty Chan

Matheus Wisniewski

Trang Pham

Jingwei Zhang

Conglong Li

Dario de Cesare

Art Khurshudov

Alex Vasiloff

Melissa Tan

Zoe Ashwood

Bobak Shahriari

Maryam Majzoubi

Garrett Tanzer

Olga Kozlova

Robin Alazard

James Lee-Thorp

Nguyet Minh Phu

Isaac Tian

Junwhan Ahn

Andy Crawford

Lauren Lax

Yuan (June) Shangguan

Iftekhar Naim

David Ross

Oleksandr Ferludin

Tongfei Guo

Andrea Banino

Hubert Soyer

Xiaoen Ju

Dominika Rogozińska

Ishaan Malhi

Marcella Valentine

Daniel Balle

Apoorv Kulshreshtha

Maciej Kula

Yiwen Song

Sophia Austin

John Schultz

Roy Hirsch

Arthur Douillard

Apoorv Reddy

Michael Fink

Summer Yue

Khyatti Gupta

Adam Zhang

Norman Rink

Daniel McDuff

Lei Meng

András György

Yasaman Razeghi

Ricky Liang

Kazuki Osawa

Aviel Atias

Matan Eyal

Tyrone Hill

Nikolai Grigorev

Zhengdong Wang

Nitish Kulkarni

Rachel Soh

Ivan Lobov

Zachary Charles

Sid Lall

Kazuma Hashimoto

Ido Kessler

Victor Gomes

Zelda Mariet

Danny Driess

Alessandro Agostini

Canfer Akbulut

Jingcao Hu

Marissa Ikonomidis

Emily Caveness

Kartik Audhkhasi

Saurabh Agrawal

Ioana Bica

Evan Senter

Jayaram Mudigonda

Kelly Chen

Jingchen Ye

Xuanhui Wang

James Svensson

Philipp Fränken

Josh Newlan

Li Lao

Eva Schnider

Sami Alabed

Joseph Kready

Jesse Emond

Afief Halumi

Tim Zaman

Chengxi Ye

Naina Raisinghani

Vilobh Meshram

Bo Chang

Ankit Singh Rawat

Axel Stjerngren

Sergey Levi

Rui Wang

Xiangzhu Long

Mitchelle Rasquinha

Steven Hand

Aditi Mavalankar

Lauren Agubuzu

Sudeshna Roy

Junquan Chen

Jarek Wilkiewicz

Hao Zhou

Michal Jastrzebski

Qiong Hu

Agustin Dal Lago

Ramya Sree Boppana

Wei-Jen Ko

Jennifer Prendki

Yao Su

Zhi Li

Eliza Rutherford

Girish Ramchandra Rao

Ramona Comanescu

Adrià Puigdomènech

Qihang Chen

Dessie Petrova

Christine Chan

Vedrana Milutinovic

Felipe Tiengo Ferreira

Chin-Yi Cheng

Ming Zhang

Tapomay Dey

Sherry Yang

Ramesh Sampath

Quoc Le

Howard Zhou

Chu-Cheng Lin

Hoi Lam

Christine Kaeser-Chen

Kai Hui

Dean Hirsch

Tom Eccles

Basil Mustafa

Shruti Rijhwani

Morgane Rivière

Yuanzhong Xu

Junjie Wang

Xinyang Geng

Xiance Si

Arjun Khare

Cheolmin Kim

Vahab Mirrokni

Kamyu Lee

Khuslen Baatarsukh

Nathaniel Braun

Lisa Wang

Pallavi LV

Richard Tanburn

Yuvein (Yonghao) Zhu

Fangda Li

Setareh Ariafar

Dan Goldberg

Ken Burke

Daniil Mirylenka

Meiqi Guo

Olaf Ronneberger

Hadas Natalie Vogel

Liqun Cheng

Nishita Shetty

Johnson Jia

Thomas Jimma

Corey Fry

Ted Xiao

Martin Sundermeyer

Ryan Burnell

Yannis Assael

Mario Pinto

JD Chen

Rohit Sathyanarayana

Donghyun Cho

Jing Lu

Rishabh Agarwal

Sugato Basu

Lucas Gonzalez

Dhruv Shah

Meng Wei

Dre Mahaarachchi

Rohan Agrawal

Tero Rissa

Yani Donchev

Ramiro Leal-Cavazos

Adrian Hutter

Markus Mircea

Alon Jacovi

Faruk Ahmed

Jiageng Zhang

Shuguang Hu

Bo-Juen Chen

Jonni Kanerva

Guillaume Desjardins

Andrew Lee

Nikos Parotsidis

Asier Mujika

Tobias Weyand

Jasper Snoek

Jo Chick

Kai Chen

Paul Chang

Ethan Mahintorabi

Zi Wang

Tolly Powell

Orgad Keller

Abhirut Gupta

Claire Sha

Kanav Garg

Nicolas Heess

Ágoston Weisz

Cassidy Hardin

Bartek Wydrowski

Ben Coleman

Karina Zainullina

Pankaj Joshi

Alessandro Epasto

Terry Spitz

Binbin Xiong

Kai Zhao

Arseniy Klimovskiy

Ivy Zheng

Johan Ferret

Itay Yona

Waleed Khawaja

Jean-Baptiste Lespiau

Maxim Krikun

Siamak Shakeri

Timothee Cour

Bonnie Li

Igor Krivokon

Dan Suh

Alex Hofer

Jad Al Abdallah

Nikita Putikhin

Oscar Akerlund

Silvio Lattanzi

Anurag Kumar

Shane Settle

Himanshu Srivastava

Folawiyo Campbell-Ajala

Edouard Rosseel

Mihai Dorin Istin

Nishanth Dikkala

Anand Rao

Nick Young

Kate Lin

Dhruva Bhaswar

Yiming Wang

Jaume Sanchez Elias

Kritika Muralidharan

James Keeling

Dayou Du

Siddharth Gopal

Gregory Dibb

Charles Blundell

Manolis Delakis

Jacky Liang

Marco Tulio Ribeiro

Georgi Karadzhov

Guillermo Garrido

Ankur Bapna

Jiawei Cao

Adam Sadovsky

Pouya Tafti

Arthur Guez

Coline Devin

Yixian Di

Jinwei Xing

Chuqiao (Joyce) Xu

Hanzhao Lin

Chun-Te Chu

Sameera Ponda

Wesley Helmholz

Fan Yang

Yue Gao

Sara Javanmardi

Wael Farhan

Alex Ramirez

Ricardo Figueira

Khe Chai Sim

Yuval Bahat

Ashwin Vaswani

Liangzhe Yuan

Gufeng Zhang

Leland Rechis

Hanjun Dai

Tayo Oguntebi

Alexandra Cordell

Eugénie Rives

Kaan Tekelioglu

Naveen Kumar

Bing Zhang

Aurick Zhou

Nikolay Savinov

Andrew Leach

Alex Tudor

Sanjay Ganapathy

Yanyan Zheng

Mirko Rossini

Vera Axelrod

Arnaud Autef

Yukun Zhu

Zheng Zheng

Mingda Zhang

Baochen Sun

Jie Ren

Nenad Tomasev

Nithish Kannan

Amer Sinha

Charles Chen

Louis O’Bryan

Alex Pak

Aditya Kusupati

Weel Yang

Deepak Ramachandran

Patrick Griffin

Seokhwan Kim

Philipp Neubeck

Craig Schiff

Tammo Spalink

Mingyang Ling

Arun Nair

Ga-Young Joung

Linda Deng

Avishkar Bhoopchand

Lora Aroyo

Tom Duerig

Jordan Griffith

Gabe Barth-Maron

Jake Ades

Alex Haig

Ankur Taly

Yunting Song

Paul Michel

Dave Orr

Dean Weesner

Corentin Tallec

Carrie Grimes Bostock

Paul Niemczyk

Andy Twigg

Mudit Verma

Rohith Vallu

Henry Wang

Marco Gelmi

Kiranbir Sodhia

Aleksandr Chuklin

Omer Goldman

Jasmine George

Liang Bai

Kelvin Zhang

Petar Sirkovic

Efrat Nehoran

Golan Pundak

Jiaqi Mu

Alice Chen

Alex Greve

Paulo Zacchello

David Amos

Heming Ge

Eric Noland

Colton Bishop

Jeffrey Dudek

Youhei Namiki

Elena Buchatskaya

Jing Li

Dorsa Sadigh

Masha Samsikova

Dan Malkin

Damien Vincent

Robert David

Rob Willoughby

Phoenix Meadowlark

Shawn Gao

Yan Li

Raj Apte

Amit Jhindal

Stein Xudong Lin

Alex Polozov

Zhicheng Wang

Tomas Mery

Anirudh GP

Varun Yerram

Sage Stevens

Tianqi Liu

Noah Fiedel

Charles Sutton

Matthew Johnson

Xiaodan Song

Kate Baumli

Nir Shabat

Muqthar Mohammad

Hao Liu

Marco Selvi

Yichao Zhou

Mehdi Hafezi Manshadi

Chu-ling Ko

Anthony Chen

Michael Bendersky

Jorge Gonzalez Mendez

Nisarg Kothari

Amir Zandieh

Yiling Huang

Daniel Andor

Ellie Pavlick

Idan Brusilovsky

Jitendra Harlalka

Sally Goldman

Andrew Lampinen

Guowang Li

Asahi Ushio

Somit Gupta

Lei Zhang

Chuyuan Kelly Fu

Madhavi Sewak

Timo Denk

Jed Borovik

Brendan Jou

Avital Zipori

Prateek Jain

Junwen Bai

Thang Luong

Jonathan Tompson

Alice Li

Li Liu

George Powell

Jiajun Shen

Alex Feng

Grishma Chole

Da Yu

Yinlam Chow

Tongxin Yin

Eric Malmi

Kefan Xiao

Yash Pande

Shachi Paul

Niccolò Dal Santo

Adil Dostmohamed

Sergio Guadarrama

Aaron Phillips

Thanumalayan Sankaranarayana Pillai

Gal Yona

Amin Ghafouri

Preethi Lahoti

Benjamin Lee

Dhruv Madeka

Eren Sezener

Simon Tokumine

Adrian Collister

Nicola De Cao

Richard Shin

Uday Kalra

Parker Beak

Emily Nottage

Ryo Nakashima

Ivan Jurin

Vikash Sehwag

Meenu Gaba

Junhao Zeng

Kevin R. McKee

Fernando Pereira

Tamar Yakar

Amayika Panda

Arka Dhar

Peilin Zhong

Daniel Sohn

Mark Brand

Lars Lowe Sjoesund

Viral Carpenter

Sharon Lin

Shantanu Thakoor

Marcus Wainwright

Ashwin Chaugule

Pranesh Srinivasan

Muye Zhu

Bernett Orlando

Jack Weber

Ayzaan Wahid

Gilles Baechler

Apurv Suman

Jovana Mitrović

Gabe Taubman

Honglin Yu

Helen King

Josh Dillon

Cathy Yip

Dhriti Varma

Tomas Izo

Levent Bolelli

Borja De Balle Pigem

Julia Di Trapani

Fotis Iliopoulos

Adam Paszke

Nishant Ranka

Joe Zou

Francesco Pongetti

Jed McGiffin

Alex Siegman

Rich Galt

Ross Hemsley

Goran Žužić

Victor Carbune

Tao Li

Myle Ott

Félix de Chaumont Quitry

David Vilar Torres

Yuri Chervonyi

Tomy Tsai

Prem Eruvbetine

Samuel Yang

Matthew Denton

Jake Walker

Slavica Andačić

Idan Heimlich Shtacher

Vittal Premachandran

Harshal Tushar Lehri

Cip Baetu

Damion Yates

Lampros Lamprou

Mariko Iinuma

Ioana Mihailescu

Ben Albrecht

Shachi Dave

Susie Sargsyan

Bryan Perozzi

Lucas Manning

Chiyuan Zhang

Denis Vnukov

Igor Mordatch

Raia Hadsell Wolfgang Macherey

Ryan Kappedal

Jim Stephan

Aditya Tripathi

Klaus Macherey

Jun Qian

Abhishek Bhowmick

Shekoofeh Azizi

Rémi Leblond

Shiva Mohan Reddy Garlapati

Timothy Knight

Matthew Wiethoff

Wei-Chih Hung

Anelia Angelova

Georgios Evangelopoulos

Pawel Janus

Dimitris Paparas

Matthew Rahtz

Ken Caluwaerts

Vivek Sampathkumar

Daniel Jarrett

Shadi Noghabi

Antoine Miech

Chak Yeung

Geoff Clark

Henry Prior

Fei Zheng

Jean Pouget-Abadie

Indro Bhattacharya

Kalpesh Krishna

Will Bishop

Zhe Yuan

Yunxiao Deng

Ashutosh Sathe

Kacper Krasowiak

Ciprian Chelba

Cho-Jui Hsieh

Kiran Vodrahalli

Buhuang Liu

Thomas Köppe

Amr Khalifa

Lubo Litchev

Pichi Charoenpanit

Reed Roberts

Sachin Yadav

Yasumasa Onoe

Desi Ivanov

Megha Mohabey

Vighnesh Birodkar

Nemanja Rakićević

Pierre Sermanet

Vaibhav Mehta

Krishan Subudhi

Travis Choma

Will Ng

Luheng He

Kathie Wang

Tasos Kementsietsidis

Shane Gu

Mansi Gupta

Andrew Nystrom

Mehran Kazemi

Timothy Chung

Nacho Cano

Nikhil Dhawan

Yufei Wang

Jiawei Xia

Trevor Yacovone

Eric Jia

Mingqing Chen

Simeon Ivanov

Ashrith Sheshan

Sid Dalmia

Paweł Stradomski

Pengcheng Yin

Salem Haykal

Congchao Wang

Dennis Duan

Neslihan Bulut

Greg Kochanski

Liam MacDermed

Namrata Godbole

Shitao Weng

Jingjing Chen

Rachana Fellinger

Ramin Mehran

Daniel Suo

Hisham Husain

Tong He

Kaushal Patel

Joshua Howland

Randall Parker

Kelvin Nguyen

Sharath Maddineni

Chris Rawles

Mina Khan

Shlomi Cohen-Ganor

Amol Mandhane

Xinyi Wu

Chenkai Kuang

Iulia Comşa

Ramya Ganeshan

Hanie Sedghi

Adam Bloniarz

Nuo Wang Pierse

Anton Briukhov

Petr Mitrichev

Anita Gergely

Serena Zhan

Allan Zhou

Nikita Saxena

Eva Lu

Josef Dean

Ashish Gupta

Nicolas Perez-Nieves

Renjie Wu

Cory McLean

Wei Liang

Disha Jindal

Anton Tsitsulin

Wenhao Yu

Kaiz Alarakyia

Tom Schaul

Piyush Patil

Peter Sung

Elijah Peake

Hongkun Yu

Feryal Behbahani

JD Co-Reyes

Alan Ansell

Sean Sun

Clara Barbu

Jonathan Lee

Seb Noury

James Allingham

Bilal Piot

Mohit Sharma

Christopher Yew

Ivan Korotkov

Bibo Xu

Demetra Brady

Goran Petrovic

Shibl Mourad

Claire Cui

Aditya Gupta

Parker Schuh

Saarthak Khanna

Anna Goldie

Abhinav Arora

Vadim Zubov

Amy Stuart

Mark Epstein

Yun Zhu

Jianqiao Liu

Yury Stuken

Ziyue Wang

Karolis Misiunas

Dee Guo

Ashleah Gill

Ale Hartman

Zaid Nabulsi

Aurko Roy

Aleksandra Faust

Jason Riesa

Ben Withbroe

Mengchao Wang

Marco Tagliasacchi

Andreea Marzoca

James Noraky

Serge Toropov

Malika Mehrotra

Bahram Raad

Sanja Deur

Steve Xu

Marianne Monteiro

Zhongru Wu

Yi Luan

Sam Ritter

Nick Li

Håvard Garnes

Yanzhang He

Martin Zlocha

Jifan Zhu

Matteo Hessel

Will Wu

Spandana Raj Babbula

Chizu Kawamoto

Yuanzhen Li

Mehadi Hassen

Yan Wang

Brian Wieder

James Freedman

Yin Zhang

Xinyi Bai

Tianli Yu

David Reitter

XiangHai Sheng

Mateo Wirth

Aditya Kini

Dima Damen

Mingcen Gao

Rachel Hornung

Michael Voznesensky

Brian Roark

Adhi Kuncoro

Yuxiang Zhou

Rushin Shah

Anthony Brohan

Kuangyuan Chen

James Wendt

David Rim

Paul Kishan Rubenstein

Jonathan Halcrow

Michelle Liu

Ty Geri

Yunhsuan Sung

Jane Shapiro

Shaan Bijwadia

Chris Duvarney

Christina Sorokin

Paul Natsev

Reeve Ingle

Pramod Gupta

Young Maeng

Ndaba Ndebele

Kexin Zhu

Valentin Anklin

Katherine Lee

Yuan Liu

Yaroslav Akulov

Shaleen Gupta

Guolong Su

Flavien Prost

Tianlin Liu

Vitaly Kovalev

Pol Moreno

Martin Scholz

Sam Redmond

Zongwei Zhou

Alex Castro-Ros

André Susano Pinto

Dia Kharrat

Michal Yarom

Rachel Saputro

Jannis Bulian

Ben Caine

Ji Liu

Abbas Abdolmaleki

Shariq Iqbal

Tautvydas Misiunas

Mikhail Sirotenko

Shefali Garg

Guy Bensky

Huan Gui

Xuezhi Wang

Raphael Koster

Mike Bernico

Da Huang

Romal Thoppilan

Trevor Cohn

Ben Golan

Wenlei Zhou

Andrew Rosenberg

Markus Freitag

Tynan Gangwani

Vincent Tsang

Anand Shukla

Xiaoqi Ren

Minh Giang

Chi Zou

Andre Elisseeff

Charline Le Lan

Dheeru Dua

Shuba Lall

Pranav Shyam

Frankie Garcia

Sarah Nguyen

Michael Guzman

AJ Maschinot

Marcello Maggioni

Ming-Wei Chang

Karol Gregor

Lotte Weerts

Kumaran Venkatesan

Bogdan Damoc

Leon Liu

Jan Wassenberg

Lewis Ho

Becca Roelofs

Majid Hadian

François-Xavier Aubet

Yu Liang

Sami Lachgar

Danny Karmon

Yong Cheng

Amelio Vázquez-Reina

Angie Chen

Zhuyun Dai

Andy Brock

Shubham Agrawal

Chenxi Pang

Peter Garst

Mariella Sanchez-Vargas

Ivor Rendulic

Aditya Ayyar

Andrija Ražnatović

Olivia Ma

Roopali Vij

Neha Sharma

Ashwin Balakrishna

Bingyuan Liu

Ian Mackinnon

Sorin Baltateanu

Petra Poklukar

Gabriel Ibagon

Colin Ji

Hongyang Jiao

Isaac Noble

Wojciech Stokowiec

Zhihao Li

Jeff Dean

David Lindner

Mark Omernick

Kristen Chiafullo

Mason Dimarco

Vitor Rodrigues

Vittorio Selo

Garrett Honke

Xintian (Cindy) Wu

Wei He

Adam Hillier

Anhad Mohananey

Vihari Piratla

Chang Ye

Chase Malik

Sebastian Riedel

Samuel Albanie

Zi Yang

Kenny Vassigh

Maria Bauza

Sheng Li

Yiqing Tao

Nevan Wichers

Andrii Maksai

Abe Ittycheriah

Ross Mcilroy

Bryan Seybold

Noah Goodman

Romina Datta

Steven M. Hernandez

Tian Shi

Yony Kochinski

Anna Bulanova

Ken Franko

Mikita Sazanovich

Nicholas FitzGerald

Praneeth Kacham

Shubha Srinivas Raghvendra

Vincent Hellendoorn

Alexander Grushetsky

Julian Salazar

Angeliki Lazaridou

Jason Chang

Jan-Thorsten Peter

Sushant Kafle

Yann Dauphin

Abhishek Rao

Filippo Graziano

Izhak Shafran

Yuguo Liao

Tianli Ding

Geng Yan

Grace Chu

Zhao Fu

Vincent Roulet

Gabriel Rasskin

Duncan Williams

Shahar Drath

Alex Mossin

Raphael Hoffmann

Jordi Orbay

Francesco Bertolini

Hila Sheftel

Justin Chiu

Siyang Xue

Yuheng Kuang

Ferjad Naeem

Swaroop Nath

Nana Nti

Phil Culliton

Kashyap Krishnakumar

Michael Isard

Pei Sun

Ayan Chakrabarti

Nathan Clement

Regev Cohen

Arissa Wongpanich

GS Oh

Ashwin Murthy

Hao Zheng

Jessica Hamrick

Oskar Bunyan

Suhas Ganesh

Nitish Gupta

Roy Frostig

John Wieting

Yury Malkov

Pierre Marcenac

Zhixin (Lucas) Lai

Xiaodan Tang

Mohammad Saleh

Fedir Zubach

Chinmay Kulkarni

Huanjie Zhou

Vicky Zayats

Nan Ding

Anshuman Tripathi

Arijit Pramanik

Patrik Zochbauer

Harish Ganapathy

Vedant Misra

Zach Behrman

Hugo Vallet

Mingyang Zhang

Mukund Sridhar

Ye Jin

Mohammad Babaeizadeh

Siim Põder

Megha Goel

Divya Jain

Tajwar Nasir

Shubham Mittal

Tim Dozat

Diego Ardila

Aliaksei Severyn

Fabio Pardo

Sammy Jerome

Siyang Qin

Louis Rouillard

Amir Yazdanbakhsh

Zizhao Zhang

Shivani Agrawal

Kaushik Shivakumar

Caden Lu

Praveen Kallakuri

Rachita Chhaparia

Kanishka Rao

Charles Kwong

Asya Fadeeva

Shitij Nigam

Yan Virin

Yuan Zhang

Balaji Venkatraman

Beliz Gunel

Marc Wilson

Huiyu Wang

Abhinav Gupta

Xiaowei Xu

Adrien Ali Taïga

Kareem Mohamed

Doug Fritz

Daniel Rodriguez

Zoubin Ghahramani

Harry Askham

Lior Belenki

James Zhao

Rahul Gupta

Krzysztof Jastrzębski

Takahiro Kosakai

Kaan Katircioglu

Jon Schneider

Rina Panigrahy

Konstantinos Bousmalis

Peter Grabowski

Prajit Ramachandran

Chaitra Hegde

Mihaela Rosca

Angelo Scorza Scarpati

Kyriakos Axiotis

Ying Xu

Zach Gleicher

Assaf Hurwitz Michaely

Mandar Sharma

Sanil Jain

Christoph Hirnschall

Tal Marian

Xuhui Jia

Kevin Mather

Kilol Gupta

Linhai Qiu

Nigamaa Nayakanti

Lucian Ionita

Steven Zheng

Lucia Loher

Kurt Shuster

Igor Petrovski

Roshan Sharma

Rahma Chaabouni

Angel Yeh

James An

Arushi Gupta

Steven Schwarcz

Seher Ellis

Sam Conway-Rahman

Javier Snaider

Alex Zhai

James Atwood

Daniel Golovin

Liqian Peng

Te I

Vivian Xia

Salvatore Scellato

Mahan Malihi

Arthur Bražinskas

Vlad-Doru Ion

Younghoon Jun

James Swirhun

Soroosh Mariooryad

Jiao Sun

Steve Chien

Rey Coaguila

Ariel Brand

Yi Gao

Tom Kwiatkowski

Roee Aharoni

Cheng-Chun Lee

Mislav Žanić

Yichi Zhang

Dan Ethier

Vitaly Nikolaev

Pranav Nair

Yoav Ben Shalom

Hen Fitoussi

Jai Gupta

Hongbin Liu

Dee Cattle

Tolga Bolukbasi

Ben Murdoch

Fantine Huot

Yin Li

Chris Hahn

The development of Gemini is a large-scale collaborative effort involving over 3000 individuals across Google, including researchers, engineers, and operations staff. These individuals contributed their hard work and expertise across diverse areas, from foundational research and the development of model architecture, data, training, and infrastructure, through to evaluation and ensuring safety and security. We gratefully acknowledge the dedication and hard work of each contributor in making Gemini a reality.

We are also grateful to the Google-independent developer Joel Zhang for his work on Gemini Plays Pokémon, and for sharing with us the design of his set-up.

[H] 8 Appendix

[H] 8.1 Evaluation additional details

Please see a description of the benchmarks considered, along with details of how scores in the main text were obtained in Table
11
.

Benchmark

Description

Details

LiveCodeBench

Code generation in Python
(
Jain et al., 2024
)
.

Results are taken from
https://livecodebench.github.io/leaderboard.html
(1/1/2025 - 5/1/2025 in the UI) or, where not available, run internally by us. For Section
2.5
and Figure
3
and
4
, results are calculated on the version of the eval corresponding to 10/05/2024 - 01/04/2025 in the UI, and are based on internal results.

Aider Polyglot

Code editing in C++, Go, Java, JavaScript Python and Rust
(
Gauthier, 2025
)
. See
https://aider.chat/2024/12/21/polyglot.html#the-polyglot-benchmark
for a full description of this task.

We report results on the “diff” or “diff-fenced” edit format (see
https://aider.chat/docs/more/edit-formats.html
for a description of the different formats). The score reported are the pass rate average of 3 trials. Numbers come from
https://aider.chat/docs/leaderboards/

SWE-bench Verified

Agentic coding: evaluates AI agents on real-world programming tasks from GitHub
(
Jimenez et al., 2024
;
Chowdhury et al., 2024
)
.

Gemini uses an internal agentic harness equipped with tools to navigate the repo, edit files, and test the code.
We report scores for two modes: performance of a single agentic trace (“single attempt”), and performance of a scaffold that samples multiple agentic traces and re-reranks them before evaluation using Gemini’s own judgement (“multiple attempts”).
All evaluations are done with temperature=1, topp=0.99, topk=1024.

GPQA
(diamond)

Challenging dataset of questions written by domain experts in biology, physics, and chemistry
(
Rein et al., 2024
)
.

Humanity’s Last Exam

Challenging dataset of questions written by domain experts in a wide range of disciplines, including mathematics, physics, chemistry, biology and computer science
(
Phan et al., 2025
)
.

No tool use variant.
Reported results are from
https://scale.com/leaderboard/humanitys_last_exam
.
For DeepSeek they are taken from
https://scale.com/leaderboard/humanitys_last_exam_text_only
(leaderboard for performance on the text-only questions) and in the case of the Gemini 2.0 models, these results are on an earlier HLE dataset, obtained from
https://scale.com/leaderboard/humanitys_last_exam_preview
(indicated with a
†
\dagger
in Table
3
)

SimpleQA

World knowledge factuality with no search enabled
(
Wei et al., 2024
)
.

F1 scores are obtained from
https://github.com/openai/simple-evals
and, where not available, run internally by us.

FACTS Grounding

Ability to provide factually correct responses given documents and diverse user requests.
(
Jacovi et al., 2025
)

Results are sourced from
https://www.kaggle.com/benchmarks/google/facts-grounding

Global MMLU (Lite)

MMLU translated by human translators into 15 languages.
(
Singh et al., 2024
)

The lite version includes 200 Culturally Sensitive and 200 Culturally Agnostic samples per language, see
https://huggingface.co/datasets/CohereLabs/Global-MMLU-Lite

ECLeKTic

A closed-book QA dataset that evaluates cross-lingual knowledge transfer
(
Goldman et al., 2025
)
.

AIME 2025

Performance on 30 questions from American Invitational Mathematics Examination from 2025
(
Balunović et al., 2025
)
.

Results are sourced from
https://matharena.ai/
.

HiddenMath-Hard

Competition-level math problems, Held out dataset AIME/AMC-like, crafted by experts and not leaked on the web.

LOFT (hard retrieval subset)

Long context multi-hop and multi-needle retrieval evaluation of 300 queries
(
Lee et al., 2024
)
.

We report the results on two variants: an up to 128K average context length variant to ensure they can be comparable with other models and a pointwise value for 1M context window to show the capability of the model at full length.

MRCR-V2 (8-needle)

MRCR-V2 is a significantly harder instance of the MRCR family of long-context evaluations
(
Vodrahalli et al., 2024
)
. Compared to MRCR-V1, we increase the nesting of the dictionary size to depth 3 rather than 2 by including a style parameter (for instance, an example key might be “write a poem about penguins in an archaic style”, rather than just “write a poem about penguins”).

The methodology has changed compared to previously published results: we focus on a harder, 8-needle version (compared to the 4-needle version used before).
We report the results on two variants: an up to 128K average context length variant to ensure they can be comparable with other models and a pointwise value for 1M context window to show the capability of the model at full length.

MMMU

Multi-discipline college-level multimodal image understanding and reasoning problems.
(
Yue et al., 2024
)

Vibe-Eval (Reka)

Image understanding evaluation, featuring particularly challenging examples.
(
Padlewski et al., 2024
)

Gemini is used as a judge.

ZeroBench

Challenging image understanding evaluation that requires multi-step reasoning.
(
Roberts et al., 2025
)

Gemini is used as a judge. Average over 4 runs.

BetterChartQA

A comprehensive chart understanding evaluation that covers 9 disjoint capability buckets. The chart images are randomly sampled from the web and QA pairs are written by professional human annotators to reflect the wide distribution of chart styles and real-world cases.
(
Gemini Team, 2024
)

Gemini is used as a judge.

FLEURS

Automatic speech recognition
(
Conneau et al., 2023
)
.

0-shot queries to public APIs for all models.
Used a subset of 53 languages (out of 102); we filtered languages for which either model responses were too incompatible to ground truth responses to be fairly scored.
We use Word-Error-Rate WER (lower is better) except for four segmented languages where we aggregate Character-Error-Rates (Chinese, Japanese, Korean and Thai).

CoVoST 2

Speech to text translation
(
Wang et al., 2020
)
.

0-shot queries to public APIs for all models.
We report BLEU scores for translating 21 languages to English.

ActivityNet-QA

General video understanding
(
Yu et al., 2019
)

Test subset, 0-shot.
Videos were processed at 1fps and linearly subsampled to a maximum of
N
f
​
r
​
a
​
m
​
e
​
s
=
1024
N_{frames}=1024
frames. For GPT 4.1, we used 500 frames due to API limitations.

EgoTempo

Egocentric video understanding
(
Plizzari et al., 2025
)

Test subset, 0-shot.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
256
N_{frames}=256
.

Perception Test

Perceptual understanding/reasoning
(
Patraucean et al., 2023
)

Test subset, 0-shot.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
256
N_{frames}=256
.

QVHighlights

Moment retrieval
(
Lei et al., 2021
)

Validation subset, 4-shots. Accuracy measured with R1@0.5.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
256
N_{frames}=256
.

VideoMMMU

Video knowledge acquisition
(
Hu et al., 2025
)

Test subset, 0-shot.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
256
N_{frames}=256
.

1H-VideoQA

Hour-long video understanding
(
Gemini Team, 2024
)

Test subset, 0-shot.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
7200
N_{frames}=7200
.

LVBench

Long video understanding
(
Wang et al., 2024
)

Test subset, 0-shot.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
1024
N_{frames}=1024
.

VideoMME

Long video understanding
(
Fu et al., 2025
)

0-shot. Audio + visual uses the Long subset of test set, audio + visual + subtitles uses full test set.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
1024
N_{frames}=1024
.

VATEX

General video captioning
(
Wang et al., 2019
)

Test subset, 4-shots. CIDEr score.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
64
N_{frames}=64
.

VATEX-ZH

Chinese video captioning
(
Wang et al., 2019
)

Validation subset, 4-shots. CIDEr score.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
64
N_{frames}=64
.

YouCook2 Cap

Instructional video captioning
(
Zhou et al., 2018
)

Validation subset, 4-shots. CIDEr score.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
256
N_{frames}=256
.

Minerva

Complex video reasoning
(
Nagrani et al., 2025a
)

Test subset, 0-shot.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
1024
N_{frames}=1024
.

Neptune

Long video understanding
(
Nagrani et al., 2025b
)

Test subset, 0-shot.
Same processing as above with
N
f
​
r
​
a
​
m
​
e
​
s
=
1024
N_{frames}=1024
.

[CAPTION] Table 11: Description of the benchmarks used, along with extra details about subsets, variants and model specifications.

[H] 8.2 Gemini Plays Pokémon Additional Details

Changing the model used by the Gemini Plays Pokémon agent had a strong effect on performance, as can be seen in Figure
4.1
.

[CAPTION] Figure 13: The model matters: Same agentic harness, different Gemini models. All runs have the same starter (Charmander). Note that measuring in units of hours also controls for the fact that each of 2.5 Flash’s actions was significantly faster (though it requires more actual actions to achieve its goals). X marks the end of gameplay and is a lower bound on the time to complete the next milestone.

[H] Additional Harness Details

[CAPTION] Figure 14: An overview of the agent harness (Zhang, 2025). The overworld fog-of-war map automatically stores a tile once explored and labels it with a visited counter. The type of tile is recorded from RAM. The agentic tools (pathfinder, boulder_puzzle_strategist) are prompted instances of Gemini 2.5 Pro. pathfinder is used for navigation and boulder_puzzle_strategist solves boulder puzzles in the Victory Road dungeon.

The Gemini Plays Pokémon agent
(
Zhang, 2025
)
receives a subset of RAM information, intended to give sufficient information to play the game, partially overlaid with a screenshot of the Game Boy screen. Gemini is prompted with a system prompt telling it that it is playing Pokémon Blue and that its goal is to beat the game, as well as descriptive information to help it understand the conventions in the translation from vision to text and a small number of general tips for gameplay. Gemini then takes actions, translated to button presses. The sequence of actions is stored in context, followed by a summary clear every 100 turns. The summaries are stored in context as well. Every 1000 turns GPP compresses the existing summaries again. Additionally, Gemini keeps track of three main goals (primary, secondary, and tertiary) as well as several additional goals (contingency plans, preparation, exploration, team composition). Every 25 turns, another prompted instance of Gemini (Guidance Gemini, or GG) observes the same context as the main Gemini and critiques performance and attempts to point out hallucinations and so on. The overworld fog-of-war map is stored in the context in XML, where coordinates which have not been seen cannot be viewed until explored. Crucially, in the system prompt, Gemini is instructed to explore. Once a tile is explored, however, the coordinate is automatically stored in the map memory and labeled with a visited counter. Tiles are also labeled by type (water, ground, cuttable, grass, spinner, etc.), and warp points to different maps are also labeled as such. Gemini also has access to two agentic tools, which are both instances of Gemini equipped with a more specialized prompt - the
pathfinder
tool, and the
boulder_puzzle_strategist
tool. In the
pathfinder
prompt, Gemini is prompted to mentally simulate a path-finding algorithm, which is left unspecified, and to verify that the path is valid against the map information available. In the
boulder_puzzle_strategist
tool, Gemini is prompted to solve special boulder puzzles that are present in Pokémon Blue in the Victory Road dungeon - these puzzles are similar to the game Sokoban - again, by mentally simulating sequences of actions that lead to solutions to the puzzle. The prompt describes the physics and the task of the boulder puzzle, as well as the desired output of solutions. The tool was added after Gemini had solved 2/4 of the puzzles in Victory Road on its own, but progress was slow on the 3rd and 4th puzzles.

[H] Additional Examples of Capabilities

[H] Long Context Agentic Tooling

The model is able to identify a complex path through a maze with auto-movement only specified by direction (Rocket Hideout spinner puzzles), solve multiple shortest path problems across multiple maps with limited resources (Safari Zone), perform maze solving on mazes with large description length (Route 13), and solve complex boulder-pushing puzzles across a multi-map 3D maze (Seafoam Islands). It is perhaps even more impressive that it appears to be possible for the model to solve these problems only with textual descriptions of the problems. On the other hand, other models, like Gemini 2.5 Flash, were not able to perform similarly long pathfinding tasks, and often failed to find simpler paths. This gap highlights the superior long context reasoning capability of Gemini 2.5 Pro (as also evidenced by other evaluations).

boulder_puzzle_strategist
is similarly impressive. The boulder puzzles in Pokémon Blue are Sokoban-like puzzles that require the player character to maneuver boulders on to switches and through holes in order to open up a pathway through a cave with multiple levels. The puzzles can become quite complex, requiring long circuitous pathways and multi-level movement in order to solve the puzzle. With only a prompt describing boulder physics and a description of how to verify a valid path, Gemini 2.5 Pro is able to one-shot some of these complex boulder puzzles, which are required to progress through Victory Road.

pathfinder
and
boulder_puzzle_strategist
are currently the only two agentic tools that the Gemini Plays Pokémon developer has implemented. In future runs, there are plans to explore tool-creation tools where the model can create new tools with only a prompt. Since most of the prompts for
pathfinder
and
boulder_puzzle_strategist
were actually written by Gemini 2.5 Pro itself, it is quite plausible that autonomous tool creation is possible for the current 2.5 Pro model.

[H] General Reasoning

Gemini 2.5 Pro is able to reason through complex game puzzles in Pokémon quite well. In this section, we present two examples.

Catching a Pokémon that is quick to flee:
In one of the runs, the Gemini 2.5 Pro agent was attempting to catch an Abra, and planned to use Pikachu’s Thunder Wave to paralyze the Abra, simultaneously making it less likely that Abra could Teleport out of the battle while also improving the catching rate. After multiple attempts, the agent caught Abra with this strategy.

Creatively escaping a softlock caused by bugs in game I/O:
On the Cycling Road, the slope forces southward movement at all times unless there is an obstacle. It turns out there are two tiles on the Cycling Road that result in a softlock as a result of this behavior. In the GPP framework, button presses are limited by time delays, and in order for a player to escape those two tiles (blocked on all sides except the north), the player would have to input a sequence of button presses more quickly than the GPP framework allows. Gemini 2.5 Pro unluckily found itself in one of these two spots – luckily, it was not a softlock, because 2.5 Pro had already taught one of its party members HM02 FLY - which allows for travel to any town it has been to. FLY is not typically used as an escape mechanism (unlike the item ESCAPE ROPE and the move DIG, both of which fail in this situation). After 4 hours of trying many approaches to escape (including movement, ESCAPE ROPE, DIG, all of which are blocked), the Gemini 2.5 Pro agent came up with the idea to use FLY to escape from the softlock successfully. This reasoning action is especially impressive since this situation can never occur in an existing game – and thus, it is certain that information from training data for this behavior has not leaked into the model’s knowledge base!

[H] Long Horizon Task Coherence

There are several additional interesting case studies of shorter planning sequences throughout Pokémon Blue that Gemini 2.5 Pro in the GPP harness was able to solve:

Training team to prepare for upcoming battles:

In one run where Gemini picked Charmander, the Fire-type starter, Gemini 2.5 Pro lost to Misty, the Water-type Gym Leader, the first time. To prepare for the rematch, Gemini 2.5 Pro spent over 24 hours leveling up a Pikachu and a Bellsprout (both super-effective against Water types) by around 25 levels in total to successfully defeat Misty.

Acquiring Hidden Moves (HMs) for game progression:

In many parts of the game, it is necessary to first acquire an HM before game progression is possible. Two examples are HM01 CUT and HM05 FLASH. Acquiring the ability to use CUT and FLASH each require four steps: 1) obtaining the HM item itself, 2) acquiring a compatible Pokémon which can learn the move, 3) adding the compatible Pokémon to the player’s team, 4) teaching the HM move to the compatible Pokémon. In many cases, each step requires many steps itself. As an example, in run 1, Gemini 2.5 Pro had to a) retrieve CUT by completing the S.S. Anne quest, b) identify a Pokémon which could learn CUT and catch it (CHOPPY the Bellsprout), c) add CHOPPY to the team and d) teach CUT. Similarly, for HM05 FLASH, Gemini 2.5 Pro had to a) first catch 10 Pokémon to fill out the Pokedex, b) backtrack to find an Aide who gives HM05 Flash, c) catch a Pokémon (ZAP the Pikachu) in Viridian Forest, use the PC to deposit a Pokémon and withdraw ZAP, d) teach HM05 FLASH to Zap.

Solving the Safari Zone:

The Safari Zone is another location with required HMs (both HM03 SURF and HM04 Strength). However, it has an extra constraint - it requires 500¥ to enter each time, and the player is limited to only 500 total steps in the Safari Zone. As a result, if the player is unable to reach the required items in the limited number of steps, the player loses 500¥ and is required to re-start! As a result, it is possible to essentially softlock if the player takes too many attempts to complete the Safari Zone. Solving the Safari Zone itself requires traversing across four different maps and not getting lost. Gemini 2.5 Pro was able to get both required HMs in 17 attempts in run 1, and in only 5 attempts in run 2.

Finding hidden keys in dungeons:

Another method of progression in Pokémon is to find hidden keys and solve complex multi-floor dungeons. In particular, in Rocket Hideout, the player must recover the LIFT KEY on the fourth basement floor (dropped after beating a specific Team Rocket Grunt) in order to unlock the elevator to find the evil Giovanni, leader of Team Rocket. In Silph Co., the player must find the CARD KEY in order to open multiple doors to find the path across eleven floors of the building to rescue the President from Giovanni. To open the seventh gym on Cinnabar Island, the player must enter the Pokémon Mansion and traverse three floors in order to find the SECRET KEY which unlocks the gym door. All of these cases require maintaining the goals over large numbers of actions and many local puzzles (like spinner puzzles in Rocket Hideout, and switch puzzles in Pokémon Mansion), in addition to maintaining the health of the Pokémon on the player’s team and managing wild encounters, trainer battles, and other items.

Puzzle solving over complex multi-level dungeons:

The Seafoam Islands contain 5 floors involving multiple boulder puzzles which require the player to navigate mazes and push boulders through holes across multiple floors using HM04 STRENGTH in order to block fast-moving currents that prevent the player from using HM03 Surf in various locations in this difficult dungeon. As a result, the player must track information across five different maps in order to both deduce the goal (push two boulders into place in order to block a specific current) as well as engage in multi-level (effectively 3D) maze solving to find the way out. It is likely the most challenging dungeon in the game. Only the second run of GPP went through Seafoam Islands, as it is not required to progress.

[H] Additional Challenges

[H] Hallucinations and Fixations on Delusions

While game knowledge can sometimes leak and be quite beneficial to the ability of the model to progress, it can also hinder the model in surprising ways due to hallucinations, delusions, and mix ups with other generations of Pokémon games. One example of this phenomenon is the TEA item. In Pokémon Red/Blue, at one point the player must purchase a drink (FRESH WATER, SODA POP, or LEMONADE) from a vending machine and hand it over to a thirsty guard, who then lets the player pass through. In Pokémon FireRed/LeafGreen, remakes of the game, you must instead bring the thirsty guard a special TEA item, which does not exist in the original game. Gemini 2.5 Pro at several points was deluded into thinking that it had to retrieve the TEA in order to progress, and as a result spent many, many hours attempting to find the TEA or to give the guard TEA.

In Run 2, the model was explicitly prompted to act as a player completely new to the game, and to disregard prior knowledge about game events, item locations, and Pokémon spawn points, in order to mitigate hallucinations from model pretraining knowledge and to also attempt to perform a cleaner test of the model’s ability to reason through the game. It appears to have at least partially worked - multiple hallucinations from other games have been avoided in the second run. On the flip side, this prompt may have also harmed the model’s ability to utilize information from its common knowledge about the game, hindering overall performance in a few critical places.

Fixations on delusions due to goal-setting and also due to the Guidance Gemini instance are not an uncommon occurrence in watching Gemini Plays Pokémon - the TEA incidence is hardly the only example of this behavior. An especially egregious form of this issue can take place with “context poisoning” – where many parts of the context (goals, summary) are “poisoned” with misinformation about the game state, which can often take a very long time to undo. As a result, the model can become fixated on achieving impossible or irrelevant goals. This failure mode is also highly related to the looping issue mentioned above. These delusions, though obviously nonsensical to a human (“Let me try to go through the entrance to a house and back out again. Then, hopefully the guard who is blocking the entrance might move.”), by virtue of poisoning the context in many places, can lead the model to ignore common sense and repeat the same incorrect statement. Context poisoning can also lead to strategies like the “black-out” strategy (cause all Pokémon in the party to faint, “blacking out” and teleporting to the nearest Pokémon Center and losing half your money, instead of attempting to leave).

[H] Topological Traps in Thinking Patterns

One recurring pattern in particularly-difficult-to-solve puzzles and mazes for Gemini 2.5 Pro consists of a “topological trap” - the topology of the reasoning graph required to solve the maze or puzzle has a distinctive shape. Namely, the desired objective appears to be nearby and easily reachable (an “attractor”), but the correct solution requires taking a detour in order to arrive at the correct solution. We observed this phenomenon in multiple parts of the game. In the spinner puzzle on B3F of Rocket Hideout
(
Zerokid, 2024
)
, the map positions both an item and the correct staircase to the south, but they are only accessible by going the long way around. The Route 13 maze has only one correct route through - the upper narrow pass. Finally, the Victory Road 3F boulder puzzle requires the player to push the boulder in the upper right all the way to the upper left switch, while ignoring the boulder puzzles, ladders, and exits to the south.

Notably, if the model is instructed to solve a given puzzle at all once (e.g., via
pathfinder
), it can manage to do so if the context length is not too long. For instance,
pathfinder
implemented with Gemini 2.5 Pro is able to solve the B3F spinner trap in one shot.

[H] Agent Panic

Over the course of the playthrough, Gemini 2.5 Pro gets into various situations which cause the model to simulate “panic”. For example, when the Pokémon in the party’s health or power points are low, the model’s thoughts repeatedly reiterate the need to heal the party immediately or escape the current dungeon (e.g., famously using the move DIG or an ESCAPE ROPE item). Quite interestingly, this mode of model performance appears to correlate with a qualitatively observable degradation in the model’s reasoning capability – for instance, completely forgetting to use the
pathfinder
tool in stretches of gameplay while this condition persists. This behavior has occurred in enough separate instances that the members of the Twitch chat have actively noticed when it is occurring.

[H] Actions vs. Game Milestones

For completeness, we plot the number of actions/steps required to achieve each game milestone (see Figure
15
). An action consists of each bucketed instance where the agent outputs a sequence of button presses to the game (note that other AI agents playing Pokémon may output different numbers of button presses per action, define what constitutes a button press differently, or define an action/step differently). However, it is important to consider action-milestone plots in conjunction with information about the time and/or cost in order to obtain the full picture about the agent’s performance.

[CAPTION] (a) The fully autonomous Run 2 milestones as a function of the number of individual actions.

[CAPTION] (b) Comparison of 2.5 Pro and 2.5 Flash in terms of actions to milestones.

[CAPTION] Figure 15: Analog of Figure 6 and 15(b), in terms of actions instead of hours.

[H] 8.3 Frontier Safety Framework Evaluations Additional Details: Frontier Safety Correctness Tests

For each testing environment, we performed basic correctness checks by looking at how the agents behaved. This involved combining AI and manual reviews of the agents’ actions to flag potential issues.

On RE-Bench, we examined the best, median and lowest scoring trajectories. For cybersecurity environments (InterCode CTFs, Internal CTFs, Hack the Box), we carefully inspected at least one successful attempt (where available) from each environment, and otherwise examined an unsuccessful attempt. We also performed checks on sample situational awareness and stealth evaluations. This involved basic spot checks to ensure that the prompt and shell outputs were correctly formatted.

We used AI assistance to monitor for obvious instances of cheating, and did not find any. For the RE-Bench tests specifically, we also looked at how the best-performing agent achieved its score to ensure that it was a plausible approach, rather than exploiting an obvious reward hack. Overall, we did not observe errors that we believe would invalidate the results of the benchmarks.

[H] 8.4 Image to Code Demo

We prompted Gemini 1.5 Pro and Gemini 2.5 Pro to generate an SVG representation of an image and found Gemini 2.5 Pro generates better reconstructions.

Prompt

Gemini 1.5 Pro

Gemini 2.5 Pro

Please convert this image into SVG and try to reconstruct the spatial arrangement of the objects.

[CAPTION] Figure 16: Comparison of Gemini 1.5 Pro and Gemini 2.5 Pro responses to image-to-SVG reconstruction prompt.

[H] 8.5 Long Context Video Understanding Demo

We prompted Gemini with the full video and the following prompt to assess its ability to find a very specific scene (which only lasts 1 second) out of the full 46 minutes video.

Video (46 minutes):
Redefining Robotics with Carolina Parada

Prompt
: What color is the T-shirt the robot arms are trying to fold, and at what timecode does this occur?

[CAPTION] Figure 17: Frame and timestamp the model is meant to recall. Taken from https://www.youtube.com/watch?v=Rgwty6dGsYI at timestamp 27:29.

See Table
12
for samples. Gemini 1.5 Pro gets the color right in 1/3 cases, and gets the timestamp in 0/3 cases (quite far away). Gemini 2.5 Pro gets the color in 3/3 cases, and gets the timestamp in 1/3 cases (remaining 2/3 are within 3 seconds close).

Model

Trial

Model response

Gemini 1.5 Pro

1

The t-shirt the robot arms are trying to fold is a dark teal or turquoise blue color. The folding begins at and continues at [11:06–11:15].

2

The T-shirt is orange, and the robot can be seen trying to fold it starting at [00:00:26–00:00:28].

3

The t-shirt being folded is dark blue and can be seen between [00:00:56–00:00:58] and [00:07:56–00:07:59].

2.5 Pro Preview 05-06

1

The T-shirt the robot arms are trying to fold is blue. This scene occurs at timecode 27:29.

2

The T-shirt the robot arms are trying to fold is blue. This occurs at timecode 27:26.

3

The T-shirt the robot arms are interacting with (as part of the "fold laundry" task mentioned by the speaker) is blue. This occurs at timestamp 0:27:28.

[CAPTION] Table 12: Gemini models samples for T-shirt folding scene recall

Experimental support, please
view the build logs
for errors. Generated by
L
A
T
E
xml
.

[H] Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue"
(
)
button, located in the page header.

Tip:
You can select the relevant text first, to include it in your report.

Our team has already identified
the following issues
. We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a
list of packages that need conversion
, and welcome
developer contributions
.

We gratefully acknowledge support from our
major funders
,
member institutions
,
, and all contributors.

About
·
Help
·
Contact
·
Subscribe
·
Copyright
·
Privacy
·
Accessibility
·
Operational Status
(opens in new tab)

Major funding support from

