SIMA 2: A Gemini-Powered AI Agent for 3D Virtual Worlds — Google DeepMind

Last year, we introduced SIMA (Scalable Instructable Multiworld Agent), a generalist AI that could follow basic instructions across a wide range of virtual environments. SIMA was a crucial first step in teaching AI to translate language into meaningful action in rich, 3D worlds.
Today we’re introducing SIMA 2, the next milestone in our research creating general and helpful AI agents. By integrating the advanced capabilities of our Gemini models, SIMA is evolving from an instruction-follower into an interactive gaming companion. Not only can SIMA 2 follow human-language instructions in virtual worlds, it can now also think about its goals, converse with users, and improve itself over time.
This is a significant step in the direction of Artificial General Intelligence (AGI), with important implications for the future of robotics and AI-embodiment in general.
The first version of SIMA learned to perform over 600 language-following skills, like “turn left,” “climb the ladder,” and “open the map,” across a diverse set of commercial video games. It operated in these environments as a person might, by “looking” at the screen and using a virtual keyboard and mouse to navigate, without access to the underlying game mechanics.
With SIMA 2, we’ve moved beyond instruction-following. By embedding a Gemini model as the agent's core, SIMA 2 can do more than just respond to instructions, it can think and reason about them.
MineDojo: SIMA 1 (left) attempts to follow the instruction while SIMA 2 (right) successfully completes the task in a game it has never seen before.
ASKA: SIMA 1 (left) attempts to follow the instruction “Find a campfire” while SIMA 2 (right) successfully completes the task in a game it has never seen before.
SIMA 2’s new architecture integrates Gemini’s powerful reasoning abilities to help it understand a user’s high-level goal, perform complex reasoning in pursuit, and skillfully execute goal-oriented actions within games.
We trained SIMA 2 using a mixture of human demonstration videos with language labels as well as Gemini-generated labels. As a result, SIMA 2 can now describe to the user what it intends to do and detail the steps it's taking to accomplish its goals.
Moving beyond simple instruction following: SIMA 2 can answer the user’s questions and also reasons about its own behavior as well as its environment.
Moving beyond simple instruction following: SIMA 2 can answer the user’s questions and also reasons about its own behavior as well as its environment.
Moving beyond simple instruction following: SIMA 2 can answer the user’s questions and also reasons about its own behavior as well as its environment.
In testing, we have found that interacting with the agent feels less like giving it commands and more like collaborating with a companion who can reason about the task at hand.
And thanks to our collaboration with our existing and new game partners (see, Acknowledgements), we have been able to train and evaluate SIMA 2 on a wider array of games.
This is the power of Gemini brought to embodied AI: a world-class reasoning engine that can now perceive, understand, and take action in complex, interactive 3D environments.
SIMA 2 interprets abstract concepts and logical commands by reasoning about its environment and the user's intent.
SIMA 2 interprets abstract concepts and logical commands by reasoning about its environment and the user's intent.
SIMA 2 interprets abstract concepts and logical commands by reasoning about its environment and the user's intent.
SIMA 2 interprets abstract concepts and logical commands by reasoning about its environment and the user's intent.
The addition of Gemini has also led to improved generalization and reliability. SIMA 2 can now understand more complex and nuanced instructions than its predecessor and is far more successful at carrying them out, particularly in situations or games on which it’s never been trained, such as the new Viking survival game, ASKA, or MineDojo - a research implementation of the popular open-world sandbox game, Minecraft.
SIMA 2 is successful at carrying out long and complex instructions.
SIMA 2 tackles a completely new game with no prior training, demonstrating impressive progress.
SIMA 2 is successful at carrying out long and complex instructions.
SIMA 2 is successful at carrying out long and complex instructions.
User is drawing a sketch on the screen.
User is drawing a sketch on the screen.
User is drawing a sketch on the screen.
See how it correctly interprets emojis to execute tasks.
See how it follows commands in different languages to execute tasks.
Moreover, its capacity to transfer learned concepts — for instance, taking its understanding of "mining" in one game and applying it to "harvesting" in another —is foundational to achieving the kind of broad generalization seen in human cognition. Indeed, as a result of this ability, SIMA 2’s performance is significantly closer to that of a human player on a wide range of tasks.
SIMA 2 can generalise actions across multiple games, including games it wasn’t trained on (like MineDojo and ASKA).
Task completion success rates for SIMA 1, SIMA 2, and humans across a set of evaluation tasks for all training game environments, showing SIMA 2 closing a significant portion of the gap to human performance. Note that the SIMA 1 performance reported here is with respect to our new, expanded, and much more difficult set of evaluations, across a wider set of environments and more complex instructions
Task completion success rates for SIMA 1 and SIMA 2 on held-out (never before seen during training) games: ASKA and MineDojo (a Minecraft research implementation).
To test the limits of SIMA 2’s generalization abilities, we combined it with another groundbreaking research project, Genie 3, which can generate new, real-time 3D simulated worlds from a single image or text prompt.
When we challenged SIMA 2 to play in these newly generated worlds, we found it was able to sensibly orient