ICYMI: Google DeepMind previews Genie 4 with native audio

Google DeepMind has introduced Genie 4, a system that creates realistic digital worlds in real time. These worlds have smooth movement and can include background sounds, music-like audio, and spoken conversations. Gemini Omni helps make the worlds look and behave like the real world, while other Gemini models help keep everything consistent. DeepMind sees Genie 4 as a tool for training robots and AI agents, as well as for creating new kinds of interactive experiences.

Users can move around these worlds with keyboard controls and change what happens by typing instructions. For example, they can change the weather, move the camera, or add a person or object without starting over. Gemini understands each request based on what is happening in the scene. It also keeps track of the world and guides what happens between user commands.

DeepMind says Genie 4 can model how air moves around objects, how things collide, how people move naturally, and how they react to their surroundings. Characters can also be controlled. Genie 4 is the company's first world model that works in real time and creates audio as part of the experience. Its sounds can include wind, waves, and conversations that change in response to instructions or events.

For tasks that involve several steps, Gemini 3.8 Flash acts as a guide. It watches the scenes, breaks a goal into smaller steps, checks progress and results, and changes its plan if the world changes or the user steps in. DeepMind is also testing Gemini 4 Argon with Genie 4 for robotics. In one demonstration, Argon's task plan becomes live captions that Genie uses to create a scene. In another, an Apptronik Apollo humanoid robot plays checkers.

The research preview is being made available to trusted testers working in creative industries, journalism, education, gaming, and engineering. Tests include planning film scenes before filming, creating simulations from old 2D photographs, developing game ideas, and exploring business uses. These are experiments, not products available to everyone. Access for consumers and developers has not yet launched. Google AI Ultra subscribers will soon be able to use Genie 4 through Project Genie. Some developers will also get early access through the Gemini API and Google AI Studio. A waitlist is available for Project Genie.

DeepMind says videos and audio made by Genie 4 include SynthID watermarks to help identify AI-generated content. The system also has safety tests and safeguards. However, it is still experimental. Visual details and the continued presence of objects can become inconsistent during long sessions, repeated complex edits, or movement through a scene. Other challenges include realistic physical contact, sound quality, readable text, accurate space and distance, and keeping the world consistent over time. DeepMind's long-term goal is to combine Gemini's ability to plan how events and scenes fit together over time with Genie's ability to create images, sounds, and responses immediately.

Sources and related context

• Genie 3: A new frontier for world models: DeepMind's earlier announcement explains the real-time interactive world model that came before Genie 4.
• Project Genie: Experimenting with infinite, interactive worlds: Google's January announcement explains the existing Project Genie prototype and how Google AI Ultra subscribers can access it.

Source