On Tuesday, Google DeepMind unveiled Genie 4, its most advanced world model to date, as a research preview. This cutting-edge model creates incredibly realistic and interactive virtual environments, enabling users to alter scenes in real-time by inputting prompts during simulations. Furthermore, Genie 4 produces synchronized real-time audio, encompassing music, background sounds, and conversational elements, to complement the simulated world.
The introduction of Genie 4 follows Runway’s launch of GWM Worlds 2, their second world model, a few months prior, which also offers interactive gameplay and audio generation. Concurrently, the world model sector is seeing significant consolidation, with AMD set to acquire Fei-Fei Li’s World Labs, Amazon reportedly negotiating to purchase Decart, and Rivian recently acquiring Third Dimension to integrate simulation features for its autonomous vehicle development.
With Genie 4, Google aims to maintain its competitive edge in this rapidly evolving and crowded domain.
Over a year ago, Google introduced Genie 3 as a research preview. Genie 4 surpasses its predecessor by offering significantly enhanced photorealism and integrated audio. It also supports dynamic, freeform modifications such as real-time weather and camera angle adjustments, or the addition of new individuals and items. The new model can sustain a coherent scene for up to two minutes, twice the duration of Genie 3, while also featuring improved physics and more lifelike characters.
Google’s AI model, Gemini, is now more tightly integrated with Genie 4, functioning as a director that devises actions and verifies frames as Genie generates the visual content. An experiment involving Gemini Robotics demonstrated how a Gemini 4 Argon model could decompose tasks into sequential steps, which Genie 4 then simulated instantaneously. This research highlights a potential method for testing robotic operations in a virtual environment before their real-world execution.
During a briefing, Tejas Kulkarni, research director at Google, predicted, “We will progressively witness hybrid world models in the future, where Gemini combines code and text generation, and Genie processes this information instantly to simulate reality across various levels of abstraction.”
Similar to other world models, proposed commercial applications span gaming, interactive education, and robotics. An imaginative instance of interactive learning could involve stepping into a historical painting and engaging in conversation with its depicted figures about the events shown.
However, “trusted testers,” who are granted early access to Google’s innovative products, are identifying a novel application: film prototyping.
During a press briefing, Diego Rivas, a Google DeepMind engineer, explained that “the real-time capability proves most beneficial in creative domains where maintaining a continuous flow is highly advantageous. This includes, for instance, during the ideation phase, storyboarding, or any situation demanding quick conceptualization.”
Additionally, Google provides Veo, an AI video generation model capable of producing high-definition clips with integrated synchronized audio from text, image, or video inputs, a tool gaining traction among filmmakers.
Rivas further noted that, eventually, users might leverage Gemini Omni, a multimodal large language model that generates video from images, audio, and text, to transform these prototypes into final creative assets, though he cautioned that these functionalities are still distant.
Throughout the briefing, Rivas emphasized that Genie 4 is in its nascent stages, and the complete user experience remains undefined. The team also conceded that Genie 4 has limitations: details and objects may become inconsistent during extended simulations, and challenges persist in ensuring audio accuracy and the legibility of in-world text.
Currently, Genie 4 is accessible as a research preview for a select group of trusted testers. In the upcoming weeks, the model will be deployed to Google AI Ultra subscribers within Project Genie, Google’s AI platform for creating interactive worlds from text or image prompts, as well as to early access partners of the Gemini API and Google AI Studio.