GenEx: Mental Exploration and 3D World Generation for Embodied AI Agents

Introduction

Planning with partial observation is a fundamental challenge in embodied AI. Traditional approaches often involve agents physically exploring their environment to update their beliefs about the world state. However, humans can mentally explore unseen parts of the world and revise their beliefs based on imagined observations. This ability allows them to make more informed decisions without having to physically navigate first. To bridge this gap, we introduce the Generative World Explorer (GenEx), a video generation model that enables AI agents to mentally explore large-scale 3D environments and acquire imagined observations.

GenEx Architecture

To support mental exploration, GenEx is designed as a variant of a video generation model. Specifically, it is defined as ( f_\theta: \mathbb{R}^{H \times W} \rightarrow \mathbb{R}^{T \times H \times W} ), where the input is an image with height ( H ) and width ( W ), and the output is a sequence of ( T ) images representing a predicted video. The goal is to generate realistic 360-degree panoramic video sequences that simulate forward movement through any environment.

• Diffusion Model: GenEx leverages image-to-video diffusion models to create these sequences.
• Spherical Consistency: To ensure the generated videos are consistent with spherical navigation, we introduce a spherical consistent learning approach. This ensures that the panoramic views remain coherent as the agent moves and turns.

GenEx Navigation

Once trained, the diffuser acts as a world explorer, supporting navigation in an imagined environment. We use large language models (LLMs) like GPT-4 to control the agent's movements through an egocentric view. The agent can move forward or change direction:

• Forward Movement: Generated by the video generator.
• Angle Changes: Handled by the panoramic image.

The agent can perform unlimited actions, making our diffuser a generative world explorer that can navigate complex 3D environments without physical constraints.

Exploration Consistency

To ensure high-quality generation and navigational consistency, GenEx employs navigational cycle consistency. This involves navigating a randomly sampled closed path and returning to the origin. In optimal scenarios, the start and end views are identical, ensuring that the world model remains consistent.

GenEx for Embodied AI

GenEx significantly enhances the intelligence of embodied AI agents by enabling them to make more informed decisions through mental exploration. Here are a few practical examples:

• Single-Agent Scenario: Imagine driving down a street when you hear a siren but can’t see the source. With GenEx, the agent projects what’s ahead and reveals an ambulance just around the corner. You stop crossing the street to make space for it to pass.
• Multi-Agent Scenario: Consider waiting at a red light where you see a pedestrian and an approaching car. Everything seems fine until GenEx helps you recognize that the pedestrian can’t see the car, and vice versa. Your vehicle is blocking their views. Realizing this, you act immediately to warn them, preventing a potential collision.

Interactive Demo

To experience the capabilities of GenEx firsthand, check out our interactive demo:

<Your browser does not support the video tag>

Instructions:

• Use the controls to navigate through the environment.
• Observe how the agent uses mental exploration to make informed decisions.