OccSora: A 4D Occupancy Generation Model for Autonomous Driving World Simulation

In a recent paper, researchers from Beihang University, UC Berkeley, and Tsinghua University introduced OccSora, a novel diffusion-based model designed to generate high-quality 4D occupancy scenes for autonomous driving. Unlike traditional autoregressive models that predict the next token in a sequence, OccSora leverages a diffusion framework to efficiently model long-term temporal evolutions. This approach not only enhances the realism of generated scenes but also provides a robust foundation for decision-making in autonomous vehicles.

Key Contributions

• Diffusion-Based 4D Occupancy Generation: OccSora uses a diffusion model to generate 4D occupancy scenes, which include spatial and temporal dimensions.
• 4D Scene Tokenizer: A compact tokenizer compresses 4D scenes into discrete representations, enabling efficient processing and high-quality reconstruction.
• Trajectory Conditioning: The model can generate 16-second videos with authentic 3D layouts and temporal consistency, conditioned on a given trajectory.

Architecture Details

4D Occupancy Scene Tokenizer

The 4D occupancy scene tokenizer is the first component of OccSora. Its primary function is to encode and compress 4D scenes into high-dimensional features, which are then decoded to retrieve spatiotemporal physical characteristics. This tokenizer:

• Encodes: Converts a 4D occupancy input (spatial coordinates + time) into a compact discrete representation.
• Compresses: Reduces the dimensionality of the scene while preserving essential information.
• Decodes: Reconstructs the original spatiotemporal characteristics from the compressed representation.

Diffusion-Based World Model

The diffusion-based world model is the core of OccSora. It uses a transformer architecture to generate 4D occupancy scenes conditioned on a trajectory prompt. The process involves:

• Tokenization: Converting 4D occupancy into a sequence of tokens using the optimal codebook obtained from training the 4D scene tokenizer.
• Denoising Training: Combining these tokens with an ego vehicle trajectory and random noise to train the model for denoising.
• Generation: Generating high-quality 4D scenes that adhere to physical logic and are controllable based on different trajectories.

Results

OccSora demonstrates significant advancements in generating realistic 4D occupancy scenes for autonomous driving. Key results include:

• High-Quality Reconstruction: The 4D scene tokenizer can compress long video sequences into compact representations, which are then accurately reconstructed.
• Controllable Scene Generation: The model can generate 16-second videos with authentic 3D layouts and temporal consistency, conditioned on various trajectories.
• Physical Logic Adherence: Generated scenes respect physical constraints and exhibit realistic behavior, making them suitable for training and testing autonomous driving systems.

Visualization of Results

• Reconstruction: Direct compression of long video sequences into spatial-temporal scene representations, followed by accurate reconstruction.
• Trajectory Video Generation: Generating 4D occupancy scenes that align with given trajectories, showcasing the model's ability to understand and simulate complex driving scenarios.

Why It Matters for Practitioners

For practitioners in autonomous driving, OccSora offers a powerful tool for simulating real-world driving conditions. The ability to generate high-fidelity 4D scenes can significantly enhance the training and testing of decision-making algorithms, leading to more robust and reliable autonomous vehicles. Additionally, the controllable nature of the generated scenes allows for tailored simulations that can cover a wide range of driving scenarios, improving overall system performance.