SynCity: Training-Free 3D World Generation with Text Prompts

SynCity, a new research project from the Visual Geometry Group at the University of Oxford, introduces a groundbreaking method for generating complex and immersive 3D worlds using text prompts. Unlike traditional approaches that require extensive training or optimization, SynCity leverages pre-trained models to create detailed environments on the fly.

Technical Overview

The key innovation in SynCity is its training-free approach, which combines two powerful pre-trained models: Flux for 2D image generation and TRELLIS for 3D geometry. Here’s a step-by-step breakdown of how it works:

• 2D Prompting: Each tile in the world is generated as a 2D image. The system first renders a view of where the new tile should be placed, taking into account context from adjacent tiles to ensure consistency.
• 3D Conversion: The 2D image is then converted into a 3D model using TRELLIS. To improve blending with neighboring tiles, SynCity adds a wider base under the tile before conversion.
• 3D Blending: The 3D models are often not well-integrated with the rest of the scene initially. To address this, SynCity renders views of the new tile next to each adjacent tile and uses an image inpainting model to fill in the gaps. This refined view is then used to condition the final blending of the 3D tiles.

Architecture Details

• Flux: A pre-trained 2D image generator that ensures artistic diversity and contextual coherence.
• TRELLIS: A pre-trained 3D geometry generator that provides accurate and detailed 3D models.
• Tile-Based Rendering: The world is built incrementally in a grid, with each tile being generated and blended individually.

Implementation Notes

• Contextual Consistency: To maintain consistency across tiles, the system uses context from adjacent tiles during both 2D prompting and 3D blending. This ensures that the final 3D environment is coherent and navigable.
• Image Inpainting: The image inpainting model is crucial for refining the regions between tiles, ensuring seamless transitions in the 3D space.

Results

The results from SynCity are impressive, demonstrating rich detail and immersive environments. Here are a few key points:

• Rich Detail: The generated worlds feature intricate details that make them feel realistic and engaging.
• Immersive Nature: Users can fully explore the 3D worlds, with smooth transitions between tiles and a consistent visual experience.
• Visual Enhancements: A sky box has been added to enhance the visual appeal of the generated environments.

Exploring Generated Worlds

SynCity’s 3D worlds can be explored in full detail. The following videos showcase example trajectories that highlight the rich detail and immersive nature of the generations:

• Video 1
• Video 2
• Video 3

Conclusion

SynCity represents a significant step forward in the field of 3D world generation. By leveraging pre-trained models and a tile-based approach, it achieves high-quality results without the need for extensive training or optimization. This makes it an attractive solution for practitioners looking to quickly generate detailed and immersive 3D environments.