Progressive Rendering Distillation: Efficient Text-to-Mesh Generation with Stable Diffusion and Minimal 3D Data

In a significant advancement for text-to-3D generation, researchers from The Hong Kong Polytechnic University and the Chinese Academy of Sciences have introduced Progressive Rendering Distillation (PRD). This novel training scheme adapts the popular Stable Diffusion (SD) model to generate high-quality 3D meshes directly from text prompts, all while requiring only a minimal increase in parameters and no explicit 3D data.

What Changed

The key innovation here is the ability to train a 3D generator without needing extensive 3D ground-truth data. Traditional approaches to generating 3D models from text often suffer from poor quality due to the scarcity of high-quality 3D training data. PRD overcomes this by distilling knowledge from multi-view diffusion models, effectively teaching SD to generate 3D meshes in a few steps.

How It Works

• Progressive Denoising: In each training iteration, PRD uses the U-Net component of SD to progressively denoise the latent space from random noise. This process is broken down into several steps.
• Multi-View Distillation: At each step of the denoising process, the model decodes the denoised latent into a 3D output. Multi-view diffusion models like MVDream and RichDreamer are used to distill text-consistent textures and geometries into these 3D outputs.
• Efficiency and Quality: By leveraging this progressive rendering approach, PRD can generate high-quality 3D meshes in just a few steps, significantly reducing the computational overhead compared to traditional methods.

Implementation Details

• Model Architecture:
  • U-Net for Denoising: The U-Net architecture is crucial for the denoising process. It progressively refines the latent representation from noise.
  • Triplane Representation: PRD adapts SD to generate a Triplane, which is a set of three orthogonal planes that encode the 3D geometry and texture information.
• Training Data:
  • No Explicit 3D Ground-Truths: One of the most significant advantages of PRD is its ability to train without explicit 3D ground-truth data. This makes it easier to scale up training datasets, leading to better generalization for challenging text prompts.
• Performance Metrics:
  • Inference Speed: TriplaneTurbo, the model trained using PRD, can generate high-quality 3D meshes in just 1.2 seconds.
  • Parameter Efficiency: The adaptation of SD to support Triplane generation adds only 2.5% extra trainable parameters.

Benchmarks and Results

• Quality: TriplaneTurbo outperforms previous text-to-3D generators in both efficiency and quality. It can produce high-quality 3D meshes that generalize well for a wide range of text inputs, including those with creative and challenging concepts.
• Speed: The ability to generate 3D models in just 1.2 seconds is a significant improvement over existing methods, making it practical for real-time applications.

Why It Matters

For practitioners in the field of AI and 3D generation, PRD offers several key benefits:

• Data Efficiency: By eliminating the need for extensive 3D ground-truth data, PRD makes it easier to train high-quality 3D generators.
• Performance: The combination of efficiency and quality makes TriplaneTurbo a compelling choice for applications requiring rapid and accurate 3D model generation from text.

Conclusion

Progressive Rendering Distillation (PRD) represents a significant step forward in the field of text-to-3D generation. By leveraging multi-view diffusion models and progressive denoising, PRD enables the efficient training of high-quality 3D generators without the need for explicit 3D data. This opens up new possibilities for real-time applications and creative use cases.