MVPaint: Synchronized Multi-View Diffusion for High-Fidelity 3D Texturing

MVPaint, a novel framework for 3D texturing, addresses the longstanding issues of local discontinuities and multi-view inconsistencies in existing Text-to-Texture (T2T) generation methods. Developed by researchers from StepFun, Shanghai AI Laboratory, Tencent PCG, S-Lab NTU, Tsinghua University, AGI Lab at Westlake University, and ByteDance, MVPaint introduces a synchronized multi-view diffusion approach to generate high-resolution, seamless textures with enhanced cross-view consistency.

Key Challenges in 3D Texturing

Traditional T2T methods often suffer from several limitations:

• Local Discontinuities: Inconsistencies within small regions of the texture.
• Multi-View Inconsistencies: Differences in texture quality and appearance across different views of the same 3D model.
• UV Unwrapping Dependence: Heavy reliance on UV unwrapping, which can introduce artifacts and discontinuities.

MVPaint Framework

MVPaint is designed to overcome these challenges through a three-stage process:

1. Synchronized Multi-view Generation (SMG):

   • Input: A 3D mesh model.
   • Process: The SMG module generates multiple views of the 3D model simultaneously, producing coarse texturing results with unpainted parts due to missing observations.
   • Output: Initial multi-view images with basic textures.

2. Spatial-aware 3D Inpainting (S3I):

   • Input: Coarse texturing results from SMG.
   • Process: The S3I method focuses on completing the 3D texturing by effectively texturing previously unobserved areas. This ensures that all parts of the 3D model are fully textured.
   • Output: Complete and consistent 3D textures.

3. UV Refinement (UVR):

   • Input: Textured 3D model from S3I.
   • Process: The UVR module improves texture quality in the UV space through two steps:
     • UV-space Super-Resolution: Enhances the resolution of the UV map to produce higher-quality textures.
     • Spatial-aware Seam-Smoothing Algorithm: Revises spatial texturing discontinuities caused by UV unwrapping, ensuring a seamless final result.
   • Output: High-fidelity textures with minimal Janus issues and enhanced cross-view consistency.

Evaluation Benchmarks

To evaluate the performance of MVPaint, the researchers established two T2T evaluation benchmarks:

• Objaverse T2T Benchmark: Based on high-quality 3D meshes from the Objaverse dataset.
• GSO T2T Benchmark: Using the entire GSO dataset.

Extensive experimental results demonstrate that MVPaint outperforms existing state-of-the-art methods. Notably, MVPaint generates textures with minimal Janus issues (inconsistencies between different views) and highly enhanced cross-view consistency.

Implementation Details

• SMG Model: Utilizes a multi-view diffusion model to generate initial texturing results.
• S3I Method: Employs a spatial-aware inpainting technique to fill in missing textures.
• UVR Module: Combines UV-space super-resolution with a seam-smoothing algorithm to refine the final texture.

Conclusion

MVPaint represents a significant advancement in 3D texturing by addressing key challenges such as local discontinuities, multi-view inconsistencies, and UV unwrapping artifacts. With its synchronized multi-view diffusion approach and advanced refinement techniques, MVPaint sets a new standard for high-fidelity 3D texture generation.