HEADLINE: Generating 3D Objects with UV Maps Using Image Diffusion Models

In a significant advancement for 3D generative models, researchers from Simon Fraser University and City University of Hong Kong have introduced Object Images (Omages), a novel approach to generating realistic 3D objects with detailed UV maps. This method, which won the Best Paper Award at 3DV 2025, effectively converts complex 3D shapes into manageable 64x64 pixel images, making it possible to use image generation models like Diffusion Transformers for 3D shape creation.

Technical Overview

The core idea behind Omages is to encapsulate the surface geometry, appearance, and patch structures of a 3D object within a 2D image format. This addresses the challenges of geometric and semantic irregularity in polygonal meshes, which are common in high-quality human-made 3D assets but difficult to capture with traditional 3D generative models.

• UV Mapping: The process starts by UV-unwrapping 3D shapes into 1024x1024 images. This step ensures that the geometric and semantic details of the object are preserved in a 2D format.
• Downsampling: These high-resolution images are then carefully downsampled to 64x64 pixels, maintaining the essential features while reducing computational complexity.
• Flattening: The resulting omages are flattened into sequences, which can be processed by image generation models like Diffusion Transformers.

Motivation

Recent advancements in 3D generative models have been impressive, but they often treat 3D shapes as static "statues," lacking the rich geometric and semantic details found in human-made 3D assets. For example, a high-quality 3D model of a headphone with intricate parts (like the one shown below) is difficult to capture using current methods. Similarly, a pack of books standing closely together (another example provided) presents significant challenges for single-view reconstruction techniques.

• Geometric and Semantic Irregularity: The irregularity in geometric connectivity and semantic part structures is a major challenge for 3D generation. Most recent techniques require regular, tensorial input, which can be difficult to achieve with complex 3D shapes.
• Omages Solution: By packing the geometry, patch structures, and material into an image format, Omages effectively handle these irregularities. This makes it possible to use powerful image generation models for 3D shape creation.

Methodology

1. Preprocessing:

   • UV-Unwrapping: The 3D shapes are UV-unwrapped into high-resolution (1024x1024) images, preserving the geometric and semantic details.
   • Downsampling: These images are then downsampled to 64x64 pixels with special care to maintain essential features.

2. Generation:

   • The 64x64 omages are flattened into sequences and fed into a Diffusion Transformer model for generation.
   • This approach leverages the strengths of image generation models, which are highly effective at capturing complex patterns and structures.

Evaluation

The researchers evaluated their method on the ABO dataset, comparing it to recent 3D generative models. The results showed that Omages achieve point cloud FID (Fréchet Inception Distance) scores comparable to state-of-the-art methods, while naturally supporting PBR (Physically Based Rendering) material generation.

• Point Cloud FID: The generated shapes with patch structures achieved FID scores on par with recent 3D generative models.
• PBR Material Generation: Omages naturally support the generation of high-quality materials, making them suitable for use in various applications, including video games and CGI.

Conclusion

Object Images (Omages) represent a significant step forward in 3D generative modeling. By converting complex 3D shapes into manageable 2D images, this approach addresses the challenges of geometric and semantic irregularity, enabling the use of powerful image generation models for 3D shape creation. The results on the ABO dataset demonstrate the effectiveness of Omages, making them a promising tool for practitioners in fields such as 3D modeling, animation, and interactive applications.