DSO: Enhancing 3D Generators with Simulation Feedback for Physical Soundness

DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness

By Ruining Li, Chuanxia Zheng, Christian Rupprecht, and Andrea Vedaldi
Visual Geometry Group, University of Oxford
ICCV 2025 (Highlight)

[Paper][1] | [Code][2] | [Evaluation Data][3]

Abstract

Most 3D object generators prioritize aesthetic quality over physical constraints, which can be a significant drawback in practical applications. One crucial constraint is that the generated 3D objects should remain stable under gravity. Previous methods used differentiable physics simulators to optimize geometry at test-time, but this approach is slow, unstable, and prone to local optima.

To address these issues, we introduce Direct Simulation Optimization (DSO), a framework that leverages feedback from a non-differentiable simulator to increase the likelihood of generating stable 3D objects. By constructing a dataset of 3D objects labeled with stability scores and fine-tuning the generator using these scores, DSO achieves faster and more reliable results compared to test-time optimization. Notably, DSO can operate without ground-truth 3D objects for training, allowing the model to self-improve through simulation feedback.

Motivation

Recent advancements in image-to-3D reconstruction have focused on enhancing the quality of the generated 3D geometry and appearance, but often overlook physical soundness. For instance, state-of-the-art generators like TRELLIS and Hunyuan3D 2.0 can fail to produce stable objects even when given images of inherently stable items. The failure rate is 15% for training data and significantly higher for new objects, such as clocks and motorcycles.

Stability is a fundamental property of natural and man-made objects and is crucial in various applications, including fabrication and simulation. Therefore, it is essential to develop methods that ensure the physical soundness of reconstructed 3D objects.

Method

Overview

The DSO framework consists of several key steps:

1. Data Collection: We start by generating a dataset of 3D objects using a base model ( p_{\text{ref}} ). Each object is labeled with a stability score obtained from a physics simulator.
2. Fine-Tuning: The 3D generator is fine-tuned using the stability scores as an alignment metric. This can be done through either direct preference optimization (DPO) or direct reward optimization (DRO), both of which are novel objectives introduced in this work.

Data Collection

• Base Model Generation: We use a pre-trained 3D generator ( p_{\text{ref}} ) to generate a set of 3D objects from image prompts.
• Stability Scoring: Each generated object is evaluated using a physics simulator to determine its stability score. The stability score reflects how well the object can remain balanced under gravity.

Fine-Tuning

• Direct Preference Optimization (DPO): DPO involves training the generator to maximize the probability of generating objects with high stability scores.
• Direct Reward Optimization (DRO): DRO introduces a novel objective that aligns diffusion models without requiring pairwise preferences. This makes it more efficient and scalable.

Experiments

We conducted extensive experiments to evaluate the effectiveness of DSO. The results demonstrate that:

• Speed: The fine-tuned feed-forward generator, using either DPO or DRO, is significantly faster than test-time optimization.
• Stability: The generated objects are much more likely to be stable under gravity.
• Self-Improvement: DSO can operate without ground-truth 3D objects, allowing the model to improve over time by automatically collecting simulation feedback.

Conclusion

DSO represents a significant step forward in aligning 3D generators with physical constraints. By leveraging non-differentiable simulators and novel optimization techniques, DSO ensures that generated 3D objects are not only visually appealing but also physically sound. This framework has the potential to revolutionize applications in fabrication, simulation, and other domains where stability is crucial.

Source: ruiningli.com