NoPoSplat: Real-Time 3D Gaussian Reconstruction from Unposed Sparse Images

NoPose, NoProblem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images

At ICLR 2025 (Oral, top 1.8%), a team of researchers from ETH Zurich, NVIDIA, Microsoft, and UC Merced introduced NoPoSplat, a groundbreaking feed-forward model for 3D scene reconstruction using 3D Gaussians from unposed sparse multi-view images. This method, trained solely with photometric loss, achieves real-time 3D Gaussian reconstruction during inference, making it highly practical for real-world applications.

Key Contributions

• Real-Time Reconstruction: NoPoSplat can reconstruct 3D scenes in real-time without requiring accurate camera poses.
• Canonical Space Anchoring: By anchoring one input view's local camera coordinates as the canonical space, the model avoids errors associated with per-frame Gaussians and pose estimation.
• Intrinsic Embedding: A novel intrinsic embedding method converts camera intrinsics into token embeddings, enabling accurate scene scale prediction.
• Superior Performance: Experimental results show that NoPoSplat outperforms state-of-the-art methods in novel view synthesis (NVS) and pose estimation, especially in scenarios with limited input image overlap.

Method Overview

Overall Framework

NoPoSplat's architecture is designed to handle sparse unposed images by directly reconstructing 3D Gaussians in a canonical space. Here’s a breakdown of the key components:

• Canonical Space: One input view's local camera coordinates are used as the canonical space, simplifying the reconstruction process.
• Feed-Forward Network: The model is trained using photometric loss to predict Gaussian primitives for all views within this canonical space.
• Intrinsic Embedding: Camera intrinsics are converted into token embeddings and concatenated with image tokens as input to the network. This helps in resolving scale ambiguity.

Two-View Setup Example

For simplicity, let’s consider a two-view setup:

1. Input Preparation: Sparse unposed images are fed into the network.
2. Canonical Space Anchoring: One view's camera coordinates serve as the canonical space.
3. Gaussian Prediction: The network predicts 3D Gaussians for all views within this canonical space.
4. Intrinsic Embedding: Camera intrinsics are converted to token embeddings and concatenated with image tokens.
5. Output: Reconstructed 3D Gaussians are used for novel view synthesis and pose estimation.

Quantitative Comparisons on NVS

Novel View Synthesis Performance

NoPoSplat was evaluated on the RealEstate10k dataset, a challenging benchmark for novel view synthesis. The results demonstrate its superior performance compared to previous pose-free methods:

• DUSt3R: NoPoSplat outperforms DUSt3R in all overlap settings.
• Splatt3R: NoPoSplat also surpasses Splatt3R, particularly in scenarios with limited input image overlap.

Pose Estimation

NoPoSplat’s two-stage coarse-to-fine pipeline for pose estimation is another significant contribution. Trained without ground truth depth or explicit matching loss, the method significantly outperforms state-of-the-art methods:

• Coarse Stage: Initial pose estimation using reconstructed 3D Gaussians.
• Fine Stage: Refinement of poses to achieve higher accuracy.

Implementation Details

• Training Data: The model is trained on photometric loss, which measures the difference between predicted and actual pixel values.
• Inference Time: Real-time reconstruction is achieved during inference, making NoPoSplat suitable for dynamic environments.
• Architecture: A feed-forward network with a novel intrinsic embedding mechanism ensures efficient and accurate 3D Gaussian prediction.

Conclusion

NoPoSplat represents a significant advance in pose-free generalizable 3D reconstruction. By simplifying the reconstruction process and addressing scale ambiguity, it achieves superior performance in novel view synthesis and pose estimation tasks. This work has broad implications for real-world applications, from augmented reality to robotics.