GauSTAR: Gaussian Surface Tracking and Reconstruction for Dynamic Scenes with Topology Changes

Gaussian Splatting has been a game-changer for rendering static 3D scenes, but handling dynamic scenes-especially those with topology changes like surfaces appearing or disappearing-has remained a challenge. Enter GauSTAR (Gaussian Surface Tracking And Reconstruction), a novel method developed by researchers at ETH Zürich and HKUST, which addresses these issues head-on.

What Changed Technically?

• Dynamic Surface Handling: GauSTAR introduces two key mechanisms to handle dynamic surfaces:

  • Consistent Tracking for Stable Surfaces: For regions with consistent topology, GauSTAR maintains the mesh topology and tracks the meshes using Gaussians.
  • Dynamic Surface Generation: For newly appearing geometry, GauSTAR adaptively unbinds Gaussians from the mesh, enabling accurate registration and generation of new surfaces.

• Surface-Based Scene Flow: A surface-based scene flow method is introduced to provide robust initialization for tracking between frames. This ensures that the system can accurately predict how surfaces move and change over time.

Why It Matters

For practitioners working with 3D rendering and dynamic scenes, GauSTAR offers several significant advantages:

• Photo-Realistic Rendering: The method enables high-fidelity rendering of dynamic objects, making it suitable for applications like virtual reality (VR), augmented reality (AR), and film production.
• Accurate Surface Reconstruction: By binding Gaussians to mesh faces, GauSTAR can accurately reconstruct surfaces even when they undergo topology changes.
• Reliable 3D Tracking: The consistent tracking mechanism ensures that the system can maintain a stable representation of dynamic objects over time.

Method Overview

GauSTAR operates on multi-view captures and processes each frame sequentially. Here’s a breakdown of the key steps:

1. Scene Flow Warping:

   • The previous frame's result is warped using scene flow to predict the initial position of surfaces in the current frame.

2. Fixed-Topology Reconstruction:

   • For regions with consistent topology, GauSTAR reconstructs Gaussian Surfaces (Gaussian-attached mesh) by maintaining the mesh topology and tracking the meshes using Gaussians.

3. Adaptive Unbinding for Topology Changes:

   • When topology changes occur, such as surfaces appearing or disappearing, GauSTAR adaptively unbinds Gaussians from the mesh. This allows for accurate registration and generation of new surfaces based on optimized Gaussians.

4. Surface-Based Scene Flow Initialization:

   • A surface-based scene flow method is used to initialize tracking between frames. This ensures that the system can accurately predict how surfaces move and change, providing robust initialization for the next frame.

Experiments and Results

Experiments demonstrate that GauSTAR effectively tracks and reconstructs dynamic surfaces, even in challenging scenarios with topology changes. The method has been tested on a variety of dynamic scenes, showing its versatility and reliability. Key applications include:

• Virtual Reality (VR): High-fidelity rendering of dynamic environments.
• Augmented Reality (AR): Real-time tracking and reconstruction of objects in the real world.
• Film Production: Accurate representation of complex, moving surfaces.

Conclusion

GauSTAR represents a significant step forward in handling dynamic 3D scenes with topology changes. By combining consistent tracking for stable surfaces and adaptive unbinding for new geometry, GauSTAR enables photo-realistic rendering, accurate surface reconstruction, and reliable 3D tracking. For more details, including the paper, supplementary materials, video, and code, check out the project page.