Align3R: Temporally Consistent Monocular Depth Estimation for Dynamic Videos

Introduction

Monocular depth estimation has seen significant advancements, enabling high-quality depth predictions from single images. However, maintaining temporal consistency across video frames remains a challenge. Recent methods have addressed this by using computationally expensive video diffusion models that produce scale-invariant depth values without camera poses. The new Align3R model, presented at CVPR 2025, tackles this issue by aligning monocular depth maps and reconstructing both depth and camera poses.

Technical Overview

Key Components

• DUSt3R Model: A foundational model for dynamic scene understanding.
• Fine-Tuning with Monocular Depth: Enhancing DUSt3R with additional monocular depth inputs.
• Global Alignment: Ensuring consistent depth maps, camera poses, and point clouds across frames.

How It Works

1. Input Frames:

   • Two consecutive frames from a video are fed into the system.

2. ViT-Based Encoder and Decoder:

   • A Vision Transformer (ViT) encoder processes the input frames to extract features.
   • These features are then passed through a ViT-based decoder to predict pairwise point maps.

3. Monocular Depth Estimation:

   • An external monocular depth estimator generates initial depth maps for each frame.
   • These depth maps are further processed by a new ViT-based encoder to extract additional features.

4. Feature Injection:

   • The features from the new ViT-based encoder are injected into the DUSt3R decoder, which has zero convolution layers.
   • This step enhances the depth estimation with context-aware information from the dynamic scenes.

5. Global Alignment:
   • During inference, global alignment techniques ensure that the depth maps, camera poses, and point clouds remain consistent across all frames.
   • This is crucial for maintaining temporal coherence in dynamic videos.

Performance and Results

DAVIS Dataset

The Align3R model was evaluated on the DAVIS dataset, a benchmark for video object segmentation. The results demonstrate superior performance compared to baseline methods:

• Input Video: Original video frames.
• DUSt3R: Depth maps generated by the DUSt3R model without additional monocular depth inputs.
• MonST3R: Another baseline method.
• Ours w/ Depth Pro: Align3R with the proposed enhancements.

Key Findings

• Temporal Consistency: Align3R maintains consistent depth maps and camera poses across frames, addressing a key limitation of previous methods.
• Efficiency: The approach is more computationally efficient compared to video diffusion models, making it suitable for real-time applications.
• Accuracy: Extensive experiments show that Align3R outperforms baseline methods in terms of depth accuracy and temporal consistency.

Conclusion

Align3R represents a significant step forward in monocular depth estimation for dynamic videos. By leveraging the DUSt3R model and enhancing it with global alignment techniques, this method ensures both high-quality depth maps and consistent camera poses. This makes Align3R a valuable tool for applications ranging from autonomous driving to augmented reality.