ReCapture: Generating New Videos with Novel Camera Trajectories from a Single Input

Models & Research

The Engineer

19 Nov 2024 · 3 min read

ReCapture transforms single input videos into new cinematic experiences by generating unseen angles and movements, expanding creative possibilities beyond the original footage's constraints.

ReCapture is a groundbreaking method for generating new videos with novel camera trajectories from a single user-provided video. This technique allows you to re-render the original video from different angles and with cinematic motion, while preserving all existing scene movements. Notably, ReCapture can even hallucinate parts of the scene that were not visible in the original video.

How It Works

ReCapture addresses a significant limitation in current video generation models: they typically cannot handle user-provided videos directly. Instead, these models are trained on synthetic data and struggle with real-world inputs. ReCapture overcomes this by using a two-step process:

Generating Noisy Anchor Video:
- Multiview Diffusion Models: These models generate a noisy anchor video with the desired camera trajectory. They use multiple views of the scene to infer the 3D structure and create a plausible new perspective.
- Depth-Based Point Cloud Rendering: This approach uses depth information from the original video to render a point cloud, which is then used to generate the noisy anchor video.
Regenerating Clean Video:
- Masked Video Fine-Tuning: The noisy anchor video is refined using a fine-tuning technique that ensures temporal consistency and clarity. This step involves masking out parts of the video that need improvement and re-generating them to match the original content.

Technical Details

Multiview Diffusion Models:
- These models are trained on large datasets of multi-view images to learn the relationships between different camera angles.
- They use a diffusion process to generate new frames that are consistent with the desired camera trajectory.
- The generated anchor video is noisy but contains the correct perspective and motion.
Depth-Based Point Cloud Rendering:
- Depth information is extracted from the original video using depth estimation techniques.
- A point cloud is created, which represents the 3D structure of the scene.
- This point cloud is then rendered from the new camera angles to generate the noisy anchor video.
Masked Video Fine-Tuning:
- The noisy anchor video is fine-tuned to improve its quality and temporal consistency.
- A masking technique is used to identify and correct areas that need improvement.
- The fine-tuned model generates a clean, high-quality video with smooth transitions and consistent motion.

Gallery

The following gallery showcases the outcomes of various camera trajectories using ReCapture. You can select different camera motions by pressing the buttons below each example:

Tilt Up
Tilt Down
Pan Left
Pan Right
Zoom In (with Rotation)
Zoom Out (with Rotation)
Orbit Shot (with Rotation)

Example 1: Tilt Up

Source Video

[Source Video]

ReCapture

[ReCapture Video]

Example 2: Tilt Down

Source Video

[Source Video]

ReCapture

[ReCapture Video]

Conclusion

ReCapture represents a significant advancement in video generation, enabling the creation of new videos with novel camera trajectories from a single user-provided input. By combining multiview diffusion models and depth-based point cloud rendering with masked video fine-tuning, ReCapture can generate high-quality, temporally consistent videos that preserve the original scene motion while introducing cinematic effects.