InstantDrag: Real-Time Drag-Based Image Editing Without Optimization

InstantDrag: Real-Time Drag-Based Image Editing Without Optimization

By Joonghyuk Shin, Daehyeon Choi, and Jaesik Park
Seoul National University & POSTECH
SIGGRAPH Asia 2024

arXiv | Code | BibTex

Overview

Drag-based image editing has become a popular technique for its interactive and precise nature. However, existing methods often struggle with performance, requiring computationally intensive per-image optimization or additional inputs like masks and text prompts. This can significantly hinder the interactivity of the editing process.

InstantDrag, introduced by researchers from Seoul National University and POSTECH, addresses these challenges by providing an optimization-free pipeline that performs drag edits in about a second using minimal inputs. The key innovation lies in decomposing the task into two subtasks: motion generation and motion-conditioned image generation. This approach not only enhances interactivity but also maintains photo-realistic quality.

Key Features

• Optimization-Free: No need for per-image optimization, making it faster and more efficient.
• Minimal Inputs: Requires only an image and a drag instruction, eliminating the need for masks or text prompts.
• Real-Time Performance: Edits are performed in about a second, suitable for interactive applications.

Inference Pipeline

1. FlowGen (Drag-Conditioned Optical Flow Generator)

   • Task: Generate dense optical flow from sparse user drag inputs.
   • Design: Conceptualized as a translation problem, mapping an RGB image with drag instructions (sparse flow) to a dense optical flow.
   • Training: Uses a Pix2Pix-like framework, combining adversarial and reconstruction losses. Trained from scratch on real-world video datasets.

2. FlowDiffusion (Optical Flow-Conditioned Diffusion Model)

   • Task: Edit the original image using the dense optical flow as guidance.
   • Design: A diffusion model conditioned on the optical flow to generate the final edited image.
   • Training: Trained to conditionally generate images based on the provided optical flow, ensuring that the edits are photo-realistic and coherent with the original content.

Model Architectures

• FlowGen

  • Input: 5-channel tensor (3 RGB channels + 2 sparse flow channels).
  • Output: Dense optical flow.
  • Training Losses:
    • Adversarial loss to ensure realistic flow generation.
    • Reconstruction loss to ensure the generated flow accurately reflects the user's drag inputs.

• FlowDiffusion

  • Input: Original image + dense optical flow.
  • Output: Edited image.
  • Training Techniques:
    • Conditioned on the dense optical flow to guide the diffusion process.
    • Uses a combination of loss functions to ensure high-quality, photo-realistic edits.

Experiments and Results

• Facial Video Datasets: InstantDrag successfully performs fast, photo-realistic edits on facial videos without requiring masks or text prompts.
• General Scenes: Demonstrates robust performance across various general scenes, maintaining the integrity of the original content while accurately reflecting user drag inputs.
• Benchmarks:
  • Edits are performed in about a second, making it suitable for real-time applications.
  • Maintains high-quality results comparable to more computationally intensive methods.

Conclusion

InstantDrag represents a significant advancement in drag-based image editing by providing an optimization-free pipeline that is both fast and interactive. By decomposing the task into motion generation and motion-conditioned image generation, InstantDrag achieves real-time performance with minimal inputs, making it a promising solution for interactive, real-time applications.