T2V-Turbo: Accelerating Text-to-Video Generation with Mixed Reward Feedback

T2V-Turbo, a new approach to text-to-video (T2V) generation, is breaking the quality bottleneck of video consistency models (VCMs) while maintaining fast inference speeds. Developed by researchers from UC Santa Barbara, Google, and the University of Waterloo, T2V-Turbo integrates feedback from differentiable reward models into the consistency distillation (CD) process of pre-trained T2V models. This innovation allows for high-quality video generation in just a few steps, significantly outperforming existing methods.

Key Technical Changes

• Mixed Reward Feedback: T2V-Turbo introduces a novel mechanism that combines feedback from both image-text and video-text reward models (RMs). These RMs provide differentiable rewards during the training process, which are directly optimized alongside the CD loss.
• Single-Step Generation: By focusing on single-step generations, T2V-Turbo avoids the memory constraints associated with backpropagating gradients through iterative sampling processes. This approach enables faster and more efficient training.

Why It Matters to Practitioners

For practitioners working in video generation, T2V-Turbo offers a significant improvement in both speed and quality. Traditional diffusion models require many steps for high-quality output, making them computationally expensive and slow. VCMs, while faster, often sacrifice quality. T2V-Turbo bridges this gap by delivering high-quality videos with fewer steps, making it an attractive solution for real-time applications and large-scale deployments.

Technical Details

• Training Pipeline:

  • Reward Models: The system uses two types of reward models: image-text RM and video-text RM. These RMs provide feedback on the quality of generated frames and sequences.
  • Consistency Distillation (CD): The CD process distills knowledge from a pre-trained T2V model to a smaller, faster model. T2V-Turbo enhances this by integrating reward feedback directly into the single-step generation process.
  • Backpropagation: Gradients are backpropagated through the single-step generation process, optimizing both the consistency loss and the rewards.

• Benchmarks:

  • VBench: On the VBench dataset, T2V-Turbo (VC2) achieves the highest total score, outperforming Gen-2 and Pika.
  • Human Evaluations: Human evaluators preferred the 4-step generations from T2V-Turbo over the 50-step DDIM samples from their teacher models. This represents more than a tenfold acceleration while improving video quality.

Implementation Notes

• 4-Step vs. 8-Step Generations: The researchers provide both 4-step and 8-step generation results, demonstrating the flexibility of T2V-Turbo. While the 4-step version offers the best balance between speed and quality, the 8-step version can be used for even higher fidelity if needed.
• Code and Model Availability: The code and model weights are publicly available on GitHub and Hugging Face, making it easy for researchers and developers to experiment with T2V-Turbo.

Try It Out

You can try out T2V-Turbo yourself using the Hugging Face Space. This interactive platform allows you to generate videos based on text prompts, giving you a hands-on experience with the model's capabilities.