TransPixeler: Extending Text-to-Video Models for RGBA Generation with Transparency

Text-to-video generative models have seen remarkable advancements, opening up new possibilities in entertainment, advertising, and education. However, one significant challenge remains: generating videos with alpha channels (RGBA) for transparency. Alpha channels are crucial for visual effects (VFX), enabling elements like smoke, reflections, and other transparent objects to blend seamlessly into scenes. To address this, a team from HKUST(GZ), HKUST, and Adobe Research has introduced TransPixeler, a method that extends pretrained video models to generate RGBA videos while maintaining the quality of RGB outputs.

Key Technical Changes

• Alpha-Specific Tokens: TransPixeler introduces new tokens specifically for alpha channel generation. These tokens are integrated into the diffusion transformer (DiT) architecture, allowing the model to jointly generate RGB and alpha channels.
• Reinitialized Positional Embeddings: The positional embeddings in the DiT are reinitialized to accommodate the additional alpha-specific tokens. This ensures that the model can effectively learn the spatial relationships between the new and existing tokens.
• Zero-Initialized Domain Embedding: A zero-initialized domain embedding is added to distinguish the alpha channel generation from the RGB token generation. This helps the model understand the context and generate consistent RGBA outputs.

Method Overview

TransPixeler builds upon state-of-the-art DiT-like video generation models by making several key modifications:

• Diffusion Transformer (DiT) Architecture: The core of TransPixeler is a diffusion transformer, which is known for its ability to generate high-quality RGB videos. By extending this architecture, TransPixeler can leverage the strengths of the original model while adding alpha channel generation.
• LoRA-Based Fine-Tuning: To fine-tune the model for RGBA generation, TransPixeler uses LoRA (Low-Rank Adaptation). This approach allows for efficient and effective adaptation of the pretrained model without overfitting to the limited training data.
• Optimized Attention Mechanisms: The attention mechanisms in the DiT are optimized to ensure strong alignment between the RGB and alpha channels. This optimization is crucial for maintaining consistency and quality in the generated videos, even with limited training data.

Results

TransPixeler demonstrates its capabilities through a series of impressive results:

Text-to-RGBA Video Generation

• Transparent Glass of Water with Ice Cubes: The model generates a video where a glass of water containing ice cubes is gently swirling. The transparency of the glass and the movement of the ice are accurately captured.
• Asteroid Belt Swirling in Space: A chaotic asteroid belt swirling through space, with the transparency of the asteroids and the background stars blending seamlessly.
• Squirrel's Bushy Tail Flicking: A squirrel's bushy tail flicking quickly, with the transparency of the fur and the environment accurately represented.
• Cloud of Dust Exploding: A cloud of dust erupting and dispersing like an explosion, with the transparency of the dust particles and the background clearly visible.

Image-to-RGBA Video Generation

• Input Image to Generated Video: TransPixeler can also generate RGBA videos from static images. For example, an input image of a landscape can be transformed into a video where elements like water, smoke, or leaves are transparent and move naturally.

Impact on VFX and Interactive Content Creation

The ability to generate high-quality RGBA videos has significant implications for the visual effects industry. TransPixeler's approach allows for more realistic and seamless integration of transparent elements into scenes, enhancing the overall quality of VFX in movies, games, and other media. Additionally, this technology opens up new possibilities for interactive content creation, such as augmented reality (AR) and virtual reality (VR) applications.

Conclusion

TransPixeler represents a significant step forward in text-to-video generation by addressing the challenge of alpha channel generation. By extending pretrained models with alpha-specific tokens, reinitialized positional embeddings, and optimized attention mechanisms, TransPixeler achieves strong alignment between RGB and alpha channels, even with limited training data. This advancement paves the way for more realistic and versatile VFX applications in various industries.