Test-Time Training Layers Enable Transformers to Generate Coherent One-Minute Videos

Transformers have made significant strides in various domains, but generating one-minute videos from text prompts remains a challenge. Traditional self-attention layers are inefficient for long sequences, and alternatives like Mamba layers struggle with complex multi-scene stories due to less expressive hidden states. A recent paper introduces Test-Time Training (TTT) layers, which can be integrated into pre-trained Transformers to generate coherent one-minute videos from text storyboards.

What Changed Technically

• Introduction of TTT Layers: TTT layers have hidden states that are themselves neural networks, making them more expressive and capable of handling complex temporal dynamics.
• Integration with Pre-Trained Models: By adding TTT layers to a pre-trained Transformer (5B parameters), the model can generate one-minute videos with strong temporal consistency and smooth motion.

Key Benefits

• Temporal Consistency: TTT layers maintain character and scene consistency across the entire duration of the video.
• Motion Smoothness: Actions are smoother and more natural compared to baselines.
• Story Coherence: The generated videos tell complex stories, which is crucial for narrative coherence.

Implementation Details

• Dataset: A dataset based on Tom and Jerry cartoons was curated for proof-of-concept experiments.
• Baselines:
  • Mamba 2: Struggles with complex multi-scene stories due to less expressive hidden states.
  • Gated DeltaNet: Lacks temporal consistency across different angles of characters.
  • Sliding-Window Attention: Alters the environment and duplicates objects, leading to inconsistencies.

Performance Metrics

• Human Evaluation: TTT layers outperform all baselines in human evaluation, scoring 34 Elo points higher in a comparison of 100 videos per method.
• Temporal Consistency:
  • Local Attention: Struggles with consistency in character colors and scene elements (e.g., Tom's color, Jerry's mousehole).
  • TTT-MLP: Demonstrates strong character and temporal consistency across the entire video duration.
• Motion Smoothness:
  • Gated DeltaNet: Lacks smooth transitions between different angles of characters.
  • TTT-MLP: Produces smooth actions, maintaining consistency over scene changes.

Limitations

Despite the promising results, there are notable artifacts in the generated videos:

• Temporal Consistency: Boxes morph between 3-second segments of the same scene.
• Motion Naturalness: The cheese hovers in mid-air instead of falling naturally to the ground.
• Aesthetics: Lighting changes dramatically as characters move (e.g., the kitchen becoming brighter when Tom turns around).

Future Directions

The current implementation is resource-constrained, limiting experiments to one-minute videos. However, the approach can be extended to longer videos and more complex stories with improved efficiency and a more capable pre-trained model.

Acknowledgements

We thank Hyperbolic Labs for compute support, Yuntian Deng for help with running experiments, and Aaryan Singhal, Arjun Vikram, and Ben Spector for assistance with systems questions. Yue Zhao would like to thank Philipp Krä.