MovieAgent: Automating Movie Generation with Multi-Agent CoT Planning

---

MovieAgent: A New Approach to Automated Movie Generation

MovieAgent, a groundbreaking framework developed by researchers at the National University of Singapore's Show Lab, aims to transform the way movies are created. By leveraging multi-agent Chain-of-Thought (CoT) planning and advanced generative models, MovieAgent can convert script synopses into coherent, multi-scene videos with minimal human intervention. This article delves into the technical details and implications for practitioners.

The Overall Pipeline of MovieAgent

The core innovation in MovieAgent lies in its hierarchical CoT reasoning process, which involves three types of agents: director, scene plan, and shot plan. Here’s how it works:

• Director Agent: Oversees the entire movie generation process, ensuring consistency in character identities, relationships, and overall narrative flow.
• Scene Plan Agent: Breaks down the script into individual scenes, each with a specific plot point, emotional tone, and key themes.
• Shot Plan Agent: Further decomposes scenes into shots, specifying camera angles, movements, and other visual elements to bring the story to life.

Task 1: Script Synopsis to Keyframe/Storyboard Generation

One of the initial steps in MovieAgent's pipeline is converting a script synopsis into a storyboard. This involves:

• Hierarchical Data: The director agent uses hierarchical data to maintain coherence across different scenes and shots.
• Character Consistency: Existing methods are employed to ensure that character identities and relationships remain consistent throughout the movie.

Task 2: Script Synopsis to Movie Generation

To illustrate the capabilities of MovieAgent, let’s consider a script synopsis for an adventure story:

Script Synopsis/Raw Story

Elsa, Anna, Kristoff, Olaf, and Mattias embark on a journey to uncover the truth behind the mysterious voice calling Elsa. As they travel through the enchanted forest, they discover secrets about their kingdom and Elsa’s powers. Mattias, a loyal Arendelle soldier trapped in the forest for years, helps them navigate the tensions between Arendelle and the Northuldra people.

Input: Script Synopsis and Character Bank (Image, Name)

Process with MovieAgent (GPT4-o + ROICtrl + Kling 1.6)

• Sub-Script 1: The Call and the Journey Begins

  • Plot: Anna, Elsa, Kristoff, and Olaf set out on a journey toward a mysterious voice calling Elsa. Supernatural weather linked to Elsa’s emotions challenges them. In the enchanted forest, they encounter frozen relics of a past conflict between Arendelle and the Northuldra. Elsa’s magic calms elemental spirits blocking their path, and they uncover cryptic symbols tied to Elsa’s childhood memories.
  • Involving Characters: Elsa, Anna, Kristoff, and Olaf.
  • Emotional Tone: Curiosity and determination.
  • Key Themes: Adventure, discovery, and mystery.

• Scene 1 - Shot 1

  • Plot: Elsa stands alone in her ice dress, listening to a mysterious, ethereal voice. Her expression is a mix of curiosity and concern as she looks towards the distant horizon.
  • Involving Characters: Elsa.
  • Shot Type: Close-up.
  • Camera Movement: Static shot.
  • Subtitle: {Elsa: I can hear it again... the voice. It's calling me.}

• Scene 1 - Shot 3

  • Plot: A wide shot reveals the enchanted forest bathed in ethereal light. Glowing plants and shimmering air create a magical atmosphere, emphasizing the vastness and wonder of the setting.
  • Involving Characters: None.
  • Shot Type: Wide shot.
  • Camera Movement: Static shot.

Technical Details and Benchmarks

• Models Used:

  • GPT4-o: For generating coherent sub-scripts and maintaining narrative consistency.
  • ROICtrl: For controlling regions of interest in the generated scenes, ensuring that important elements are highlighted.
  • Kling 1.6: For rendering high-quality visuals using Stable Diffusion.

• Performance:

  • MovieAgent can generate a multi-scene, multi-shot video with a coherent narrative from a script synopsis in under