Auffusion: Bridging Text and Audio with Diffusion Models and LLMs

Recent advancements in diffusion models and large language models (LLMs) have significantly advanced the field of Artificial Intelligence Generated Content (AIGC). One exciting application within this domain is Text-to-Audio (TTA), which aims to generate audio from natural language prompts. However, existing TTA systems often struggle with generation quality and text-audio alignment, especially for complex inputs. Enter Auffusion, a new TTA system developed by researchers at Beijing University of Posts and Telecommunications.

Key Technical Innovations

• Inspiration from Text-to-Image (T2I) Models: Auffusion draws heavily on the success of T2I models, which have shown exceptional generative capabilities and precise cross-modal alignment. By adapting these frameworks to the TTA task, Auffusion leverages the inherent strengths of diffusion models.
• Cross-Modal Alignment: One of the critical challenges in TTA is ensuring that the generated audio accurately matches the textual description. Auffusion addresses this by using a U-Net architecture initialized with a pretrained text-to-image latent diffusion model (LDM). This initialization helps in achieving fine-grained details and object bindings, which are crucial for high-quality audio generation.

Architecture Overview

The Auffusion system involves back-and-forth transformations between four feature spaces: audio, spectrogram, pixel, and latent space. Here’s a breakdown of the process:

• Audio to Spectrogram: The input audio is first converted into a spectrogram, which captures the frequency content over time.
• Spectrogram to Pixel Space: The spectrogram is then transformed into a pixel representation using a U-Net architecture. This step is crucial for leveraging the pretrained text-to-image LDM.
• Pixel to Latent Space: The pixel representation is mapped to a latent space, where the diffusion process takes place. This latent space is where the model generates the audio content conditioned on the text input.
• Latent to Audio: Finally, the generated latent representation is transformed back into an audio signal.

Implementation Details

• Training Setup: Auffusion’s LDM is trained on a single A6000 GPU using Stable Diffusion with cross attention. This setup allows for efficient training and inference, making it accessible even with limited computational resources.
• Data Efficiency: Despite the complexity of the task, Auffusion demonstrates impressive performance using relatively small datasets, which is a significant advantage over other TTA systems.

Evaluation and Findings

To evaluate Auffusion’s effectiveness, the researchers conducted both objective and subjective assessments. Key findings include:

• Superior Generation Quality: Auffusion outperformed previous TTA approaches in terms of audio quality and text-audio alignment.
• Text-Audio Alignment: Through comprehensive ablation studies and innovative cross-attention map visualizations, the researchers provided detailed insights into how well Auffusion aligns audio with textual descriptions. These visualizations revealed that Auffusion can accurately bind specific words to corresponding audio segments.

Applications

Auffusion’s strong text-audio alignment enables several advanced applications:

• Text-Guided Audio Style Transfer: Users can modify the style of generated audio by providing different text prompts.
• Audio Inpainting: Auffusion can fill in missing parts of an audio clip based on textual descriptions, making it useful for repairing damaged or incomplete audio files.
• Attention-Based Replacement: The system allows for replacing specific segments of an audio clip with new content guided by the text input.

Conclusion

Auffusion represents a significant step forward in the field of Text-to-Audio generation. By leveraging the strengths of diffusion models and LLMs, it achieves high-quality audio generation and precise text-audio alignment. The system’s efficiency and versatility make it a promising tool for various applications, from content creation to audio repair.