Unpacking the Connection Between Diffusion Models and Autoregressive Models in the Frequency Domain

If you’ve been keeping up with recent advancements in machine learning, you might have noticed a growing interest in understanding how different model architectures relate to each other. One particularly intriguing insight is that diffusion models of images perform approximate autoregression in the frequency domain. This connection is not just theoretical but has practical implications for practitioners working with both diffusion and autoregressive models.

Two Forms of Iterative Refinement

To start, let's consider how both diffusion models and autoregressive models handle iterative refinement:

• Diffusion Models: These models work by gradually denoising a random noise image to produce a coherent output. The process is guided by a series of learned steps that iteratively refine the image.
• Autoregressive Models: These models, on the other hand, generate data one step at a time, conditioning each new step on the previous ones. For images, this means generating pixels sequentially.

The key insight here is that both models can be seen as performing iterative refinement, but in different domains. Diffusion models do this in the spatial domain, while autoregressive models can be understood as operating in the frequency domain.

A Spectral View of Diffusion

To understand how diffusion models perform approximate autoregression in the frequency domain, let's dive into the spectral perspective:

• Fourier Transform: The Fourier transform is a mathematical tool that decomposes a signal into its constituent frequencies. For images, this means breaking down the image into different spatial frequencies.
• Diffusion as Frequency Filtering: When you apply the Fourier transform to an image and then run it through a diffusion model, you can observe how the model affects different frequency components. Diffusion models tend to smooth out high-frequency noise while preserving or enhancing lower frequencies.

This spectral view reveals that diffusion models are effectively performing a form of autoregressive modeling in the frequency domain. By iteratively refining the image in the spatial domain, they indirectly influence the frequency spectrum, which is a key characteristic of autoregressive processes.

What About Sound?

The connection between diffusion and autoregressive models isn't limited to images. For sound data, the same principles apply:

• Sound as a Signal: Audio signals can be represented as time-series data, much like pixel values in an image.
• Spectral Domain for Sound: The Fourier transform can also be applied to audio signals to decompose them into their frequency components.
• Diffusion Models for Sound: When diffusion models are used to generate or denoise sound, they similarly affect the frequency spectrum. This means that diffusion models for audio can also be seen as performing approximate autoregression in the frequency domain.

Unstable Equilibrium

One of the interesting aspects of this connection is the concept of an "unstable equilibrium":

• Stability and Convergence: In both diffusion and autoregressive models, the iterative refinement process must converge to a stable state. However, the nature of this convergence can be different.
• Diffusion Models: These models are designed to denoise and refine images or sounds until they reach a coherent output. This process is inherently stable but can sometimes get stuck in local minima.
• Autoregressive Models: These models generate data step-by-step, which can lead to instability if the model is not well-tuned. However, this instability can also be a source of creativity and diversity in the generated outputs.

Closing Thoughts

The connection between diffusion models and autoregressive models in the frequency domain is more than just a theoretical curiosity. It provides a new perspective on how these models work and opens up possibilities for cross-pollination of ideas and techniques. For practitioners, this means that insights from one domain can be applied to improve or innovate in the other.

To explore this further, I’ve created a Python notebook using Google Colab. This notebook includes all the code used to produce the plots and animations discussed in this post. I encourage you to try it out, modify the parameters, and see how the models behave.