Hugging Face has introduced a new approach to building diffusion pipelines called Modular Diffusers. This framework allows developers to construct complex workflows by composing reusable blocks, making it easier to tailor models to specific needs without starting from scratch. This article will guide you through the basics of Modular Diffusers, how to use pre-built blocks, and how to create custom ones.

Quickstart

If you're familiar with Hugging Face's DiffusionPipeline, the transition to Modular Diffusers is straightforward. Here’s a quick example using the FLUX.2 Klein 4B model:

import torch
from diffusers import ModularPipeline

# Define the modular pipeline (model weights are not loaded yet)
pipe = ModularPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-klein-4B"
)

# Load the model weights and configure dtype, quantization, etc.
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")

# Generate an image
image = pipe(
    prompt="a serene landscape at sunset",
    num_inference_steps=4,
).images[0]

image.save("output.png")

This code produces the same output as a standard DiffusionPipeline, but under the hood, it uses composable blocks. Each block is a self-contained unit that can be mixed and matched to create custom pipelines.

Custom Blocks

One of the key features of Modular Diffusers is the ability to create and use custom blocks. This flexibility allows you to:

• Extend existing functionality: Add new features or modify existing ones.
• Optimize performance: Fine-tune blocks for specific hardware or use cases.
• Share and collaborate: Contribute your blocks to the community.

To create a custom block, you need to subclass DiffusionBlock and implement the necessary methods. Here’s a basic example:

from diffusers import DiffusionBlock

class CustomTextEncoder(DiffusionBlock):
    def __init__(self, model_name: str):
        self.model = SomeTextEncoderModel.from_pretrained(model_name)

    def forward(self, prompt: str):
        return self.model.encode(prompt)

Once you have your custom block, you can integrate it into a modular pipeline:

from diffusers import ModularPipeline

Define the pipeline and add the custom block

pipe = ModularPipeline() pipe.add_block("text_encoder", CustomTextEncoder(model_name="your-model"))

Load other components and run inference

pipe.load_components(torch_dtype=torch.bfloat16) pipe.to("cuda")

image = pipe( prompt="a serene landscape at sunset", num_inference_steps=4, ).images[0]

image.save("output.png")

### Modular Repositories

Hugging Face has also introduced **Modular Repositories** to facilitate sharing and collaboration. These repositories contain pre-built blocks that you can easily integrate into your pipelines. You can find a growing collection of modular components on the Hugging Face Hub.

To use a block from a modular repository:

```python
from diffusers import ModularPipeline

# Define the pipeline and load a block from a modular repository
pipe = ModularPipeline()
pipe.load_block("text_encoder", "some-user/some-repo")

# Load other components and run inference
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")

image = pipe(
    prompt="a serene landscape at sunset",
    num_inference_steps=4,
).images[0]

image.save("output.png")

Community Pipelines

The Modular Diffusers ecosystem is enriched by community contributions. Users can share their custom pipelines and blocks, making it easier for others to build on top of existing work. This collaborative approach accelerates innovation and ensures that the framework remains versatile and adaptable.

To explore community pipelines:

1. Visit the Hugging Face Hub.
2. Search for modular pipelines or blocks.
3. Load them into your own projects using the load_block method.

Integration with Mellon

For those who prefer a visual workflow, Modular Diffusers integrates seamlessly with Mellon, a node-based interface. Mellon allows you to drag and drop blocks to create complex workflows without writing code. This is particularly useful