XLabs AI Releases FLUX.1-dev LoRA Checkpoints for Enhanced Artistic Control

The team at XLabs AI has released a collection of trained LoRAs (Low-Rank Adaptation) for the FLUX.1-dev model by Black Forest Labs, aimed at enhancing artistic control and creativity in AI-generated art. This release is particularly exciting for practitioners working with models like ComfyUI and looking to fine-tune their outputs with specific styles or themes.

Key Technical Changes

• LoRA Checkpoints: The repository now includes pre-trained LoRAs for the FLUX.1-dev model, which can be easily integrated into various workflows.
• Training Scripts: XLabs AI has published detailed training scripts and configurations for both LoRAs and ControlNet, making it easier to fine-tune models with custom datasets.

ComfyUI Workflows

For those using ComfyUI, the team has provided a GitHub repository with specific workflows. You can find these at:

• ComfyUI Workflows

Training Details

The training scripts and configurations are available in another GitHub repository. These scripts support both LoRA and ControlNet fine-tuning, providing a robust foundation for custom model development.

• LoRA: Low-Rank Adaptation is a technique that allows you to fine-tune large models with minimal additional parameters, making it efficient and effective.
• ControlNet: This module can be used to guide the generation process based on additional inputs like edge maps or depth maps.

You can access these scripts and configurations at:

• Training Scripts and Configs

Training Dataset

The training dataset follows a structured format, which is crucial for ensuring consistent and high-quality outputs. The dataset directory structure looks like this:

├── images/
│    ├── 1.png
│    ├── 1.json
│    ├── 2.png
│    ├── 2.json
│    ├── ...

Each image (e.g., 1.png) is paired with a JSON file (e.g., 1.json) that contains a "caption" field. This caption serves as the text prompt for the training process.

Inference Examples

To help you get started, here are some example commands for generating images using the pre-trained LoRAs:

Furry_LoRA

python3 main.py \
 --prompt "Female furry Pixie with text 'hello world'" \
 --lora_repo_id XLabs-AI/flux-furry-lora --lora_name furry_lora.safetensors --device cuda --offload --use_lora \
 --model_type flux-dev-fp8 --width 1024 --height 1024 \
 --timestep_to_start_cfg 1 --num_steps 25 --true_gs 3.5 --guidance 4

python3 main.py \
 --prompt "Male furry Lycanthrope with fur-covered body in ancient ruins, howling at the full moon, surrounded by eerie mist, werewolf transformation, elder scrolls, eslweyr, glitch aesthetic, anime-inspired, digital illustration, artstation, furry" \
 --lora_repo_id XLabs-AI/flux-furry-lora --lora_name furry_lora.safetensors --device cuda --offload --use_lora \
 --model_type flux-dev-fp8 --width 1024 --height 1024 \
 --timestep_to_start_cfg 1 --num_steps 25 --true_gs 3.5

MJV6_LoRA

python3 main.py \
--prompt "A handsome man in a suit, 25 years old, cool, futuristic" \
--lora_repo_id XLabs-AI/flux-lora-collection --lora_name mjv6_lora.safetensors \
--device cuda:4 --offload --use_lora --model_type flux-dev-fp8 --width 1024 --height 1024

python3 main.py \
--prompt "A girl in a suit covered with bold tattoos and holding a vest pistol, beautiful woman, 25 years old, cool