EvolveDirector: Training Text-to-Image Models with Public Data and Vision-Language Guidance

EvolveDirector: A New Approach to Advanced Text-to-Image Generation

Recent advancements in generative models have revolutionized content creation, but they often rely on proprietary data and closed APIs. This limits their broader application and research potential. Enter EvolveDirector, a novel framework introduced by researchers from various institutions, including Rui Zhao and Hangjie Yuan. EvolveDirector aims to train text-to-image generation models using publicly available resources while achieving performance comparable to state-of-the-art models.

Key Technical Changes and Why They Matter

1. Public Data Utilization: Unlike proprietary models that use high-quality but private datasets, EvolveDirector leverages public data sources.
2. API Interaction for Data Generation: The framework interacts with advanced models via their APIs to generate text-image pairs, which are then used to train the base model.
3. Pre-trained Vision-Language Models (VLMs) as Guides: VLMs continuously evaluate and refine the training dataset through dynamic operations like discrimination, expansion, deletion, and mutation.

How EvolveDirector Works

Initial Data Generation

• API Calls: The framework starts by calling APIs of advanced models to generate initial text-image pairs.
• Data Scale: To achieve comparable performance, a large-scale dataset of 10 million or more samples is required. This can be time-consuming and expensive due to API call costs.

Training the Base Model

• Base Model Training: The generated data is used to train the base model.
• Performance Approximation: Experiments show that with extensive data, the base model can approximate the generation capabilities of advanced models.

VLM-Guided Evolution

• Dynamic Dataset Refinement:
  • Discrimination: Identifies and removes low-quality samples.
  • Expansion: Adds new, high-quality samples.
  • Deletion: Removes redundant or irrelevant data.
  • Mutation: Modifies existing samples to improve diversity and quality.
• Continuous Evaluation: VLMs provide ongoing feedback, ensuring the base model evolves efficiently.

Experimental Results

• Data Volume Reduction: The use of VLMs significantly reduces the required dataset size, making the training process more efficient and cost-effective.
• Balanced Abilities: When trained with data from multiple advanced models, EvolveDirector can select the best samples, leading to a model with powerful and balanced generation capabilities.

Implementation Details

• Model Architecture:
  • The base model is typically a deep neural network designed for text-to-image generation.
  • VLMs are pre-trained on large vision-language datasets and fine-tuned for specific tasks within EvolveDirector.
• Benchmarks:
  • Initial experiments show that the base model trained with public data can achieve FID (Fréchet Inception Distance) scores comparable to those of proprietary models, given a sufficiently large dataset.
  • The VLM-guided evolution further improves these scores by optimizing the training data.

Conclusion

EvolveDirector represents a significant step forward in making advanced text-to-image generation more accessible. By leveraging public data and pre-trained vision-language models, it offers a practical solution to the limitations imposed by proprietary datasets and closed APIs. This framework not only reduces costs but also enhances the quality and versatility of generated content, opening new avenues for research and application.