Detecting Human Artifacts in Text-to-Image Models with HAD Dataset and HADM

Recent advancements in text-to-image generation models have been impressive, but one persistent issue remains: human artifacts. These artifacts manifest as poorly generated human bodies, including distorted, missing, or extra body parts, which can significantly impair the visual fidelity of the images. A new paper by Kaihong Wang, Lingzhi Zhang, and Jianming Zhang introduces a solution to this problem through the Human Artifact Dataset (HAD) and the Human Artifact Detection Models (HADM).

What Changed

The researchers have created HAD, the first large-scale dataset specifically designed to identify and localize human artifacts in generated images. This dataset comprises over 37,000 images from several popular text-to-image models, each annotated for artifact localization. Using HAD, they trained HADM, a model that can detect various types of artifacts across multiple generative domains.

Key Details

• Dataset Creation:

  • Size: Over 37,000 images
  • Sources: Images generated by popular text-to-image models like DALL-E and Stable Diffusion
  • Annotations: Detailed annotations for human artifact localization (e.g., missing limbs, extra fingers)

• Model Training:

  • Architecture: HADM is a deep learning model designed to identify specific types of artifacts.
  • Training Data: The dataset is split into training and validation sets to ensure robustness.
  • Performance: HADM demonstrates strong generalization capabilities, even on images from unseen generators.

• Applications:

  • Feedback for Generators: The predictions from HADM can be used as feedback during the finetuning of diffusion models. This process helps improve the structural coherence of generated human figures.
  • Iterative Inpainting: A novel application where HADM is used in an iterative inpainting framework to correct artifacts directly in images.

Implementation and Results

• Finetuning Diffusion Models:

  • Process: The researchers used HADM predictions to guide the finetuning of diffusion models.
  • Outcome: This approach led to a noticeable reduction in human artifacts, improving the overall quality of generated images.

• Iterative Inpainting Framework:

  • Concept: HADM identifies artifacts, and an inpainting model corrects them iteratively.
  • Benefits: This method can be applied to any image, not just those from specific generators, making it a versatile tool for improving image quality.

Benchmarks and Experiments

• Quantitative Analysis:

  • Metrics: The researchers used metrics like Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR) to evaluate the effectiveness of HADM.
  • Results: Significant improvements in these metrics were observed, confirming the model's ability to reduce artifacts.

• Qualitative Analysis:

  • Visual Inspections: Before-and-after comparisons showed a clear reduction in human artifacts, with generated images appearing more natural and coherent.

Conclusion

The introduction of HAD and HADM represents a significant step forward in addressing one of the most challenging issues in text-to-image generation. By providing a large-scale dataset and a robust detection model, researchers and practitioners can now better understand and mitigate human artifacts, leading to higher-quality generated images. The applications of this work, from finetuning generative models to iterative inpainting, demonstrate its practical utility in improving the fidelity of AI-generated content.