Personalize Anything: Zero-Shot Subject Reconstruction with Diffusion Transformers

Haoran Feng, Zehuan Huang, Lin Li, and Lu Sheng have introduced a groundbreaking method called Personalize Anything in their recent AAAI 2026 paper. This framework leverages diffusion transformers (DiTs) to achieve personalized image generation without the need for additional fine-tuning. The key innovation lies in replacing denoising tokens with those of a reference subject, enabling zero-shot subject reconstruction and versatile editing capabilities.

Technical Breakdown

Personalize Anything introduces several novel techniques that enhance both identity preservation and generative flexibility:

• Timestep-Adaptive Token Replacement: This method ensures subject consistency by injecting reference tokens early in the denoising process. Later stages introduce regularization to maintain flexibility.
• Patch Perturbations: By perturbing patches of the token injections, the model can generate structurally diverse images while preserving the subject's identity.
• Mask-Guided Token Replacement: This technique uses masks to guide where tokens are replaced, ensuring that the subject's identity is maintained in specific regions.

Methodology

1. Early Denoising with Mask-Guided Token Replacement

   • Preserved Positional Encoding: The positional encoding of the reference subject's tokens is retained during replacement.
   • Identity Anchoring: This early injection ensures that the subject's identity is firmly anchored in the generated image.

2. Late-Stage Multi-Modal Attention

   • Semantic Fusion with Text: In later denoising steps, multi-modal attention allows for semantic fusion between the reference tokens and text prompts.
   • Flexibility Enhancement: This late-stage regularization enhances the model's ability to generate diverse images while maintaining subject consistency.

3. Patch Perturbations

   • Variation Injection: By perturbing patches of the injected tokens, the model can produce a wide range of structural variations.
   • Balanced Identity and Flexibility: This technique balances the preservation of the subject's identity with the flexibility to generate diverse images.

Applications

Personalize Anything supports various advanced use cases:

• Layout-Guided Generation: By translating token-injected regions, the model can generate images that adhere to specific layouts.
• Multi-Subject Composition: Sequential token injection allows for the composition of multiple subjects in a single image.
• Inpainting and Outpainting: Specifying masks and increasing token replacement enables precise editing within or beyond the original image boundaries.

Qualitative Results

The method demonstrates strong identity preservation and versatility across different scenarios:

• Single-Subject Personalization: The model accurately reconstructs the subject's identity while allowing for flexible edits.
• Layout-Guided Subject Personalization: Images generated with specific layouts maintain high fidelity to the reference subject.
• Multi-Subject Personalization: Composing multiple subjects in a single image while preserving each subject's identity.
• Mask-Controlled Editing: Inpainting and outpainting results show precise control over the editing process.

Evaluation

Evaluations of Personalize Anything demonstrate state-of-the-art performance in:

• Identity Preservation: The model maintains high fidelity to the reference subject, even in complex scenarios.
• Versatility: It supports a wide range of personalization and editing tasks without additional fine-tuning.

This work not only establishes new insights into diffusion transformers but also delivers a practical paradigm for efficient personalization. By leveraging simple yet effective token replacement techniques, Personalize Anything opens up new possibilities in personalized image generation.