MM1.5: Enhancing Multimodal LLMs with Data-Centric Fine-Tuning and Diverse Data Mixtures

Apple Research has released a new family of multimodal large language models (MLLMs) called MM1.5, which aims to significantly enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. This latest iteration builds on the MM1 architecture by adopting a data-centric approach to model training, exploring the impact of diverse data mixtures throughout the entire training lifecycle.

Key Technical Changes and Why They Matter

Data-Centric Approach

• Diverse Data Mixtures: The team systematically explored the use of high-quality OCR (Optical Character Recognition) data and synthetic captions for continual pre-training. This ensures that the models can better understand text within images.
• Visual Instruction-Tuning Data Mixture: For supervised fine-tuning, an optimized visual instruction-tuning data mixture was used. This helps in improving the model's ability to follow complex instructions involving both text and visual elements.

Model Variants

• Parameter Range: MM1.5 models range from 1B to 30B parameters, including both dense and mixture-of-experts (MoE) variants.
• Performance at Small Scales: The research demonstrates that with careful data curation and training strategies, even smaller-scale models (1B and 3B) can achieve strong performance.

Implementation Details

Continual Pre-Training

• High-Quality OCR Data: This data is crucial for improving the model's ability to recognize and understand text within images.
• Synthetic Captions: These captions are generated to provide additional context and variability during training, helping the model generalize better.

Supervised Fine-Tuning

• Optimized Visual Instruction-Tuning Data Mixture: The team carefully curated this data to include a wide range of visual instructions, ensuring that the models can handle complex tasks involving both text and images.
• Training Strategies: The use of diverse data mixtures and careful curation allows for more effective fine-tuning, leading to better performance in downstream tasks.

Benchmarks and Performance

The MM1.5 family of models demonstrates significant improvements over its predecessor, especially in tasks that require understanding and reasoning about text within images. Here are some key benchmarks:

• Text-Rich Image Understanding: Improved accuracy in recognizing and interpreting text within images.
• Visual Referring and Grounding: Enhanced ability to correctly identify and refer to specific objects or regions within an image based on textual descriptions.
• Multi-Image Reasoning: Better performance in tasks that require reasoning across multiple images, such as understanding the relationship between different visual elements.

Practical Implications for Practitioners

For practitioners working with multimodal data, MM1.5 offers several key benefits:

• Enhanced Multimodal Capabilities: The models are better equipped to handle a variety of tasks involving both text and images.
• Data-Centric Approach: The emphasis on diverse data mixtures provides a roadmap for improving model performance through careful data curation.
• Scalability: With models ranging from 1B to 30B parameters, there are options suitable for different resource constraints.

Conclusion

MM1.5 represents a significant step forward in the development of multimodal large language models. By adopting a data-centric approach and exploring diverse data mixtures, the team has created models that can better understand and reason about text within images. This research provides valuable insights for practitioners looking to enhance their own multimodal systems.