Pixtral Large: Deprecation and Legacy of a 124B Multimodal Model

Pixtral Large: A Technical Retrospective

Nov 18, 2024

Mistral AI Team

Pixtral Large, once a flagship model in the Mistral AI lineup, has been deprecated. The team now focuses on more advanced vision and multimodal models. However, it's worth revisiting what made Pixtral Large such a significant milestone in the field.

Key Features of Pixtral Large:

• Frontier-Class Multimodal Performance: Achieved state-of-the-art results on MathVista, DocVQA, and VQAv2.
• 124B Parameters: Comprised a 123B multimodal decoder and a 1B parameter vision encoder.
• 128K Context Window: Capable of processing at least 30 high-resolution images in a single context window.
• Text Performance: Maintained the strong text capabilities of Mistral Large 2.

Architecture and Implementation

Pixtral Large was built on top of Mistral Large 2, extending its multimodal capabilities without compromising text performance. The model's architecture includes:

• Multimodal Decoder (123B Parameters): Handles both textual and visual inputs.
• Vision Encoder (1B Parameters): Specializes in processing images.

The large context window (128K tokens) allowed the model to process multiple high-resolution images simultaneously, which is crucial for tasks like document analysis and chart recognition. This feature was a significant improvement over previous models, which often struggled with handling complex visual data efficiently.

Performance Benchmarks

Pixtral Large was evaluated against other leading models on several benchmarks:

• MathVista: Achieved 69.4%, outperforming all other models in this benchmark.
• ChartQA and DocVQA: Surpassed GPT-4o and Gemini-1.5 Pro, demonstrating superior reasoning over complex charts and documents.
• MM-MT-Bench: Outperformed Claude-3.5 Sonnet (new), Gemini-1.5 Pro, and GPT-4o (latest).
• LMSys Vision Leaderboard: Best open-weights model by a substantial margin, outperforming proprietary models like GPT-4o (August '24).

Use Cases

Pixtral Large was versatile in its applications:

• Document Analysis: Capable of understanding and reasoning over complex documents.
• Chart Recognition: Excellent at interpreting and explaining charts.
• Natural Images: Strong performance on tasks involving natural images.

Example: Multilingual OCR and Reasoning

Prompt: I bought the coffee and the sausage. How much do I owe? Add an 18% tip.

Response: To calculate the total amount you owe, including the 18% tip, follow these steps:

1. Calculate the cost of the coffee and the sausage:

   • 2x Latte Macchiato: 2 * 4.50 CHF = 9.00 CHF
   • 1x Sausage: 6.00 CHF

2. Total before tip: 9.00 CHF + 6.00 CHF = 15.00 CHF

3. Calculate the 18% tip:

   • Tip: 15.00 CHF * 0.18 = 2.70 CHF

4. Total amount you owe: 15.00 CHF + 2.70 CHF = 17.70 CHF

Availability and Licensing

Pixtral Large was available under the Mistral Research License (MRL) for research and educational use. For commercial purposes, including experimentation, testing, and production, the model could be licensed through the Mistral Commercial License.

Despite its deprecation, Pixtral Large remains a testament to the rapid advancements in multimodal AI and sets a high bar for future models.