Qwen1.5: Enhanced Multilingual Models and Developer Experience

Introducing Qwen1.5

February 4, 2024 · 14 min read · 2895 words · Qwen Team

As we approach the Chinese New Year, the Qwen team is excited to announce Qwen1.5, the latest iteration in our series of language models. This update focuses on developing a "good" model while significantly improving the developer experience. Here’s what you need to know:

Key Features and Improvements

• Open-Sourced Models: We are open-sourcing base and chat models across six sizes: 0.5B, 1.8B, 4B, 7B, 14B, 32B, 72B, and 110B parameters.
• MoE Model: In addition to the standard models, we are also releasing an MoE (Mixture of Experts) model for more specialized use cases.
• Quantized Models: To support a wide range of deployment scenarios, we provide Int4 and Int8 GPTQ models, as well as AWQ and GGUF quantized models.
• Enhanced Developer Experience: Qwen1.5’s code has been merged into Hugging Face transformers (version 4.37.0 and above), making it accessible without needing trust_remote_code.

Collaborations and Integrations

We have collaborated with several frameworks to ensure broad support for Qwen1.5:

• vLLM: For efficient deployment.
• SGLang: For seamless integration into various applications.
• AutoAWQ and AutoGPTQ: For advanced quantization techniques.
• Axolotl and LLaMA-Factory: For fine-tuning capabilities.
• llama.cpp: For local LLM inference.

Availability

The Qwen1.5 series is available on platforms like:

• Ollama
• LMStudio

API services are also provided on:

• DashScope
• together.ai (global accessibility)

You can get started with the API at api.together.ai. We recommend trying out the Qwen1.5-72B-chat model, which is available on together.xyz/playground/chat/Qwen/Qwen1.5-72B-Chat.

Performance Enhancements

Qwen1.5 brings substantial improvements in several key areas:

• Alignment with Human Preferences: The chat models are better aligned with human preferences, ensuring more natural and coherent conversations.
• Multilingual Capabilities: Enhanced multilingual support across all models, making them suitable for a broader range of applications.
• Context Length: All models now uniformly support a context length of up to 32768 tokens, allowing for more extensive and detailed interactions.

Minor Improvements in Base Models

While the focus has been on chat model improvements, we have also made minor enhancements to the quality of base language models. These improvements can benefit your fine-tuning endeavors, providing a solid foundation for custom applications.

Technical Details

• Model Sizes: The range of available models from 0.5B to 110B parameters caters to different performance and resource requirements.
• Quantization Techniques: Int4 and Int8 GPTQ models, along with AWQ and GGUF quantized models, ensure efficient deployment on a variety of hardware.
• Context Length: The uniform support for up to 32768 tokens across all models is a significant improvement in handling long-form content.

Conclusion

Qwen1.5 represents a step forward in our mission to create a truly "good" model while enhancing the developer experience. With improved alignment, enhanced multilingual capabilities, and broader availability, we believe Qwen1.5 will be a valuable addition to your toolkit.