A Deep Dive into the Open Source AI Stack: 845 Repos and Counting

Four years ago, I analyzed the open source machine learning (ML) ecosystem. Since then, the landscape has evolved significantly, especially around foundation models. This time, I focused exclusively on this stack to understand its current state.

Data Collection

I started by searching GitHub using keywords like gpt, llm, and generative ai. The sheer volume of results is staggering-118K for gpt alone. To narrow it down, I filtered the repos to those with at least 500 stars. This gave me:

• 590 results for llm
• 531 results for gpt
• 38 results for generative ai

I also checked GitHub trending and social media for new repositories. After many hours, I identified 896 repos. Of these, 51 are tutorials (e.g., dair-ai/Prompt-Engineering-Guide) and aggregated lists (e.g., f/awesome-chatgpt-prompts). While these are helpful, I focused my analysis on the 845 software repositories.

The New AI Stack

The current AI stack can be broken down into three layers: infrastructure, model development, and application development.

Infrastructure

At the foundation of this stack is infrastructure. This layer includes tools for:

• Serving: Repos like vllm and NVIDIA’s Triton help in efficiently serving models.
• Compute Management: Tools like skypilot manage compute resources, ensuring smooth operations.
• Vector Search and Database: Projects such as faiss, milvus, qdrant, and lancedb provide efficient vector search and database solutions.

Model Development

The model development layer is where the magic happens. It includes:

• Frameworks for Modeling & Training: Libraries like transformers, pytorch, and DeepSpeed are essential for building and training models.
• Inference Optimization: Tools such as ggml and openai/triton optimize inference performance.
• Dataset Engineering: This involves tools for creating, processing, and managing datasets.
• Evaluation: Frameworks to evaluate model performance.

Anything that involves changing a model’s weights-such as finetuning-happens in this layer.

Application Development

The top layer is application development. It includes:

• Instruction Tuning: Tools like dair-ai/Prompt-Engineering-Guide help in crafting effective prompts.
• Integration: Libraries and tools that facilitate the integration of models into applications.

Key Observations

1. Collaborative Ecosystem: The open source community is incredibly collaborative, with contributions from developers worldwide.
2. Divergent Ecosystems: China’s open source ecosystem diverges significantly from the Western one, reflecting different priorities and regulations.

Conclusion

The open source AI stack is rich and diverse, with a plethora of tools and frameworks supporting each layer. This ecosystem is not just about building models but also about serving them efficiently, managing compute resources, and developing applications that leverage these models. As the landscape continues to evolve, staying updated with new repositories and tools will be crucial for practitioners.