Hugging Face Acquires XetHub to Revolutionize AI Repository Management

Hugging Face has announced the acquisition of XetHub, a Seattle-based company founded by Yucheng Low, Ajit Banerjee, and Rajat Arya. These founders previously worked at Apple, where they built and scaled the internal ML infrastructure. XetHub's mission is to bring software engineering best practices to AI development, and their technology allows Git to handle terabyte-scale repositories efficiently.

Why This Matters

The acquisition of XetHub by Hugging Face (HF) marks a significant step in enhancing the management of large datasets and models. Julien Chaumond, HF CTO, emphasized that this move will "unlock the next 5 years of growth" for HF datasets and models by transitioning to a more optimized storage backend.

The Current State

When HF built the first version of the Hub in 2020, they chose Git LFS (Large File Storage) as the foundation. Git LFS was a reasonable choice at the time due to its familiarity and adequacy for bootstrapping the Hub's usage. However, it has limitations when dealing with the massive files common in AI development.

The XetHub Solution

XetHub’s technology addresses these limitations by enabling:

• Chunked Files: Breaking large files into smaller, manageable chunks.
• Deduplication: Storing only unique data chunks to save storage space.
• Efficient Versioning: Allowing for granular updates without re-uploading entire files.

Future Use Cases

1. Efficient Data Updates

Imagine you have a 10GB Parquet file and need to add a single row. With current methods, you would need to re-upload the entire 10GB file. XetHub’s chunked files and deduplication will allow you to upload only the few chunks containing the new data, significantly reducing storage and bandwidth usage.

2. Granular Model Updates

For example, let's say a user like @bartowski wants to update a single metadata value in the GGUF header of a Llama 3.1 405B model repository. With XetHub’s technology, they can re-upload only the specific chunk containing the new data, making the process much more efficient.

The Team

XetHub brings a talented team of 12 members to Hugging Face. Yucheng Low and his co-founders have already made significant contributions to the AI community, and their expertise will be crucial in driving HF’s next phase of growth. You can follow the new XetHub team at hf.co/xet-team.

Looking Ahead

As the field moves toward trillion-parameter models, the need for efficient storage and versioning solutions becomes even more critical. Hugging Face’s acquisition of XetHub is a strategic move that will not only enhance their existing offerings but also set new standards in AI repository management.