Cloudflare's Omni Platform: Running More AI Models on Fewer GPUs

2025-08-27
By Sven Sauleau and Mari Galicer

As the demand for AI products continues to grow, developers are creating a wider variety of models. At Cloudflare, we've been adding new models to our growing catalog on Workers AI, but noticed that not all models are used equally-leaving infrequently used ones occupying valuable GPU space. Given that efficiency is a core value at Cloudflare and GPUs are a scarce resource, we built Omni, an internal platform designed to maximize GPU usage by running and managing multiple models on a single machine.

What Changed Technically

Omni introduces several key innovations:

• Single Control Plane for Multiple Models: Instead of dedicating a container or VM to each model, Omni allows us to spawn and manage multiple models from a single control plane. This reduces the overhead of managing individual stacks and makes it easier to scale.
• Lightweight Process Isolation: Models can spin up and down quickly without affecting other processes, thanks to lightweight isolation techniques.
• File System Isolation: Each model has its own isolated file system, making dependency management straightforward.
• GPU Memory Overcommitment: By over-committing GPU memory, we can run more models on a single GPU, optimizing resource usage.

How Omni Works

Architecture Overview

At a high level, Omni is a platform for running AI models. Here’s how it processes inference requests:

1. Request Ingestion: When an inference request comes in via Workers AI, the model configuration is loaded from Workers KV.
2. Routing: The routing layer forwards the request to the closest Omni instance with available capacity. For Asynchronous Batch API requests, we route to an idle Omni instance, typically located in a region where it’s night.
3. Pre and Post Processing: Omni runs checks on the inference request and performs model-specific pre and post processing.
4. Model Execution: The request is then handed over to the appropriate model for execution.

Elastic Scaling

Traditionally, each AI model would run in its own container or VM with a dedicated GPU. This setup is straightforward but inefficient at scale due to the overhead of managing multiple stacks. Omni addresses this by:

• Centralized Management: A single control plane manages the lifecycle of all models, from provisioning and configuration to execution.
• Dynamic Resource Allocation: Models can be spun up and down as needed, ensuring that GPU resources are used efficiently.
• Isolated Environments: Each model runs in its own isolated environment, preventing conflicts and simplifying dependency management.

Benefits

1. Improved Model Availability: By running more models on each node, we increase the availability of AI services across our network.
2. Reduced Latency: Requests are routed to the closest available Omni instance, minimizing latency.
3. Energy Efficiency: Idle GPUs consume less power, and by optimizing resource usage, we reduce overall energy consumption.

Implementation Details

• Lightweight Isolation: We use lightweight process isolation techniques to ensure that models can be quickly spun up and down without affecting other processes.
• File System Management: Each model has its own isolated file system, allowing for easy management of dependencies and configurations.
• GPU Memory Overcommitment: By over-committing GPU memory, we can run more models on a single GPU. This is achieved by carefully managing memory allocation and ensuring that the total memory usage does not exceed physical limits.

Conclusion

Omni represents a significant step forward in how Cloudflare manages AI models on our edge nodes. By maximizing GPU usage and improving model availability, we can provide better services to developers and users alike. As AI continues to evolve, platforms like Omni will be crucial in ensuring that resources are used efficiently and effectively.