Benchmarking Agentic Workloads for Modern Inference Engines

Large language model (LLM) inference engines have traditionally been benchmarked using simple, single-turn workloads. These benchmarks typically involve sending a prompt of P tokens and generating D tokens, measuring metrics like time-to-first-token, tokens-per-second, and completion throughput. Examples include 1k/8k for decode-heavy, 8k/1k for prefill-heavy, and 1k/1k for balanced workloads. However, the rise of agentic applications has introduced a new set of challenges that these traditional benchmarks fail to capture.

The Shift to Agentic Workloads

Agentic applications, such as multi-turn chatbots, tool-using agents, and interactive coding assistants, operate under a fundamentally different workload pattern. These applications involve multiple turns of interaction, where the model generates responses, calls external tools, receives outputs, and continues generating based on new context. This loop can repeat dozens or even hundreds of times until the agent completes its task.

Key Challenges:

• KV Cache Management: Long-running traces require efficient management of key-value (KV) caches to store intermediate states.
• Scheduler Pressure: A high volume of short output requests puts significant pressure on schedulers, which must handle rapid context switching.
• Heavy-Tailed Token Distributions: The distribution of token lengths in agentic workloads is often heavy-tailed, meaning some interactions produce very long outputs while others are much shorter.

New Workload Profiles and Benchmarking Tools

To address these challenges, Applied Compute has released three distinct workload profiles and an open-source benchmarking harness. These tools aim to provide a more accurate representation of modern agentic workloads, helping developers optimize inference engines and hardware accelerators for real-world performance.

Workload Profiles:

• Interactive Agent: Simulates a multi-turn conversation where the model generates responses, calls tools, and processes tool outputs.
• Batch Processing: Models long-running tasks that involve multiple rounds of prefill and generation.
• Background Agent: Represents scenarios where the agent performs background tasks with minimal user interaction.

Benchmarking Harness:

The open-source benchmarking harness allows developers to replay these workload profiles in their own environments. This enables more realistic load testing and performance optimization, ensuring that inference engines can handle the complexities of agentic applications.

Metrics for Different Deployment Contexts

Understanding the right metrics is crucial for optimizing inference engines in various deployment contexts:

• Interactive Agents: Focus on latency and responsiveness. Key metrics include time-to-first-token and tokens-per-second.
• Batch Processing: Emphasize throughput and resource utilization. Metrics like completion throughput and memory usage are critical.
• Background Agents: Balance between efficiency and reliability. Metrics such as energy consumption and error rates become important.

Real-World Observations

Over 100 production multi-turn post-training runs sampled from different deployments, Applied Compute observed a mean:

• Number of Turns per Session: Varies widely but averages around 50 turns.
• Token Distribution: Heavy-tailed, with some sessions generating thousands of tokens while others produce only a few dozen.
• Tool Call Latency: Significantly impacts overall performance, often introducing delays that affect the model's ability to generate coherent responses.

Conclusion

The shift towards agentic workloads has introduced new complexities in LLM inference. Traditional benchmarks are no longer sufficient to capture these nuances. By using the new workload profiles and benchmarking harness, developers can better understand and optimize their inference engines for real-world applications. This will ultimately lead to more efficient, responsive, and reliable agentic systems.