MLPerf Storage v3.0 Adds S3 Support, KV Cache and Vector DB Tests to Track Modern AI Workloads

MLCommons just shipped version 3.0 of MLPerf Storage, and it's a fairly significant overhaul of a benchmark that had been showing its age. The old version tested storage systems against three training workloads, Unet3D, Cosmoflow, and Resnet50, using Nvidia A100 and H100 GPUs as reference points. That made sense a couple of years ago. It makes a lot less sense now, when most of the interesting storage pressure in AI systems comes from inference, not training.

V3.0 fixes that by restructuring the whole suite around four workload categories, and adding an S3 object storage layer alongside the existing POSIX filesystem interface. If you've been trying to figure out whether object storage can actually keep pace with filesystem-based storage for AI workloads, this round gives you real numbers to look at instead of vendor claims.

The four workloads now look like this:

• Training: Unet3D and RetinaNet, simulated with Nvidia B200 or AMD MI355 GPUs, measuring the maximum number of accelerators you can keep above 90 percent utilization
• Checkpointing: Llama 3-style models at 8B, 70B, 405B, and 1.25T parameters, timing 10x write and 10x read cycles
• Vector database (VDB): a Milvus database holding 1 million vectors at 1,536 dimensions, measuring queries per second while tracking latency and recall percentage
• KV cache: measuring how many simultaneous conversations a storage system can support

That last one is the tell here. KV cache, the key-value cache that inference engines use to avoid recomputing attention over previous tokens, has become a serious storage and memory bottleneck as context windows balloon into the hundreds of thousands of tokens. Checkpointing and vector databases are the other two big inference-adjacent workloads that didn't exist in earlier versions of this benchmark, and their inclusion tells you where MLCommons thinks the pain points actually are.

Brian Belgodere, co-chair of the MLPerf Storage working group, put it plainly: "These new additions to the benchmark suite round out the test collection, covering a larger range of AI inference workloads that drive storage needs. On the training side, we have training and checkpointing tests, and on the inference side we now have KV cache and vector database tests. Including tests that decompose monolithic AI systems and focus on specific storage uses and patterns, such as checkpointing, KV caching and vector databases, gives stakeholders a much clearer idea of how to engineer and provision AI systems to minimize storage performance bottlenecks."

Why S3 Support Matters Now

The addition of S3 as a supported data access layer is arguably the bigger structural change here. It's now supported for training, checkpointing, and some of the VDB tests, and roughly one-sixth of all submissions in this round actually used it.

Curtis Anderson, the other working group co-chair, framed the reasoning around scale: "As the scale of AI contexts reaches into the trillions, we expect object-based storage systems to emerge as a viable, and possibly preferred, alternative to filesystem-based storage. By enabling S3 support now, we are ensuring that stakeholders will have the performance information they need to make smart decisions."

That's a notable bet. POSIX filesystems have long been the default for high-performance ML training because of their low-latency random access patterns, but object storage scales more cheaply and more simply, especially in cloud-native deployments. If S3-backed systems can hold their own on checkpointing and training workloads at trillion-parameter scale, that's a real signal for anyone architecting new AI infrastructure rather than just inheriting whatever HPC filesystem was already sitting in the datacenter.

Nineteen suppliers submitted results this round. Worth flagging who's absent: DDN, Dell, Huawei, IBM, NetApp, VAST Data, and WEKA are all missing from the list, despite being names you'd expect to see chasing AI storage benchmarks hard. MLPerf's own framing was blunt about it: "Traditional NAS is thin: 19 orgs present; legacy enterprise incumbents are mostly missing."

The results themselves are a lot to digest. There's no single leaderboard number here, there are four, one per workload category, each with a pile of sub-parameters. The full results spreadsheet runs to roughly 145 rows and 55 columns, nearly 8,000 cells, and a good chunk of them are blank because vendors are picking and choosing which workloads to submit for. The KV cache workload alone eats 12 columns, covering things like:

• KVCache llama3.1-8b Storage Only - Throughput (tok/s)
• KVCache llama3.1-8b Storage Only - Read/Write B/W (GiB/s)
• KVCache llama3.1-8b Storage Only - P95 Read Latency (ms)
• KVCache llama3.1-8b Storage + Mem - Throughput (tok/s), Read/Write B/W, P95 Read Latency
• The same set of metrics again for the llama3.1-70b model

MLCommons also suggests normalizing results per rack unit and per watt, which is a smart move given how power-constrained a lot of AI datacenter buildouts have become. On-premises checkpointing write submissions hit a median of 14 GB/s/watt, with the best result reaching 201 GB/s/watt. UNet3D read tests came in at a median of 34 GB/s/watt, topping out at 277.

Selectivity in submissions is worth calling out too. Only four vendors, NewFW, Samsung, SuzhouZishan, and TTA, actually submitted VDB workload results, which suggests vector database benchmarking is either harder to run or less of a priority for most storage vendors right now.

Everpure, formerly known as Pure Storage, put up strong numbers on the higher end of the checkpointing and KV cache tests. Its FlashBlade//EXA claimed the top spot across both the 405B and 1.25T parameter model checkpointing categories, and in KV cache performance too. At the 1.25T parameter scale, using 30 data nodes and 1,024 simulated accelerators, FlashBlade//EXA hit 877.52 GiB/s write bandwidth (finishing in 17.74 seconds) and 588.28 GiB/s read bandwidth (28.99 seconds). Scaling looked clean too: write throughput went from 327.65 GiB/s with 10 data nodes up to that 877.52 GiB/s figure at 30 nodes, which is roughly linear scaling, always a good sign for anyone trying to plan capacity growth.

Key Takeaways

If you're evaluating storage for an AI inference pipeline, this version of MLPerf Storage is finally testing things that actually matter to you: KV cache pressure, checkpoint recovery times at frontier model scale, and vector database query performance under load. The S3 layer addition means you can now get real comparative data on object storage versus filesystem storage instead of relying on vendor whitepapers. Just go in expecting a genuinely complex results matrix, and don't assume every vendor tested every workload. The absence of several major enterprise storage vendors from this round is also worth watching. Either they're not ready to compete on these new inference-focused workloads yet, or they're betting the benchmark won't matter enough to bother.