Qdrant Releases 10-Billion-Vector Benchmark to Stress-Test Search at Real Scale

If you've ever tried to benchmark a vector database honestly, you know the annoying part isn't running the queries, it's knowing whether the answers you got back are actually correct. Qdrant just released something aimed squarely at that problem: a 10-billion-document dataset called Qdrant-FineWeb-10B, built with cloud provider Vultr, along with an open source benchmarking tool called Supernova that anyone can point at Qdrant or a competing system.

The scale here is the headline number, but the harder engineering problem was figuring out ground truth. To validate results against roughly 120,000 test queries, Qdrant had to run more than a quadrillion distance calculations. That's the unglamorous, expensive work that usually keeps benchmarks small: it's one thing to generate billions of vectors, it's another to know with certainty which results are the "right" ones for a given query.

Why does that matter? Because without ground truth, you can measure speed all day and still have no idea if the system is any good. A vector search engine that returns results in two milliseconds is useless if those results aren't relevant. Recall, the fraction of true nearest neighbors actually retrieved, only means something when you have a verified answer key to check against. Qdrant built that answer key at a scale nobody else has published, then made it public instead of keeping it as an internal testing asset.

Why Scale Changes the Engineering Problem

Vector search benchmarking already exists in decent shape at smaller scales. Big ANN Benchmarks, one of the more widely cited efforts in the space, has datasets reaching a billion vectors. There's also no shortage of smaller test sets that vendors use to make apples-to-apples comparisons. What's been missing is anything approaching the size that large enterprises are actually starting to run in production.

Qdrant-FineWeb-10B comes in at about 10.07 billion documents, roughly 24.5 TB of vector data. That's an order of magnitude past where most public benchmarks stop, and the jump isn't just "more of the same but bigger." A few things change qualitatively once you cross into double-digit billions:

• Memory pressure compounds. You can't just keep everything in RAM anymore; index structures and caching strategies that work fine at a billion vectors start to strain or break entirely.
• Index build time becomes its own bottleneck. Constructing something like an HNSW graph (a common approximate nearest neighbor index structure) over 10 billion points is a meaningfully different job than doing it over a million.
• Distributed sharding gets unavoidable. No single machine is handling this workload alone, which introduces network overhead, coordination costs, and new failure modes that don't show up in single-node tests.
• Speed and accuracy start trading off harder. A system can juice its throughput numbers by cutting corners on recall, and at this scale those corners get bigger and easier to hide.

That last point is really what Supernova is built to surface. The tool measures throughput, indexing performance, recall, and latency at the p50, p95, and p99 percentiles (the median, and the slower 5% and 1% tails, respectively, which matter a lot for real-world SLAs). It's designed to work against vector search systems beyond Qdrant's own, so in theory a competitor could run it and publish results too.

André Zayarni, Qdrant's CEO and co-founder, framed the release as a response to a recurring pain point. "Accurate benchmarking is one of the hardest things we get asked about, both in our open-source community and in customer discussions," he said. "Teams need real data at the size they actually run, and they need to know what the correct answers are. Assembling both yourself is expensive. We did it once and made it public."

That framing is worth taking at face value, but also worth putting in context. Qdrant sells a vector database. It has an obvious interest in defining what "good" benchmarking looks like in ways that flatter systems built the way it builds things. And realistically, most companies deploying RAG (retrieval-augmented generation, where a model pulls in external documents before generating a response) or enterprise search today aren't anywhere near 10 billion vectors. Bigger doesn't automatically mean more representative of what most teams need to test.

Still, the direction of travel is pretty clear. RAG pipelines, enterprise search, and increasingly agent-based applications are all pushing larger and larger corpora of corporate data behind retrieval layers. Document collections that used to live in the millions are creeping toward the billions as companies index everything from support tickets to internal wikis to years of historical records. At some point, "does this scale" stops being a marketing slide and becomes a genuine engineering question that needs a real test to answer.

Key Takeaways

The concrete value here isn't the marketing angle, it's the cost Qdrant absorbed on everyone else's behalf. Building a 24.5 TB dataset and computing verified ground truth across 120,000 queries with over a quadrillion distance calculations is not a weekend project. Most teams evaluating vector databases don't have the budget or patience to do that work themselves, so they either skip large-scale testing entirely or trust vendor claims without a way to verify them.

By publishing both the dataset and Supernova as open tools, Qdrant has effectively lowered the bar for anyone who wants to run a serious large-scale comparison. That includes competitors, which is either a confident move or a savvy one, depending on how their systems actually perform under Supernova's numbers. Either way, the excuse of "we didn't test at that scale because nobody could" just got a lot weaker for the whole industry.