Benchmarks

The benchmarks measure three criteria: speed, accuracy and cost.

The short version

The headline numbers primarily revolve around costs. About $3.20 per million searches served straight from cloud storage with no index in RAM. That's at the 8-region setting, which measured 84% accuracy on the 1M test; the 98%-accuracy setting checks 32 regions and costs about $12.80. And about $0.01 per million writes through a write log kept directly in the bucket, with no separate queue server (like Kafka) in the middle. When everything is hot, polign_db's in-RAM path answers in well under a millisecond, though the RAM-resident engines in the comparison below are faster still.

Comparison with other databases

• In memory search
  • Qdrant: open benchmark (2024, 8-vCPU server) reports ~1,260 searches/sec at 96.7% accuracy on 1M 1536-dim vectors
  • Weaviate close behind (~1,140 at 97.5%)
  • Milvus trailing on queries (~220/sec) while indexing far faster than everyone.
  • Polign measures ~505/sec at 94.6% on the same core count, but on 768-dim vectors, half the width Qdrant was carrying, and each 1536-dim distance computation costs roughly twice as much. Adjusted like-for-like, Qdrant is roughly 3–5× faster here, and Weaviate reaches it at higher accuracy than Polign's 94.6%. In-memory speed is not where Polign wins; the cloud-storage rows below are.
• Cloud native storage search is the more meaningful comparison, because almost nobody else publishes it.
  • turbopuffer reports ~14–16 ms typical for cached 1M-document namespaces (a namespace is their unit of a database) and ~400–900 ms uncached
  • Polign measures 18 ms against a same-machine store and ~144 ms typical (p99 ~354 ms) at 98% accuracy against Amazon S3; lower-accuracy settings answer in as little as ~80 ms
  • besides turbopuffer, the other systems in this comparison either don't serve straight from cloud storage or don't publish numbers for it.
• Writes. Both engines append writes to a log kept in object storage, where writes that queue up behind an upload ride along on the next one, so the per-log rate is the honest head-to-head.
  • turbopuffer documents 10,000+ writes/sec per namespace at ~165 ms typical confirmation
  • Polign measures ~6,200 writes/sec per log at ~161 ms typical, and turbopuffer's published figure is ~1.6× higher at the same confirmation-latency class
  • Polign's default deployment runs 16 independent logs, putting its aggregate ceiling around 100,000 writes/sec, but that is arithmetic from the measured per-log rate, not a separate measurement, and turbopuffer namespaces multiply out the same way.

Cold and warm, in the format the category publishes

Every object-storage-backed engine has a cold number and a warm number, and the honest way to read this table is that the cold column is the cost of not renting RAM. We publish ours in the same shape everyone else does rather than quoting only the warm figure:

Read across the cold column and the numbers land in the same band, because they are all measuring the same physical thing: an object-storage round trip. Ours is measured end-to-end at a stated recall on a stated instance; several of the others are published as a range or a design target. Where the designs actually diverge is the warm column and what it costs to get there, which is the next table.

Scale-to-zero rehydration: what a cold fleet costs to bring back

If nodes are disposable, the number that decides your bill is not the first query's latency but how long a brand-new node takes to become useful, and how much of that is rebuild work versus a download. We have not found this published by anyone else in the category, so here is ours. A node boots by fetching a pre-built index generation from the bucket; it never rebuilds:

The build runs once per published index version; every boot, restart, and scale-out after that reuses it. So the cost of scaling to zero overnight is one boot-time fetch per node in the morning, plus whatever cold queries run before the heat mechanism promotes the hot collections back into RAM. Two caveats: the boot-time fetch is single-threaded today, and parallelizing it is the next engine change; and the 100M boot figure is from local disk, so the same-region S3 equivalent will be somewhat higher.

The architectural comparison behind these benchmarks is detailed here - comparison page.

Search speed — in memory

Each curve below is a trade-off from fast-and-rough to slow-and-thorough.

Search speed — cloud-native storage

Polign cold search is executed with no index in memory at all. Each search downloads just the pieces of the index it needs from a storage bucket (S3, GCS, Azure Blob, MinIO) and searches them on the spot. That makes the cost of a search unusually easy to state, because cloud storage bills per download: a search that checks 8 regions of the index makes 8 downloads, and at S3's price of $0.40 per million downloads, a million such searches cost about $3.20.

How big each download is gets decided when the index is built: splitting the same data into fewer regions means bigger files but fewer downloads per search. Here is what three different splits of the same 1-million-vector dataset produced:

Cold search costs

The defaults are S3 list prices; downloads within the same AWS region carry no bandwidth charge, so bandwidth only matters for cross-region or internet clients.

Write speed and cost

In this setup, every write is appended to a log that lives in the storage bucket itself. When many writers are active at once, writes that queue up behind an upload already in flight all ride along in the next one. So a single upload can carry hundreds of writes.

That batching is the entire difference between 33 and 6,233 writes per second measured against the same S3 bucket. It also sets the cost: S3 charges $5.00 per million uploads, so the cost of a write is that price divided by how many writes share an upload.

The default deployment spreads writes across 16 logs, and because the logs never coordinate, throughput scales with the log count. The measured ~6,200/sec per log puts the 16-log default at roughly 100,000 writes/sec aggregate.

Compression. Smaller index, nearly the same answers

Instead of keeping every full vector, Polign can keep a compact fingerprint of each one, 8–64× smaller in the tests below (the headline numbers on this page use the 32× setting). Searching fingerprints alone gives up some accuracy, so an optional second pass re-checks the top candidates against the true vectors before answering. The charts show how much accuracy the fingerprints alone give up, and how much of it the re-check wins back.

Scale test with 10 million and 100 million vectors

This section reruns the same three paths, memory, cloud storage and compression, at 10× and 100× the data, using the 10M and 100M slices of the BigANN benchmark (SIFT1B). BigANN publishes official answer keys at exactly these sizes, so the accuracy numbers are verified against the same ground truth.

In short: at scale, search time is still bounded by the storage round trip. A one-region cold search costs ~80 ms median at 1M, ~87 ms at 10M, and ~96 ms at 100M. The median region file grew from ~0.6 MB at 1M to ~3.5 MB at 100M and the median barely moved, because S3's per-download first-touch price dominates the bytes.

Compression at scale: fingerprints are 8–64× smaller than the raw vectors, and the compressed tier checks a fixed 128 regions per search regardless of corpus size, so speed barely moves with scale: ~300 searches/sec for 32-byte codes at both 10M (8 vCPU) and 100M (16 vCPU), at 25 and 46 ms median. Deepening the re-check pool buys back the accuracy the fixed probe gives up: at 100M, 32-byte codes reach 98.8% with a 100-deep re-check and 16-byte codes 98.6% re-checking 500; at 10M the same settings reach 99.4%+. Two caveats: the bounded scan concedes the last point of accuracy (a wide probe hit 99.75% at 100M, but at 1/70th the speed), and 8-byte codes stay weak at scale, reaching just 91.1% at 100M even re-checking 500, their ranking too coarse for all the true neighbours to make the pool. With the full vectors in same-region S3 instead of RAM, re-check depth is fetch traffic: 50/100/200/500 finalists cost 26/52/104/255 MB per search for 89.3/96.7/99.3/99.7% accuracy at 0.46–2.5 s median (measured at 10M). The chart's data table has the full breakdown.

Serving from a compressed replica

A replica node holds only the fingerprints in RAM (1.6 GB of fingerprints, ~2 GB structure in total, at 100M); the full vectors stay in cloud storage, and each search re-checks its finalists by downloading only the few kilobytes it needs from inside each file.

• Fetch cost: a 100-deep re-check downloads 51 KB at 201 ms median (10M, same-region S3). The earlier engine fetched whole 512 KB blocks, which came to ~52 MB and 628 ms for the same accuracy; reading only the needed byte ranges is ~1,000× less data.
• Accuracy (100M): 93.0% with a 100-deep re-check, 98.3% at 500. Fingerprints only (the opt-in fast tier): 45.9% at 13 ms, zero downloads.
• Boot: nodes fetch the pre-built structure instead of rebuilding. That takes 5.2 s at 10M from same-region S3 (3.3 s from local disk; 36 s at 100M, measured on local disk), vs. index builds of 18 min (8 vCPU) and 8.4 h (16 vCPU). The build runs once per published index version; every boot, restart and scale-out reuses it. It is single-threaded today; parallelizing it is the next engine change.
• Exact reads for stored vectors is about 25 ms per single fetch from same-region S3 (0.01 ms cache-warm), ~1 ms/vector in 100-vector batches.

Micro-benchmarks: transport and engine

Server on an 8-vCPU Amazon EC2 instance (Ubuntu, us-west-2); the client's 16 workers on a separate t3.small in the same availability zone. That's a network hop, so the server's CPU is all its own. dim=128, k=10. Illustrative, not guarantees, because your hardware will differ.

Transport overhead (10k×128, 16 workers)

Over http/grpc the server seeds its default IVF-over-HNSW backend, so the local rows show both that default (the apples-to-apples transport comparison) and the pure-HNSW in-process ceiling. The local rows run on the server instance itself, so they're in-process by definition.

gRPC beats HTTP/JSON by ~23% on throughput with a tighter tail (p99 2.86 vs 3.90 ms). That's protobuf + HTTP/2 vs JSON + HTTP/1.1. One oddity, deliberately left visible: the local IVF p99 (~21 ms) is worse than gRPC's, and it is not a loopback artifact. It persists on an otherwise idle machine. At in-process saturation (~15k QPS) the driving process shares the engine's GC, so stop-the-world pauses land in its own tail; the served paths run the same backend ~30% below that saturation and don't show it.

Engine hot path: allocation-free search (50k×128, all cores)

A matched before → after pair on the same 8-vCPU EC2 instance, isolating the search hot-path optimization, which is SIMD distance kernels plus a pooled, allocation-free search scratchpad (visit-version visited set). BenchmarkSearchParallel reports per-parallel-op wall time, so aggregate throughput ≈ 1e9 / ns·op.

Stripping per-query allocation out of the engine (1,162 → 1 allocs/op on the raw index) is what lifts concurrent throughput: fewer allocations mean less GC work competing across cores, and the ~5–8× AVX2+FMA distance kernels compound the win. The service search calls straight into the engine, so it inherits both.

Scaling: HNSW vs IVF-PQ as the corpus grows

How these numbers were produced

Everything on this page comes from polign_db's own benchmark harness, run on standard EC2 instances against local disk, MinIO, and same-region Amazon S3, over public datasets: SIFT1M, VectorDBBench's cohere_medium_1m, and BigANN subsets for the 10M/100M tiers. Each chart states its instance type, dataset, and configuration, and every chart has a "view data as table" toggle backed by the raw JSON reports, including median and worst-case timings for every point.