Benchmarks

The benchmarks measure three criteria: speed, accuracy and cost.

The short version

The headline numbers primarily revolve around costs. About $3.20 per million searches served straight from cloud storage with no index in RAM — that's at the 8-region setting, which measured 84% accuracy on the 1M test; the 98%-accuracy setting checks 32 regions and costs about $12.80. And about $0.01 per million writes through a write log kept directly in the bucket — no separate queue server (like Kafka) in the middle. When everything is hot, polign_db's in-RAM path answers in well under a millisecond — though the RAM-resident engines in the comparison below are faster still.

Comparison with other databases

• In memory search
  • Qdrant: open benchmark (2024, 8-vCPU server) reports ~1,260 searches/sec at 96.7% accuracy on 1M 1536-dim vectors
  • Weaviate close behind (~1,140 at 97.5%)
  • Milvus trailing on queries (~220/sec) while indexing far faster than everyone.
  • Polign measures ~505/sec at 94.6% on the same core count — but on 768-dim vectors, half the width Qdrant was carrying, and each 1536-dim distance computation costs roughly twice as much. Adjusted like-for-like, Qdrant is roughly 3–5× faster here, and Weaviate reaches it at higher accuracy than Polign's 94.6%. In-memory speed is not where Polign wins; the cloud-storage rows below are.
• Cloud native storage search — the more meaningful comparison, because almost nobody else publishes it.
  • turbopuffer reports ~14–16 ms typical for cached 1M-document namespaces (a namespace is their unit of a database) and ~400–900 ms uncached
  • Polign measures 18 ms against a same-machine store and ~144 ms typical (p99 ~354 ms) at 98% accuracy against Amazon S3; lower-accuracy settings answer in as little as ~80 ms
  • besides turbopuffer, the other systems in this comparison either don't serve straight from cloud storage or don't publish numbers for it.
• Writes — both engines append writes to a log kept in object storage, where writes that queue up behind an upload ride along on the next one, so the per-log rate is the honest head-to-head.
  • turbopuffer documents 10,000+ writes/sec per namespace at ~165 ms typical confirmation
  • Polign measures ~6,200 writes/sec per log at ~161 ms typical — turbopuffer's published figure is ~1.6× higher at the same confirmation-latency class
  • Polign's default deployment runs 16 independent logs, putting its aggregate ceiling around 100,000 writes/sec — but that is arithmetic from the measured per-log rate, not a separate measurement, and turbopuffer namespaces multiply out the same way.

The architectural comparison behind these benchmarks is detailed here - comparison page.

Search speed — in memory

Each curve below is a trade-off from fast-and-rough to slow-and-thorough.

Search speed — cloud-native storage

Polign cold search is executed with no index in memory at all. Each search downloads just the pieces of the index it needs from a storage bucket (S3, GCS, Azure Blob, MinIO) and searches them on the spot. That makes the cost of a search unusually easy to state, because cloud storage bills per download: a search that checks 8 regions of the index makes 8 downloads, and at S3's price of $0.40 per million downloads, a million such searches cost about $3.20.

How big each download is gets decided when the index is built: splitting the same data into fewer regions means bigger files but fewer downloads per search. Here is what three different splits of the same 1-million-vector dataset produced:

Cold search costs

The defaults are S3 list prices; downloads within the same AWS region carry no bandwidth charge, so bandwidth only matters for cross-region or internet clients.

Write speed and cost

In this setup, every write is appended to a log that lives in the storage bucket itself. When many writers are active at once, writes that queue up behind an upload already in flight all ride along in the next one. So a single upload can carry hundreds of writes.

Compression. Smaller index, nearly the same answers

Instead of keeping every full vector, Polign can keep a compact fingerprint of each one — 8–64× smaller in the tests below (the headline numbers on this page use the 32× setting). Searching fingerprints alone gives up some accuracy, so an optional second pass re-checks the top candidates against the true vectors before answering. The charts show how much accuracy the fingerprints alone give up, and how much of it the re-check wins back.

Scale test with 10 million and 100 million vectors

This section reruns the same three paths — memory, cloud storage, compression — at 10× and 100× the data, using the 10M and 100M slices of the BigANN benchmark (SIFT1B). BigANN publishes official answer keys at exactly these sizes, so the accuracy numbers are verified against the same ground truth.

In short: at scale, search time is still bounded by the storage round trip. A one-region cold search costs ~80 ms median at 1M, ~87 ms at 10M, and ~96 ms at 100M. The median region file grew from ~0.6 MB at 1M to ~3.5 MB at 100M and the median barely moved, because S3's per-download first-touch price dominates the bytes.

Compression at scale: fingerprints are 8–64× smaller than the raw vectors, and the compressed tier checks a fixed 128 regions per search regardless of corpus size — so speed barely moves with scale: ~300 searches/sec for 32-byte codes at both 10M (8 vCPU) and 100M (16 vCPU), at 25 and 46 ms median. Deepening the re-check pool buys back the accuracy the fixed probe gives up: at 100M, 32-byte codes reach 98.8% with a 100-deep re-check and 16-byte codes 98.6% re-checking 500; at 10M the same settings reach 99.4%+. Two caveats: the bounded scan concedes the last point of accuracy (a wide probe hit 99.75% at 100M, but at 1/70th the speed), and 8-byte codes stay weak at scale — 91.1% at 100M even re-checking 500, their ranking too coarse for all the true neighbours to make the pool. With the full vectors in same-region S3 instead of RAM, re-check depth is fetch traffic: 50/100/200/500 finalists cost 26/52/104/255 MB per search for 89.3/96.7/99.3/99.7% accuracy at 0.46–2.5 s median (measured at 10M) — the chart's data table has the full breakdown.

Serving from a compressed replica

A replica node holds only the fingerprints in RAM (1.6 GB of fingerprints, ~2 GB structure in total, at 100M); the full vectors stay in cloud storage, and each search re-checks its finalists by downloading only the few kilobytes it needs from inside each file.

• Fetch cost: a 100-deep re-check downloads 51 KB at 201 ms median (10M, same-region S3). The earlier engine fetched whole 512 KB blocks — ~52 MB and 628 ms for the same accuracy; reading only the needed byte ranges is ~1,000× less data.
• Accuracy (100M): 93.0% with a 100-deep re-check, 98.3% at 500. Fingerprints only (the opt-in fast tier): 45.9% at 13 ms, zero downloads.
• Boot: nodes fetch the pre-built structure instead of rebuilding — 5.2 s at 10M from same-region S3 (3.3 s from local disk; 36 s at 100M, measured on local disk), vs. index builds of 18 min (8 vCPU) and 8.4 h (16 vCPU). The build runs once per published index version; every boot, restart and scale-out reuses it. It is single-threaded today; parallelizing it is the next engine change.
• Exact reads for stored vectors is about 25 ms per single fetch from same-region S3 (0.01 ms cache-warm), ~1 ms/vector in 100-vector batches.

Run these yourself

Everything on this page can be regenerated from the repository:

# fetch datasets into bench/ann/data (SIFT1M + VectorDBBench cohere_medium_1m)
go build -tags cloud -o bench/ann/ann ./bench/ann
bench/ann/run_all.sh          # writes bench/ann/results/*.json
SCALE=10m bench/ann/run_scale.sh  # the 10M/100M tiers (SCALE=100m; BigANN subsets, see bench/ann/README.md)
bench/ann/gen_webdata.py      # regenerates assets/js/bench-data.js for this page

Every chart has a "view data as table" toggle, and the raw JSON reports (including median and worst-case timings for every point) are committed alongside the harness.