12.5 million Wikipedia passages at 2,000 searches a second

The Wikipedia demo has been running on a t4g.small since August, which answers the question of how cheaply polign_db can serve a large corpus. It does not answer how fast. So we pointed a dedicated benchmark at the same bucket the demo reads, gave it real hardware, and measured.

The corpus is every English Wikipedia article: 12,519,135 passages, an 81.7 GiB index living in S3 across 142,653 objects. Nothing is preloaded into memory. The server starts empty, reads index segments out of object storage as queries arrive, and keeps whatever it has room for.

On one 16-vCPU instance, that arrangement answered 2,033 searches a second with a 40 ms p99, and 1,027 a second at the higher of the two accuracy settings. Both figures held flat for twenty minutes: 2.44 million queries in the first case, zero errors in either. Below are the numbers and the exact configuration that produced them.

The best result

A c7g.4xlarge (16 vCPU, 30 GiB), reading the index from S3 with a warm local cache, at concurrency 32:

Twenty-minute runs, zero errors in both, and each reproduces a ten-minute run of the same configuration to within 0.2%. Latency is measured by a separate load generator on its own instance and spans the full round trip through the HTTP endpoint, including receipt of the complete JSON response with passage text and metadata. Query embedding happens on the client and is excluded, because this measures the database. Hit@10 here is scored on the 10,000-question load set described below.

A single 4-vCPU box does the same work at a quarter of the scale, which is the more interesting number if you are sizing for a real application rather than a headline. On a c7g.xlarge (4 vCPU, 8 GiB) at concurrency 8:

Ten-minute runs, zero errors in both. This node is bounded by its volume rather than its processors, so it holds the same rate at concurrency 16 and gains nothing above it.

What the accuracy is

Throughput without accuracy is not a result, so both were measured on the same index. The retrieval set is NQ-Open, a published question answering benchmark. A hit means a returned passage contains the annotated answer. Two disjoint sets were used: 100 held-out questions for the accuracy sweep below, and 10,000 training questions to drive the load runs.

100 held-out NQ-Open questions, scored sequentially with no concurrent load. Answer presence in a retrieved passage is a retrieval proxy, not generated-answer accuracy: short or common answers can produce false positives, and a 2023 Wikipedia snapshot can disagree with date-sensitive questions. The 10,000-question load set scores lower in absolute terms (0.508 at nprobe=8) because it is a different, larger set of questions, not because those runs retrieved worse.

nprobe=4 is the setting worth knowing about. Going from 8 probes to 4 costs two points of Hit@10, from 0.600 to 0.580, and doubles throughput. Going to 2 probes doubles throughput again but costs ten points, which is a different product. If you want maximum accuracy, nprobe=8 still answers a thousand searches a second on one instance.

The configuration

Nothing here is exotic. The whole result comes from four settings.

# c7g.4xlarge, 16 vCPU / 30 GiB, GOMEMLIMIT=20GiB
# disk cache 80 GiB, enough for the whole 81.7 GiB index
# segment cache 2 GiB, held in memory
polign-server \
  -segment-stores s3://your-bucket/your-prefix \
  -cold-first=true \
  -disk-cache-dir /var/lib/polign/cache \
  -disk-cache-bytes 85899345920 \
  -segment-cache-bytes 2147483648 \
  -hedge-reads 0 \
  -http 0.0.0.0:23000

Give the disk cache a volume worth caching on

This is the setting that mattered most, and it is easy to get wrong. The cache lives on an EBS volume, and a gp3 volume ships at 125 MiB/s by default. That is slower than the network the cache is supposed to be replacing, so a default volume makes the cache a downgrade. Provisioned at 1,000 MiB/s and 16,000 IOPS, the same cache is transformative:

Both rows are twenty-minute runs at concurrency 32, and both scored an identical Hit@10 of 0.470 on the same question set. The cache changes how fast an answer arrives, never what the answer is.

The reason the gap is twentyfold is worth stating plainly. Each search reads about 17 MiB of index at nprobe=4, and twice that at nprobe=8, because this collection stores full-precision vectors and a search reads whole index cells. Served from S3, throughput is simply the instance's network allowance divided by that number. Once the cache is warm, the object store leaves the request path entirely, the reads come from local storage and page cache, and the limit becomes the processor instead. Every cached run above recorded essentially zero bytes from S3.

The cache warms itself. There is no preload step and no separate build: the first pass of traffic populates it, and a ten-minute warm run was enough to reach steady state on this corpus. The volume needs to be big enough to hold the working set, which is why the example above gives it 80 GiB for an 81.7 GiB index.

Turn hedged reads off

Hedging issues a duplicate read when the first one is slow, spending bandwidth to cut tail latency. That is a good trade when bandwidth is spare and a bad one when it is the constraint. Switching it off was worth 19% more throughput on a matched pair of runs, at 3.3 hedged reads per query.

Size memory for the concurrency you want

Each in-flight search holds the cells it is reading, so transient memory runs at roughly concurrency × nprobe × 8 MiB. At nprobe=8 and concurrency 32 that is about 2 GiB of working buffers on top of the segment cache. Budget for it and the node is stable at any concurrency tested; budget under it and the process is killed rather than slowed. Concurrency 32 was the sweet spot on 16 vCPU, and 8 on 4 vCPU.

Cold start

A node with an empty cache answers its first query in 3.1 seconds, then settles. There is no index load, no warmup pass and no restore step, because the bucket is the system of record and the server reads from it directly. That is what makes the node disposable: you can replace it, scale it out, or run it on spot capacity, and the only cost of losing one is the cache it had warmed.

What this adds up to

One instance, one bucket, no preloading, and no separate serving tier: 12.5 million passages searched two thousand times a second with a 40 ms p99, or a thousand times a second at the better accuracy setting. On a 4-vCPU box the same corpus serves 65 searches a second, which is more than most applications ever ask of it, for the bill described here.

The headline numbers come from a big volume and a warm cache. The architecture that makes them possible is the same one that lets the demo run on a t4g.small for $4.60 a month: the corpus lives in object storage, the node holds nothing durable, and you decide how much hardware to point at it. The benchmarks page has the ANN measurements, and the architecture page covers how the read path works.

Try it

The demo is live at demo.polign.com, serving the same index these numbers were measured against, on a much smaller box. To run this on your own bucket, start with the getting started guide.