Polign and DuckDB: Shared retrieval

I kept getting asked how Polign compares with DuckDB, so I put this comparison together. My focus is agent memory with Polign Recall, and helping multiple agents share a knowledge base through vector and semantic retrieval.

Polign Recall is the agent memory layer built on polign_db, which provides the underlying storage and retrieval service. Sharing that memory across agents raises practical questions, how do they read and write the same data and coordinate work? This post looks at how Polign (0.7.1) handles those questions, then compares its vector retrieval with DuckDB on one million vectors. The benchmark measures the database layer. I will have a separate post about evaluating Recall's memory behavior.

Shared service · Coordination · Cross-node reads · Retrieval and operations · Benchmark

The short answer

DuckDB across processes

While one process held the DuckDB database open for writing, a second process failed to open it in either read-write or read-only mode, with a conflicting-file-lock error. DuckDB's concurrency documentation describes the direct-file choices as one writing process, or multiple read-only processes with no writer. Multiple threads within the owning process are supported, as the throughput runs demonstrate.

The documentation also describes Quack, a client-server protocol in beta, and DuckLake with a PostgreSQL catalog for coordinated access. We did not test either, including their interaction with HNSW. Polign exposes HTTP and gRPC server interfaces directly; multiple clients can use them to read and write a shared collection.

From an application database to a shared service

Consider an agent application with a web server handling requests, a worker ingesting documents, and several agents reading the resulting memory. They may run on different machines and restart independently. Alongside search quality, the application needs a durable place for writes, a way to serve reads from another node, and rules for who updates the index.

With Polign's -store deployment, the write log, persisted segments and index metadata live in the configured store. Serving nodes keep caches and working state that can be rebuilt from it. Multiple writable servers can use the same store; adding another server does not require making its local disk the authoritative copy of the database. Across machines, use a shared object store. Two private fs: directories are separate stores.

You can also run a dedicated read tier with -read-only and read-only bucket credentials. Those nodes reject mutations and do no persistence or maintenance. They refresh published segment generations, so they can lag the writable nodes' latest log entries. A replacement writable node opens the same store and rebuilds its serving state from persisted data and the retained write tail. Recovery time depends on that state and the storage service; the process-restart timings later in this post are local measurements.

How the nodes coordinate

The object store supplies both durable storage and the conditional writes used for coordination. This deployment does not require a separate log broker or coordination service. Three mechanisms handle different jobs:

1. Ordering incoming writes. A write is acknowledged after its log batch reaches the store. Writers claim log sequence numbers with conditional object creation; a writer that loses a claim retries. Writes for a collection share a log partition and are replayed in order. This lets several processes append without overwriting each other's log entries. A conditional schema claim also makes concurrent first writes agree on a collection's dimension and distance metric before they are logged. This ordering does not provide a transaction spanning multiple collections, and contention or an unavailable store can still cause a write to fail.
2. Assigning background work. Persistence turns the log into segments; maintenance compacts and builds indexes. Bucket-backed leases select an active worker for each guarded job while others stand by. A standby can acquire an expired lease if its holder disappears. The default lease lifetime is 30 seconds, with takeover also depending on polling and storage latency. Lease safety relies on bounded clock skew and stopping guarded work when the lease is no longer valid. This is distinct from the ability of several nodes to accept application writes.
3. Assigning collections to serving nodes. Optional -node-id and -peers configuration enables deterministic collection placement using rendezvous hashing. Every node must agree on membership. In the 0.7.1 server CLI, each collection has one assigned owner in this mode; requests to another node return an owner hint, surfaced as HTTP 421 or gRPC FailedPrecondition. The client or gateway must route accordingly. This distributes collections across a fleet; it does not split every query across all nodes.

The Go service layer also has configurable owner sets, draining and load-based expansion of a busy collection's owner set. These are integration controls: the 0.7.1 server CLI does not expose all of them, and its peer configuration is static. Fleet discovery, membership changes and machine autoscaling need an external deployment or control layer. The architecture guide describes the placement model.

What happens when the next read goes to another node?

An acknowledgement establishes that the write reached the store. Another serving node may still be catching up. Writable nodes merge a log-tail overlay with persisted search results, but remote writes are discovered by polling. After an idle period, that polling interval can reach 10 seconds, plus storage latency; a successful write in the same process wakes its own readers immediately.

For a workflow such as “the worker writes a document, then an agent on another host reads it,” Polign 0.7.1 provides an explicit HTTP contract. Store-backed, cold-first writes return X-Polign-Write-Token. Send that receipt as X-Polign-Require-Write-Token on a subsequent point read or vector-only query. The receiving node must establish that its serving state covers the write before answering.

The coverage check waits up to five seconds, or the client's earlier cancellation. If coverage cannot be established, it returns 503 with a retry hint instead of silently answering from state that predates the write. A read-only node can satisfy the token once its loaded persisted index includes that write. The token does not accelerate replication or freeze a snapshot: later updates and deletes can still be visible, and approximate search can still miss a neighbour. This contract currently covers HTTP point reads and vector-only queries, not text/hybrid queries or gRPC convenience calls.

What else the service handles

• Vector, keyword and hybrid retrieval. Polign supports metadata-filtered vector search, BM25 keyword search and hybrid queries combining both. Text and hybrid search use the segment index. An application can search for an exact product name and a semantically related description through the same service.
• Record lifecycle. Batch upserts and gets, filtered listing, metadata patches and deletion by id or filter support ingestion and document cleanup. Metadata patches avoid making the client resend an embedding just to change a document's label or access-list metadata.
• Different serving budgets. The cold path reads persisted segments on demand; the optional hot tier brings busy collections' compressed search structures into memory. Segment and disk caches have configurable budgets. A configurable cap on unpersisted records applies backpressure if persistence falls behind. These controls address different sources of memory use; a cache budget is not a total process RAM cap.
• Service access and data ownership. Deployments can require API keys, scope keys to namespaces, enable TLS, and encrypt stored objects inside the process. The optional collection registry can bind collections to customer-owned S3 or GCS backends. These controls are enabled through deployment configuration. The encryption guide explains the storage encryption options.

These capabilities were checked against the 0.7.1 release source and documentation. The measurements below test a single local serving node. They do not measure fleet scaling, lease takeover, cross-host freshness or object-store performance.

What we measured on one machine

At about 0.98 recall, Polign's p50 latency was 4.5 ms and DuckDB's was 5.8 ms. With eight clients, Polign served 650 queries a second and DuckDB 930. The measurements help choose a local serving configuration; the architecture above addresses how the service is shared and operated.

Loading the data

These are one-time builds of the same corpus. DuckDB takes 2 m 54 s from an Arrow table through a checkpointed HNSW index. Polign takes 5 m 49 s from the binary vector file through trained and packed IVF-PQ segments. Each system retains the full vectors as well as its search index. Build and disk-size measurements are single trials.

Search latency and recall

Each row pairs a DuckDB setting with a Polign setting at similar measured recall, ordered from lower to higher recall. These are approximate matches; the exact recall for each engine is shown. The final row compares the highest recall tested for each. Latency is one client issuing queries serially. Throughput comes from separate 15-second runs with eight client threads.

Approximate-search entries are medians of three rounds of 1,000 queries per setting, with shuffled setting order and alternating engine order. QPS is also the median of three runs. The two remaining sweep settings are shown below to preserve the full results; they are not a matched pair.

DuckDB without an index (exact scan) reached 1.0000 recall, with 294 ms p50 and 632 ms p99. This was a fresh single run of 20 queries before index creation; its p99 has a small sample and its throughput was not measured.

At roughly 0.98 recall, Polign nprobe 64 reaches 0.9786 recall at 4.5 ms p50; DuckDB ef_search 256 reaches 0.9789 at 5.8 ms. With eight clients the same settings serve 650 and 930 queries a second respectively. Single-query latency and throughput under concurrent load are separate measurements; choose the column that matches your application's traffic.

At those settings, p50 across the three rounds ranged from 4.5 to 5.3 ms for Polign and 5.6 to 5.8 ms for DuckDB. Eight-client throughput ranged from 624 to 667 QPS for Polign and 861 to 992 QPS for DuckDB.

Polign's automatic setting probes 124 of the 1,000 IVF cells and re-ranks 100 candidates against their full vectors. It reaches 0.9904 recall here. The explicit 250-probe setting reaches 0.9962. DuckDB's largest tested ef_search, 1024, reaches 0.9933. These are measured points on each system's speed/recall curve, not their maximum possible recall. Neither HNSW construction parameters nor Polign's rerank pool were swept.

DuckDB queries use precomputed vector literals in SQL and the verified HNSW_INDEX_SCAN plan. Polign queries use precomputed Python vector lists over gRPC. Timings include request execution and result handling, but exclude construction of those literals and lists. This is a comparison of these application paths, not isolated index kernels.

Memory and startup

Startup and first-answer RSS are medians of three fresh processes using each engine's default search setting. DuckDB's timer includes opening the file, loading VSS and answering the query. Polign's includes starting the server, connecting and answering. OS caches were not cleared, so these are warm-cache starts, not cold-disk measurements.

RSS means resident process memory. The DuckDB measurement includes its Python host; the Polign measurement is the server alone. Polign's separate Python client used 87–90 MiB under load. The load ranges span samples after all settings in all three rounds; they are not peak measurements. Neither figure includes the operating system's file cache or compressed-out pages. macOS memory compression was active during this run. We did not impose a memory limit or isolate the machine from its other applications.

DuckDB's VSS documentation says its HNSW index must fit in RAM and is outside the buffer manager and memory_limit. Polign reads the probed cells' compressed codes and fetches full vectors for reranking. Its segment cache was the default 256 MiB. The measured RSS difference is specific to this corpus and cache configuration; it does not establish a minimum machine size or how memory grows at larger scales.

Writes and durability

This table uses a matched batch of 1,000 new vectors, added after an initial 1,000-write batch: both systems start this trial with 1,001,000 records. DuckDB used precomputed SQL literals, one autocommit insert per row into the indexed table; Polign used one gRPC put per row. All inserted records were present with exact float32 payloads in both systems. The table separately reports whether approximate search returned each vector as its own top-1 result at the default settings. An ANN miss is distinct from a missing stored record. Both engines answered a query before write timing started. These write timings are one run, including client overhead.

The Python call path matters. In the initial matched batch, DuckDB inserts using bound vector parameters measured 37.9 ms p50 and 26 writes/s; Polign measured 10.1 ms and 55 writes/s. The table uses the literal path to avoid making Python parameter binding the headline database comparison. Literal formatting is outside the timer, as in the search test; scientific notation forces DOUBLE literals before the FLOAT cast to preserve the input float32 values. Both batches' stored vectors were checked byte-for-byte. These are application-path timings, not isolated storage-engine timings.

Polign appends a write to the configured store's log before acknowledging it. DuckDB uses its write-ahead log, but its VSS persistence documentation identifies incomplete recovery support for custom indexes and possible index corruption or data loss after an unexpected shutdown with uncommitted changes. Persisting HNSW requires hnsw_enable_experimental_persistence = true. We enabled it and did not crash-test DuckDB.

• Polign process crashes. Four clients wrote continuously to a separate small store while we killed the server with SIGKILL, in three trials. Every one of 2,690 acknowledged records returned with its exact float32 payload after restart. The last acknowledged record was searchable in each trial. Restart to the first answer took 0.95–1.19 s on that store.
• Graceful restart. After the first write batch, the million-vector Polign store, including those 1,000 added writes, answered after 2.46 s at 127 MiB RSS. All 1,000 added vectors were verified byte-for-byte.
• Machine loss. These tests leave the local disk intact. They do not establish survival of disk loss, power loss or an object-store outage.

Using both

DuckDB can export ids and vectors as Parquet for Polign to import:

-- in DuckDB
COPY (SELECT 'doc-' || id AS id, v FROM items) TO 'items.parquet' (FORMAT parquet);

# then load it into a Polign store
polign-import -store fs:./memory -collection items \
  -metric cosine -vector-column v items.parquet

We ran this path on 20,000 benchmark rows and checked every imported id and vector byte-for-byte. This makes it possible to keep SQL analysis in DuckDB and serve a shared vector collection through Polign.

How we ran it

• Versions. Polign server and importer 0.7.1, release commit 361df1009c31, built with the release flags. DuckDB 1.5.5, VSS b833341. Python 3.14.7 and Polign's Python SDK 0.7.0. Full package versions, binary checksums and input checksums are in the downloadable evidence.
• Index settings. DuckDB: cosine HNSW, M 16, M0 32, ef_construction 128, 12 engine threads. Polign: 1,000 IVF cells, 192-byte PQ codes, compacted and packed segments, exact rerank pool 100. The server ran with -store fs:DIR -hot-max 0 -maintain 0 and telemetry disabled.
• Ground truth. Exhaustive float64 cosine top 10 across exactly the same million rows in both engines, recomputed for all 1,000 queries. DuckDB's unindexed scan independently matched the ground truth on its 20 test queries. We also recomputed exact ground truth after SQL literal conversion: all 1,000 top-10 sets were unchanged. All search measurements preceded the write tests.
• Repetitions and caches. Three fresh processes per engine, alternating which engine runs first, with settings shuffled inside each round. Each process received 200 warmup queries. Engines ran sequentially. OS caches were not cleared. "Hot tier off" means Polign did not build its optional in-memory serving index; it does not mean cold disk access.
• Throughput. Eight Python client threads with staggered positions in the query corpus, a shared start barrier, and 15 seconds of requests. QPS divides completed requests by actual elapsed time, including the last in-flight requests. Errors fail the run.

Build and write timings are single trials; the repeated search results describe variability on this machine with one index build per engine.

What this test does not cover

• Object storage: neither engine was benchmarked against S3, GCS or Azure Blob.
• Fleet behavior: this benchmark did not test multi-node throughput, lease takeover, placement changes or cross-host write-token reads.
• Other serving modes: Polign's hot tier, Quack and DuckLake were not benchmarked.
• Other workloads: no filters, text search, hybrid search, mixed read/write traffic or network between machines.
• Other sizes: one machine, one dataset, one corpus size, no enforced RAM cap.
• Exhaustive tuning: no sweep of HNSW construction parameters, larger ef_search values or Polign rerank settings.