Install & deploy polign_db

One static binary; one flag for durability. This page covers installing polign_db, operating it in production, and the reference material — API, tuning, benchmarks. Once your server is up, the user guide walks through adding data and searching it.

Install

Each release archive ships static binaries for linux/darwin (amd64/arm64) with S3, GCS, and Azure Blob support built in — no Go toolchain, no Docker. Download from the releases page, unpack, run:

tar -xzf polign_db_darwin_arm64.tar.gz

# for development: in-memory, zero config, write and query immediately
./polign-server

# durable, local: point it at a directory — no cloud account needed
./polign-server -store fs:/var/lib/polign

# durable, S3: point it at your bucket — that's the whole deployment
./polign-server -store s3://my-bkt/polign

Either way you now have an HTTP API on port 23000 and a gRPC API on port 23001. The archive also contains polign, the command-line client the user guide uses — it talks to http://localhost:23000 by default, and its every command is in the CLI reference.

-store turns on the full data plane inside that one process: a durable write-ahead log in the bucket (.wal/, appended before every ack — no broker), persistence, bucket-backed serving with fresh reads, a bounded local disk cache, and the background maintenance that keeps big collections memory-efficient. The API is identical with or without it, so upgrading later is adding the flag.

export AWS_ACCESS_KEY_ID=AKIA... AWS_SECRET_ACCESS_KEY=... AWS_REGION=us-east-1
aws s3 mb s3://my-bkt --region us-east-1   # create the bucket once

./polign-server -store s3://my-bkt/polign

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
gcloud storage buckets create gs://my-bkt   # create the bucket once

./polign-server -store gcs://my-bkt/polign

az storage container create --account-name myaccount --name my-ctr   # create the container once

./polign-server -store az://myaccount/my-ctr/polign

./polign-server -store fs:/var/lib/polign

To run from source instead, use Go 1.25+ (the cloud tag builds in S3/GCS/Azure support, which release binaries already have):

git clone https://github.com/Polign/polign_db.git && cd polign_db
go run ./cmd/server                                          # dev
go run -tags cloud ./cmd/server -store s3://my-bkt/polign    # durable

Advanced installs — optional operational packaging on top of the same server and API. They don't add durability or features.

Advanced: run it as a service

The whole data plane is one process, so a long-lived host needs nothing beyond your init system — Restart=on-failure does the supervising. A minimal systemd unit:

[Unit]
Description=polign_db server
After=network-online.target

[Service]
ExecStart=/usr/local/bin/polign-server -store s3://my-polign-bucket/polign
Environment=AWS_REGION=us-east-1
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target

The server binds 127.0.0.1 by default — reachable only from that host. Serving beyond it is an explicit choice: pass wider listen addresses, put a gateway or network boundary in front, and enable TLS. A listener exposed without TLS gets a loud warning in the server log at startup:

# expose beyond localhost — put a gateway in front and enable TLS
./polign-server -store s3://my-polign-bucket/polign \
  -http :23000 -grpc :23001 \
  -tls-cert cert.pem -tls-key key.pem

Advanced: pin the release archive

The same archives "Install" uses, for when you want to pin a version in your own provisioning (built with GoReleaser). Pre-releases aren't served as "latest" — pin explicitly:

V=v0.1.0-rc.9
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -fsSL "https://github.com/Polign/polign_db/releases/download/$V/polign_db_${OS}_${ARCH}.tar.gz" | tar -xz

Each archive ships static binaries: polign-server, polign (the command-line client), polign-persistor, polign-apikey for minting API keys, and polign-maintain for running index maintenance from cron.

Operating it

Day-2 operations stay small on purpose: point servers at the bucket, let the background maintenance converge. Here's what each task you'll actually face looks like.

Scale out

Start another node with the same -store flag and put it behind your load balancer:

./polign-server -store s3://my-bkt/polign

It boots in seconds from published snapshots plus the recent log tail, serves immediately (queries it hasn't warmed yet are answered from the bucket while the disk cache fills), and reads the fleet's persisted heat map to pre-promote what's already known to be hot. Bucket-backed leases coordinate the fleet: exactly one node persists, exactly one runs maintenance, and it doesn't matter which. Scaling down is draining a node and terminating it — nothing on a node is authoritative.

Replace a dead node

Start a replacement with the same flag. Every acknowledged write reached the bucket's write log before it was applied, so nothing durable is lost. The replacement restores from the latest snapshots, replays only the log tail, refuses to serve if the restore fails (never stale data), and pre-warms from the heat map. If the dead node held a lease, a peer takes it over within one lease TTL.

Let collections grow

Nothing to do. When a collection grows past ~10k records, the maintenance pass (in-process by default, or polign-maintain from cron) republishes its index sized for the corpus and publishes a compressed replica alongside — busy collections serve from codes in RAM at a fraction of the footprint, while full-precision vectors live only in the bucket. No migration, no index-type decision: the PQ code length is chosen automatically for any embedding dimension. The one knob you might touch is per-query: rescore trades recall against latency on that request.

Control the bill

Full-precision vectors live only in object storage, the busiest collections are codes-in-RAM, and everything else serves from the bucket through a local disk cushion — the shape is already the cost-efficient one. The knobs that matter map directly to line items: -hot-max (how many collections stay promoted in RAM), -disk-cache-bytes (local disk budget), and -keep-generations on the persistor/maintenance side (how much bucket history to keep). None of them can lose data; the worst a wrong setting costs is latency.

API reference

Both transports expose the same operations — the user guide walks through each one. Collections are auto-created on first PutVector, inferring their dimension from the vector. Batch puts take up to 5,000 vectors per request, validated as a whole before anything is applied.

Servers running the collections registry (-byo-store) additionally expose collection-management calls (create/get/list/delete/verify) — see the gRPC API guide; on other servers those return "not enabled".

# Put (auto-creates the "docs" collection as 3-dimensional)
curl -X PUT localhost:23000/v1/collections/docs/vectors/a \
  -d '{"values":[1,0,0],"metadata":{"label":"first","text":"a friendly hello"}}'

# Nearest-neighbour search; filters take operators ($in, $gte, $exists, …)
# composed with $and/$or/$not
curl -X POST localhost:23000/v1/collections/docs/query \
  -d '{"values":[0.9,0.1,0],"k":5,"filter":{"label":"first"}}'

Text & hybrid search

The same query endpoint also runs BM25 keyword search and hybrid search. Send text alone for pure BM25, or text plus values to run both legs concurrently and get one fused ranking (Reciprocal Rank Fusion by default, or a weighted linear blend). Each hit carries a score (larger is better). The text index is built by the persistor over a metadata field (default "text", set with -text-field), so text search needs the segment store (-segment-stores):

# pure BM25 keyword search
curl -X POST localhost:23000/v1/collections/docs/query \
  -d '{"text":"friendly hello","k":5}'

# hybrid: vector + text, fused server-side (RRF is the default)
curl -X POST localhost:23000/v1/collections/docs/query \
  -d '{"values":[0.9,0.1,0],"text":"friendly hello","k":5,
       "fusion":{"method":"linear","alpha":0.7}}'

Client libraries

Python — the polign package lives in the repo under sdk/python. The HTTP client is pure standard library — zero dependencies — and a gRPC transport with the identical API is an optional extra. Not on PyPI yet; install from the repo:

# HTTP client — no dependencies
pip install "git+https://github.com/Polign/polign_db.git#subdirectory=sdk/python"

# with the gRPC transport (adds grpcio + protobuf)
pip install "polign[grpc] @ git+https://github.com/Polign/polign_db.git#subdirectory=sdk/python"

The Python SDK guide covers every operation, search options, auth/TLS, and the typed-error table.

Go — github.com/Polign/polign_db/client talks to a running server over gRPC (:23001) and is the supported integration point for services built on top; they import only this package, never the internal ones. The gRPC API guide covers the full client, typed filters, and hybrid search options.

Tuning the index

HNSW is a stack of proximity-graph layers. Three knobs trade recall against speed and memory:

Benchmarks

Server on an 8-vCPU Amazon EC2 instance (Ubuntu, us-west-2); the client's 16 workers on a separate t3.small in the same availability zone — a network hop, so the server's CPU is all its own. dim=128, k=10. Illustrative, not guarantees — run cmd/loadtest on your own hardware.

Transport overhead (10k×128, 16 workers)

Over http/grpc the server seeds its default IVF-over-HNSW backend, so the local rows show both that default (the apples-to-apples transport comparison) and the pure-HNSW in-process ceiling. The local rows run on the server instance itself — in-process by definition.

gRPC beats HTTP/JSON by ~23% on throughput with a tighter tail (p99 2.86 vs 3.90 ms) — protobuf + HTTP/2 vs JSON + HTTP/1.1. One oddity, deliberately left visible: the local IVF p99 (~21 ms) is worse than gRPC's, and it is not a loopback artifact — it persists on an otherwise idle machine. At in-process saturation (~15k QPS) the driving process shares the engine's GC, so stop-the-world pauses land in its own tail; the served paths run the same backend ~30% below that saturation and don't show it.

Engine hot path: allocation-free search (50k×128, all cores)

A matched before → after pair on the same 8-vCPU EC2 instance, isolating the search hot-path optimization — SIMD distance kernels plus a pooled, allocation-free search scratchpad (visit-version visited set). BenchmarkSearchParallel reports per-parallel-op wall time, so aggregate throughput ≈ 1e9 / ns·op.

Stripping per-query allocation out of the engine (1,162 → 1 allocs/op on the raw index) is what lifts concurrent throughput: fewer allocations mean less GC work competing across cores, and the ~5–8× AVX2+FMA distance kernels compound the win. The service search calls straight into the engine, so it inherits both.

Scaling: HNSW vs IVF-PQ as the corpus grows