Operate in production

Everything past the laptop: run the server as a service, scale the fleet, control the bill. When one bucket stops being enough, bind each collection to its own bucket with verified wiring and API keys. This page assumes you've been through Get started.

Run it as a service

The whole data plane is one process, so a long-lived host needs nothing beyond your init system. Restart=on-failure does the supervising. A minimal systemd unit:

[Unit]
Description=polign_db server
After=network-online.target

[Service]
ExecStart=/usr/local/bin/polign-server -store s3://my-polign-bucket/polign
Environment=AWS_REGION=us-east-1
Restart=on-failure
RestartSec=2

[Install]
WantedBy=multi-user.target

The server binds 127.0.0.1 by default, so it's reachable only from that host. Serving beyond it is an explicit choice: pass wider listen addresses, put a gateway or network boundary in front, and enable TLS. A listener exposed without TLS gets a loud warning in the server log at startup:

# expose beyond localhost — put a gateway in front and enable TLS
./polign-server -store s3://my-polign-bucket/polign \
  -http :23000 -grpc :23001 \
  -tls-cert cert.pem -tls-key key.pem

Pin the release archive

The same archives Install uses, for when you want to pin a version in your own provisioning (built with GoReleaser). Resolve the current stable tag at provision time, or hard-code one; pre-releases aren't served as "latest", so reaching a pre-release always means naming its tag:

V=$(curl -fsSL https://dl.polign.com/latest/version)   # or pin: V=vX.Y.Z
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -fsSL "https://dl.polign.com/$V/polign_db_${OS}_${ARCH}.tar.gz" | tar -xz

dl.polign.com redirects to the GitHub release assets, and https://dl.polign.com/latest/version returns the latest stable tag if you'd rather resolve it at provision time.

Each archive ships static binaries: polign-server, polign (the command-line client), polign-persistor, polign-apikey for minting API keys, polign-maintain for running index maintenance from cron, and polign-import for bulk-loading vectors from JSONL or big-ann files.

Operating the fleet

Day-2 operations stay small on purpose: point servers at the bucket, let the background maintenance converge. Here's what each task you'll actually face looks like.

Scale out

Start another node with the same -store flag and put it behind your load balancer:

./polign-server -store s3://my-bkt/polign

It boots in seconds from published snapshots plus the recent log tail, serves immediately (queries it hasn't warmed yet are answered from the bucket while the disk cache fills), and reads the fleet's persisted heat map to pre-promote what's already known to be hot. Bucket-backed leases coordinate the fleet: exactly one node persists, exactly one runs maintenance, and it doesn't matter which. Scaling down is draining a node and terminating it, because nothing on a node is authoritative.

Replace a dead node

Start a replacement with the same flag. Every acknowledged write reached the bucket's write log before it was applied, so nothing durable is lost. The replacement restores from the latest snapshots, replays only the log tail, refuses to serve if the restore fails (never stale data), and pre-warms from the heat map. If the dead node held a lease, a peer takes it over within one lease TTL.

Let collections grow

Nothing to do. When a collection grows past ~10k records, the maintenance pass (in-process by default, or polign-maintain from cron) republishes its index sized for the corpus and publishes a compressed replica alongside. Busy collections serve from codes in RAM at a fraction of the footprint, while full-precision vectors live only in the bucket. No migration, no index-type decision: the PQ code length is chosen automatically for any embedding dimension. The one knob you might touch is per-query: rescore trades recall against latency on that request.

Control the bill

Full-precision vectors live only in object storage, the busiest collections are codes-in-RAM, and everything else serves from the bucket through a local disk cushion, so the shape is already the cost-efficient one. The knobs that matter map directly to line items: -hot-max (how many collections stay promoted in RAM), -disk-cache-bytes (local disk budget), and -keep-generations on the persistor/maintenance side (how much bucket history to keep). None of them can lose data; the worst a wrong setting costs is latency.

The write-side pair to watch is the freshness overlay, which buffers unpersisted writes and grows only when the persistor lags: -overlay-warn-buffered sets the depth at which the server starts warning, and -overlay-max-buffered is the hard cap at which writes get 503 + Retry-After until the persistor catches up. That trade is deliberate: backpressure instead of unbounded writer memory.

Tuning the index

HNSW is a stack of proximity-graph layers. Three parameters trade recall against speed and memory; these are the shipped defaults, and there is deliberately no server flag to change the build-time pair — the maintenance pass sizes indexes for you:

The dials you actually reach for are per-query — ef, nprobe, and rescore on the search request — plus polign-maintain's -pqm and -min-corpus on the maintenance side.

Per-collection buckets

Out of the box, everything lives in the one bucket you gave -store, which is the right default. The rest of this page is for when one bucket stops being enough: the analytics corpus should carry its own storage bill, the EU collection has to stay in an EU bucket, prod and staging shouldn't share anything. You can bind each collection to its own bucket, and because pointing a database at the wrong bucket is the kind of mistake you want caught immediately, doing so is an explicit, verified step. The whole flow is one small pattern, create, claim, use, repeated per collection.

Step 1 — Start the server

Same binary and -store flag as the install section, plus one more:

./polign-server -store s3://myco-control/polign -byo-store

Your -store bucket becomes the control store. Collections you don't say otherwise about keep living here. -byo-store adds the collection API: the endpoints for binding a collection to a bucket of its own. Off (the default), those endpoints answer "not implemented" (HTTP 501) and the server behaves exactly as it always has.

Step 2 — Mint your keys

The collection API takes an API key, because its routes point your server at buckets, which anonymous callers shouldn't drive. Reading and writing vectors needs no key (step 5). Keys live in your control store under .auth/, so there's no separate auth database. Two mints and you're set up. First, once, with bucket credentials, the admin key, which is the credential that manages other keys:

polign-apikey -stores s3://myco-control/polign admin create -note "ops"

admin key id:  9c01f7aa20b643d1
plgn_9c01f7aa20b643d1_77e01b39c2aa41d6…

Copy the key. This is the only time it's shown; only its hash is stored. From here on, key management happens over HTTPS with no bucket credentials involved:

export POLIGN_ADMIN_KEY=plgn_9c01f7aa20b643d1_…

# the key your tooling will use for the collection API
polign-apikey -api https://db.example.com:23000 create -note "infra scripts"

key id:  24d2512e6a993fe0
plgn_24d2512e6a993fe0_b2b5eaa30f2c2cd3…

The two kinds of key never cross: an admin key manages keys and nothing else; a regular key opens the collection API and can't mint keys. Export it and the polign CLI picks it up from there:

export POLIGN_API_KEY=plgn_24d2512e6a993fe0_…
export POLIGN_URL=https://db.example.com:23000

Step 3 — A collection with its own bucket

The prod document corpus should live in its own bucket, with its own lifecycle policy and its own line on the bill. Register the collection against it:

polign collections create prod-docs -bucket s3://myco-prod-vectors/polign

prod-docs: pending
claim token (shown once): plgnclaim_8d1f0c2ab34e…
finish with:
  polign collections claim prod-docs plgnclaim_8d1f0c2ab34e…

One step to finish: prove the URI you typed really is a bucket you control. The claim command writes the token into the bucket with your credentials, not the server's, and that's the proof. Then it asks the server to verify:

polign collections claim prod-docs plgnclaim_8d1f0c2ab34e…

prod-docs: active

Between those two lines the server probed the bucket with put, get, list, conditional-put, and delete, so a missing permission fails now, naming the step, instead of in production later. (No CLI on the machine that owns the bucket? The claim is one object write: aws s3 cp - s3://myco-prod-vectors/polign/.polign/claim <<< "plgnclaim_…", and the server verifies on its own within ~30 seconds.)

Step 4 — A second collection, second bucket

The analytics corpus gets the same treatment. Nothing new, same two commands, different bucket:

polign collections create analytics-events -bucket s3://myco-analytics/polign
polign collections claim analytics-events plgnclaim_51c9e0d47f21…

analytics-events: active

That's the whole pattern, however many collections you add: create, claim, active. Each collection binds to exactly one bucket, so prod-docs and analytics-events never share storage, and your control store carries only the bookkeeping. (Bucket in a different AWS account? Same flow with an IAM role and no claim step. See cross-account buckets.)

Step 5 — Use the collections

From here, both collections behave exactly like the ones in Get started: same endpoints, same search, same filters, and no API key. The data plane is open; keys guard only the collection API.

# lands in s3://myco-prod-vectors/polign
polign put prod-docs guide-1 -values @embedding.json \
  -meta title="Getting started" -meta text="How to get started with…"

polign search prod-docs -values @query.json -k 5

The same goes for curl, Python, and Go. Every client works unchanged (the CLI reference has the full flag list for these commands), with no api_key for day-to-day work. analytics-events is identical, with the same API and a different bucket underneath. Everything follows a collection into its bucket, the write log and every index file, so each collection's storage bills to its own bucket.

Step 6 — Day-2 operations

A bucket stops answering. Say a security sweep tightens the analytics bucket's policy and the server loses access. The next read or write fails once, the server records why, and the collection flips to degraded, after which requests fail fast with the reason instead of timing out against a dead bucket:

polign collections list

NAME              STATUS    BUCKET
analytics-events  degraded  s3://myco-analytics/polign     # reads and writes fail with 503
prod-docs         active    s3://myco-prod-vectors/polign

You don't babysit it: the same background pass that activates pending collections re-probes degraded ones and reactivates them the moment access is restored.

Rotate or revoke a key.

# mint the replacement first, then disable the old one
polign-apikey -api https://db.example.com:23000 create -note "infra scripts v2"
polign-apikey -api https://db.example.com:23000 disable -id 24d2512e6a993fe0

Revocation is a record change, not a restart. Running servers notice within the auth cache TTL (-auth-cache-ttl, 5 minutes by default). Shorten it for faster revocation, lengthen it for fewer bucket reads. enable undoes a disable; delete removes the record for good.

Retire a collection.

polign collections delete analytics-events

analytics-events unregistered (the bucket's contents are untouched)

This unregisters the collection and drops the server's local index for it. It does not delete the bucket's contents. The objects stay exactly where they are, under your retention rules, until you decide otherwise. Unregistering a database from a bucket and destroying the data are different decisions, and the API only makes the first one.

Cross-account buckets

When a bucket lives in an AWS account the server's credentials don't reach, the server assumes an IAM role that account grants. Writing cross-account IAM by hand is where the typos live, so ask the server for the setup instead:

polign setup -bucket s3://myco-eu/polign

{"external_id": "plgn-ext-4f8a…", "trust_policy": {…}, "permissions_policy": {…},
 "terraform": "…", "cli": ["aws iam create-role …", …]}

Apply the recipe in the other account, as pasted policies, Terraform, or the CLI lines, and create the collection with the resulting role:

polign collections create eu-docs \
  -bucket s3://myco-eu/polign \
  -role arn:aws:iam::999888777666:role/polign-access

eu-docs: active

active in one call, no claim step: the trust policy only admits assumptions carrying the server's stable external_id, so a successful assumption already proves the account's owner authorized this server. The same five-capability probe runs before activation. Start the server with -fleet-principal arn:aws:iam::…:role/polign-fleet so generated recipes print your real principal instead of a placeholder.

Quick reference

The collection API

All of these need an API key. gRPC names in parentheses.

When it refuses

The SDKs surface these as typed errors, so code branches on meaning, not status numbers: Go's ErrPending, ErrNotEnabled, ErrPermissionDenied, ErrUnavailable; Python's ConflictError, NotEnabledError, PermissionDeniedError, UnavailableError.

Scaling this up

Per-collection buckets isolate storage (where the bytes live, whose bill they're on, which region they stay in), not serving: every collection on a server shares its memory, maintenance passes, and blast radius, and any valid key opens the whole collection API. When you need serving isolation too, run separate deployments. That's cheap here, because servers read from the bucket on demand and an idle instance costs close to nothing. Scaling one deployment is more copies of step 1 pointed at the same control store: key and collection records are bucket records, so every node sees the same state with nothing to sync.

Where to go next

• Start with Get started if you landed here first: install, add data, and search.
• The HTTP API, Python SDK, and gRPC API cover every operation from the clients, with error tables.