Advanced user guide

One workflow, start to finish: give each collection its own bucket — per project, per region, per cost center — with the wiring verified the moment you create it.

Out of the box, everything lives in the one bucket you gave -store — the right default. This page is for when one bucket stops being enough: the analytics corpus should carry its own storage bill, the EU collection has to stay in an EU bucket, prod and staging shouldn't share anything. You can bind each collection to its own bucket, and because pointing a database at the wrong bucket is the kind of mistake you want caught immediately, doing so is an explicit, verified step. The whole flow is one small pattern — create, claim, use — repeated per collection.

Step 1 — Start the server

Same binary and -store flag as the install guide, plus one more:

./polign-server -store s3://myco-control/polign -byo-store

Your -store bucket becomes the control store — collections you don't say otherwise about keep living here. -byo-store adds the collection API: the endpoints for binding a collection to a bucket of its own. Off (the default), those endpoints answer "not implemented" (HTTP 501) and the server behaves exactly as it always has.

Step 2 — Mint your keys

The collection API takes an API key — its routes point your server at buckets, which anonymous callers shouldn't drive. Reading and writing vectors needs no key (step 5). Keys live in your control store under .auth/ — no separate auth database. Two mints and you're set up. First, once, with bucket credentials, the admin key — the credential that manages other keys:

polign-apikey -stores s3://myco-control/polign admin create -note "ops"

admin key id:  9c01f7aa20b643d1
plgn_9c01f7aa20b643d1_77e01b39c2aa41d6…

Copy the key — this is the only time it's shown; only its hash is stored. From here on, key management happens over HTTPS with no bucket credentials involved:

export POLIGN_ADMIN_KEY=plgn_9c01f7aa20b643d1_…

# the key your tooling will use for the collection API
polign-apikey -api https://db.example.com:23000 create -note "infra scripts"

key id:  24d2512e6a993fe0
plgn_24d2512e6a993fe0_b2b5eaa30f2c2cd3…

The two kinds of key never cross: an admin key manages keys and nothing else; a regular key opens the collection API and can't mint keys. Export it and the polign CLI picks it up from there:

export POLIGN_API_KEY=plgn_24d2512e6a993fe0_…
export POLIGN_URL=https://db.example.com:23000

Step 3 — A collection with its own bucket

The prod document corpus should live in its own bucket — its own lifecycle policy, its own line on the bill. Register the collection against it:

polign collections create prod-docs -bucket s3://myco-prod-vectors/polign

prod-docs: pending
claim token (shown once): plgnclaim_8d1f0c2ab34e…
finish with:
  polign collections claim prod-docs plgnclaim_8d1f0c2ab34e…

One step to finish: prove the URI you typed really is a bucket you control. The claim command writes the token into the bucket with your credentials, not the server's — that's the proof — then asks the server to verify:

polign collections claim prod-docs plgnclaim_8d1f0c2ab34e…

prod-docs: active

Between those two lines the server probed the bucket — put, get, list, conditional-put, delete — so a missing permission fails now, naming the step, instead of in production later. (No CLI on the machine that owns the bucket? The claim is one object write: aws s3 cp - s3://myco-prod-vectors/polign/.polign/claim <<< "plgnclaim_…", and the server verifies on its own within ~30 seconds.)

Step 4 — A second collection, second bucket

The analytics corpus gets the same treatment. Nothing new — same two commands, different bucket:

polign collections create analytics-events -bucket s3://myco-analytics/polign
polign collections claim analytics-events plgnclaim_51c9e0d47f21…

analytics-events: active

That's the whole pattern, however many collections you add: create, claim, active. Each collection binds to exactly one bucket, so prod-docs and analytics-events never share storage — and your control store carries only the bookkeeping. (Bucket in a different AWS account? Same flow with an IAM role and no claim step — see cross-account buckets.)

Step 5 — Use the collections

From here, both collections behave exactly like the ones in the user guide — same endpoints, same search, same filters, and no API key: the data plane is open; keys guard only the collection API.

# lands in s3://myco-prod-vectors/polign
polign put prod-docs guide-1 -values @embedding.json \
  -meta title="Getting started" -meta text="How to get started with…"

polign search prod-docs -values @query.json -k 5

The same goes for curl, Python, and Go — every client works unchanged (the CLI reference has the full flag list for these commands), with no api_key for day-to-day work. analytics-events is identical — same API, different bucket underneath. Everything follows a collection into its bucket — the write log and every index file — so each collection's storage bills to its own bucket.

Step 6 — Day-2 operations

A bucket stops answering

Say a security sweep tightens the analytics bucket's policy and the server loses access. The next read or write fails once, the server records why, and the collection flips to degraded — after which requests fail fast with the reason instead of timing out against a dead bucket:

polign collections list

NAME              STATUS    BUCKET
analytics-events  degraded  s3://myco-analytics/polign     # reads and writes fail with 503
prod-docs         active    s3://myco-prod-vectors/polign

You don't babysit it: the same background pass that activates pending collections re-probes degraded ones and reactivates them the moment access is restored.

Rotate or revoke a key

# mint the replacement first, then disable the old one
polign-apikey -api https://db.example.com:23000 create -note "infra scripts v2"
polign-apikey -api https://db.example.com:23000 disable -id 24d2512e6a993fe0

Revocation is a record change, not a restart. Running servers notice within the auth cache TTL (-auth-cache-ttl, 5 minutes by default — shorten it for faster revocation, lengthen it for fewer bucket reads). enable undoes a disable; delete removes the record for good.

Retire a collection

polign collections delete analytics-events

analytics-events unregistered (the bucket's contents are untouched)

This unregisters the collection and drops the server's local index for it. It does not delete the bucket's contents — the objects stay exactly where they are, under your retention rules, until you decide otherwise. Unregistering a database from a bucket and destroying the data are different decisions, and the API only makes the first one.

Cross-account buckets

When a bucket lives in an AWS account the server's credentials don't reach, the server assumes an IAM role that account grants. Writing cross-account IAM by hand is where the typos live, so ask the server for the setup instead:

polign setup -bucket s3://myco-eu/polign

{"external_id": "plgn-ext-4f8a…", "trust_policy": {…}, "permissions_policy": {…},
 "terraform": "…", "cli": ["aws iam create-role …", …]}

Apply the recipe in the other account — as pasted policies, Terraform, or the CLI lines — and create the collection with the resulting role:

polign collections create eu-docs \
  -bucket s3://myco-eu/polign \
  -role arn:aws:iam::999888777666:role/polign-access

eu-docs: active

active in one call, no claim step: the trust policy only admits assumptions carrying the server's stable external_id, so a successful assumption already proves the account's owner authorized this server. The same five-capability probe runs before activation. Start the server with -fleet-principal arn:aws:iam::…:role/polign-fleet so generated recipes print your real principal instead of a placeholder.

Quick reference

The collection API

All of these need an API key. gRPC names in parentheses.

When it refuses

The SDKs surface these as typed errors — Go's ErrPending, ErrNotEnabled, ErrPermissionDenied, ErrUnavailable; Python's ConflictError, NotEnabledError, PermissionDeniedError, UnavailableError — so code branches on meaning, not status numbers.

Scaling this up

Per-collection buckets isolate storage — where the bytes live, whose bill they're on, which region they stay in — not serving: every collection on a server shares its memory, maintenance passes, and blast radius, and any valid key opens the whole collection API. When you need serving isolation too, run separate deployments — cheap here, because servers read from the bucket on demand and an idle instance costs close to nothing. Scaling one deployment is more copies of step 1 pointed at the same control store: key and collection records are bucket records, so every node sees the same state with nothing to sync.

Where to go next

• Install & deploy — production installs, fleet mechanics, the API reference.
• Python SDK and gRPC API — every operation from the clients, with error tables.
• Architecture — how a collection reaches down through the write log, the segment layout, and the bucket.