Kubernetes Agent Sandbox
Agent Sandbox is a Kubernetes project that runs an AI agent as one isolated, stateful pod with a stable name, on gVisor or Kata where the cluster offers them. Sandboxes are disposable, and everything the agent learned goes with the pod unless its memory lives outside. This page puts Recall outside it: the pod proves who it is with the token Kubernetes already gives it, Recall mints a key for that sandbox alone, and a sandbox recreated under the same name picks up its memory and its half-finished work.
Before you start
- A cluster with agent-sandbox v1.0.6 or later installed. The demo below uses kind; nothing in it is specific to one cloud.
- A store for Recall: polign_db through the polign chart (the backend with leases, which the resume tools need), or a Qdrant you already run.
recall0.16.0 or later and therecallHelm chart 0.1.0 or later.
How it works
No secret is written into a sandbox and nothing injects one. The sandbox pod carries a
projected service account token with audience recall, which Kubernetes
mints and rotates itself. The agent posts that token to the Recall server. The server asks
the API server whether the token is real and which pod holds it, then mints a key that
lasts fifteen minutes and that the agent renews.
Everything in the key's identity comes from the cluster, not from the caller:
| Part | Where it comes from |
|---|---|
| Tenant | The pod's Kubernetes namespace, or a label the cluster admin put on the service account. Never a label on a template a tenant can edit. |
| Agent | The pod's name, which the API server reports from the token itself. For pods a warm pool hands out under generated names, a label the claim puts on its pod, inside the tenant's own namespace. |
| Session | The pod's name, so the record says which pod did the work. |
The key reaches one Recall namespace, <tenant>:<agent>, and
can resume one agent, the same one. A request that names another is refused. Memory written
with the key lives in the store, in your bucket when the store is polign_db, and outlives the
sandbox.
Run it
-
Install the Recall server with token exchange on, pointed at your store. With the polign chart in namespace
polign:kubectl create namespace recall kubectl -n recall create secret generic polign-key \ --from-literal=api-key="$(kubectl -n polign get secret polign-key -o jsonpath='{.data.api-key}' | base64 -d)" helm install recall oci://ghcr.io/polign/charts/recall -n recall \ --set backend.url=http://polign.polign.svc:23000 \ --set backend.existingSecret=polign-key \ --set kubernetes.enabled=true \ --set kubernetes.agentLabel=sandbox.users.io/agentThe chart creates the ClusterRole the server needs to review tokens. Tenants are Kubernetes namespaces unless you set
kubernetes.tenantLabel. -
Create the tenant's namespace and a service account for its sandboxes. The admin creates the service account; a tenant who edits a template cannot change what it says.
kubectl create namespace agents kubectl -n agents create serviceaccount runner
-
Give the sandbox the token and nothing else. The controller already turns off the default token mount; this adds one projected token with the right audience:
apiVersion: agents.x-k8s.io/v1beta1 kind: Sandbox metadata: name: demo-1 namespace: agents spec: podTemplate: spec: serviceAccountName: runner containers: - name: agent image: your-agent-image env: - {name: RECALL_URL, value: http://recall.recall.svc:8080} - {name: RECALL_TOKEN_FILE, value: /var/run/secrets/recall/token} volumeMounts: - {name: recall-token, mountPath: /var/run/secrets/recall, readOnly: true} volumes: - name: recall-token projected: sources: - serviceAccountToken: {path: token, audience: recall, expirationSeconds: 3600} -
Inside the sandbox, the agent needs those two variables and nothing else. An MCP host that speaks stdio runs
recall connect; Python usespolign_recall.Client.from_env(). Both exchange the token, renew the key, and exchange again if the server ever refuses it.
The complete demo, with a SandboxTemplate, a warm pool and a claim as well
as the plain Sandbox, and a Makefile that runs it on kind, is in
deploy/agent-sandbox.
Two things to know about templates
Sandboxes made from a SandboxTemplate behave differently from one created
directly, in two ways that matter here.
- They are cut off from the cluster by default. With
networkPolicyManagement: Managed, the controller writes a policy that allows egress to public addresses only and points the pod at public DNS resolvers, so an in-cluster Recall cannot be resolved. Give the template an explicitnetworkPolicythat allows egress to cluster DNS and to therecallService; the controller then gives the pod cluster DNS and keeps everything else closed. The demo template shows the exact rules. - A claim's pod carries the pool's generated name. The identity the
token proves is the pod's, so it would change with every claim. Set a label on the claim's
pod with the agent's name (
additionalPodMetadata.labels; the controller allowlists the label domains a claim may use,sandbox.users.ioby default) and setkubernetes.agentLabelon the chart to that key. The key is then foragents:run-1, andrun-1keeps its memory across claims.
What the first run showed
Measured on a kind cluster on a laptop (Kubernetes v1.35, runc, image already on the node), October 2026. Numbers from a real cluster with gVisor or Kata will differ, above all the cold start, and will be published when we have them.
| What | Result |
|---|---|
Fresh Sandbox applied to pod Ready | 0.5 s |
| Token exchange (API server review plus key mint), from inside the sandbox | p50 2.4 ms over 20 calls |
| recall over HTTP, from inside the sandbox | p50 1.0 ms, p99 2.7 ms over 40 calls |
| Revoked key refused | At once with one server replica; within 30 s across replicas |
| Resident memory, Recall server and polign_db | 17 MB and 14 MB, on a handful of records |
Deleting the sandbox and creating it again under the same name gave a new pod and a new
key for the same namespace. The remembered preference and the agent's working state were
there, and the resume tools picked up from the first pod. A key for demo-1
could not resume run-1.
Limitations
- A deleted sandbox's key stays valid until it expires, fifteen minutes by default,
unless an admin revokes it through
/v1/keys. The projected token dies with the pod, so nothing can mint a new key for a deleted sandbox; setkubernetes.keyTTLlower to shorten the window. A key renews itself without the token today, so a key copied out of a pod keeps renewing until revoked; renewal will require the pod's token in the next recall release. A revoked key is still accepted for the server's auth cache TTL (-auth-cache-ttl, thirty seconds by default) on replicas other than the one that revoked it. - Keys are not rotated while a sandbox runs. The agent exchanges a new one when the old one is refused.
- The resume tools need a store with leases: polign_db today. On Qdrant, remember, recall, forget and history work; resume does not.
- Tenants map to Kubernetes namespaces unless the cluster admin controls service account
labels and sets
kubernetes.tenantLabel. - Every token exchange mints a key.
recall connectand the Python client exchange once and renew; a loop that exchanges on every call leaves keys behind until they expire.
If something goes wrong
| What you see | What to do |
|---|---|
Could not resolve host: recall.recall.svc from a templated sandbox | The managed network policy gave the pod public DNS. Add a networkPolicy to the template that allows cluster DNS and the recall Service, as in the demo. |
Claim stays InvalidMetadata | The label domain is not allowlisted. Use a key under sandbox.users.io, or add your domain to allowed-label-domains in the controller's agent-sandbox-config ConfigMap. |
401 unauthenticated: token audience is not "recall" | The projected token was minted for another audience. Set audience: recall on the serviceAccountToken source. |
401 ... the token is not bound to a pod | A long-lived service account Secret was used instead of a projected token. Project the token into the pod. |
502 upstream on exchange | The server could not reach the API server or lacks the TokenReview ClusterRole. Check kubernetes.rbac.create and the server's logs. |
Resume refused with namespace_mismatch | The sandbox asked to resume an agent other than itself. Use its own name, or the claim's agent label. |
Source & license
The server, the stdio bridge, the Python client and the demo manifests are part of Recall, open source under the Apache License 2.0, in github.com/Polign/recall. The Helm chart is in github.com/Polign/polign. The contract every client is written against is the Memory API, version 1.