Kubernetes Agent Sandbox

Agent Sandbox is a Kubernetes project that runs an AI agent as one isolated, stateful pod with a stable name, on gVisor or Kata where the cluster offers them. Sandboxes are disposable, and everything the agent learned goes with the pod unless its memory lives outside. This page puts Recall outside it: the pod proves who it is with the token Kubernetes already gives it, Recall mints a key for that sandbox alone, and a sandbox recreated under the same name picks up its memory and its half-finished work.

Two agents paint the same picture while their sandboxes are killed 12 times between them. The one restoring a pod snapshot every 2.5 seconds repaints 234 cells and never finishes; the one resuming from Recall picks up at the exact stroke each time and paints every stroke once.
Two agents paint the same picture while their sandboxes are killed and recreated. Left: restore from a pod snapshot taken every 2.5 seconds (simulated), which repaints work and never finishes. Right: resume from Recall at the exact stroke where the last pod stopped.

Before you start

How it works

No secret is written into a sandbox and nothing injects one. The sandbox pod carries a projected service account token with audience recall, which Kubernetes mints and rotates itself. The agent posts that token to the Recall server. The server asks the API server whether the token is real and which pod holds it, then mints a key that lasts fifteen minutes and that the agent renews.

Everything in the key's identity comes from the cluster, not from the caller:

PartWhere it comes from
TenantThe pod's Kubernetes namespace, or a label the cluster admin put on the service account. Never a label on a template a tenant can edit.
AgentThe pod's name, which the API server reports from the token itself. For pods a warm pool hands out under generated names, a label the claim puts on its pod, inside the tenant's own namespace.
SessionThe pod's name, so the record says which pod did the work.

The key reaches one Recall namespace, <tenant>:<agent>, and can resume one agent, the same one. A request that names another is refused. Memory written with the key lives in the store, in your bucket when the store is polign_db, and outlives the sandbox.

Run it

  1. Install the Recall server with token exchange on, pointed at your store. With the polign chart in namespace polign:

    kubectl create namespace recall
    kubectl -n recall create secret generic polign-key \
      --from-literal=api-key="$(kubectl -n polign get secret polign-key -o jsonpath='{.data.api-key}' | base64 -d)"
    helm install recall oci://ghcr.io/polign/charts/recall -n recall \
      --set backend.url=http://polign.polign.svc:23000 \
      --set backend.existingSecret=polign-key \
      --set kubernetes.enabled=true \
      --set kubernetes.agentLabel=sandbox.users.io/agent

    The chart creates the ClusterRole the server needs to review tokens. Tenants are Kubernetes namespaces unless you set kubernetes.tenantLabel.

  2. Create the tenant's namespace and a service account for its sandboxes. The admin creates the service account; a tenant who edits a template cannot change what it says.

    kubectl create namespace agents
    kubectl -n agents create serviceaccount runner
  3. Give the sandbox the token and nothing else. The controller already turns off the default token mount; this adds one projected token with the right audience:

    apiVersion: agents.x-k8s.io/v1beta1
    kind: Sandbox
    metadata:
      name: demo-1
      namespace: agents
    spec:
      podTemplate:
        spec:
          serviceAccountName: runner
          containers:
            - name: agent
              image: your-agent-image
              env:
                - {name: RECALL_URL, value: http://recall.recall.svc:8080}
                - {name: RECALL_TOKEN_FILE, value: /var/run/secrets/recall/token}
              volumeMounts:
                - {name: recall-token, mountPath: /var/run/secrets/recall, readOnly: true}
          volumes:
            - name: recall-token
              projected:
                sources:
                  - serviceAccountToken: {path: token, audience: recall, expirationSeconds: 3600}
  4. Inside the sandbox, the agent needs those two variables and nothing else. An MCP host that speaks stdio runs recall connect; Python uses polign_recall.Client.from_env(). Both exchange the token, renew the key, and exchange again if the server ever refuses it.

The complete demo, with a SandboxTemplate, a warm pool and a claim as well as the plain Sandbox, and a Makefile that runs it on kind, is in deploy/agent-sandbox.

Two things to know about templates

Sandboxes made from a SandboxTemplate behave differently from one created directly, in two ways that matter here.

What the first run showed

Measured on a kind cluster on a laptop (Kubernetes v1.35, runc, image already on the node), October 2026. Numbers from a real cluster with gVisor or Kata will differ, above all the cold start, and will be published when we have them.

WhatResult
Fresh Sandbox applied to pod Ready0.5 s
Token exchange (API server review plus key mint), from inside the sandboxp50 2.4 ms over 20 calls
recall over HTTP, from inside the sandboxp50 1.0 ms, p99 2.7 ms over 40 calls
Revoked key refusedAt once with one server replica; within 30 s across replicas
Resident memory, Recall server and polign_db17 MB and 14 MB, on a handful of records

Deleting the sandbox and creating it again under the same name gave a new pod and a new key for the same namespace. The remembered preference and the agent's working state were there, and the resume tools picked up from the first pod. A key for demo-1 could not resume run-1.

Limitations

If something goes wrong

What you seeWhat to do
Could not resolve host: recall.recall.svc from a templated sandboxThe managed network policy gave the pod public DNS. Add a networkPolicy to the template that allows cluster DNS and the recall Service, as in the demo.
Claim stays InvalidMetadataThe label domain is not allowlisted. Use a key under sandbox.users.io, or add your domain to allowed-label-domains in the controller's agent-sandbox-config ConfigMap.
401 unauthenticated: token audience is not "recall"The projected token was minted for another audience. Set audience: recall on the serviceAccountToken source.
401 ... the token is not bound to a podA long-lived service account Secret was used instead of a projected token. Project the token into the pod.
502 upstream on exchangeThe server could not reach the API server or lacks the TokenReview ClusterRole. Check kubernetes.rbac.create and the server's logs.
Resume refused with namespace_mismatchThe sandbox asked to resume an agent other than itself. Use its own name, or the claim's agent label.

Source & license

The server, the stdio bridge, the Python client and the demo manifests are part of Recall, open source under the Apache License 2.0, in github.com/Polign/recall. The Helm chart is in github.com/Polign/polign. The contract every client is written against is the Memory API, version 1.