Memory that outlives the sandbox

The problem

Agent Sandbox, a Kubernetes project, runs an AI agent as one isolated pod with a stable name, on gVisor or Kata where the cluster has them. It is the right shape for code that an agent wrote a moment ago: the pod is walled off, it can be paused, and when the task is over it is thrown away.

Thrown away is the problem. Everything the agent learned about the user, every correction it took, every decision it made about the task, goes with the pod. The next sandbox starts from nothing. And the usual fix, handing each sandbox a credential to some memory service, means a secret inside the box, which is exactly what a sandbox exists to avoid, plus a controller to write one secret per sandbox and clean it up.

Introducing Recall for Agent Sandbox

Recall now runs as a server that many agents share, each inside its own namespace, and it understands Kubernetes workload identity. A sandbox pod carries one thing: the service account token Kubernetes already mints and rotates for it, projected with the audience recall. The agent hands that token to the Recall server. The server asks the API server whether the token is real and which pod holds it, and mints a fifteen minute key for <tenant>:<that pod>. The agent renews the key while it runs.

Nothing in the key's identity comes from the caller. The tenant is the pod's namespace, or a label the cluster admin put on the service account. The agent is the pod's name from the token, or a label the claim sets inside that namespace. A request that names any other namespace or agent is refused. The memory itself lives in the store behind the server, in your own bucket when that store is polign_db, and it is still there when the sandbox is gone.

No secret is written per sandbox. No webhook injects anything. Nothing a tenant controls decides which tenant a sandbox belongs to. Within its own namespace a tenant can name its agents however it likes, and that is fine, because a tenant can only ever reach its own memory.

How it works

1. The pod gets a token. The Sandbox pod spec names a service account and mounts one projected token with audience: recall. The controller already turns off the default token mount, so this is the only credential in the pod.
2. The agent exchanges it. recall connect (MCP over stdio for the agent, HTTP to the server) or the Python client's Client.from_env() reads RECALL_URL and RECALL_TOKEN_FILE, posts the token, and keeps the key renewed. If the server ever refuses the key, it exchanges again.
3. The server verifies and mints. One call to the API server's TokenReview with the same audience. The answer carries the service account and the pod name. The key's namespace, agent and session are built from those and nothing else. The server needs one piece of RBAC for this, a ClusterRole that may create TokenReviews; the Helm chart creates it, and the integration page lists the rest.
4. Memory outlives the pod. Records go to the store under the key's namespace. A sandbox recreated with the same name gets a new key for the same namespace, finds its memory, and agent_resume picks up its working state.

Two details came out of running this against agent-sandbox v1.0.6 and shaped the manifests. Sandboxes made from a template are cut off from the cluster by default: the managed network policy allows public egress only and gives the pod public DNS, so the template needs an explicit policy that opens cluster DNS and the Recall service. And a sandbox handed out by a warm pool carries the pool's generated name, so a claim labels its pod with the agent's name and the server reads that label. Both are in the integration page.

What we measured

On a kind cluster on a laptop, Kubernetes v1.35, runc, with the image already on the node. A cluster with gVisor or Kata will show a slower cold start; we will publish those numbers when we have them.

The memory row is the one to read twice. Those two processes serve every sandbox on the cluster. They grow with the records, not with the number of sandboxes reading them, which matters when the whole point of Agent Sandbox is how many agents fit on a node. The numbers are from the demo's handful of records.

The part that matters is not a latency number, but here is one anyway: delete the sandbox, apply it again under the same name, and the new pod is Ready in the same half second as a fresh one and reasoning with its old state on its first call. The new pod got a new key for the same namespace. The remembered preference came back with its current value, and the value it had superseded was still readable as history. The resume tools returned the working state the first pod had written. A key minted for one sandbox could not resume another sandbox's run.

Limitations

• A deleted sandbox's key lives until it expires, fifteen minutes by default, unless an admin revokes it. The projected token dies with the pod, so nothing can mint a new key for a deleted sandbox and the pod itself can no longer renew. The exposure is the key TTL, which the admin sets with kubernetes.keyTTL in the chart. One gap today: a key renews itself without the token, so a key copied out of a pod before deletion can keep renewing until an admin revokes it. Renewal will require the pod's token in the next recall release, after which a copied key dies at its TTL.
• Revocation takes up to thirty seconds to reach every replica. Each server caches key checks for its auth cache TTL, -auth-cache-ttl, thirty seconds by default. The replica that revoked the key refuses it at once; the others refuse it when their cache entry expires. Lower the TTL to shorten the lag at the cost of more key reads.
• No rotation in flight. The agent exchanges a new key when the old one is refused.
• Resume needs leases. polign_db has them; Qdrant does not. On Qdrant the memory tools work and the resume tools do not.
• Tenants are Kubernetes namespaces unless the cluster admin controls service account labels.
• Measured on kind, not on a sandbox runtime. The cold start above is runc with a cached image.

Availability

• recall 0.16.0: recall serve, recall connect, recall key, and the Kubernetes token exchange. Apache-2.0 in github.com/Polign/recall, with the demo manifests under deploy/agent-sandbox.
• polign-recall 0.16.0 on PyPI: Client(url=..., token_file=...) and Client.from_env().
• recall Helm chart 0.1.0 in github.com/Polign/polign.
• The contract behind all of it is the Memory API, version 1.