Agent Substrate

Agent Substrate runs each agent as an actor on a pool of warm Kubernetes workers, inside a gVisor or microVM sandbox. When an actor goes idle, Substrate snapshots its process memory and files and frees the worker; the next request restores it. A snapshot belongs to one actor and ends when the actor is deleted. This page puts Recall beside it: what the agent learned and where it was in its work live outside the actor, the actor holds no key for them, and an actor created again under the same name gets them back.

Before you start

• A cluster with Agent Substrate v1.0.0 or later installed. The example below uses the kind cluster from Substrate's quickstart.
• recall 0.25.0 or later, which adds -substrate-issuer.
• A store for Recall. The example keeps memory in a local database inside the Recall pod; a real install names a backend you run.

How it works

An actor on Substrate has no credentials of its own. Every outbound request goes through Substrate's egress gateway, and an actor's egress policy can tell the gateway to replace a request header with a short-lived token that Substrate mints for that actor. Recall accepts that token directly:

1. The Recall server is started with -substrate-issuer, the URL of the install's token issuer. It trusts that issuer and no other.
2. The actor calls Recall with any placeholder in its Authorization header.
3. The gateway discards the placeholder and sends the actor's token in its place.
4. Recall checks the token's signature against the issuer's published keys, its audience, and its expiry, and reads which actor it names.

Everything in the caller's identity comes from Substrate, not from the actor:

There is no exchange for a Recall key. A key would sit in the actor's memory and be written into its snapshots; a token the gateway adds on the way out never enters the actor. The token names the actor and not one incarnation of it, so deleting an actor and creating it again under the same name reaches the same memory.

Run it

1. Start the Recall server with the issuer, and give it the CA that signs Substrate's in-cluster certificates so it can fetch the issuer's keys:

   containers:
   - name: recall
     image: ghcr.io/polign/recall:0.25.0
     args:
     - serve
     - -substrate-issuer=https://idp.ate-system.svc
     - -substrate-ca-file=/run/servicedns-ca/trust-bundle.pem
     volumeMounts:
     - {name: servicedns-ca, mountPath: /run/servicedns-ca, readOnly: true}
   volumes:
   - name: servicedns-ca
     projected:
       sources:
       - clusterTrustBundle:
           signerName: servicedns.podcert.ate.dev/identity
           labelSelector:
             matchLabels: {podcert.ate.dev/canarying: live}
           path: trust-bundle.pem

2. Give the actor Recall's address and nothing else. In the ActorTemplate:

   containers:
   - name: agent
     image: your-agent-image@sha256:...
     env:
     - name: RECALL_URL
       value: http://recall.recall-system.svc.cluster.local
     - name: RECALL_API_KEY
       value: placeholder

   The Python client and the integrations built on it send RECALL_API_KEY as the bearer, so the placeholder is all they need.

3. Create the actor's egress policy. It allows Recall and has the gateway add the actor's token:

   rules:
   - http:
       hostnames: ["recall.recall-system.svc.cluster.local"]
       effects:
         replaceHeaders:
         - header: Authorization
           prefix: "Bearer "
           actorJwt:
             audiences: ["recall"]
             expirationSeconds: 900

   kubectl ate create egress-policy my-agent -a my-atespace -f egress-policy.yaml

The complete example, with the server manifest, a worker pool, a demo actor, and a script that runs it on kind and checks the results, is in deploy/agent-substrate.

Hermes Agent on Substrate

The Recall memory provider for Hermes Agent runs inside an actor without changes. It reads RECALL_URL and RECALL_API_KEY from the actor's environment, as in step 2, and the egress policy from step 3 does the rest. An example that drives the provider through a completed turn, the start of the next turn, a tool call, a suspend, and a deleted actor is in plugins/hermes/substrate.

Three things to know about actors

• Name the Service in full. An actor's DNS answers only the names its egress policy lists, and it matches the name as it is looked up, which for a Service is the fully qualified one. Use recall.recall-system.svc.cluster.local in the policy; a policy that lists recall.recall-system.svc gets no such host.
• Each actor has its own egress policy. A policy belongs to one actor and is not inherited from the template. An actor with no policy reaches nothing, Recall included, so create the policy when you create the actor.
• Connections do not survive a suspend. Substrate does not keep open network connections in a snapshot. The Python client opens a connection per call, so it works after a resume without being reopened.

What the first run showed

Run on a kind cluster on a laptop with Substrate v1.0.0 and gVisor workers, November 2026. The timing is from that laptop, with the snapshot on the same node, and will differ on a real cluster.

Limitations

• A token stays valid until it expires. Substrate does not revoke actor tokens, so a deleted actor's last token works for the rest of its lifetime: fifteen minutes with the policy above, five at the shortest Substrate allows.
• Whoever may write an atespace's egress policies chooses where its actors' tokens are sent. Recall accepts only tokens minted for its own audience, recall by default, so a token minted for another service is refused.
• An actor created again under a deleted actor's name finds its agent still resumed for the old actor. agent_resume answers 409 already_resumed; the new actor calls agent_release and resumes, and gets the working state back.
• Memory is per actor name. Two actors that should share what they learn need -substrate-scope atespace, which applies to the whole server.
• The recall Helm chart has no settings for Substrate yet. Use the manifest in the example.

If something goes wrong

Source & license

The server and the example are part of Recall, open source under the Apache License 2.0, in github.com/Polign/recall. The contract for an actor's identity is in the State API, version 1.