Durable agent actions
A resumed agent now knows which of its tool calls a crash left unfinished, and settles each one before it takes another step, so it never sends the same email twice.
The problem
Resume brings a killed agent back from its own records: its goal, plan, progress and last turns. Those records say what the agent was doing. They cannot say whether its last tool call happened.
Kill an agent while send_email is running and the email may or may not have gone out.
The next process sees its own call with no result, and the natural thing for it to do is call it
again. For a search that costs a few tokens. For an email, a payment or a ticket, the customer gets
two. A snapshot does not help either: restoring the process brings back the moment before the send,
and the agent sends it again.
This matters most for:
- Support and operations agents that send messages, open tickets or issue refunds.
- Coding agents that push, merge or deploy.
- Any agent on spot capacity, autoscaled pods or disposable sandboxes, where being killed mid-call is routine.
In short
- Each tool call is recorded before it runs and closed with its result when it returns.
- Each tool has a class: pure, idempotent or external, which decides what happens to a call a crash interrupted.
- On resume, unfinished calls are settled before the model takes another step, and the outcome goes into the agent's history.
- The database refuses a write from a process that lost its lease, on polign_db, PostgreSQL and MongoDB.
- In 540 kill trials, durable tools never repeated or lost a side effect.
Introducing durable actions
Durable actions add one record per tool call to the agent's Recall records. The harness writes it before the call runs and closes it when the call returns. A crash between the two leaves it open, and an open call is the one thing a resumed agent must settle first.
| Resume | Resume with durable actions | |
|---|---|---|
| Killed after the email went out | Sees its call with no result and sends again | Sees the call is unfinished and checks first |
| Killed in the middle of a safe write | May write it twice | Writes again with the original key, so the store keeps one |
| Killed during a read | Reads again | Reads again |
| A frozen process wakes after a takeover | Refused once its client sees the lease is gone | Refused by the database on its next write |
How it works
Record before, close after
begin_action writes the tool, its arguments, its class and a key, then the tool runs.
end_action closes the record with the result, or with the error if the tool reported
one. The key is fixed when the action begins, so every retry of that call presents the same one.
A killed process or a cancelled call leaves the action open, because nothing says whether it took
effect.
Three classes
| Class | Use it for | If a crash interrupted it |
|---|---|---|
pure |
Lookups, searches, listings | Closed. The agent runs it again if it still needs the answer. |
idempotent |
A PUT by id, a push of a known commit, a payment with an idempotency key | Run again with its original key before the agent's next step. |
external |
Send, post, charge, delete | Checked against the outside system by a resolver you write. If nothing can tell, it stays at the top of the briefing as a call to check before repeating. |
Every durable tool needs a class. A tool without one is refused when the agent is built, not given a default, because guessing is exactly how a resumed agent sends the email twice.
Settling before the next step
On resume, Recall closes interrupted reads, the LangGraph package re-runs interrupted safe writes with their keys, and the resolver checks the rest. Each settled outcome is written into the agent's turns, so the model reads "sent, found in the sent folder" in its history instead of a call with no result. Only then does the model get its first briefing. Recall never runs a tool on its own: the re-run is the harness calling your tool, with the key.
Writes the database refuses
Resume already gives one process at a time a lease on the agent, an epoch that only increases. A process that freezes past its lease while another takes over used to be stopped by its own check of the lease, which it makes before writing. Now the backend checks too. polign_db applies an agent write only while the lease is still the writer's. PostgreSQL reads the lease row in the write's own transaction. MongoDB commits the write together with a write to the lease document, so a takeover and a late write conflict and only one of them commits.
Configuring it
Step 1: install
pip install "recall-langgraph>=0.2.0"
Step 2: classify your tools
from recall_langgraph import ActionEnd, RecallResume, recall_tools
def sent_folder(action):
if MAILBOX.has_sent_since(action.started_at, subject_in=action.args):
return ActionEnd("done", resolution="found in the sent folder")
return None # cannot tell
resume = RecallResume("AGENT_ID", resolver=sent_folder)
tools = resume.durable(
[lookup_order, save_invoice, send_email],
classes={"lookup_order": "pure", "save_invoice": "idempotent", "send_email": "external"},
)
agent = create_react_agent(MODEL, [*tools, *recall_tools(resume)],
pre_model_hook=resume.pre_model_hook,
post_model_hook=resume.post_model_hook)
Step 3: pass the key to safe writes
@tool
def save_invoice(invoice: str, total: int,
idempotency_key: Annotated[str, InjectedToolArg] = "") -> str:
"""Save an invoice."""
return BILLING.put(invoice, total, idempotency_key=idempotency_key)
The model never sees idempotency_key; the wrapper fills it with the action's key.
Step 4: keep the records where you want them
Records go to polign_db by default. Set RECALL_BACKEND to keep them in a database you
already run:
| Backend | Variables |
|---|---|
| polign_db | POLIGN_URL, POLIGN_API_KEY |
| PostgreSQL + pgvector | RECALL_BACKEND=pgvector, DATABASE_URL |
| MongoDB (new) | RECALL_BACKEND=mongodb, MONGODB_URI |
MongoDB is new in this release, on Atlas or self-hosted, with every Recall capability and exact vector search that needs no search index.
What we measured
The kill test runs an agent through three tools in turn, a lookup (pure), an invoice save (idempotent) and an email (external), and kills it with SIGKILL, together with its Recall subprocess, at one of three points: after the call is recorded but before it takes effect, after it takes effect but before the tool returns, and after the tool returns but before the next model call. A new process then resumes the agent and finishes. The mail server and invoice store are files outside both processes, so every side effect is counted.
The agent is a scripted, cautious policy rather than a model: it skips a tool whose result it can see, and does not repeat a call its briefing says may have happened. What it can see is exactly what Recall hands it, so the test measures what the harness tells a resumed agent. 20 trials per setup, tool and kill point, 540 in all, on polign_db 0.15.0 and again on MongoDB 8.0.
| Setup | Duplicate emails | Duplicate invoices | Lost emails | Left for a person to check |
|---|---|---|---|---|
| Resume, ordinary tools | 20 of 60 | 20 of 60 | 0 | 0 |
| Durable tools, no resolver | 0 | 0 | 0 | 40 of 60 |
| Durable tools with a resolver | 0 | 0 | 0 | 0 |
Counts are for the 60 trials that killed the agent in that tool. Results were identical on
polign_db and MongoDB. All 540 trials were killed at their point and every resumed run finished.
Harness: python/recall-langgraph/bench/kill_actions.py.
With ordinary tools, every duplicate came from the kill after the effect: the email went out, its result was never recorded, and the resumed agent sent it again. Kills before the effect or after the tool returned did no harm. With durable tools the interrupted invoice was saved again under its original key, so the store kept one. Without a resolver, the 40 interrupted emails were never re-sent; they waited at the top of the briefing for someone to check. With a resolver reading the mail log, emails already sent were recorded as done and emails never sent were sent once.
Availability
Available today:
- polign_db 0.15.0: the server checks the lease on agent writes.
- Recall 0.21.0: actions, reconcile, fenced writes, and the MongoDB backend.
brew install polign/tap/recallorpip install polign-recall. - recall-langgraph 0.2.0:
RecallResume.durablefor LangGraph agents.
Recall and the Python packages are Apache-2.0 in
github.com/Polign/recall and
github.com/Polign/polign.
polign_db is free to self-host. Other harnesses use the same records through the
begin_action, end_action and pending_actions MCP tools; see
the architecture docs.
Try it
Running agents that send, charge or deploy? We'll help you classify their tools and wire this in, in your own cloud.