Recall on LongMemEval: accuracy, tokens and latency

Agent memory is worth having when it puts the right facts in front of the model for a fraction of the tokens of the whole conversation, and answers fast enough to sit on every turn. We measured Recall 0.12 on that: how often the answer is right, how many tokens it sends, how long a read takes, and what it costs to write a user's history into memory.

Accuracy depends on the model reading the memory

Every method hands the same kind of model a context and asks it to answer. We ran each one with two answering models: gpt-4o, which the benchmark was published with, and gpt-6-astra, a current frontier model. A gpt-4o judge scored every answer, the same judge for every row.

Keyword search returns the 10 best-matching turns of the conversation. The oracle is handed only the sessions that contain the answer, so it is the ceiling for any retrieval method. "With embeddings" adds OpenAI text-embedding-3-small to Recall's word search.

With gpt-4o, Recall answers 81.3% of the questions against 73.8% for keyword search, winning 27 questions and losing 7 (sign test p = 0.0008), close to the 85.0% gpt-4o reaches when handed only the right sessions. At that point the answering model is the limit.

A frontier model changes that. gpt-6-astra improves every method, keyword search most of all, and given the right sessions it answers 96.6%. Now retrieval is the limit. Recall with word search alone is only 2.2 points ahead of keyword search (11 won, 5 lost), which is not a significant difference. With an embedding model Recall reaches 89.9%, six points ahead (19 won, 3 lost, p = 0.0009), on the same 4,400 tokens per question. If a frontier model reads your agent's memory, run Recall with an embedding model.

By kind of question

gpt-6-astra answering, 267 questions. The 15 questions that test whether the model knows it does not know are also counted inside their own kind.

Recall is strongest where memory goes wrong in production: facts that changed, questions about time, and declining to answer when nothing was said. It refuses more reliably than the oracle because it only returns what was believed by the question date. The largest gaps to the ceiling are questions about time (83.3% against 93.3%) and preferences (80% against 100%). Both are retrieval misses, and they are where we are working next.

What changed in Recall 0.12

Two changes took Recall from 77.5% to 82.0% with gpt-4o, with no extra tokens and no extra latency:

gpt-4o answering. A later run of the released 0.12 binary retrieved exactly the same memories for all 267 questions and scored 81.3%: gpt-4o answered four of them differently. Runs with the same retrieval vary by about two questions.

The time window is the bigger of the two. Most of the misses before it were questions like "what gardening did I do two weeks ago?", where word search found gardening from any month. Recall now reads the phrase against the date the question is asked, searches that span first, and still returns everything else after it. Nothing is dropped. A caller can also set the window explicitly with observed_after and observed_before.

Tokens: what memory costs on every turn

The tokens memory adds to each request are a cost an agent pays again on every turn. Recall sends about 4,400 tokens per question, including the original sentence each fact came from. Sending the whole history is about 104,000.

Mean over the 267 questions. Monthly cost assumes 10,000 users each making 200 memory reads a month at $2.50 per million input tokens, and counts only the memory context, not the user's message or the reply.

Change the numbers to see the monthly input cost of the memory context at your own volume:

Latency

Recall runs beside your agent and keeps its memory in your own polign_db, so a read makes no call to a third-party service. We timed the released recall binary through its Python client:

300 rounds on the managed local store that recall setup creates, on a laptop.

In the benchmark, where each user's store holds about 550 memories and every result comes back with its source conversation, a recall took 83 ms at the median and 138 ms at p95. The time-window search added nothing measurable. An embedding model adds the time to embed the question: about 350 ms per read through a hosted embedding API, which a local embedding model would avoid.

A write is readable on the very next call, so an agent that records a correction can rely on it in the same conversation. With memory in object storage, the bucket sets the floor:

Measured September 18, 2026 on a small store, with the server in the bucket's region. Ranges span polign_db's two serving modes.

Writing a user's history into memory

A model has to read new conversation to turn it into typed facts. Recall makes about one extraction call per session, and every fact keeps a link to the sentence it came from. You can also skip extraction: Recall then keeps each turn as a note and searches those.

One user history is a LongMemEval_S conversation: 48 sessions, 494 messages, 490 KB. Extraction by gpt-4o-mini at its list price. Storage is your own bucket; an idle Recall node costs a few dollars a month.