We let a model decide which memories to retire. It was wrong 32 times out of 35.

An agent's memory has to change when the user's life changes: the job ends, the plan is called off, the thing they liked is now the thing they avoid. The obvious design is to let a model read each new statement and decide which earlier memories it makes untrue. We built that, ran it on real conversations, and checked every memory it retired against the conversation. Recall 0.15 ships without it. This post is the measurement.

The setup

Recall keeps typed memories: a subject, a predicate, a value. user likes: mangoes. A predicate declares whether it holds one value or many, so a new value for a single-valued predicate replaces the old one by rule. What a rule cannot do is notice that user dislikes: mangoes ends user likes: mangoes, or that "we broke up" ends relationship: girlfriend Sarah. Those corrections cross predicates, and somebody has to say which belief ended.

So we gave the job to a model. For every statement the extractor produced, the server searched the subject's current beliefs for candidates: the same value under another predicate first, then the nearest in meaning. The corrector saw the new statement, the sentence it was drawn from, and the candidates with their dates, and answered with the beliefs the statement retires and why. An answer naming a belief it was not shown was dropped. In the version measured here it was only asked about statements whose own words announce a change: no longer, quit, cancelled, broke up, gave away, finished, avoid, and about forty more cues. Before that gate, in a pilot on the first five conversations, it retired 98 beliefs with one prompt and 65 with a tightened one, among them likes: bird photography, plan: make pasta from scratch soon and preference: journaling.

We ran the gated version twice: with a hand-tuned registry of predicates, and with an open vocabulary where the extractor coins predicates as it goes, starting from Recall's personal starter set.

What the model retired

Every retired belief leaves a retraction in its history, linked to the statement that retired it, so the full list can be pulled from the stores afterwards. Here is a sample. The full scored list of 35 is in the repository as retired-judged.jsonl.

Verdicts are ours, from reading the conversation around each sentence. "Unclear" was used twice, where the conversation does not say whether the belief ended; both are counted neither as right nor as wrong above.

The pattern is the same in nearly every row. The sentence shares a word or a topic with the belief (issues, party, Rachel, shoes, subscribers) and announces that something changed, and the model reads "related and newer" as "replaces". Nothing about the gate or the candidate search fixes that, because the model is doing exactly what it was asked to do: find a belief the new statement could plausibly end. On conversations about ordinary life, there is nearly always one.

The benchmark did not notice

LongMemEval scores answers to questions, and a knowledge-update question asks for the newest value. A retired belief that was still true only costs a point when a question asks for it, and these questions rarely do. So the usual accuracy number barely moved while the memory underneath it was losing facts.

77 questions each; the open-vocabulary difference is one question. gpt-4o answered and judged, the same prompts as our LongMemEval post. An open vocabulary gives the corrector more to retire because its beliefs are filed under more, and looser, predicates.

This is the part we would want anyone evaluating a memory system to take away. A question benchmark measures whether the right fact is in front of the model at answer time. It does not measure what the memory silently dropped. If a vendor's memory "resolves conflicts automatically", ask for the list of what it resolved.

What Recall does instead

Recall 0.15 handles corrections across predicates with three mechanisms, none of which asks a model to decide.

• Rules in the registry. A predicate lists the predicates it contradicts, value for value: dislikes excludes likes, gave_away excludes owns, cancelled excludes plans. A value stated under one ends the same value under the other, and the read reports what was replaced. It is applied when the log is read, so it covers statements written before the pairing existed.
• Explicit corrections. A statement can name the beliefs it makes untrue, under any predicate. Each one still believed is withdrawn with a link back to the statement. Anything named that does not hold is reported back as unreplaced, never dropped quietly. On every typed write Recall returns related, the beliefs that may conflict, so an agent that wants to confirm a correction has the candidates in hand and the decision is its own.
• Reads show the conflict. Every belief read lists, under elsewhere, the same value held under other predicates, with dates. "likes mangoes" comes back beside "dislikes mangoes" and the newer one usually holds. Nothing is decided at read time; it is shown.

The model extractor still files statements. What changed is that a statement which denies, reverses, or ends something is filed under the predicate that contradicts the earlier one, with the same value wording, and the rule does the rest. On our own set of 40 corrections across predicates and 28 lookalikes that only sound like corrections ("I also like papayas"), gpt-4o-mini's filing let the rule make 17 of 39 corrections and left all 26 lookalikes alone; qwen3:8b, a local model, made 17 of 37 and also left all 26 alone. That set is eval/corrections in the repository, and it runs through the product path with any extraction model.

Limitations

• The rule path misses corrections the extractor files under a predicate the registry does not pair: "I quit caffeine" filed as cancelled: caffeine does not end likes: morning coffee, because the values differ. That is the 22 of 39 above. Those conflicts surface only if the agent names them, and the fix is a better pairing or a better filing, both of which are visible and editable, not a model's judgment call.
• The verdicts are ours. The list is published so anyone can disagree row by row.
• One corrector model, gpt-4o-mini, with two prompts. A larger model would retire fewer true beliefs; we did not measure how many fewer, and the failure mode is in the task, not the model size: with no ground truth for "this belief ended", there is nothing to check a confident wrong answer against.
• 35 retirements is a small count. The conclusion rests on the proportion, not the total, and on the pilots that retired 98 and 65 beliefs in five conversations before the gate.