Agent Memory & Context Evaluation Benchmark

Overview

Persistent memory for AI agents is an ecosystem full of confident claims and short on controlled measurement. Vendors assert that memory makes agents better; practitioners feel that it sometimes does and sometimes derails a session with stale or irrelevant recall; almost nobody publishes numbers a skeptic could re-run. I am designing an open benchmark to replace that argument with an experiment.

The testbed is infrastructure I already operate daily: MCP Context Server, my persistent context store for LLM agents, exercised through real coding-agent workflows. That gives the benchmark something most memory evaluations lack — a production system whose retrieval modes (full-text, semantic, hybrid, hybrid with reranking) can be swapped as controlled conditions rather than simulated.

The Hypothesis

Structured persistent context improves multi-session agent task completion compared with no memory and with file-based handoff, without introducing unacceptable retrieval errors, latency, or cost. Every clause of that sentence is falsifiable, and the design treats the negative result as a legitimate finding: if markdown handoff files match a retrieval store on task success at lower cost, that is worth knowing and publishing.

What It Will Measure

• Baselines that bracket the question. An agent with no persistent memory; file-based handoff (the CLAUDE.md-style pattern); automatic conversation summarization; and the context server under full-text, semantic, hybrid, and hybrid-plus-reranking retrieval, with and without summarization.
• Task families drawn from real multi-session work. Factual recall, cross-session state, project constraints, tool selection, long-document navigation, conflicting memories, stale information, failure recovery, continuation after context compaction, and multi-agent handoff — on the order of one to two hundred reproducible scenarios.
• Metrics on both sides of the trade. End-to-end task success, factual correctness, unsupported claims, retrieval recall and ranking quality, tool-selection accuracy — against tokens, latency, cost, and tool-call counts. Human scores and LLM-judge scores, with the agreement between them reported rather than assumed.

Methodology Commitments

These are the commitments the project is being designed around, stated up front because they are the point: a versioned dataset with a held-out test set; fixed model and prompt versions per run; repeated runs with confidence intervals rather than single-shot anecdotes; ablations that isolate each retrieval stage's contribution; an error taxonomy instead of a bare failure count; a reproducible environment; published raw results next to the aggregate numbers; and a regression gate so the benchmark can hold its own future versions to its own standard.

Status

Concept, announced ahead of the build deliberately. The design work — task taxonomy, baselines, and metric definitions — is where this project currently lives, and the repository, harness, and first report will follow as they materialize. It is the evaluation layer my own agent-infrastructure work has earned: the systems exist, they run in production, and the discipline of measuring them properly is the next capability I am building in the open.