---
title: "Agent Memory & Context Evaluation Benchmark: Does Persistent Context Actually Help?"
description: "An open benchmark I am designing to answer a question the agent-memory ecosystem mostly asserts instead of measuring: does structured persistent context improve multi-session agent task completion over no memory and file-based handoff — and at what cost in retrieval errors, latency, and tokens? Versioned dataset, held-out test set, confidence intervals, ablations, published raw results."
date: "2026-08-27T07:00:00Z"
last_updated: "2026-08-28"
build_time: "2026-08-28T03:11:43Z"
project_name: "Agent Memory & Context Evaluation Benchmark"
project_description: "An open benchmark, in design, that measures whether structured persistent context improves multi-session agent task completion over no memory and file-based handoff — and at what cost in retrieval errors, latency, and tokens. Versioned dataset, held-out test set, repeated runs with confidence intervals, ablations, error taxonomy, published raw results."
status: "Concept"
project_group: "open-source-experiments"
period_from: "2026-08-27"
related_skills: ["artificial-intelligence","claude-code","mcp","python","system-analysis","quality-assurance"]
related_projects: ["mcp-context-server"]
keywords: ["Agent Evaluation","Evals","Benchmark","Agent Memory","Persistent Context","Context Engineering","Retrieval Evaluation","LLM as Judge","Evaluation Methodology","MCP","Claude Code","Reproducible Experiments"]
license: "https://creativecommons.org/licenses/by/4.0/"
canonical: "https://www.alexfeel.info/projects/agent-memory-context-benchmark/"
---

## Overview

Persistent memory for AI agents is an ecosystem full of confident claims and short on controlled measurement. Vendors assert that memory makes agents better; practitioners feel that it sometimes does and sometimes derails a session with stale or irrelevant recall; almost nobody publishes numbers a skeptic could re-run. I am designing an open benchmark to replace that argument with an experiment.

The testbed is infrastructure I already operate daily: [MCP Context Server](/projects/mcp-context-server/), my persistent context store for LLM agents, exercised through real coding-agent workflows. That gives the benchmark something most memory evaluations lack — a production system whose retrieval modes (full-text, semantic, hybrid, hybrid with reranking) can be swapped as controlled conditions rather than simulated.

## The Hypothesis

Structured persistent context improves multi-session agent task completion compared with no memory and with file-based handoff, without introducing unacceptable retrieval errors, latency, or cost. Every clause of that sentence is falsifiable, and the design treats the negative result as a legitimate finding: if markdown handoff files match a retrieval store on task success at lower cost, that is worth knowing and publishing.

## What It Will Measure

- **Baselines that bracket the question.** An agent with no persistent memory; file-based handoff (the CLAUDE.md-style pattern); automatic conversation summarization; and the context server under full-text, semantic, hybrid, and hybrid-plus-reranking retrieval, with and without summarization.
- **Task families drawn from real multi-session work.** Factual recall, cross-session state, project constraints, tool selection, long-document navigation, conflicting memories, stale information, failure recovery, continuation after context compaction, and multi-agent handoff — on the order of one to two hundred reproducible scenarios.
- **Metrics on both sides of the trade.** End-to-end task success, factual correctness, unsupported claims, retrieval recall and ranking quality, tool-selection accuracy — against tokens, latency, cost, and tool-call counts. Human scores and LLM-judge scores, with the agreement between them reported rather than assumed.

## Methodology Commitments

These are the commitments the project is being designed around, stated up front because they are the point: a versioned dataset with a held-out test set; fixed model and prompt versions per run; repeated runs with confidence intervals rather than single-shot anecdotes; ablations that isolate each retrieval stage's contribution; an error taxonomy instead of a bare failure count; a reproducible environment; published raw results next to the aggregate numbers; and a regression gate so the benchmark can hold its own future versions to its own standard.

## Status

Concept, announced ahead of the build deliberately. The design work — task taxonomy, baselines, and metric definitions — is where this project currently lives, and the repository, harness, and first report will follow as they materialize. It is the evaluation layer my own agent-infrastructure work has earned: the systems exist, they run in production, and the discipline of measuring them properly is the next capability I am building in the open.


## Site index

- [llms.txt](https://www.alexfeel.info/llms.txt)
- [Complete index](https://www.alexfeel.info/llms-index.txt)