Independent audits and repeated evaluations will determine whether the Agent Memory Leaderboard produces reproducible, decision-useful comparisons across open-source and commercial agent-memory systems.

state: expiredheat: lowuncertainty: highconvergesscott: mediumagent-memory memory-evaluation agent-harnessesAgent Memory Leaderboard

What is this?

The Agent Memory Leaderboard is a public evaluation platform intended to compare agent-memory systems under shared conditions, with participants exposing Add/Search APIs and being scored on textual-memory and coding-agent-memory tracks. Its stated motivation is that vendors currently report results using different datasets, answer models, and judges, making scores difficult to compare. The supplied snippets include secondary claims that Hindsight leads several benchmarks, but they do not establish independent audits, repeated reproductions, or enough methodology and primary-result detail to conclude that the leaderboard’s comparisons are yet decision-useful.

Why it matters to Scott

The leaderboard independently moves toward Scott’s vendor-neutral, model-plus-harness evaluation position and directly overlaps his trace-backed agent-comparison work. It creates a practical comparison and publishing opportunity, but the supplied evidence does not yet show the audits, replayable traces, controlled harness disclosure, or repeated evaluations needed to establish decision-useful results.

ip:concept.model-plus-harness-benchmark-unitip:concept.capability-auditdev:concept.trace-backed-agent-comparisondev:project.thinkerradar:memory-bench-layer-baseline-validityradar:concept.agent-memoryradar:concept.agent-evaluationradar:concept.agent-benchmarks

queries asked of Scott's wikis

agent memory evaluation methodology and reproducibility
memory benchmarks for coding agents and harnesses
file vs vector vs graph memory architectures
agent-maintained wiki memory evaluation
retrieval quality versus downstream agent performance
vendor-neutral benchmarks and evaluation harnesses

Measured heat

no measured readings yet — the hourly heat pass fills this in

How the heat travelled

no chain yet — the hourly chain pass fills this in

Evidence (5) — ⭐ canonical anchor

source	object	author	score	comments
🟧 hn	Show HN: Agent Memory Leaderboard – first public results for AI memory systems	IreneAI	2	0
🟧 echo.other ⭐	This is the primary results publication, not a report quoting an earlier announcement. The site publishes the academic textual leaderboard a	Agent Memory Leaderboard Organizers	—	—
🟧 hn	Show HN: I evaluated file, vector, graph and RL based memory frameworks	pinglin	10	0
🟠 reddit	A scale for your AI's notes. Notes that pay their rent stay. Notes that don't get thrown out. And it refuses to claim a saving it can't prove. ClaudeAI	tvuk	1	2
🟧 hn	AI Memory for Claude: An Honest 4-Way Comparison	labyrinthAC	1	0

Interpretation history

2026-08-22T17:27:58Z
Repeated reviews have produced only adjacent evaluation examples and minor amplification, with no audit, replayable traces, or versioned rerun testing the inaugural rankings. The episode has faded; a direct reproduction or new leaderboard run should open a fresh case.
2026-08-20T16:41:15Z
Minor engagement on an adjacent comparison adds no methodology, replay materials, audit, or repeated leaderboard run. The case remains a cold, prospective benchmark-validation question despite the broader topic’s heat.
2026-08-18T15:46:09Z
The four-way Claude memory comparison remains a title-only adjacent evaluation with no disclosed methods, results, traces, or replay materials. It neither reproduces nor audits the Agent Memory Leaderboard, so the case remains an unvalidated first run awaiting direct independent testing.
2026-08-18T14:23:50Z
evidence attached: hn.story.49345976 — This hunted comparison is independent evaluation evidence bearing directly on whether Claude memory approaches can be compared in a decision-useful way.
2026-08-17T17:41:03Z
No audit, reproduction, disclosed trace set, or repeated versioned run has appeared; the attached items remain adjacent evaluation examples rather than validation of the leaderboard. The case stays open but cold until evidence directly tests the inaugural rankings.
2026-08-15T16:39:27Z
Token Warden adds a concrete A/B harness pattern for testing whether persistent memory rules justify their context cost, but it neither audits nor reproduces the Agent Memory Leaderboard. The case remains an unvalidated first leaderboard run awaiting independent replay, disclosed traces, or a repeated versioned evaluation.
2026-08-15T16:23:04Z
evidence attached: reddit.post.1vp6178 — The released token-warden plugin introduces a concrete task-based method for measuring whether persistent coding-agent memory rules repay their context cost.
2026-08-15T15:36:55Z
grounded: converges/medium — The leaderboard independently moves toward Scott’s vendor-neutral, model-plus-harness evaluation position and directly overlaps his trace-backed agent-compariso
2026-08-15T15:34:13Z
The newly attached comparison is only a bare title and does not disclose methods, versions, results, or any audit of the leaderboard, so it does not yet provide an independent line of validation. The case remains about whether the released inaugural rankings can withstand reproduction and repeated versioned runs.
2026-08-15T15:23:04Z
evidence attached: hn.story.49310834 — A comparative evaluation of file, vector, graph, and RL memory frameworks materially informs whether agent-memory comparisons are decision-useful.
2026-08-15T04:25:04Z
Only minor engagement arrived; there is still no independent audit, reproduction, or second versioned run to validate the leaderboard’s rankings. The case remains a prospective benchmark-integrity question rather than established comparative infrastructure.
2026-08-13T03:35:45Z
grounded: known/medium — Scott already holds the core position in “Model-Plus-Harness Benchmark Unit” and “Trace-backed agent comparison”: comparisons must bind results to a disclosed,
2026-08-13T03:33:16Z
No independent audit, repeated run, or implementation evidence has arrived; this is unchanged first-party publication rather than validation of reproducibility or decision usefulness. The cached grounding is now misleading because the inaugural results have been released.
2026-08-13T03:31:51Z
grounded: known/low — Scott already holds the core position in Model-Plus-Harness Benchmark Unit and Trace-backed agent comparison, while the radar tracks substantially the same vali
2026-08-13T03:29:22Z
origin walked (codex/luna, conf 0.9): anchor hn.story.49281370 -> echo.other.a705d6dc35 by Agent Memory Leaderboard Organizers
2026-08-13T03:27:27Z
case created — The first public evaluation covering 69 memory frameworks is a substantive artifact that could become useful comparative infrastructure if its methodology holds up.

Decision trace

08-23 03:27expireRepeated reviews have produced only adjacent evaluation examples and minor amplification, with no audit, replayable traces, or versioned rerun testing the inaugural rankings. The episode has faded; a
08-23 03:27alert_silentThe latest delta is only staleness and negligible engagement, with no consequential validation event to surface.
08-23 03:27alert_routeThe latest delta is only staleness and negligible engagement, with no consequential validation event to surface.
08-21 02:41repriceMinor engagement on an adjacent comparison adds no methodology, replay materials, audit, or repeated leaderboard run. The case remains a cold, prospective benchmark-validation question despite the bro
08-21 02:41alert_silentNo consequential new event occurred; the delta is only staleness plus negligible engagement, so it can wait for a direct reproduction, audit, trace release, or versioned rerun.
08-21 02:41alert_routeNo consequential new event occurred; the delta is only staleness plus negligible engagement, so it can wait for a direct reproduction, audit, trace release, or versioned rerun.
08-19 01:46repriceThe four-way Claude memory comparison remains a title-only adjacent evaluation with no disclosed methods, results, traces, or replay materials. It neither reproduces nor audits the Agent Memory Leader
08-19 01:46alert_silentThe new attachment does not establish a consequential event or materially advance the leaderboard’s reproducibility; it can wait for routine review unless methodology or results appear.
08-19 01:46alert_routeThe new attachment does not establish a consequential event or materially advance the leaderboard’s reproducibility; it can wait for routine review unless methodology or results appear.
08-19 00:25alert_silentThe new item is only a low-engagement title/link for a four-way Claude memory comparison; the supplied evidence includes no methodology, results, replayable traces, controlled harness, or connection t
08-19 00:25alert_routeThe new item is only a low-engagement title/link for a four-way Claude memory comparison; the supplied evidence includes no methodology, results, replayable traces, controlled harness, or connection t
08-19 00:23attachThis hunted comparison is independent evaluation evidence bearing directly on whether Claude memory approaches can be compared in a decision-useful way.
08-19 00:23propose_attachThis hunted comparison is independent evaluation evidence bearing directly on whether Claude memory approaches can be compared in a decision-useful way.
08-18 03:41repriceNo audit, reproduction, disclosed trace set, or repeated versioned run has appeared; the attached items remain adjacent evaluation examples rather than validation of the leaderboard. The case stays op
08-18 03:41alert_silentThis look contains only a staleness trigger and no consequential new evidence; there is nothing Scott needs before the normal briefing.
08-18 03:41alert_routeThis look contains only a staleness trigger and no consequential new evidence; there is nothing Scott needs before the normal briefing.
08-16 05:21sensor_dirtyengagement_update
08-16 02:39repriceToken Warden adds a concrete A/B harness pattern for testing whether persistent memory rules justify their context cost, but it neither audits nor reproduces the Agent Memory Leaderboard. The case rem
08-16 02:39alert_silentThe implementation is relevant as an evaluation technique but provides no validation of the leaderboard and has no established methodological rigor, results, or adoption; normal briefing is sufficient
08-16 02:39alert_routeThe implementation is relevant as an evaluation technique but provides no validation of the leaderboard and has no established methodological rigor, results, or adoption; normal briefing is sufficient
08-16 02:23alert_silentToken Warden presents a directly relevant, testable A/B approach to measuring whether persistent agent-memory rules repay their token cost, but the supplied evidence is only the author's low-deta
08-16 02:23surface_candidateToken Warden presents a directly relevant, testable A/B approach to measuring whether persistent agent-memory rules repay their token cost, but the supplied evidence is only the author's low-deta
08-16 02:23alert_routeToken Warden presents a directly relevant, testable A/B approach to measuring whether persistent agent-memory rules repay their token cost, but the supplied evidence is only the author's low-deta
08-16 02:23attachThe released token-warden plugin introduces a concrete task-based method for measuring whether persistent coding-agent memory rules repay their context cost.
08-16 02:22propose_attachThe released token-warden plugin introduces a concrete task-based method for measuring whether persistent coding-agent memory rules repay their context cost.
08-16 01:36repriceThe newly attached comparison is only a bare title and does not disclose methods, versions, results, or any audit of the leaderboard, so it does not yet provide an independent line of validation. The
08-16 01:36groundThe leaderboard independently moves toward Scott’s vendor-neutral, model-plus-harness evaluation position and directly overlaps his trace-backed agent-comparison work. It creates a practical compariso
08-16 01:34alert_silentThe independent-looking comparison could become consequential, but the available evidence establishes no methodology or finding and does not validate the leaderboard; fuller evidence can wait for the
08-16 01:34alert_routeThe independent-looking comparison could become consequential, but the available evidence establishes no methodology or finding and does not validate the leaderboard; fuller evidence can wait for the
08-16 01:23alert_silentA new independent-looking comparison across file, vector, graph, and RL memory approaches could become useful validation evidence, but the visible evidence contains only the post title and establishes
08-16 01:23surface_candidateA new independent-looking comparison across file, vector, graph, and RL memory approaches could become useful validation evidence, but the visible evidence contains only the post title and establishes
08-16 01:23alert_routeA new independent-looking comparison across file, vector, graph, and RL memory approaches could become useful validation evidence, but the visible evidence contains only the post title and establishes
08-16 01:23attachA comparative evaluation of file, vector, graph, and RL memory frameworks materially informs whether agent-memory comparisons are decision-useful.
08-16 01:22propose_attachA comparative evaluation of file, vector, graph, and RL memory frameworks materially informs whether agent-memory comparisons are decision-useful.
08-15 14:25repriceOnly minor engagement arrived; there is still no independent audit, reproduction, or second versioned run to validate the leaderboard’s rankings. The case remains a prospective benchmark-integrity que
08-15 14:25alert_silentNo consequential new evidence has arrived beyond the already assessed first-party release; wait for an independent audit, reproduced ranking, or repeated versioned evaluation.
08-15 14:25alert_routeNo consequential new evidence has arrived beyond the already assessed first-party release; wait for an independent audit, reproduced ranking, or repeated versioned evaluation.
08-13 13:35repriceNo independent audit, repeated run, or implementation evidence has arrived; this is unchanged first-party publication rather than validation of reproducibility or decision usefulness. The cached groun
08-13 13:35groundScott already holds the core position in “Model-Plus-Harness Benchmark Unit” and “Trace-backed agent comparison”: comparisons must bind results to a disclosed, versioned harness and retain auditable e
08-13 13:33alert_silentThis look adds no consequential delta beyond the already-routed inaugural release; wait for an independent reproduction, methodology audit, or second versioned run.