RESEARCH
Agent memory: findings from my experiments
How does memory for my agents actually work? A week with Hindsight, a plain SQLite memory built with my agents, and what a 15-question comparison measured.
- METHOD
- My agent threads, frozen run artifacts and a read-only diagnosis. No new runs for this article.
- WINDOW
- 5–11 Oct 2026. Fifteen questions, one frozen comparison.
- RESULT
- 13 of the 14 points of the gap came from data my memory never indexed. Retrieval and packing showed two mechanisms worth changing.
- my memory's score on 15 questions; Hindsight scored 23 / 30
- 9 / 30
- points of the gap from questions it had no data for
- 13 of 14
- questions where its keyword search returned anything
- 0 of 15
- where search put the right message; the 900-character excerpt stopped before the answer
- rank 2
The question: what does an agent's memory actually do?
I run Hermes, my self-hosted agent setup, as daily infrastructure. Several agent profiles talk to me over Discord and Telegram and run cron jobs. Each profile keeps its own conversation history in SQLite.
Memory is the layer that should let an agent answer “what did we decide about this last week?” without me pasting the thread again. I wanted to know what that layer really does between a conversation and an answer: what it stores, how it finds things, and what ends up in the model’s context. Until this week the layer was Hindsight, an open-source memory server, and I wasn’t happy with what it recalled. So I ran two experiments. First I tuned Hindsight and watched how it behaved on my setup. Then I built a simpler memory of my own with my agents and compared the two on 15 questions about my own history.
Most of the hands-on work was done by agents: Hermes itself and the Claude Code workers it launched. I made the calls and read the reports. Some of those reports overstated a result and were corrected later; where that happened, I say so.
The short version: my homemade memory scored 9/30 against Hindsight’s 23/30. Tracing every question showed that most of that gap measured the test, not retrieval. Underneath it, the trace found two mechanisms in my own code, one in retrieval and one in context packing.
Text version of the figure
Hindsight (stages from its docs; settings from my config on 7 October)
- Conversation, retained every 10 turns.
- Retain: an LLM writes facts and observations. What is stored is extracted text.
- Recall: semantic, BM25 keyword, graph and temporal search, fused by reciprocal rank and reranked.
- Budget: the
lowbudget passes the top 60 candidates to the reranker; the result fills a 512-token memory block. - The model gets the memory only if it arrives within Hermes’ 8-second wait.
hermes-mem (the tested 0.3 candidate)
- Conversation, with tool output and other noise dropped.
- Store: the messages themselves, BGE-M3 vectors and a few agent-written cards.
- Search: keyword (strict AND) and semantic.
- Excerpts: the head of each message, cut to 900 characters.
- The model gets the excerpts that fit the budget.
The public 0.2.0 returns whole episodes instead of single messages and keeps reviewed records instead of cards.
Hindsight in use: extraction, recall and the deadline
Hindsight has three verbs. Retain uses an LLM to turn conversations into facts, experiences and consolidated “observations”. Recall searches them: semantic, BM25 keyword, graph and temporal search, fused by reciprocal rank, reranked, then trimmed to a token budget. Reflect runs a slower reasoning loop over the memories. Memories live in isolated banks. All of that is from Hindsight’s own docs; what follows is what I observed on my setup.
My Hermes plugin config, as an agent printed it on 7 October: automatic recall on, asynchronous, observation-only, the low budget, a 512-token memory block, a 20-second timeout. Retain ran every ten turns. Reranking went through Luna, an LLM reached through my ChatGPT subscription. The embedding model was E5-small. My agents never recorded the Hindsight version or the retain model in those threads, so I can’t give you either.
What I saw falls into three layers.
What goes in (our setup). We fed cron output into retain, and one cron job alone produced 2,526 of 7,786 consolidated records, 32.4%. The low budget passes the top 60 candidates to the reranker. In one audit, 19 of 36 records we had agreed were useful were cut before the reranker ever saw them. Raising that cap to 250 made reranking take 11–28 seconds instead of about 8. The experiment also used up a local 100-calls-a-day reranker limit that production shared, so for a while my live agents ran without reranking. Several test runs stopped on our own orchestration: broken task contracts, a supervisor relaunch loop, a cron job that couldn’t report back.
How it reaches a turn (the integration). Hermes waits at most 8 seconds for memory before a turn. In one screening run, recall missed that deadline in 99 of 177 successful requests. One answer came back from Hindsight in 8.48 seconds and Hermes dropped it. In asynchronous mode, a turn gets the memory that was fetched for the previous turn.
How extraction treated my data. In an audit of stored memories, two of my agent’s recommendations had been saved as my decisions, and it happened again later in live context. Retain rules were ours too, so I count that as mixed. Long Ukrainian questions went over the 500-token query limit and got an HTTP 400. Reflect was too slow to run every turn.
Then the tuning. Doubling the memory block from 512 to 1,024 tokens looked like a win on 20 tuning questions: 4 fully supported answers against 2. On held-out questions, both settings fully supported 1 of 9. The agent’s own verdict that night was that the settings we tested did not solve the problem. On 10 October I stopped the Hindsight experiments and moved to the second one.
A plain memory to compare against: hermes-mem
The second experiment was deliberately plain: take the messages, drop tool output and other noise, add an embedding to each one so the agent can search by meaning, and keep a few large cards about me, my intentions and the assistant itself. The agent writes those cards with a tool when it notices something worth keeping, or when I ask it to. That became hermes-mem: SQLite, FTS5 for keywords, and BGE-M3 for embeddings.
The model choice came from 22 synthetic Ukrainian and English queries run on a Mac.
Text version of the figure
| Model | Right dialogue ranked first (of 22) | Median warm query |
|---|---|---|
| BGE-M3 (INT8) | 21 | 52 ms |
| Qwen3-Embedding-0.6B | 21 | 1,065 ms |
| MiniLM | 17 | 13 ms |
BGE-M3 matched Qwen3’s 21 of 22 at a fraction of its query time. On eight extra queries with mixed-language wording, BGE-M3 got 8 of 8 and the other two 7 of 8. That is a small synthetic set.
Two infrastructure incidents from that week tend to get retold as model results, so here is what they were. The first Mac run stopped because Docker hit the macOS keychain before the container even started. Separately, an E5-small benchmark on my VPS was OOM-killed at 1,536 MiB while the model was still loading, before it had embedded a single text. The 41 seconds in that log is the time to the crash, not embedding throughput. Neither incident says anything about embedding quality.
Then the default profile’s history went in. Its database held 171,270 rows in 443 sessions, but rows aren’t messages: 74,115 were tool output, 55,861 were empty assistant turns, and plenty were copies left behind by context compaction. After filtering there were 2,834 messages, forming 1,288 episodes and 3,567 chunks to embed.
The first pilot embedded 148 of those chunks, about 4%. At roughly four seconds a chunk on the VPS, the rest would have taken about four hours. The agent reported the profile as “activated”. Search worked; the backfill hadn’t run and automatic sync was off. I switched my default profile to hermes-mem and back again the same evening.
Next we tried to carry over what Hindsight knew. Its export held 20,320 memory units, 8,107 of them observations, built from 654 source documents. Only 1,420 observations made it into a test database. Not because the rest were worthless:
Text version of the figure
| Import outcome | Observations |
|---|---|
| Imported into the test database | 1,420 |
| Conversation belongs to another profile | 6,216 |
| No link to a conversation | 252 |
| No owner tag | 140 |
| From scheduler sessions | 79 |
| All observations in the export | 8,107 |
The import filtered by access rules, and I had first read the number as a value judgement. Once that was clear, we started the design over with a smaller core.
The minimum version kept only messages, embeddings and cards. Review caught a privacy bug before acceptance: calls without a session fell into one shared session:unknown bucket. A second fix kept source references stable when a message was rolled back. The accepted revision passed 81 tests on each of two Python versions, 31 of 31 integration checks and 43 of 43 install checks. It is an unpushed 0.3 candidate. The public repository still holds 0.2.0, which returns whole episodes instead of single messages and keeps reviewed records instead of cards.
A 15-question comparison
The comparison used 15 questions about my own history, each worth 2 points under a written rubric. The candidate searched the default profile only: 2,914 messages, 4,384 of 4,384 chunks embedded, nine cards. Hindsight answered from its usual bank.
The result: my candidate 9/30, Hindsight 23/30. Hindsight did better on 10 questions, mine on 1, and 4 were ties.
The agent’s report did say the corpora weren’t equal. I read the headline as a verdict on my retrieval anyway, and that reading was the next thing to test.
One thing to keep straight before reading further: 9/30 is graded answer points. It is not recall@k, and it doesn’t tell you how often search found the right message.
What the score measured
So I went through it question by question: is the answer not in the database, or is it there and search didn’t find it? A separate agent traced each question through every stage, from source to corpus, search, context, answer and grade. It changed nothing and ran no new answers or judges.
Text version of the chart
| What happened | Kind | Questions | Points lost |
|---|---|---|---|
| Missing source dataThe answer was only in the two agent profiles it never indexed | Test setup | 9 | 18 |
| Context clippedFound the right message at rank 2, then cut the answer off at 900 characters | hermes-mem packing | 1 | 2 |
| Incomplete answerHad the facts in context, left part of the answer out (Hindsight lost the same point) | Answer step | 1 | 1 |
| Full-mark answers | Answered in full | 4 | 0 |
| Total | 15 | 21 |
The 14-point gap to Hindsight (23 − 9) splits by the same causes: 13 corpus + 2 packing + 0 answer − 1 where mine scored higher. An attribution of one frozen run, not a re-score. Also test setup: the answer key was written from Hindsight's own memories, and the test sent no reworded queries. Also a retrieval finding: hybrid keyword search found nothing on all 15 questions.
The corpus decided most of it. Hindsight’s bank mixes the history of three agent profiles. My candidate indexed one. In 9 of the 15 questions, every fact in the expected answer came from the two profiles the candidate never saw. Those nine questions account for 13 of the 14 points of the gap. In nine of my candidate’s ten zero-point answers, it honestly said the notes didn’t have it. The tenth was a real invention: it pulled in an unrelated detail.
The answer key leaned one way. The expected answers were written from Hindsight’s own observations. That bias didn’t come from leakage; everything was frozen in the right order. It came from using one contender’s memories as the answer key. In one question the key said I had decided to add a table to a document. The source conversation holds my request for a review and the agent’s recommendation to add that table. There is no decision from me. My candidate answered that no decision was recorded, which is true, and scored zero.
| Primary cause | Questions | Points my candidate lost | Points of the gap |
|---|---|---|---|
| Answer not in its corpus | 9 | 18 | 13 |
| Retrieved, then cut from the context | 1 | 2 | 2 |
| Support present, answer incomplete | 1 | 1 | 0 |
| Full-mark answers | 4 | 0 | −1 |
| Total | 15 | 21 | 14 |
The table attributes one frozen result to causes. It doesn’t claim what an equal test would score.
Retrieval and packing: two mechanisms in my code
Inside the data it did have, my own code limited what reached the model in two ways.
Hybrid search was semantic-only. In hybrid mode the keyword arm requires every word of the question to match, first as words, then as trigrams. Relaxing to OR is switched off there. On natural-language questions that never fires: no keyword hits on any of the 15 original questions or on 28 of 30 reworded variants. The strictness was deliberate. In the project’s first evaluation, OR matches on common words flooded the fusion and cut hybrid hit@1 from 0.93 to 0.14. It still left hybrid search running on one arm. Where the answer was in the corpus, semantic search usually found it: rank 1 or 2 for three questions. One more sat at rank 15 for the original wording and moved to rank 2 with a short Ukrainian rewording. The tool accepts such rewordings from the agent. The test harness passed none.
The right message, the wrong passage. Search returns the head of each message, cut to 900 characters; 59 of the 120 results returned in the run were cut. In the table question above, search put the right message at rank 2, but the table starts near character 4,435.
Text version of the figure
- Search ranked the right message 2nd.
- Inside that message, the table starts near character 4,435.
- An excerpt of 900 characters (the default) left the table out.
- An excerpt of 4,000 characters (the configured maximum) also left it out.
- An excerpt of 6,000 characters included the table, but then only three results fit the budget instead of eight.
My first idea was to centre the excerpt on the best-matching chunk. The trace ruled it out. The best-matching chunk of that message was the first one (similarity 0.619); the chunk holding the table scored 0.562. Search found the right message for the wrong part of it, and the plugin has no tool to read a whole message.
Both behaviours are in the public 0.2.0 on GitHub as well as the unpushed candidate, as of 11 October. Neither has been changed yet.
What to test next
Same data on both sides, first. Either index all three profiles for my candidate or restrict Hindsight to the one profile. Until that runs, I don’t know how much of the 13 points would close. Some facts could live in places neither side looked.
An answer key that doesn’t come from one contender’s memory. Write it from the original conversations, and check who actually said what.
Score the stages separately. Is the answer in the corpus? At what rank did search put it? Did it survive packing? Was the answer right? Graded points alone collapse all four into one number.
Agent status reports as claims to check. “Activated” meant search worked on 4% of the chunks. “45%” wasn’t accuracy. The reports were usually honest a message later, which is a fine trait in a colleague and a bad one in a dashboard.
The next experiments for hermes-mem are proposals, not shipped changes: return the matching passages instead of message heads, or add a bounded tool that reads a whole message; let the keyword arm fall back to OR or BM25 at a lower weight; pass the agent’s rewordings in the test the way the product intends. Another open-source memory tool, Basic Memory, documents both of the first two ideas: a matched passage next to the head preview and a full-note read tool. In my table case only the full read would have helped.
None of this crowns Hindsight or rules it out. It beat my candidate on a test that leaned its way, and on my setup its integration had real costs: the 8-second deadline, mixed-up authorship, a reranker cap that cut useful records. Those are separate questions, and this week answered none of them cleanly.
Method, sources and limits
Sources. Two private Discord threads between me and my agents (5–11 October 2026), the frozen comparison artifacts, and a read-only diagnosis that replayed the accepted search code with the pinned BGE-M3 model. Specific numbers come from tool output or reports in those threads; I kept a private ledger linking each one to its message and time. Raw transcripts, memory contents and private paths are not published.
What was not done. No equal-corpus rerun. The diagnosis didn’t open the other two profiles’ databases, so “not in the corpus” is bounded to the one profile and Hindsight’s export. The lookups used to find each answer were written after the fact to locate support, not to measure search. The exports are incomplete history; a missing detail isn’t proof that it never happened. Hindsight’s version and retain model weren’t recorded in the threads I used.
Public sources. Hindsight README and docs for retain, recall and reflect; hermes-mem (public 0.2.0 at 587a162; the tested 0.3 candidate 56120d6 is local); SQLite FTS5; the BGE-M3 model card; Basic Memory’s search docs. Vendor docs describe claimed behaviour, not my measurements.
The chart. Drawn for this article as SVG: a plain bar chart of where the 21 unscored points of the frozen run went, by cause. Its frame, axis grid, legend and direct labels borrow the grammar of the charts on Anthropic’s Claude Opus 5.5 page. The text version under the chart carries the same content.