Autonomous Agent Memory: Two-Tier Architecture
The prevailing myth in AI agent memory architecture is seductive in its simplicity: give your agent a vector database, embed everything it experiences, and retrieve the top-k results at query time. Done. Solved. Ship it.
I believed this too — right up until Familiar started confidently acting on facts that were three weeks stale.
Familiar is an autonomous agent system we built internally, designed to run persistent loops across long-horizon tasks: research, code review workflows, cross-session planning. The kind of work where context isn't just helpful — it's load-bearing. When the memory layer fails in a system like that, the agent doesn't crash. It just quietly becomes wrong. And quiet wrongness in an autonomous system is catastrophically worse than a loud failure.
Here's what we actually learned, and why almost everything the AI community currently believes about agent memory is optimized for demos, not production.
The Myth: Retrieval Is the Hard Problem in LLM Memory
The conventional wisdom is that the core challenge of agent memory is retrieval quality — getting the right embeddings, tuning your similarity thresholds, picking the right vector store. The entire ecosystem has organized itself around this belief. LangChain's memory abstractions, LlamaIndex's retrieval pipelines, the explosion of managed vector databases — all of it is fundamentally retrieval infrastructure.
Why do people believe this? Because in benchmarks and demos, it's true. You load a corpus, you ask questions, you measure recall. Retrieval quality is measurable. It produces clean numbers. It's the part of the problem that looks like a machine learning problem, which means it attracts ML researchers, which means it gets published, which means it becomes the frame everyone uses.
The actual hard problem isn't retrieval. It's memory lifecycle management: what happens to memories over time, how they get invalidated, and whether the agent can tell the difference between something it learned yesterday and something that was true six months ago and has since changed.
What Actually Breaks in Production
When we first deployed Familiar's memory layer, we used a fairly standard setup: everything the agent observed got embedded and stored, retrieval happened via cosine similarity, and we tuned the threshold until recall looked good on our test cases. It worked beautifully in staging.
In production, three failure modes emerged within the first two weeks.
Duplication without reconciliation. The agent would encounter the same fact — say, the current status of a particular codebase module — in multiple sessions. Each observation got stored as a separate embedding. When retrieved together, they appeared to corroborate each other, which the LLM interpreted as high-confidence signal. But they were just echoes of the same original observation. The agent was essentially counting one data point as three, and treating the artificial consensus as ground truth.
Staleness with no expiry signal. We had memories from early system setup that described architectural decisions that had since been reversed. Those memories had high semantic relevance to current queries — of course they did, they were about the same system. But they were wrong. The vector store had no concept of a memory being outdated. It just served whatever was most similar.
Context window blowout during real-time loops. This one almost sunk the whole project. In long-running agent loops, we were injecting retrieved memories at each step. As the loop progressed, the context window filled with memory payloads, leaving progressively less room for the actual task. By step 15 or 20 of a complex loop, the agent was operating with a degraded view of its current task because its own memory system was consuming the budget.
None of these are retrieval quality problems. They're architecture problems. And they don't show up in benchmarks.
The Two-Tier Solution We Actually Built
After a painful two weeks of watching Familiar make confident, well-reasoned, completely wrong decisions, we scrapped the single-store model and built a two-tier system.
Tier 1: Operational Memory (Markdown Files)
The first tier is intentionally primitive. It's a set of markdown files that the agent can read and write mid-run. These files are scoped to the current operational context — think of them as the agent's working notepad. They're ephemeral by design: they exist for the duration of a task or session, and they get archived (not deleted, archived) when the session closes.
Why markdown? Because it's directly readable by the LLM without any retrieval layer. The agent loads the relevant operational files at the start of each loop step, reads them in full, and updates them as it makes progress. There's no embedding, no similarity search, no threshold tuning. The agent writes ## Current Status and ## Decisions Made and ## Open Questions in plain text, and the next step of the loop reads that plain text.
This solved the context window blowout problem immediately. Operational files are bounded in size by convention — we enforce a soft limit of around 800 tokens per file. If an operational file is getting long, the agent is instructed to summarize and consolidate before continuing. The working memory stays lean.
Tier 2: Curated Long-Term Memory (YAML Frontmatter)
The second tier is where things get interesting. Long-term memories in Familiar are stored as markdown files with YAML frontmatter — not as raw vector embeddings. The frontmatter carries metadata that the retrieval layer actually uses to make lifecycle decisions:
---
memory_id: "arch-decision-2026-06-14-001"
created: "2026-06-14"
last_validated: "2026-07-01"
expires: "2026-09-01"
confidence: 0.87
source_session: "planning-loop-44"
tags: ["architecture", "database", "postgres"]
supersedes: []
superseded_by: null
---
The expires field is the one that changed everything. Every long-term memory has a TTL. When the retrieval layer pulls candidates, it filters out expired memories before scoring. The agent never sees a stale memory — it simply doesn't exist in the retrieval set.
The supersedes / superseded_by fields handle the duplication problem. When the agent forms a new memory about a topic it has prior memories on, it doesn't just append — it explicitly links the new memory to the old one and marks the old one as superseded. The retrieval layer follows these links and excludes superseded memories from results. What looked like corroborating evidence in our old system now correctly resolves to a single authoritative memory.
last_validated is used by a lightweight background process that periodically resurfaces memories to the agent with a simple prompt: "Is this still accurate given what you know?" It's not sophisticated, but it catches the worst cases of silent staleness.
Why This Feels Counterintuitive to ML Engineers
There's a real tension here that I want to name directly. The two-tier architecture I'm describing is, in some ways, less "AI" than the vector-store approach. It's more file system. More structured. More explicit. And that makes ML engineers uncomfortable, because it feels like you're fighting the medium.
The current discourse around LLM burnout captures something real about where the field is right now — there's a growing fatigue with approaches that pile LLM-on-LLM without asking whether the underlying architecture is sound. The agent memory problem is a perfect example. We reached for embeddings because embeddings are the LLM-native tool. But the actual failure modes we hit were data lifecycle problems, and data lifecycle problems have known solutions that don't require neural networks.
The YAML frontmatter approach is also interesting from a durability standpoint. There's been good discussion recently about durable execution patterns — the idea that long-running processes need explicit checkpointing and state management to survive failures. Agent memory is exactly this problem. A memory that can't be invalidated, versioned, or expired isn't durable — it's a liability. The YAML frontmatter gives us the checkpointing semantics we need.
What the Retrieval Layer Actually Does Now
The vector store didn't disappear — it got demoted. We still embed the long-term memory files and use cosine similarity for initial candidate retrieval. But the retrieval layer is now a pipeline with multiple gates:
- Semantic similarity (vector search, top-20 candidates)
- Expiry filter (remove anything past TTL)
- Supersession filter (remove anything with a non-null
superseded_by) - Confidence threshold (remove anything below 0.6)
- Token budget enforcement (trim to fit available context window)
The result is that the agent sees a small, curated, high-confidence set of memories at each step — not a raw similarity dump. The LLM doesn't have to reason about whether a memory is still valid. That work happens before the context is assembled.
This is a meaningful architectural shift. We moved validity reasoning out of the LLM (where it's expensive, inconsistent, and burns context tokens) and into deterministic pre-retrieval filters (where it's cheap, consistent, and invisible to the context window).
The Metrics After Two Months
I want to be specific here because vague claims about improvement are worthless.
Before the two-tier system: Familiar made decisions that we could trace back to stale or duplicated memories in roughly 23% of long-horizon task sessions. That number came from a manual audit of 60 sessions where we tracked the provenance of every significant agent decision.
After the two-tier system, over 90 sessions: that number dropped to 4%. The remaining cases were all situations where the agent formed a new memory that was itself incorrect — a ground truth problem, not a memory architecture problem. That's a different failure mode and a different fix.
Context window utilization at step 20 of a long loop: was averaging 71% consumed by memory payloads before. Now averages 28%. That's not a small improvement — that's the difference between an agent that can still reason coherently mid-task and one that's operating in a fog of its own history.
What to Actually Build
If you're designing AI agent memory architecture today, the framing I'd push back on hardest is "retrieval is the core problem." It's not. The core problem is memory validity over time.
Start with TTLs on every memory. Non-negotiable. If you can't expire a memory, you can't trust your memory layer in production.
Separate operational memory (short-lived, directly readable, scoped to current task) from long-term memory (curated, validated, explicitly versioned). Don't try to serve both use cases from a single store.
Move validity reasoning into deterministic pre-retrieval filters. Don't ask the LLM to figure out if a memory is stale — it will sometimes get it right and sometimes confidently get it wrong, and you won't be able to predict which.
And audit your memory layer against real production sessions, not benchmark datasets. The failure modes that will kill you in production are invisible in benchmarks because benchmarks don't have time. They don't have sessions that run for weeks. They don't have architectural decisions that get reversed. Production does.
Familiar is still running. The memory layer is still evolving. But it's no longer the part of the system that keeps me up at night — and that's the only metric for an infrastructure component that really matters.