Memory — messages, context, and the art of forgetting

The 30-second version: LLMs have no memory — every call is a fresh start where Pi must re-send the whole conversation. That makes context Pi's most precious resource, and it manages it like one: trim inputs, assemble the prompt carefully, and when history gets too heavy, compress it into a summary instead of dropping it.

The goldfish problem

Here's the fundamental awkwardness of LLMs: they don't remember anything. Every single model call is a stranger walking in and needing the entire story re-told — your prompt, everything said so far, every tool result. The "conversation" you experience is an illusion maintained by the harness, which resends the full transcript each round.

The only thing limiting that transcript is the context window — a hard token budget per call (say, 200k tokens on Claude). Fill it, and the API simply refuses: prompt is too long. Game over.

So a coding agent — whose tool results can be enormous (think npm install output, or reading a 3,000-line file) — lives under constant threat of overflow. Pi's answer is three lines of defense.

Defense 1: trim at the door

The cheapest token is the one you never send. Pi trims tool output before it enters history:

bash output keeps the last ~2,000 lines or 50 KB (whichever hits first) — because errors and results live at the end of logs. The full output gets saved to a temp file, and the truncation notice tells the model where to find it — so it can read more if it really needs to.
read on huge files is similarly capped, keeping the beginning — because imports, types, and structure live at the top of source files.

Notice the elegance: truncation isn't silent data loss. It's a note that says "here's the gist; the archive is over there if you want it." The model stays in control of its own curiosity.

Defense 2: assemble the prompt like a care package

Every call to the model gets a carefully packed context:

system prompt      who you are, ground rules, tool list
context files      AGENTS.md from your project (and parent dirs)
skills index       names + descriptions of available skills
conversation       the transcript so far (trimmed, translated)

Two details are worth savoring.

Context files are recursive. Pi collects AGENTS.md / CLAUDE.md from your working directory, every parent directory, and a global one in ~/.pi/agent/. Monorepo? Your org conventions, team conventions, and project conventions all stack automatically, most specific last. Project knowledge without any hardcoding.

Skills use the library-card trick. Pi may know dozens of skills, but their full text stays out of the prompt — only a small index goes in (name + one-line description + file path). When a task matches, the model reads the skill file itself with the read tool. Instead of pre-loading every possible capability (expensive, mostly wasted), Pi lets the model pull knowledge on demand. Pay for what you use.

Defense 3: when the window fills up — compaction

Trimming helps, but long sessions still grow. When the transcript approaches the window's edge, Pi performs compaction: it asks the model to summarize the older part of the conversation, then swaps the originals for the summary.

The trigger is a simple inequality:

contextTokens  >  contextWindow − reserveTokens     (reserve defaults to 16,384)

That reserve is breathing room for the model's reply — you never fill a window to the brim.

The interesting part is where to cut. Pi walks backwards from the newest message, accumulating tokens until it has saved about 20k (configurable) of recent history — because recent context is what the model is actively working with. Everything older than that cut point becomes summary material.

And not a free-form summary — Pi asks for a structured one, with fixed sections: the goal, constraints, progress (done / in progress / blocked), key decisions, next steps, and critical context. It also appends running lists of files read and modified, accumulated across every compaction. Run three compactions deep into a session and Pi still knows every file it has ever touched.

what the model sees after compaction:

┌────────┬──────────────┬───────────────────────────────────┐
│ system │   summary    │ recent messages, kept verbatim    │
│ prompt │ (the past,   │ (the working present)             │
│        │  compressed) │                                   │
└────────┴──────────────┴───────────────────────────────────┘

The raw messages aren't deleted anywhere — remember, sessions are append-only — they're simply not sent anymore. Compaction is a view, not a shredder.

One last detail we like: these one-off summarization calls use fresh routing IDs and skip prompt-cache writes where the provider allows — a small courtesy that says "this prompt will never be reused, don't pollute the cache with it."

The plumbing underneath: two kinds of messages

A peek at the machinery, kept short. Inside Pi, the transcript holds AgentMessages — a flexible family that includes the three things LLMs understand (user, assistant, toolResult) plus app-specific types: compaction summaries, branch summaries, UI-only records.

But the LLM only speaks those three standard types. So right before every model call, a translator function (convertToLlm) converts the rich internal format into the strict external one. Rich on the inside, strict at the door — the same pattern as everything else in Pi.

This is also how tricks like "the user ran a command whose output the model should never see" work: the record exists for the UI, and the translator simply doesn't translate it.

The takeaway

Context engineering sounds grand, but in Pi it's three homely habits:

Don't send what you don't need (trim outputs, index skills instead of embedding them).
Send what you do need, well-organized (layered context files, structured prompts).
When the past gets heavy, summarize it — never just amputate it.

A forgetful model plus disciplined note-taking is surprisingly close to memory.

Sources & further reading: Compaction & Branch Summarization · pi-agent-core message flow