(feat): Agent Component: summarize older chat history instead of sending the last 100 messages verbatim #15799

Feature Request

Feature Request

Problem scenario

A support assistant is built with the Agent component, using the default n_messages = 100.

1. In the first message, the user says: "I'm working on project PRJ-123 on the EU cluster."
2. Over the next hour they troubleshoot together: 60 exchanges, so 120 stored messages. Along the way the user pastes a long error log.
3. They then ask: "Which cluster were we on again?"

What happens today:

• The first message is now outside the 100-message window, so the Agent no longer has PRJ-123 or "EU cluster". It answers wrongly or asks again.
• Every model call in that hour carried up to 100 raw messages, the pasted log included. A tool-using turn makes several model calls, so one answer can cost well over 100k input tokens.
• Setting n_messages to 10 brings the cost down, but then the project and cluster are forgotten after five exchanges.
• Attaching a Memory Base as a tool doesn't close the gap. It ingests once every 50 runs, so with N = 10 most of the last 50 exchanges are neither in the window nor in the Memory Base. Even for older messages, the model only gets them back if it decides to search and writes a query that finds them.

What's needed is to keep the key information (project, cluster, decisions) for the whole session and send only a few recent messages verbatim.

Proposal

Add an opt-in chat history mode to the Agent component that keeps a rolling summary of older messages plus a small window of recent messages, instead of loading the last n_messages (default 100) rows verbatim on every turn.

The Agent's existing Number of Chat History Messages (n_messages) setting stays as it is. One new advanced switch is added, Summarize Older Messages, off by default:

• Off: today's behaviour. Existing flows don't change.
• On:
  • The last N messages are still sent verbatim.
  • Messages older than N are are summerized and only key messae are remembered. Once 20 of them have built up, they're folded into a per-session summary, at one extra model call per fold.
  • The summary keeps pinned keys (a new optional input, for example project_id, customer), which the summarizer may update but never drops, and a bullet summary of up to 300 words.
  • Since older messages now live on in the summary, N can be lowered (for example to 10) without losing the start of the session. The summary covers everything older than N, whether or not a Memory Base has ingested it yet.
  • The Memory Base tool keeps its role: on-demand search for specific older details and for recall across sessions. The key facts no longer depend on it.
  • The summary reaches the model through the same channel as today's history. get_memory_data() returns [summary message, *recent messages], and the summary is one user-role message ("Here is a summary of the conversation to date: …"), the same convention as LangChain's SummarizationMiddleware. The system prompt isn't changed, and the summary isn't stored as a chat message.

Storage and safety:

• A new session_memory_state table, one row per (flow_id, user_id, session_id, context_id), the same scope chat-history reads already use. A (timestamp, id) cursor records the last message folded in.
• Message rows aren't modified. Summaries don't show up in the Playground, in raw-mode history or in Memory Base ingestion.
• The row is deleted together with the session through the existing session-delete endpoints.
• Concurrent runs on one session use optimistic versioning. The first save wins and the other run reuses its result, with no locks.
• If the summary call fails, the turn still runs with the previous summary.
• Without a Langflow database (standalone lfx), the mode falls back to last-N.

Motivation

Today the model gets memory through two paths:

• (A) the last n_messages stored messages, on every turn;
• (B) a Memory Base search, when the model calls the Memory Base tool. It returns up to 5 chunks, from the same session by default, and ingestion runs once per 50 runs.

That combination has these problems:

1. Every model call carries the last 100 stored messages verbatim. n_messages defaults to 100. get_memory_data() → aget_agent_chat_history() fetches the newest N rows and build_initial_messages() sends each one as a Human or AI message. Once a session reaches 100 rows, every call pays for all of them. At around 300 tokens per message that's about 30k tokens, repeated on each model call of a tool-using turn.
2. The limit counts rows and ignores size and importance. A pasted 5,000-word document and "ok" each count as one row, and one exchange is two rows. The window can't be sized to a context budget.
3. Older messages are dropped silently. Message N+1 falls out of the window and no summary replaces it. Facts stated early in the session, such as the project, customer or a decision, are lost once the session passes N messages.
4. There's no session-level state. MessageTable is the only session store, so nothing can hold a handful of key facts for the whole session.
5. The one setting trades cost against memory. Lowering N makes problem 3 worse, raising it makes problem 1 worse. In the Playground, a request with no session id uses the flow id as its session, so every run of a flow shares one growing session and both problems show up quickly.
6. Messages between the window and the last Memory Base ingestion are invisible. Path A covers the last N messages, and path B covers only what has been ingested, about every 100 messages. Lower N to save tokens and the messages in between reach the model through neither path.
7. Key facts depend on the model choosing to search. Facts that apply to every turn, such as the project or "keep answers short", only come back through path B if the model calls the tool with a query that matches them. Each search adds a model call, and its results ride along on every remaining call of the turn.

Related: #9973 (memory controls removed from the Agent) and #15685 (system prompt stability for prompt caching). The system prompt stays identical across turns, and user-derived summary text is never given system-prompt authority.

Your Contribution

Potential implementation, in the order it would be built:

1. Storage. A new table session_memory_state with columns flow_id, user_id, session_id, context_id (unique together), summary, pinned (JSON), covered_until_message_id, covered_until_timestamp, absorbed_count, summary_model, version and timestamps. It comes with an EXPAND Alembic migration that only creates the table.

2. Data access in langflow/session_memory.py, re-exported from langflow.memory:

   • aget_session_memory_state();
   • asave_session_memory_state(expected_version=…), which inserts when the version is 0 and otherwise does a compare-and-swap update on version. It returns False when another run saved first;
   • adelete_session_memory_states() for session deletes.

   All three go through the same _message_scope() flow/owner check as aget_messages(). A new aget_messages_after() reads messages after a (timestamp, id) cursor, ordered the same way as Memory Base ingestion.

3. lfx dispatch. The same functions are added to lfx.memory and its stubs. The stubs report the feature as unsupported, so standalone lfx falls back to last-N.

4. Compressor in lfx/base/memory/session_compressor.py, storage-agnostic so it can be unit-tested. Each turn it:

   1. loads the state and the last N messages (n_messages), excluding the current input;
   2. loads the messages after the cursor;
   3. if at least 20 of them are older than the last N, makes one model call with the previous key facts, the summary and the batch (at most 60 messages, each truncated to 4,000 characters), asking for JSON {"pinned": {...}, "summary": "..."};
   4. merges pinned keys (an omitted key keeps its value, null clears it, keys that weren't configured are ignored), clips the summary to the word budget, and saves with compare-and-swap. If the save loses the race, it reloads the winning state;
   5. if the model call fails, keeps the previous state so the turn still runs.

5. Agent. Two new advanced inputs: summarize_older_messages (a boolean, off by default) and pinned_keys (shown when the switch is on). n_messages keeps its current meaning, and n_messages = 0 still turns memory off. When the switch is on, get_memory_data() returns [summary message, *last N messages], where the summary is a user-role Message, and they go through the existing chat_history → build_initial_messages() path. Folding uses the Agent's model with streaming off.

6. Cleanup. The monitor session-delete endpoints also delete the session's state row, next to the existing Memory Base purge.

7. Tests:

   • compressor unit tests with a fake store and LLM: threshold, cursor, pinned-key merge, model failure, version conflict, catch-up on long sessions;
   • backend tests: insert and compare-and-swap, scope isolation, cursor reads, purge;
   • migration upgrade and downgrade.

Questions to maintainer

1. Should folding run inline before the agent call, or as a background task after the run? Inline keeps the summary current but adds latency to one turn in every 20 messages.
2. Should there be a separate summary model input, or should it reuse the Agent's model?
3. Should the fold batch (20) and the summary word budget (300) be exposed as settings, or stay internal defaults?
4. Should the Message History component get the same switch in a follow-up?