agent memory: LongShortTermMemoryStorage

[yuyijiong]

This is a simple design of agent memory, consisting of core memory, long-term memory and short-term memory. There is no RAG, instead, we simply ask the agent itself to summarize and consolidate its history into memory blocks, and recall the details of the blocks when needed.

We implement LongShortTermMemoryStorage in autogen/beta/example_LSTMS.py. The usage example is in autogen/beta/long_short_term_memory_storage.py.

[CLAassistant]

All committers have signed the CLA.

[jsun-m]

@Lancetnik Hey Nikita, this is one of the research students I'm working with for implementing a memory module

@yuyijiong maybe we move the actual files under autogen/beta/strategies/memory

[Lancetnik]

@yuyijiong Hi! Sorry for the delay. I like your feature and the PR looks great. However, I feel that it's more like a plugin than an integrated part of the framework. This week, we want to create and release a policy for third-party plugins. I believe it will be called autogen.beta.contrib, with a migration to ag-community in the future, or something similar. Therefore, I would like to revisit this PR after the creation of this policy and merge it into a plugin. Would that be okay with you?

[jsun-m]

@yuyijiong
can you look to move the documentation under here?
https://github.com/ag2ai/ag2/tree/main/website/docs/beta

[yuyijiong]

@yuyijiong Hi! Sorry for the delay. I like your feature and the PR looks great. However, I feel that it's more like a plugin than an integrated part of the framework. This week, we want to create and release a policy for third-party plugins. I believe it will be called autogen.beta.contrib, with a migration to ag-community in the future, or something similar. Therefore, I would like to revisit this PR after the creation of this policy and merge it into a plugin. Would that be okay with you?

Yes. It's OK.

[yuyijiong]

Update since the first commit:

1. Improving prompts for summarization, emphasizing the importance of specific details such as time, place name, person name.
2. When compressing L1 blocks, we exclude messages of tool calls of memory_lookup, so that we would not compress the content of an old L1 block into a new one.
3. Evaluating this memory on LongMemEval-s (using gpt-5-mini). Since most benchmarks are actually just testing short-term memory, we only evaluate the short-term memory function. The results show it’s good at single-session tasks but weaker at others:

Overall:  304/500  (60.80%)

By question_type:
 knowledge-update:  46/78  (58.97%)
 multi-session:  78/133  (58.65%)
 single-session-assistant:  49/56  (87.50%)
 single-session-preference:  0/30  (0.00%)
 single-session-user:  58/70  (82.86%)
 temporal-reasoning:  73/133  (54.89%)

4. Add documentation about this memory system in ag2\website\docs\beta\memory.mdx

[marklysze]

Thanks for the LongShortTermMemoryStorage contribution — the core idea (core + long-term + short-term memory blocks with LLM-driven consolidation, no RAG) is interesting.

However, the PR has several issues that need to be fixed before review:

1. Unintended deletions in core files

The diff removes plugins: Iterable["Plugin"] from all Agent.__init__ overloads in autogen/beta/agent.py, deletes ~81 lines from anthropic/mappers.py, ~106 lines from gemini/mappers.py, and entirely removes autogen/beta/tools/builtin/skills.py and autogen/beta/tools/toolkits/skills/local_skills/loader.py. None of these changes are related to the memory feature — they appear to be regressions from a stale fork.

2. CI config modification

The PR modifies .github/workflows/pr-checks.yml (removing permissions: { contents: write, pull-requests: read } → permissions: {}). Feature PRs should not touch the CI configuration.

3. Limited CI coverage

Only 2 checks ran (labeler + CLA) rather than the full 19-check suite. This usually means the CI trigger didn't fire correctly.

Root cause: Your fork is likely behind upstream main. Please rebase your branch on the latest ag2ai/ag2 main, resolve conflicts carefully — keeping upstream's changes for all files outside the memory feature — and push again. The PR should only add:

• autogen/beta/strategies/memory/long_short_term_memory_storage.py (and related)
• Tests for the new storage
• Any docs/notebook

No modifications to agent.py, mapper files, CI config, or skills files.

[ferhimedamine]

This three-tier design resonates — I have been building something similar and the biggest lesson was that LLM-driven summarization alone is lossy in ways that are hard to predict. Specific details (entity names, timestamps, numeric values) get silently dropped during consolidation, which then breaks downstream recall on factual questions. What worked better for us is combining the summarization with a knowledge graph that extracts entities and relationships before compression happens, so the graph preserves the structured facts even after the prose summary loses them. The other thing worth considering is cross-agent memory: if agent A consolidates a memory block, agent B working on the same task should be able to recall from it via a shared namespace rather than re-ingesting the raw history. We implemented this with session-scoped namespaces — the JS SDK shows the pattern: https://github.com/Dakera-AI/dakera-js