AutoMem vs PlugMem: knowledge-centric graph architecture comparison + eval run #26

Migrated from AutoHub issue #782. AutoHub is the private automation-hub tracker; this work belongs in verygoodplugins/automem-evals.

PlugMem comparison and benchmark work is AutoMem evaluation research and belongs with the benchmark/eval tracker rather than the AutoHub automation backlog.

Original source: https://github.com/verygoodplugins/autohub/issues/782

Background

PlugMem (Microsoft Research / TIMAN Group, UIUC — paper, code) is a task-agnostic plugin memory module for LLM agents that structures episodic memories as a propositional + prescriptive knowledge graph. It departs from entity-centric GraphRAG by using knowledge units as graph nodes rather than named entities or text chunks.

AutoMem uses a relation-typed graph with LEADS_TO, INVALIDATED_BY, EVOLVED_INTO, REINFORCES, and PART_OF edges — a structurally different approach that is well-suited to the same task-agnostic scenario PlugMem targets.

Why This Matters

PlugMem is evaluated on 3 heterogeneous benchmarks (code available on GitHub). AutoMem has not been evaluated on these same benchmarks, which means:

1. No direct architectural comparison exists in the literature
2. AutoMem's write-time edge quality (LEADS_TO, INVALIDATED_BY) has not been empirically compared to PlugMem's propositional knowledge approach
3. A published comparison positions AutoMem against the closest published MS Research competitor

Evaluation Matrix

Total: 28/30 — highest-scoring open lead this scout cycle.

Acceptance Criteria

• Clone PlugMem repo and document their benchmark suite (datasets, metrics, eval script entry points)
• Identify which PlugMem benchmarks overlap with AutoMem's current eval suite
• Run AutoMem against at least one PlugMem shared benchmark; record scores
• Produce a structured architectural comparison table:
  • Memory unit representation (knowledge vs entity vs relation)
  • Write-time graph construction approach
  • Retrieval/traversal mechanism
  • Cross-task generalization evidence
• Draft an architecture comparison note (markdown or blog draft) highlighting AutoMem differentiators:
  • INVALIDATED_BY / t_invalid for temporal memory drift
  • current_only=true guard vs PlugMem's prescriptive invalidation
  • AutoMem's expand_entities multi-hop traversal
• Optional: open outreach issue to TIMAN group (UIUC) with comparison results

PlugMem Key Details

• Graph type: Propositional (factual statements) + Prescriptive (behavioral rules)
• Benchmarks: 3 heterogeneous task sets — check repo for exact names
• Key gap vs AutoMem: PlugMem does not handle temporal invalidation (no equivalent to AutoMem's t_invalid)
• Key strength: Knowledge-unit nodes allow cross-task generalization without entity disambiguation

References

• Paper: https://www.microsoft.com/en-us/research/publication/plugmem-a-task-agnostic-plugin-memory-module-for-llm-agents/
• Code: https://github.com/TIMAN-group/PlugMem
• Related: AutoMem FAMA/Memora eval (#711), LongMemEval-V2 scout candidate

AutoHub Context

• Frontier candidate: opportunity:plugmem-architecture-comparison (priority 74)
• AutoMem components relevant: memory_store_memory, memory_recall_memory, memory_associate_memories, Falkor graph backend
• Output format: markdown architecture comparison doc in docs/research/ + eval score table