AML Cycle 2: AutoMem adapter readiness (opens 2026-09-20)

[jack-arturo]

AML Cycle 2 readiness (Cycle 2 opens 2026-09-20)

Track AutoMem's entry in the Agent Memory Leaderboard Cycle 2. AML owns answering and scoring. We provide a synchronous Add/Search API that stores messages and returns ranked memory evidence.

Adapter

runners/aml/ (PR linked below). It implements the three AML endpoints:

• POST /add writes every message to AutoMem (POST /memory) before echoing request_id/user_id/session_id. It is idempotent per request_id.
• POST /search wraps AutoMem GET /recall. It returns AML's data evidence capped at top_k and isolates each exact user_id with a hashed tag. The recall call pins tag_match=exact and disables relation/entity expansion and scope fallback.
• GET /health for AML's unauthenticated probe.

It ships with a Dockerfile and contract tests. An earlier version passed a real-AutoMem Add → Search → isolation smoke.

The public AML smoke can't run yet: it needs an Eval Key issued after Cycle 2 opens.

Routes

AML accepts either:

• a hosted, publicly reachable synchronous Add/Search API with an issued Eval Key; or
• the academic code route: a public repo with Docker startup instructions for maintainer deployment.

Remaining steps

1. Owner picks the evaluation division and route, freezes the AutoMem version under test, and confirms the public deployment target.
2. Deploy the adapter over HTTPS with a dedicated AML_ADAPTER_API_KEY (Memory System Key). AutoMem credentials go in as deploy secrets.
3. Set up deletion of aml-evaluation records within 30 days of a run. The adapter already sets t_invalid at 30 days, so records drop out of recall.
4. Once Cycle 2 opens, request evaluation access, bind the endpoints and key, and run AML's non-scored compatibility smoke.
5. Human owner sign-off is required before any AML evaluation request, smoke submission, Full run, or publication request.

References: AML rules, AML public repository.

Moved here from a private tracker, where it was filed in the wrong repo.

[jack-arturo]

Cycle 2 gap analysis — Streaming + Coding (2026-09-19)

Sources reviewed

• AML Cycle 2 rules and Add/Search API guide — Streaming writes “frozen incremental session or task units”; the platform may Search the same question at multiple increments. Add requests contain a unique request_id, ordered messages, user_id, and session_id; an HTTP 200 is allowed only after all writes are persisted and immediately searchable. The platform retries a logical Add with the same request_id and payload.
• AML public repository — Coding Memory measures reuse of prior engineering experience; its public description lists 12 repositories, 150 base engineering tasks, and relevant/noisy historical conditions. The official rules identify CAMBench Coding as the current scored suite: 150 tasks × relevant/noisy = 300 scored attempts.
• Cycle-2 announcement.

Streaming-memory finding

No separate endpoint or “partial Add” schema is required. Streaming is repeated, ordinary synchronous POST /add calls for incremental units, with POST /search potentially interleaved between them. The adapter already accepts independent Add chunks and returns only after AutoMem /memory writes complete, so it is the right shape.

Two low-risk correctness gaps were found and fixed locally in commit 2c72f45:

1. The adapter used Promise.all for a request’s messages, which could persist them out of source order. It now writes them sequentially.
2. Its completed-request_id cache had a race: two concurrent retries could both observe no completed entry and write duplicates. It now coalesces same-body in-flight retries; a different-body replay remains HTTP 409.

New contract tests cover delayed-first-message ordering and concurrent same-request_id retries. node --test runners/aml/adapter.test.mjs passes (7/7). No deployment, Eval Key request, credentials, or AML smoke/full evaluation was performed.

Remaining deployment gate: deduplication is process-local. Before a real multi-replica deployment, use a durable shared idempotency store or load-balancer affinity/single writer for same-request_id retries. This is an operational correctness requirement, not a reason to invent a new Streaming protocol.

Coding-track finding and recommendation

Go: Textual track only for Cycle 2, conditional on the remaining deployment gate. No-go: Coding track this cycle.

The generic API envelope can carry Coding strings, but AutoMem’s current adapter is a text/graph evidence retriever. It does not provide a code/repository-aware ingestion model, task-history representation, or retrieval/reranking strategy tailored to CAMBench’s same-repository relevant-vs-noisy engineering histories. Entering Coding now would be technically contract-shaped but not a credible, purpose-built evaluation. Keep the Cycle-2 scope focused on Textual (including Streaming); consider Coding only after a separate, benchmark-driven design and local relevant/noisy validation.

The owner-sign-off gates already listed in this issue remain unchanged: do not deploy, request access, submit Smoke/Full, or publish without explicit approval.

[autojack-bot]

AML Cycle 2 has opened today (2026-09-20) — confirmed, readiness needs an update pass

Scout run confirms via agentmemoryleaderboard.ai and independent field coverage (Superpower Daily) that Cycle 2 is live today with scope beyond the original Add/Search contract PR #29 was built against:

• Streaming Memory is a new component in the textual track — messages arrive incrementally rather than in batch, which changes assumptions about idempotency/ordering that runners/aml/ should re-verify.
• A coding track now evaluates 150 software-engineering tasks under relevant-history and noisy-history conditions — out of scope for our current adapter, worth an explicit opt-out decision rather than silence.
• A multimodal track also launches this cycle — likewise needs an explicit scope decision.
• Rules reconfirmed: user isolation, immediate searchability after synchronous Add, and idempotent request handling for retries — all already covered by PR feat(aml): AML Cycle 2 Add/Search adapter for AutoMem #29's design (tag_match=exact, request_id idempotency, 409 on replay).

PR #29 (feat/aml-cycle2-adapter) is still open/unmerged and was written before these Cycle 2 specifics were public. Recommend before requesting an Eval Key:

1. Merge feat(aml): AML Cycle 2 Add/Search adapter for AutoMem #29 once CI is green (tests already show 5/5 local pass).
2. Confirm scope: textual track only (with Streaming Memory support) vs. also entering coding/multimodal.
3. If entering textual only, verify /add behaves correctly for incremental/streamed message delivery, not just batch.
4. Re-affirm human owner sign-off is still required before any evaluation request, smoke, Full run, or publication — this comment does not authorize that step.

Sources: https://agentmemoryleaderboard.ai/, https://superpowerdaily.com/posts/agent-memory-challenge-will-open-a-shared-test-for-long-running-ai-agents, #29

[GiulioDER]

Hi Jack,

I found your Cycle 2 readiness notes here while looking for other AML participants. I am also doing the same Textual track, and my Full run is in progress now.

A few things I ran into that may save you time:

The platform retries a failed Add with the same request_id, many times, so a deterministic validation error (an empty message, an unexpected field) never clears. I relaxed my validation to return 422 only for input that cannot be stored at all.
Some Adds are large, and on my side the biggest took minutes before I optimised them. It is worth timing your slowest Add before the Full.
The organisers have not answered any issue on the public repo, and some email to them has gone unanswered too, so I would not count on clarifications arriving before your run.

Would you be open to comparing notes as the cycle goes on? I am happy to share anything else I find.

Giulio

[autojack-bot]

Recurring finding: review/merge throughput, not benchmark sourcing, is the bottleneck (Scout run 2026-09-27)

Posting here (most-recently-touched tracking issue) since there's no dedicated backlog issue yet — happy to move to a new issue if preferred.

Evidence: automem-evals currently has 11 open PRs and 29 open issues. Concretely:

• feat(aml): AML Cycle 2 Add/Search adapter for AutoMem #29 (AML Cycle 2 adapter) — open since 2026-09-18, gated on owner sign-off per AML Cycle 2: AutoMem adapter readiness (opens 2026-09-20) #28, no update in 9 days
• feat(mem2actbench): port AutoMem evidence-retrieval pilot #34 (Mem2ActBench pilot, full 400-task run: 77.81% evidence F1) — open since 2026-09-18, unreviewed
• [preserved] Repo: verygoodplugins/automem-evals. Implement the AMA-Bench AutoMem adapter tra #41 (AMA-Bench adapter) — draft, preserved-on-failure state since 2026-09-24/25: "Not reviewed. Mark ready for review to request Codex, or close to discard" — still undecided
• feat(bench): add DolphinBench AutoMem protocol smoke #46 (DolphinBench protocol smoke) — babysit:ready (Codex cleared, human may merge) since 2026-09-25, still unmerged 2 days later
• plus MemoryArena/Memora-FAMA PRs referenced in the 2026-09-25 scout run, also unmerged at last check

This is the fourth consecutive scheduled AutoMem Opportunity Scout run (2026-08-26, 2026-09-19, 2026-09-25, 2026-09-27) to observe this same pattern: sourcing keeps producing real, evidence-backed pilots faster than they get triaged/merged. Adding another new benchmark this cycle would make the pile worse, not better.

Recommended next action (owner decision needed):

1. A merge/triage sweep of the babysit:ready and stale draft PRs above (feat(aml): AML Cycle 2 Add/Search adapter for AutoMem #29, feat(mem2actbench): port AutoMem evidence-retrieval pilot #34, [preserved] Repo: verygoodplugins/automem-evals. Implement the AMA-Bench AutoMem adapter tra #41, feat(bench): add DolphinBench AutoMem protocol smoke #46 at minimum) — either merge, request changes, or close-and-discard each.
2. Consider whether automem-evals's babysit policy should auto-merge on babysit:ready for this repo (low production risk — it's an evals sandbox, not automem's own server) to stop the queue from growing unbounded.

No frontier action beyond this comment — closing the loop here rather than delegating a fifth benchmark-sourcing task into an already-backlogged queue.