feat: cancel timed-out tasks; stable refs that late writes cannot overwrite

README.md

@@ -366,9 +366,17 @@ type ToolContext = {
   get refMap(): Map<string, string>; // lazy — returns empty map if no session
   workflowContext: WorkflowContext;
   toolRegistry: Map<string, ToolFunction<unknown, unknown>>;
+  driver?: IPlatformDriver;
+  signal?: AbortSignal; // aborted when the request times out
+  setRefMap?: (
+    map: Map<string, string>,
+    nodes?: A11yNodeTrimmed[],
+  ) => { refMap: Map<string, string>; nodes?: A11yNodeTrimmed[] };
 };
 ```
 
+Tools check `signal?.aborted` between steps and stop. Tools that build refs call `context.setRefMap(map, nodes)` (fall back to `sessionManager.setRefMap` when it is absent) and return the `nodes` it returns: the daemon may renumber refs so elements the agent already holds keep their refs. The daemon's `RefStore` drops a write from a request that timed out or was replaced by a newer one, so a cancelled call cannot overwrite the refs of later calls. In the daemon, `context.sessionManager.setRefMap` goes through the same guard (without renumbering).
+
 The daemon routes `POST /tool/:name` requests through the registry, applies Zod validation on inputs, executes the tool through the request queue, and captures observations (test IDs, a11y snapshot) according to the call's `observe` level (see [Observations](#observations)).
 
 **Registered tools:**
@@ -443,6 +451,10 @@ mm type e3 "hello"    → types into the element mapped to e3
 
 This accessibility-first approach provides reliable element targeting that survives minor UI changes.
 
+Refs stay valid while the element stays on screen. Each observation, `describe_screen` and `accessibility_snapshot` is renumbered against the last tree the agent received. Elements are matched by key (browser: role + name + testId; mobile: testId, else role + name). A key found once in each tree keeps its ref. Same key: matched in screen order (the k-th element with that key keeps the ref of the k-th element with that key in the last tree), so the many unnamed containers of a mobile screen keep their refs and an unchanged screen diffs as unchanged. When an element is inserted in the middle of a group that shares a key, the later elements of that group shift by one ref (each takes its predecessor's ref; the last gets a fresh one); elements with other keys are not affected. Other elements get fresh refs above the highest ref handed out so far, so an element that leaves the screen and comes back gets a fresh ref, and a removed ref is never reused for another element within a session (numbering restarts at `launch` and `cleanup`). Ref numbers can therefore grow within a session: new elements, options and collapsed runs get fresh refs, and do not reuse lower numbers. Options (`role: option`) and elements inside collapsed runs (`… 4 more`) are not stabilized: they always get fresh, contiguous refs so ranges (`e2–e56`) stay valid. If a tool set refs during a call (for example `describe_screen` with `observe`), the observation keeps them and adds its new refs, so every ref in the response resolves. `list_testids` does not change refs.
+
+On mobile, after a mutating or batch call, the daemon may take the snapshot again after 300 ms: while it shows a busy element (a label ending in `, busy`, an Android `ProgressBar`, or an iOS `ActivityIndicator` / `ProgressIndicator` / `UIActivityIndicatorView`), and at most once when it still equals the previous screen. In total at most 3 extra snapshots and about 2 s. The test IDs come from the same snapshot as the tree. `describe_screen` on mobile also takes one snapshot for both.
+
 ### Session Metadata
 
 Consumers can persist launch metadata with `writeSessionMetadata(artifactsDir, metadata)`. It writes `<artifactsDir>/<sessionId>/session.json` (for example `test-artifacts/<sessionId>/session.json` with `artifactsDir: 'test-artifacts'`) and returns the file path.
@@ -541,6 +553,8 @@ For `run_steps`, the top-level `observe` sets the level of the one observation c
 
 Observations contain `testIds` and `a11y` only. They do not include app state; call `get_state` for that.
 
+A call that times out is cancelled: its signal is aborted, it stops at its next step, it collects no observation, and its refs are dropped.
+
 ## CLI Reference
 
 The `mm` CLI provides a unified interface for agents and developers. All commands communicate with the daemon over HTTP — the daemon is auto-started on `mm launch` if not already running.
@@ -795,6 +809,7 @@ Tool errors are classified into specific error codes for structured handling:
 | `MM_PORT_IN_USE`                    | Port already in use                                                                                     |
 | **System**                          |                                                                                                         |
 | `MM_UNKNOWN_TOOL`                   | Unknown tool name                                                                                       |
+| `MM_REQUEST_ABORTED`                | Internal: a cancelled (timed-out) call stopped; the caller gets the timeout error                       |
 | `MM_INTERNAL_ERROR`                 | Internal server error                                                                                   |
 | `MM_BATCH_TIMEOUT`                  | `batchTimeoutMs` deadline exceeded in run_steps                                                         |
 | `MM_CDP_BLOCKED`                    | CDP method is blocked (destructive to session); on mobile the Hermes blocklist maps here too            |

SKILL.md

@@ -18,7 +18,7 @@ mm cleanup --shutdown      # 5. Clean up when done
 
 **Critical rules:**
 
-- **Always `describe-screen` before interacting.** Refs like `e1`, `e2` are ephemeral — they change after every action.
+- **Always `describe-screen` before interacting.** Refs like `e1`, `e2` are short-lived: a ref stays valid while its element stays on screen. New elements get new refs, an element that leaves and comes back gets a new ref, and a removed ref is never reused for another element (until `launch`/`cleanup`).
 - **Always `describe-screen` after interacting** — OR use inline `observations` from mutating tool responses. Mutating tools (click, type, navigate, etc.) return an `observations` object with fresh `testIds` and `a11y` refs. You can use these refs directly for the next interaction without calling `describe-screen`. Call `describe-screen` when you need the full tree or screenshots, and `get-state` when you need app state (observations carry none).
 - **One target per command.** Specify exactly ONE of: a11y ref (`e5`), testId, or CSS selector.
 - **Errors are structured.** Check the `error.code` field to decide recovery strategy (see Error Codes below).
@@ -85,7 +85,9 @@ After a mutating action, the response includes fresh screen data:
 }
 ```
 
-You can use the `ref` values from `observations.a11y.nodes` for the next interaction — no `describe-screen` needed. Note that refs in compacted observations may be summary nodes (e.g., `"55 options (refs e2–e56)"`) when there are 3+ options under a combobox or listbox.
+You can use the `ref` values from `observations.a11y.nodes` for the next interaction — no `describe-screen` needed. Refs stay valid while the element stays on screen: an unchanged element keeps the ref you already hold (from `describe-screen` or an earlier observation), even when other elements appear before it. A new element gets a new, higher ref; an element whose name changed, or that left the screen and came back, gets a new ref, so ref numbers grow during a session. Elements with the same key (for example unnamed containers on mobile) are matched in screen order: if one is inserted among them, the later ones of that group shift by one ref. Note that refs in compacted observations may be summary nodes (e.g., `"55 options (refs e2–e56)"`) when there are 3+ options under a combobox or listbox.
+
+**Timeouts cancel the call:** when a call times out (`Tool execution timed out`), it is cancelled. It stops at its next step and cannot overwrite the refs of later calls. The action itself may still have happened, so run `describe-screen` before retrying.
 
 **Quick reference:**
 

src/index.ts

@@ -10,6 +10,7 @@ export type {
   TypeActionResult,
   GetTextActionResult,
   PlatformScreenshotOptions,
+  ScreenCapture,
 } from './platform';
 export { PlaywrightPlatformDriver, MobilePlatformDriver } from './platform';
 

src/platform/index.ts

@@ -7,6 +7,7 @@ export type {
   PlatformScreenshotOptions,
   WithinScope,
   IPlatformDriver,
+  ScreenCapture,
 } from './types.js';
 
 export { PlaywrightPlatformDriver } from './playwright-driver.js';

src/platform/mobile-platform-driver.test.ts

@@ -71,6 +71,52 @@ describe('MobilePlatformDriver', () => {
     vi.restoreAllMocks();
   });
 
+  describe('captureScreen', () => {
+    it('returns the tree and the test IDs of one snapshot', async () => {
+      const backend = createMockBackend({
+        getWindowSize: vi.fn().mockResolvedValue({ width: 402, height: 874 }),
+        snapshot: vi
+          .fn()
+          .mockResolvedValueOnce({
+            platform: 'ios',
+            hierarchy: [
+              makeElement({ identifier: 'login', label: 'Unlock' }),
+              makeElement({
+                identifier: 'below',
+                type: 'StaticText',
+                frame: { x: 0, y: 2000, width: 10, height: 10 },
+              }),
+            ],
+            raw: '[]',
+            timestamp: Date.now(),
+          })
+          .mockResolvedValueOnce({
+            platform: 'ios',
+            hierarchy: [makeElement({ identifier: 'wallet' })],
+            raw: '[]',
+            timestamp: Date.now(),
+          }),
+      });
+      const driver = new MobilePlatformDriver(backend);
+
+      const screen = await driver.captureScreen();
+      const testIds = await screen.getTestIds();
+
+      expect(backend.snapshot).toHaveBeenCalledTimes(1);
+      expect(screen.nodes.map((node) => node.testId)).toStrictEqual([
+        'login',
+        'below',
+      ]);
+      expect(screen.refMap.get('e1')).toBe('identifier:login');
+      // Same items, order and visibility as getTestIds on that snapshot.
+      expect(testIds).toStrictEqual([
+        { testId: 'login', tag: 'Button', text: 'Unlock', visible: true },
+        { testId: 'below', tag: 'StaticText', text: undefined, visible: false },
+      ]);
+      expect(await screen.getTestIds(1)).toHaveLength(1);
+    });
+  });
+
   describe('getPlatform', () => {
     it('returns ios for ios backend', () => {
       const driver = new MobilePlatformDriver(createMockBackend());

src/platform/mobile-platform-driver.ts

@@ -22,6 +22,7 @@ import type {
   GetTextActionResult,
   PlatformScreenshotOptions,
   PlatformType,
+  ScreenCapture,
   WithinScope,
 } from './types.js';
 import type {
@@ -208,6 +209,33 @@ export class MobilePlatformDriver implements IPlatformDriver {
    */
   async getTestIds(limit?: number): Promise<TestIdItem[]> {
     const snapshot = await this.#backend.snapshot();
+    return this.#testIdsOf(snapshot.hierarchy, limit);
+  }
+
+  /**
+   * Takes one device snapshot and returns its accessibility tree and test
+   * IDs, so both describe the same screen.
+   *
+   * @returns The nodes, refMap and a test ID reader for the same snapshot.
+   */
+  async captureScreen(): Promise<ScreenCapture> {
+    const snapshot = await this.#backend.snapshot();
+    return {
+      ...normalizeSnapshot(snapshot.hierarchy),
+      getTestIds: async (limit?: number) =>
+        this.#testIdsOf(snapshot.hierarchy, limit),
+    };
+  }
+
+  /**
+   * @param hierarchy - The UIElement tree of a snapshot.
+   * @param limit - Maximum number of test IDs to return.
+   * @returns The test IDs, with visibility against the device viewport.
+   */
+  async #testIdsOf(
+    hierarchy: UIElement[],
+    limit?: number,
+  ): Promise<TestIdItem[]> {
     const items: TestIdItem[] = [];
     const max = limit ?? OBSERVATION_TESTID_LIMIT;
     let viewport: { width: number; height: number } | undefined;
@@ -217,7 +245,7 @@ export class MobilePlatformDriver implements IPlatformDriver {
       // Without a viewport we cannot judge visibility; report everything.
       viewport = undefined;
     }
-    collectTestIds(snapshot.hierarchy, items, max, viewport);
+    collectTestIds(hierarchy, items, max, viewport);
     return items;
   }
 

src/platform/types.ts

@@ -48,6 +48,21 @@ export type WithinScope = {
   value: string;
 };
 
+/**
+ * One snapshot of the screen: the accessibility tree, and the test IDs of the
+ * same snapshot.
+ */
+export type ScreenCapture = {
+  nodes: A11yNodeTrimmed[];
+  refMap: Map<string, string>;
+  /**
+   * Lists the test IDs of this snapshot (no new snapshot).
+   *
+   * @param limit - Maximum number of test IDs to return.
+   */
+  getTestIds(limit?: number): Promise<TestIdItem[]>;
+};
+
 export type IPlatformDriver = {
   click(
     targetType: TargetType,
@@ -88,6 +103,13 @@ export type IPlatformDriver = {
 
   getTestIds(limit?: number): Promise<TestIdItem[]>;
 
+  /**
+   * Takes one snapshot and returns both the accessibility tree and its test
+   * IDs, so they describe the same screen. Optional: without it, callers use
+   * `getAccessibilityTree` and `getTestIds` (two snapshots).
+   */
+  captureScreen?(): Promise<ScreenCapture>;
+
   screenshot(options: PlatformScreenshotOptions): Promise<ScreenshotResult>;
 
   getAppState(): Promise<ExtensionState>;