Skip to content

feat: filter mobile status/nav bars and dev overlay; viewport from snapshot - #67

Draft
cryptotavares wants to merge 1 commit into
cryptotavares/mmcli-p1-11-exact-refsfrom
cryptotavares/mmcli-p1-13-mobile-snapshot
Draft

cryptotavares wants to merge 1 commit into
cryptotavares/mmcli-p1-11-exact-refsfrom
cryptotavares/mmcli-p1-13-mobile-snapshot

Conversation

@cryptotavares

Copy link
Copy Markdown
Collaborator

Description

Mobile observations carried noise and made more device calls than needed:

  • Android status bar: on Android the status bar is a separate top-level window with 22 com.android.systemui:* ids. The clock, signal and battery change between snapshots, so they made up most of the 35 login testIds, and unchanged screens never produced an empty diff.
  • Expo dev menu button: the button (gearshape.fill + "Tools") showed up as an app element on both platforms.
  • Window size: every testId read made an extra getWindowSize() device call for the visibility check.

Changes:

  • filterAppHierarchy (new src/platform/mobile-hierarchy.ts) runs once per snapshot. The a11y tree, testIds and generate_locators are all built from its output, so they agree.
    • Android: a top-level window is dropped only if it is shaped like a bar (full width or full height, at most 15 % thick) and all its ids are com.android.systemui:*. That covers the status bar and nav bar. Other system windows (biometric prompt, consent and permission dialogs, keyboard) stay. In a flat hierarchy, only status/nav bar ids are dropped.
    • Both platforms: the Expo dev menu button is dropped. The new MobilePlatformDriver option excludeDevOverlay (default true; type MobilePlatformDriverOptions is exported) turns this off.
  • Viewport from the snapshot: iOS uses the root frame and Android the extent of the top-level windows. getWindowSize() is now only a cached fallback.
  • Mobile settle:
    • the time budget now includes the first capture;
    • an unchanged screen is not captured again when the first capture already took ≥ 1 s (MOBILE_SETTLE_SLOW_CAPTURE_MS); on Android one capture is an am instrument run;
    • busy screens still retry.
  • Tests confirm that observations, describe_screen, list_testids and accessibility_snapshot take one snapshot each (P1-08's captureScreen).

Part of the mm CLI v2 effort: ticket P1-13 (Phase 1).
Stacked on #66 (review that first).

Breaking changes

None. The constructor option is optional. Mobile testIds and trees no longer contain Android status/nav bar windows or the Expo dev menu button.

Consumer impact

Consumer Change needed Done on branch
Extension test/e2e/playwright/llm-workflow/ None (mobile only) n/a
Mobile tests/llm-workflow/ None. Its agent-device snapshot output on Android (pixel frames, full resource-ids) is handled. n/a

Proof

Tested with yalc builds in Mobile (dev client) on the iOS simulator and the Android emulator. Before = #66 build, after = this branch. Click and benchmark rows are 5 runs each; host load was 5–14.

Check Before (#66) After (3cc8aa9)
Android login testIds / com.android.systemui 35 / 22 13 / 0
Android wallet describe-screen root windows / systemui nodes 2 / 22 1 / 0
Android generate-locators on login – 13 locators, 0 systemui, 0 dev menu
Android click observationMs p50 / p95 963 / 1048 891 / 914 (−7 % / −13 %)
Android unchanged-screen click: empty diffs 0 of 5 1–2 of 5 (7.8 KB)
iOS login testIds 10 (incl. gearshape.fill) 9
iOS click observationMs p50 / p95 1170 / 1232 972 / 1005 (−17 %)
iOS list-testids / describe-screen toolMs p50 1162 / 1516 895 / 1284
iOS idb describe (window size) per click / describe / list 5 / 2 / 2 4 / 1 / 1
Device snapshots per click / describe / list (both platforms) 3 / 1 / 1 3 / 1 / 1

AC3 ("Android click observationMs roughly halves") is not met, and core cannot meet it.

  • Halving assumed 2 snapshots per observation. P1-08 already brought that down to 1.
  • Each click takes 3 device snapshots: 2 inside the backend tap (waitForElement + tapElement, in device-mcp / the Mobile agent-device helper) and 1 for the observation.
  • Removing the tap's snapshots needs a change in device-mcp or the consumer, for example tapping at coordinates already in the last observation. That is listed as a follow-up.

Still noisy on Android:

  • Some unchanged-screen diffs are still 30–135 KB, because the app keeps mounting zero-height ViewGroups. P1-14 (drop layout-only nodes) covers this.
  • On iOS, 2–4 full-screen Toolbar / FloatingBarHostingView nodes remain. They are most likely UIKit's floating bar per navigation controller, not the dev launcher. They are kept for now, pending a check on a release build.
Unit tests
Test Files  52 passed (52)
     Tests  1723 passed (1723)
Type Errors no errors
All files | 97.49 | 94.79 | 96.69 | 97.63
yarn lint → exit 0

New tests:

  • mobile-hierarchy.test.ts:
    • bar-window rule, plus a biometric prompt window that is kept;
    • flat-hierarchy fallback, dev overlay and the excludeDevOverlay option, empty-container pruning;
    • viewport from the root frame or Android extent, including an off-origin window and a status-bar-only snapshot;
    • a real Android login snapshot fixture (src/platform/__fixtures__/, excluded from the build), which reduces to 1 root and 0 systemui testIds.
  • Driver: 1 snapshot per call, getWindowSize at most once, generateLocators filtered.
  • Server: 1 snapshot per observation; the settle budget includes the first capture; slow first capture → no unchanged-screen retry, busy screens still retry.

@cryptotavares
cryptotavares added this pull request to stack #63 October 25, 2026 09:32
@cryptotavares
cryptotavares force-pushed the cryptotavares/mmcli-p1-13-mobile-snapshot branch from 3cc8aa9 to 52e4d72 Compare October 25, 2026 09:39
@cryptotavares
cryptotavares force-pushed the cryptotavares/mmcli-p1-13-mobile-snapshot branch from 52e4d72 to 9c6fb65 Compare October 25, 2026 09:47
…apshot

Ticket P1-13.

- filterAppHierarchy (new src/platform/mobile-hierarchy.ts) runs once per
  snapshot, before the a11y tree, testIds and generated locators are built,
  so all of them agree:
  - Android: drop top-level windows shaped like a bar (full width or
    height, at most 15 % thick) whose ids are all `com.android.systemui:*`
    (status bar: clock, signal, battery change on every snapshot; nav bar).
    Other system windows (biometric prompt, consent dialogs, permission
    dialogs, keyboard) stay. Flat hierarchies: drop only status/nav bar ids.
  - iOS / Android: drop the Expo dev menu button (`gearshape.fill` plus its
    "Tools" label). `MobilePlatformDriver` option `excludeDevOverlay`
    (default true, new exported `MobilePlatformDriverOptions`).
- The testId viewport comes from the snapshot (iOS root frame, Android
  extent of the top-level windows); getWindowSize() is called (and cached)
  only when the snapshot has none. Before, every testId read made one extra
  device call.
- Mobile settle: the time budget includes the first capture, and an
  unchanged screen is not re-captured when the first capture already took
  >= 1 s (MOBILE_SETTLE_SLOW_CAPTURE_MS). Busy screens still retry.
- Observations, describe_screen, list_testids and accessibility_snapshot
  take one snapshot each (P1-08 captureScreen); tests lock that in.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant