Skip to content

feat!: per-call observe level; no state in observations - #62

Draft
cryptotavares wants to merge 2 commits into
cryptotavares/mmcli-p1-05-observe-mutating-onlyfrom
cryptotavares/mmcli-p1-06-observe-level-no-state
Draft

cryptotavares wants to merge 2 commits into
cryptotavares/mmcli-p1-05-observe-mutating-onlyfrom
cryptotavares/mmcli-p1-06-observe-level-no-state

Conversation

@cryptotavares

Copy link
Copy Markdown
Collaborator

Description

Two observation problems, fixed together because they touch the same code (tickets P1-06 and P1-07):

  1. Callers can't choose the observation. An agent that chains clicks wants none (fast). An agent that just navigated wants the whole screen. A read sometimes wants one too. Before this PR the category decided, and run_steps had its own includeObservations switch.
  2. Observations carried app state, which was wrong on mobile and slow on browser. On mobile it was a stub with invented values (isUnlocked: true, currentScreen: "unknown", nulls). On browser it cost a getExtensionState call plus a 500 ms re-check loop whenever the screen was unknown.

Changes:

  • observe: 'none' | 'diff' | 'full' on every tool input. It is one shared observeLevelSchema, added to every toolSchemas entry with withObserve(). The daemon reads it and strips it (omitObserve) before the tool runs, so consumers' sessionManager.launch() never sees it.
    • Defaults: diff for mutating tools and run_steps, none for readonly, discovery and cdp.
    • full returns the whole (option-collapsed) tree without diffing, and becomes the new diff baseline.
  • CLI: a global --observe <level> flag, accepted anywhere on the command line. It is added to POST /tool/*, /launch and /cleanup bodies.
  • run_steps: includeObservations is removed and now fails validation with a hint to use observe. The single batch observation uses the top-level observe, else the last executed step's observe, else diff.
  • No state in observations. getExtensionState/getAppState and the 500 ms re-check are gone from the observation path. Use get_state for state.
  • Mobile getAppState() returns only what it knows: { isLoaded, extensionId, source: 'stub' }. ExtensionState fields other than isLoaded and extensionId are optional, and source: 'stub' marks partial state.
  • The animation-settle wait now runs for every category except readonly and discovery, so cdp --observe full, which may change the screen, also waits.

Part of the mm CLI v2 effort: tickets P1-06 + P1-07 (Phase 1), combined into one PR.
Stacked on #61 (review that first).

Breaking changes

  • run_steps.includeObservations is removed. Sending it is a validation error (includeObservations was removed; use observe). The 'failures' mode is gone.
  • Observations no longer contain state:
    • StepRecordObservation has no state field;
    • createDefaultObservation(testIds, nodes) drops its first parameter.
  • ExtensionState: currentUrl, isUnlocked, currentScreen, accountAddress, networkName, chainId and balance are optional. New field: source?: 'stub'.
  • MobilePlatformDriver.getAppState() returns { isLoaded, extensionId, source: 'stub' } only.
  • shouldCollectObservations and shouldIncludeObservationsInResponse are replaced by resolveObserveLevel(category, input, toolResult).
  • toolSchemas entries are now intersections that include observe. The individual *InputSchema exports are unchanged.

Consumer impact

Consumer Change needed Done on branch
Extension test/e2e/playwright/llm-workflow/ None. Its own ExtensionState with required fields still type-checks against the looser core type, and nothing reads observations.state or includeObservations. Jest: 144 passed. n/a
Mobile tests/llm-workflow/ The state snapshot capability stops inventing isUnlocked/currentScreen/null fields: with no session it reports {isLoaded:false, extensionId:'', source:'stub'}, and it omits currentScreen when no known testId matches. Jest: 232 passed. cryptotavares/mm-cli-v2-mobile 35fc72cc609

Follow-up, not in this PR: on mobile, core get_state calls the driver directly, so the Mobile capability's screen detection only runs on browser. Phase 2 mm state (P2-09) covers this.

Proof

Tested with a yalc build of this branch in the Extension (e2e build, headless) and Mobile (dev client on iOS sim and Android emulator), on wallet home after unlock. Before = #61 tip; its proof is reused, except the Extension click bench, which was run on both builds back to back.

Same command at each level (after; bytes / totalMs):

Platform none diff full
ext (click a home tab) 197 / 17 6760 / 581 9441 / 586
iOS (wait-for wallet-screen) 183 / 880 7495 / 2923 58865 / 2905
Android (click tab-bar-item-Wallet) 193 / 1794 7672 / 3671 1059191 / 3666
Check Result
click --observe none no observations, observationMs 0, on all 3 platforms
get-state --observe full observation with testIds + a11y only: no state, no diff
run-steps, top-level observe: none / last step observe: full no observations / full tree without diff (all 3 platforms)
run-steps with includeObservations exit 1: includeObservations was removed; use observe
observations.state in any of the 55 after outputs none
Mobile get-state, before → after isUnlocked: true, currentScreen: "unknown", currentUrl: "", 4 × null → {isLoaded, extensionId, source: "stub"}
ext click observationMs p50 (bench, 5 runs, back to back) 564 → 555 ms; bytes 7335 → 6760 (the state block)
ext smoke click-unlock (unlock → home) observationMs 71 → 15 ms (the re-check loop no longer runs while the screen is unknown)

The bench gain is small because on home the screen is already known, so the 500 ms re-check rarely ran. Most of the remaining ~550 ms is the animation wait plus the a11y snapshot. P1-10 targets the animation wait.

Transcripts (ext)
$ yarn mm click --testid unlock-submit --observe none
{
  "clicked": true,
  "target": "testId:unlock-submit",
  "meta": { "timing": { "queueMs": 0, "toolMs": 32, "observationMs": 0, "totalMs": 33 } }
}

$ yarn mm run-steps '{"steps":[{"tool":"get_state"}],"includeObservations":"none"}'
Error: includeObservations: includeObservations was removed; use observe
exit=1

$ yarn mm --observe bogus get-state
Error: --observe requires one of: none, diff, full
exit=1
Unit tests
Test Files  46 passed (46)
     Tests  1450 passed (1450)
Type Errors no errors
All files | 96.94 | 93.52 | 94.91 | 97.12
yarn lint → exit 0

New or changed:

  • resolveObserveLevel: category defaults, an explicit level wins, an invalid level is ignored, run_steps last executed step and failure fallback.
  • Endpoint tests: click with none; get_state with full/diff; cdp with full waits for animations; observe is not passed to the handler; /cleanup body.
  • No getExtensionState during observation (spy).
  • Mobile getAppState shape.
  • withObserve/omitObserve, includeObservations rejected, and the CLI --observe parsing and injection.

Callers can now choose the observation they get back, and observations no
longer carry app state (which was wrong on mobile and slow on browser).

- Every tool input accepts `observe: 'none' | 'diff' | 'full'` (shared
  observeLevelSchema, added to toolSchemas with withObserve()). The daemon
  reads it and strips it before the tool runs. Defaults: diff for mutating
  tools and run_steps, none for readonly/discovery/cdp. `full` returns the
  whole (option-collapsed) tree without diffing and becomes the new diff
  baseline.
- CLI: global `--observe <none|diff|full>` flag, added to POST /tool/*,
  /launch and /cleanup bodies.
- run_steps: `includeObservations` is removed. The batch observation uses the
  top-level `observe`, else the last executed step's `observe`, else diff.
- Observations no longer include `state`. The getExtensionState/getAppState
  calls and the 500 ms unknown-screen recheck loop are gone from the
  observation path. Use get_state for state.
- The mobile driver's getAppState reports only what it knows:
  { isLoaded, extensionId, source: 'stub' }. `source: 'stub'` marks partial
  state; fields not read from the app are omitted.

BREAKING CHANGE: `run_steps.includeObservations` is removed and now fails
validation with a hint to use `observe` (the 'failures' mode is gone).
MobilePlatformDriver.getAppState() now returns only
{ isLoaded, extensionId, source: 'stub' } (no isUnlocked, currentScreen,
currentUrl or null wallet fields). Observations no
longer contain `state`; StepRecordObservation has no `state` field and
createDefaultObservation(testIds, nodes) drops its state parameter.
ExtensionState fields other than isLoaded and extensionId are optional and a
`source?: 'stub'` field is added. shouldCollectObservations and
shouldIncludeObservationsInResponse are replaced by resolveObserveLevel.
toolSchemas entries are now intersections that include `observe`.
Comment thread src/server/create-server.ts Fixed
@cryptotavares

Copy link
Copy Markdown
Collaborator Author

Fixed CodeQL js/unvalidated-dynamic-method-call in 1bbfc70 (check the registry entry is a function before calling it).

@cryptotavares
cryptotavares force-pushed the cryptotavares/mmcli-p1-06-observe-level-no-state branch from 1bbfc70 to 2ee766a Compare October 24, 2026 01:32
CodeQL (js/unvalidated-dynamic-method-call) flagged the tool call because
the tool name comes from the request path. Look the tool up only when the
registry has the name, and check the entry is a function.
@cryptotavares
cryptotavares force-pushed the cryptotavares/mmcli-p1-06-observe-level-no-state branch from 2ee766a to 9f88039 Compare October 24, 2026 01:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants