Skip to content

feat: add meta.timing to every daemon response - #57

Draft
cryptotavares wants to merge 1 commit into
mainfrom
cryptotavares/mmcli-p1-01-meta-timing
Draft

cryptotavares wants to merge 1 commit into
mainfrom
cryptotavares/mmcli-p1-01-meta-timing

Conversation

@cryptotavares

Copy link
Copy Markdown
Collaborator

Description

Every speed or size claim in the mm CLI v2 work needs a measurement. Today only wall clock from outside the CLI can be measured, and it mixes yarn/node start-up, queue wait, the tool, observation collection and serialization. The server already computes a duration for the knowledge step (src/server/create-server.ts:377), but does not return it.

  • Every /tool/*, /launch and /cleanup response body now ends with meta: { timing: { queueMs, toolMs, observationMs, totalMs } }. This applies to success and error bodies: 404 unknown tool, 503, 400 validation, platform gate, 500 tool throw, and 500 from the express error middleware (malformed JSON).
    • queueMs: enqueue → task start. toolMs: the tool call. observationMs: observation collection (0 when skipped). totalMs: request in → response out. The start time is recorded before express.json, so body parsing is included.
    • All values are whole ms, ≥ 0, and totalMs ≥ queueMs + toolMs + observationMs. New RequestTimer in src/server/response-timing.ts uses the monotonic performance.now().
    • New public types: ToolResponseMeta, ToolResponseTiming (src/types/http.ts).
  • CLI: meta is printed as the last key of the JSON output. String results print unchanged, without meta. Arrays and primitives are wrapped as { result, meta }. Error exits do not print meta yet; structured CLI errors come in a later ticket.
  • scripts/bench.mjs (not shipped): runs mm commands N times against a live session and prints TSV with the median and p95 of wall clock, bytes and each meta.timing field.

Part of the mm CLI v2 effort: ticket P1-01 (Phase 1).
Base: main (bottom of the stack).

Note: run_steps already has per-step steps[].meta inside result. The new meta is top-level, next to ok and result, so they do not collide.

Breaking changes

None. The field is additive.

Consumer impact

Consumer Change needed Done on branch
Extension test/e2e/playwright/llm-workflow/ None (nothing parses the response body or CLI stdout strictly) n/a
Mobile tests/llm-workflow/ None n/a

Proof

Tested with a yalc build of this branch in the Extension (e2e test build 13.56.0, headless Chrome) and Mobile (dev client 8.19.0, iOS sim + Android emulator). Full proof: proof/p1-01/PROOF.md in the mm-cli-v2 workspace.

Check Before (0.12.0) After
meta.timing on JSON outputs (smoke, 3 platforms) absent 25/25 outputs, valid numbers
meta.timing on HTTP 404 / 400 / 500 (malformed JSON) absent present on all three
get-state bytes, Extension (pure cost of meta) 682 809 (pretty-printed)
bench.mjs --runs 5 on Extension and iOS n/a TSV, 0 failures

The new numbers already show where time goes (bench p50, live session). This is the baseline for the observation tickets:

Platform Command toolMs observationMs totalMs wall ms
ext cdp Runtime.evaluate 1+1 2 571 573 1072
ext get-state 7 584 592 1090
ext describe-screen 578 569 1147 1613
ios cdp Runtime.evaluate 1+1 10 4788 4797 6006
ios describe-screen 5202 4966 10298 11611
Transcripts
$ curl -X POST -d '{}' http://127.0.0.1:<port>/tool/nonexistent
{"ok":false,"error":{"code":"TOOL_NOT_FOUND","message":"Unknown tool: nonexistent"},"meta":{"timing":{"queueMs":0,"toolMs":0,"observationMs":0,"totalMs":0}}}
HTTP 404
$ curl -X POST -d '{}' http://127.0.0.1:<port>/tool/click
{"ok":false,"error":{"code":"VALIDATION_ERROR","message":"Exactly one of a11yRef, testId, or selector must be provided"},"meta":{"timing":{"queueMs":0,"toolMs":0,"observationMs":0,"totalMs":2}}}
HTTP 400
$ curl -X POST -d '{not json' http://127.0.0.1:<port>/tool/get_state
{"ok":false,"error":{"code":"INTERNAL_ERROR","message":"Expected property name or '}' in JSON at position 1 (line 1 column 2)"},"meta":{"timing":{"queueMs":0,"toolMs":0,"observationMs":0,"totalMs":1}}}
HTTP 500

$ yarn mm get-state     # Extension, tail
  "meta": {
    "timing": {
      "queueMs": 0,
      "toolMs": 10,
      "observationMs": 87,
      "totalMs": 105
    }
  }
}

meta.timing per platform (smoke run, ms: queue / tool / observation / total)
            describe-screen      click unlock        cdp eval
ext         0/591/567/1159       0/22/17/40          0/2/564/567
ios         0/2570/2517/5088     0/1592/2331/3924    0/7/2461/2469
android     0/5759/3228/8995     0/2484/5006/7494    0/3161/6760/9933
Unit tests
 Test Files  49 passed (49)
      Tests  1639 passed (1639)
Type Errors  no errors
All files          |    96.4 |    91.64 |   94.71 |   96.64 |

New tests: src/server/response-timing.test.ts; meta.timing block in src/server/create-server.test.ts (success, tool vs observation split, no session, queue wait, tool throw, validation, unknown tool, platform gate, malformed JSON measured from arrival); meta block in src/cli/mm.test.ts.

Every /tool/*, /launch and /cleanup response body now ends with
`meta: { timing: { queueMs, toolMs, observationMs, totalMs } }`, on
success and on errors (404, 400 validation, platform gate, 500 tool throw,
500 from the error middleware for malformed JSON). The request start is
taken before the body parser, so totalMs covers request in to response out.

The mm CLI keeps `meta` as the last key of its JSON output. String results
are printed as before, without meta.

Add scripts/bench.mjs (not shipped) to run mm commands N times against a
live session and print median/p95 of wall clock, bytes and meta.timing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant