teach-mode: learn-from-demonstration → compiled skill (the one real gap vs Grok Bot) #1

Activity

1. bferanmi806-sketch commented on Aug 16, 2026

   

   bferanmi806-sketch

   on Aug 16, 2026

   Last edited by bferanmi806-sketch

   Contributor

   More actions

   I’d love to work on this. I traced the current v0.2 path from the computer panel through the Docker/noVNC browser, the gateway’s Playwright CDP connection, routines, and the agent tool loop before proposing an implementation.

   One thing I wanted to confirm first: this issue predates the v0.2 self-built-core merge and still describes compiling demonstrations into an OpenClaw SKILL.md. Current main no longer appears to have an OpenClaw skill runtime, so I think the record → compile → deterministic replay goal still fits, but the artifact probably wants to be native to OpenGrokBot now.

   My current direction would be:

   • record against the bot’s existing persistent Chromium while the operator drives it through the current noVNC panel, rather than launch a second Codegen browser;
   • instrument the existing Playwright/CDP context to capture semantic navigation/click/input/select events;
   • compile those into a small typed workflow rather than arbitrary generated JS;
   • never persist password/secret input values;
   • replay deterministic steps first, then hand control to the agent only when a locator/page has drifted;
   • keep learned workflows separate from routines: a workflow describes how, while a routine can later schedule when it runs.

   I’d probably split it into recorder/trace foundation → deterministic workflow runner → agent/routine integration rather than land teach-mode as one large PR.

   Does that match where you want #1 to go after the v0.2 rewrite? If so, I’m happy to take the first slice
2. wolfqing commented on Oct 30, 2026

   

   wolfqing

   on Oct 30, 2026

   OwnerAuthor

   More actions

   Thanks for the thorough read, and sorry for the long silence. Yes, your direction matches where #1 should go after the v0.2 rewrite: record against the bot's own persistent Chromium through the existing CDP context, compile into a small typed workflow rather than generated JS, never persist secret input, replay deterministically and hand off to the agent on drift, and keep workflows ("how") separate from routines ("when"). Splitting it into recorder/trace → deterministic runner → agent/routine integration is the right shape too.

   I've refreshed the issue body to drop the OpenClaw SKILL.md framing. If you're still interested, the first slice (recorder + trace format) is yours; a short design note on the trace schema before code would be ideal.

   ❤️React with ❤️1Reacted by Enoch
3. bferanmi806-sketch commented on Oct 31, 2026

   

   bferanmi806-sketch

   on Oct 31, 2026

   Contributor

   More actions

   Trace schema v1

   I propose keeping a recorded workflow as a versioned, typed JSON document, separate from routines. A workflow describes how to do something; a routine may later decide when to run it. The first slice records traces only and does not add replay or scheduling.

   type WorkflowTrace = {
     format: 'opengrokbot.workflow'
     version: 1
     title: string
     steps: WorkflowStep[]
   }
   
   type Locator = {
     role?: { name: string; role: string }
     label?: string
     placeholder?: string
     text?: string
     testId?: string
     css?: string // last-resort fallback
   }
   
   type WorkflowStep =
     | { type: 'navigate'; url: string }
     | { type: 'click'; target: Locator }
     | { type: 'fill'; target: Locator; value: { kind: 'literal'; text: string } | { kind: 'redacted'; reason: 'sensitive-input' } }
     | { type: 'select'; target: Locator; value: { kind: 'literal'; text: string } | { kind: 'redacted'; reason: 'sensitive-input' } }

   The recorder should attach to the teammate's existing persistent Chromium Page supplied by the shared BrowserSession; it must not launch a second browser or replace the logged-in context. A recording session explicitly starts and stops event capture on that page. Each action stores semantic locators in a resilient preference order (role/name, label, placeholder, test id, then CSS), without generating executable JavaScript.

   Redaction is enforced before serialization. Password and one-time-code fields, sensitive autocomplete values, and fields whose accessible/name metadata indicates a password, secret, token, or API key produce a redacted fill/select step with no input value. Navigation URLs lose user-info and fragments; query values are omitted by default so session tokens cannot leak through URLs. The trace never stores page text, screenshots, cookies, or other browser state.

   The format leaves room for later deterministic replay: locator candidates are tried in order, and a redacted step can ask the operator for a value at replay time. That runner and routine integration remain out of this first slice. If this shape fits, I’ll proceed with the recorder and schema validation/tests only.
4. wolfqing commented on Nov 1, 2026

   

   wolfqing

   on Nov 1, 2026

   OwnerAuthor

   More actions

   Thanks — this is a clear note, and the overall shape is right: one document per workflow with a version number, a recorder that attaches to the teammate's existing browser, redaction before anything is written, and no generated JS. Please go ahead with the recorder and schema slice. A few things I'd like folded in first:

   1. Build on v0.4.0. A teammate's computer can now be local, not only Docker. Since your note, v0.4.0 added local computers: on macOS and Linux a teammate can get a browser on the host instead of a container. Both kinds share BrowserSession (gateway/src/browser-page.ts), so attaching to session.page() works for both. Two things differ for recording:

   • On a local computer the panel only shows a CDP screencast, which you can watch but not click or type into. The operator drives the real window instead, the one showWindow() opens for sign-in. So recording on a local computer means showing that window first.
   • showWindow() / hideWindow() close the browser and reopen it with or without a window (see relaunch in computer-local.ts), so the Page you attached to is gone afterwards. The recorder needs to re-attach after a relaunch, or recording needs to stop cleanly when one happens.

   Please test both Docker and local computers.

   2. A few more step types, or at least room for them. navigate/click/fill/select cover a lot, but real workflows quickly need:

   • press (Enter to submit a search box is the common case)
   • check / uncheck
   • clicks that open a new tab or popup
   • actions inside an iframe

   Leaving some of these out of v1 is fine, as long as the format can add them without a version bump. Maybe that means an optional frame / page reference on every step.

   3. Make the locator candidates an ordered list. "Tried in order" is really a property of the list, so target: { candidates: Locator[] } (most stable first, each candidate one strategy) says it directly. A single object with optional fields leaves the order to whoever reads it.

   4. Leave room for intent now. The compile step will add each step's intent and the page state it expects, and the agent fallback depends on both. An optional intent?: string on every step (and on the trace) costs nothing now and avoids a v2 later.

   5. Literal values are still personal data. Redacting secrets is right, but a fill with an email address or a street address is still stored as is. Before saving, please show the operator what was captured (even a plain step list), so they can turn any value into a placeholder or drop a step. Recording query keys with their values removed, so a replay knows which parameters were there, also sounds good.

   6. Two small scope questions.

   • Storage: everything else a teammate owns (routines, approvals, the action log) lives in SQLite (gateway/src/db.ts). A workflows table next to routines would match that.
   • UI: does this slice include start/stop controls in the computer panel, or only the API? Either is fine; just say which in the PR.

   Happy to review the PR as soon as it's ready.
5. wolfqing

   added 2 commits that reference this issue on Nov 1, 2026

   docs(readme): describe local computers wherever the READMEs assumed D…

   ...

   52a5284

   feat(gateway): record typed browser workflows (#11)

   ...

   cf6f17c