feat(gateway): record typed browser workflows

CHANGELOG.md

@@ -6,6 +6,12 @@ still change behaviour.
 
 ## [Unreleased]
 
+### Added
+
+- First teach-mode slice: record semantic browser actions on the teammate's existing page, review the draft before
+  saving it to SQLite, redact sensitive input, and reattach across local-computer window relaunches. API-only; replay
+  and routines remain out of scope.
+
 ### Security
 
 - A local computer's screen stream (`/screen/:botId/ws`) was served without the gateway's token and origin

README.md

@@ -68,6 +68,10 @@ Every line below is implemented and covered by tests — nothing here is a roadm
   routine a secret webhook address instead, for GitHub, a form or any script to call. "Keep an eye on the status
   page" makes a watch routine: read-only tools, and silence unless something needs you. Pause, run or delete them
   from the computer panel — or tell the bot to stop one.
+- **Recorded workflow traces (first slice).** API-only controls start and stop a typed recording on a teammate's
+  existing browser page. The operator reviews the draft before saving it to SQLite; password, one-time-code and
+  other sensitive-looking fields are redacted automatically. Local-computer recordings continue across window
+  relaunches. Replay and routine scheduling are not part of this slice. See [docs/workflows.md](docs/workflows.md).
 - **Shared skills.** Drop a Markdown playbook into `<data>/skills/` and every teammate sees it and reads it
   before work it covers. Memory and routines stay with each teammate; skills are shared.
 - **See the work, stop the work.** Every step on its computer — the command, the page, the click — shows live
@@ -229,7 +233,8 @@ limits), and on Windows it is still the only way to give teammates a computer. S
 [docs/local-computers.md](docs/local-computers.md).
 
 More in [docs/faq.md](docs/faq.md) · [docs/multi-bot.md](docs/multi-bot.md) · [docs/always-on.md](docs/always-on.md) ·
-[docs/security.md](docs/security.md) · [docs/comparison.md](docs/comparison.md) · [CHANGELOG.md](CHANGELOG.md)
+[docs/security.md](docs/security.md) · [docs/workflows.md](docs/workflows.md) · [docs/comparison.md](docs/comparison.md) ·
+[CHANGELOG.md](CHANGELOG.md)
 
 ## Disclaimer
 

docs/workflows.md

@@ -0,0 +1,125 @@
+# Recorded workflow traces
+
+This first teach-mode slice records a browser trace for a later deterministic workflow runner. It does not replay,
+schedule, or hand steps to the agent when a page changes. The slice exposes API controls only; it adds no start/stop
+buttons or workflow editor to the product UI.
+
+Recordings attach to the page already owned by the teammate's shared `BrowserSession`. They do not launch a second
+browser or replace its logged-in context. In v0.4.0 local-computer mode, `showWindow()` and `hideWindow()` relaunch
+Chromium; the recorder reattaches to the replacement page and continues recording. Show the local browser window
+before recording: its panel screencast is view-only, so the operator drives the real window. Docker recording also
+attaches to the current page. The recorder captures navigation, clicks, text entry, selection, checkbox state, and
+Enter presses as typed steps. A navigation caused by an action is recorded as that action's expected URL, and a
+popup is marked as unrecorded. It does not store screenshots, cookies, arbitrary page dumps, or generated JavaScript.
+
+## Start, stop, review, and save
+
+All endpoints require the gateway's normal authentication token. Drive the teammate's current page through its
+computer screen while recording is active.
+
+```http
+POST /api/bots/{botId}/workflows/recording
+Content-Type: application/json
+
+{"title":"Open the project dashboard","intent":"Find the current project status"}
+```
+
+`intent` is optional. The gateway responds with `202` and starts recording. Only one recording or unsaved draft can
+exist per teammate. Stop capture to receive a review draft:
+
+```http
+POST /api/bots/{botId}/workflows/recording/stop
+```
+
+Stopping does not save anything to SQLite. Review the returned trace, remove steps that should not be retained,
+replace literal values with `{ "kind": "placeholder", "key": "customerEmail" }`, and then save the edited trace:
+
+```http
+POST /api/bots/{botId}/workflows
+Content-Type: application/json
+
+{"draftId":"<id returned by stop>","trace":{"format":"opengrokbot.workflow","version":1,"title":"...","steps":[]}}
+```
+
+The submitted trace is validated and sensitive fields are redacted again before persistence. To discard a draft:
+
+```http
+DELETE /api/bots/{botId}/workflows/drafts/{draftId}
+```
+
+Drafts are held in gateway memory until saved or discarded. Restarting the gateway discards pending drafts. Saved
+traces are stored as rows in the gateway SQLite database, alongside routines. List and read saved traces with:
+
+```http
+GET /api/bots/{botId}/workflows
+GET /api/bots/{botId}/workflows/{workflowId}
+```
+
+The computer status response includes `workflowRecording` during capture and `workflowDraft` while review is
+pending. An active recording prevents idle computer cleanup until it is stopped.
+
+## Trace format v1
+
+The trace is JSON. Each locator candidate uses exactly one strategy; the ordered array preserves preference from
+stable semantic locator to structural CSS fallback.
+
+```ts
+type WorkflowTrace = {
+  format: 'opengrokbot.workflow'
+  version: 1
+  title: string
+  intent?: string
+  steps: WorkflowStep[]
+}
+
+type WorkflowStepContext = {
+  intent?: string
+  page?: string
+  frame?: string
+}
+
+type WorkflowActionContext = WorkflowStepContext & { expect?: { url: string } }
+
+type WorkflowStep = WorkflowStepContext & (
+  | { type: 'navigate'; url: string }
+  | (WorkflowActionContext & { type: 'click'; target: { candidates: Locator[] } })
+  | (WorkflowActionContext & { type: 'fill'; target: { candidates: Locator[] }; value: WorkflowValue })
+  | (WorkflowActionContext & { type: 'select'; target: { candidates: Locator[] }; value: WorkflowValue })
+  | (WorkflowActionContext & { type: 'check' | 'uncheck'; target: { candidates: Locator[] } })
+  | (WorkflowActionContext & { type: 'press'; key: 'Enter'; target: { candidates: Locator[] } })
+  | { type: 'unrecorded-popup' }
+)
+
+type Locator =
+  | { role: { role: string; name: string } }
+  | { label: string }
+  | { placeholder: string }
+  | { testId: string }
+  | { text: string }
+  | { css: string }
+
+type WorkflowValue =
+  | { kind: 'literal'; text: string }
+  | { kind: 'placeholder'; key: string }
+  | { kind: 'redacted'; reason: 'sensitive-input' }
+```
+
+`page`, `frame`, and `intent` are optional on every step so recorder metadata can grow while the trace remains v1.
+The current recorder emits one locator candidate for each recognized strategy, ordered as role/name, label,
+placeholder, test ID, text, then CSS. CSS fallbacks are positional paths and do not copy element IDs or classes.
+
+## Redaction and limits
+
+The browser checks input type, autocomplete, labels, accessible names, placeholders, and field names before sending
+values to the gateway. Password and one-time-code fields, credit-card autocomplete fields, fields marked with
+sensitive terms such as `secret`, `token`, or `api key`, known credential formats, and high-entropy strings are
+stored as redacted values without their text. The gateway validates every event again and re-applies field-name and
+credential-pattern checks before a draft is returned or a saved trace is written. URL user info and fragments are
+removed; query keys are retained with empty values. High-entropy path segments, including UUIDs, are replaced with
+`:redacted`; those navigation steps cannot be replayed as the original record-specific URL. Only HTTP and HTTPS
+navigation is recorded.
+
+V1 is bounded to 1,000 steps, 120 title characters, 500 intent characters, 2,000 input characters per step, and
+5 MB per trace. Secret detection relies on field metadata and credential-pattern checks; avoid entering secrets
+into fields that are not marked or named as sensitive. The operator review before saving lets the operator remove
+personal values that do not match these automatic checks.

gateway/src/browser-page.ts

@@ -23,20 +23,28 @@ export type PageLike = {
   goBack?(opts?: unknown): Promise<unknown>
   waitForLoadState?(state?: string, opts?: unknown): Promise<void>
   evaluate?(script: string): Promise<unknown>
+  context?(): PageContextLike
+  mainFrame?(): unknown
+  on?(event: 'framenavigated' | 'close' | 'popup', listener: (value?: unknown) => void): void
+  off?(event: 'framenavigated' | 'close' | 'popup', listener: (value?: unknown) => void): void
   viewportSize(): { width: number; height: number } | null
 }
 
 /** 够用就好的 CDP 会话：本机模式用它拿浏览器画面 */
 export type CdpSessionLike = {
   send(method: string, params?: Record<string, unknown>): Promise<unknown>
-  on(event: string, listener: (params: never) => void): unknown
+  on(event: string, listener: (params: unknown) => void): unknown
+  off?(event: string, listener: (params: unknown) => void): unknown
   detach(): Promise<void>
 }
 
-export type ContextLike = {
+export type PageContextLike = {
+  newCDPSession?(page: unknown): Promise<CdpSessionLike>
+}
+
+export type ContextLike = PageContextLike & {
   pages(): PageLike[]
   newPage(): Promise<PageLike>
-  newCDPSession?(page: PageLike): Promise<CdpSessionLike>
 }
 
 export type BrowserLike = {

gateway/src/computer-docker.ts

@@ -2,6 +2,7 @@ import { chromium } from 'playwright-core'
 import { createBrowserSession, pageOperations, type BrowserLike } from './browser-page.js'
 import type { BotComputer, ShellResult } from './computer.js'
 import type { Endpoints } from './containers.js'
+import { createWorkflowRecorder } from './workflow-recorder.js'
 
 export type DockerComputerDeps = {
   fetchImpl?: typeof fetch
@@ -83,6 +84,9 @@ export function createDockerComputer(endpoints: Endpoints, deps: DockerComputerD
     async writeFile(path, content) {
       await post('/write', { path, content })
     },
+    async startWorkflowRecording(title, intent) {
+      return createWorkflowRecorder(await session.page(), title, intent)
+    },
     ...pageOperations(() => session.page()),
     dispose: () => session.dispose(),
   }

gateway/src/computer-local.ts

@@ -4,6 +4,8 @@ import { dirname, isAbsolute, join, relative, resolve, sep } from 'node:path'
 import { chromium } from 'playwright-core'
 import { createBrowserSession, pageOperations, type BrowserLike, type ContextLike } from './browser-page.js'
 import type { BotComputer, ShellResult } from './computer.js'
+import { createWorkflowRecorder } from './workflow-recorder.js'
+import type { WorkflowRecording } from './workflow-recorder.js'
 
 /**
  * 本机模式的电脑：浏览器是宿主上直接拉起来的 Chromium（一位同事一个配置目录，登录态留在里面），
@@ -16,6 +18,18 @@ const MAX_FILE_BYTES = 16 * 1024 * 1024
 /** bot 的提示词和工具说明里都叫 /workspace；本机模式把它映射到这位同事的文件夹 */
 const VIRTUAL_ROOT = '/workspace'
 
+type ScreencastFrame = { data: string; sessionId: number; metadata: { deviceWidth: number; deviceHeight: number } }
+
+function isScreencastFrame(value: unknown): value is ScreencastFrame {
+  if (value === null || typeof value !== 'object' || Array.isArray(value)) return false
+  if (!('data' in value) || !('sessionId' in value) || !('metadata' in value)) return false
+  const metadata = value.metadata
+  if (metadata === null || typeof metadata !== 'object' || Array.isArray(metadata)) return false
+  if (!('deviceWidth' in metadata) || !('deviceHeight' in metadata)) return false
+  return typeof value.data === 'string' && typeof value.sessionId === 'number'
+    && typeof metadata.deviceWidth === 'number' && typeof metadata.deviceHeight === 'number'
+}
+
 export type LocalLaunch = (opts: { profileDir: string; executablePath: string; headless: boolean }) => Promise<BrowserLike>
 
 export type LocalShell = {
@@ -92,6 +106,7 @@ export function createLocalComputer(opts: LocalComputerOptions): BotComputer {
   const launch = opts.launch ?? defaultLaunch
   let headless = opts.headless ?? true
   const session = createBrowserSession(() => launch({ profileDir: opts.profileDir, executablePath: opts.executablePath, headless }))
+  let activeWorkflowRecording: WorkflowRecording | null = null
 
   /**
    * Chromium 不能在运行中切换有没有窗口：关掉再以另一种方式开同一个配置。
@@ -107,11 +122,19 @@ export function createLocalComputer(opts: LocalComputerOptions): BotComputer {
   async function relaunch(nextHeadless: boolean): Promise<void> {
     let url: string | undefined
     if (session.state() === 'open') url = await session.page().then((p) => p.url()).catch(() => undefined)
+    try {
+      await activeWorkflowRecording?.detach?.()
+    } catch {
+      activeWorkflowRecording?.markIncomplete?.()
+    }
     await session.dispose()
     endStreams()
     headless = nextHeadless
     const p = await session.page()
+    let reattachError: unknown
+    try { await activeWorkflowRecording?.reattach(p) } catch (error) { reattachError = error }
     if (url && url !== 'about:blank') await p.goto(url, { waitUntil: 'domcontentloaded', timeout: 45_000 }).catch(() => {})
+    if (reattachError) throw reattachError
   }
 
   return {
@@ -125,12 +148,12 @@ export function createLocalComputer(opts: LocalComputerOptions): BotComputer {
       if (!context.newCDPSession) throw new Error('this browser cannot stream its screen')
       const page = await session.page()
       const cdp = await context.newCDPSession(page)
-      type Frame = { data: string; sessionId: number; metadata: { deviceWidth: number; deviceHeight: number } }
       let stopped = false
       const end = () => { stopped = true; hooks.onEnd?.() }
       watchers.add(end)
-      cdp.on('Page.screencastFrame', (frame: Frame) => {
-        if (stopped) return
+      cdp.on('Page.screencastFrame', (value) => {
+        if (stopped || !isScreencastFrame(value)) return
+        const frame = value
         onFrame({ data: frame.data, width: frame.metadata.deviceWidth, height: frame.metadata.deviceHeight })
         // 不确认上一帧，Chromium 就不发下一帧——借这一点把帧率压在 10 帧/秒上下，面板和网关都轻松
         setTimeout(() => { void cdp.send('Page.screencastFrameAck', { sessionId: frame.sessionId }).catch(() => {}) }, 100)
@@ -181,6 +204,23 @@ export function createLocalComputer(opts: LocalComputerOptions): BotComputer {
         await handle.close()
       }
     },
+    async startWorkflowRecording(title, intent) {
+      const recorder = await createWorkflowRecorder(await session.page(), title, intent)
+      const active: WorkflowRecording = {
+        reattach: (page) => recorder.reattach(page),
+        async detach() { await recorder.detach?.() },
+        markIncomplete() { recorder.markIncomplete?.() },
+        async stop() {
+          try {
+            return await recorder.stop()
+          } finally {
+            if (activeWorkflowRecording === active) activeWorkflowRecording = null
+          }
+        },
+      }
+      activeWorkflowRecording = active
+      return active
+    },
     ...pageOperations(() => session.page()),
     async dispose() {
       await session.dispose()

gateway/src/computer.ts

@@ -1,3 +1,5 @@
+import type { WorkflowRecording } from './workflow-recorder.js'
+
 export type ShellResult = { stdout: string; stderr: string; exitCode: number }
 export type Shot = { buffer: Buffer; width: number; height: number }
 export type PageLink = { text: string; href: string }
@@ -36,6 +38,8 @@ export interface BotComputer {
   scroll(direction: 'up' | 'down'): Promise<void>
   back(): Promise<{ url: string; title: string }>
   screenshot(): Promise<Shot>
+  /** Starts event capture on this computer's existing persistent browser page. */
+  startWorkflowRecording?(title: string, intent?: string): Promise<WorkflowRecording>
   dispose(): Promise<void>
 }
 
@@ -53,6 +57,7 @@ export function createFakeComputer(
   const pageLinks = overrides.pageLinks ?? []
   const base: BotComputer = {
     kind: overrides.kind ?? 'docker',
+    ...(overrides.startWorkflowRecording ? { startWorkflowRecording: overrides.startWorkflowRecording } : {}),
     async ping() { return true },
     async shell(cmd) { calls.push(`shell:${cmd}`); return { stdout: 'hi\n', stderr: '', exitCode: 0 } },
     async readFile(path) { calls.push(`read:${path}`); return 'file contents' },

gateway/src/db.ts

@@ -156,6 +156,17 @@ const MIGRATIONS: Migration[] = [
   `),
   // v4：本机电脑的命令行开关，按同事存（操作者在面板上做的决定，要看得见、能撤）
   (db) => db.exec(`ALTER TABLE bots ADD COLUMN shell_enabled INTEGER NOT NULL DEFAULT 0;`),
+  // v5：recorded workflows belong beside routines and actions in the teammate database, not as loose files.
+  (db) => db.exec(`
+    CREATE TABLE IF NOT EXISTS workflows (
+      id TEXT PRIMARY KEY,
+      bot_id TEXT NOT NULL,
+      title TEXT NOT NULL,
+      trace_json TEXT NOT NULL,
+      created_at INTEGER NOT NULL
+    );
+    CREATE INDEX IF NOT EXISTS idx_workflows_bot ON workflows(bot_id, created_at DESC, id DESC);
+  `),
 ]
 
 export const SCHEMA_VERSION = MIGRATIONS.length