Skip to content

Refactor instruction runtime for generic agent actions - #53

Merged
jeremymcs merged 3 commits into
open-gsd:mainfrom
Solvely-Colin:feat/instruction-core-planner
Jun 14, 2026
Merged

jeremymcs merged 3 commits into
open-gsd:mainfrom
Solvely-Colin:feat/instruction-core-planner

Conversation

@Solvely-Colin

@Solvely-Colin Solvely-Colin commented Jun 13, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Split instruction handling into parser, planner, page model, action runtime, workflow runtime, and verification modules
  • Expand generic action planning across forms, custom controls, shadow DOM, iframes, scoped workflows, visual interactions, uploads, navigation, and command surfaces
  • Add reusable instruction-runtime validation and cleanup scripts to avoid stale browser/session state during full runs
  • Update MCP docs to surface browser_act_instruction as part of the semantic action surface

Validation

  • cargo fmt --check
  • git diff --check
  • cargo check -p gsd-browser
  • cargo clippy -p gsd-browser --tests (passes with existing warnings)
  • scripts/run-instruction-runtime.sh --summary --results-file /tmp/gsd-browser-instruction-runtime-rebased-20260613.tsv -> PASS=232 FAIL=0 MISSING=0 TOTAL=232
  • hardcode scan for benchmark/domain/task names returned no matches
  • final browser/test-runner process scan clean

View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.

Copilot AI review requested due to automatic review settings June 13, 2026 18:09
@Solvely-Colin
Solvely-Colin force-pushed the feat/instruction-core-planner branch from 9a153fd to cd99827 Compare June 13, 2026 18:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors the act_instruction runtime into clearer modules (planning/page-model/action/verification), expands generic action handling across more UI surfaces (custom controls, shadow DOM/iframes, uploads, etc.), and adds reusable local-run scripts to avoid stale browser/session state during full instruction-runtime sweeps.

Changes:

  • Introduces an instruction page model and before/after verification summary to make act_instruction effects observable and auditable.
  • Improves interaction primitives (e.g., richer click parameters, stronger enabled/visible checks, more robust upload resolution) and form analysis/fill logic to support custom controls and nested DOM roots.
  • Adds scripts for cleaning up managed browser runs and running instruction-runtime tests serially with locking/retries; updates docs to surface browser_act_instruction.

Reviewed changes

Copilot reviewed 22 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
SKILL.md Documents page-model grounding and verification summaries for instruction plans.
scripts/run-instruction-runtime.sh New runner for serial instruction_runtime tests with locking/retries/results logging.
scripts/cleanup-browser-runs.sh New cleanup tool for managed daemon/Chrome/test-runner processes and temp artifacts.
gsd-browser-skill/references/command-reference.md Updates act-instruction docs to reference the page model and verification signals.
docs/mcp.md Surfaces browser_act_instruction in the MCP semantic action surface.
docs/AGENT-BEST-PRACTICES.md Updates guidance for when/how to use browser_act_instruction.
cli/src/output.rs CLI output now prints page-model summary and verification status for act_instruction.
cli/src/moonshot_tool_schema.rs Test-only helpers/constants are now #[cfg(test)].
cli/src/mcp.rs Avoids non-test imports; keeps Moonshot schema assertions in tests only.
cli/src/daemon/state.rs Adds PageRegistry::replace_page_handle to support page-handle recovery.
cli/src/daemon/mod.rs Wires updated act_instruction handler signature; minor lock usage improvements.
cli/src/daemon/inspection.rs Expands visibility/enabled checks; improves custom control semantics in JS actions.
cli/src/daemon/handlers/pages.rs Adds active-page refresh from browser target list for detached session recovery.
cli/src/daemon/handlers/interaction.rs Adds click button/count support; improves typed results, drag center resolution, upload targeting via objectId, and checked-state reporting.
cli/src/daemon/handlers/intent.rs Updates click helper callsite for new click signature.
cli/src/daemon/handlers/instruction/verification.rs New verification engine producing structured “signals” from before/after page models + dispatch results.
cli/src/daemon/handlers/instruction/planner.rs New planner building action plans from parsed instruction intents and DOM planner JS.
cli/src/daemon/handlers/instruction/page_model.rs New page-model capture (visible elements, labels, grouping, affordances) used by planner/verification.
cli/src/daemon/handlers/instruction/model.rs New shared instruction analysis/intent models and JSON helpers.
cli/src/daemon/handlers/forms.rs Expands form field discovery/label resolution across shadow roots/iframes + custom controls.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +455 to +458
acquire_lock "$@"
if [[ -n "$runner_log_file" ]]; then
: >"$runner_log_file"
fi
Comment on lines +290 to 292
const elements = [...fieldElements(form), ...all(BUTTON_SELECTOR, form).filter(el => !fieldElements(form).includes(el))];
const fields = [];
const submitButtons = [];
@jeremymcs
jeremymcs merged commit 1041aaf into open-gsd:main Jun 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants