Repository navigation
Refactor instruction runtime for generic agent actions - #53
Merged
jeremymcs merged 3 commits intoJun 14, 2026
Merged
Conversation
Solvely-Colin
force-pushed
the
feat/instruction-core-planner
branch
from
June 13, 2026 18:10
9a153fd to
cd99827
Compare
There was a problem hiding this comment.
Pull request overview
Refactors the act_instruction runtime into clearer modules (planning/page-model/action/verification), expands generic action handling across more UI surfaces (custom controls, shadow DOM/iframes, uploads, etc.), and adds reusable local-run scripts to avoid stale browser/session state during full instruction-runtime sweeps.
Changes:
- Introduces an instruction page model and before/after verification summary to make
act_instructioneffects observable and auditable. - Improves interaction primitives (e.g., richer click parameters, stronger enabled/visible checks, more robust upload resolution) and form analysis/fill logic to support custom controls and nested DOM roots.
- Adds scripts for cleaning up managed browser runs and running instruction-runtime tests serially with locking/retries; updates docs to surface
browser_act_instruction.
Reviewed changes
Copilot reviewed 22 out of 26 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| SKILL.md | Documents page-model grounding and verification summaries for instruction plans. |
| scripts/run-instruction-runtime.sh | New runner for serial instruction_runtime tests with locking/retries/results logging. |
| scripts/cleanup-browser-runs.sh | New cleanup tool for managed daemon/Chrome/test-runner processes and temp artifacts. |
| gsd-browser-skill/references/command-reference.md | Updates act-instruction docs to reference the page model and verification signals. |
| docs/mcp.md | Surfaces browser_act_instruction in the MCP semantic action surface. |
| docs/AGENT-BEST-PRACTICES.md | Updates guidance for when/how to use browser_act_instruction. |
| cli/src/output.rs | CLI output now prints page-model summary and verification status for act_instruction. |
| cli/src/moonshot_tool_schema.rs | Test-only helpers/constants are now #[cfg(test)]. |
| cli/src/mcp.rs | Avoids non-test imports; keeps Moonshot schema assertions in tests only. |
| cli/src/daemon/state.rs | Adds PageRegistry::replace_page_handle to support page-handle recovery. |
| cli/src/daemon/mod.rs | Wires updated act_instruction handler signature; minor lock usage improvements. |
| cli/src/daemon/inspection.rs | Expands visibility/enabled checks; improves custom control semantics in JS actions. |
| cli/src/daemon/handlers/pages.rs | Adds active-page refresh from browser target list for detached session recovery. |
| cli/src/daemon/handlers/interaction.rs | Adds click button/count support; improves typed results, drag center resolution, upload targeting via objectId, and checked-state reporting. |
| cli/src/daemon/handlers/intent.rs | Updates click helper callsite for new click signature. |
| cli/src/daemon/handlers/instruction/verification.rs | New verification engine producing structured “signals” from before/after page models + dispatch results. |
| cli/src/daemon/handlers/instruction/planner.rs | New planner building action plans from parsed instruction intents and DOM planner JS. |
| cli/src/daemon/handlers/instruction/page_model.rs | New page-model capture (visible elements, labels, grouping, affordances) used by planner/verification. |
| cli/src/daemon/handlers/instruction/model.rs | New shared instruction analysis/intent models and JSON helpers. |
| cli/src/daemon/handlers/forms.rs | Expands form field discovery/label resolution across shadow roots/iframes + custom controls. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+455
to
+458
| acquire_lock "$@" | ||
| if [[ -n "$runner_log_file" ]]; then | ||
| : >"$runner_log_file" | ||
| fi |
Comment on lines
+290
to
292
| const elements = [...fieldElements(form), ...all(BUTTON_SELECTOR, form).filter(el => !fieldElements(form).includes(el))]; | ||
| const fields = []; | ||
| const submitButtons = []; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
browser_act_instructionas part of the semantic action surfaceValidation
cargo fmt --checkgit diff --checkcargo check -p gsd-browsercargo clippy -p gsd-browser --tests(passes with existing warnings)scripts/run-instruction-runtime.sh --summary --results-file /tmp/gsd-browser-instruction-runtime-rebased-20260613.tsv->PASS=232 FAIL=0 MISSING=0 TOTAL=232Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.