[saruar-dev]

Description

Adds Nace Drex Max as an opt-in decision model for Skyvern's bounded-choice prompts. Drex Max is not a chat model: one POST /v1/systemone request carries a state text, up to eight images and a few bounded questions (noul yes/no, choice over fixed options), and the reply is a probability per option instead of generated text. Several Skyvern prompts are bounded decisions already, so this PR rewrites them into System One questions and rebuilds the JSON object their existing parsers expect. LLM_KEY keeps doing the action planning and extraction; the main action-planning loop is untouched.

Disclosure: I work at nace.ai, which serves Drex Max.

What's included

• skyvern/forge/sdk/api/llm/nace_decision_llm_caller.py (new) — the System One recipes and a small httpx client, following the shape of the UI-TARS / Yutori Navigator callers (Add Yutori Navigator as a CUA engine #5513):
  • check-user-goal, check-user-goal-after-click → one noul → user_goal_achieved
  • check-user-goal-with-termination → choice complete/terminate/continue (+ a failure category when terminating)
  • decisive-criterion-validate → choice COMPLETE/TERMINATE → actions[]
  • page-classify → choice over the options rendered in the prompt (including UNKNOWN)
  • infer-action-type → choice over the action types, inferred_actions ranked by probability
  • auto-completion-choose-option → noul for the attempt + choice over the element ids rendered in the prompt (restricted to the emerging ids when present, NONE option) + direct_searching for search bars
  • Screenshots go as the top-level images array (data URLs, first 8 kept); the prompt text is the state; choice criteria is an object of strings. Any other prompt is rejected before a request is made. thoughts/reasoning fields carry the decision and its probability, since the model generates no text.
• Routing — LLMCaller._dispatch_llm_call sends the NACE_DREX_MAX key to _call_nace_decision (like _call_ui_tars / _call_yutori_navigator), and get_llm_api_handler builds that key as an LLMCaller so it never reaches litellm.acompletion. Artifacts, token accounting (get_call_stats prices input tokens only, as billed) and error classification (4xx deterministic, 429/5xx/529 retryable) stay in the base class.
• Config — ENABLE_NACE, NACE_API_KEY, NACE_API_BASE (default https://console.nace.ai), NACE_MODEL (default drex-max) in config.py, .env.example; registry entry under if settings.ENABLE_NACE.
• forge_app.py — NACE_CLIENT and DECISION_LLM_API_HANDLER, both None unless enabled (a missing key logs a warning and keeps the defaults).
• Call sites — the bounded prompts resolve their handler through get_decision_llm_api_handler(default), which returns default unless ENABLE_NACE is on: complete verification and infer-action-type in agent.py, the after-click check and both auto-completion-choose-option calls in webeye/actions/handler.py, page-classify and infer-action-type in real_skyvern_page_ai.py. Organization defaults and per-run llm_key overrides still wrap the result, so they win over the deployment-wide opt-in. decisive-criterion-validate is rendered inside the planning path, so it is only answered by Drex Max when a run selects the NACE_DREX_MAX key through its llm_key override.
• Docs — "Nace Drex Max (decision model)" section in docs/reference/configuration.mdx.

Nothing changes for deployments that do not set ENABLE_NACE=true. Not included on purpose: the skyvern init wizard, per-prompt-type routing settings, and any benchmark numbers.

How Has This Been Tested?

Unit tests in tests/unit/test_nace_decision_llm_caller.py (mocked httpx.MockTransport, no network): request shapes (noul/choice only, string instructions, choice criteria as an object of strings, images capped at eight), answer decoding for every supported prompt against the real templates, HTTP error classification, the chat-shaped ModelResponse round-tripping through parse_api_response, factory routing away from litellm, input-token-only cost, and the ENABLE_NACE fallback helper.

uv sync --all-extras
ruff format <touched files>                      # 3 files reformatted, 6 unchanged
ruff check <touched files>                       # no findings in the changed lines / new files
mypy skyvern/forge/sdk/api/llm/nace_decision_llm_caller.py skyvern/forge/sdk/api/llm/api_handler_factory.py skyvern/forge/forge_app.py
                                                 # Success: no issues found in 3 source files
pytest tests/unit/test_nace_decision_llm_caller.py
                                                 # 31 passed
pytest tests/unit/test_api_handler_factory.py tests/unit/test_check_user_goal_llm_override.py \
       tests/unit/test_llm_response_parsing.py tests/unit/test_complete_verify_goal_unwrap.py \
       tests/unit/test_complete_verify_span.py tests/unit/test_verifier_goal_unwrap.py \
       tests/unit/test_select_shadow_match.py tests/unit/test_custom_llms.py tests/unit/test_internal_llms.py \
       tests/unit/test_llm_handler_parity_characterization.py tests/unit/test_llm_prompt_config.py \
       tests/unit/test_llm_attribution.py tests/unit/test_llm_handler_tracing.py tests/unit/test_copilot_model_usage.py \
       tests/unit/test_nace_decision_llm_caller.py
                                                 # 625 passed
pytest <the 58 tests/unit files referencing the touched call-site modules>
                                                 # 4077 passed, 131 skipped

Two manual live calls against https://console.nace.ai/v1/systemone through call_nace_decision (a page-classify prompt with a synthetic login-form screenshot, and a check-user-goal prompt with the same screenshot) returned {"classification": "LOGIN", "confidence": 0.998, ...} and {"user_goal_achieved": true, "confidence_float": 0.984, ...} with usage.input_tokens populated, confirming the request shape the docs specify (object criteria of strings, top-level images array, no box question type).

Not run: the full unit suite, the Playwright/integration suites, and pre-commit run --all-files (which with ruff 0.16 reports pre-existing findings in the large files this PR touches).

Artifacts (if appropriate):

Example parsed replies from the live calls above:

{"reasoning": "Drex Max decision: classification LOGIN (p=0.998)", "classification": "LOGIN", "confidence": 0.998}
{"thoughts": "Drex Max decision: user goal achieved (p=0.984)", "user_goal_achieved": true, "confidence_float": 0.984}

🤖 Generated with Claude Code

[mintlify]

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit bde46bc.

Approve preview