Skip to content

fix(ci): stabilize GPU E2E tests on shared RDP agents - #1416

Merged
Yue Sun (KayMKM) merged 3 commits into
mainfrom
yuesu/fix-rdp-gpu-e2e
Sep 14, 2026
Merged

Yue Sun (KayMKM) merged 3 commits into
mainfrom
yuesu/fix-rdp-gpu-e2e

Conversation

@KayMKM

Copy link
Copy Markdown
Contributor

RDP transitions on the shared NPU-OV2 agent leave duplicate GPU routes in ORT, including a route that fails with DXGI_ERROR_DEVICE_REMOVED. This causes DML evaluation failures and makes adapter-selection tests fail even when the physical GPU remains usable.

This change opts NPU-OV2 into physical-GPU selection in the E2E harness. DML Eval resolves the sole DXCore GPU and passes its current LUID to winml perf. The perf pytest step uses that GPU while preserving the requested EP, including OpenVINO GPU. Both adapter-selection cases still execute inference independently through --device-luid and --ep-options device_id, and verify the monitor binding. Missing hardware/LUIDs and unrelated extra devices still fail; normal agents retain strict inventory and default-selection checks. No tests are skipped or marked xfail, and shipped product code is unchanged.

The keepalive installer also includes the enhanced worker already deployed on the agent: network-source filtering, Session 1 support, occupied-console protection, verified tscon retries, and cleanup limited to non-present RDP display nodes. The task now passes the source address directly, and the documentation explains that console recovery does not prove DML readiness.

Validation:

  • Eight targeted hardware E2E cases passed on NPU-OV2 with RDP connected: both selectors, DML binding/inference, DML and OpenVINO monitoring, and CLI argument validation; no skips (September 11).
  • 201 evaluation-harness unit tests passed (September 11).
  • 42 targeted checks passed before publication, including PowerShell parsing, mocked keepalive transitions, worker equivalence, and pinning regressions (September 14).
  • Ruff and git diff --check passed; pipeline YAML parsed successfully.

The pin is opt-in through dmlPinnedAgents (initially NPU-OV2); an empty list disables it. It does not guarantee that in-flight inference survives a driver reset during an RDP switch.

@KayMKM
Yue Sun (KayMKM) requested a review from a team as a code owner September 14, 2026 03:27

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the physical-GPU pinning and keepalive changes. One actionable issue in the new retry path is noted below. Validation included a Windows PowerShell 5.1 native-stderr reproduction; the shared-agent hardware E2E suite was not rerun.

Comment thread scripts/agent_setup/setup_rdp_gpu_keepalive.ps1 Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the changes since 090f9cb. The earlier native-stderr finding is addressed for both tscon and pnputil: Continue is scoped to native invocation, the exit code is captured, and the original error preference is restored in finally. A harmless native-stderr reproduction in Windows PowerShell confirmed that three attempts complete, exit code 7 is retained, and Stop is restored. All reported PR checks pass; no remaining blocking findings. Shared-agent GPU/RDP end-to-end behavior was not rerun locally.

@KayMKM
Yue Sun (KayMKM) merged commit a04129c into main Sep 14, 2026
9 checks passed
@KayMKM
Yue Sun (KayMKM) deleted the yuesu/fix-rdp-gpu-e2e branch September 14, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants