Repository navigation
fix(ci): stabilize GPU E2E tests on shared RDP agents - #1416
Conversation
Qiong Wu (qiowu) (DingmaomaoBJTU)
left a comment
There was a problem hiding this comment.
Reviewed the physical-GPU pinning and keepalive changes. One actionable issue in the new retry path is noted below. Validation included a Windows PowerShell 5.1 native-stderr reproduction; the shared-agent hardware E2E suite was not rerun.
Qiong Wu (qiowu) (DingmaomaoBJTU)
left a comment
There was a problem hiding this comment.
Re-reviewed the changes since 090f9cb. The earlier native-stderr finding is addressed for both tscon and pnputil: Continue is scoped to native invocation, the exit code is captured, and the original error preference is restored in finally. A harmless native-stderr reproduction in Windows PowerShell confirmed that three attempts complete, exit code 7 is retained, and Stop is restored. All reported PR checks pass; no remaining blocking findings. Shared-agent GPU/RDP end-to-end behavior was not rerun locally.
RDP transitions on the shared NPU-OV2 agent leave duplicate GPU routes in ORT, including a route that fails with DXGI_ERROR_DEVICE_REMOVED. This causes DML evaluation failures and makes adapter-selection tests fail even when the physical GPU remains usable.
This change opts NPU-OV2 into physical-GPU selection in the E2E harness. DML Eval resolves the sole DXCore GPU and passes its current LUID to winml perf. The perf pytest step uses that GPU while preserving the requested EP, including OpenVINO GPU. Both adapter-selection cases still execute inference independently through --device-luid and --ep-options device_id, and verify the monitor binding. Missing hardware/LUIDs and unrelated extra devices still fail; normal agents retain strict inventory and default-selection checks. No tests are skipped or marked xfail, and shipped product code is unchanged.
The keepalive installer also includes the enhanced worker already deployed on the agent: network-source filtering, Session 1 support, occupied-console protection, verified tscon retries, and cleanup limited to non-present RDP display nodes. The task now passes the source address directly, and the documentation explains that console recovery does not prove DML readiness.
Validation:
The pin is opt-in through dmlPinnedAgents (initially NPU-OV2); an empty list disables it. It does not guarantee that in-flight inference survives a driver reset during an RDP switch.