Skip to content

[LAPACK][cuSOLVER] Fix incorrect QR results on repeated runs (#626) - #748

Open
zjin-lcf wants to merge 1 commit into
uxlfoundation:developfrom
zjin-lcf:fix/cusolver-qr-stream-sync-626
Open

[LAPACK][cuSOLVER] Fix incorrect QR results on repeated runs (#626)#748
zjin-lcf wants to merge 1 commit into
uxlfoundation:developfrom
zjin-lcf:fix/cusolver-qr-stream-sync-626

Conversation

@zjin-lcf

@zjin-lcf zjin-lcf commented Jul 28, 2026

Copy link
Copy Markdown

Summary

Fixes #626.

QR decomposition (geqrf + orgqr) on the cuSOLVER backend returned incorrect results on repeated runs: for a diagonal input matrix the diagonal of Q should be 1, but from the second run onwards (for n >= 256) some entries stayed at the input value 2. The corruption was nondeterministic and size-dependent, and did not occur on the Intel backends.

Root cause

The Householder-family cuSOLVER routines (geqrf, orgqr, ormqr, gebrd, orgbr, orgtr, ormtr, and the complex ung*/unm* variants) passed nullptr for cuSOLVER's devInfo argument and skipped the lapack_info_check call that every other routine (getrf, potrf, …) performs.

Beyond losing all error reporting, this also skipped the implicit synchronization that lapack_info_check performs: it reads devInfo back through a blocking queue.wait(). Without that wait, the SYCL event returned from the native-command submission could signal before the cuSOLVER kernels had actually finished, so the subsequent memcpy read partially-computed data — producing the observed run-2+ corruption.

Fix

  • Allocate a real devInfo and call lapack_info_check in all of these routines (both buffer and USM paths), matching the established pattern used by the working routines. This restores error checking and removes the race.
  • Align CusolverScopedContextHandler::get_stream with the cuBLAS backend by returning the interop handle's native queue (ih.get_native_queue()) instead of the queue's default stream.

Test plan — validated end-to-end on NVIDIA sm_103 device, CUDA 13.2

Built oneMath with -DENABLE_CUSOLVER_BACKEND=ON -DENABLE_CUBLAS_BACKEND=ON -DTARGET_DOMAINS=lapack using an open-source DPC++ toolchain (intel/llvm nightly, CUDA backend), and ran the #626 reproducer (QR of a diagonal matrix, checking Q diagonals across repeated runs):

Build n=512, 6 runs
develop (unpatched) FAIL — run 2+ have dozens of wrong diagonal entries
This PR PASS — all runs, all diagonals == 1

Also verified n = 256, 512, 1024 all pass with the fix.

Attribution note for reviewers

I isolated the two changes on the device:

  • devInfo / lapack_info_check change only (stream change reverted): PASS.
  • get_stream change only (devInfo change reverted): FAIL — still races.

So the devInfo/lapack_info_check addition is what actually resolves #626 (via the synchronization it introduces and by matching the working routines). The get_stream change is included as a correctness/consistency alignment with the cuBLAS backend, not as the fix itself; happy to drop it if maintainers prefer a minimal change.

  • End-to-end reproduce + fix (n=256/512/1024).
  • devInfo/lapack_info_check counts balanced across buffer and USM paths.
  • CI: cuSOLVER LAPACK functional tests.

Suggestion: add a functional test that runs geqrf+orgqr repeatedly on a known-diagonal matrix to guard against regressions.

The Householder-family cuSOLVER routines (geqrf, orgqr, ormqr, gebrd,
orgbr, orgtr, ormtr and the complex ung*/unm* variants) passed nullptr
for cuSOLVER's devInfo argument and skipped the lapack_info_check that
every other routine (getrf, potrf, ...) performs.

Besides losing all error reporting, this omitted the implicit
synchronization that lapack_info_check performs (it reads devInfo back
via a blocking queue.wait()). Without it, the SYCL event returned from
the native-command submission could signal before the cuSOLVER kernels
had finished, so a subsequent memcpy read partially-computed data. This
produced nondeterministic, size-dependent wrong results - e.g. QR of a
diagonal matrix returning Q diagonals stuck at the input value on the
second and later runs for n >= 256 (issue uxlfoundation#626).

Allocate a real devInfo and call lapack_info_check in all of these
routines (buffer and USM paths), matching the established pattern. This
both restores error checking and removes the race.

Also align CusolverScopedContextHandler::get_stream with the cuBLAS
backend by returning the interop handle's native queue
(ih.get_native_queue()) instead of the queue's default stream, so
cuSOLVER work is enqueued on the stream the SYCL runtime tracks for
native-command completion.

Fixes uxlfoundation#626.
@zjin-lcf
zjin-lcf requested a review from a team as a code owner July 28, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[cuSOLVER] Incorrect results in QR decomposition on CUDA

1 participant