rocm: model weights in host RAM on integrated APUs (Strix Halo) - #1028
Open
Piega wants to merge 1 commit into
Open
rocm: model weights in host RAM on integrated APUs (Strix Halo)#1028Piega wants to merge 1 commit into
Piega wants to merge 1 commit into
Conversation
Piega
force-pushed
the
strixhalo-host-ram-weights
branch
2 times, most recently
from
September 12, 2026 21:23
663462c to
8b46251
Compare
Strix Halo (gfx1151) exposes a small device memory pool (~62 GiB set by the BIOS UMA frame buffer), so a resident DeepSeek V4 Flash Q2 (80.76 GiB) failed with OOM while preparing model tensor spans. On integrated APUs the model arena now allocates via hipHostMalloc (fallback hipMallocManaged), mirroring llama.cpp GGML_HIP_UMA: weights live in host RAM and stay GPU visible over the UMA fabric, bypassing the device pool ceiling. Discrete GPUs are unchanged; DS4_ROCM_FORCE_DEVICE_MEM restores previous behavior. Verified on Bosgame M5 (Ryzen AI Max+ 395, Radeon 8060S, 128 GB, ROCm 10): 80.76 GiB resident in ~30 s, prefill ~273-284 tok/s (ctx 4-8k), steady generation ~13 tok/s.
Piega
force-pushed
the
strixhalo-host-ram-weights
branch
from
September 12, 2026 21:24
8b46251 to
c8b47e5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes OOM loading resident DeepSeek V4 Flash Q2 (80.76 GiB) on Strix Halo systems, and unlocks the other 128 GB UMA resident models on ROCm.
Problem
The ROCm model arena allocates weights with
hipMalloc, which on integrated APUs is bound by the small device memory pool exposed to the GPU (~62 GiB, set by the BIOS UMA frame buffer). A resident Flash Q2 (80.76 GiB) therefore failed during tensor-span preparation:Fix
On integrated APUs, allocate model arena chunks via
hipHostMalloc(falling back tohipMallocManaged), mirroring how llama.cpp'sGGML_HIP_UMAkeeps weights in host RAM while they stay GPU-visible over the UMA fabric. Discrete GPUs keep the existinghipMallocpath untouched;DS4_ROCM_FORCE_DEVICE_MEM=1restores the previous behavior.Verified
Bosgame M5 (Ryzen AI Max+ 395, Radeon 8060S, 128 GB, ROCm 10.0, this branch).
DeepSeek V4 Flash models, resident:
All load in ~30 s; KV is 0.41-0.46 GiB. GLM 5.3 Flash Q2 also runs resident (90 GiB) with
DS4_GLM_MEMORY_GUARD=0(the guard budgets from the ~61 GiB device pool); vision mode (V4 Flash Vision-Exp + encoder) works on ROCm.Note: this approach is complementary to #361 (opt-in
DS4_CUDA_MANAGEDfor high-context OOM on large-carveout UMA APUs): #361 addresses long-context scratch by defaulting weight/scratch placement policy on demand, while the change here makes resident weights work on integrated APUs out of the box by routing the weight arena to host memory.