Koboldcpp uses significant system RAM despite having enough VRAM

[interested-user-1]

Before loading Gemma-4-E4B QAT Q4_0, my VRAM is at 0.1/8.0GB. After loading it, it jumps to 5.0/8.0GB (expected), while using 3.0GB of system RAM (the Koboldcpp program, not the entire system).

It doesn't seem to matter whether it's generating tokens at the moment because a few minutes ago it was at 3.0GB system RAM, but now it's at 81MB, but it changes from time to time. Sometimes it stays at 3.0GB until I restart.

I don't think it's due to a limitation in VRAM, because when I started generating, the VRAM stayed around 5.0GB and system RAM increased from 81MB to 91MB but no higher.

Settings:

• SWA enabled, SWA Padding tokens = 0
• Flash Attention enabled
• KV Cache f16
• Fast Forwarding disabled, Context Shift disabled, SmartCache disabled

I'm wondering if this had anything to do with it:

done_getting_tensors: tensor 'token_embd.weight' (q6_K) (and 1 others) moved from CUDA_Host, using CPU instead
load_tensors: offloading output layer to GPU
load_tensors: offloading 41 repeating layers to GPU
load_tensors: offloaded 43/43 layers to GPU
load_tensors:          CPU model buffer size =  2730.00 MiB

Hardware and Drivers: Nvidia 5060 (mobile), GeForce Game Ready Driver version 610.74

Let me know if more information is needed

[LostRuins]

E4B does use quite a bit of RAM for the KV cache, if i remember correctly. You can confirm this by loading a different model with the same settings, e.g. Mistral

[aelfwyne]

I just encountered this exact issue.

It is 100% NOT a lack of vram problem. In my case the vram usage was 19.2 out of 32.0gb, and the system spill was 5.1 gb on a 31b gemma.

It was even spilling on 12b gemma with MTP loaded on a QAT gguf. There is definitely a serious problem where it's using system ram when there's plenty of vram available.

[aelfwyne]

After further testing, I've narrowed this down specifically to unsloth GGUF's. QAT/MTP is the worst, UD/MTP has some spill, and even plain UD unsloth without MTP is having spillover, as well as buffer size errors when there are multiples of GB free. "WARNING: Requested buffer size (4638400288) exceeds device max_buffer_size limit (2147483648)!". These errors do not present when using an identical size and quant model from a different source (non unsloth).

[interested-user-1]

I'm experiencing this issue on the models Google quantised itself though: https://huggingface.co/google/gemma-4-E4B-it-qat-q4_0-gguf

Also note that Google has recently released new versions of the models under the same name. https://www.reddit.com/r/LocalLLaMA/comments/1uxfu4k/google_is_updating_gemma_4s_chat_templates/ That's what I was using. Their update seems to have broken Kobold's preset system prompts for the E4B non-thinking too. (It occasionally adds a /turn tag at the end of responses and there are some other quirks)

[interested-user-1]

I think I've confirmed that it's not due to KV Cache, at least to my knowledge, it depends on context, so, I currently have 6k/7k context filled with Gemma-4-E4B and RAM usage remains at 141MB (one of the times it's not 3GB). VRAM usage also remains at 5.1GB, so neither RAM nor VRAM has increased significantly. It seems random to me whether it will use 3.0+GBs of RAM or just around 100MB

Do correct me if I'm wrong about KV Cache, iirc it's saved context and since I'm near the limit (I always use 7K max with this model) and the RAM usage isn't at 3GB like it sometimes gets, I don't think KV Cache is the reason behind this. Plus, I'd been told that VRAM for context is reserved on loading anyway (not sure if it counts for KV Cache too though).

[aelfwyne]

I think I've confirmed that it's not due to KV Cache, at least to my knowledge, it depends on context, so, I currently have 6k/7k context filled with Gemma-4-E4B and RAM usage remains at 141MB (one of the times it's not 3GB). VRAM usage also remains at 5.1GB, so neither RAM nor VRAM has increased significantly. It seems random to me whether it will use 3.0+GBs of RAM or just around 100MB

I have a similar experience. Sometimes it's a massive spill, but frequently I see small amounts of a few hundred megabytes still hit shared GPU, even when the massive spill doesn't happen. It seems Unsloth triggers it to be the massive GB leak over into system ram, but even other models are touching system ram more than they should.

[LostRuins]

I don't think there's anything unusual set up for this model. Could you give the rolling release a try? https://github.com/LostRuins/koboldcpp/releases/tag/rolling