koboldcpp 1.117.1 high speed with 1 GPU and very low speed with 2 GPU

[xelercen]

koboldcpp 1.117.1
Windows 10 Ent
Nvidia driver 553.74 (CUDA Toolkit: 12.4)

Use 1 GPU: speed 140 Token/s, load GPU 90-100%
Use 1 and 2 GPU: speed 12 Token/s, load GPU 5-10%

**System use CPU + GPU?
I have very little RAM (4 GB), how can I disable CPU calculation? (this may not be the problem).
I set all GPUs to use and set the tensor split to 1,1.
I set the weight split to 1,1 (not the tensor split). I left the leaf split to automatic.
I tried setting the tensor split to 1,1, but it didn't help much.

Log consol:

---

Welcome to KoboldCpp - Version 1.117.1
For command line arguments, please run --help in the terminal.
Note: The GUI mode is not accessible to screen readers.

---

Auto Selected CUDA Backend (flag=0)

Loading Chat Completions Adapter: C:\Users\xxx\AppData\Local\Temp_MEI49522\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
Auto Recommended GPU Layers: 35
System: Windows 10.0.19044 AMD64 Intel64 Family 6 Model 167 Stepping 0, GenuineIntel
Detected Available GPU Memory: 23040 MB
Detected Available RAM: 2733 MB
Initializing dynamic library: koboldcpp_cublas.dll

Namespace(admin=False, admindir='', adminpassword='', adminunloadtimeout=0, allow_config_onready=False, analyze='', autofit=False, autofitpadding=1024, autoswapmode=False, baseconfig='', batchsize=512, benchmark=None, blasthreads=None, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=12288, debugmode=0, defaultgenamt=1536, device='', downloaddir='', draftamount=4, draftgpulayers=999, draftgpusplit=None, draftmodel=None, embeddingsgpu=False, embeddingsmaxctx=4096, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=False, forceversion=False, foreground=False, gendefaults='', gendefaultsoverwrite=False, genlimit=0, gpulayers=35, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='', ignoremissing=False, jinja=False, jinja_kwargs='', jinja_tools=False, jinjatemplate='', jinjathink='default', launch=True, lora=None, loramult=1.0, lowvram=False, maingpu=0, maxrequestsize=32, mcpfile=None, mmproj=None, mmprojcpu=False, model=[], model_param='C:/Users/xxx/Desktop/soft/Qwen3.6-35B-A3B-Q4_K_M.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=10, musicdiffusion='', musicembeddings='', musicllm='', musiclowvram=False, musicvae='', noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, noflashattention=False, nommap=False, nommq=True, nomodel=False, nopipelineparallel=False, noshift=False, noswa=False, onready='', overridekv=None, overridenativecontext=0, overridetensors=None, parallelrequests=1, password=None, pipelineparallel=False, port=5001, port_param=5001, preloadstory=None, prompt='', proxy_port=None, quantkv='q8_0', quiet=False, ratelimit=0, reasoningeffort='default', remotetunnel=False, reqtimeout=600, ropeconfig=[0.0, 10000.0], routermode=False, rpcdevice='', rpchost='0.0.0.0', rpcmode='disabled', rpcport=5551, rpctargets='', savedatafile=None, sdaudiovae='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipdevice=-2, sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults=False, sdlora=[], sdloramult=[1.0], sdmaingpu=-1, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=7, sdtiledvae=640, sdupscaler='', sdvae='', sdvaeauto=False, sdvaecpu=False, sdvaedevice=-1, sdvramlimit=0, showgui=False, singleinstance=False, skiplauncher=False, smartcache=0, smartcontext=False, splitmode='layer', ssl=None, swapadding=0, tensor_split=[1.0, 1.0], testmemory=False, threads=7, ttsdir='', ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=7, ttswavtokenizer='', unpack='', usecpu=False, usecuda=['normal'], usemlock=False, usemmap=False, usemtp=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, visionmaxtokens=-1, visionmintokens=-1, websearch=False, whispermodel='')

Loading Text Model: C:\Users\xxx\Desktop\soft\Qwen3.6-35B-A3B-Q4_K_M.gguf

The reported GGUF Arch is: qwen35moe
Arch Category: 33

---

Identified as GGUF model.
Attempting to Load...

Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 45875 MiB):
Device 0: Tesla P40, compute capability 6.1, VMM: no, VRAM: 22937 MiB
Device 1: Tesla P40, compute capability 6.1, VMM: no, VRAM: 22937 MiB
Number of Backends: 3
Backend 0: CUDA
Backend 1: RPC
Backend 2: CPU
CUDA MMQ: False

Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...

Applying Tensor Split...
llama_model_loader: loaded meta data with 41 key-value pairs and 733 tensors from C:\Users\xxx\Desktop\soft\Qwen3.6-35B-A3B-Q4_K_M.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file type = unknown, may not work
print_info: file size = 19.70 GiB (4.88 BPW)
llama_prepare_model_devices: using device CUDA0 (Tesla P40) (0000:02:00.0) - 22783 MiB free
llama_prepare_model_devices: using device CUDA1 (Tesla P40) (0000:03:00.0) - 22783 MiB free
init_tokenizer: initializing tokenizer for type 2
load: 0 unused tokens
load: setting token '' (248069) attribute to USER_DEFINED (16), old attributes: 16
load: setting token '' (248068) attribute to USER_DEFINED (16), old attributes: 16
load: printing all EOG tokens:
load: - 248044 ('<|endoftext|>')
load: - 248046 ('<|im_end|>')
load: - 248063 ('<|fim_pad|>')
load: - 248064 ('<|repo_name|>')
load: - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
print_info: arch = qwen35moe
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd_inp = 2048
print_info: n_embd = 2048
print_info: n_embd_out = 2048
print_info: n_layer = 40
print_info: n_layer_all = 40
print_info: n_head = 16
print_info: n_head_kv = 2
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 256
print_info: n_embd_head_v = 256
print_info: n_gqa = 8
print_info: n_embd_k_gqa = 512
print_info: n_embd_v_gqa = 512
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: f_attn_value_scale = 0.0000
print_info: n_ff = 0
print_info: n_expert = 256
print_info: n_expert_used = 8
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = -1
print_info: rope type = 40
print_info: rope scaling = linear
print_info: freq_base_train = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: mrope sections = [11, 11, 10, 0]
print_info: ssm_d_conv = 4
print_info: ssm_d_inner = 4096
print_info: ssm_d_state = 128
print_info: ssm_dt_rank = 32
print_info: ssm_n_group = 16
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 35B.A3B
print_info: model params = 34.66 B
print_info: general.name = Qwen_Qwen3.6 35B A3B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 248044 '<|endoftext|>'
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248044 '<|endoftext|>'
print_info: LF token = 198 'Д?'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
done_getting_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) moved from CUDA_Host, using CPU instead
ggml_cuda_host_malloc: failed to allocate 3067.92 MiB of pinned memory: out of memory
load_tensors: offloading output layer to GPU
load_tensors: offloading 34 repeating layers to GPU
load_tensors: offloaded 35/41 layers to GPU
load_tensors: CPU model buffer size = 272.81 MiB
load_tensors: CUDA0 model buffer size = 8567.04 MiB
load_tensors: CUDA1 model buffer size = 8267.94 MiB
load_tensors: CPU model buffer size = 3067.92 MiB
load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
...........................................load_all_data: using async uploads for device CUDA1, buffer type CUDA1, backend CUDA1
........................................................

This model does not use SWA
RNN or Hyrbid model with FF and shifting flags enabled - SmartCache will be enabled with extra slots. Disable CtxShift if you do not want this.
SmartCache: Prepared 7 KV slots

MRope is used, context shift will be disabled!
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 12544
llama_context: n_ctx_seq = 12544
llama_context: n_batch = 1024
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = enabled
llama_context: kv_unified = true
llama_context: freq_base = 10000000.0
llama_context: freq_scale = 1
llama_context: n_rs_seq = 0
llama_context: n_outputs_max = 1024
llama_context: n_ctx_seq (12544) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
set_abort_callback: call
llama_context: CUDA_Host output buffer size = 0.95 MiB
llama_kv_cache: CPU KV buffer size = 13.02 MiB
llama_kv_cache: CUDA0 KV buffer size = 65.08 MiB
llama_kv_cache: CUDA1 KV buffer size = 52.06 MiB
llama_kv_cache: size = 130.16 MiB ( 12544 cells, 10 layers, 1/1 seqs), K (q8_0): 65.08 MiB, V (q8_0): 65.08 MiB
llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 256
llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 256
llama_memory_recurrent: layer 3: skipped
llama_memory_recurrent: layer 7: skipped
llama_memory_recurrent: layer 11: skipped
llama_memory_recurrent: layer 15: skipped
llama_memory_recurrent: layer 19: skipped
llama_memory_recurrent: layer 23: skipped
llama_memory_recurrent: layer 27: skipped
llama_memory_recurrent: layer 31: skipped
llama_memory_recurrent: layer 35: skipped
llama_memory_recurrent: layer 39: skipped
llama_memory_recurrent: CPU RS buffer size = 10.47 MiB
llama_memory_recurrent: CUDA0 RS buffer size = 27.22 MiB
llama_memory_recurrent: CUDA1 RS buffer size = 25.12 MiB
llama_memory_recurrent: size = 62.81 MiB ( 1 cells, 40 layers, 1 seqs 0 rs_seq), R (f32): 2.81 MiB, S (f32): 60.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
sched_reserve: reserving ...
sched_reserve: max_nodes = 23456
sched_reserve: reserving full memory module
sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
sched_reserve: resolving fused Gated Delta Net support:
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve: CUDA0 compute buffer size = 299.25 MiB
sched_reserve: CUDA1 compute buffer size = 497.00 MiB
sched_reserve: CUDA_Host compute buffer size = 32.63 MiB
sched_reserve: graph nodes = 3847
sched_reserve: graph splits = 131 (with bs=512), 13 (with bs=1)
sched_reserve: reserve took 178.24 ms, sched copies = 1
Threadpool set to 7 threads and 7 blasthreads...
attach_threadpool: call

This architecture has explicitly disabled the BOS token - if you need it, you must add it manually.
Starting model warm up, please wait a moment...
Load Text Model OK: True
Chat completion heuristic: ChatML (Generic)
Embedded KoboldAI Lite loaded.
Embedded API docs loaded.
Llama.cpp UI loaded.

Active Modules: TextGeneration
Inactive Modules: ImageGeneration VoiceRecognition MultimodalVision MultimodalAudio NetworkMultiplayer ApiKeyPassword WebSearchProxy TextToSpeech VectorEmbeddings AdminControl MCPBridge MusicGen RouterMode
Enabled APIs: KoboldCppApi OpenAiApi OllamaApi AnthropicApi
Note: For third party Ollama API Emulation, you should set the port to 11434.
Starting Kobold API on port 5001 at http://localhost:5001/api/
Starting OpenAI Compatible API on port 5001 at http://localhost:5001/v1/
Starting llama.cpp secondary WebUI at http://localhost:5001/lcpp/

Please connect to custom endpoint at http://localhost:5001

Input: {"n": 1, "max_context_length": 12288, "max_length": 1024, "rep_pen": 1.05, "temperature": 0.75, "top_p": 0.92, "top_k": 100, "top_a": 0, "typical": 1, "tfs": 1, "rep_pen_range": 360, "rep_pen_slope": 0.7, "sampler_order": [6, 0, 1, 3, 4, 2, 5], "memory": "", "trim_stop": true, "genkey": "KCPP6598", "min_p": 0, "dynatemp_range": 0, "dynatemp_exponent": 1, "smoothing_factor": 0, "smoothing_curve": 1, "nsigma": 0, "banned_tokens": [], "render_special": false, "logprobs": false, "replace_instruct_placeholders": true, "presence_penalty": 0, "logit_bias": {}, "adaptive_target": -1, "adaptive_decay": 0.9, "stop_sequence": ["{{[INPUT]}}", "{{[OUTPUT]}}"], "use_default_badwordsids": false, "bypass_eos": false, "prompt": "{{[INPUT]}}Why availability llm for articles?{{[OUTPUT]}}"}

Processing Prompt (15 / 15 tokens)
Generating (54 / 1024 tokens)
[22:31:44] CtxLimit:69/12288, Init:0.15s, Processed:15 in 3.27s (4.58T/s), Generated:54/1024 in 6.27s (8.61T/s), Total:9.70s
Output:
Here's a thinking process:

1. Analyze User Input:

• Query: "Why availability llm for articles?"
• Key Concepts:
  • "availability" (likely referring to whether LLMs are

Generation Aborted

Input: {"n": 1, "max_context_length": 12288, "max_length": 1024, "rep_pen": 1.05, "temperature": 0.75, "top_p": 0.92, "top_k": 100, "top_a": 0, "typical": 1, "tfs": 1, "rep_pen_range": 360, "rep_pen_slope": 0.7, "sampler_order": [6, 0, 1, 3, 4, 2, 5], "memory": "", "trim_stop": true, "genkey": "KCPP2104", "min_p": 0, "dynatemp_range": 0, "dynatemp_exponent": 1, "smoothing_factor": 0, "smoothing_curve": 1, "nsigma": 0, "banned_tokens": [], "render_special": false, "logprobs": false, "replace_instruct_placeholders": true, "presence_penalty": 0, "logit_bias": {}, "adaptive_target": -1, "adaptive_decay": 0.9, "stop_sequence": ["{{[INPUT]}}", "{{[OUTPUT]}}"], "use_default_badwordsids": false, "bypass_eos": false, "prompt": "{{[INPUT]}}Why availability llm for articles?\n{{[OUTPUT]}}"}

[SmartCache RNN No Match, Saving into slot 5...]
state_write_data: writing state
state_write_data: - writing memory module
state_write_data: writing state
state_write_data: - writing memory module

KV Save State 5: Created SaveState of 69 tokens, costing 63 MB.

Processing Prompt (16 / 16 tokens)
Generating (214 / 1024 tokens)
[22:32:26] CtxLimit:230/12288, Init:0.12s, Processed:16 in 0.53s (30.36T/s), Generated:214/1024 in 17.61s (12.16T/s), Total:18.25s
Output:
Here's a thinking process:

1. Analyze User Input:

• Query: "Why availability llm for articles?"
• Key Concepts: "availability", "LLM" (Large Language Model), "articles"
• Interpretation: The user is asking why Large Language Models (LLMs) are available/accessible for use with articles, or perhaps why there's a need/availability of LLMs specifically for article-related tasks (like writing, summarizing, analyzing, etc.). The phrasing is a bit fragmented, so I need to clarify the intent while addressing the core question.

2. Identify Core Questions/Intent:

• Why are LLMs available for article-related tasks?
• What makes LLMs suitable/useful for articles?
• Why has the industry developed/made LLMs accessible for this purpose?
• Could also mean: "Why are LLMs becoming widely available for processing/writing articles?"

Generation Aborted

[xelercen]

I checked, it turned out that some of the layers are running on the CPU, fixed it, now the speed is 32 tokens per second.

---

Welcome to KoboldCpp - Version 1.117.1
For command line arguments, please run --help in the terminal.
Note: The GUI mode is not accessible to screen readers.

---

Auto Selected CUDA Backend (flag=0)

Loading Chat Completions Adapter: C:\Users\xxx\AppData\Local\Temp_MEI46042\kcpp_adapters\AutoGuess.json
Chat Completions Adapter Loaded
System: Windows 10.0.19044 AMD64 Intel64 Family 6 Model 167 Stepping 0, GenuineIntel
Detected Available GPU Memory: 23040 MB
Detected Available RAM: 2614 MB
Initializing dynamic library: koboldcpp_cublas.dll

Namespace(admin=False, admindir='', adminpassword='', adminunloadtimeout=0, allow_config_onready=False, analyze='', autofit=False, autofitpadding=1024, autoswapmode=False, baseconfig='', batchsize=512, benchmark=None, blasthreads=None, chatcompletionsadapter='AutoGuess', cli=False, config=None, contextsize=12288, debugmode=0, defaultgenamt=1536, device='', downloaddir='', draftamount=4, draftgpulayers=999, draftgpusplit=None, draftmodel=None, embeddingsgpu=False, embeddingsmaxctx=4096, embeddingsmodel='', enableguidance=False, exportconfig='', exporttemplate='', failsafe=False, flashattention=False, forceversion=False, foreground=False, gendefaults='', gendefaultsoverwrite=False, genlimit=0, gpulayers=41, highpriority=False, hordeconfig=None, hordegenlen=0, hordekey='', hordemaxctx=0, hordemodelname='', hordeworkername='', host='', ignoremissing=False, jinja=False, jinja_kwargs='', jinja_tools=False, jinjatemplate='', jinjathink='default', launch=True, lora=None, loramult=1.0, lowvram=False, maingpu=0, maxrequestsize=32, mcpfile=None, mmproj=None, mmprojcpu=False, model=[], model_param='C:/Users/xxx/Desktop/soft/Qwen3.6-35B-A3B-Q4_K_M.gguf', moecpu=0, moeexperts=-1, multiplayer=False, multiuser=10, musicdiffusion='', musicembeddings='', musicllm='', musiclowvram=False, musicvae='', noavx2=False, noblas=False, nobostoken=False, nocertify=False, nofastforward=False, noflashattention=False, nommap=False, nommq=True, nomodel=False, nopipelineparallel=False, noshift=False, noswa=False, onready='', overridekv=None, overridenativecontext=0, overridetensors=None, parallelrequests=1, password=None, pipelineparallel=False, port=5001, port_param=5001, preloadstory=None, prompt='', proxy_port=None, quantkv='q8_0', quiet=False, ratelimit=0, reasoningeffort='default', remotetunnel=False, reqtimeout=600, ropeconfig=[0.0, 10000.0], routermode=False, rpcdevice='', rpchost='0.0.0.0', rpcmode='disabled', rpcport=5551, rpctargets='', savedatafile=None, sdaudiovae='', sdclamped=0, sdclampedsoft=0, sdclip1='', sdclip2='', sdclipdevice=-2, sdclipgpu=False, sdconfig=None, sdconvdirect='off', sdflashattention=False, sdgendefaults=False, sdlora=[], sdloramult=[1.0], sdmaingpu=-1, sdmodel='', sdnotile=False, sdoffloadcpu=False, sdphotomaker='', sdquant=0, sdt5xxl='', sdthreads=7, sdtiledvae=640, sdupscaler='', sdvae='', sdvaeauto=False, sdvaecpu=False, sdvaedevice=-1, sdvramlimit=0, showgui=False, singleinstance=False, skiplauncher=False, smartcache=0, smartcontext=False, splitmode='layer', ssl=None, swapadding=0, tensor_split=[1.0, 1.0], testmemory=False, threads=7, ttsdir='', ttsgpu=False, ttsmaxlen=4096, ttsmodel='', ttsthreads=7, ttswavtokenizer='', unpack='', usecpu=False, usecuda=['normal'], usemlock=False, usemmap=False, usemtp=False, useswa=False, usevulkan=None, version=False, visionmaxres=1024, visionmaxtokens=-1, visionmintokens=-1, websearch=False, whispermodel='')

Loading Text Model: C:\Users\xxx\Desktop\soft\Qwen3.6-35B-A3B-Q4_K_M.gguf

The reported GGUF Arch is: qwen35moe
Arch Category: 33

---

Identified as GGUF model.
Attempting to Load...

Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 45875 MiB):
Device 0: Tesla P40, compute capability 6.1, VMM: no, VRAM: 22937 MiB
Device 1: Tesla P40, compute capability 6.1, VMM: no, VRAM: 22937 MiB
Number of Backends: 3
Backend 0: CUDA
Backend 1: RPC
Backend 2: CPU
CUDA MMQ: False

Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...

Applying Tensor Split...
llama_model_loader: loaded meta data with 41 key-value pairs and 733 tensors from C:\Users\xxx\Desktop\soft\Qwen3.6-35B-A3B-Q4_K_M.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file type = unknown, may not work
print_info: file size = 19.70 GiB (4.88 BPW)
llama_prepare_model_devices: using device CUDA0 (Tesla P40) (0000:02:00.0) - 22783 MiB free
llama_prepare_model_devices: using device CUDA1 (Tesla P40) (0000:03:00.0) - 22783 MiB free
init_tokenizer: initializing tokenizer for type 2
load: 0 unused tokens
load: setting token '' (248069) attribute to USER_DEFINED (16), old attributes: 16
load: setting token '' (248068) attribute to USER_DEFINED (16), old attributes: 16
load: printing all EOG tokens:
load: - 248044 ('<|endoftext|>')
load: - 248046 ('<|im_end|>')
load: - 248063 ('<|fim_pad|>')
load: - 248064 ('<|repo_name|>')
load: - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
print_info: arch = qwen35moe
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd_inp = 2048
print_info: n_embd = 2048
print_info: n_embd_out = 2048
print_info: n_layer = 40
print_info: n_layer_all = 40
print_info: n_head = 16
print_info: n_head_kv = 2
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 256
print_info: n_embd_head_v = 256
print_info: n_gqa = 8
print_info: n_embd_k_gqa = 512
print_info: n_embd_v_gqa = 512
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: f_attn_value_scale = 0.0000
print_info: n_ff = 0
print_info: n_expert = 256
print_info: n_expert_used = 8
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = -1
print_info: rope type = 40
print_info: rope scaling = linear
print_info: freq_base_train = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: mrope sections = [11, 11, 10, 0]
print_info: ssm_d_conv = 4
print_info: ssm_d_inner = 4096
print_info: ssm_d_state = 128
print_info: ssm_dt_rank = 32
print_info: ssm_n_group = 16
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 35B.A3B
print_info: model params = 34.66 B
print_info: general.name = Qwen_Qwen3.6 35B A3B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 248044 '<|endoftext|>'
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248044 '<|endoftext|>'
print_info: LF token = 198 'Д?'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
done_getting_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) moved from CUDA_Host, using CPU instead
load_tensors: offloading output layer to GPU
load_tensors: offloading 39 repeating layers to GPU
load_tensors: offloaded 41/41 layers to GPU
load_tensors: CPU model buffer size = 272.81 MiB
load_tensors: CUDA0 model buffer size = 10206.43 MiB
load_tensors: CUDA1 model buffer size = 9696.47 MiB
load_all_data: using async uploads for device CUDA0, buffer type CUDA0, backend CUDA0
..................................................load_all_data: using async uploads for device CUDA1, buffer type CUDA1, backend CUDA1
...............................................

This model does not use SWA
RNN or Hyrbid model with FF and shifting flags enabled - SmartCache will be enabled with extra slots. Disable CtxShift if you do not want this.
SmartCache: Prepared 7 KV slots

MRope is used, context shift will be disabled!
Automatic RoPE Scaling: Using model internal value.
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 12544
llama_context: n_ctx_seq = 12544
llama_context: n_batch = 1024
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = enabled
llama_context: kv_unified = true
llama_context: freq_base = 10000000.0
llama_context: freq_scale = 1
llama_context: n_rs_seq = 0
llama_context: n_outputs_max = 1024
llama_context: n_ctx_seq (12544) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
set_abort_callback: call
llama_context: CUDA_Host output buffer size = 0.95 MiB
llama_kv_cache: CUDA0 KV buffer size = 65.08 MiB
llama_kv_cache: CUDA1 KV buffer size = 65.08 MiB
llama_kv_cache: size = 130.16 MiB ( 12544 cells, 10 layers, 1/1 seqs), K (q8_0): 65.08 MiB, V (q8_0): 65.08 MiB
llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 256
llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 256
llama_memory_recurrent: layer 3: skipped
llama_memory_recurrent: layer 7: skipped
llama_memory_recurrent: layer 11: skipped
llama_memory_recurrent: layer 15: skipped
llama_memory_recurrent: layer 19: skipped
llama_memory_recurrent: layer 23: skipped
llama_memory_recurrent: layer 27: skipped
llama_memory_recurrent: layer 31: skipped
llama_memory_recurrent: layer 35: skipped
llama_memory_recurrent: layer 39: skipped
llama_memory_recurrent: CUDA0 RS buffer size = 33.50 MiB
llama_memory_recurrent: CUDA1 RS buffer size = 29.31 MiB
llama_memory_recurrent: size = 62.81 MiB ( 1 cells, 40 layers, 1 seqs 0 rs_seq), R (f32): 2.81 MiB, S (f32): 60.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: pipeline parallelism enabled
sched_reserve: reserving ...
sched_reserve: max_nodes = 23456
sched_reserve: reserving full memory module
sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
sched_reserve: resolving fused Gated Delta Net support:
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve: CUDA0 compute buffer size = 109.56 MiB
sched_reserve: CUDA1 compute buffer size = 530.07 MiB
sched_reserve: CUDA_Host compute buffer size = 33.07 MiB
sched_reserve: graph nodes = 3847
sched_reserve: graph splits = 3
sched_reserve: reserve took 94.44 ms, sched copies = 2
Threadpool set to 7 threads and 7 blasthreads...
attach_threadpool: call

This architecture has explicitly disabled the BOS token - if you need it, you must add it manually.
Starting model warm up, please wait a moment...
Load Text Model OK: True
Chat completion heuristic: ChatML (Generic)
Embedded KoboldAI Lite loaded.
Embedded API docs loaded.
Llama.cpp UI loaded.

Active Modules: TextGeneration
Inactive Modules: ImageGeneration VoiceRecognition MultimodalVision MultimodalAudio NetworkMultiplayer ApiKeyPassword WebSearchProxy TextToSpeech VectorEmbeddings AdminControl MCPBridge MusicGen RouterMode
Enabled APIs: KoboldCppApi OpenAiApi OllamaApi AnthropicApi
Note: For third party Ollama API Emulation, you should set the port to 11434.
Starting Kobold API on port 5001 at http://localhost:5001/api/
Starting OpenAI Compatible API on port 5001 at http://localhost:5001/v1/
Starting llama.cpp secondary WebUI at http://localhost:5001/lcpp/

Please connect to custom endpoint at http://localhost:5001

Input: {"n": 1, "max_context_length": 12288, "max_length": 1024, "rep_pen": 1.05, "temperature": 0.75, "top_p": 0.92, "top_k": 100, "top_a": 0, "typical": 1, "tfs": 1, "rep_pen_range": 360, "rep_pen_slope": 0.7, "sampler_order": [6, 0, 1, 3, 4, 2, 5], "memory": "", "trim_stop": true, "genkey": "KCPP4677", "min_p": 0, "dynatemp_range": 0, "dynatemp_exponent": 1, "smoothing_factor": 0, "smoothing_curve": 1, "nsigma": 0, "banned_tokens": [], "render_special": false, "logprobs": false, "replace_instruct_placeholders": true, "presence_penalty": 0, "logit_bias": {}, "adaptive_target": -1, "adaptive_decay": 0.9, "stop_sequence": ["{{[INPUT]}}", "{{[OUTPUT]}}"], "use_default_badwordsids": false, "bypass_eos": false, "prompt": "{{[INPUT]}}Why availability llm for articles?{{[OUTPUT]}}"}

Processing Prompt (15 / 15 tokens)
Generating (1024 / 1024 tokens)
[08:59:12] CtxLimit:1039/12288, Init:0.00s, Processed:15 in 0.25s (60.00T/s), Generated:1024/1024 in 31.70s (32.30T/s), Total:31.96s
Output:
Here's a thinking process:

1. Analyze User Input:

• Query: "Why availability llm for articles?"
• Key Concepts: "availability", "llm" (Large Language Model), "articles"
• Interpretation: The user is asking about the availability of LLMs for use with/processing/creating articles. This could mean:
  • Why are LLMs available for article-related tasks?
  • Where/how can I access LLMs for articles?
  • What's the reason behind the current availability of LLMs in the context of articles/content creation?
  • Is there a specific platform/tool they're referring to?
• Grammar/Clarity: The phrasing is a bit fragmented. It likely means "Why are LLMs available for articles?" or "What's the availability of LLMs for article-related tasks?"

2. Identify Core Questions to Address:

• Why are LLMs widely available now?
• How do LLMs relate to articles (creation, analysis, editing, summarization, etc.)?
• What factors drove this availability?
• Where can users access them?
• Are there limitations or considerations?

3. Brainstorming Key Points:

• Technical Advances: Open-source models (Llama, Mistral, Qwen), improved training data, better inference efficiency.
• Commercial Demand: High demand for content generation, SEO, research, academic writing, journalism.
• Infrastructure: Cloud APIs (OpenAI, Anthropic, Google, AWS, Azure), open-weight models, local deployment options.
• Use Cases for Articles: Drafting, editing, summarizing, fact-checking, translation, SEO optimization, research assistance.
• Why Now? Democratization of AI, competitive market, reduced compute costs, community contributions, regulatory/ethical frameworks evolving.
• Access Methods: Web interfaces, APIs, local installs (Ollama, LM Studio), WordPress/Notion integrations, specialized platforms.
• Considerations: Accuracy issues, bias, cost, privacy, need for human oversight.

4. Structure the Response:

• Clarify the question first
• Explain why LLMs are widely available for article-related tasks
• List key drivers/factors
• Describe how they're used with articles
• Provide access options
• Note limitations/best practices
• Keep it concise and actionable

5. Draft Response (Mental Refinement):
   Your question seems to ask why Large Language Models (LLMs) are now widely available for article-related tasks like writing, editing, summarizing, or research. Here’s why:

1. Rapid Technical & Open-Source Advances
Models like Llama, Mistral, Qwen, and others are open-weight or freely accessible, lowering barriers to entry. Improved training techniques and efficient inference make them practical for everyday use.

2. High Commercial & Creative Demand
Publishers, journalists, academics, and marketers need help drafting, optimizing SEO, summarizing research, and translating content. LLMs directly address these workflows.

3. Cloud Infrastructure & APIs
Providers (OpenAI, Anthropic, Google, AWS, Azure, etc.) offer reliable, scalable APIs. Local tools (Ollama, LM Studio, Hugging Face) enable offline/private use.

4. Integration Ecosystem
LLMs are built into writing tools (Notion, WordPress, Grammarly, Jasper), browsers, and research platforms, making them “available by default” for article workflows.

How to Access Them for Articles:

• Web/API: OpenAI, Claude, Gemini, Perplexity, Poe
• Free/Open: Hugging Face, Ollama, LM Studio, OpenRouter
• Integrated: Writer.com, Copy.ai, SurferSEO, Hemingway + AI plugins

Important Considerations:

• LLMs can hallucinate—always fact-check.
• Tone/style may need human refinement.
• Copyright & originality policies vary by platform.
• Privacy: Avoid sharing sensitive/unpublished drafts with public APIs.

If you meant a specific platform, use case, or region’s availability, let me know and I’ll tailor the answer!

6. Self-Correction/Verification:

• Is the response accurate? Yes, covers technical, commercial, infrastructural, and practical aspects.
• Does it address "availability"? Yes, explains why they're available now and how to access them.
• Is it clear given the fragmented query? Yes, acknowledges ambiguity and invites clarification.

[LostRuins]

Glad you got it working. Generally, you can simply enable autofit and let it handle the offload automatically.