Qwen3.8 27B Q5_K_M + 7900xtx + DeepSeek Harness实战平均 52 t/s

[exllm]

@6cccccc 你BIOS开above 4G decoding了吗？

[terry]

@exllm 开不开影响都不大，我实测没感觉出来区别，但是有大神测了有区别。

[懒人烘培]

@6cccccc
刚才实际测试了，如果开启Thinking了，速度也只有16，把Thinking Off后，速度我就恢复了70了。注意

[Michael Gill]

@exllm @terry 感谢二位，以为本地部署没啥希望，刚好是7900xtx，简单测试

[farmer node]

此主題已被删除！

[6cccccc]

@exllm 我一会试试

[6cccccc]

@懒人烘培 嗯，我一会试试，好像开了后，mtp命中率很低

[laobenxiong]

感谢楼主. 7900xtx 能跑,不错. 明天试试dsh. 今天dsh+v4pro花了100多, 肉疼(不过确实强, 长链自主调试FPGA, 综合+ila probe+烧写+测试+uart输出分析一条龙, 我基本可以不用管, 花token就行).

bruin@lmde7 ~ $ ./run-3.8-q5.sh
0.00.035.069 W Setting 'enable_thinking' via --chat-template-kwargs is deprecated. Use --reasoning on / --reasoning off instead.
0.00.035.097 W DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead
0.00.039.005 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.039.491 W srv  llama_server: -----------------
0.00.039.495 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.039.495 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.039.495 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.039.495 W srv  llama_server: -----------------
0.00.040.765 I srv    load_model: loading model '/opt/gguf-models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-Q5_K_S.gguf'
0.08.836.391 I cmn          init: llama threadpool init, n_threads = 10
0.09.247.607 I common_speculative_init_result: creating MTP draft context against the target model '/opt/gguf-models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-Q5_K_S.gguf'
0.09.318.357 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 131072, kv_unified = 'true'
0.09.457.688 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.09.457.746 I srv  llama_server: model loaded
0.09.457.750 I srv  llama_server: listening on http://0.0.0.0:8000
0.31.502.907 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.31.503.144 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.33.674.917 I slot print_timing: id  0 | task 0 | prompt eval time =    1607.08 ms /   333 tokens (    4.83 ms per token,   207.21 tokens per second)
0.33.674.922 I slot print_timing: id  0 | task 0 |        eval time =     564.47 ms /    34 tokens (   17.11 ms per token,    58.46 tokens per second)
0.33.674.923 I slot print_timing: id  0 | task 0 |       total time =    2171.56 ms /   367 tokens
0.33.674.928 I slot print_timing: id  0 | task 0 |    graphs reused =         11
0.33.674.930 I slot print_timing: id  0 | task 0 | draft acceptance = 0.72727 (   24 accepted /    33 generated), mean len =  3.18
0.33.674.999 I slot      release: id  0 | task 0 | stop processing: n_tokens = 368, truncated = 0
0.39.250.971 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.963 (> 0.100 thold), f_keep = 1.000
0.39.251.208 I slot launch_slot_: id  0 | task 16 | processing task, is_child = 0
0.40.592.295 I slot print_timing: id  0 | task 16 | prompt eval time =     368.90 ms /    14 tokens (   26.35 ms per token,    37.95 tokens per second)
0.40.592.303 I slot print_timing: id  0 | task 16 |        eval time =     972.01 ms /    67 tokens (   14.73 ms per token,    67.90 tokens per second)
0.40.592.303 I slot print_timing: id  0 | task 16 |       total time =    1340.91 ms /    81 tokens
0.40.592.305 I slot print_timing: id  0 | task 16 |    graphs reused =         31
0.40.592.307 I slot print_timing: id  0 | task 16 | draft acceptance = 0.71429 (   45 accepted /    63 generated), mean len =  3.14
0.40.592.367 I slot      release: id  0 | task 16 | stop processing: n_tokens = 448, truncated = 0

llama.cpp 我用的最新 b10485; 完全抄作业:

bruin@lmde7 ~ $ cat run-3.8-q5.sh
#!/bin/bash

# ref: https://lcz.me/topic/1157/qwen3.8-27b-q5_k_m-7900xtx-deepseek-harness%E5%AE%9E%E6%88%98%E5%B9%B3%E5%9D%87-52-t-s

LLAMA_SERVER=/home/bruin/llama-server-vulkan-b10485
MAIN_MODEL="/opt/gguf-models/unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-Q5_K_S.gguf"
MTMD_MODEL="/opt/gguf-models/unsloth/Qwen3.8-27B-GGUF/mmproj-F16.gguf"
#--mmproj "${MTMD_MODEL}" \

${LLAMA_SERVER} \
--device Vulkan0 \
--model "${MAIN_MODEL}" \
-t 10 \
-b 512 \
-ub 256 \
--spec-draft-n-max 3 \
--fit off \
--no-context-shift \
--metrics \
--kv-unified \
--jinja \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
-fa on \
--spec-type draft-mtp \
--ctx-size 131072 \
--parallel 1 \
-ngl -1 \
--host 0.0.0.0 \
--port 8000 \
--chat-template-kwargs '{"enable_thinking": true, "preserve_think": false, "reasoning_effort": "medium"}' \
--no-mmap \
--temp 0.6 \
--top-p 0.5 \
--top-k 15 \
--repeat-penalty 1.0 \
--override-tensor blk\.\d+\.ffn_.*_exps\.=CPU \
--alias qwen3.8-27b-q5

[6cccccc]

@laobenxiong 确实，用了你这个q5ks的模型，我的速度终于正常了，也不用关thinking，不知道为什么其他人的q5km可以

[xl]

关于--parallel 2 的测试

软硬件环境：
cpu： AMD Ryzen 7 9700X
RAM：DDR5 64GB 6000
GPU：R9700 32G
OS： Ubuntu 24.04
软件 ： llama.cpp + vulkan

结论：将并行设为1或者2，大部分情况下，性能变化属于正常波动，没有明显差别。

但是，即使有32G显存，有些任务还是会报以下错误：
26.48.401.777 E init_batch: failed to prepare attention ubatches
26.48.401.778 W decode: failed to find a memory slot for batch of size 16
26.48.401.779 W srv decode: failed to find free space in the KV cache, retrying with smaller batch size, off = 190, n_batch = 8, ret = 1
26.48.403.275 E init_batch: failed to prepare attention ubatches
26.48.403.277 W decode: failed to find a memory slot for batch of size 8
26.48.403.277 W srv decode: failed to find free space in the KV cache, retrying with smaller batch size, off = 190, n_batch = 4, ret = 1
虽然使用了 --cache-type-k/v q8_0 做 KV 量化来节省显存，但是 128K 的超长上下文，加上 --parallel 2，对 KV Cache 的要求依然极其恐怖。模型权重稳稳地占了 17GB。但当开启并发，且 Prompt 超长时，给 128K 上下文预留的 KV Cache 空间，在高峰期（特别是并发请求时）瞬间耗尽了，导致没有空位给下一个 Token 的生成。
好消息是，llama.cpp不能为KV Cache分配空间，并不会导致崩溃或任务中断（我没有碰到过），只会导致llama.cpp降级重试，严重的情况下会感觉明显卡顿而已，不用过于担心。
所以，用7900XTX的朋友，把并发设为1可能是更好的选择。

[Enigma]

感谢分享，学习中。。。