抡锤者

TLDR

先上個實體圖

Beelink Ser 8 8745HS 用Oculink連接 RTX Pro 4500

跑在Ubuntu 26.04, Kernel 7.0

啓動咒語, 注意這個是我在vLLM cu130 nightly (0.20)設立的, cu129 0.22估計會有更多優化, 我會試試看其他版本

docker run -d \
  --name vllm-Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP \
  --restart unless-stopped \
  --ipc host \
  --gpus '"device=0"' \
  -p 0.0.0.0:7380:8000 \
  -v "~/vllm/models:/models:ro" \
  -v "~/vllm/.cache/huggingface:/root/.cache/huggingface" \
  -e GPU_MEMORY_UTILIZATION="0.95" \
  -e HF_HUB_OFFLINE="1" \
  -e KV_CACHE_DTYPE="fp8" \
  -e MAX_MODEL_LEN="230400" \
  -e MODEL_PATH="/models/sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP" \
  -e PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True" \
  -e SERVED_MODEL_NAME="Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP" \
  -e VLLM_ATTENTION_BACKEND="FLASHINFER" \
  -e VLLM_EXTRA_ARGS='--quantization modelopt --trust-remote-code --enable-chunked-prefill --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice --max-num-seqs 1 --max-num-batched-tokens 4096 --speculative-config {"method":"qwen3_5_mtp","num_speculative_tokens":3} --language-model-only --performance-mode interactivity --attention-backend flashinfer --skip-mm-profiling --enable-prefix-caching --no-disable-hybrid-kv-cache-manager' \
  -e VLLM_LOGGING_LEVEL="INFO" \
  -e VLLM_NVFP4_GEMM_BACKEND="flashinfer-cutlass" \
  -e VLLM_USE_FLASHINFER_MOE_FP4="0" \
  -e VLLM_USE_FLASHINFER_SAMPLER="1" \
  --health-cmd 'curl -fsS http://localhost:8000/v1/models || exit 1' \
  --health-timeout 5s \
  --health-interval 30s \
  --health-retries 5 \
  --health-start-period 5m \
  --entrypoint /bin/bash \
  vllm/vllm-openai:cu130-nightly \
  -lc 'exec vllm serve "$MODEL_PATH" --served-model-name "$SERVED_MODEL_NAME" --host 0.0.0.0 --port 8000 --max-model-len "$MAX_MODEL_LEN" --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" --kv-cache-dtype "$KV_CACHE_DTYPE" $VLLM_EXTRA_ARGS'

llama-benchy benchmark

llama-benchy \
  --base-url "http://localhost:7380/v1" \
  --model "Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP" \
  --tokenizer "$HOME/vllm/models/sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP" \
  --pp 2048 \
  --tg 480 \
  --depth 0 1000 5000 10000 20000 50000 100000 150000 200000 \    #(不同上下文長度)
  --latency-mode generation \
  --skip-coherence \
  --concurrency 1 \

效果

| model                                    |             test |               t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:-----------------------------------------|-----------------:|------------------:|-------------:|------------------:|------------------:|------------------:|
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |           pp2048 | 7741.01 ± 1375.30 |              |    373.94 ± 54.49 |    274.26 ± 54.49 |    373.94 ± 54.49 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |            tg480 |      68.87 ± 6.65 | 81.33 ± 3.68 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   pp2048 @ d1000 |   8136.73 ± 32.84 |              |     474.32 ± 1.44 |     374.64 ± 1.44 |     474.32 ± 1.44 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |    tg480 @ d1000 |      67.73 ± 5.06 | 88.00 ± 5.72 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   pp2048 @ d5000 |   6615.23 ± 22.79 |              |    1165.21 ± 3.86 |    1065.53 ± 3.86 |    1165.21 ± 3.86 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |    tg480 @ d5000 |      72.92 ± 3.56 | 89.33 ± 3.77 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  pp2048 @ d10000 |   6008.73 ± 10.16 |              |    2104.88 ± 3.47 |    2005.20 ± 3.47 |    2104.88 ± 3.47 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   tg480 @ d10000 |      65.25 ± 2.21 | 82.00 ± 4.32 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  pp2048 @ d20000 |    5152.21 ± 0.52 |              |    4379.13 ± 0.52 |    4279.45 ± 0.52 |    4380.19 ± 0.46 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   tg480 @ d20000 |      70.45 ± 1.27 | 89.67 ± 0.47 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  pp2048 @ d50000 |    3690.36 ± 5.88 |              |  14203.66 ± 22.59 |  14103.98 ± 22.59 |  14205.86 ± 22.80 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   tg480 @ d50000 |      67.03 ± 1.67 | 84.67 ± 0.47 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP | pp2048 @ d100000 |    2528.58 ± 0.55 |              |   40457.51 ± 8.72 |   40357.83 ± 8.72 |   40461.50 ± 8.69 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  tg480 @ d100000 |      60.96 ± 0.75 | 78.33 ± 3.68 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP | pp2048 @ d150000 |    1922.36 ± 0.98 |              |  79194.84 ± 39.68 |  79095.17 ± 39.68 |  79201.49 ± 39.50 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  tg480 @ d150000 |      62.53 ± 3.29 | 76.33 ± 1.89 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP | pp2048 @ d200000 |    1556.00 ± 0.99 |              | 129951.65 ± 82.49 | 129851.97 ± 82.49 | 129959.72 ± 82.53 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  tg480 @ d200000 |      59.58 ± 1.31 | 69.67 ± 1.70 |                   |                   |                   |

---

碎碎唸, 講一下參數選擇邏輯

GPU_MEMORY_UTILIZATION => 0.95, Headless伺服器, 顯示輸出由iGPU負責
KV_CACHE_DTYPE => FP8, Ada架構以後基本統一FP8
MAX_MODEL_LEN => 230K, 之前有嘗試試過極限拉到240K左右, 但是會在部分長上下文出現OOM, 穩定點用230K

PYTORCH_CUDA_ALLOC_CONF => Pytorch實驗性參數, 透過呼叫CUDA内核API整理VRAM碎塊, 降低OOM機會
VLLM_ATTENTION_BACKEND => FLASHINFER, 很奇怪的是vLLM是推薦用這個而不是Flash Attention, 理論上在NVFP4在sm 12X (Desktop Blackwell)還沒完善下的情況用FA估計會比較好, 在sm 10X (Datacenter Blackwell)則FLASHINFER比較好

quantization => modelopt, vllm會跑去讀hf_quant_config.json裏的quant_algo, 這個模型是nvfp4
enable-chunked-prefill => 必開不解釋, 優化VRAM避免Spike導致OOM
speculative-config => 2 或者 3 都可, 激進點就用了3
skip-mm-profiling => 因爲這個模型只支持Text, 所以不需要multi model設定,省點VRAM
enable-prefix-caching => 降低TTRT
no-disable-hybrid-kv-cache-manager => 避免因爲Qwen模型的混合Attention導致挂掉

VLLM_NVFP4_GEMM_BACKEND => 叫vLLM 使用 FlashInfer/Cutlass NVFP4 kernels進行矩陣計算, Blackwell特點
VLLM_USE_FLASHINFER_MOE_FP4 (0) + VLLM_USE_FLASHINFER_SAMPLER (1) => 優化CUDA内核

@Tony-Wang

對, 就算是沒load也會固定在30%風扇, RTX Pro系列風扇都不可調

補一補圖, 今晚可能會發個文簡單講一下

@rolex-lo

基本上我是把上下文拉爆(日間Coding需要), 然後concurrency單純調1, 並沒有特別針對hermes做什麼特別優化 (也許研究一下會更好, 不過得要有空)

@tony-wang @rolex-lo

趁現在午休的時候跑了一下llama benchy

llama-benchy \
  --base-url "http://localhost:7380/v1" \
  --model "Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP" \
  --tokenizer "$HOME/vllm/models/sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP" \
  --pp 2048 \
  --tg 480 \
  --depth 0 1000 5000 10000 20000 50000 100000 150000 200000 \    #(不同上下文長度)
  --latency-mode generation \
  --skip-coherence \
  --concurrency 1 \

Context Ladder

| model                                    |             test |               t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:-----------------------------------------|-----------------:|------------------:|-------------:|------------------:|------------------:|------------------:|
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |           pp2048 | 7741.01 ± 1375.30 |              |    373.94 ± 54.49 |    274.26 ± 54.49 |    373.94 ± 54.49 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |            tg480 |      68.87 ± 6.65 | 81.33 ± 3.68 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   pp2048 @ d1000 |   8136.73 ± 32.84 |              |     474.32 ± 1.44 |     374.64 ± 1.44 |     474.32 ± 1.44 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |    tg480 @ d1000 |      67.73 ± 5.06 | 88.00 ± 5.72 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   pp2048 @ d5000 |   6615.23 ± 22.79 |              |    1165.21 ± 3.86 |    1065.53 ± 3.86 |    1165.21 ± 3.86 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |    tg480 @ d5000 |      72.92 ± 3.56 | 89.33 ± 3.77 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  pp2048 @ d10000 |   6008.73 ± 10.16 |              |    2104.88 ± 3.47 |    2005.20 ± 3.47 |    2104.88 ± 3.47 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   tg480 @ d10000 |      65.25 ± 2.21 | 82.00 ± 4.32 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  pp2048 @ d20000 |    5152.21 ± 0.52 |              |    4379.13 ± 0.52 |    4279.45 ± 0.52 |    4380.19 ± 0.46 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   tg480 @ d20000 |      70.45 ± 1.27 | 89.67 ± 0.47 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  pp2048 @ d50000 |    3690.36 ± 5.88 |              |  14203.66 ± 22.59 |  14103.98 ± 22.59 |  14205.86 ± 22.80 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |   tg480 @ d50000 |      67.03 ± 1.67 | 84.67 ± 0.47 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP | pp2048 @ d100000 |    2528.58 ± 0.55 |              |   40457.51 ± 8.72 |   40357.83 ± 8.72 |   40461.50 ± 8.69 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  tg480 @ d100000 |      60.96 ± 0.75 | 78.33 ± 3.68 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP | pp2048 @ d150000 |    1922.36 ± 0.98 |              |  79194.84 ± 39.68 |  79095.17 ± 39.68 |  79201.49 ± 39.50 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  tg480 @ d150000 |      62.53 ± 3.29 | 76.33 ± 1.89 |                   |                   |                   |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP | pp2048 @ d200000 |    1556.00 ± 0.99 |              | 129951.65 ± 82.49 | 129851.97 ± 82.49 | 129959.72 ± 82.53 |
| Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP |  tg480 @ d200000 |      59.58 ± 1.31 | 69.67 ± 1.70 |                   |                   |                   |

Token速度相當可用, 200K上下都能大約有60 tks

上班的時候我vLLM會拿來當成agent來用, 跟claude跟gpt計劃完後implementation就會用了, 有時候涉及一些低層邏輯或者敏感資料也會只使用自己的私人模型

睡覺的時候就叫hermes看一看reddit上面Locallama的熱文，臨近起床的時候去看一下+整理股票價格

@bily-j

我自己有在用vLLM搭建API配合cline使用, 可以給點意見

除了跟@xiaote 說的用有寫abliterated的模型, 也可以用有寫heretic的模型

heretic在 重新訓練的數據集比較好 的情況下, 模型的思考能力, 連貫性跟邏輯會比abliterated更強, 也相對會更加少出現拒絕回答的情況

---

以下是碎碎唸跟簡單講一下原理, 有興趣可以讀一下

在hf的model card上通常都會有refusals rate, 我拿兩個模型作為例子

llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-GGUF

wangzhang/Qwen3.6-35B-A3B-abliterated-GGUF

KL越接近基礎模型越好, 代表兩個模型的思維偏差越低, 在這兩個例子下就是越接近0越好, KL基本上低於0.3就沒什麼大問題

heretic跟abliterated都是去審核的方式

abliterated只能說是減低拒絕的概率(refusal-reduced), 因為只單純控制權重(Weight Projection), 不涉及微調

heretic則是利用貝葉斯優化(Bayesian optimization)微調模型

但很遺憾的是沒有一個通用準則說什麼時候用abliterated, 什麼時候用heretic

---

然後就有人腦洞大開會跑去弄個混合模型

Youssofal/Qwen3.6-35B-A3B-Abliterated-Heretic-GGUF

@rolex-lo

我現在就是用RTX Pro 4500, 也許晚上我發個文?

@CS6

Vulkan支持混合卡, 把sm86變成sm120應該就可以, 畢竟CS你應該也在用vulkan吧

B70的話還是避開吧, 這張卡很多測試情景都是用Intel自己docker image, 適用性可能無限趨近0

@rolex-lo

一分錢一分強

而且tks都是貴的, 本地單純是一次性, API就訂閱制

這個nvfp4準確度我也沒特別測量

見到各位大神有在分享設置, 然後我這個小白最近在公司還蠻幸運能用上AWS G5dn x12large配置

現在就來分享一下我在試驗時候(私心)測試後比較穩定的配置(至少沒有OOM).

A10G可以視作被削弱的3090, 所以配置可以視作3090/3090Ti適用, 擁有3090等級或者以上的24GB卡也用來參考 (基本上只會比這個更快, 不會更慢).

注意我目前還沒有額外時間跑太多的Benchmark, 而且有跑的Benchmark基本上是vLLm自己官網上有寫的. 因此下面有寫的基本上就是成功架構并且能達到普(老)通(闆)人不會等到不耐煩的速度.

使用場景: v0.22.0-cu129-ubuntu2404

1) Gemma 4 31B

試過2 * A10G 跟 4 * A10G, 以下是2 * A10G設置

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  -e VLLM_MODEL_NAME="Gemma-4-31B-it" \
  vllm/vllm-openai:v0.22.0-cu129-ubuntu2404 \
  --model Intel/gemma-4-31B-it-int4-AutoRound \
  --served-model-name Gemma-4-31B-it \
  --dtype float16 \
  --quantization auto_round \
  --gpu-memory-utilization 0.90 \
  --max-model-len 192768 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --data-parallel-size 1 \
  --attention-backend TRITON_ATTN \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":4}' \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

Gemma 4 31B不能在單A10G的情況下使用, 就算是Int 4也不行, 請研究關於MoE的配置

來解釋一下配置原因:

base: Intel/gemma-4-31B-it-int4-AutoRound (~22 GB)
drafter: google/gemma-4-31B-it-assistant (0.5B / ~927 MB BF16)

Gemma 4 MTP跟Qwen 3.6不一樣, Qwen内置MTP head, 因此可以不需要額外帶上另外一個drafter.
Gemma 4并沒有内置MTP head, 所以需要一個基於Gemma 4提煉出來的drafter. 理論上也可以使用gemma 4 E2B 跟 E4B作爲drafter, 但是這兩個比31B自己的0.5B drafter還要大

dtype : float16, 理論上bf16也可以, 3090 支持float 16跟bfloat 16
quantization: autoround, 低精度下保持相對高的token質量, int4/int8必用

gpu-memory-utilization: 0.9, 有其他東西需要VRAM所以我限制在0.9, 理論上Headless Server可以把這個推到0.95的話, 然後2張卡可以上260K長度
max-num-seqs: 1作爲POC超長上下文使用, 普通長上下文Agent可以設成2, 短上下文Agent可以上4

max-num-batched-tokens: vision tower需要至少2496以上
attention-backend: TRITON_ATTN, Gemma 4 理論上支持 FA2, 但是基於Head Dimension跟Hybrid Attention的關係用FA會爆炸, 現階段還是用triton attention

speculative-config: MTP加速

值得一提的是kv-cache-dtype理論上int8_per_token_head, 我也曾經在Reddit上看到有人成功使用, 可是我自己不行, 有3090的朋友可以試試看

Benchmark

--dataset-name random --input-len 1024 --output-len 256 --num-prompts 100 --request-rate inf --ignore-eos

============ Serving Benchmark Result ============
Successful requests:                     100       
Failed requests:                         0         
Benchmark duration (s):                  376.11    
Total input tokens:                      103747    
Total generated tokens:                  25600     
Request throughput (req/s):              0.27      
Output token throughput (tok/s):         68.06     
Peak output token throughput (tok/s):    32.00     
Peak concurrent requests:                100.00    
Total token throughput (tok/s):          343.91    
---------------Time to First Token----------------
Mean TTFT (ms):                          190345.47 
Median TTFT (ms):                        190211.51 
P99 TTFT (ms):                           370098.98 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.63     
Median TPOT (ms):                        10.63     
P99 TPOT (ms):                           13.52     
---------------Inter-token Latency----------------
Mean ITL (ms):                           32.00     
Median ITL (ms):                         32.28     
P99 ITL (ms):                            32.95     
---------------Speculative Decoding---------------
Acceptance rate (%):                     51.67     
Acceptance length:                       3.07      
Drafts:                                  8369      
Draft tokens:                            33476     
Accepted tokens:                         17296     
Per-position acceptance (%):
  Position 0:                            77.55     
  Position 1:                            58.00     
  Position 2:                            41.55     
  Position 3:                            29.57     
==================================================

---

2) Gemma 4 26B A4B

單純4 * A10G設立過, 但并沒有真正使用過, 沒加上MTP, 原則上跟Gemma 4 31B的思路差不多

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  -e VLLM_MODEL_NAME="Gemma-4-26B-A4B-it" \
  vllm/vllm-openai:v0.22.0-cu129-ubuntu2404 \
  --model Intel/gemma-4-26B-A4B-it-int4-mixed-AutoRound \
  --served-model-name Gemma-4-26B-A4B-it \
  --dtype float16 \
  --quantization auto_round \
  --gpu-memory-utilization 0.90 \
  --max-model-len 192768 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 8192 \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --data-parallel-size 1 \
  --enable-expert-parallel true \
  --attention-backend TRITON_ATTN \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

model: Intel/gemma-4-26B-A4B-it-int4-mixed-AutoRound (~15 GB)

請注意跟自行修改tensor-parallel-size跟enable-expert-parallel

tensor-parallel-size: 4, 因爲x12Large有4張卡, 這個按照你卡的數量以2的次方為準 (2, 4, 8 etc)
enable-expert-parallel: true, 這個沒有NVLINK那些的話就直接false就可以了, 有興趣的可以問我, 之後再回答

以下預測MTP的配置:

drafter: google/gemma-4-26B-A4B-it-assistant (~0.4B / ~800 MB BF16)
--speculative-config '{"method":"mtp","model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":4}'

如果想要2卡的話max-model-len很大機會需要乘上0.75 或 0.5

---

3. Qwen 3.6 27B

有一個比較嚴重的問題是Qwen 3.6在長上下文的情況下思考時間過長(Overthinking), 試過關閉enable_thinking跟限制thinking token數量但效果不太明顯.

雖然質量出色但因爲本身System Prompt加上上下文一多就會導致TTFT太長, 需要更多時間研究

試過2 * A10G 跟 4 * A10G, 以下是2 * A10G設置

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  -e VLLM_MODEL_NAME="Qwen3.6-27B" \
  vllm/vllm-openai:v0.22.0-cu129-ubuntu2404 \
  --model cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 \
  --tensor-parallel-size 2 \
  --max-model-len 192768 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 2 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 (~ 15GB)

沒特別研究所以還沒看autoround (Lorbus/Qwen3.6-27B-int4-AutoRound ) 以及其他人有關fp8 kv cache的設定

---

4. Qwen 3.6 35B A3B

未嘗試, 但估計思路跟從Gemma 4 31B轉換到26B-A4B類似

然後這個是Gemma 4 31B (2 * A10G)

vllm serve \
  --model Intel/gemma-4-31B-it-int4-AutoRound \
  --host 0.0.0.0 \
  --port 8000 \
  --generation-config vllm \
  --served-model-name Gemma-4-31B-it \
  --dtype float16 \
  --quantization auto_round \
  --gpu-memory-utilization 0.95 \        #(需要上到0.95不然OOM)
  --max-model-len 192768 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \ #(8192降到4096)
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --data-parallel-size 1 \
  --language-model-only \
  --attention-config.backend TRITON_ATTN \
  --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"mtp","model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":4}' \
  --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

以下是更新版的Benchmark

### Workload

| Metric                     | Run 08:58            | Run 09:07            | Run 09:34            |
| -------------------------- | -------------------- | -------------------- | -------------------- |
| dataset                    | random               | random               | random               |
| input length arg           | 1024                 | 1024                 | 1024                 |
| output length arg          | 256                  | 256                  | 256                  |
| input tokens mean/min/max  | 1037.5 / 1037 / 1039 | 1037.5 / 1037 / 1039 | 1037.5 / 1037 / 1039 |
| output tokens mean/min/max | 256.0 / 256 / 256    | 256.0 / 256 / 256    | 256.0 / 256 / 256    |
| num prompts                | 100                  | 100                  | 100                  |
| request rate               | inf                  | inf                  | inf                  |

### Request Outcome

| Metric                 | Run 08:58 | Run 09:07 | Run 09:34 |
| ---------------------- | --------- | --------- | --------- |
| successful requests    | 100       | 100       | 100       |
| failed requests        | 0         | 0         | 0         |
| benchmark duration (s) | 462.51    | 457.19    | 462.96    |

### Latency

| Metric           | Run 08:58 | Run 09:07 | Run 09:34 |
| ---------------- | --------- | --------- | --------- |
| mean TTFT (ms)   | 233010.17 | 229102.51 | 231664.74 |
| median TTFT (ms) | 234769.52 | 232669.51 | 231388.69 |
| P99 TTFT (ms)    | 453358.78 | 449056.54 | 454054.81 |
| mean TPOT (ms)   | 13.96     | 13.75     | 13.98     |
| P99 TPOT (ms)    | 18.09     | 17.01     | 18.07     |
| mean ITL (ms)    | 42.03     | 41.92     | 42.02     |
| P99 ITL (ms)     | 43.59     | 43.59     | 43.67     |

### Throughput

| Metric                          | Run 08:58 | Run 09:07 | Run 09:34 |
| ------------------------------- | --------- | --------- | --------- |
| request throughput (req/s)      | 0.216     | 0.219     | 0.216     |
| output token throughput (tok/s) | 55.35     | 55.99     | 55.30     |
| total token throughput (tok/s)  | 279.66    | 282.92    | 279.39    |
| prefill throughput (tok/s)      | 4.5       | 4.5       | 4.5       |

### Memory And Cache

| Metric                      | Run 08:58                  | Run 09:07                  | Run 09:34                  |
| --------------------------- | -------------------------- | -------------------------- | -------------------------- |
| VRAM before (MiB)           | 20825                      | 20947                      | 20825                      |
| VRAM peak (MiB)             | 20947                      | 20947                      | 20947                      |
| VRAM peak per GPU (MiB)     | 20947, 20947, 3, 3        | 20947, 20947, 3, 3        | 20947, 20947, 3, 3        |
| RAM used peak (MiB)         | 14713                      | 14706                      | 14809                      |
| vLLM process RSS peak (MiB) | 2117                       | 2117                       | 2133                       |
| gpu/kv_cache_usage peak     | 4.4%                       | 4.4%                       | 4.4%                       |
| prefix caching enabled      | false                      | false                      | false                      |
| prefix cache hit rate       | 0.00% (0/103761)           | 0.00% (0/103761)           | 0.00% (0/103761)           |

### Speculative Decoding

| Metric              | Run 08:58 | Run 09:07 | Run 09:34 |
| ------------------- | --------- | --------- | --------- |
| acceptance rate (%) | 51.55     | 52.60     | 51.51     |
| acceptance length   | 3.06      | 3.10      | 3.06      |

@CS6

個人覺得, 與其折騰哪個引擎比較好, 倒不如理解它們背後的運作原理跟優勢更好, 尤其是Paged Attention (vLLM) 跟 Radix Attention (SGLang)

Paged Attention能消除記憶體浪費跟支援平行採樣
Radix Attention的前綴緩存(Prefix Caching)做得比vLLM好, 首字延遲(TTFT)更低

引用一下一個大神寫的文

至於家用的話, 個人覺得SGLang跟vLLM並不會有太大分別, 因為真的需要壓榨全部性能的情況太少了 (不是沒有, 除非你喜歡折騰), 付出的時間成本跟精力不成正比

@terry

抱歉, 因為是軟件工程出身跟泡太多關於榨乾硬件性能的Reddit帖子 / Github Repo, 所以關於配置跟benchmark可能會帶有比較專業的名詞跟滿多數據, 之後會在回覆開頭加個tldr

這個帖子估計也會持續更新, 在實體空(摸)閒(魚)的時候拿來當實驗紀錄

Qwen 27B 參數 (2 * A10G)

Docker Image: vllm-openai:v0.22.0-cu129-ubuntu2404

vllm serve \
  --model Lorbus/Qwen3.6-27B-int4-AutoRound \
  --host 0.0.0.0 \
  --port 8000 \
  --generation-config vllm \
  --served-model-name Qwen-3.6-27B-autoround \
  --dtype float16 \
  --quantization auto_round \
  --kv-cache-dtype fp8_e5m2 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 192768 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --data-parallel-size 1 \
  --language-model-only \
  --enable-auto-tool-choice \
  --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

---

碎碎唸

思路基本上跟A10G * 4一樣, batch token 降到4096, gpu memory utilization 上到0.95

以下是更新版的Benchmark

### Workload

| Metric                     | Run 07:09            | Run 07:17            | Run 07:26            |
| -------------------------- | -------------------- | -------------------- | -------------------- |
| dataset                    | random               | random               | random               |
| input length arg           | 1024                 | 1024                 | 1024                 |
| output length arg          | 256                  | 256                  | 256                  |
| input tokens mean/min/max  | 1034.4 / 1033 / 1036 | 1034.4 / 1033 / 1036 | 1034.4 / 1033 / 1036 |
| output tokens mean/min/max | 256.0 / 256 / 256    | 256.0 / 256 / 256    | 256.0 / 256 / 256    |
| num prompts                | 100                  | 100                  | 100                  |
| request rate               | inf                  | inf                  | inf                  |

### Request Outcome

| Metric                 | Run 07:09  | Run 07:17  | Run 07:26  |
| ---------------------- | ---------- | ---------- | ---------- |
| successful requests    | 100        | 100        | 100        |
| failed requests        | 0          | 0          | 0          |
| benchmark duration (s) | 463.34     | 478.80     | 474.50     |

### Latency

| Metric           | Run 07:09   | Run 07:17   | Run 07:26   |
| ---------------- | ----------- | ----------- | ----------- |
| mean TTFT (ms)   | 232418.08   | 238435.64   | 236922.49   |
| median TTFT (ms) | 231770.91   | 238065.71   | 238316.95   |
| P99 TTFT (ms)    | 455414.07   | 470471.84   | 466104.09   |
| mean TPOT (ms)   | 14.38       | 15.00       | 14.83       |
| P99 TPOT (ms)    | 24.48       | 20.19       | 22.90       |
| mean ITL (ms)    | 39.04       | 39.49       | 39.32       |
| P99 ITL (ms)     | 41.72       | 42.91       | 42.08       |

### Throughput

| Metric                          | Run 07:09 | Run 07:17 | Run 07:26 |
| ------------------------------- | --------- | --------- | --------- |
| request throughput (req/s)      | 0.216     | 0.209     | 0.211     |
| output token throughput (tok/s) | 55.25     | 53.47     | 53.95     |
| total token throughput (tok/s)  | 278.50    | 269.50    | 271.94    |
| prefill throughput (tok/s)      | 4.5       | 4.3       | 4.4       |

### Memory And Cache

| Metric                      | Run 07:09                  | Run 07:17                  | Run 07:26                  |
| --------------------------- | -------------------------- | -------------------------- | -------------------------- |
| VRAM before (MiB)           | 20731                      | 21693                      | 21693                      |
| VRAM peak (MiB)             | 21693                      | 21693                      | 21693                      |
| VRAM peak per GPU (MiB)     | 21691, 21693, 3, 3         | 21691, 21693, 3, 3         | 21691, 21693, 3, 3         |
| RAM used peak (MiB)         | 16572                      | 15092                      | 15119                      |
| vLLM process RSS peak (MiB) | 1837                       | 1837                       | 1837                       |
| gpu/kv_cache_usage peak     | 3.1%                       | 3.1%                       | 3.1%                       |
| prefix caching enabled      | false                      | false                      | false                      |
| prefix cache hit rate       | n/a                        | n/a                        | n/a                        |

### Speculative Decoding

| Metric              | Run 07:09 | Run 07:17 | Run 07:26 |
| ------------------- | --------- | --------- | --------- |
| acceptance rate (%) | 58.40     | 55.60     | 56.28     |
| acceptance length   | 2.75      | 2.67      | 2.69      |

@c0aster

Deepseek在伺服器那邊基本上都會有Prompting優化, 本地的AGENT.md跟Rules基本上不會有同樣的效果

倒不如說根本追不上, 就算是同樣的Prompting, 一個27B跟一個1600B-A49B (1.6T-A49B, DeepSeek-V4-Pro), 基本上就是螞蟻跟大象的分別

本地最大的優勢就只是在處理敏感資料跟不會額外收費而已

然後給一下Qwen 27B 參數 (4 * A10G)

Docker Image: vllm-openai:v0.22.0-cu129-ubuntu2404

vllm serve \
  --model Lorbus/Qwen3.6-27B-int4-AutoRound \
  --host 0.0.0.0 \
  --port 8000 \
  --generation-config vllm \
  --served-model-name Qwen-3.6-27B-autoround \
  --dtype float16 \
  --quantization auto_round \
  --kv-cache-dtype fp8_e5m2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 192768 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 8192 \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1 \
  --data-parallel-size 1 \
  --language-model-only \
  --enable-auto-tool-choice \
  --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

---

碎碎唸

基本上跟Gemma 4一樣，使用auto round來節省model weight

kv cache則使用僅有支持Ampere架構的fp8_e5m2, vllm可以透過fp8_e5m2模仿bfloat16, 並且轉換成int8獲得硬件加速, fp8_e4m3架構則不支持模仿

強烈不建議使用 --default-chat-template-kwargs '{"enable_thinking": false}', Token質量會斷崖式下降

以下是更新版的Benchmark

### Workload

| Metric                     | Run 05:17            | Run 05:28            | Run 05:36            |
| -------------------------- | -------------------- | -------------------- | -------------------- |
| dataset                    | random               | random               | random               |
| input length arg           | 1024                 | 1024                 | 1024                 |
| output length arg          | 256                  | 256                  | 256                  |
| input tokens mean/min/max  | 1034.4 / 1033 / 1036 | 1034.4 / 1033 / 1036 | 1034.4 / 1033 / 1036 |
| output tokens mean/min/max | 256.0 / 256 / 256    | 256.0 / 256 / 256    | 256.0 / 256 / 256    |
| num prompts                | 100                  | 100                  | 100                  |
| request rate               | inf                  | inf                  | inf                  |

### Request Outcome

| Metric                 | Run 05:17 | Run 05:28 | Run 05:36 |
| ---------------------- | --------- | --------- | --------- |
| successful requests    | 100       | 100       | 100       |
| failed requests        | 0         | 0         | 0         |
| benchmark duration (s) | 430.22    | 427.70    | 443.72    |

### Latency

| Metric           | Run 05:17 | Run 05:28 | Run 05:36 |
| ---------------- | --------- | --------- | --------- |
| mean TTFT (ms)   | 214258.88 | 211603.07 | 217519.66 |
| median TTFT (ms) | 211865.20 | 210793.65 | 213751.71 |
| P99 TTFT (ms)    | 422468.83 | 418775.36 | 435311.06 |
| mean TPOT (ms)   | 13.11     | 13.01     | 13.63     |
| P99 TPOT (ms)    | 21.82     | 16.84     | 19.43     |
| mean ITL (ms)    | 35.67     | 35.94     | 36.59     |
| P99 ITL (ms)     | 38.89     | 39.51     | 40.25     |

### Throughput

| Metric                          | Run 05:17 | Run 05:28 | Run 05:36 |
| ------------------------------- | --------- | --------- | --------- |
| request throughput (req/s)      | 0.232     | 0.234     | 0.225     |
| output token throughput (tok/s) | 59.50     | 59.85     | 57.69     |
| total token throughput (tok/s)  | 299.94    | 301.70    | 290.81    |
| prefill throughput (tok/s)      | 4.8       | 4.9       | 4.8       |

### Memory And Cache

| Metric                      | Run 05:17                  | Run 05:28                  | Run 05:36                  |
| --------------------------- | -------------------------- | -------------------------- | -------------------------- |
| VRAM before (MiB)           | 20261                      | 21143                      | 21143                      |
| VRAM peak (MiB)             | 21143                      | 21143                      | 21143                      |
| VRAM peak per GPU (MiB)     | 21143, 21143, 21143, 21143 | 21143, 21143, 21143, 21143 | 21143, 21143, 21143, 21143 |
| RAM used peak (MiB)         | 22076                      | 20870                      | 20798                      |
| vLLM process RSS peak (MiB) | 1825                       | 1825                       | 1825                       |
| gpu/kv_cache_usage peak     | 1.2%                       | 1.2%                       | 1.2%                       |
| prefix caching enabled      | false                      | false                      | false                      |
| prefix cache hit rate       | n/a                        | n/a                        | n/a                        |

### Speculative Decoding

| Metric              | Run 05:17 | Run 05:28 | Run 05:36 |
| ------------------- | --------- | --------- | --------- |
| acceptance rate (%) | 58.75     | 60.16     | 57.49     |
| acceptance length   | 2.76      | 2.80      | 2.72      |

---

@dreamy2k

好消息是你可以混合使用A + N卡, 你可以用Vulkan來將model分到兩張卡的VRAM上面, 然後llamacpp選用Vulkan, 我也曾經在Reddit上面聽過有人混合RTX 5070 Ti + RX 9070, 除了prefill速度慢了跟沒有特別優化之外應該沒什麼問題

壞消息是你需要自己編譯Vulkan內核

如果是普通人不太想太深入研究的話推薦直接買多一張A10G, 或者賣A10G換成R9700

---

碎碎念一下

跑去llamacpp看了一下, 很不負責地給一下編譯command

強烈建議使用docker container + Linux Kernel, 不要在Window底下編譯, 可以用這個試試看

編譯
rm -rf build && \
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" cmake -B build \
  -DBUILD_SHARED_LIBS=ON \
  -DGGML_BACKEND_DL=ON \
  -DGGML_NATIVE=OFF \
  -DGGML_CPU_ALL_VARIANTS=ON \
  -DGGML_CUDA=ON \              
  -DGGML_HIP=ON \
  -DGPU_TARGETS=gfx1201 \                           #(R9700 AI 架構)
  -DGGML_HIP_ROCWMMA_FATTN=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES="86" && \           #(3090 SM86架構)
cmake --build build --config Release -j 64

啟動
${HOME}/code/llama.cpp/build/bin/llama-server \
	--port 1234 --host 0.0.0.0 \
	--models-preset <你模型的啟動參數>.ini \
	--device CUDA0,ROCm0 --fit-target 3072,512        #(假設你第一張卡是插屏幕，需要預留多點VRAM)

@rolex-lo

可以這樣說, AMD在原生的Linux内核會比WSL 2來得好, 畢竟WSL 2再怎麽貼近Linux 内核, 它的本質還是Hyper V, 不多不少都會有影響

@CS6

如果像4090一樣應該可以...吧, 到時候我們就知道了

現在就是迷你電腦配上RTX Pro 4500

@CS6

https://lcz.me/topic/431/对-m5-max-跑本地大模型有点失望/28

我在這裏簡單用llama benchy測試了一下, 可以參考看看

5000 Pro, 6000 Pro那些應該只會更快不會更慢