求助4卡r9700 用vllm跑qwen3.8 27b fp8怎么优化
-
问题不在解码速度，在 prefill。你 benchmark 测的是 output（192 tok/s），但 Hermes 那种 agent 负载是「每回合都把上下文整包重送」，经常 3-6 万 token 丢进去，输出才几百 token——全耗在 prefill 上。所以单开 opwebui 短对话还行，一接 agent 就崩，多人挤一起直接个位数。
再叠一层：你
--max-model-len 180000是预分配，不是弹性。这模型是 hybrid-attention，只有 16 层缓存 KV，FP8 约 32KiB/token，180000×24≈134GiB，超过你 4×32=128GiB 的池子。长上下文一挤，vLLM 就 preemption（踢出去重算），这就是掉到个位数的直接原因。按影响改几个：
--max-model-len降到 65536 或 98304 就够，别 180000。KV 启动时就分配好，调低等于把池子腾给并发。- 日志里
num_preemptions_total在涨就把--max-num-seqs降到 8-12。agent 会话是占坑的，不是快查询。 --max-num-batched-tokens24576 偏大偏吞吐，多人求快降到 8192-16384。--enable-prefix-caching先关掉量一次。hybrid 模型的 prefix cache 有已知坑（零命中/白算），agent 最需要它却常不中用。- MTP 你 94% accept 说明能跑，但多人吞吐帮助有限，反而多一层 verify，可关可留。
- 确认下你
vllm-openai-rocm:latest是不是真原生 FP8，还是偷偷反量化回 FP32。是的话加VLLM_ROCM_USE_AITER=0跑，顺便python3 -c "import aiter.ops.triton.utils.arch_info as a; a._ARCH_TO_DEVICE['gfx1201']='MI350X'"。
量测看
curl localhost:8000/metrics | grep -E "num_preemptions_total|spec_decode_num_accepted_tokens_total"，改一处重启对照。把一轮 Hermes 的 prompt 长度贴出来更好判断。 -
有分析投标文件是否雷同的需求，所以设置了180k，太大了么？有点奇怪的是qwen3.6这个参数设200k都可以啊，为什么3.8就不行了。vllm-openai-rocm:latest是前天拉的最新的，显示vllm版本是0.28.0，这个怎么判断是不是原生fp8？用bf16的我试了试，结果都一样
-
楼上把主因说透了：你这负载是 prefill 瓶颈，不是 decode。补几个能直接落地的：
1. 为什么 3.6 换 3.8 明显变慢（架构原因，不是参数）
Qwen3.8-27B 是 hybrid Mamba+Attention：只有 16 层 attention 需要缓存 KV，其余是 mamba state。prefix caching 在这类模型上有已知坑（零命中/白算），等于 Hermes 每回合整包重发上下文时 prefill 全量重算，缓存基本白开；3.6 是纯 attention，cache 还能命中一段。所以"3.6 还行、3.8 拉胯"大概率不是调参能救的，是模型架构对 agent 重 prefill 场景更不友好。2. 两个数先查
- curl localhost:8000/metrics | grep num_preemptions_total——只要在涨，就是 180K 长上下文把 KV 池挤爆被踢重算，把 --max-model-len 砍到 65536/98304、--max-num-seqs 降到 8-12（楼上已说，我同意）。
- fp8 是否真生效：看 vLLM 启动日志有没有 FP8 kernel 选择（gfx1201 的 FP8 走 AITER/半官方路径，不对会静默回退 BF16，权重和 KV 白翻倍）。
3. Flash-Next：别上，更别为它加卡
Flash-Next 176B，FP8 权重就要 ~176GB，4 卡 128G、加第 5 卡 160G 都装不下；只能跑 IQ4_XS ~100GB 的量化。但你这场景是 agent 多用户、每回合 3-6 万 token 重 prefill，176B 的 prefill 比 27B 慢一个数量级，加卡只会更惨。27B 就是这个负载的甜点；真想提质量，试 35B 级模型更实际。4. 并发才是多人变慢的主开关
24 个 seq 槽对 agent"占坑型"会话太多：5 个同事的飞书会话能占满长上下文坑位互相挤。降到 8-12 之外，飞书接入端也限制同时活跃的会话数，比调 vLLM 参数立竿见影。改完把一轮 Hermes 的实际 prompt 长度（日志里的 input tokens）贴出来，能帮你把 KV 预算算死。
-
之前的启动日志里，有{"log":"(Worker_TP1 pid=728) WARNING 08-27 12:59:09 [fp8_utils.py:852] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=3584,K=5120,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json\n","stream":"stdout","time":"2026-08-27T12:59:09.814899751Z"}，这个是之前启动的日志里面的，应该是在FP8下运行了吧
-
(APIServer pid=1) INFO: 127.0.0.1:51838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51872 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:29:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 60.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 75.5%
(APIServer pid=1) INFO 08-29 15:29:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.81, Accepted throughput: 27.10 tokens/s, Drafted throughput: 33.50 tokens/s, Accepted: 271 tokens, Drafted: 335 tokens, Per-position acceptance rate: 0.809, Avg Draft acceptance rate: 80.9%
(APIServer pid=1) INFO 08-29 15:29:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 75.5%
(APIServer pid=1) INFO: 100.69.44.242:43230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:31:14 [loggers.py:310] Engine 000: Avg prompt throughput: 331.0 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 77.1%
(APIServer pid=1) INFO 08-29 15:31:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.70, Accepted throughput: 0.16 tokens/s, Drafted throughput: 0.23 tokens/s, Accepted: 16 tokens, Drafted: 23 tokens, Per-position acceptance rate: 0.696, Avg Draft acceptance rate: 69.6%
(APIServer pid=1) INFO 08-29 15:31:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 77.1%
(APIServer pid=1) INFO 08-29 15:31:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.73, Accepted throughput: 2.20 tokens/s, Drafted throughput: 3.00 tokens/s, Accepted: 22 tokens, Drafted: 30 tokens, Per-position acceptance rate: 0.733, Avg Draft acceptance rate: 73.3%
(APIServer pid=1) INFO: 100.69.44.242:42916 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 100.69.44.242:42922 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=1) INFO: 100.69.44.242:42924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:31:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 77.1%
(APIServer pid=1) INFO 08-29 15:31:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 1.50 tokens/s, Drafted throughput: 1.50 tokens/s, Accepted: 15 tokens, Drafted: 15 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 08-29 15:31:44 [loggers.py:310] Engine 000: Avg prompt throughput: 1610.9 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:31:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.50, Accepted throughput: 0.10 tokens/s, Drafted throughput: 0.20 tokens/s, Accepted: 1 tokens, Drafted: 2 tokens, Per-position acceptance rate: 0.500, Avg Draft acceptance rate: 50.0%
(APIServer pid=1) INFO 08-29 15:31:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:31:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.73, Accepted throughput: 1.60 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 16 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.727, Avg Draft acceptance rate: 72.7%
(APIServer pid=1) INFO 08-29 15:32:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.78, Accepted throughput: 1.80 tokens/s, Drafted throughput: 2.30 tokens/s, Accepted: 18 tokens, Drafted: 23 tokens, Per-position acceptance rate: 0.783, Avg Draft acceptance rate: 78.3%
(APIServer pid=1) INFO 08-29 15:32:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.86, Accepted throughput: 1.90 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 19 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.864, Avg Draft acceptance rate: 86.4%
(APIServer pid=1) INFO 08-29 15:32:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 2.20 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 22 tokens, Drafted: 22 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 08-29 15:32:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.82, Accepted throughput: 1.80 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 18 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.818, Avg Draft acceptance rate: 81.8%
(APIServer pid=1) INFO 08-29 15:32:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.73, Accepted throughput: 1.60 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 16 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.727, Avg Draft acceptance rate: 72.7%
(APIServer pid=1) INFO 08-29 15:32:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.59, Accepted throughput: 1.30 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 13 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.591, Avg Draft acceptance rate: 59.1%
(APIServer pid=1) INFO 08-29 15:33:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:33:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.67, Accepted throughput: 1.40 tokens/s, Drafted throughput: 2.10 tokens/s, Accepted: 14 tokens, Drafted: 21 tokens, Per-position acceptance rate: 0.667, Avg Draft acceptance rate: 66.7%
(APIServer pid=1) INFO 08-29 15:33:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:33:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.73, Accepted throughput: 1.60 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 16 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.727, Avg Draft acceptance rate: 72.7%
(APIServer pid=1) INFO: 100.69.44.242:47770 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=1) INFO: 100.69.44.242:47780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:33:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:33:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.90, Accepted throughput: 1.80 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 18 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.900, Avg Draft acceptance rate: 90.0%
(APIServer pid=1) INFO 08-29 15:33:34 [loggers.py:310] Engine 000: Avg prompt throughput: 299.8 tokens/s, Avg generation throughput: 2.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:33:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.59, Accepted throughput: 1.00 tokens/s, Drafted throughput: 1.70 tokens/s, Accepted: 10 tokens, Drafted: 17 tokens, Per-position acceptance rate: 0.588, Avg Draft acceptance rate: 58.8%
(APIServer pid=1) INFO 08-29 15:33:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:33:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.86, Accepted throughput: 1.90 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 19 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.864, Avg Draft acceptance rate: 86.4%
(APIServer pid=1) INFO 08-29 15:33:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:33:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.48, Accepted throughput: 1.00 tokens/s, Drafted throughput: 2.10 tokens/s, Accepted: 10 tokens, Drafted: 21 tokens, Per-position acceptance rate: 0.476, Avg Draft acceptance rate: 47.6%
(APIServer pid=1) INFO 08-29 15:34:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:34:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.86, Accepted throughput: 1.90 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 19 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.864, Avg Draft acceptance rate: 86.4%
(APIServer pid=1) INFO 08-29 15:34:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:34:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.95, Accepted throughput: 2.10 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 21 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.955, Avg Draft acceptance rate: 95.5%
(APIServer pid=1) INFO: 100.69.44.242:49454 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=1) INFO: 100.69.44.242:49458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:34:24 [loggers.py:310] Engine 000: Avg prompt throughput: 693.1 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:34:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 0.60 tokens/s, Drafted throughput: 0.60 tokens/s, Accepted: 6 tokens, Drafted: 6 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 08-29 15:34:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:34:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.67, Accepted throughput: 1.40 tokens/s, Drafted throughput: 2.10 tokens/s, Accepted: 14 tokens, Drafted: 21 tokens, Per-position acceptance rate: 0.667, Avg Draft acceptance rate: 66.7%
(APIServer pid=1) INFO 08-29 15:34:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:34:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.55, Accepted throughput: 1.10 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 11 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.550, Avg Draft acceptance rate: 55.0%
(APIServer pid=1) INFO 08-29 15:34:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:34:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.45, Accepted throughput: 0.90 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 9 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.450, Avg Draft acceptance rate: 45.0%
(APIServer pid=1) INFO 08-29 15:35:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.90, Accepted throughput: 1.80 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 18 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.900, Avg Draft acceptance rate: 90.0%
(APIServer pid=1) INFO 08-29 15:35:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 1.50 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 15 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.750, Avg Draft acceptance rate: 75.0%
(APIServer pid=1) INFO 08-29 15:35:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 1.50 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 15 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.750, Avg Draft acceptance rate: 75.0%
(APIServer pid=1) INFO 08-29 15:35:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.65, Accepted throughput: 1.30 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 13 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.650, Avg Draft acceptance rate: 65.0%
(APIServer pid=1) INFO 08-29 15:35:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 1.50 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 15 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.750, Avg Draft acceptance rate: 75.0%
(APIServer pid=1) INFO 08-29 15:35:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.50, Accepted throughput: 1.00 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 10 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.500, Avg Draft acceptance rate: 50.0%
(APIServer pid=1) INFO 08-29 15:36:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.55, Accepted throughput: 1.10 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 11 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.550, Avg Draft acceptance rate: 55.0%
(APIServer pid=1) INFO 08-29 15:36:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.65, Accepted throughput: 1.30 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 13 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.650, Avg Draft acceptance rate: 65.0%
(APIServer pid=1) INFO 08-29 15:36:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 1.50 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 15 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.750, Avg Draft acceptance rate: 75.0%
(APIServer pid=1) INFO 08-29 15:36:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 2.00 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 20 tokens, Drafted: 20 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO: 100.69.44.242:56230 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 100.69.44.242:56236 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=1) INFO 08-29 15:36:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 0.20 tokens/s, Drafted throughput: 0.20 tokens/s, Accepted: 2 tokens, Drafted: 2 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 08-29 15:36:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 79.2%
还是很慢 -
@gk20082000 你這個內存有點少 看有沒有機會加到256以上吧，可以用但不太夠
我這邊可以分享一下雙卡 VMML遇到的問題
如果沒特別需求，會建議參考 https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference
或者是四張卡分別跑一個27B 可以做到50 t/s 以上[實測] R9700 (gfx1201) 雙卡別再試 vLLM tensor-parallel — 附四卡該怎麼配
看到有人問四卡要怎麼處理,先把我們踩過的坑跟數字放上來,省下你重走一遍。
TL;DR
- vLLM TP/PP 在 gfx1201 VM上跑不起來,裸機也一樣,不是你設定錯
- 就算修好,卡越多越慢 —— 有人 8 卡 TP=8 跑到 3.9 tok/s,輸我們 2 卡的 16
- 四卡的正解是四個獨立實例,不是把一個模型切四份
症狀
雙 R9700,
--tensor-parallel-size 2:NCCL error: unhandled cuda error HIP failure 'the operation cannot be performed in the present state' at .../rccl/build/hipify/src/enqueue.cc:1926關鍵細節:
ncclCommInitRank是成功的(tuning index 7 有對到 gfx1201),
死在後面實際 kernel enqueue。所以不是「認不到卡」或「拓樸不通」。試過但沒用的(全部同一個死法)
# 配方 結果 1 kyuz0 image + P2P 全開,不加任何 disable 
2 官方 vllm-openai-rocm:v0.20.2+NCCL_PROTO=Simple3 NCCL_P2P_DISABLE=1+NCCL_SHM_DISABLE=0+HSA_ENABLE_SDMA=0+HSA_FORCE_FINE_GRAIN_PCIE=14 上述 + guest kernel iommu=pt跟 image 版本、傳輸方式(P2P/SHM)、SDMA、fine-grain、IOMMU 全部無關。
這不是虛擬化造成的
Puget Systems 的裸機實測明確寫:
stock vLLM cannot serve a model across two R9700s in either multi-GPU mode
TP 和 PP 都不行 —— vLLM v1 引擎不管哪種平行模式,distributed init
都會做一次 RCCL all-reduce。最貴的一課:修好 P2P 沒有用
我們花了不少工夫把 VM passthrough 的 ReBAR 修好
(BAR0 從 256MB resize 到 32GB + OVMFX-PciMmio64Mb加大 MMIO 窗口)。
修完之後rocm-smi --showtopo確認兩卡 PCIE 直連,
RCCL log 也看得到via P2P/IPC+Connected all rings。然後 TP=2 死在同一行
enqueue.cc。P2P/ReBAR 修復是必要但不充分:它讓 RCCL 通道建立成功了,
但過不了 collective kernel launch。(ReBAR 對 llama.cpp 是純加分,卡間傳輸現在走直連 —— 那部分值得修。)
數字:為什麼「卡越多越慢」
同一顆 Qwen3.8-27B,llama.cpp + Vulkan,ReBAR 已修復:
單卡(模型裝得下)
TTFT 0.1s 解碼 64.6 tok/s(三次重測 70.1 / 69.3 / 69.0) 雙卡 split-mode 對照(llama-bench)
split-mode pp512 tg64 layer1001 t/s 16.4 t/s row906 15.9 tensor(真 TP)507 13.7 llama.cpp 自己也有 TP 型的切法,三種都能跑,但 layer 全面最快 ——
PCIe x8 的卡間延遲吃掉了 TP 的並行收益。八卡 TP=8(Level1Techs,裸機 Gen5 x16,自製修補 RCCL 2.30.4)
~3.9 tok/s43 層 × 86 次同步點被 PCIe 延遲拖垮。他們的硬體條件(裸機直插)
比我們的 VM passthrough 好,8 張卡還是打不過我們 2 張的 16.4。把三組放在一起
配置 解碼 1 卡,模型完整放進去 64.6 tok/s 2 卡 layer-split 16.4 2 卡 tensor-split 13.7 8 卡 TP=8(修補 RCCL) 3.9 在這個硬體上,layer split 勝過 tensor parallel 是物理規律,不是軟體沒調好。
而「不切」又勝過「切」—— 差了 4 倍。
所以四卡該怎麼配?
先問一句:你的模型放得進單張 32GB 嗎?
放得下 → 四個獨立實例,不要切
card0 llama-server instance A ─┐ card1 llama-server instance B ─┤ card2 llama-server instance C ─┼─→ 前面掛一個 OpenAI 相容閘道做負載平衡 card3 llama-server instance D ─┘- 每個實例都是滿速 ~65 tok/s
- 四個併發槽,聚合吞吐 ~260 tok/s
- 一張卡掛掉只損失 1/4,不是整組死
- 完全不碰 RCCL
我們兩卡就是這樣跑的(前面掛 LiteLLM 做別名與 key 管理),
Qwen3.8-27B Q4_K_XL 只有 16.4 GiB,單卡綽綽有餘。放不下 → 用剛好夠的卡數做 layer-split,不是全部下去
一個 60GB 的模型,用 2 張 layer-split,剩下 2 張跑第二個實例 ——
不要 4 張切一份。每多一張卡就多一組同步點,而收益是零。# llama.cpp,不經 RCCL llama-server -m model.gguf --split-mode layer -ngl 99 ... # 預設就會自動分卡,不需要 tensor-split真的非要 vLLM TP 不可 → 兩條路,都不推薦
- GPU 改裸機直跑(VM passthrough 這條已確認無解)
- 自己編譯修補版 RCCL 2.30.4
(修掉 TP≥2 觸發的 DDA IPC 初始化路徑問題,官方 nightly 沒這個補丁)
配NCCL_P2P_DISABLE=1+NCCL_SHM_DISABLE=0
—— 注意是保留 SHM,跟多數人試的「兩個都關」相反
但看上面那張表:修好之後大概率比 llama.cpp 慢。 工程成本換來更慢的東西。
追蹤中
ROCm/rocm-systems#5480
—— 標題就是「RCCL deadlock during vLLM TP=2 inference on dual R9700 (gfx1201/RDNA4)」,
同代 Ryzen X3D + 2× R9700,跟我們幾乎同一台機器。查證時狀態:Open,已指派 @tcgu-amd,無官方回應/時程。
確認是 RCCL 2.27.3 → 2.27.7 之間的回歸性 bug。
回報者自己試NCCL_P2P_DISABLE=1無效
(跟 Level1Techs 8 卡帖裡有效的 workaround 不同 —— 那是另一組版本組合,不能直接套)。要問雙卡/四卡 vLLM TP 有沒有進展,先去看 #5480 狀態變了沒。
觀察名單:SGLang
有人在裸機雙 R9700 跑通 TP=2
(mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference
—— SGLang v0.5.16 + 69 個自製 patch + RCCL 2.27.7 釘版)。裸機 only,而且 TP=2 重啟會間歇 RCCL coredump(作者自己掛 watchdog),
不適合當常駐後端。他的高速主要來自 MoE 模型(A3B 每 token 只動約 3B 參數);
dense 27B 在 gfx1201 填不滿 CU 是他親測結論 —— 跟我們的實測一致。
環境:PVE + VM passthrough,2× AMD Radeon AI PRO R9700 (gfx1201),
ROCm 7.14,llama.cpp + Vulkan (kyuz0/amd-r9700-toolboxes:vulkan-radv)。 -
谢谢各位大佬，看来可以试试每个卡跑一个4bit量化的27b模型。就是不知道4位量化和现在的fp8会不会能力差的比较大
-
差距有，但没想象中大：FP8 基本是无损档（27B FP8 体积约 28GB，业界当"近无损"用，通用评分掉 0.5% 以内）；4bit（Q4_K_M / IQ4_XS 约 16GB）大约掉 2~4%。实际使用区别集中在"精确性"场景：代码生成、长文本引用、严格格式输出这类，4bit 偶尔会错引用符号或格式，FP8 基本不会。
你的情况：4×R9700 每卡 32G，跑 27B FP8 本来就是正解，tensor 并行下每卡权重才 7G，KV 余量很足。4bit 的真正意义是"用质量换容量/并发"——想每卡独立跑一个实例、或把上下文拉得极长时再降不迟。如果当前 FP8 已经跑顺了，没必要为省显存降级。
