抡锤者

gk20082000

感谢各位大佬，最后用双卡7900xtx VLLM qwen3.8 爽玩agent pp1600 tg 160+ (附 mtp/7900xtx全攻略）这个帖子里大佬给的https://hub.docker.com/r/capicua25x/vllm-rocm-rdna4，用这个镜像成功解决问题了

gk20082000

@iamvirus

@iamvirus 说:

@gk20082000
我又来做好人了，这里有现成的作业不去抄？
https://hub.docker.com/r/capicua25x/vllm-rocm-rdna4

太感谢了，这个镜像直接把我的4卡R9700带到起飞了，直接解决了困扰我好多天的问题。

gk20082000

r9700，rdna4能按这个跑么？

gk20082000

谢谢各位大佬，看来可以试试每个卡跑一个4bit量化的27b模型。就是不知道4位量化和现在的fp8会不会能力差的比较大

gk20082000

(APIServer pid=1) INFO: 127.0.0.1:51838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 127.0.0.1:51872 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:29:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 60.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 75.5%
(APIServer pid=1) INFO 08-29 15:29:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.81, Accepted throughput: 27.10 tokens/s, Drafted throughput: 33.50 tokens/s, Accepted: 271 tokens, Drafted: 335 tokens, Per-position acceptance rate: 0.809, Avg Draft acceptance rate: 80.9%
(APIServer pid=1) INFO 08-29 15:29:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 75.5%
(APIServer pid=1) INFO: 100.69.44.242:43230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:31:14 [loggers.py:310] Engine 000: Avg prompt throughput: 331.0 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 77.1%
(APIServer pid=1) INFO 08-29 15:31:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.70, Accepted throughput: 0.16 tokens/s, Drafted throughput: 0.23 tokens/s, Accepted: 16 tokens, Drafted: 23 tokens, Per-position acceptance rate: 0.696, Avg Draft acceptance rate: 69.6%
(APIServer pid=1) INFO 08-29 15:31:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 77.1%
(APIServer pid=1) INFO 08-29 15:31:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.73, Accepted throughput: 2.20 tokens/s, Drafted throughput: 3.00 tokens/s, Accepted: 22 tokens, Drafted: 30 tokens, Per-position acceptance rate: 0.733, Avg Draft acceptance rate: 73.3%
(APIServer pid=1) INFO: 100.69.44.242:42916 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 100.69.44.242:42922 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=1) INFO: 100.69.44.242:42924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:31:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 77.1%
(APIServer pid=1) INFO 08-29 15:31:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 1.50 tokens/s, Drafted throughput: 1.50 tokens/s, Accepted: 15 tokens, Drafted: 15 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 08-29 15:31:44 [loggers.py:310] Engine 000: Avg prompt throughput: 1610.9 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:31:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.50, Accepted throughput: 0.10 tokens/s, Drafted throughput: 0.20 tokens/s, Accepted: 1 tokens, Drafted: 2 tokens, Per-position acceptance rate: 0.500, Avg Draft acceptance rate: 50.0%
(APIServer pid=1) INFO 08-29 15:31:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:31:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.73, Accepted throughput: 1.60 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 16 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.727, Avg Draft acceptance rate: 72.7%
(APIServer pid=1) INFO 08-29 15:32:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.78, Accepted throughput: 1.80 tokens/s, Drafted throughput: 2.30 tokens/s, Accepted: 18 tokens, Drafted: 23 tokens, Per-position acceptance rate: 0.783, Avg Draft acceptance rate: 78.3%
(APIServer pid=1) INFO 08-29 15:32:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.86, Accepted throughput: 1.90 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 19 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.864, Avg Draft acceptance rate: 86.4%
(APIServer pid=1) INFO 08-29 15:32:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 2.20 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 22 tokens, Drafted: 22 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 08-29 15:32:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.82, Accepted throughput: 1.80 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 18 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.818, Avg Draft acceptance rate: 81.8%
(APIServer pid=1) INFO 08-29 15:32:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.73, Accepted throughput: 1.60 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 16 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.727, Avg Draft acceptance rate: 72.7%
(APIServer pid=1) INFO 08-29 15:32:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:32:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.59, Accepted throughput: 1.30 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 13 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.591, Avg Draft acceptance rate: 59.1%
(APIServer pid=1) INFO 08-29 15:33:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:33:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.67, Accepted throughput: 1.40 tokens/s, Drafted throughput: 2.10 tokens/s, Accepted: 14 tokens, Drafted: 21 tokens, Per-position acceptance rate: 0.667, Avg Draft acceptance rate: 66.7%
(APIServer pid=1) INFO 08-29 15:33:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:33:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.73, Accepted throughput: 1.60 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 16 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.727, Avg Draft acceptance rate: 72.7%
(APIServer pid=1) INFO: 100.69.44.242:47770 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=1) INFO: 100.69.44.242:47780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:33:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 76.0%
(APIServer pid=1) INFO 08-29 15:33:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.90, Accepted throughput: 1.80 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 18 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.900, Avg Draft acceptance rate: 90.0%
(APIServer pid=1) INFO 08-29 15:33:34 [loggers.py:310] Engine 000: Avg prompt throughput: 299.8 tokens/s, Avg generation throughput: 2.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:33:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.59, Accepted throughput: 1.00 tokens/s, Drafted throughput: 1.70 tokens/s, Accepted: 10 tokens, Drafted: 17 tokens, Per-position acceptance rate: 0.588, Avg Draft acceptance rate: 58.8%
(APIServer pid=1) INFO 08-29 15:33:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:33:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.86, Accepted throughput: 1.90 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 19 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.864, Avg Draft acceptance rate: 86.4%
(APIServer pid=1) INFO 08-29 15:33:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:33:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.48, Accepted throughput: 1.00 tokens/s, Drafted throughput: 2.10 tokens/s, Accepted: 10 tokens, Drafted: 21 tokens, Per-position acceptance rate: 0.476, Avg Draft acceptance rate: 47.6%
(APIServer pid=1) INFO 08-29 15:34:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:34:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.86, Accepted throughput: 1.90 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 19 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.864, Avg Draft acceptance rate: 86.4%
(APIServer pid=1) INFO 08-29 15:34:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 78.1%
(APIServer pid=1) INFO 08-29 15:34:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.95, Accepted throughput: 2.10 tokens/s, Drafted throughput: 2.20 tokens/s, Accepted: 21 tokens, Drafted: 22 tokens, Per-position acceptance rate: 0.955, Avg Draft acceptance rate: 95.5%
(APIServer pid=1) INFO: 100.69.44.242:49454 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=1) INFO: 100.69.44.242:49458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-29 15:34:24 [loggers.py:310] Engine 000: Avg prompt throughput: 693.1 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:34:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 0.60 tokens/s, Drafted throughput: 0.60 tokens/s, Accepted: 6 tokens, Drafted: 6 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 08-29 15:34:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:34:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.67, Accepted throughput: 1.40 tokens/s, Drafted throughput: 2.10 tokens/s, Accepted: 14 tokens, Drafted: 21 tokens, Per-position acceptance rate: 0.667, Avg Draft acceptance rate: 66.7%
(APIServer pid=1) INFO 08-29 15:34:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:34:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.55, Accepted throughput: 1.10 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 11 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.550, Avg Draft acceptance rate: 55.0%
(APIServer pid=1) INFO 08-29 15:34:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:34:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.45, Accepted throughput: 0.90 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 9 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.450, Avg Draft acceptance rate: 45.0%
(APIServer pid=1) INFO 08-29 15:35:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.90, Accepted throughput: 1.80 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 18 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.900, Avg Draft acceptance rate: 90.0%
(APIServer pid=1) INFO 08-29 15:35:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 1.50 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 15 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.750, Avg Draft acceptance rate: 75.0%
(APIServer pid=1) INFO 08-29 15:35:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 1.50 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 15 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.750, Avg Draft acceptance rate: 75.0%
(APIServer pid=1) INFO 08-29 15:35:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.65, Accepted throughput: 1.30 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 13 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.650, Avg Draft acceptance rate: 65.0%
(APIServer pid=1) INFO 08-29 15:35:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 1.50 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 15 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.750, Avg Draft acceptance rate: 75.0%
(APIServer pid=1) INFO 08-29 15:35:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:35:54 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.50, Accepted throughput: 1.00 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 10 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.500, Avg Draft acceptance rate: 50.0%
(APIServer pid=1) INFO 08-29 15:36:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:04 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.55, Accepted throughput: 1.10 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 11 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.550, Avg Draft acceptance rate: 55.0%
(APIServer pid=1) INFO 08-29 15:36:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:14 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.65, Accepted throughput: 1.30 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 13 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.650, Avg Draft acceptance rate: 65.0%
(APIServer pid=1) INFO 08-29 15:36:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:24 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.75, Accepted throughput: 1.50 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 15 tokens, Drafted: 20 tokens, Per-position acceptance rate: 0.750, Avg Draft acceptance rate: 75.0%
(APIServer pid=1) INFO 08-29 15:36:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:34 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 2.00 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 20 tokens, Drafted: 20 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO: 100.69.44.242:56230 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=1) INFO: 100.69.44.242:56236 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=1) INFO 08-29 15:36:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 79.2%
(APIServer pid=1) INFO 08-29 15:36:44 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.00, Accepted throughput: 0.20 tokens/s, Drafted throughput: 0.20 tokens/s, Accepted: 2 tokens, Drafted: 2 tokens, Per-position acceptance rate: 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=1) INFO 08-29 15:36:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 79.2%
还是很慢

gk20082000

之前的启动日志里，有{"log":"(Worker_TP1 pid=728) WARNING 08-27 12:59:09 [fp8_utils.py:852] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=3584,K=5120,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json\n","stream":"stdout","time":"2026-08-27T12:59:09.814899751Z"}，这个是之前启动的日志里面的，应该是在FP8下运行了吧

gk20082000

有分析投标文件是否雷同的需求，所以设置了180k，太大了么？有点奇怪的是qwen3.6这个参数设200k都可以啊，为什么3.8就不行了。vllm-openai-rocm:latest是前天拉的最新的，显示vllm版本是0.28.0，这个怎么判断是不是原生fp8？用bf16的我试了试，结果都一样

gk20082000

整机硬件是技嘉G292-Z20服务器,CPU 是AMD 7K62，内存128G

gk20082000

装了一台4卡r9700的服务器，之前用vllm跑的qwen3.6 27b fp8，然后用hermes通过飞书接入，提供给同事使用。之前感觉速度还不错，前两天改成qwen3.8 27b fp8之后，刚起来用opwebui直接聊天感觉速度还行，用hermes就感觉慢，特别是一多人共用之后，速度拉胯到都没法用了，看日志都不到10个token了。求助各位大佬能怎么优化一下么。启动命令也是deepseek调整了好几次，好像效果都差不多

docker run -d --name qwen38_27B --restart unless-stopped --network host \
  --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --shm-size 64g \
  -e HIP_VISIBLE_DEVICES=0,1,2,3 \
  -e GPU_MAX_HW_QUEUES=1 \
  -e NCCL_PROTO=Simple \
  -v /home/models:/models \
  vllm/vllm-openai-rocm:latest \
  --model /models/Qwen/Qwen3.8-27B-FP8 \
  --served-model-name Qwen3.8-27B-FP8 \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --max-num-seqs 24 \                      # ⭐ 修正为 24
  --max-model-len 180000 \
  --max-num-batched-tokens 24576 \        # ⭐ 保留 24576
  --dtype auto \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --quantization fp8 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

按deepseek给的命令，启动后测试结果好像还可以，但是hermes一工作，就慢到不行了。

============ Serving Benchmark Result ============
Successful requests:                     100
Failed requests:                         0
Maximum request concurrency:             10
Request rate configured (RPS):           10.00
Benchmark duration (s):                  265.96
Total input tokens:                      204800
Total generated tokens:                  51200
Request throughput (req/s):              0.38
Output token throughput (tok/s):         192.51
Peak output token throughput (tok/s):    180.00
Peak concurrent requests:                14.00
Total token throughput (tok/s):          962.56
---------------Time to First Token----------------
Mean TTFT (ms):                          1712.86
Median TTFT (ms):                        1349.10
P99 TTFT (ms):                           4440.84
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          48.11
Median TPOT (ms):                        43.79
P99 TPOT (ms):                           94.28
---------------Inter-token Latency----------------
Mean ITL (ms):                           93.44
Median ITL (ms):                         57.48
P99 ITL (ms):                            1047.97
---------------Speculative Decoding---------------
Acceptance rate (%):                     94.33
Acceptance length:                       1.94
Drafts:                                  26526
Draft tokens:                            26526
Accepted tokens:                         25021
Per-position acceptance (%):
  Position 0:                            94.33
==================================================

还有各位大神，我这配置能跑qwen3.8 flash next么，另一台电脑还有一块r9700，需要的话还能再加一块

gk20082000

我的r9700，用vllm和ollama都能跑起来，ollama跑qwen3.5 27b速度感觉很快，vllm，qwen3.5那个官方int4好像有点问题，爆显存跑不起来，后来跑了qwen3 32b，不知道为什么，比ollama慢太多了