rtx pro 6000 max-q配合qwen3.6 MTP开到20收获前所有见的效率

[Tony Xu 0]

之前一直在做模型输出结构优化，MTP一直用在3，生成速度大概稳定在85 tokens/s，无意间发现MTP第三位居然稳定在0.97以上，后续索性把MTP加到10后第十位也居然稳定在0.95以上，干脆一不做二不休再次翻倍开20，单次请求得到如下log。还没有完全优化完，只是边做工作边刻意的想着优化结构化输出，先把工作完成等到后续性能调优再看能不能让数据更漂亮
你怎么看@xiaote

(APIServer pid=1565552) INFO 07-28 20:35:19 [loggers.py:273] Engine 000: Avg prompt throughput: 1818.1 tokens/s, Avg generation throughput: 71.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.7%, Prefix cache hit rate: 18.3%
(APIServer pid=1565552) INFO 07-28 20:35:19 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 15.13, Accepted throughput: 0.19 tokens/s, Drafted throughput: 0.22 tokens/s, Accepted: 664 tokens, Drafted: 769 tokens, Per-position acceptance rate: 0.979, 0.894, 0.894, 0.894, 0.872, 0.872, 0.872, 0.809, 0.787, 0.766, 0.745, 0.681, 0.638, 0.553, 0.532, 0.532, 0.468, 0.447, 0.447, 0.447, Avg Draft acceptance rate: 86.3%
(APIServer pid=1565552) INFO 07-28 20:35:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 241.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.3%, Prefix cache hit rate: 18.3%
(APIServer pid=1565552) INFO 07-28 20:35:29 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 19.05, Accepted throughput: 229.21 tokens/s, Drafted throughput: 249.11 tokens/s, Accepted: 2292 tokens, Drafted: 2491 tokens, Per-position acceptance rate: 1.000, 0.992, 0.976, 0.976, 0.969, 0.961, 0.953, 0.953, 0.937, 0.929, 0.913, 0.913, 0.898, 0.866, 0.850, 0.835, 0.819, 0.795, 0.764, 0.748, Avg Draft acceptance rate: 92.0%
(APIServer pid=1565552) INFO 07-28 20:35:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 244.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 18.3%
(APIServer pid=1565552) INFO 07-28 20:35:39 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 19.55, Accepted throughput: 231.89 tokens/s, Drafted throughput: 247.69 tokens/s, Accepted: 2319 tokens, Drafted: 2477 tokens, Per-position acceptance rate: 1.000, 0.992, 0.992, 0.992, 0.992, 0.984, 0.984, 0.984, 0.976, 0.976, 0.968, 0.944, 0.920, 0.904, 0.880, 0.880, 0.824, 0.808, 0.792, 0.760, Avg Draft acceptance rate: 93.6%
(APIServer pid=1565552) INFO 07-28 20:35:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 248.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 26.0%, Prefix cache hit rate: 18.3%
(APIServer pid=1565552) INFO 07-28 20:35:49 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 20.22, Accepted throughput: 236.40 tokens/s, Drafted throughput: 245.90 tokens/s, Accepted: 2364 tokens, Drafted: 2459 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 0.984, 0.984, 0.984, 0.967, 0.967, 0.959, 0.951, 0.943, 0.927, 0.902, 0.894, 0.886, 0.870, Avg Draft acceptance rate: 96.1%
(APIServer pid=1565552) INFO: 127.0.0.1:34096 - "POST /v1/chat/completions HTTP/1.1" 200 OK

[kos or]

@Tony-Xu-0 said:

Avg generation throughput: 241.9 tokens/s

MTP = 20 可以高達這個速度嗎？

[Tony Xu 0]

@kos-or 说:

@Tony-Xu-0 said:

Avg generation throughput: 241.9 tokens/s

MTP = 20 可以高達這個速度嗎？

是的，但是要优化输出结构，实际之后又试了MTP=30/40，请求处理跑起来以后生成速度稳定上350 tokens/s，但是效率收益递减，没再开

[kop wang]

但是prefill速度也降了。总体上来讲，目前的重Agent场景，总体性能，prefill和decode速度各占一半。

[terry]

@kop-wang Agent场景的话就用SG-Lang了，如果要是考虑高并发API访问，VLLM有自己的优势，吐字速度确实是比SG-Lang强。

[Tony Xu 0]

@kop-wang 说:

但是prefill速度也降了。总体上来讲，目前的重Agent场景，总体性能，prefill和decode速度各占一半。

agent得另算，我使用场景关注并发，不关注TTFT等延迟。这个log现在不优化直接开batch=4并发已经可以输出接近1000 tokens/s，还是在我把vllm的gpu利用率限制在0.55情况下，这要升级到0.92后续继续优化我都不敢想，没准直接上1800了