两张Intel B70 共64G显存 vllm 部署 QWEN3.8-27B FP8 的过程及数据。
-
@张光璞 主要是这张卡也不便宜啊
-
对已经 intel上车的同志。这是一个 灯塔。
-
分享一下我的结果 TP=2 PCIe4.0x8 虽然是一个月前测试的了但或许能帮上忙
docker run -d --name vllmb70 --ipc=host --shm-size=32g --dns 192.168.1.242 --device=/dev/dri:/dev/dri --privileged -p 1234:8000 \ -e VLLM_WORKER_MULTIPROC_METHOD=spawn \ -e ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE \ -e ZE_AFFINITY_MASK=0,1 \ -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ -e PYTORCH_ALLOC_CONF=expandable_segments:True \ -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \ -e CCL_SYCL_ALLREDUCE_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_REDUCE_SCATTER_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_ALLGATHERV_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_ALLTOALL_TMP_BUF=1 \ -e http_proxy= \ -e https_proxy= \ -e no_proxy= \ --entrypoint /bin/bash intel/llm-scaler-vllm:0.21.0-b2 -c " python3 -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen3.8-27B-FP8 \ --tokenizer Qwen/Qwen3.8-27B-FP8 \ --gpu-memory-utilization 0.97 \ --max-model-len 262144 \ --kv-cache-dtype auto \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --default-chat-template-kwargs '{\"enable_thinking\": false}' \ --trust-remote-code \ --port 8000 \ --tensor-parallel-size 2 \ --pipeline-parallel-size 1 \ --cudagraph-capture-sizes 1 2 4 \ --dtype half \ "测试结果来自3.6-27b
单请求:- pp16384: ~2400tps
- tg512: 33.3tps
共享上下文大概480k at FP16 KV Cache，fp8的kv cache能上1M
在我的agent swarm 真实使用 16并行峰值能到吐字300tps
···
(APIServer pid=1) INFO 08-03 09:00:52 [loggers.py:273] Engine 000: Avg prompt throughput: 31.9 tokens/s, Avg generation throughput: 294.1 tokens/S, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 50.6%, Prefix cache hit rate: 88.0%
(APIServer pid=1) INFO 08-03 09:01:02 loggers py:273] Engine 000: Avg prompt throughput: 60.7 tokens/s, Avg generation throughput: 280.0 tokens /S, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 52.0%, Prefix cache hit rate: 88.0%
···XPU Graph 有时不太稳定但大致上能接受，但是不能接受intel摆烂的态度 https://github.com/intel/llm-scaler/issues/641
-
，系统 取消固定了此主题
-
分享一下我的结果 TP=2 PCIe4.0x8 虽然是一个月前测试的了但或许能帮上忙
docker run -d --name vllmb70 --ipc=host --shm-size=32g --dns 192.168.1.242 --device=/dev/dri:/dev/dri --privileged -p 1234:8000 \ -e VLLM_WORKER_MULTIPROC_METHOD=spawn \ -e ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE \ -e ZE_AFFINITY_MASK=0,1 \ -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ -e PYTORCH_ALLOC_CONF=expandable_segments:True \ -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \ -e CCL_SYCL_ALLREDUCE_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_REDUCE_SCATTER_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_ALLGATHERV_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_ALLTOALL_TMP_BUF=1 \ -e http_proxy= \ -e https_proxy= \ -e no_proxy= \ --entrypoint /bin/bash intel/llm-scaler-vllm:0.21.0-b2 -c " python3 -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen3.8-27B-FP8 \ --tokenizer Qwen/Qwen3.8-27B-FP8 \ --gpu-memory-utilization 0.97 \ --max-model-len 262144 \ --kv-cache-dtype auto \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --default-chat-template-kwargs '{\"enable_thinking\": false}' \ --trust-remote-code \ --port 8000 \ --tensor-parallel-size 2 \ --pipeline-parallel-size 1 \ --cudagraph-capture-sizes 1 2 4 \ --dtype half \ "测试结果来自3.6-27b
单请求:- pp16384: ~2400tps
- tg512: 33.3tps
共享上下文大概480k at FP16 KV Cache，fp8的kv cache能上1M
在我的agent swarm 真实使用 16并行峰值能到吐字300tps
···
(APIServer pid=1) INFO 08-03 09:00:52 [loggers.py:273] Engine 000: Avg prompt throughput: 31.9 tokens/s, Avg generation throughput: 294.1 tokens/S, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 50.6%, Prefix cache hit rate: 88.0%
(APIServer pid=1) INFO 08-03 09:01:02 loggers py:273] Engine 000: Avg prompt throughput: 60.7 tokens/s, Avg generation throughput: 280.0 tokens /S, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 52.0%, Prefix cache hit rate: 88.0%
···XPU Graph 有时不太稳定但大致上能接受，但是不能接受intel摆烂的态度 https://github.com/intel/llm-scaler/issues/641
-
分享一下我的结果 TP=2 PCIe4.0x8 虽然是一个月前测试的了但或许能帮上忙
docker run -d --name vllmb70 --ipc=host --shm-size=32g --dns 192.168.1.242 --device=/dev/dri:/dev/dri --privileged -p 1234:8000 \ -e VLLM_WORKER_MULTIPROC_METHOD=spawn \ -e ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE \ -e ZE_AFFINITY_MASK=0,1 \ -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ -e PYTORCH_ALLOC_CONF=expandable_segments:True \ -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \ -e CCL_SYCL_ALLREDUCE_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_REDUCE_SCATTER_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_ALLGATHERV_SIMPLE_THRESHOLD=4294967296 \ -e CCL_SYCL_ALLTOALL_TMP_BUF=1 \ -e http_proxy= \ -e https_proxy= \ -e no_proxy= \ --entrypoint /bin/bash intel/llm-scaler-vllm:0.21.0-b2 -c " python3 -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen3.8-27B-FP8 \ --tokenizer Qwen/Qwen3.8-27B-FP8 \ --gpu-memory-utilization 0.97 \ --max-model-len 262144 \ --kv-cache-dtype auto \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --default-chat-template-kwargs '{\"enable_thinking\": false}' \ --trust-remote-code \ --port 8000 \ --tensor-parallel-size 2 \ --pipeline-parallel-size 1 \ --cudagraph-capture-sizes 1 2 4 \ --dtype half \ "测试结果来自3.6-27b
单请求:- pp16384: ~2400tps
- tg512: 33.3tps
共享上下文大概480k at FP16 KV Cache，fp8的kv cache能上1M
在我的agent swarm 真实使用 16并行峰值能到吐字300tps
···
(APIServer pid=1) INFO 08-03 09:00:52 [loggers.py:273] Engine 000: Avg prompt throughput: 31.9 tokens/s, Avg generation throughput: 294.1 tokens/S, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 50.6%, Prefix cache hit rate: 88.0%
(APIServer pid=1) INFO 08-03 09:01:02 loggers py:273] Engine 000: Avg prompt throughput: 60.7 tokens/s, Avg generation throughput: 280.0 tokens /S, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 52.0%, Prefix cache hit rate: 88.0%
···XPU Graph 有时不太稳定但大致上能接受，但是不能接受intel摆烂的态度 https://github.com/intel/llm-scaler/issues/641
