抡锤者

kaifan

还有好几个错误更正后的

Hardware Metric	Arc Pro B70	Radeon AI PRO R9700	RX 7900 XTX	RTX 3090	RTX 4070	RTX 5060 Ti 16GB	RTX 5070	RTX 5070 Ti	RTX 4090	RTX 5090
Architecture	Xe2	RDNA4	RDNA3	Ampere	Ada	Blackwell	Blackwell	Blackwell	Ada	Blackwell
VRAM	32 GB	32 GB	24 GB	24 GB	12 GB	16 GB	12 GB	16 GB	24 GB	32 GB
Memory Bus	256-bit	256-bit	384-bit	384-bit	192-bit	128-bit	192-bit	256-bit	384-bit	512-bit
Memory Bandwidth	608 GB/s	644.6 GB/s	960 GB/s	936 GB/s	504 GB/s	448 GB/s	672 GB/s	896 GB/s	1008 GB/s	1792 GB/s
FP32	22.9 TF	47.8 TF	61.4 TF	35.6 TF	29.2 TF	23.7 TF	30.8 TF	43.9 TF	82.6 TF	104.8 TF
FP16/BF16 Vector	45.9 TF	95.7 TF	122.8 TF	71.2 TF	58.3 TF	47.4 TF	61.6 TF	87.8 TF	165.2 TF	209.6 TF
FP16/BF16 Matrix Dense	~183 TF*	191.4 TF	~123 TF†	142 TF	233 TF	189.6 TF	248 TF	351.4 TF	330.3 TF	838.4 TF
FP16/BF16 Matrix Sparse	—	382.8 TF	—	284 TF	466 TF	379.2 TF	496 TF	702.8 TF	660.6 TF	1676.8 TF
FP8 Matrix Dense	~367 TF*	382.8 TF	Emulated	Emulated	466 TF	379.2 TF	496 TF	702.8 TF	660.6 TF	1676.8 TF
FP8 Matrix Sparse	—	765.6 TF	—	—	932 TF	758.4 TF	992 TF	1405.6 TF	1321 TF	3353.6 TF
INT8 Dense	367 TOPS	382.8 TOPS	~246 TOPS†	142 TOPS	233 TOPS	189.6 TOPS	248 TOPS	351.4 TOPS	330.3 TOPS	838.4 TOPS
INT8 Sparse	—	765.6 TOPS	—	284 TOPS	466 TOPS	379.2 TOPS	496 TOPS	702.8 TOPS	660.6 TOPS	1676.8 TOPS
INT4 Dense	~734 TOPS*	1531 TOPS	~246 TOPS†	284 TOPS	466 TOPS	379.2 TOPS	496 TOPS	702.8 TOPS	660.6 TOPS	1676.8 TOPS
INT4 Sparse	—	3062 TOPS	—	568 TOPS	932 TOPS	758.4 TOPS	992 TOPS	1405.6 TOPS	1321 TOPS	3353.6 TOPS
Native FP8	Estimated	Yes	No	No	Yes	Yes	Yes	Yes	Yes	Yes
Native FP4	No	No	No	No	No	Yes	Yes	Yes	No	Yes
FP4 Dense	—	—	—	—	—	758.4 TF	992 TF	1405.6 TF	—	1676.8 TF
FP4 Sparse	—	—	—	—	—	1516.8 TF	1984 TF	2811.2 TF	—	3353.6 TF

kaifan

分享一下我的结果 TP=2 PCIe4.0x8 虽然是一个月前测试的了但或许能帮上忙

docker run -d --name vllmb70 --ipc=host --shm-size=32g --dns 192.168.1.242 --device=/dev/dri:/dev/dri --privileged -p 1234:8000 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE \
  -e ZE_AFFINITY_MASK=0,1 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e PYTORCH_ALLOC_CONF=expandable_segments:True \
  -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \
  -e CCL_SYCL_ALLREDUCE_SIMPLE_THRESHOLD=4294967296 \
  -e CCL_SYCL_REDUCE_SCATTER_SIMPLE_THRESHOLD=4294967296 \
  -e CCL_SYCL_ALLGATHERV_SIMPLE_THRESHOLD=4294967296 \
  -e CCL_SYCL_ALLTOALL_TMP_BUF=1 \
  -e http_proxy= \
  -e https_proxy= \
  -e no_proxy= \
  --entrypoint /bin/bash intel/llm-scaler-vllm:0.21.0-b2 -c "
    python3 -m vllm.entrypoints.openai.api_server \
      --model Qwen/Qwen3.8-27B-FP8 \
      --tokenizer Qwen/Qwen3.8-27B-FP8 \
      --gpu-memory-utilization 0.97 \
      --max-model-len 262144 \
      --kv-cache-dtype auto \
      --enable-prefix-caching \
      --enable-auto-tool-choice \
      --tool-call-parser qwen3_coder \
      --default-chat-template-kwargs '{\"enable_thinking\": false}' \
      --trust-remote-code \
      --port 8000 \
      --tensor-parallel-size 2 \
      --pipeline-parallel-size 1 \
      --cudagraph-capture-sizes 1 2 4 \
      --dtype half \
  "

测试结果来自3.6-27b
单请求:

pp16384: ~2400tps
tg512: 33.3tps

共享上下文大概480k at FP16 KV Cache，fp8的kv cache能上1M
在我的agent swarm 真实使用 16并行峰值能到吐字300tps
···
(APIServer pid=1) INFO 08-03 09:00:52 [loggers.py:273] Engine 000: Avg prompt throughput: 31.9 tokens/s, Avg generation throughput: 294.1 tokens/S, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 50.6%, Prefix cache hit rate: 88.0%
(APIServer pid=1) INFO 08-03 09:01:02 loggers py:273] Engine 000: Avg prompt throughput: 60.7 tokens/s, Avg generation throughput: 280.0 tokens /S, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 52.0%, Prefix cache hit rate: 88.0%
···

XPU Graph 有时不太稳定但大致上能接受，但是不能接受intel摆烂的态度 https://github.com/intel/llm-scaler/issues/641

抡锤者

kaifan

帖子