目前窮人思想只能跑跑Qwen 3.8 27B
跑vllm
mkdir -p /tmp/$USER/flashinfer
apptainer exec --nv
--bind /tmp:/tmp
--home /tmp/$USER
--env FLASHINFER_WORKSPACE_DIR=/tmp/$USER/flashinfer
/tmp/$USER/vllm_box
vllm serve /tmp/$USER/Qwen3.8-27B-FP8
--speculative-config '{"method": "dflash", "model": "/tmp/u2809001/Qwen3.8-27B-DFlash2", "num_speculative_tokens": 7}'
--tensor-parallel-size 2
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser qwen_25
--gpu-memory-utilization 0.90
--max-model-len 254000
--host 0.0.0.0
--port 8000
低並發下的單請求生成爆發力
02:41:18（18 reqs）：總生成速度 4,181.9 tokens/s，單請求平均約 232 tokens/s。
02:41:28（11 reqs）：總生成速度 3,995.1 tokens/s，單請求平均約 363 tokens/s。
02:41:38（2 reqs）：總生成速度 2,229.7 tokens/s，單請求平均突破 1,114 tokens/s。
以下數據
(APIServer pid=1251807) INFO 09-04 02:43:58 [loggers.py:310] Engine 000: Avg prompt throughput: 232.9 tokens/s, Avg generation throughput: 3247.7 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 4.4%
(APIServer pid=1251807) INFO 09-04 02:43:58 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.63, Accepted throughput: 2546.81 tokens/s, Drafted throughput: 4907.53 tokens/s, Accepted: 25469 tokens, Drafted: 49077 tokens, Per-position acceptance rate: 0.793, 0.650, 0.557, 0.479, 0.427, 0.383, 0.343, Avg Draft acceptance rate: 51.9%
=1251807) INFO 09-04 02:41:18 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4181.9 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0%
(APIServer pid=1251807) INFO 09-04 02:41:18 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.78, Accepted throughput: 3076.13 tokens/s, Drafted throughput: 7743.48 tokens/s, Accepted: 30761 tokens, Drafted: 77434 tokens, Per-position acceptance rate: 0.770, 0.556, 0.425, 0.344, 0.283, 0.227, 0.175, Avg Draft acceptance rate: 39.7%
(APIServer pid=1251807) INFO: 172.21.103.11:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:41664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO 09-04 02:41:28 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3995.1 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1251807) INFO 09-04 02:41:28 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.11, Accepted throughput: 3023.36 tokens/s, Drafted throughput: 6805.33 tokens/s, Accepted: 30237 tokens, Drafted: 68061 tokens, Per-position acceptance rate: 0.808, 0.613, 0.472, 0.391, 0.333, 0.276, 0.216, Avg Draft acceptance rate: 44.4%
(APIServer pid=1251807) INFO: 172.21.103.11:48764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:41700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:41688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:41672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO 09-04 02:41:38 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2229.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1251807) INFO 09-04 02:41:38 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.44, Accepted throughput: 1728.68 tokens/s, Drafted throughput: 3512.56 tokens/s, Accepted: 17287 tokens, Drafted: 35126 tokens, Per-position acceptance rate: 0.844, 0.677, 0.538, 0.445, 0.375, 0.311, 0.254, Avg Draft acceptance rate: 49.2%
(APIServer pid=1251807) INFO: 172.21.103.11:41676 - "POST /v1/chat/completions HTTP/1.1" 200 OK