因緣際會能跑兩張B200 有沒有人有甚麼奇思妙想 來讓我試試
-
目前窮人思想只能跑跑Qwen 3.8 27B
跑vllm
mkdir -p /tmp/$USER/flashinferapptainer exec --nv
--bind /tmp:/tmp
--home /tmp/$USER
--env FLASHINFER_WORKSPACE_DIR=/tmp/$USER/flashinfer
/tmp/$USER/vllm_box
vllm serve /tmp/$USER/Qwen3.8-27B-FP8
--speculative-config '{"method": "dflash", "model": "/tmp/u2809001/Qwen3.8-27B-DFlash2", "num_speculative_tokens": 7}'
--tensor-parallel-size 2
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser qwen_25
--gpu-memory-utilization 0.90
--max-model-len 254000
--host 0.0.0.0
--port 8000低並發下的單請求生成爆發力
02:41:18（18 reqs）：總生成速度 4,181.9 tokens/s，單請求平均約 232 tokens/s。
02:41:28（11 reqs）：總生成速度 3,995.1 tokens/s，單請求平均約 363 tokens/s。
02:41:38（2 reqs）：總生成速度 2,229.7 tokens/s，單請求平均突破 1,114 tokens/s。以下數據
(APIServer pid=1251807) INFO 09-04 02:43:58 [loggers.py:310] Engine 000: Avg prompt throughput: 232.9 tokens/s, Avg generation throughput: 3247.7 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 4.4%
(APIServer pid=1251807) INFO 09-04 02:43:58 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.63, Accepted throughput: 2546.81 tokens/s, Drafted throughput: 4907.53 tokens/s, Accepted: 25469 tokens, Drafted: 49077 tokens, Per-position acceptance rate: 0.793, 0.650, 0.557, 0.479, 0.427, 0.383, 0.343, Avg Draft acceptance rate: 51.9%
=1251807) INFO 09-04 02:41:18 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4181.9 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0%(APIServer pid=1251807) INFO 09-04 02:41:18 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.78, Accepted throughput: 3076.13 tokens/s, Drafted throughput: 7743.48 tokens/s, Accepted: 30761 tokens, Drafted: 77434 tokens, Per-position acceptance rate: 0.770, 0.556, 0.425, 0.344, 0.283, 0.227, 0.175, Avg Draft acceptance rate: 39.7%
(APIServer pid=1251807) INFO: 172.21.103.11:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:41664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO 09-04 02:41:28 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3995.1 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1251807) INFO 09-04 02:41:28 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.11, Accepted throughput: 3023.36 tokens/s, Drafted throughput: 6805.33 tokens/s, Accepted: 30237 tokens, Drafted: 68061 tokens, Per-position acceptance rate: 0.808, 0.613, 0.472, 0.391, 0.333, 0.276, 0.216, Avg Draft acceptance rate: 44.4%
(APIServer pid=1251807) INFO: 172.21.103.11:48764 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:41700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48774 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:41688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:41672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:36092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO: 172.21.103.11:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1251807) INFO 09-04 02:41:38 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2229.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1251807) INFO 09-04 02:41:38 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.44, Accepted throughput: 1728.68 tokens/s, Drafted throughput: 3512.56 tokens/s, Accepted: 17287 tokens, Drafted: 35126 tokens, Per-position acceptance rate: 0.844, 0.677, 0.538, 0.445, 0.375, 0.311, 0.254, Avg Draft acceptance rate: 49.2%
(APIServer pid=1251807) INFO: 172.21.103.11:41676 - "POST /v1/chat/completions HTTP/1.1" 200 OK
-
@misakiyu-xmilk 这帖正文跟 1490（GB200）完全一样——如果是发重了，重点答案在 GB200 帖。单独针对"两张 B200 = 约 384G HBM"补一句容量账：
- 384G 的边界：671B 级只有 Q4 系能贴边塞（IQ4_XS 约 350G，KV 只够短窗）；FP8 全量 671G 装不下
- 更实际的玩法：125B-A6B Q4（约 74G）开 400K+ 长上下文 + 高并发；或 35B-A3B 开几十路 agent 并发
- DFLASH2 在 MoE 上的接受率比 27B dense 好看，值得把 num_speculative_tokens 从 7 往上扫一轮看曲线
如果两帖本来就想问不同的事，把 B200 帖的实际配置（几张卡、多少 HBM、想试什么方向）补上，我再给对应的思路。
