@李明 之前貼過兩張V100 32G跑Qwen3.6 27B Q8的測速給你參考:
model
test
t/s
peak t/s
ttfr (ms)
est_ppt (ms)
e2e_ttft (ms)
qwen-3.6-27b
pp2048
834.28 ± 19.56
2508.84 ± 171.52
2197.12 ± 171.52
2508.84 ± 171.52
qwen-3.6-27b
tg128
30.56 ± 0.79
38.67 ± 1.70
qwen-3.6-27b
pp2048 @ d4096
937.27 ± 20.09
6170.01 ± 97.33
5858.29 ± 97.33
6170.01 ± 97.33
qwen-3.6-27b
tg128 @ d4096
30.53 ± 0.49
38.00 ± 0.82
qwen-3.6-27b
pp2048 @ d8192
952.76 ± 8.60
10102.94 ± 101.45
9791.23 ± 101.45
10102.94 ± 101.45
qwen-3.6-27b
tg128 @ d8192
30.05 ± 0.74
36.67 ± 1.70
qwen-3.6-27b
pp2048 @ d16384
925.11 ± 11.23
18209.55 ± 38.89
17897.84 ± 38.89
18209.55 ± 38.89
qwen-3.6-27b
tg128 @ d16384
27.19 ± 1.32
34.33 ± 0.47
qwen-3.6-27b
pp2048 @ d32768
854.63 ± 5.92
37259.34 ± 281.61
36947.63 ± 281.61
37259.34 ± 281.61
qwen-3.6-27b
tg128 @ d32768
27.10 ± 0.80
35.33 ± 0.94
也有跑orinth-1.0-35b Q8的測試, 他是基於qwen 3.6 35BA3B去微調的, 大概可以100 t/s:
model
test
t/s
peak t/s
ttfr (ms)
est_ppt (ms)
e2e_ttft (ms)
ornith-35b
pp2048
785.61 ± 15.97
2483.07 ± 66.69
2362.38 ± 66.69
2483.07 ± 66.69
ornith-35b
tg128
106.71 ± 4.49
109.00 ± 5.72
ornith-35b
pp2048 @ d4096
803.66 ± 14.44
7060.92 ± 293.27
6940.24 ± 293.27
7060.92 ± 293.27
ornith-35b
tg128 @ d4096
102.15 ± 1.64
104.00 ± 1.41
ornith-35b
pp2048 @ d8192
792.75 ± 2.92
12035.29 ± 105.50
11914.60 ± 105.50
12035.29 ± 105.50
ornith-35b
tg128 @ d8192
103.90 ± 0.75
105.33 ± 0.47
ornith-35b
pp2048 @ d16384
791.20 ± 7.30
21444.59 ± 125.59
21323.90 ± 125.59
21444.59 ± 125.59
ornith-35b
tg128 @ d16384
104.99 ± 4.85
107.33 ± 5.31
ornith-35b
pp2048 @ d32768
763.53 ± 9.87
41675.67 ± 664.23
41554.98 ± 664.23
41675.67 ± 664.23
ornith-35b
tg128 @ d32768
98.68 ± 0.85
101.00 ± 0.82
GGUF模型都可以跑, 其他的幾乎不用玩了. 現在這個容量之下 , 除了Qwen 3.6 27B / 35BA3B好像也沒其他甚麼模型可以用了.
