有人試過這個嗎? 7900xtx 運行Qwen 3.8 FLash Next 有 30T/S, 7百多 PP

[franklee006]

https://www.bilibili.com/video/BV1ALha63Eyd/?trackid=web_pegasus_0.router-web-pegasus-2479516-f6fct.1790052429481.857&vd_source=0f4d937cc5bbfdd01e38a4c0a96ff348

https://github.com/Headmaster218/MoE4All

[laobenxiong]

@franklee006 说:
https://github.com/Headmaster218/MoE4All

这个仓库好像有点搞头啊. 让 AI 看了下仓库, 让它回答了几个关键问题:

---

MoE4All Summary

1. What Inference Engine Does It Use?

MoE4All is a custom Rust-based inference engine forked from infr (kryptic-sh/infr). It's not based on llama.cpp, vLLM, or SGLang. It's a native Rust implementation.

From Cargo.toml:

• infr-core — Core engine
• infr-vulkan — AMD Vulkan backend (main focus)
• infr-metal — Apple Metal backend
• infr-cpu — CPU fallback
• infr-gguf — GGUF model loading

2. Does It Support TP=2?

Yes! It supports tensor parallelism.

From infr.example.toml:

# Device lists: [0, 1] or ["Vulkan0", "Vulkan1"]. pipeline and tensor_parallel
# tensor_parallel = [0, 1]
tp_p2p = true

So you can configure multiple GPUs for tensor parallelism.

3. How Difficult to Convert to Linux?

Medium-High difficulty:

Challenge	Difficulty
Build system	Easy (Cargo works on Linux)
Vulkan backend	Easy (works on Linux with AMDVLK/Mesa)
GPU memory management	Medium (Linux uses different APIs)
Multi-GPU coordination	Medium (different than Windows)
Performance tuning	High (requires testing on Linux)

The code is already cross-platform Rust, but:

• Windows-specific GPU driver APIs may need Linux equivalents
• Memory management differs between Windows and Linux
• The Vulkan layer is already cross-platform

4. Does It Only Use Vulkan?

No! It supports multiple backends:

Backend	Status
Vulkan	Main (AMD GPU focus)
Metal	For Apple GPUs
CPU	Fallback

The project is designed with swappable backends (infr-vulkan, infr-metal, infr-cpu).

Summary

Question	Answer
Inference engine	Custom Rust (fork of infr)
TP=2 support	Yes (tensor_parallel config)
Linux port difficulty	Medium-High (Rust is cross-platform, but GPU APIs need adaptation)
Vulkan only?	No — also Metal and CPU

ReportID: d7eb9374-e784-4ef9-af49-1621406bfd74
ConversationID: a7dbb0f5-74ca-48e0-b9fd-2cfc3d04848a

[Xiaote]

MoE4All 是 Rust 写的自定义推理引擎（fork 自 infr，Vulkan 后端主攻 AMD），不属于 llama.cpp / vLLM / SGLang 那一系。Vulkan 路线的意义主要是：不用等 ROCm 把新算子补齐，直接用 Vulkan compute 跑自定义 kernel。

30 t/s、700+ pp 这个量级是否可信，取决于三件事：

1. 模型是 MoE 且激活参数小（总参数大、每次只过少量专家），24G 显存放得下量化后的权重与 KV；
2. prefill 是算力型，7900XTX 的 fp16/INT8 峰值够，700 pp 不离谱；decode 是带宽型，30 t/s 要看权重加 KV 的等效带宽在 960GB/s 量级内是否站得住；
3. 视频里的设置有没有开 n-gram / 投机解码、KV 是什么 dtype、上下文多长、batch 多少。

单开发者引擎加新后端，建议先用自己固定的 prompt 和上下文长度复现，再把 pp/tg 分开和 llama.cpp 对比，之后决定要不要迁。

[williamlouis]

不要发这种就只有链接的帖子。

[terry]

@laobenxiong 别发大段英文