跳转至内容
  • 版块
  • 最新
  • 标签
  • 热门
  • 用户
  • 群组
皮肤
  • 浅色
  • Brite
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • 深色
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor
  • 默认（不使用皮肤）
  • 不使用皮肤
折叠
品牌标识

抡锤者
Billy ShenB

Billy Shen

@Billy Shen
取消关注 关注
关于
帖子
13
主题
4
分享
0
群组
0
粉丝
0
关注
0

帖子

最新 最佳 有争议的
  • vllm v0.22.0 加载qwen3.6 报错 #flashinfer
    Billy ShenB Billy Shen

    @williamlouis 显卡的驱动已经安装了。 GPU Name: NVIDIA RTX 5880 Ada Generation | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 |

    LLM讨论区 vllm
  • 分享一个 vllm v0.22.0版本的安装脚本
    Billy ShenB Billy Shen 
    #!/bin/bash
# ============================================================
#  setup_vllm.sh
#  vLLM 0.22.0 全新环境重建 & 部署脚本
#
#  包管理: uv
#  目标机器: Ubuntu 24.04 + NVIDIA RTX 5880 Ada (SM 8.9)
#            CUDA Driver 595.x  |  CUDA Toolkit 13.2
#            Python 3.12
#
#  用法:
#    chmod +x setup_vllm.sh
#    bash setup_vllm.sh
#
#  脚本执行流程:
#    1. 切换 nvcc → CUDA 13.2
#    2. 安装 / 升级 uv
#    3. 销毁旧虚拟环境
#    4. 用 uv 创建全新 Python 3.12 虚拟环境
#    5. 用 uv 安装 vLLM 0.22.0（--torch-backend=cu130）
#    6. Patch flashinfer sampling.cuh（FlagHeads → SubtractLeft）
#    7. 安装 flashinfer-jit-cache（消除运行时 JIT 编译）
#    8. 清除编译缓存
#    9. 验证环境
# ============================================================

set -euo pipefail

# ── 颜色输出 ──────────────────────────────────────────────
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
info()    { echo -e "${CYAN}[INFO]${NC}  $*"; }
ok()      { echo -e "${GREEN}[OK]${NC}    $*"; }
warn()    { echo -e "${YELLOW}[WARN]${NC}  $*"; }
error()   { echo -e "${RED}[ERROR]${NC} $*"; exit 1; }
section() {
    echo -e "\n${BOLD}══════════════════════════════════════════${NC}"
    echo -e "${BOLD}  $*${NC}"
    echo -e "${BOLD}══════════════════════════════════════════${NC}"
}

# ── 配置变量（按需修改）──────────────────────────────────
VLLM_VERSION="0.22.0"
CUDA_TOOLKIT_VER="13.2"                            # 系统已安装的 CUDA Toolkit
CUDA_PATH="/usr/local/cuda-${CUDA_TOOLKIT_VER}"   # nvcc 所在路径
TORCH_BACKEND="cu130"                              # uv --torch-backend 参数
VENV_DIR="$HOME/vllm_env"                         # 虚拟环境目录
PYTHON_VER="3.12"

# ── 开始 ──────────────────────────────────────────────────
echo ""
echo -e "${BOLD}╔══════════════════════════════════════════════╗${NC}"
echo -e "${BOLD}║   vLLM ${VLLM_VERSION} 全新环境重建脚本 (uv)       ║${NC}"
echo -e "${BOLD}╚══════════════════════════════════════════════╝${NC}"
echo ""

# ============================================================
# Step 1: 切换 nvcc 到 CUDA 13.2
# ============================================================
section "Step 1/9  切换 nvcc → CUDA ${CUDA_TOOLKIT_VER}"

if [ ! -d "$CUDA_PATH" ]; then
    warn "未找到 $CUDA_PATH，当前可用版本："
    ls /usr/local/ | grep cuda || true
    error "请修改脚本顶部 CUDA_TOOLKIT_VER 为实际已安装版本"
fi

# 幂等写入 bashrc
if ! grep -q "cuda-${CUDA_TOOLKIT_VER}/bin" "$HOME/.bashrc" 2>/dev/null; then
    {
        echo ""
        echo "# === CUDA ${CUDA_TOOLKIT_VER} (added by setup_vllm.sh) ==="
        echo "export PATH=${CUDA_PATH}/bin:\$PATH"
        echo "export LD_LIBRARY_PATH=${CUDA_PATH}/lib64:\$LD_LIBRARY_PATH"
        echo "export CUDA_HOME=${CUDA_PATH}"
    } >> "$HOME/.bashrc"
    ok "已写入 ~/.bashrc"
else
    ok "~/.bashrc 中已有 CUDA ${CUDA_TOOLKIT_VER} 配置，跳过"
fi

# 当前 shell 立即生效
export PATH="${CUDA_PATH}/bin:$PATH"
export LD_LIBRARY_PATH="${CUDA_PATH}/lib64:${LD_LIBRARY_PATH:-}"
export CUDA_HOME="${CUDA_PATH}"

NVCC_OUT=$(nvcc --version 2>/dev/null | grep "release" | awk '{print $5,$6}' | tr -d ',' || echo "未找到")
ok "nvcc: ${NVCC_OUT}"

# ============================================================
# Step 2: 安装 / 升级 uv
# ============================================================
section "Step 2/9  安装 / 升级 uv"

if command -v uv &>/dev/null; then
    UV_VER=$(uv --version 2>/dev/null | awk '{print $2}')
    info "已检测到 uv ${UV_VER}，执行升级..."
    uv self update 2>/dev/null && ok "uv 已升级" || warn "uv 升级失败，继续使用当前版本"
else
    info "未检测到 uv，正在安装..."
    curl -LsSf https://astral.sh/uv/install.sh | sh
    ok "uv 安装完成"
fi

# 确保 uv 在 PATH 中
export PATH="$HOME/.local/bin:$PATH"
if ! grep -q '\.local/bin' "$HOME/.bashrc" 2>/dev/null; then
    echo 'export PATH="$HOME/.local/bin:$PATH"' >> "$HOME/.bashrc"
fi

UV_VER=$(uv --version 2>/dev/null | awk '{print $2}' || echo "未知")
ok "uv ${UV_VER} 就绪"

# ============================================================
# Step 3: 销毁旧虚拟环境
# ============================================================
section "Step 3/9  销毁旧虚拟环境 (${VENV_DIR})"

if [ -d "$VENV_DIR" ]; then
    info "正在删除 ${VENV_DIR} ..."
    rm -rf "$VENV_DIR"
    ok "已删除旧虚拟环境"
else
    ok "旧虚拟环境不存在，跳过"
fi

# ============================================================
# Step 4: 用 uv 创建全新虚拟环境
# ============================================================
section "Step 4/9  用 uv 创建 Python ${PYTHON_VER} 虚拟环境"

uv venv "$VENV_DIR" --python "${PYTHON_VER}"
source "$VENV_DIR/bin/activate"
ok "虚拟环境已创建并激活: ${VENV_DIR}"

# ============================================================
# Step 5: 用 uv 安装 vLLM 0.22.0
# ============================================================
section "Step 5/9  用 uv 安装 vLLM ${VLLM_VERSION} (--torch-backend=${TORCH_BACKEND})"

# uv --torch-backend=cu130 自动从 PyTorch 官方 whl 索引拉取 torch 2.11.0+cu130
# vLLM wheel 会同时携带匹配版本的 flashinfer-python / flashinfer-cubin
info "正在安装 vllm==${VLLM_VERSION}（torch 2.11+cu130 约 600MB，请耐心等待）..."

uv pip install "vllm==${VLLM_VERSION}" \
    --torch-backend="${TORCH_BACKEND}" \
    2>&1 | tee /tmp/vllm_install.log \
         | grep -E "^(Resolved|Prepared|Installed|Uninstalled|error|Error|warning)" \
    || true

# 验证安装
INSTALLED_VLLM=$(uv pip show vllm 2>/dev/null | grep "^Version:" | awk '{print $2}' || echo "")
if [ "$INSTALLED_VLLM" != "$VLLM_VERSION" ]; then
    error "vLLM 安装失败（期望 ${VLLM_VERSION}，实际 ${INSTALLED_VLLM:-未安装}）\n       详情: /tmp/vllm_install.log"
fi
ok "vLLM ${INSTALLED_VLLM} 安装成功"

TORCH_VER=$(python -c "import torch; print(torch.__version__)" 2>/dev/null || echo "未知")
ok "torch: ${TORCH_VER}"

FI_PY=$(uv pip show flashinfer-python 2>/dev/null | grep "^Version:" | awk '{print $2}' || echo "未安装")
FI_CB=$(uv pip show flashinfer-cubin  2>/dev/null | grep "^Version:" | awk '{print $2}' || echo "未安装")
ok "flashinfer-python: ${FI_PY}"
ok "flashinfer-cubin:  ${FI_CB}"

# ============================================================
# Step 6: Patch flashinfer sampling.cuh
#         FlagHeads → SubtractLeft
#         原因: flashinfer-cubin 0.6.x 内置 CUB v3.3.2 删除了
#               BlockAdjacentDifference::FlagHeads，改名为 SubtractLeft
# ============================================================
section "Step 6/9  Patch flashinfer sampling.cuh (FlagHeads → SubtractLeft)"

FLASHINFER_DIR=$(python -c \
    "import flashinfer, os; print(os.path.dirname(flashinfer.__file__))" \
    2>/dev/null || echo "")

if [ -z "$FLASHINFER_DIR" ]; then
    warn "无法找到 flashinfer 安装路径，跳过 patch"
else
    SAMPLING_CUH="${FLASHINFER_DIR}/data/include/flashinfer/sampling.cuh"

    if [ ! -f "$SAMPLING_CUH" ]; then
        warn "未找到 ${SAMPLING_CUH}，可能此版本已官方修复，跳过"
    else
        FLAG_COUNT=$(grep -c "\.FlagHeads(" "$SAMPLING_CUH" 2>/dev/null || echo 0)
        PATCHED_COUNT=$(grep -c "\.SubtractLeft(" "$SAMPLING_CUH" 2>/dev/null || echo 0)

        if [ "$FLAG_COUNT" -eq 0 ] && [ "$PATCHED_COUNT" -gt 0 ]; then
            ok "sampling.cuh 已是 patch 后版本，跳过"
        elif [ "$FLAG_COUNT" -eq 0 ]; then
            ok "未发现 FlagHeads，无需 patch"
        else
            cp "$SAMPLING_CUH" "${SAMPLING_CUH}.bak"
            sed -i 's/\.FlagHeads(/.SubtractLeft(/g' "$SAMPLING_CUH"
            ok "Patch 完成：${FLAG_COUNT} 处 FlagHeads → SubtractLeft（原文件已备份）"
        fi
    fi
fi

# ============================================================
# Step 7: 安装 flashinfer-jit-cache（消除运行时 JIT 编译）
# ============================================================
section "Step 7/9  安装 flashinfer-jit-cache（预编译 kernel）"

info "从 flashinfer.ai 安装 jit-cache (cu130)..."

uv pip install flashinfer-jit-cache \
    --index-url "https://flashinfer.ai/whl/cu130" \
    2>&1 | grep -E "^(Resolved|Prepared|Installed|error|Error|No solution|warning)" \
    || true

JIT_VER=$(uv pip show flashinfer-jit-cache 2>/dev/null | grep "^Version:" | awk '{print $2}' || echo "")
if [ -n "$JIT_VER" ]; then
    ok "flashinfer-jit-cache ${JIT_VER} 安装成功（首次启动无需 JIT 编译）"
else
    warn "flashinfer-jit-cache 未安装（cu130 wheel 可能尚未发布）"
    warn "首次启动会自动 JIT 编译，约需 2~5 分钟，属正常现象"
fi

# ============================================================
# Step 8: 清除编译缓存
# ============================================================
section "Step 8/9  清除所有编译缓存"

for CACHE_DIR in \
    "$HOME/.cache/flashinfer" \
    "$HOME/.cache/vllm/torch_compile_cache" \
    "$HOME/.cache/vllm/torch_aot_compile"
do
    if [ -d "$CACHE_DIR" ]; then
        rm -rf "$CACHE_DIR"
        ok "已清除: $CACHE_DIR"
    else
        info "不存在，跳过: $CACHE_DIR"
    fi
done

# ============================================================
# Step 9: 验证环境
# ============================================================
section "Step 9/9  验证环境"

echo ""
echo "  ┌───────────────────────────────────────────────────┐"
echo "  │  环境验证报告                                      │"
echo "  ├───────────────────────────────────────────────────┤"

_row() { printf "  │  %-22s %-24s │\n" "$1" "$2"; }

_row "uv:"                "$(uv --version 2>/dev/null | awk '{print $2}' || echo '未找到')"
_row "nvcc:"              "$(nvcc --version 2>/dev/null | grep 'release' | awk '{print $5,$6}' | tr -d ',' || echo '未找到')"
_row "Python:"            "$(python --version 2>&1)"
_row "torch:"             "$(python -c 'import torch; print(torch.__version__)' 2>/dev/null || echo '未安装')"
_row "torch CUDA:"        "$(python -c 'import torch; print(torch.version.cuda)' 2>/dev/null || echo 'N/A')"
_row "GPU:"               "$(python -c 'import torch; print(torch.cuda.get_device_name(0))' 2>/dev/null || echo '无法获取')"
_row "vllm:"              "$(uv pip show vllm 2>/dev/null | grep '^Version:' | awk '{print $2}' || echo '未安装')"
_row "flashinfer-python:" "$(uv pip show flashinfer-python 2>/dev/null | grep '^Version:' | awk '{print $2}' || echo '未安装')"
_row "flashinfer-cubin:"  "$(uv pip show flashinfer-cubin  2>/dev/null | grep '^Version:' | awk '{print $2}' || echo '未安装')"
_row "flashinfer-jit:"    "$(uv pip show flashinfer-jit-cache 2>/dev/null | grep '^Version:' | awk '{print $2}' || echo '未安装')"

echo "  ├───────────────────────────────────────────────────┤"

# fast_decode_plan 导入测试
if python -c "from flashinfer.decode import fast_decode_plan" 2>/dev/null; then
    _row "fast_decode_plan:" "✅ 可导入"
else
    _row "fast_decode_plan:" "❌ 导入失败"
fi

# sampling.cuh patch 检查
if [ -n "${FLASHINFER_DIR:-}" ]; then
    CUH="${FLASHINFER_DIR}/data/include/flashinfer/sampling.cuh"
    if [ -f "$CUH" ]; then
        REMAIN=$(grep -c "\.FlagHeads(" "$CUH" 2>/dev/null || echo 0)
        [ "$REMAIN" -eq 0 ] \
            && _row "sampling.cuh patch:" "✅ 已修复" \
            || _row "sampling.cuh patch:" "❌ 仍有 FlagHeads"
    fi
fi

echo "  └───────────────────────────────────────────────────┘"

# ============================================================
# 完成提示
# ============================================================
echo ""
echo -e "${BOLD}╔══════════════════════════════════════════════╗${NC}"
echo -e "${BOLD}║   ✅  环境重建完成                           ║${NC}"
echo -e "${BOLD}╚══════════════════════════════════════════════╝${NC}"
echo ""
echo "  激活环境并启动 vLLM："
echo ""
echo -e "  ${CYAN}source ${VENV_DIR}/bin/activate${NC}"
echo -e "  ${CYAN}bash ~/vllm/start_vllm.sh${NC}"
echo ""
echo "  启动成功标志（日志中出现以下两行）："
echo "    INFO [topk_topp_sampler.py] Using FlashInfer for top-p & top-k sampling."
echo "    INFO [api_server.py] Application startup complete."
echo ""
    LLM讨论区 vllm
  • vllm v0.22.0 加载qwen3.6 报错 #flashinfer
    Billy ShenB Billy Shen

    vllm 升级到v0.22.0后加载大模型会报错，Deepseek分析vllm的模型加载日志后，提示是从排查flashinfer的问题入手。

    群友遇到过这样的问题吗？ 是怎么解决的？

    GPU Information

    Sat May 30 19:14:11 2026
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 |
    +-----------------------------------------+------------------------+----------------------+
    | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
    | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
    | | | MIG M. |
    |=========================================+========================+======================|
    | 0 NVIDIA RTX 5880 Ada Gene... Off | 00000000:AF:00.0 Off | Off |
    | 30% 51C P8 31W / 285W | 15MiB / 49140MiB | 0% Default |
    | | | N/A |
    +-----------------------------------------+------------------------+----------------------+

    +-----------------------------------------------------------------------------------------+

    Processes:
    GPU GI CI PID Type Process name GPU Memory
    ID ID Usage
    0 N/A N/A 4146 G /usr/lib/xorg/Xorg 4MiB

    +-----------------------------------------------------------------------------------------+

    ====================================================
    CUDA Version

    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2023 NVIDIA Corporation
    Built on Fri_Jan__6_16:45:21_PST_2023
    Cuda compilation tools, release 12.0, V12.0.140
    Build cuda_12.0.r12.0/compiler.32267302_0

    ====================================================
    Python Version

    Python 3.12.3

    ====================================================
    vLLM Package

    Name: vllm
    Version: 0.22.0
    Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
    Home-page: https://github.com/vllm-project/vllm
    Author: vLLM Team
    Author-email:
    License-Expression: Apache-2.0
    Location: /home/ames/vllm_env/lib/python3.12/site-packages
    Requires: aiohttp, anthropic, apache-tvm-ffi, blake3, cachetools, cbor2, cloudpickle, compressed-tensors, depyf, diskcache, einops, fastapi, fastsafetensors, filelock, flashinfer-cubin, flashinfer-python, gguf, humming-kernels, ijson, lark, llguidance, lm-format-enforcer, mcp, mistral_common, model-hosting-container-standards, msgspec, ninja, numba, numpy, nvidia-cudnn-frontend, nvidia-cutlass-dsl, openai, openai-harmony, opencv-python-headless, opentelemetry-api, opentelemetry-exporter-otlp, opentelemetry-sdk, opentelemetry-semantic-conventions-ai, outlines_core, partial-json-parser, pillow, prometheus-fastapi-instrumentator, prometheus_client, protobuf, psutil, py-cpuinfo, pybase64, pydantic, python-json-logger, pyyaml, pyzmq, quack-kernels, regex, requests, safetensors, sentencepiece, setproctitle, setuptools, six, tiktoken, tilelang, tokenizers, tokenspeed-mla, torch, torchaudio, torchvision, tqdm, transformers, typing_extensions, watchfiles, xgrammar
    Required-by:

    ====================================================
    FlashInfer Package

    Name: flashinfer-python
    Version: 0.6.11.post2
    Summary: FlashInfer: Kernel Library for LLM Serving
    Home-page: https://github.com/flashinfer-ai/flashinfer
    Author: FlashInfer team
    Author-email:
    License-Expression: Apache-2.0
    Location: /home/ames/vllm_env/lib/python3.12/site-packages
    Requires: apache-tvm-ffi, click, cuda-tile, einops, ninja, numpy, nvidia-cudnn-frontend, nvidia-cutlass-dsl, nvidia-ml-py, packaging, requests, tabulate, torch, tqdm
    Required-by: vllm

    ====================================================
    Core AI Packages

    flashinfer-cubin 0.6.11.post2
    flashinfer-python 0.6.11.post2
    tokenspeed-triton 3.7.10.post20260505
    torch 2.11.0+cu130
    torch_c_dlpack_ext 0.1.5
    torchaudio 2.11.0+cu130
    torchvision 0.26.0+cu130
    triton 3.6.0
    vllm 0.22.0

    ====================================================
    PyTorch CUDA Information

    PyTorch Version: 2.11.0+cu130
    CUDA Available: True
    CUDA Version: 13.0
    GPU Count: 1
    GPU Name: NVIDIA RTX 5880 Ada Generation
    GPU Capability: (8, 9)

    ====================================================
    FlashInfer Cache

    总计 27K
    drwxrwxr-x 3 ames ames 3 5月 30 18:55 .
    drwx------ 19 ames ames 21 5月 30 18:55 ..
    drwxrwxr-x 3 ames ames 3 5月 30 18:55 0.6.11.post2

    LLM讨论区 vllm
  • 我的 AI 工作站硬件配置报告
    Billy ShenB Billy Shen

    目前跑的是Qwen/Qwen3.6-35B-A3B-FP8, vllm version 0.21.0, 局域网组网使用了Tailscale, 远程登录没问题. 客户机里面运行Hermes agent. , vLLM + tensor-parallel 的参数自己不会搞, 准备让hermes自己优化.

    分享个启动脚本

    ames@ames-P10DRG:~$ cat vllm/start_vllm.sh

    #!/bin/bash
# =============================================================================
#  Qwen3.6-35B-A3B-FP8  vLLM Optimized Startup Script  v4.0
#  基于 start_qwen3.6_claude.sh v3.1 的架构，针对 RTX 5880 Ada 全面优化
#  优化项:
#    1. max-num-batched-tokens 8192 -> 32768  (吞吐提升 ~4x)
#    2. max-num-seqs 16 -> 32                (并发请求翻倍)
#    3. max-model-len 262144 -> 40960        (MoE 线性注意力层 KV cache 实际上限 ~40K)
#    4. gpu-memory-utilization 0.92 -> 0.90  (更安全，避免 OOM)
#    5. 保留动态 NUMA 检测 + v0.21.0 内置 --numa-bind
# =============================================================================

set -euo pipefail

# --- 配置区 ---
VENV_PATH="$HOME/vllm_env/bin/activate"
MODEL_NAME="Qwen/Qwen3.6-35B-A3B-FP8"
SERVED_NAME="qwen3.6-35b"
PORT=8000

# --- 1. 激活虚拟环境 ---
echo "[1/3] 正在激活 Python 虚拟环境..."
source "$VENV_PATH"

# --- 2. 硬件级 NUMA 优化 ---
# 动态检测 GPU 所在 NUMA 节点，避免硬编码（双路 CPU 架构下降低跨桥延迟）
echo "[2/3] 正在检测显卡 NUMA 拓扑节点..."
NUMA_NODE=$(cat /sys/bus/pci/devices/0000:af:00.0/numa_node 2>/dev/null || echo "0")
if [[ "$NUMA_NODE" == "-1" ]]; then NUMA_NODE="0"; fi
echo "      已锁定至 NUMA Node ${NUMA_NODE}"

# --- 3. 启动 vLLM ---
echo "[3/3] 正在拉起 vLLM 推理引擎..."

VLLM_ARGS=(
    # ── 模型 ─────────────────────────────────────────────────────────────────
    --model                   "$MODEL_NAME"
    --served-model-name       "$SERVED_NAME"
    --port                    "$PORT"

    # ── GPU 资源 ──────────────────────────────────────────────────────────────
    --tensor-parallel-size    1
    --gpu-memory-utilization  0.90

    # ── NUMA 绑定（v0.21.0 内置，无需 numactl）───────────────────────────────
    --numa-bind
    --numa-bind-nodes         "$NUMA_NODE"

    # ── 上下文窗口 ────────────────────────────────────────────────────────────
    # MoE 模型 40 层中 30 层为线性注意力，每位置 KV cache 约 250 KB
    # 48 GB GPU 扣除 ~35 GB 权重后，KV 预算仅 ~9 GB ≈ 39K positions
    --max-model-len           40960
    --max-num-batched-tokens  32768
    --max-num-seqs            32

    # ── 推理解析 ──────────────────────────────────────────────────────────────
    --reasoning-parser        qwen3

    # ── 工具调用 ──────────────────────────────────────────────────────────────
    --enable-auto-tool-choice
    --tool-call-parser        qwen3_coder

    # ── 投机解码（MTP）───────────────────────────────────────────────────────
    # num_speculative_tokens=1 已验证接受率最佳，>1 会降低整体吞吐
    --speculative-config      '{"method":"mtp","num_speculative_tokens":1}'

    # ── FP8 KV Cache（节省约 30% VRAM）───────────────────────────────────────
    --kv-cache-dtype          fp8

    # ── 性能优化 ──────────────────────────────────────────────────────────────
    --performance-mode        throughput
    --enable-chunked-prefill
    --enable-prefix-caching

    # ── 日志精简 ──────────────────────────────────────────────────────────────
    --disable-uvicorn-access-log
    --disable-log-stats

    --trust-remote-code
)

exec python3 -m vllm.entrypoints.openai.api_server "${VLLM_ARGS[@]}"
    AI硬件
  • 交作业 — 淘宝 华南金牌旗舰店 硬件 - RTX 3090 NVLink + 双路 Xeon E5-2686 v4
    Billy ShenB Billy Shen

    这一整套主机花了多少钱?

    AI硬件 nvidia rtx3090
  • 我的 AI 工作站硬件配置报告
    Billy ShenB Billy Shen

    下面的硬件配置, 适合跑哪个模型?

    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    AI 工作站硬件配置报告
    主机名 : ames-P10DRG
    检测时间: 2026-05-28 16:47:46
    操作系统: Ubuntu 24.04.4 LTS
    内核版本: 6.17.0-29-generic
    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    CPU 处理器
    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    total used free shared buff/cache available
    内存： 125Gi 33Gi 58Gi 64Mi 34Gi 91Gi
    交换： 8.0Gi 18Mi 8.0Gi

    ── 内存插槽详情 ──
    Thu May 28 16:47:46 2026
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 |
    +-----------------------------------------+------------------------+----------------------+
    | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
    | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
    | | | MIG M. |
    |=========================================+========================+======================|
    | 0 NVIDIA RTX 5880 Ada Gene... Off | 00000000:AF:00.0 Off | Off |
    | 30% 41C P8 10W / 285W | 43014MiB / 49140MiB | 0% Default |
    | | | N/A |
    +-----------------------------------------+------------------------+----------------------+

    +-----------------------------------------------------------------------------------------+

    Processes:
    GPU GI CI PID Type Process name GPU Memory
    ID ID Usage
    0 N/A N/A 5507 G /usr/lib/xorg/Xorg 4MiB
    0 N/A N/A 12641 C VLLM::EngineCore 42990MiB

    +-----------------------------------------------------------------------------------------+

    ── CUDA 版本 ──
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2023 NVIDIA Corporation
    Built on Fri_Jan__6_16:45:21_PST_2023
    Cuda compilation tools, release 12.0, V12.0.140
    Build cuda_12.0.r12.0/compiler.32267302_0
    ── 磁盘列表 ──
    NAME SIZE ROTA MODEL
    sda 7.3T 1 ST8000NM017B-2TJ103
    nvme0n1 1.8T 0 KINGSTON SNV3S2000G

    AI硬件
  • 关于本地部署，局域网可以怎么共享使用么？
    Billy ShenB Billy Shen

    我是用Tailscale组SDWAN实现的， 可以异地或者手机调取AI服务器的大模型。

    AI硬件
  • 分享一下我的ai主機
    Billy ShenB Billy Shen

    @Sam-Hsu 需要找个图床，将图片的链接嵌入回复里面。

    AI硬件
  • 交作业：X99 CD3+E5 2666 +128G DDR3 + AMD AI Pro 9700，成功安装ubuntu，等不及跟大家分享。后续再更新
    Billy ShenB Billy Shen

    @Brian 说:

    24

    Ubuntu 24.04 LTS， Python 3.12， 要装个完整的 nVidia toolkit

    AI硬件 amd r9700
  • NVIDIA Quadro RTX 5880 Ada Generation 48GB + vLLM + Qwen3.6-35B-A3B-FP8 + tailscale + 多个Hermes Agent客户端
    Billy ShenB Billy Shen

    最近用工作站搭建了vLLM+Hermes agent, 分享下教程

    Linux OS： Ubuntu 24.04 LTS
    · CPU：2 颗 Intel Xeon 8168（每颗 24 核 48 线程，基础频率 2.7GHz）
    · 主板：Intel Xeon 1代/2代 PIODRG 双路主板
    · 内存：三星 DDR4 4*32GB 2933MHz RECC
    · 系统盘：金士顿NVMe SSD，2TB容量
    · 存储盘：希捷企业级 8TB 硬盘，256MB 缓存，7200RPM SATA
    · 显卡：NVIDIA Quadro RTX 5880 Ada Generation，48GB 显存

    我安装的是Ubuntu 24.04 LTS桌面版，系统安装完成之后
    1 安装tailscale 并登录
    2 安装Nvidia-toolkit的最新版

    安装环境Ubuntu 24.04
    安装cuda-toolkit
    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
    sudo dpkg -i cuda-keyring_1.1-1_all.deb
    sudo apt install nvidia-cuda-toolkit

    本地 Python 环境安装
    sudo apt update
    sudo apt install -y python3-venv python3-pip git build-essential

    1. 安装 uv

    curl -LsSf https://astral.sh/uv/install.sh | sh
    source $HOME/.local/bin/env

    2. 创建并激活 Python 3.12 虚拟环境

    uv venv --python 3.12 --seed vllm_env
    source vllm_env/bin/activate

    3. 安装 vLLM

    uv pip install vllm --torch-backend=auto

    installing to /home/ames/.local/bin
    uv
    uvx
    everything's installed!

    To add $HOME/.local/bin to your PATH, either restart your shell or run:

    source $HOME/.local/bin/env (sh, bash, zsh)                                                                         
source $HOME/.local/bin/env.fish (fish)

    uv pip install --upgrade huggingface_hub

    echo 'export HF_ENDPOINT="https://hf-mirror.com"' >> ~/.bashrc
    source ~/.bashrc

    uv pip install hf_transfer

    uvx hf auth login #登录Hugging Face （使用APIkey登录）

    uvx hf download

    #然后就是下载模型了，由于模型比较大，偶尔会掉线，需要重连下载。

    ls -lh ~/.cache/huggingface/hub/models

    /home/ames/.cache/huggingface/hub

    vllm 启动 Qwen/Qwen3.6-35B-A3B-FP8 参数

    我把启动程序，直接做了个shell脚本：

    #!/bin/bash

    =============================================================================

    Qwen3.6-35B-A3B-FP8 vLLM 生产级启动脚本

    针对 Dual Xeon + RTX 5880 Ada + Hermes Agent 深度优化

    =============================================================================

    set -euo pipefail

    --- 配置区 ---

    VENV_PATH="$HOME/vllm_env/bin/activate"
    MODEL_NAME="Qwen/Qwen3.6-35B-A3B-FP8"
    SERVED_NAME="qwen3.6-35b" # 务必在 Hermes config.yaml 中将 model.default 设为此别名
    PORT=8000

    --- 1. 激活虚拟环境 ---

    echo "[1/3] 正在激活 Python 虚拟环境..."
    source "$VENV_PATH"

    --- 2. 硬件级 NUMA 优化 ---

    双路 CPU 架构下，强制将计算与内存绑定在与显卡物理直连的 CPU 节点上，降低跨桥延迟

    echo "[2/3] 正在检测显卡 NUMA 拓扑节点..."
    NUMA_NODE=$(cat /sys/bus/pci/devices/0000:af:00.0/numa_node 2>/dev/null || echo "0")
    if [[ "$NUMA_NODE" == "-1" ]]; then NUMA_NODE="0"; fi
    NUMA_CMD="numactl --cpunodebind=${NUMA_NODE} --membind=${NUMA_NODE}"
    echo " 已锁定至 NUMA Node ${NUMA_NODE}"

    --- 3. 启动 vLLM ---

    echo "[3/3] 正在拉起 vLLM 推理引擎..."
    eval $NUMA_CMD python3 -m vllm.entrypoints.openai.api_server \
    --model "$MODEL_NAME" \
    --served-model-name "$SERVED_NAME" \
    --port "$PORT" \
    --quantization fp8 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 65536 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 128 \
    --trust-remote-code \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

    ~/vllm$ ./start_qwen3.6.sh                                                                                                                                        
[1/3] 正在激活 Python 虚拟环境...                                                                                                                                                  
[2/3] 正在检测显卡 NUMA 拓扑节点...                                                                                                                                                
      已锁定至 NUMA Node 1                                                                                                                                                         
[3/3] 正在拉起 vLLM 推理引擎...                                                                                                                                                    
(APIServer pid=138029) INFO 05-26 09:27:38 [utils.py:306]                                                                                                                          
(APIServer pid=138029) INFO 05-26 09:27:38 [utils.py:306]        █     █     █▄   ▄█                                                                                               
(APIServer pid=138029) INFO 05-26 09:27:38 [utils.py:306]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.21.0                                                                               
(APIServer pid=138029) INFO 05-26 09:27:38 [utils.py:306]   █▄█▀ █     █     █     █  model   Qwen/Qwen3.6-35B-A3B-FP8                                                             
(APIServer pid=138029) INFO 05-26 09:27:38 [utils.py:306]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀                                                                                               
(APIServer pid=138029) INFO 05-26 09:27:38 [utils.py:306]                                                                                                                          
(APIServer pid=138029) INFO 05-26 09:27:38 [utils.py:240] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'hermes', 'model': 'Qwen/Qwen3.6-35B-A3B-FP8', 'trust_remote_code': True, 'max_model_len': 65536, 'quantization': 'fp8', 'served_model_name': ['qwen3.6-35b'], 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': True, 'max_num_batched_tokens': 4096, 'max_num_seqs': 128, 'enable_chunked_prefill': True}                                                                                                      
(APIServer pid=138029) INFO 05-26 09:28:11 [model.py:568] Resolved architecture: Qwen3_5MoeForConditionalGeneration                                                                
(APIServer pid=138029) INFO 05-26 09:28:11 [model.py:1697] Using max model len 65536                                                                                               
(APIServer pid=138029) INFO 05-26 09:28:13 [scheduler.py:239] Chunked prefill is enabled with

    vLLM加载完了模型之后， Tailscale网络里面的客户机，就可以设置URL http:// tailscale IP:8000/v1, 调用服务器的Token了

    LLM讨论区 nvidia vllm hermes
  • 5090 + vLLM + Qwen3.6-27B 成功分享
    Billy ShenB Billy Shen

    我用的是以下配置，跑的是Qwen3.6-35B-A3B-FP8

    Linux OS： Ubuntu 24.04 LTS
    · CPU：2 颗 Intel Xeon 8168（每颗 24 核 48 线程，基础频率 2.7GHz）
    · 主板：Intel Xeon 1代/2代 PIODRG 双路主板
    · 内存：三星 DDR4 4*32GB 2933MHz RECC
    · 系统盘：金士顿NVMe SSD，2TB容量
    · 存储盘：希捷企业级 8TB 硬盘，256MB 缓存，7200RPM SATA
    · 显卡：NVIDIA Quadro RTX 5880 Ada Generation，48GB 显存

    LLM讨论区 rtx5090 vllm
  • 运营油管对IP有要求吗？
    Billy ShenB Billy Shen

    建议买个欧洲的VPS

    自媒体
  • 电商， 硬件需要本地部署大模型加跑马Hermes 加 Comfyui。
    Billy ShenB Billy Shen

    @Chuyao-Chen 澳洲是个好地方

    随便聊聊 comfyui hermes
  • 登录
  • 没有帐号？ 注册
  • 登录或注册以进行搜索。
  • 第一个帖子
    最后一个帖子
0
  • 版块
  • 最新
  • 标签
  • 热门
  • 用户
  • 群组