B200 SXM 服务器 CUDA 驱动安装实录

  1. 安装尝试
    1. 清理cuda 13.0
    2. 网上参考
    3. 尝试使用cuda13.1
    4. 重新检查
  2. 最终可用方式
    1. 清理脚本
    2. 部署流程
    3. 验证留存
  • 总结
  • 遇到的问题
    1. 20260707-重新安装B200的驱动时出问题
  • B200 SXM

    环境:

    • B200 SXM
    • 超微服务器
    • 有9张IB卡
    • ubuntu 24.04 live

    安装尝试

    cuda 13.1

    sudo apt-get install -y nvidia-open
    
    apt install python3.12-venv -y
    
    python3 -m venv torch_env source torch_env/bin/activate
    pip install torch -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
    # 安装 cuda13.0 版本的 pytorch pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu130

    pytorch 版本

    缺少nvcc 环境变量

    root@gpu-node:~# find /usr/local /usr -name nvcc 2>/dev/null
    /usr/local/cuda-13.0/bin/nvcc
    /usr/local/cuda-13.0/bin/nvcc
    
    # 写入bashrc(推荐) echo 'export CUDA_HOME=/usr/local/cuda-13.0' >> ~/.bashrc echo 'export PATH=$CUDA_HOME/bin:$PATH' >> ~/.bashrc echo 'export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
    # 立即生效 source ~/.bashrc
    # 再给venv环境写一个 cat >> torch_env/bin/activate << 'EOF'
    # CUDA 13.0 环境变量 export CUDA_HOME=/usr/local/cuda-13.0 export PATH=$CUDA_HOME/bin:$PATH export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH EOF source torch_env/bin/activate
    >>> import torch
    >>> print(torch.version.cuda)
    12.8
    >>> print(torch.cuda.is_available())
    /root/torch_130_env/lib/python3.12/site-packages/torch/cuda/__init__.py:182: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:119.)
      return torch._C._cuda_getDeviceCount() > 0
    False
    
    # 一键检测 python -c "import torch;print(torch.cuda.is_available());"

    测试失败,reboot 重启尝试,也不行

    尝试使用 torch 2.9.0, 2.9.1 , 2.11.0.dev20251215+cu128 、2.9.1+cu130 版本

    尝试切换到 cuda 12.8 版本,驱动版本570

    尝试使用 torch 2.9.0, 2.9.1 , 2.11.0.dev20251215+cu128 版本,仍然不行

    清理cuda 13.0

    #!/bin/bash
    echo "开始彻底清理CUDA 13.0..."
    
    # APT卸载 sudo apt-get --purge remove cuda-toolkit-13-0 "*cuda*" "*cublas*" "*cudnn*" "*cufft*" "*curand*" "*cusolver*" "*cufft*" -y sudo apt-get autoremove -y sudo apt-get autoclean
    # 手动删除文件 sudo rm -rf /usr/local/cuda-13.0 /usr/local/cuda sudo rm -f /etc/ld.so.conf.d/cuda-13-0.conf sudo rm -f /etc/apt/sources.list.d/cuda-13-0-local.list sudo rm -f /etc/profile.d/cuda.sh sudo rm -f /usr/share/keyrings/cuda-archive-keyring.gpg
    # 清理环境变量(在.bashrc中) sed -i '/cuda-13.0/d' ~/.bashrc source ~/.bashrc
    echo "清理完成!请重启系统以彻底清除内核缓存。" echo "重启命令:sudo reboot"
    # 1. 验证APT包是否清空
    dpkg -l | grep -i cuda
    # 应该**没有任何输出**
    
    # 2. 验证文件系统残留 find /usr/local -name "*cuda*" 2>/dev/null # 应该**没有结果**
    # 3. 验证环境变量 echo $CUDA_HOME # 应该**为空**
    # 4. 验证nvcc命令 which nvcc # 应该**返回空**(或指向旧版本)
    # 5. **重启系统**(必须!清除内核模块缓存) sudo reboot

    网上参考

    modprobe ib_umad
    

    启用后测试仍然不行

    尝试安装IB驱动

    尝试使用cuda13.1

    使用cuda13.1 并且安装sudo apt-get install -y cuda-drivers

    直接无法找到 nvidia-smi

    【教程】保姆级安装NVIDIA CUDA、CUDNN、NCCL环境教程+解决SSH一段时间自动断开报Destination Host Unreachable

    重新检查

    检查步骤

    # 1. 确认系统能识别GPU硬件
    lspci | grep -i nvidia
    
    # 2. 检查NVIDIA内核模块状态 lsmod | grep nvidia
    # 3. 查看驱动安装情况 dpkg -l | grep nvidia
    # 4. 检查内核日志 dmesg | grep -i nvidia

    demsg中

    NVRM: The NVIDIA GPU ... requires use of the NVIDIA open kernel modules.
    NVRM: loading NVIDIA UNIX x86_64 Kernel Module  590.44.01
    

    你的系统处于矛盾状态:安装了nvidia-driver-open包(开源驱动元包),但内核模块签名验证失败,且缺少关键的Fabric Manager服务。

    问题根源

    B200 GPU要求:

    1. 必须使用开源内核模块(日志明确提示)
    2. 必须安装并启动nvidia-fabricmanager服务(多GPU通信必需)
    3. 模块签名问题导致驱动无法完全初始化

    安装nvidia-fabricmanager需要IB驱动

    # 安装IB驱动
    tar -zxvf MLNX_OFED_LINUX-24.10-3.2.5.0-ubuntu24.04-x86_64.tgz
    cd MLNX_OFED_LINUX-24.10-3.2.5.0-ubuntu24.04-x86_64
    ./mlnxofedinstall
    /etc/init.d/openibd restart
    
    # 验证IB ibstat
    # B200多GPU系统必须安装 sudo apt-get install -y nvidia-fabricmanager-590
    # 启动并设置开机自启 sudo systemctl enable nvidia-fabricmanager sudo systemctl start nvidia-fabricmanager
    # 查看安装包 apt-cache search nvidia | grep -E '(nvlink|fabric)'

    仍然有问题,感觉是包冲突

    安装cuda, 特别要注意版本, 动不动就版本冲突


    cat /proc/driver/nvidia/version # 内核使用的版本 dkms status | grep nvidia /usr/local/cuda-13.1/bin/nvcc -V /usr/bin/nv-fabricmanager -v

    FabricManager官方指导

    缺少nvlsm模块, apt库中没有相关包

    apt-cache search nvidia | grep nvlinksm
    # 无模块
    

    最终可用方式

    排查过程中试过多套组合,最终可用的安装顺序见下。

    cd /root
    sudo dpkg -i cuda-repo-ubuntu2404-13-1-local_13.1.0-590.44.01-1_amd64.deb
    sudo apt update
    sudo apt install -y nvidia-fabricmanager-590
    sudo apt install -y infiniband-diags 
    
    sudo apt install -y curl gnupg ca-certificates curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb -o cuda-keyring.deb sudo dpkg -i cuda-keyring.deb sudo apt update sudo apt install -y nvlsm
    sudo systemctl enable nvidia-fabricmanager sudo systemctl start nvidia-fabricmanager sudo systemctl status nvidia-fabricmanager
    python - <<'EOF' import ctypes cuda = ctypes.CDLL("libcuda.so") res = cuda.cuInit(0) print("cuInit =", res) EOF

    清理脚本

    #!/bin/bash
    set -e
    
    echo "========================================" echo " NVIDIA/CUDA 完全通用清理工具" echo "========================================" echo "⚠️ 警告:这将自动检测并移除系统上所有 NVIDIA/CUDA 版本" echo ""
    read -p "是否继续? (y/N): " -n 1 -r echo if [[ ! $REPLY =~ ^[Yy]$ ]]; then echo "操作已取消" exit 1 fi
    # 1. 停止所有 NVIDIA 相关服务 echo "停止 NVIDIA 服务..." sudo systemctl stop nvidia-persistenced.service 2>/dev/null || true sudo systemctl stop nvidia-fabricmanager.service 2>/dev/null || true sudo systemctl stop nvidia-powerd.service 2>/dev/null || true
    # 2. 自动检测所有 CUDA 软件包 echo "检测 CUDA 软件包..." CUDA_PKGS=$(dpkg -l | grep -E '^ii.*cuda' | awk '{print $2}' | tr '\n' ' ') if [ ! -z "$CUDA_PKGS" ]; then echo "找到以下 CUDA 软件包:" echo "$CUDA_PKGS" | tr ' ' '\n' | sed 's/^/- /' echo "正在卸载..." sudo apt-get --purge remove -y $CUDA_PKGS else echo "未发现 CUDA 软件包" fi
    # 3. 自动检测所有已安装的 NVIDIA 软件包 echo "检测 NVIDIA 软件包..." NVIDIA_PKGS=$(dpkg -l | grep -E '^ii.*(nvidia-driver|nvidia-kernel|nvidia-fabricmanager|cuda)' | awk '{print $2}' | tr '\n' ' ') if [ ! -z "$NVIDIA_PKGS" ]; then echo "找到以下 NVIDIA 软件包:" echo "$NVIDIA_PKGS" | tr ' ' '\n' | sed 's/^/- /' echo "正在卸载..." sudo apt-get --purge remove -y $NVIDIA_PKGS else echo "未发现 NVIDIA 软件包" fi
    # 4. 清理其他相关包 echo "检测其他相关包..." OTHER_PKGS=$(dpkg -l | grep -E '^ii.*(libnvidia|libxnvctrl|libcublas|libcufft|libcurand)' | awk '{print $2}' | tr '\n' ' ') if [ ! -z "$OTHER_PKGS" ]; then sudo apt-get --purge remove -y $OTHER_PKGS 2>/dev/null || true fi
    # 5. 清理 DKMS 模块(自动检测所有版本) echo "清理 DKMS 模块..." if command -v dkms &> /dev/null; then NVIDIA_DKMS=$(dkms status | grep -i nvidia | awk -F: '{print $1}' | tr '\n' ' ') if [ ! -z "$NVIDIA_DKMS" ]; then for module in $NVIDIA_DKMS; do echo "移除 DKMS 模块: $module" sudo dkms remove "$module" --all 2>/dev/null || true done fi fi
    # 6. 删除所有 NVIDIA/CUDA 仓库配置 echo "清理仓库配置..." sudo rm -f /etc/apt/preferences.d/cuda-repository-pin-600 sudo rm -f /etc/apt/sources.list.d/cuda* sudo rm -f /etc/apt/sources.list.d/nvidia* sudo rm -f /usr/share/keyrings/cuda-archive-keyring.gpg sudo rm -f /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    # 7. 删除残留文件和目录 echo "清理残留文件..." sudo rm -rf /usr/local/cuda* sudo rm -rf /opt/cuda* sudo rm -rf /usr/lib/cuda* sudo rm -rf /usr/include/cuda* sudo rm -rf /usr/share/cuda* sudo rm -rf /etc/cuda* sudo rm -rf /var/lib/dkms/nvidia* sudo rm -rf /usr/src/nvidia-* sudo rm -rf /usr/lib/nvidia-*
    ## 清理内核文件 sudo find /lib/modules/ -path "*/updates/dkms/nvidia*" -delete
    ## stubs 目录 rm -rf /usr/lib/x86_64-linux-gnu/stubs ## 重建 stubs 目录,避免下次安装报错 mkdir -p /usr/lib/x86_64-linux-gnu/stubs
    # 8. 清理 modprobe 配置 echo "清理 modprobe 配置..." sudo rm -f /etc/modprobe.d/nvidia*.conf sudo rm -f /lib/modprobe.d/nvidia*.conf
    # 9. 清理 udev 规则 echo "清理 udev 规则..." sudo rm -f /etc/udev/rules.d/99-nvidia*.rules
    # 10. 清理 X11 配置 echo "清理 X11 配置..." sudo rm -f /etc/X11/xorg.conf.d/10-nvidia.conf sudo rm -f /etc/X11/xorg.conf
    # 11. 清理环境变量 echo "清理环境变量..." for rcfile in ~/.bashrc ~/.zshrc ~/.profile; do if [ -f "$rcfile" ]; then cp "$rcfile" "$rcfile.backup.$(date +%Y%m%d%H%M%S)" 2>/dev/null || true sed -i '/CUDA_HOME/d' "$rcfile" 2>/dev/null || true sed -i '/export PATH=.*cuda/d' "$rcfile" 2>/dev/null || true sed -i '/export LD_LIBRARY_PATH=.*cuda/d' "$rcfile" 2>/dev/null || true sed -i '/Nsight/d' "$rcfile" 2>/dev/null || true fi done
    # 12. 更新系统 echo "更新系统..." sudo apt-get autoremove -y sudo apt-get autoclean -y # sudo apt-get update sudo update-initramfs -u
    # 13. 最终验证 echo "========================================" echo "验证清理结果:" echo "--- 剩余 NVIDIA 软件包 ---" dpkg -l | grep -E '^ii.*nvidia' || echo "✅ 无 NVIDIA 软件包"
    echo "--- 剩余 CUDA 软件包 ---" dpkg -l | grep -E '^ii.*cuda' || echo "✅ 无 CUDA 软件包"
    echo "--- 剩余相关库 ---" dpkg -l | grep -E '^ii.*(libnvidia|libxnvctrl)' || echo "✅ 无相关库"
    echo "--- DKMS 模块状态 ---" dkms status | grep -i nvidia || echo "✅ 无 NVIDIA DKMS 模块"
    echo "========================================" echo "✅ 通用清理完成!" echo "系统已完全准备好安装任何版本 NVIDIA/CUDA" echo "建议重启: sudo reboot" echo "========================================"

    部署流程

    系统为Ubuntu 24.04 TLS

    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-ubuntu2404.pin
    sudo cp cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600
    
    # 如果没有deb, 下载个 wget https://developer.download.nvidia.com/compute/cuda/13.1.0/local_installers/cuda-repo-ubuntu2404-13-1-local_13.1.0-590.44.01-1_amd64.deb
    sudo dpkg -i cuda-repo-ubuntu2404-13-1-local_13.1.0-590.44.01-1_amd64.deb sudo cp /var/cuda-repo-ubuntu2404-13-1-local/cuda-*-keyring.gpg /usr/share/keyrings/ sudo apt-get update
    apt install -y nvidia-fabricmanager=590.44.01-0ubuntu1 apt install -y infiniband-diags apt install -y nvlsm=2025.06.10-1 apt-get install -y nvidia-open sudo apt-get -y install cuda-toolkit-13-1
    # 确认内核版本是否统一 cat /proc/driver/nvidia/version # 内核使用的版本 dkms status | grep nvidia # 可以使用的版本,如果和上面的不一致,说明需要reboot重启下 /usr/local/cuda-13.1/bin/nvcc -V # 指的是 cuda-toolkit 版本 /usr/bin/nv-fabricmanager -v
    # 启动 nvidia-fabricmanager modprobe ib_umad echo "ib_umad" > /etc/modules-load.d/ib.conf sudo systemctl enable --now nvidia-fabricmanager sudo systemctl enable --now nvidia-persistenced
    # 如果之前是清理过驱动, 此处强烈建议 reboot
    # 必须运行 NVSwich互联 sudo systemctl status nvidia-fabricmanager nvidia-smi
    # 如果没有这个 nvidia-smi 启动非常慢 sudo systemctl status nvidia-persistenced
    # 验证, 返回 cuInit = 0 为正确 python3 -c "import ctypes; print(ctypes.CDLL('libcuda.so').cuInit(0))"

    安装 nvidia-container-runtime

    which nvidia-container-runtime
    
    # 1. 下载到当前目录 curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ -o nvidia.gpg
    curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ -o nvidia.list
    # 2. 本地处理 gpg --dearmor -o nvidia-container-toolkit.gpg nvidia.gpg
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#g' \ nvidia.list > nvidia-container-toolkit.list
    # 3. 查看/校验(可选,确认无误后再执行下一步) cat nvidia-container-toolkit.list
    # 4. 安装(仅此处用 sudo) sudo cp nvidia-container-toolkit.gpg /usr/share/keyrings/ sudo cp nvidia-container-toolkit.list /etc/apt/sources.list.d/
    # 替换国内源 sed -i 's#nvidia.github.io#mirrors.ustc.edu.cn#g' /etc/apt/sources.list.d/nvidia-container-toolkit.list
    sudo apt update sudo apt install -y nvidia-container-toolkit
    # 重新生成 CDI 配置,让 toolkit 识别新驱动 590.44.01 sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml sudo nvidia-ctk runtime configure --set-as-default --runtime=containerd # or docker sudo systemctl restart containerd # or docker
    # 检查是否写入 containerd config dump 2>/dev/null | grep -A10 nvidia

    验证留存

    root@gpu-node-02:~# cat /proc/driver/nvidia/version
    NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  590.44.01  Release Build  (dvs-builder@U22-I3-A14-02-4)  Fri Nov 21 13:14:57 UTC 2025
    GCC version:  gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04) 
    root@gpu-node-02:~#  
    root@gpu-node-02:~# 
    root@gpu-node-02:~# dkms status | grep nvidia 
    nvidia/590.44.01, 6.8.0-90-generic, x86_64: installed
    root@gpu-node-02:~# 
    root@gpu-node-02:~# 
    root@gpu-node-02:~# /usr/local/cuda-13.1/bin/nvcc -V
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2025 NVIDIA Corporation
    Built on Fri_Nov__7_07:23:37_PM_PST_2025
    Cuda compilation tools, release 13.1, V13.1.80
    Build cuda_13.1.r13.1/compiler.36836380_0
    root@gpu-node-02:~# 
    root@gpu-node-02:~# 
    root@gpu-node-02:~# /usr/bin/nv-fabricmanager -v
    Fabric Manager version is : 590.44.01
    Build Info - Branch: rel/gpu_drv/r590/r591_37-155, Changelist: 0
    
    root@gpu-node-02:~# systemctl status nvidia-persistenced ● nvidia-persistenced.service - NVIDIA Persistence Daemon Loaded: loaded (/usr/lib/systemd/system/nvidia-persistenced.service; enabled; preset: enabled) Active: active (running) since Thu 2026-03-05 11:24:03 CST; 1 day 2h ago Main PID: 4721 (nvidia-persiste) Tasks: 1 (limit: 629145) Memory: 1.1M (peak: 1.9M) CPU: 356ms CGroup: /system.slice/nvidia-persistenced.service └─4721 /usr/bin/nvidia-persistenced --user nvidia-persistenced
    # nvidia-container-runtime 的验证 # 检查 CDI 配置是否已生成 ls -la /etc/cdi/nvidia.yaml 2>/dev/null || true cat /etc/cdi/nvidia.yaml 2>/dev/null | head -20
    # 验证生成的配置里指向的驱动版本 grep -E "driver|version|nvidia" /etc/cdi/nvidia.yaml | head -20
    # 确认配置被正确写入 grep -A10 "nvidia" /etc/containerd/config.toml

    总结

    1. 由于这次的B200 是SXM的制式, 采用第四代NVSwitch,
      1. 加入了NVLSM
      2. 所以需要安装 FabricManager
    2. FabricManager 依赖 nvlsm, 所以也需要同步安装
    3. 需要保证 nvidia 驱动版本, cuda 版本完全一致才行, 否则直接重装
    4. nvidia-smi 是安装cuda驱动才可以使用的
    5. 要使用开源的nvidia驱动(nvidia-open), 否则会直接找不到device(在dmesg中会明确写出来)
    6. 清理cuda 和nvidia驱动, 使用的是4层清理法
    7. 当出现问题, 从nvidia 驱动版本, cuda 驱动版本, nvidia-fabricmanager 运行状态, dmesg 日志等地方来排查

    cuda toolkit下载网站

    遇到的问题

    20260707-重新安装B200的驱动时出问题

    安装 nvidia-open 的时候,无法正常安装

    Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia.ko.zst already installed at version 590.44.01, override by specifying --force
    Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia-modeset.ko.zst already installed at version 590.44.01, override by specifying --force
    Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia-drm.ko.zst already installed at version 590.44.01, override by specifying --force
    Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia-uvm.ko.zst already installed at version 590.44.01, override by specifying --force
    Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia-peermem.ko.zst already installed at version 590.44.01, override by specifying --force
    

    DKMS 编译成功,但安装模块时检测到旧版本的.ko.zst** 文件已存在,拒绝覆盖**,导致 nvidia-dkms-open 的 post-install 脚本失败

    解决方案

    sudo find /lib/modules/ -path "*/updates/dkms/nvidia*" -delete && \
    sudo dkms install nvidia/590.44.01 --force && \
    sudo dpkg --configure -a && \
    sudo apt-get install -f
    

    转载请注明来源,欢迎对文章中的引用来源进行考证,欢迎指出任何有错误或不够清晰的表达。可以在下面评论区评论,也可以邮件至 365433079@qq.com