B200 SXM
环境:
- B200 SXM
- 超微服务器
- 有9张IB卡
- ubuntu 24.04 live
安装尝试
cuda 13.1
- https://developer.nvidia.com/cuda-13-1-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=24.04&target_type=deb_local
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-ubuntu2404.pin sudo cp cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600 wget https://developer.download.nvidia.com/compute/cuda/13.0.0/local_installers/cuda-repo-ubuntu2404-13-0-local_13.0.0-580.65.06-1_amd64.deb sudo dpkg -i cuda-repo-ubuntu2404-13-0-local_13.0.0-580.65.06-1_amd64.deb sudo cp /var/cuda-repo-ubuntu2404-13-0-local/cuda-*-keyring.gpg /usr/share/keyrings/ sudo apt-get update sudo apt-get -y install cuda-toolkit-13-0
sudo apt-get install -y nvidia-open
apt install python3.12-venv -y
python3 -m venv torch_env
source torch_env/bin/activate
pip install torch -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
# 安装 cuda13.0 版本的 pytorch
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu130
pytorch 版本
缺少nvcc 环境变量
root@gpu-node:~# find /usr/local /usr -name nvcc 2>/dev/null
/usr/local/cuda-13.0/bin/nvcc
/usr/local/cuda-13.0/bin/nvcc
# 写入bashrc(推荐)
echo 'export CUDA_HOME=/usr/local/cuda-13.0' >> ~/.bashrc
echo 'export PATH=$CUDA_HOME/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
# 立即生效
source ~/.bashrc
# 再给venv环境写一个
cat >> torch_env/bin/activate << 'EOF'
# CUDA 13.0 环境变量
export CUDA_HOME=/usr/local/cuda-13.0
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
EOF
source torch_env/bin/activate
>>> import torch
>>> print(torch.version.cuda)
12.8
>>> print(torch.cuda.is_available())
/root/torch_130_env/lib/python3.12/site-packages/torch/cuda/__init__.py:182: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:119.)
return torch._C._cuda_getDeviceCount() > 0
False
# 一键检测
python -c "import torch;print(torch.cuda.is_available());"
测试失败,reboot 重启尝试,也不行
尝试使用 torch 2.9.0, 2.9.1 , 2.11.0.dev20251215+cu128 、2.9.1+cu130 版本
尝试切换到 cuda 12.8 版本,驱动版本570
尝试使用 torch 2.9.0, 2.9.1 , 2.11.0.dev20251215+cu128 版本,仍然不行
清理cuda 13.0
#!/bin/bash
echo "开始彻底清理CUDA 13.0..."
# APT卸载
sudo apt-get --purge remove cuda-toolkit-13-0 "*cuda*" "*cublas*" "*cudnn*" "*cufft*" "*curand*" "*cusolver*" "*cufft*" -y
sudo apt-get autoremove -y
sudo apt-get autoclean
# 手动删除文件
sudo rm -rf /usr/local/cuda-13.0 /usr/local/cuda
sudo rm -f /etc/ld.so.conf.d/cuda-13-0.conf
sudo rm -f /etc/apt/sources.list.d/cuda-13-0-local.list
sudo rm -f /etc/profile.d/cuda.sh
sudo rm -f /usr/share/keyrings/cuda-archive-keyring.gpg
# 清理环境变量(在.bashrc中)
sed -i '/cuda-13.0/d' ~/.bashrc
source ~/.bashrc
echo "清理完成!请重启系统以彻底清除内核缓存。"
echo "重启命令:sudo reboot"
# 1. 验证APT包是否清空
dpkg -l | grep -i cuda
# 应该**没有任何输出**
# 2. 验证文件系统残留
find /usr/local -name "*cuda*" 2>/dev/null
# 应该**没有结果**
# 3. 验证环境变量
echo $CUDA_HOME
# 应该**为空**
# 4. 验证nvcc命令
which nvcc
# 应该**返回空**(或指向旧版本)
# 5. **重启系统**(必须!清除内核模块缓存)
sudo reboot
网上参考
- https://forums.developer.nvidia.com/t/cuda-initialization-failure-with-error-error-802-system-not-yet-initialized/337819
这篇文章遇到同样的问题,没有明确说明是什么问题,但是涉及到IB的内核模块问题
modprobe ib_umad
启用后测试仍然不行
尝试安装IB驱动
尝试使用cuda13.1
使用cuda13.1 并且安装sudo apt-get install -y cuda-drivers
直接无法找到 nvidia-smi
【教程】保姆级安装NVIDIA CUDA、CUDNN、NCCL环境教程+解决SSH一段时间自动断开报Destination Host Unreachable
重新检查
检查步骤
# 1. 确认系统能识别GPU硬件
lspci | grep -i nvidia
# 2. 检查NVIDIA内核模块状态
lsmod | grep nvidia
# 3. 查看驱动安装情况
dpkg -l | grep nvidia
# 4. 检查内核日志
dmesg | grep -i nvidia
demsg中
NVRM: The NVIDIA GPU ... requires use of the NVIDIA open kernel modules.
NVRM: loading NVIDIA UNIX x86_64 Kernel Module 590.44.01
你的系统处于矛盾状态:安装了nvidia-driver-open包(开源驱动元包),但内核模块签名验证失败,且缺少关键的Fabric Manager服务。
问题根源
B200 GPU要求:
- 必须使用开源内核模块(日志明确提示)
- 必须安装并启动nvidia-fabricmanager服务(多GPU通信必需)
- 模块签名问题导致驱动无法完全初始化
安装nvidia-fabricmanager需要IB驱动
# 安装IB驱动
tar -zxvf MLNX_OFED_LINUX-24.10-3.2.5.0-ubuntu24.04-x86_64.tgz
cd MLNX_OFED_LINUX-24.10-3.2.5.0-ubuntu24.04-x86_64
./mlnxofedinstall
/etc/init.d/openibd restart
# 验证IB
ibstat
# B200多GPU系统必须安装
sudo apt-get install -y nvidia-fabricmanager-590
# 启动并设置开机自启
sudo systemctl enable nvidia-fabricmanager
sudo systemctl start nvidia-fabricmanager
# 查看安装包
apt-cache search nvidia | grep -E '(nvlink|fabric)'
仍然有问题,感觉是包冲突
安装cuda, 特别要注意版本, 动不动就版本冲突
cat /proc/driver/nvidia/version # 内核使用的版本
dkms status | grep nvidia
/usr/local/cuda-13.1/bin/nvcc -V
/usr/bin/nv-fabricmanager -v
FabricManager官方指导
缺少nvlsm模块, apt库中没有相关包
apt-cache search nvidia | grep nvlinksm
# 无模块
最终可用方式
排查过程中试过多套组合,最终可用的安装顺序见下。
cd /root
sudo dpkg -i cuda-repo-ubuntu2404-13-1-local_13.1.0-590.44.01-1_amd64.deb
sudo apt update
sudo apt install -y nvidia-fabricmanager-590
sudo apt install -y infiniband-diags
sudo apt install -y curl gnupg ca-certificates
curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb -o cuda-keyring.deb
sudo dpkg -i cuda-keyring.deb
sudo apt update
sudo apt install -y nvlsm
sudo systemctl enable nvidia-fabricmanager
sudo systemctl start nvidia-fabricmanager
sudo systemctl status nvidia-fabricmanager
python - <<'EOF'
import ctypes
cuda = ctypes.CDLL("libcuda.so")
res = cuda.cuInit(0)
print("cuInit =", res)
EOF
清理脚本
#!/bin/bash
set -e
echo "========================================"
echo " NVIDIA/CUDA 完全通用清理工具"
echo "========================================"
echo "⚠️ 警告:这将自动检测并移除系统上所有 NVIDIA/CUDA 版本"
echo ""
read -p "是否继续? (y/N): " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "操作已取消"
exit 1
fi
# 1. 停止所有 NVIDIA 相关服务
echo "停止 NVIDIA 服务..."
sudo systemctl stop nvidia-persistenced.service 2>/dev/null || true
sudo systemctl stop nvidia-fabricmanager.service 2>/dev/null || true
sudo systemctl stop nvidia-powerd.service 2>/dev/null || true
# 2. 自动检测所有 CUDA 软件包
echo "检测 CUDA 软件包..."
CUDA_PKGS=$(dpkg -l | grep -E '^ii.*cuda' | awk '{print $2}' | tr '\n' ' ')
if [ ! -z "$CUDA_PKGS" ]; then
echo "找到以下 CUDA 软件包:"
echo "$CUDA_PKGS" | tr ' ' '\n' | sed 's/^/- /'
echo "正在卸载..."
sudo apt-get --purge remove -y $CUDA_PKGS
else
echo "未发现 CUDA 软件包"
fi
# 3. 自动检测所有已安装的 NVIDIA 软件包
echo "检测 NVIDIA 软件包..."
NVIDIA_PKGS=$(dpkg -l | grep -E '^ii.*(nvidia-driver|nvidia-kernel|nvidia-fabricmanager|cuda)' | awk '{print $2}' | tr '\n' ' ')
if [ ! -z "$NVIDIA_PKGS" ]; then
echo "找到以下 NVIDIA 软件包:"
echo "$NVIDIA_PKGS" | tr ' ' '\n' | sed 's/^/- /'
echo "正在卸载..."
sudo apt-get --purge remove -y $NVIDIA_PKGS
else
echo "未发现 NVIDIA 软件包"
fi
# 4. 清理其他相关包
echo "检测其他相关包..."
OTHER_PKGS=$(dpkg -l | grep -E '^ii.*(libnvidia|libxnvctrl|libcublas|libcufft|libcurand)' | awk '{print $2}' | tr '\n' ' ')
if [ ! -z "$OTHER_PKGS" ]; then
sudo apt-get --purge remove -y $OTHER_PKGS 2>/dev/null || true
fi
# 5. 清理 DKMS 模块(自动检测所有版本)
echo "清理 DKMS 模块..."
if command -v dkms &> /dev/null; then
NVIDIA_DKMS=$(dkms status | grep -i nvidia | awk -F: '{print $1}' | tr '\n' ' ')
if [ ! -z "$NVIDIA_DKMS" ]; then
for module in $NVIDIA_DKMS; do
echo "移除 DKMS 模块: $module"
sudo dkms remove "$module" --all 2>/dev/null || true
done
fi
fi
# 6. 删除所有 NVIDIA/CUDA 仓库配置
echo "清理仓库配置..."
sudo rm -f /etc/apt/preferences.d/cuda-repository-pin-600
sudo rm -f /etc/apt/sources.list.d/cuda*
sudo rm -f /etc/apt/sources.list.d/nvidia*
sudo rm -f /usr/share/keyrings/cuda-archive-keyring.gpg
sudo rm -f /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
# 7. 删除残留文件和目录
echo "清理残留文件..."
sudo rm -rf /usr/local/cuda*
sudo rm -rf /opt/cuda*
sudo rm -rf /usr/lib/cuda*
sudo rm -rf /usr/include/cuda*
sudo rm -rf /usr/share/cuda*
sudo rm -rf /etc/cuda*
sudo rm -rf /var/lib/dkms/nvidia*
sudo rm -rf /usr/src/nvidia-*
sudo rm -rf /usr/lib/nvidia-*
## 清理内核文件
sudo find /lib/modules/ -path "*/updates/dkms/nvidia*" -delete
## stubs 目录
rm -rf /usr/lib/x86_64-linux-gnu/stubs
## 重建 stubs 目录,避免下次安装报错
mkdir -p /usr/lib/x86_64-linux-gnu/stubs
# 8. 清理 modprobe 配置
echo "清理 modprobe 配置..."
sudo rm -f /etc/modprobe.d/nvidia*.conf
sudo rm -f /lib/modprobe.d/nvidia*.conf
# 9. 清理 udev 规则
echo "清理 udev 规则..."
sudo rm -f /etc/udev/rules.d/99-nvidia*.rules
# 10. 清理 X11 配置
echo "清理 X11 配置..."
sudo rm -f /etc/X11/xorg.conf.d/10-nvidia.conf
sudo rm -f /etc/X11/xorg.conf
# 11. 清理环境变量
echo "清理环境变量..."
for rcfile in ~/.bashrc ~/.zshrc ~/.profile; do
if [ -f "$rcfile" ]; then
cp "$rcfile" "$rcfile.backup.$(date +%Y%m%d%H%M%S)" 2>/dev/null || true
sed -i '/CUDA_HOME/d' "$rcfile" 2>/dev/null || true
sed -i '/export PATH=.*cuda/d' "$rcfile" 2>/dev/null || true
sed -i '/export LD_LIBRARY_PATH=.*cuda/d' "$rcfile" 2>/dev/null || true
sed -i '/Nsight/d' "$rcfile" 2>/dev/null || true
fi
done
# 12. 更新系统
echo "更新系统..."
sudo apt-get autoremove -y
sudo apt-get autoclean -y
# sudo apt-get update
sudo update-initramfs -u
# 13. 最终验证
echo "========================================"
echo "验证清理结果:"
echo "--- 剩余 NVIDIA 软件包 ---"
dpkg -l | grep -E '^ii.*nvidia' || echo "✅ 无 NVIDIA 软件包"
echo "--- 剩余 CUDA 软件包 ---"
dpkg -l | grep -E '^ii.*cuda' || echo "✅ 无 CUDA 软件包"
echo "--- 剩余相关库 ---"
dpkg -l | grep -E '^ii.*(libnvidia|libxnvctrl)' || echo "✅ 无相关库"
echo "--- DKMS 模块状态 ---"
dkms status | grep -i nvidia || echo "✅ 无 NVIDIA DKMS 模块"
echo "========================================"
echo "✅ 通用清理完成!"
echo "系统已完全准备好安装任何版本 NVIDIA/CUDA"
echo "建议重启: sudo reboot"
echo "========================================"
部署流程
系统为Ubuntu 24.04 TLS
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-ubuntu2404.pin
sudo cp cuda-ubuntu2404.pin /etc/apt/preferences.d/cuda-repository-pin-600
# 如果没有deb, 下载个
wget https://developer.download.nvidia.com/compute/cuda/13.1.0/local_installers/cuda-repo-ubuntu2404-13-1-local_13.1.0-590.44.01-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2404-13-1-local_13.1.0-590.44.01-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2404-13-1-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
apt install -y nvidia-fabricmanager=590.44.01-0ubuntu1
apt install -y infiniband-diags
apt install -y nvlsm=2025.06.10-1
apt-get install -y nvidia-open
sudo apt-get -y install cuda-toolkit-13-1
# 确认内核版本是否统一
cat /proc/driver/nvidia/version # 内核使用的版本
dkms status | grep nvidia # 可以使用的版本,如果和上面的不一致,说明需要reboot重启下
/usr/local/cuda-13.1/bin/nvcc -V # 指的是 cuda-toolkit 版本
/usr/bin/nv-fabricmanager -v
# 启动 nvidia-fabricmanager
modprobe ib_umad
echo "ib_umad" > /etc/modules-load.d/ib.conf
sudo systemctl enable --now nvidia-fabricmanager
sudo systemctl enable --now nvidia-persistenced
# 如果之前是清理过驱动, 此处强烈建议 reboot
# 必须运行 NVSwich互联
sudo systemctl status nvidia-fabricmanager
nvidia-smi
# 如果没有这个 nvidia-smi 启动非常慢
sudo systemctl status nvidia-persistenced
# 验证, 返回 cuInit = 0 为正确
python3 -c "import ctypes; print(ctypes.CDLL('libcuda.so').cuInit(0))"
安装 nvidia-container-runtime
which nvidia-container-runtime
# 1. 下载到当前目录
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
-o nvidia.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
-o nvidia.list
# 2. 本地处理
gpg --dearmor -o nvidia-container-toolkit.gpg nvidia.gpg
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit.gpg] https://#g' \
nvidia.list > nvidia-container-toolkit.list
# 3. 查看/校验(可选,确认无误后再执行下一步)
cat nvidia-container-toolkit.list
# 4. 安装(仅此处用 sudo)
sudo cp nvidia-container-toolkit.gpg /usr/share/keyrings/
sudo cp nvidia-container-toolkit.list /etc/apt/sources.list.d/
# 替换国内源
sed -i 's#nvidia.github.io#mirrors.ustc.edu.cn#g' /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
# 重新生成 CDI 配置,让 toolkit 识别新驱动 590.44.01
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
sudo nvidia-ctk runtime configure --set-as-default --runtime=containerd # or docker
sudo systemctl restart containerd # or docker
# 检查是否写入
containerd config dump 2>/dev/null | grep -A10 nvidia
验证留存
root@gpu-node-02:~# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 590.44.01 Release Build (dvs-builder@U22-I3-A14-02-4) Fri Nov 21 13:14:57 UTC 2025
GCC version: gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04)
root@gpu-node-02:~#
root@gpu-node-02:~#
root@gpu-node-02:~# dkms status | grep nvidia
nvidia/590.44.01, 6.8.0-90-generic, x86_64: installed
root@gpu-node-02:~#
root@gpu-node-02:~#
root@gpu-node-02:~# /usr/local/cuda-13.1/bin/nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Nov__7_07:23:37_PM_PST_2025
Cuda compilation tools, release 13.1, V13.1.80
Build cuda_13.1.r13.1/compiler.36836380_0
root@gpu-node-02:~#
root@gpu-node-02:~#
root@gpu-node-02:~# /usr/bin/nv-fabricmanager -v
Fabric Manager version is : 590.44.01
Build Info - Branch: rel/gpu_drv/r590/r591_37-155, Changelist: 0
root@gpu-node-02:~# systemctl status nvidia-persistenced
● nvidia-persistenced.service - NVIDIA Persistence Daemon
Loaded: loaded (/usr/lib/systemd/system/nvidia-persistenced.service; enabled; preset: enabled)
Active: active (running) since Thu 2026-03-05 11:24:03 CST; 1 day 2h ago
Main PID: 4721 (nvidia-persiste)
Tasks: 1 (limit: 629145)
Memory: 1.1M (peak: 1.9M)
CPU: 356ms
CGroup: /system.slice/nvidia-persistenced.service
└─4721 /usr/bin/nvidia-persistenced --user nvidia-persistenced
# nvidia-container-runtime 的验证
# 检查 CDI 配置是否已生成
ls -la /etc/cdi/nvidia.yaml 2>/dev/null || true
cat /etc/cdi/nvidia.yaml 2>/dev/null | head -20
# 验证生成的配置里指向的驱动版本
grep -E "driver|version|nvidia" /etc/cdi/nvidia.yaml | head -20
# 确认配置被正确写入
grep -A10 "nvidia" /etc/containerd/config.toml
总结
- 由于这次的B200 是SXM的制式, 采用第四代NVSwitch,
- 加入了NVLSM
- 所以需要安装 FabricManager
- FabricManager 依赖 nvlsm, 所以也需要同步安装
- 需要保证 nvidia 驱动版本, cuda 版本完全一致才行, 否则直接重装
- nvidia-smi 是安装cuda驱动才可以使用的
- 要使用开源的nvidia驱动(nvidia-open), 否则会直接找不到device(在dmesg中会明确写出来)
- 清理cuda 和nvidia驱动, 使用的是4层清理法
- 当出现问题, 从nvidia 驱动版本, cuda 驱动版本, nvidia-fabricmanager 运行状态, dmesg 日志等地方来排查
cuda toolkit下载网站
遇到的问题
20260707-重新安装B200的驱动时出问题
安装 nvidia-open 的时候,无法正常安装
Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia.ko.zst already installed at version 590.44.01, override by specifying --force
Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia-modeset.ko.zst already installed at version 590.44.01, override by specifying --force
Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia-drm.ko.zst already installed at version 590.44.01, override by specifying --force
Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia-uvm.ko.zst already installed at version 590.44.01, override by specifying --force
Module /lib/modules/6.8.0-101-generic/updates/dkms/nvidia-peermem.ko.zst already installed at version 590.44.01, override by specifying --force
DKMS 编译成功,但安装模块时检测到旧版本的.ko.zst** 文件已存在,拒绝覆盖**,导致 nvidia-dkms-open 的 post-install 脚本失败
解决方案
sudo find /lib/modules/ -path "*/updates/dkms/nvidia*" -delete && \
sudo dkms install nvidia/590.44.01 --force && \
sudo dpkg --configure -a && \
sudo apt-get install -f
转载请注明来源,欢迎对文章中的引用来源进行考证,欢迎指出任何有错误或不够清晰的表达。可以在下面评论区评论,也可以邮件至 365433079@qq.com