解决多卡机器安装完CUDA后，出现802错误码：Fabric Manager需要和Driver具有完全一致的版本号。

现象

检查

查看service状态：

显示failed，查看nvidia-smi中的Driver版本：

切换版本

sudo yum list installed | grep nvidia
sudo yum remove nvidia-fabric-manager.x86_64
# 注意版本后的-1
sudo yum install -y nvidia-fabric-manager-515.65.01-1
# 启动服务
sudo systemctl disable nvidia-fabricmanager
sudo systemctl enable nvidia-fabricmanager
sudo systemctl start nvidia-fabricmanager
sudo systemctl status nvidia-fabricmanager

确认状态：

cuda bin可以正常工作：

参考链接

NVIDIA trouble shooting docs

On systems with NVSwitch, if you notice the CUDA_ERROR_SYSTEM_NOT_READY error being reported, then make sure that you install the same version of Fabric Manager as the CUDA driver.