文章目录
- 1.环境准备
- 1.1 关闭selinux
- 1.2 关闭防火墙
- 1.3 配置免密
- 1.4 设置yum源
- 1.5 安装依赖
- 1.6 设置时间同步
- 1.7 安装docker
- 2.安装Ceph
- 2.1 安装cephadm
- 2.2 部署ceph集群
- 2.3 集群添加节点
- 2.4 部署MON
- 2.5 部署OSD
- 2.6 部署MGR
- 2.7 集群状态
- 3.问题
- 3.1 failed to retrieve runc version: exit
1.环境准备
| 节点 | IP | 数据盘 | 角色 |
|---|---|---|---|
| ceph61 | 192.168.120.61 | ||
| ceph62 | 192.168.120.62 | ||
| ceph63 | 192.168.120.63 |
1.1 关闭selinux
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
1.2 关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
1.3 配置免密
# 在集群所有节点都要执行该命令
ssh-keygen -t rsa
ssh-copy-id ceph61
ssh-copy-id ceph62
ssh-copy-id ceph63
1.4 设置yum源
vim /etc/yum.repo.d/Centos-8.repo
[base]
name=CentOS-$releasever - Base - mirrors.aliyun.com
#failovermethod=priority
baseurl=https://mirrors.aliyun.com/centos/$releasever/BaseOS/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever/BaseOS/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/BaseOS/$basearch/os/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-Official
#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.aliyun.com
#failovermethod=priority
baseurl=https://mirrors.aliyun.com/centos/$releasever/extras/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever/extras/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/extras/$basearch/os/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-Official
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.aliyun.com
#failovermethod=priority
baseurl=https://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever/centosplus/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/centosplus/$basearch/os/
gpgcheck=1
enabled=0
gpgkey=https://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-Official
[PowerTools]
name=CentOS-$releasever - PowerTools - mirrors.aliyun.com
#failovermethod=priority
baseurl=https://mirrors.aliyun.com/centos/$releasever/PowerTools/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever/PowerTools/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/PowerTools/$basearch/os/
gpgcheck=1
enabled=0
gpgkey=https://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-Official
[AppStream]
name=CentOS-$releasever - AppStream - mirrors.aliyun.com
#failovermethod=priority
baseurl=https://mirrors.aliyun.com/centos/$releasever/AppStream/$basearch/os/
http://mirrors.aliyuncs.com/centos/$releasever/AppStream/$basearch/os/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/AppStream/$basearch/os/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-Official
1.5 安装依赖
yum clean all
yum makecache
yum install yum-utils python3 device-mapper-persistent-data lvm2 libseccomp-devel -y
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
1.6 设置时间同步
yum install -y chrony
systemctl start chronyd && systemctl enable chronyd
1.7 安装docker
# 卸载podman
yum -y erase podman buildah
yum install docker-ce docker-ce-cli containerd.io -y
systemctl enable docker && systemctl start docker
2.安装Ceph
2.1 安装cephadm
集群所有节点都要执行
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm
chmod +x cephadm && cp cephadm /usr/bin/cephadm
cephadm add-repo --release quincy
cephadm install
集群所有节点都需要安装，安装完毕之后，才可以使用 ceph orch 查看信息
cephadm install ceph-common
2.2 部署ceph集群
在集群第一个节点执行即可
cephadm bootstrap --mon-ip 192.168.120.61
# 可以访问
Ceph Dashboard is now available at:
URL: https://ceph61:8443/
User: admin
Password: jyfzjm0uj7
查看集群状态
[root@ceph61 ~]# ceph status
cluster:
id: 13be3c46-a8e6-11ed-a1fb-000c29b46455
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum ceph61 (age 23m)
mgr: ceph61.diyqxv(active, since 9m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[root@ceph61 ~]#
[root@ceph61 ~]# ceph -v
ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
2.3 集群添加节点
cd /etc/ceph
ssh-copy-id -f -i ceph.pub root@ceph61
ssh-copy-id -f -i ceph.pub root@ceph62
ssh-copy-id -f -i ceph.pub root@ceph63
禁用自动部署mon节点，否在集群添加节点，会自动部署服务
ceph orch apply mon --unmanaged
ceph orch apply mgr --unmanaged
ceph orch apply osd --all-available-devices --unmanaged
给节点打标签_admin
ceph orch host add ceph62 --labels _admin
ceph orch host add ceph63 --labels _admin
或者
ceph orch host add ceph62
ceph orch host label add ceph62 _admin
ceph orch host add ceph63
ceph orch host label add ceph63 _admin
2.4 部署MON
方法1：一般第一个节点是一个mon节点，该方法会破坏第一个节点的mon
ceph orch apply mon ceph61,ceph62,ceph63
方法2：在第一个节点是mon节点的基础上新增mon节点
ceph orch daemon add mon ceph62:192.168.120.62/24
ceph orch daemon add mon ceph63:192.168.120.63/24
2.5 部署OSD
前提条件：
设备必须没有分区
设备不得具有任何LVM状态
设备不能被mounted
该设备不得包含文件系统
该设备不得包含Ceph BlueStore OSD
设备必须大于5 GB
查看可用设备
ceph orch device ls
部署osd
方法1：在所有可用设备上自动部署osd
ceph orch apply osd --all-available-devices
方法2：指定盘上部署osd
ceph orch daemon add osd ceph61:/dev/sdb,/dev/sdc
ceph orch daemon add osd ceph62:/dev/sdb,/dev/sdc
ceph orch daemon add osd ceph63:/dev/sdb,/dev/sdc
查看osd信息
ceph osd tree
2.6 部署MGR
ceph orch apply mgr --placement=“ceph61,ceph62,ceph63”
2.7 集群状态
3.问题
3.1 failed to retrieve runc version: exit
Docker启动容器，报错，一直不成功，查看docker运行状态，又告警信息
第一步：查看是否存在多个runc，如果存在多个且md5值不一致，使用正常的覆盖不正常的
[root@ceph61 ~]# whereis runc
runc: /usr/bin/runc
正常的runc：
[root@ceph61 ~]# runc --version
runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d
spec: 1.0.2-dev
go: go1.18.10
libseccomp: 2.5.1
异常的runc：
[root@ceph61 ~]# runc --version
runc: symbol lookup error: runc: undefined symbol: seccomp_api_get
第二步：如果只存在一个，并且异常，请安装依赖，再运行，正常下显示
yum install libseccomp-devel -y
