接上篇
openEuler 22.03 LTS x86_64 cephadm 部署ceph18.2.0 未完成 笔记-CSDN博客
故障 /usr/libexec/podman/catatonit: no such file or directory
[root@node-1 ~]# cephadm bootstrap --mon-ip 10.47.76.94 --skip-pull
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 3.4.4 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 18788cd6-8825-11ee-9f59-08002726ce7d
Verifying IP 10.47.76.94 port 3300 ...
Verifying IP 10.47.76.94 port 6789 ...
Mon IP `10.47.76.94` is in CIDR network `10.47.76.0/24`
- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16 -e NODE_NAME=node-1 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16 --version
ceph: stderr Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory
Traceback (most recent call last):
File "/usr/sbin/cephadm", line 8571, in <module>
main()
File "/usr/sbin/cephadm", line 8559, in main
r = ctx.func(ctx)
File "/usr/sbin/cephadm", line 1787, in _default_image
return func(ctx)
File "/usr/sbin/cephadm", line 4257, in command_bootstrap
image_ver = CephContainer(ctx, ctx.image, 'ceph', ['--version']).run().strip()
File "/usr/sbin/cephadm", line 3420, in run
out, _, _ = call_throws(self.ctx, self.run_cmd(),
File "/usr/sbin/cephadm", line 1467, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v16 -e NODE_NAME=node-1 -e CEPH_USE_RANDOM_NONCE=1 quay.io/ceph/ceph:v16 --version
参考方案
container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory · Issue #54 · mgoltzsche/podman-static · GitHub
解决
wget https://github.com/openSUSE/catatonit/releases/download/v0.2.0/catatonit.x86_64
cp catatonit.x86_64 /usr/libexec/podman/catatonit
chmod +x /usr/libexec/podman/catatonitbootstrap
cephadm bootstrap --mon-ip 10.47.76.94 --skip-pull[root@node-1 ~]# cephadm bootstrap --mon-ip 10.47.76.94 --skip-pull
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 3.4.4 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 4466c6c0-8828-11ee-b74e-08002726ce7d
Verifying IP 10.47.76.94 port 3300 ...
Verifying IP 10.47.76.94 port 6789 ...
Mon IP `10.47.76.94` is in CIDR network `10.47.76.0/24`
- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Ceph version: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
firewalld ready
Enabling firewalld service ceph-mon in current zone...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network to 10.47.76.0/24
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 9283 ...
firewalld ready
Enabling firewalld service ceph in current zone...
firewalld ready
Enabling firewalld port 9283/tcp in current zone...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host node-1...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 9...
mgr epoch 9 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
firewalld ready
Enabling firewalld port 8443/tcp in current zone...
Ceph Dashboard is now available at:
URL: https://node-1:8443/
User: admin
Password: uqmwzrvev2
Enabling client.admin keyring and conf on hosts with "admin" label
You can access the Ceph CLI with:
sudo /usr/sbin/cephadm shell --fsid 4466c6c0-8828-11ee-b74e-08002726ce7d -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
Please consider enabling telemetry to help improve Ceph:
ceph telemetry on
For more information see:
https://docs.ceph.com/docs/pacific/mgr/telemetry/
Bootstrap complete.
dashboard
启用ceph cli（本质上是进入ceph容器）
[root@node-1 ~]# cephadm shell
Inferring fsid 4466c6c0-8828-11ee-b74e-08002726ce7d
Using recent ceph image quay.io/ceph/ceph@sha256:c00e089064cc7afd4db454ac645cc4d760f2eae86eb73655124abde9bb9058fb
[ceph: root@node-1 /]# ceph -s
cluster:
id: 4466c6c0-8828-11ee-b74e-08002726ce7d
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum node-1 (age 2h)
mgr: node-1.tmiabs(active, since 2h)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
可以看到目前有两个容器
[root@node-1 ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b647a11ed239 quay.io/ceph/ceph:v16 -n mon.node-1 -f ... 2 hours ago Up 2 hours ago ceph-4466c6c0-8828-11ee-b74e-08002726ce7d-mon-node-1
727bde63ebbc quay.io/ceph/ceph:v16 -n mgr.node-1.tmi... 2 hours ago Up 2 hours ago ceph-4466c6c0-8828-11ee-b74e-08002726ce7d-mgr-node-1-tmia
添加节点
配置免密登录（容器内）
[root@node-1 ~]# cephadm shell
Inferring fsid 4466c6c0-8828-11ee-b74e-08002726ce7d
Using recent ceph image quay.io/ceph/ceph@sha256:c00e089064cc7afd4db454ac645cc4d760f2eae86eb73655124abde9bb9058fb
[ceph: root@node-1 ~]# ceph cephadm get-pub-key > ~/ceph.pub
[ceph: root@node-1 ~]# ssh-copy-id -f -i ~/ceph.pub root@10.47.76.95
[ceph: root@node-1 ~]# ssh-copy-id -f -i ~/ceph.pub root@10.47.76.96
新节点安装podman
同步离线安装包
[root@node-1 ~]# rsync -avr -P rpmbuild/RPMS node-2:~/rpmbuild/
[root@node-1 ~]# rsync -avr -P rpmbuild/RPMS node-3:~/rpmbuild/
[root@node-1 ~]# rsync -avr -P ref node-2:~/
[root@node-1 ~]# rsync -avr -P ref node-3:~/
安装podman及依赖
[root@node-2 ~]# rpm -ivUh rpmbuild/RPMS/x86_64/*.rpm rpmbuild/RPMS/noarch/*.rpm ref/*.rpm
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Updating / installing...
1:protobuf-3.14.0-6.oe2203 ################################# [ 4%]
2:protobuf-compiler-3.14.0-6.oe2203################################# [ 8%]
3:protobuf-c-1.4.0-2.oe2203 ################################# [ 12%]
4:yajl-2.1.0-18.oe2203 ################################# [ 16%]
5:libnet-1.2-1.oe2203 ################################# [ 20%]
6:criu-3.16.1-2.oe2203 ################################# [ 24%]
7:crun-1.4.5-1 ################################# [ 28%]
Creating group dnsmasq with gid 987.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 987 and gid 987.
8:dnsmasq-2.86-2.oe2203 ################################# [ 32%]
9:container-selinux-2:2.138-4.oe220################################# [ 36%]
uavc: op=load_policy lsm=selinux seqno=3 res=1 10:containers-common-1:1.5.2-3.oe220################################# [ 40%]
11:containernetworking-plugins-1.0.1################################# [ 44%]
12:podman-gvproxy-1:3.4.4-1 ################################# [ 48%]
13:podman-plugins-1:3.4.4-1 ################################# [ 52%]
14:podman-debugsource-1:3.4.4-1 ################################# [ 56%]
15:crun-debugsource-1.4.5-1 ################################# [ 60%]
16:conmon-debugsource-2:2.1.0-1 ################################# [ 64%]
17:conmon-2:2.1.0-1 ################################# [ 68%]
18:podman-1:3.4.4-1 ################################# [ 72%]
19:podman-docker-1:3.4.4-1 ################################# [ 76%]
20:conmon-debuginfo-2:2.1.0-1 ################################# [ 80%]
21:crun-debuginfo-1.4.5-1 ################################# [ 84%]
22:podman-debuginfo-1:3.4.4-1 ################################# [ 88%]
23:podman-remote-1:3.4.4-1 ################################# [ 92%]
24:podman-help-1:3.4.4-1 ################################# [ 96%]
25:crun-help-1.4.5-1 ################################# [100%]
[root@node-3 ~]# rpm -ivUh rpmbuild/RPMS/x86_64/*.rpm rpmbuild/RPMS/noarch/*.rpm ref/*.rpm
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Updating / installing...
1:protobuf-3.14.0-6.oe2203 ################################# [ 4%]
2:protobuf-compiler-3.14.0-6.oe2203################################# [ 8%]
3:protobuf-c-1.4.0-2.oe2203 ################################# [ 12%]
4:yajl-2.1.0-18.oe2203 ################################# [ 16%]
5:libnet-1.2-1.oe2203 ################################# [ 20%]
6:criu-3.16.1-2.oe2203 ################################# [ 24%]
7:crun-1.4.5-1 ################################# [ 28%]
Creating group dnsmasq with gid 987.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 987 and gid 987.
8:dnsmasq-2.86-2.oe2203 ################################# [ 32%]
9:container-selinux-2:2.138-4.oe220################################# [ 36%]
uavc: op=load_policy lsm=selinux seqno=3 res=1 10:containers-common-1:1.5.2-3.oe220################################# [ 40%]
11:containernetworking-plugins-1.0.1################################# [ 44%]
12:podman-gvproxy-1:3.4.4-1 ################################# [ 48%]
13:podman-plugins-1:3.4.4-1 ################################# [ 52%]
14:podman-debugsource-1:3.4.4-1 ################################# [ 56%]
15:crun-debugsource-1.4.5-1 ################################# [ 60%]
16:conmon-debugsource-2:2.1.0-1 ################################# [ 64%]
17:conmon-2:2.1.0-1 ################################# [ 68%]
18:podman-1:3.4.4-1 ################################# [ 72%]
19:podman-docker-1:3.4.4-1 ################################# [ 76%]
20:conmon-debuginfo-2:2.1.0-1 ################################# [ 80%]
21:crun-debuginfo-1.4.5-1 ################################# [ 84%]
22:podman-debuginfo-1:3.4.4-1 ################################# [ 88%]
23:podman-remote-1:3.4.4-1 ################################# [ 92%]
24:podman-help-1:3.4.4-1 ################################# [ 96%]
25:crun-help-1.4.5-1 ################################# [100%]
添加主机（容器内）
[root@node-1 ~]# cephadm shell
Inferring fsid 4466c6c0-8828-11ee-b74e-08002726ce7d
Using recent ceph image quay.io/ceph/ceph@sha256:c00e089064cc7afd4db454ac645cc4d760f2eae86eb73655124abde9bb9058fb
[ceph: root@node-1 /]# ceph orch host add node-2 10.47.76.95 --labels _admin
Added host 'node-2' with addr '10.47.76.95'
[ceph: root@node-1 /]# ceph orch host add node-3 10.47.76.96 --labels _admin
Added host 'node-3' with addr '10.47.76.96'
查看主机列表
添加更多的mon节点
podman离线部署ceph镜像
[root@node-1 ~]# scp ceph-v16.tar node-2:~/
Authorized users only. All activities may be monitored and reported.
ceph-v16.tar 100% 1158MB 206.4MB/s 00:05
[root@node-1 ~]# scp ceph-v16.tar node-3:~/
Authorized users only. All activities may be monitored and reported.
ceph-v16.tar 100% 1158MB 222.8MB/s 00:05
[root@node-1 ~]#
登录node-2,node-3导入podman镜像
[root@node-2 ~]# podman load -i ceph-v16.tar
Getting image source signatures
Copying blob 602fe9f6cc5a done
Copying blob c9b8d8af634e done
Copying config ae96fc565b done
Writing manifest to image destination
Storing signatures
Loaded image(s): quay.io/ceph/ceph:v16
[root@node-3 ~]# podman load -i ceph-v16.tar
Getting image source signatures
Copying blob 602fe9f6cc5a done
Copying blob c9b8d8af634e done
Copying config ae96fc565b done
Writing manifest to image destination
Storing signatures
Loaded image(s): quay.io/ceph/ceph:v16
手动安装catatonit
[root@node-1 ~]# scp catatonit.x86_64 node-2:/usr/libexec/podman/catatonit
Authorized users only. All activities may be monitored and reported.
catatonit.x86_64 100% 3924KB 206.4MB/s 00:00
[root@node-1 ~]# ssh node-2 chmod +x /usr/libexec/podman/catatonit
Authorized users only. All activities may be monitored and reported.
[root@node-1 ~]# scp catatonit.x86_64 node-3:/usr/libexec/podman/catatonit
Authorized users only. All activities may be monitored and reported.
catatonit.x86_64 100% 3924KB 266.8MB/s 00:00
[root@node-1 ~]# ssh node-3 chmod +x /usr/libexec/podman/catatonit
Authorized users only. All activities may be monitored and reported.
添加mon(容器内)
[root@node-1 ~]# cephadm shell
Inferring fsid 4466c6c0-8828-11ee-b74e-08002726ce7d
Using recent ceph image quay.io/ceph/ceph@sha256:c00e089064cc7afd4db454ac645cc4d760f2eae86eb73655124abde9bb9058fb
[ceph: root@node-1 /]# ceph orch apply mon --unmanaged
Scheduled mon update...
[ceph: root@node-1 /]# ceph orch daemon add mon node-2:10.47.76.95
Deployed mon.node-2 on host 'node-2'
[ceph: root@node-1 /]# ceph orch daemon add mon node-3:10.47.76.96
Deployed mon.node-3 on host 'node-3'
# 把自己加回去！
[ceph: root@node-1 /]# ceph orch daemon add mon node-1:10.47.76.94
Deployed mon.node-1 on host 'node-1'
状态
提交
ceph orch apply mon --placement="node-1,node-2,node-3"
整半天，版本是16.2.14....
参考
Deploying a new Ceph cluster — Ceph Documentation
Host Management — Ceph Documentation
