bclinux aarch64 ceph 14.2.10 对象存储 http网关 CEPH OBJECT GATEWAY Civetweb

部署对象存储网关

cd /etc/ceph
ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3

成功日志

[root@ceph-0 ceph]# ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-0', 'rgw.ceph-0'), ('ceph-1', 'rgw.ceph-1'), ('ceph-2', 'rgw.ceph-2'), ('ceph-3', 'rgw.ceph-3')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0xffffa1b8faa0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0xffffa1d554d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-0:rgw.ceph-0 ceph-1:rgw.ceph-1 ceph-2:rgw.ceph-2 ceph-3:rgw.ceph-3
[ceph-0][DEBUG ] connected to host: ceph-0 
[ceph-0][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-0][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-0
[ceph-0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-0][WARNIN] rgw keyring does not exist yet, creating one
[ceph-0][DEBUG ] create a keyring file
[ceph-0][DEBUG ] create path recursively if it doesn't exist
[ceph-0][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-0 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring
[ceph-0][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-0
[ceph-0][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-0.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-0][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-0
[ceph-0][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-0 and default port 7480
dhclient(1787) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
dhclient(1787) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-1][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-1
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-1][WARNIN] rgw keyring does not exist yet, creating one
[ceph-1][DEBUG ] create a keyring file
[ceph-1][DEBUG ] create path recursively if it doesn't exist
[ceph-1][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-1 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-1/keyring
[ceph-1][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-1
[ceph-1][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-1.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-1][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-1
[ceph-1][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-1 and default port 7480
dhclient(1742) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
dhclient(1742) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
[ceph-2][DEBUG ] connected to host: ceph-2 
[ceph-2][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-2][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-2
[ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-2][WARNIN] rgw keyring does not exist yet, creating one
[ceph-2][DEBUG ] create a keyring file
[ceph-2][DEBUG ] create path recursively if it doesn't exist
[ceph-2][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-2 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-2/keyring
[ceph-2][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-2
[ceph-2][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-2.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-2][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-2
[ceph-2][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-2 and default port 7480
dhclient(1722) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
dhclient(1722) is already running - exiting. 

This version of ISC DHCP is based on the release available
on ftp.isc.org. Features have been added and other changes
have been made to the base software release in order to make
it work better with this distribution.

Please report issues with this software via: 
https://gitee.com/src-openeuler/dhcp/issues

exiting.
[ceph-3][DEBUG ] connected to host: ceph-3 
[ceph-3][DEBUG ] detect platform information from remote host
21.10U3 LTS
bclinux
[ceph-3][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-3
[ceph-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-3][WARNIN] rgw keyring does not exist yet, creating one
[ceph-3][DEBUG ] create a keyring file
[ceph-3][DEBUG ] create path recursively if it doesn't exist
[ceph-3][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-3 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-3/keyring
[ceph-3][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-3
[ceph-3][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-3.service → /usr/lib/systemd/system/ceph-radosgw@.service.
[ceph-3][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-3
[ceph-3][INFO  ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-3 and default port 7480
[root@ceph-0 ceph]# cp: cannot stat '/tmp/security_updates': No such file or directory
cp: cannot stat '/tmp/security_updates': No such file or directory
cp: cannot stat '/tmp/security_updates': No such file or directory

创建成功

验证服务状态（失败）

日志目录

/var/log/ceph

日志文件

ceph-client.rgw.ceph-0.log

清空日志文件，重启服务

cd /var/log/ceph
echo "" > ceph-client.rgw.ceph-0.log
systemctl start ceph-radosgw@rgw-ceph-0

先看服务日志

journalctl -xeu ceph-radosgw@rgw-ceph-0

日志

Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory
Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffff7a330148) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx
Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory
Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffffdc7f98c8) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx
Nov 14 13:38:08 ceph-0 radosgw[58375]: failed to fetch mon config (--no-mon-config to skip)
Nov 14 13:38:08 ceph-0 systemd[1]: ceph-radosgw@rgw-ceph-0.service: Main process exited, code=exited, status=1/FAILURE

没有按照官方文档执行网关安装有关？ceph-deploy install --rgw node..

但是这个版本的ceph-deploy指定的ceph版本太低，且是el6

解决 /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory

查看/etc/ceph目录下的keyring文件，发现已生成了ceph.bootstrap-rgw.keyring

错误的路径/var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring

正确的路径/var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring

不知道为什么路径有一个符号错误

做个软连接

cd /var/lib/ceph/radosgw
ln -s ceph-rgw.ceph-0 ceph-rgw-ceph-0

错误：

Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.076 ffff99055730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.080 ffff99856730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
Nov 14 14:11:43 ceph-0 radosgw[59264]: failed to fetch mon config (--no-mon-config to skip)
Nov 14 14:11:43 ceph-0 systemd[1]: ceph-radosgw@rgw-ceph-0.service: Main process exited, code=exited, status=1/FAILURE

重新执行部署

cd /etc/ceph
ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3

重新部署后，会生成/var/log/ceph/ceph-client.rgw.ceph-0.log日志文件

其中有日志

Numerical result out of range

2023-11-14 14:32:24.461 ffff8abe8010 0 deferred set uid:gid to 1000:1000 (ceph:ceph)

2023-11-14 14:32:24.461 ffff8abe8010 0 ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable), process radosgw, pid 60035

2023-11-14 14:32:25.509 ffff8abe8010 0 rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)

2023-11-14 14:32:25.509 ffff8abe8010 0 failed reading realm info: ret -34 (34) Numerical result out of range

2023-11-14 14:32:25.509 ffff8abe8010 0 ERROR: failed to start notify service ((34) Numerical result out of range

2023-11-14 14:32:25.509 ffff8abe8010 0 ERROR: failed to init services (ret=(34) Numerical result out of range)

2023-11-14 14:32:25.513 ffff8abe8010 -1 Couldn't init storage provider (RADOS)

设置默认pg_num pgs_num

/etc/ceph/ceph.conf

osd pool default size = 3
osd pool default min_size = 1
osd pool default pg num = 128
osd pool default pgp num = 128

重新部署(--overwrite-conf)

cd /etc/ceph
ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3

故障依旧

成功的方案

删掉所有pool，重新部署，参考以下历史记录

652 ceph osd pool ls
653 ceph osd pool rm vdbench
654 ceph osd pool rm vdbench vdbench --yes-i-really-really-mean-it
655 ls /dev/
656 mount
657 df -h
658 umount /mnt/rbd0
659 ceph osd ls
660 ceph osd pool ls
661 ceph osd volume ls
662 ceph osd dump
663 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it
664 ceph fs ls
665 ceph fs rm cephfs
666 ceph fs fail
667 ceph fs fail cephfs
668 ceph -s
669 ceph fs status cephfs
670 ceph fs ls
671 ceph fs rm cephfs
672 ceph fs rm cephfs --yes-i-really-mean-it
673 ceph fs ls
674 ceph osd pool ls
675 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-mean-it
676 ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it
677 ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it
678 ceph -
679 ceph -s
680 cd /etc/ceph
681 ceph-deploy rgw create ceph-0
682 cat /var/log/ceph/ceph-client.rgw.ceph-0.log
683 systemctl status ceph-radosgw@rgw.ceph-0
再来一次集群环境的部署

cd /etc/ceph
ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3

部分日志

ceph -s rgw已经起来了

======================

以下为参考官方文档的失败经理，留档参考

部署对象存储网关rgw（失败）

ceph-deploy install --rgw ceph-0 ceph-1 ceph-2 ceph-3

故障 Unable to find a match: epel-release

dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y

故障Status code: 404

坑爹的阿里云镜像，没有epel 8，bclinux又是el8版本！

采用huawei epel镜像

[epel-everything]
name=huawei epel mirror
baseurl=https://repo.huaweicloud.com/epel/8/Everything/aarch64/
enabled=1
gpgcheck=0

yum install epel-release --nobest --skip-broken

ceph-deploy 会强制执行yum install epel-release(失败)

注释掉epel-release的安装

修改/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 60行，注释掉epel-release的安装（前面已经手动--skip-broken安装好了）

生效了

故障 Unable to find a match: yum-plugin-priorities

参考上面思路，直接注释掉/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 61,62行

自动找到老版本ceph-release-1-0.el6 Status code: 404

手动安装

修改/usr/lib/python2.7/site-packages/ceph_deploy/install.py

安装版本参数args.release = 'mimic' 改为 args.release = 'nautilus

关于python3版本

bclinux python3.7.9

cenos7 python3.6.8

openeuler 20.03 lts sp3 python 3.7.9

参考

Ceph Object Gateway — Ceph Documentation

如何在 CentOS 8 和 RHEL 8 服务器上启用 EPEL 仓库 - 知乎 (zhihu.com)

ceph-deploy部署ceph-nautilus_ceph nautilus-CSDN博客