准备步骤:
先将192.168.152.30 现有单节点集群stop,然后克隆两台虚拟机并在workstations重置网卡MAC,配置新主机IP
[admin@oceanbase ~]$ obd cluster list
+----------------------------------------------------------------------+
| Cluster List |
+-------------+--------------------------------------+-----------------+
| Name | Configuration Path | Status (Cached) |
+-------------+--------------------------------------+-----------------+
| obce-single | /home/admin/.obd/cluster/obce-single | running |
+-------------+--------------------------------------+-----------------+
[admin@oceanbase ~]$ obd cluster stop obce-single
Get local repositories ok
Search plugins ok
Open ssh connection ok
Stop observer ok
obce-single stopped
[admin@oceanbase ~]$ free -g
total used free shared buff/cache available
Mem: 15 0 13 0 1 11
Swap: 2 0 2
开始部署:
资源列表
IP 192.168.152.30、192.168.152.31、192.168.152.32
网卡名 ens33
OS rhel80
CPU 4C
内存 总内存 16G,可用内存 11G以上
磁盘1 /dev/sdb 30G(一块也够)
磁盘2 /dev/sdc 30G
机器和角色划分
OBD 192.168.152.30 中控机,自动化部署软件
OBServer 192.168.152.30 OceanBase 数据库 zone1
OBServer 192.168.152.31 OceanBase 数据库 zone2
OBServer 192.168.152.32 OceanBase 数据库 zone3
OBProxy 192.168.152.30 OceanBase 访问反向代理 --若环境资源允许,可以每台observer均部署obproxy
OBClient 192.168.152.30 OceanBase 命令行客户端
配置互信和时钟同步
三台机子均执行如下命令:
ssh-keygen -t rsa
ssh-copy-id 192.168.152.30
ssh-copy-id 192.168.152.31
ssh-copy-id 192.168.152.32
ssh 192.168.152.30
ssh 192.168.152.31
ssh 192.168.152.32
时钟同步配置
服务端:vi /etc/chrony.conf
server 127.0.0.1
allow 192.168.0.0/16
local stratum 10
客户端:vi /etc/chrony.conf
server 192.168.152.30
systemctl enable chronyd.service
systemctl restart chronyd.service
chronyc activity
[root@obce01 ~]# chronyc sources -v
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current best, '+' = combined, '-' = not combined,
| / 'x' = may be in error, '~' = too variable, '?' = unusable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* time.neu.edu.cn 1 7 377 85 -1452us[-1502us] +/- 19ms
^- ntp.wdc2.us.leaseweb.net 2 8 377 144 +1276us[+1228us] +/- 247ms
^+ time.neu.edu.cn 1 7 377 27 +2004us[+2004us] +/- 18ms
^+ 139.199.214.202 2 8 377 93 +1098us[+1049us] +/- 56ms
^? localhost 0 9 0 - +0ns[ +0ns] +/- 0ns
sudo clockdiff 192.168.152.30
部署三节点 observer 和 obproxy 进程:
下载yaml模板
https://github.com/oceanbase/obdeploy/blob/master/example/mini-distributed-with-obproxy-example.yaml
su - admin
vi /home/admin/obcluster-3zones.yaml
## Only need to configure when remote login is required
user:
username: admin
key_file: /home/admin/.ssh/id_rsa.pub
port: your ssh port, default 22
oceanbase-ce:
servers:
- name: obce01
ip: 192.168.152.30
- name: obce02
ip: 192.168.152.31
- name: obce03
ip: 192.168.152.32
global:
devname: ens33
cluster_id: 2
memory_limit: 8G # The maximum running memory for an observer
system_memory: 3G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
datafile_size: 10G # Size of the data file.
log_disk_size: 3G # The size of disk space used by the clog files.
cpu_count: 16
production_mode: false
syslog_level: WARN # System log level. The default value is INFO.
enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
max_syslog_file_count: 4
appname: obcluster-3zones # observer cluster name, consistent with obproxy's cluster_name
root_password: 0EI5N08d# root_password: # root user password, can be empty
proxyro_password: uY7Yf8zx# proxyro_password: # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty
obce01:
mysql_port: 3881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
rpc_port: 3882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
home_path: /home/admin/oceanbase-ce
data_dir: /data
redo_dir: /redo
zone: zone1
obce02:
mysql_port: 3881
rpc_port: 3882
home_path: /home/admin/oceanbase-ce
data_dir: /data
redo_dir: /redo
zone: zone2
obce03:
mysql_port: 3881
rpc_port: 3882
home_path: /home/admin/oceanbase-ce
data_dir: /data
redo_dir: /redo
zone: zone3
obproxy:
# Set dependent components for the component.
# When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.
depends: #注意上面那行注释的意思,参数将继承depends所配置的组件
- oceanbase-ce
servers:
- 192.168.152.30
# - 192.168.152.31 #本人环境资源有限,obproxy只装在obd上
# - 192.168.152.32
global:
listen_port: 3883 # External port. The default value is 2883.
prometheus_listen_port: 3884 # The Prometheus port. The default value is 2884.
home_path: /home/admin/obproxy
rs_list: 192.168.152.30:3881;192.168.152.31:3881;192.168.152.32:3881
enable_cluster_checkout: false
cluster_name: obcluster-3zones
skip_proxy_sys_private_check: true
enable_strict_kernel_release: false
obproxy_sys_password: 0MdTv1tm # obproxy sys user password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
observer_sys_password: uY7yf8zx # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
1)若yaml文件部署报参数错误,请仔细检查参数。每个配置项后面冒号(:)跟后面的值之间必须有个空格 (’ ')。
2)若报密码obproxy_sys_password和observer_sys_password相关报错,配置文件中可以删除该部分配置,待集群创建完成后自行配置即可
OBD 部署集群:
i.复制软件到相应节点,并在各节点安装软件,本环境方案中,各节点为克隆虚拟机,无需安装。
ii.在相应节点创建相关目录。
iii.obd cluster deploy [集群名] -c 集群配置文件
obd cluster deploy obcluster-3zones -c /home/admin/obcluster-3zones.yaml
[ERROR] OBD-1002: Fail to init obce01(192.168.152.30) home path: /home/admin/oceanbase-ce is not empty.
初次安装不会报我这个错
IPS="192.168.152.30 192.168.152.31 192.168.152.32"
for IP in $IPS;do echo $IP; ssh $IP "/bin/rm -rf ~/oceanbase-ce/ /data/ /redo/ ~/obproxy/ "; done
解决办法:
配置文件中注释掉图中obproxy_sys_password: 0MdTv1tm 和observer_sys_password: uY7yf8zx,待集群创建完成后自行配置即可(后面*连接obproxy集群 部分有介绍)
初始化并启动集群
obd cluster start obcluster-3zones
如果集群节点内核参数和会话限制参数不符合要求,安装时会给出提示。 这个命令会进行几分钟的 bootstrap。当可用内存不足 8G 或者日志目录剩余可用空间比例不足 5% 的时候,这个bootstrap 是很可能会失败的。
[admin@obce01 ~]$ obd cluster list
+--------------------------------------------------------------------------------+
| Cluster List |
+------------------+-------------------------------------------+-----------------+
| Name | Configuration Path | Status (Cached) |
+------------------+-------------------------------------------+-----------------+
| obce-single | /home/admin/.obd/cluster/obce-single | destroyed |
| obcluster-3zones | /home/admin/.obd/cluster/obcluster-3zones | running |
+------------------+-------------------------------------------+-----------------+
[admin@obce01 oceanbase-ce]$ obd cluster display obcluster-3zones
Get local repositories and plugins ok
Open ssh connection ok
Cluster status check ok
Connect to observer ok
Wait for observer init ok
+--------------------------------------------------+
| observer |
+----------------+---------+------+-------+--------+
| ip | version | port | zone | status |
+----------------+---------+------+-------+--------+
| 192.168.152.30 | 3.1.0 | 3881 | zone1 | active |
| 192.168.152.31 | 3.1.0 | 3881 | zone2 | active |
| 192.168.152.32 | 3.1.0 | 3881 | zone3 | active |
+----------------+---------+------+-------+--------+
obclient -h192.168.152.30 -P3881 -uroot -p0EI5N08d -Doceanbase -A
Connect to obproxy ok
+--------------------------------------------------+
| obproxy |
+----------------+------+-----------------+--------+
| ip | port | prometheus_port | status |
+----------------+------+-----------------+--------+
| 192.168.152.30 | 3883 | 3884 | active |
+----------------+------+-----------------+--------+
若
检查 OceanBase 集群各个节点信息
i.进程
OceanBase 数据库是单进程软件,进程名为 observer ,可运行以下命令查看这个进程。
IPS="192.168.152.30 192.168.152.31 192.168.152.32"
for ob in $IPS;do echo $ob; ssh $ob "ps -ef | grep observer | grep -v grep "; done
查看observer启动目录
ls -l /proc/进程号/cwd ,与配置文件一致
[admin@obce01 ~]$ cat obcluster-3zones.yaml |grep home
home_path: /home/admin/oceanbase-ce
2.监听
IPS="192.168.152.30 192.168.152.31 192.168.152.32"
for ob in $IPS;do echo $ob; ssh $ob "netstat -ntlp"; done
3.查看目录结构
tree ~/oceanbase-ce/ /data/ /redo/ ~/obproxy/
使用ob集群
i.直连ob集群、连接obproxy集群以及通过obproxy代理连ob集群的方法
1.直连 OceanBase 集群的内部实例(sys),并查看相关资源
obclient -h192.168.152.30 -P3881 -uroot -p0EI5N08d -Doceanbase -A
select a.zone,concat(a.svr_ip,':',a.svr_port) observer, cpu_total, (cpu_total-cpu_assigned) cpu_free, round(mem_total/1024/1024/1024) mem_total_gb, round((mem_total-mem_assigned)/1024/1024/1024)
mem_free_gb, usec_to_time(b.last_offline_time) last_offline_time, usec_to_time(b.start_service_time) start_service_time, b.status, usec_to_time(b.stop_time) stop_time, b.build_version
from __all_virtual_server_stat a join __all_server b on (a.svr_ip=b.svr_ip and a.svr_port=b.svr_port)order by a.zone, a.svr_ip;
2.连接obproxy集群
obclient -h 192.168.152.30 -uroot@proxysys -P3883 -p0MdTv1tm
发现报密码相关错误:
原因如下:ymal文件在集群初始化时报错,obproxy的obproxy_sys_password和observer_sys_password被我从ymal文件注释掉了。proxysys 的 root密码为空。
obclient -h 192.168.152.30 -uroot@proxysys -P3883 -p
Enter password:此处直接回车即可
[admin@obce01 ~]$ obclient -h 192.168.152.30 -uroot@proxysys -P3883 -p
Enter password:
此处省略部分显示
MySQL [(none)]> show proxyconfig like '%sys_password%';
+-----------------------+-------+--------------------------------+-------------+---------------+
| name | value | info | need_reboot | visible_level |
+-----------------------+-------+--------------------------------+-------------+---------------+
| observer_sys_password | | password for observer sys user | false | SYS | ======================value为空
| obproxy_sys_password | | password for obproxy sys user | false | SYS |
+-----------------------+-------+--------------------------------+-------------+---------------+ ======================value为空
2 rows in set (0.006 sec)
MySQL [(none)]> alter proxyconfig set obproxy_sys_password="0MdTv1tm";
Query OK, 0 rows affected (0.007 sec)
MySQL [(none)]> alter proxyconfig set observer_sys_password="uY7Yf8zx"; =====================consistent with oceanbase-ce's proxyro_password
Query OK, 0 rows affected (0.008 sec)
MySQL [(none)]> show proxyconfig like '%sys_password%';
+-----------------------+------------------------------------------+--------------------------------+-------------+---------------+
| name | value | info | need_reboot | visible_level |
+-----------------------+------------------------------------------+--------------------------------+-------------+---------------+
| observer_sys_password | 34864da3a09bda2d587ac2c9d8e4521981586fc3 | password for observer sys user | false | SYS |
| obproxy_sys_password | 9fe82c02e25ae612ed310ab0ac81c97bd813dd4e | password for obproxy sys user | false | SYS |
+-----------------------+------------------------------------------+--------------------------------+-------------+---------------+
2 rows in set (0.015 sec)
MySQL [(none)]> Bye
带上密码重新尝试连接proxysys的root用户
obclient -h 192.168.152.30 -uroot@proxysys -P3883 -p0MdTv1tm 成功
3.使用obclient通过obproxy连接集群
obclient -h 192.168.152.30 -uroot@sys#obcluster-3zones -P3883 -p0EI5N08d -c -A oceanbase
此处在通过obproxy登录observer,用户为root,密码为observer的root_password: 0EI5N08d(不是proxyro的password)
ii.简单使用运维命令
查看参数
show parameters like '%_password%';
show parameters like 'memory_limit%';
show parameters where name in ('memory_limit','memory_limit_ percentage');
show parameters like 'config_additional_dir';
修改参数
alter system set syslog_level = 'USER_ERR' server='192.168.152.30:3882';
iii.OceanBase 集群参数文件
show parameters like 'config_additional_dir'
通常 OceanBase 集群每个节点的启动目录下都会有一个目录 etc,保存了该节点进程的参数文件observer.config.bin。
observer.config.bin 是一个 binary 类型的文件,不能直接用 cat命令读取,可使用 strings 命令读取。
该文件也不建议直接修改,可通过上面提到的命令进行修改 alter 。
observer进程默认会读取文件夹 etc 中的参数文件,其他两个目录是参数文件的备份,这个备份路径也是通过参数config_additional_dir 指定的,默认值是同一个启动目录的 etc2 和 etc3。
生产环境建议将 etc设置到其他磁盘,这样会更加安全。
[admin@obce01 etc]$ pwd
/home/admin/oceanbase-ce/etc
[admin@obce01 etc]$ strings observer.config.bin
_enable_oracle_priv_check=True
_max_trx_size=100M
all_server_list=192.168.152.30:3882,192.168.152.31:3882,192.168.152.32:3882
__min_full_resource_pool_memory=268435456
min_observer_version=3.1.0
enable_syslog_recycle=True
enable_syslog_wf=False
max_syslog_file_count=4
syslog_level=WARN
cluster_id=2
cluster=obcluster-3zones
rootservice_list=192.168.152.30:3882:3881;192.168.152.31:3882:3881;192.168.152.32:3882:3881
_partition_balance_strategy=standard
enable_one_phase_commit=False
cpu_count=16
system_memory=3G
memory_limit=8G
zone=zone1
devname=ens33
mysql_port=3881
rpc_port=3882
datafile_size=10G
data_dir=/data
使用 OBD 修改 OceanBase 集群参数:
obd cluster edit-config obcluster-3zones
obd cluster reload obcluster-3zones
手动启动observer进程(应急)
进入home_path:/home/admin/oceanbase-ce
cd /home/admin/oceanbase-ce
bin/observer -o "max_syslog_file_count=4,datafile_size=10G"