MySQL入门篇-MySQL MHA高可用实战

news2024/12/27 13:00:25

MHA简介

MHA（Master High Availability）目前在MySQL高可用方面是一个相对成熟的解决方案，它由日本DeNA公司的youshimaton（现就职于Facebook公司）开发，是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中，MHA能做到在0~30秒之内自动完成数据库的故障切换操作，并且在进行故障切换的过程中，MHA能在最大程度上保证数据的一致性，以达到真正意义上的高可用。
MHA架构图

image.png

MHA工作原理总结为以下几条：

  • 从宕机崩溃的master保存二进制日志事件（binlog events）；
  • 识别含有最新更新的slave；
  • 应用差异的中继日志（relay log）到其他slave；
  • 应用从master保存的二进制日志事件（binlog events）；
  • 提升一个slave为新master；
  • 使用其他的slave连接新的master进行复制。

MHA特点:

  • 故障切换时间 10-30秒
  • 自动监控Master以及故障转移
  • 在线切换Master到异机

MHA搭建 

 基本环境

操作系统版本：CentOS Linux release 7.8.2003 (Core)
MySQL版本 ：5.7.31
VIP ：10.31.1.241

主机信息:

配置主从 

配置 hp3、hp4两台同步hp2的主从复制，参考主从复制blog，此处略过

安装Perl等依赖模块

用root用户在所有四个节点执行下面的操作。

# 安装一个epel源
wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo
 
# 用yum安装依赖包
yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes -y

配置SSH登录无密码验证 

在hdp1 10.31.1.119（Monitor）上用root用户执行：

ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.120
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.121
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.122

在hdp2 10.31.1.120（Master）上用root用户执行：

ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.119
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.121
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.122

在hdp3 10.31.1.121（Candidate Master）上用root用户执行：

ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.119
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.120
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.122

在hdp4 10.31.1.122（Slave）上用root用户执行：

ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.119
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.120
ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.31.1.121

安装MHA Node

下载地址：https://github.com/yoshinorim/mha4mysql-manager/wiki/Downloads
在hdp2、hdp3、hdp4上用root用户执行下面的操作。

rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm

安装完成后，在/usr/bin/目录下有如下MHA相关文件：

apply_diff_relay_logs
filter_mysqlbinlog
purge_relay_logs
save_binary_logs

这些脚本工具通常由MHA Manager的脚本触发，无需人为操作。脚本说明：
apply_diff_relay_logs：识别差异的中继日志事件并将其差异的事件应用于其它slave。
filter_mysqlbinlog：去除不必要的ROLLBACK事件（MHA已不再使用这个工具）。
purge_relay_logs：清除中继日志（不会阻塞SQL线程）。
save_binary_logs：保存和复制master的二进制日志。

安装MHA Manager

下载地址：https://github.com/yoshinorim/mha4mysql-manager/wiki/Downloads
在hdp1上用root用户执行下面的操作。

rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm

安装报错

[root@10-31-1-119 src]# rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm 
错误：依赖检测失败：
        mha4mysql-node >= 0.54 被 mha4mysql-manager-0.56-0.el6.noarch 需要
        perl(MHA::BinlogManager) 被 mha4mysql-manager-0.56-0.el6.noarch 需要
        perl(MHA::NodeConst) 被 mha4mysql-manager-0.56-0.el6.noarch 需要
        perl(MHA::NodeUtil) 被 mha4mysql-manager-0.56-0.el6.noarch 需要
        perl(MHA::SlaveUtil) 被 mha4mysql-manager-0.56-0.el6.noarch 需要
[root@10-31-1-119 src]# 
[root@10-31-1-119 src]# 
[root@10-31-1-119 src]# yum -y install mha4mysql-manager-0.56-0.el6.noarch.rpm
已加载插件：fastestmirror, langpacks
正在检查 mha4mysql-manager-0.56-0.el6.noarch.rpm: mha4mysql-manager-0.56-0.el6.noarch
mha4mysql-manager-0.56-0.el6.noarch.rpm 将被安装
正在解决依赖关系
--> 正在检查事务
---> 软件包 mha4mysql-manager.noarch.0.0.56-0.el6 将被 安装
--> 正在处理依赖关系 mha4mysql-node >= 0.54，它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
Loading mirror speeds from cached hostfile
--> 正在处理依赖关系 perl(MHA::BinlogManager)，它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
--> 正在处理依赖关系 perl(MHA::NodeConst)，它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
--> 正在处理依赖关系 perl(MHA::NodeUtil)，它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
--> 正在处理依赖关系 perl(MHA::SlaveUtil)，它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
--> 解决依赖关系完成
错误：软件包：mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)
          需要：mha4mysql-node >= 0.54
错误：软件包：mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)
          需要：perl(MHA::NodeUtil)
错误：软件包：mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)
          需要：perl(MHA::BinlogManager)
错误：软件包：mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)
          需要：perl(MHA::NodeConst)
错误：软件包：mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)
          需要：perl(MHA::SlaveUtil)
 您可以尝试添加 --skip-broken 选项来解决该问题
 您可以尝试执行：rpm -Va --nofiles --nodigest
[root@10-31-1-119 src]#

监控节点需要先安装node再安装manager

[root@10-31-1-119 src]# rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm
准备中...                          ################################# [100%]
正在升级/安装...
   1:mha4mysql-node-0.56-0.el6        ################################# [100%]
[root@10-31-1-119 src]# 
[root@10-31-1-119 src]# rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm
准备中...                          ################################# [100%]
正在升级/安装...
   1:mha4mysql-manager-0.56-0.el6     ################################# [100%]
[root@10-31-1-119 src]#

安装完成后，在/usr/bin/目录下有如下MHA相关文件：

masterha_check_repl
masterha_check_ssh
masterha_check_status
masterha_conf_host
masterha_manager
masterha_master_monitor
masterha_master_switch
masterha_secondary_check
masterha_stop
apply_diff_relay_logs
filter_mysqlbinlog
purge_relay_logs
save_binary_logs

配置MHA

在hdp1上用root用户执行下面（1）、（2）、（3）的操作。
（1）建立配置文件目录

mkdir -p /etc/masterha

（2）创建配置文件/etc/masterha/app1.cnf，内容如下：

[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1.log
master_binlog_dir=/var/lib/mysql
master_ip_failover_script=/usr/bin/master_ip_failover
master_ip_online_change_script=/usr/bin/master_ip_online_change
password=abc123
ping_interval=1
remote_workdir=/tmp
repl_password=abc123
repl_user=repl
secondary_check_script=/usr/bin/masterha_secondary_check -s hp2 -s hp3 --user=root --master_host=hp2 --master_ip=10.31.1.120 --master_port=3306
shutdown_script=""
ssh_user=root
user=root
 
[server1]
hostname=10.31.1.120
port=3306
 
[server2]
candidate_master=1
check_repl_delay=0
hostname=10.31.1.121
port=3306
 
[server3]
hostname=10.31.1.122
port=3306

server default段是manager的一些基本配置参数，server1、server2、server3分别对应复制中的master、第一个slave、第二个slave。该文件的语法要求严格，变量值后不要有多余的空格。主要配置项说明如下。

manager_log：设置manager的日志文件。
manager_workdir：设置manager的工作目录。
master_binlog_dir：设置master保存binlog的位置，以便MHA可以找到master的日志，这里的也就是mysql的数据目录。
master_ip_failover_script：设置自动failover时候的切换脚本。
master_ip_online_change_script：设置手动切换时候的切换脚本。
password：设置mysql中root用户的密码。
ping_interval：设置监控主库，发送ping包的时间间隔，默认是3秒，尝试三次没有回应的时候自动进行railover。
remote_workdir：设置远端mysql在发生切换时binlog的保存位置。
repl_password：设置复制用户的密码。
repl_user：设置复制环境中的复制用户名
secondary_check_script：一旦MHA到hdp4的监控之间出现问题，MHA Manager将会尝试从hdp3登录到hdp4。
shutdown_script：设置故障发生后关闭故障主机脚本。该脚本的主要作用是关闭主机放在发生脑裂，这里没有使用。
ssh_user：设置ssh的登录用户名。
user：设置监控用户为root。
candidate_master：设置为候选master。设置该参数以后，发生主从切换以后将会将此从库提升为主库，即使这个主库不是集群中事件最新的slave。
check_repl_delay：默认情况下如果一个slave落后master 100M的relay logs的话，MHA将不会选择该slave作为一个新的master，因为对于这个slave的恢复需要花费很长时间，通过设置check_repl_delay=0，MHA触发切换在选择一个新的master的时候将会忽略复制延时，这个参数对于设置了candidate_master=1的主机非常有用，因为这个候选主在切换的过程中一定是新的master。
（3）建立软连接

-- 这一步如果 mysqlbinlog和mysql已经可以直接使用就没问题
ln -s /home/mysql/mysql-5.6.14/bin/mysqlbinlog /usr/bin/mysqlbinlog
ln -s /home/mysql/mysql-5.6.14/bin/mysql /usr/bin/mysql

（4）设置复制中Slave的relay_log_purge参数
在hdp3和hdp4上用mysql用户执行：

mysql -uroot -pabc123 -e "set global relay_log_purge=0"

注意，MHA在发生切换的过程中，从库的恢复过程中依赖于relay log的相关信息，所以这里要将relay log的自动清除设置为OFF，采用手动清除relay log的方式。默认情况下，从服务器上的中继日志会在SQL线程执行完毕后被自动删除。但是在MHA环境中，这些中继日志在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。在ext3的文件系统下，删除大的文件需要一定的时间，会导致严重的复制延时。为了避免复制延时，需要暂时为中继日志创建硬链接，因为在linux系统中通过硬链接删除大文件速度会很快。（在mysql数据库中，删除大表时，通常也采用建立硬链接的方式）


创建相关脚本

创建定期清理relay脚本

在hdp3、hdp4两台slave上建立/root/purge_relay_log.sh文件，内容如下：

#!/bin/bash
 
. /home/mysql/.bashrc
 
user=root
passwd=abc123
port=3306
log_dir='/data'
work_dir='/data'
purge='/usr/bin/purge_relay_logs'
 
if [ ! -d $log_dir ]
then
   mkdir $log_dir -p
fi
 
$purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port --workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1

purge_relay_logs的参数说明：

user mysql：MySQL用户名。
password mysql：MySQL用户密码。
port：MySQL端口号。
workdir：指定创建relay log的硬链接的位置，默认是/var/tmp。由于系统不同分区创建硬链接文件会失败，故需要执行硬链接具体位置，成功执行脚本后，硬链接的中继日志文件被删除。
disable_relay_log_purge：默认情况下，如果relay_log_purge=1，脚本会什么都不清理，自动退出。通过设定这个参数，当relay_log_purge=1的情况下会将relay_log_purge设置为0。清理relay log之后，最后将参数设置为OFF。
改模式为可执行：

mkdir -p /data
chmod 755 purge_relay_log.sh

添加到crontab中：

0 4 * * * /bin/bash /root/purge_relay_log.sh

创建自动failover脚本

在hdp1上创建/usr/bin/master_ip_failover文件，内容如下：

#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
    $command,          $ssh_user,        $orig_master_host, $orig_master_ip,
    $orig_master_port, $new_master_host, $new_master_ip,    $new_master_port
);

my $vip = '10.31.1.241';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens192:$key $vip/24";
my $ssh_stop_vip = "/sbin/ifconfig ens192:$key down";


GetOptions(
    'command=s'          => \$command,
    'ssh_user=s'         => \$ssh_user,
    'orig_master_host=s' => \$orig_master_host,
    'orig_master_ip=s'   => \$orig_master_ip,
    'orig_master_port=i' => \$orig_master_port,
    'new_master_host=s'  => \$new_master_host,
    'new_master_ip=s'    => \$new_master_ip,
    'new_master_port=i'  => \$new_master_port,
);

exit &main();

sub main {

    print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

    if ( $command eq "stop" || $command eq "stopssh" ) {

        my $exit_code = 1;
        eval {
            print "Disabling the VIP on old master: $orig_master_host \n";
            &stop_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn "Got Error: $@\n";
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "start" ) {

        my $exit_code = 10;
        eval {
            print "Enabling the VIP - $vip on the new master - $new_master_host \n";
            &start_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn $@;
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "status" ) {
        print "Checking the Status of the script.. OK \n";
        exit 0;
    }
    else {
        &usage();
        exit 1;
    }
}
sub start_vip() {
    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
    print
    "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

注意脚本中的vip部分

chmod u+x /usr/bin/master_ip_failover

创建手动failover脚本

在hdp1上创建/usr/bin/master_ip_online_change文件，内容如下：

#!/usr/bin/env perl
  
## Note: This is a sample script and is notcomplete. Modify the script based on your environment.
  
use strict;
use warnings FATAL => 'all';
  
use Getopt::Long;
use MHA::DBHelper;
use MHA::NodeUtil;
# use Time::HiRes qw( sleep gettimeofdaytv_interval );
use Time::HiRes qw(sleep gettimeofday tv_interval);
use Data::Dumper;
  
my $_tstart;
my $_running_interval = 0.1;
my (
 $command,         $orig_master_host, $orig_master_ip,
 $orig_master_port, $orig_master_user,
 $new_master_host, $new_master_ip,   $new_master_port,
 $new_master_user, 
);
  
my $vip = '10.31.1.241';  # Virtual IP 
my $key = "1"; 
my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip/24";
my $ssh_stop_vip = "/sbin/ifconfig ens160:$key down";
my $ssh_user = "root";
my $new_master_password = "123456";
my $orig_master_password = "123456";
  
GetOptions(
 'command=s'              =>\$command,
 #'ssh_user=s'             => \$ssh_user, 
 'orig_master_host=s'     =>\$orig_master_host,
 'orig_master_ip=s'       =>\$orig_master_ip,
 'orig_master_port=i'     =>\$orig_master_port,
 'orig_master_user=s'     =>\$orig_master_user,
 #'orig_master_password=s' => \$orig_master_password,
 'new_master_host=s'      =>\$new_master_host,
 'new_master_ip=s'        =>\$new_master_ip,
 'new_master_port=i'      =>\$new_master_port,
 'new_master_user=s'      =>\$new_master_user,
 #'new_master_password=s'  =>\$new_master_password,
);
  
exit &main();
  
sub current_time_us {
  my ($sec, $microsec ) = gettimeofday();
  my$curdate = localtime($sec);
 return $curdate . " " . sprintf( "%06d", $microsec);
}
  
sub sleep_until {
  my$elapsed = tv_interval($_tstart);
  if ($_running_interval > $elapsed ) {
   sleep( $_running_interval - $elapsed );
  }
}
  
sub get_threads_util {
  my$dbh                    = shift;
  my$my_connection_id       = shift;
  my$running_time_threshold = shift;
  my$type                   = shift;
 $running_time_threshold = 0 unless ($running_time_threshold);
 $type                   = 0 unless($type);
  my@threads;
  
  my$sth = $dbh->prepare("SHOW PROCESSLIST");
 $sth->execute();
  
 while ( my $ref = $sth->fetchrow_hashref() ) {
    my$id         = $ref->{Id};
    my$user       = $ref->{User};
    my$host       = $ref->{Host};
    my$command    = $ref->{Command};
    my$state      = $ref->{State};
    my$query_time = $ref->{Time};
    my$info       = $ref->{Info};
   $info =~ s/^\s*(.*?)\s*$/$1/ if defined($info);
   next if ( $my_connection_id == $id );
   next if ( defined($query_time) && $query_time <$running_time_threshold );
   next if ( defined($command)   && $command eq "Binlog Dump" );
   next if ( defined($user)      && $user eq "system user" );
   next
     if ( defined($command)
     && $command eq "Sleep"
     && defined($query_time)
     && $query_time >= 1 );
  
    if( $type >= 1 ) {
     next if ( defined($command) && $command eq "Sleep" );
      nextif ( defined($command) && $command eq "Connect" );
    }
  
    if( $type >= 2 ) {
     next if ( defined($info) && $info =~ m/^select/i );
     next if ( defined($info) && $info =~ m/^show/i );
    }
  
   push @threads, $ref;
  }
 return @threads;
}
  
sub main {
  if ($command eq "stop" ) {
    ##Gracefully killing connections on the current master
    #1. Set read_only= 1 on the new master
    #2. DROP USER so that no app user can establish new connections
    #3. Set read_only= 1 on the current master
    #4. Kill current queries
    #* Any database access failure will result in script die.
    my$exit_code = 1;
   eval {
     ## Setting read_only=1 on the new master (to avoid accident)
     my $new_master_handler = new MHA::DBHelper();
  
     # args: hostname, port, user, password, raise_error(die_on_error)_or_not
     $new_master_handler->connect( $new_master_ip, $new_master_port,
       $new_master_user, $new_master_password, 1 );
     print current_time_us() . " Set read_only on the new master..";
     $new_master_handler->enable_read_only();
     if ( $new_master_handler->is_read_only() ) {
       print "ok.\n";
     }
     else {
       die "Failed!\n";
     }
     $new_master_handler->disconnect();
  
     # Connecting to the orig master, die if any database error happens
     my $orig_master_handler = new MHA::DBHelper();
     $orig_master_handler->connect( $orig_master_ip, $orig_master_port,
       $orig_master_user, $orig_master_password, 1 );
  
      ## Drop application user so that nobodycan connect. Disabling per-session binlog beforehand
     #$orig_master_handler->disable_log_bin_local();
     #print current_time_us() . " Drpping app user on the origmaster..\n";
     #FIXME_xxx_drop_app_user($orig_master_handler);
  
     ## Waiting for N * 100 milliseconds so that current connections can exit
     my $time_until_read_only = 15;
     $_tstart = [gettimeofday];
     my @threads = get_threads_util( $orig_master_handler->{dbh},
       $orig_master_handler->{connection_id} );
     while ( $time_until_read_only > 0 && $#threads >= 0 ) {
       if ( $time_until_read_only % 5 == 0 ) {
         printf "%s Waiting all running %d threads aredisconnected.. (max %d milliseconds)\n",
           current_time_us(), $#threads + 1, $time_until_read_only * 100;
         if ( $#threads < 5 ) {
           print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump ."\n"
             foreach (@threads);
         }
       }
       sleep_until();
       $_tstart = [gettimeofday];
       $time_until_read_only--;
       @threads = get_threads_util( $orig_master_handler->{dbh},
         $orig_master_handler->{connection_id} );
     }
  
     ## Setting read_only=1 on the current master so that nobody(exceptSUPER) can write
     print current_time_us() . " Set read_only=1 on the orig master..";
     $orig_master_handler->enable_read_only();
     if ( $orig_master_handler->is_read_only() ) {
       print "ok.\n";
     }
     else {
       die "Failed!\n";
     }
  
     ## Waiting for M * 100 milliseconds so that current update queries cancomplete
     my $time_until_kill_threads = 5;
     @threads = get_threads_util( $orig_master_handler->{dbh},
       $orig_master_handler->{connection_id} );
     while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {
       if ( $time_until_kill_threads % 5 == 0 ) {
         printf "%s Waiting all running %d queries aredisconnected.. (max %d milliseconds)\n",
           current_time_us(), $#threads + 1, $time_until_kill_threads * 100;
         if ( $#threads < 5 ) {
           print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump ."\n"
             foreach (@threads);
         }
       }
       sleep_until();
       $_tstart = [gettimeofday];
       $time_until_kill_threads--;
       @threads = get_threads_util( $orig_master_handler->{dbh},
         $orig_master_handler->{connection_id} );
     }
  
                print "Disabling the VIPon old master: $orig_master_host \n";
                &stop_vip();    
  
     ## Terminating all threads
     print current_time_us() . " Killing all applicationthreads..\n";
     $orig_master_handler->kill_threads(@threads) if ( $#threads >= 0);
     print current_time_us() . " done.\n";
     #$orig_master_handler->enable_log_bin_local();
     $orig_master_handler->disconnect();
  
     ## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK
     $exit_code = 0;
    };
    if($@) {
     warn "Got Error: $@\n";
     exit $exit_code;
    }
   exit $exit_code;
  }
 elsif ( $command eq "start" ) {
    ##Activating master ip on the new master
    #1. Create app user with write privileges
    #2. Moving backup script if needed
    #3. Register new master's ip to the catalog database
  
# We don't return error even thoughactivating updatable accounts/ip failed so that we don't interrupt slaves'recovery.
# If exit code is 0 or 10, MHA does notabort
    my$exit_code = 10;
    eval{
     my $new_master_handler = new MHA::DBHelper();
  
     # args: hostname, port, user, password, raise_error_or_not
     $new_master_handler->connect( $new_master_ip, $new_master_port,
       $new_master_user, $new_master_password, 1 );
  
     ## Set read_only=0 on the new master
     #$new_master_handler->disable_log_bin_local();
     print current_time_us() . " Set read_only=0 on the newmaster.\n";
     $new_master_handler->disable_read_only();
  
     ## Creating an app user on the new master
     #print current_time_us() . " Creating app user on the newmaster..\n";
     #FIXME_xxx_create_app_user($new_master_handler);
     #$new_master_handler->enable_log_bin_local();
     $new_master_handler->disconnect();
  
     ## Update master ip on the catalog database, etc
                print "Enabling the VIP -$vip on the new master - $new_master_host \n";
                &start_vip();
                $exit_code = 0;
    };
    if($@) {
     warn "Got Error: $@\n";
     exit $exit_code;
    }
   exit $exit_code;
  }
 elsif ( $command eq "status" ) {
  
    #do nothing
   exit 0;
  }
  else{
   &usage();
   exit 1;
  }
}
  
# A simple system call that enable the VIPon the new master
sub start_vip() {
   `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIPon the old_master
sub stop_vip() {
   `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
  
sub usage {
 print
"Usage: master_ip_online_change --command=start|stop|status--orig_master_host=host --orig_master_ip=ip --orig_master_port=port--new_master_host=host --new_master_ip=ip --new_master_port=port\n";
  die;
}

授权

chmod u+x /usr/bin/master_ip_online_change

检查MHA配置

检查SSH配置

在hdp1上用root用户操作。

[root@10-31-1-119 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Mon Aug 24 17:47:04 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Aug 24 17:47:04 2020 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Mon Aug 24 17:47:04 2020 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Mon Aug 24 17:47:04 2020 - [info] Starting SSH connection tests..
Mon Aug 24 17:47:06 2020 - [debug] 
Mon Aug 24 17:47:04 2020 - [debug]  Connecting via SSH from root@10.31.1.120(10.31.1.120:22) to root@10.31.1.121(10.31.1.121:22)..
Mon Aug 24 17:47:05 2020 - [debug]   ok.
Mon Aug 24 17:47:05 2020 - [debug]  Connecting via SSH from root@10.31.1.120(10.31.1.120:22) to root@10.31.1.122(10.31.1.122:22)..
Mon Aug 24 17:47:06 2020 - [debug]   ok.
Mon Aug 24 17:47:07 2020 - [debug] 
Mon Aug 24 17:47:05 2020 - [debug]  Connecting via SSH from root@10.31.1.122(10.31.1.122:22) to root@10.31.1.120(10.31.1.120:22)..
Mon Aug 24 17:47:06 2020 - [debug]   ok.
Mon Aug 24 17:47:06 2020 - [debug]  Connecting via SSH from root@10.31.1.122(10.31.1.122:22) to root@10.31.1.121(10.31.1.121:22)..
Mon Aug 24 17:47:07 2020 - [debug]   ok.
Mon Aug 24 17:47:07 2020 - [debug] 
Mon Aug 24 17:47:05 2020 - [debug]  Connecting via SSH from root@10.31.1.121(10.31.1.121:22) to root@10.31.1.120(10.31.1.120:22)..
Mon Aug 24 17:47:06 2020 - [debug]   ok.
Mon Aug 24 17:47:06 2020 - [debug]  Connecting via SSH from root@10.31.1.121(10.31.1.121:22) to root@10.31.1.122(10.31.1.122:22)..
Mon Aug 24 17:47:06 2020 - [debug]   ok.
Mon Aug 24 17:47:07 2020 - [info] All SSH connection tests passed successfully.

检查整个复制环境状况

hp2上授权

create user 'root'@'10.31.1.%' identified by 'abc123';
grant all privileges on *.* to 'root'@'10.31.1.%';
flush privileges;

在hdp1上用root用户操作。

[root@10-31-1-119 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Mon Aug 24 17:57:22 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Aug 24 17:57:22 2020 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Mon Aug 24 17:57:22 2020 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Mon Aug 24 17:57:22 2020 - [info] MHA::MasterMonitor version 0.56.
Mon Aug 24 17:57:23 2020 - [info] GTID failover mode = 0
Mon Aug 24 17:57:23 2020 - [info] Dead Servers:
Mon Aug 24 17:57:23 2020 - [info] Alive Servers:
Mon Aug 24 17:57:23 2020 - [info]   10.31.1.120(10.31.1.120:3306)
Mon Aug 24 17:57:23 2020 - [info]   10.31.1.121(10.31.1.121:3306)
Mon Aug 24 17:57:23 2020 - [info]   10.31.1.122(10.31.1.122:3306)
Mon Aug 24 17:57:23 2020 - [info] Alive Slaves:
Mon Aug 24 17:57:23 2020 - [info]   10.31.1.121(10.31.1.121:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Aug 24 17:57:23 2020 - [info]     Replicating from 10.31.1.120(10.31.1.120:3306)
Mon Aug 24 17:57:23 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Aug 24 17:57:23 2020 - [info]   10.31.1.122(10.31.1.122:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Aug 24 17:57:23 2020 - [info]     Replicating from 10.31.1.120(10.31.1.120:3306)
Mon Aug 24 17:57:23 2020 - [info] Current Alive Master: 10.31.1.120(10.31.1.120:3306)
Mon Aug 24 17:57:23 2020 - [info] Checking slave configurations..
Mon Aug 24 17:57:23 2020 - [info]  read_only=1 is not set on slave 10.31.1.121(10.31.1.121:3306).
Mon Aug 24 17:57:23 2020 - [info]  read_only=1 is not set on slave 10.31.1.122(10.31.1.122:3306).
Mon Aug 24 17:57:23 2020 - [info] Checking replication filtering settings..
Mon Aug 24 17:57:23 2020 - [info]  binlog_do_db= , binlog_ignore_db= 
Mon Aug 24 17:57:23 2020 - [info]  Replication filtering check ok.
Mon Aug 24 17:57:23 2020 - [info] GTID (with auto-pos) is not supported
Mon Aug 24 17:57:23 2020 - [info] Starting SSH connection tests..
Mon Aug 24 17:57:26 2020 - [info] All SSH connection tests passed successfully.
Mon Aug 24 17:57:26 2020 - [info] Checking MHA Node version..
Mon Aug 24 17:57:27 2020 - [info]  Version check ok.
Mon Aug 24 17:57:27 2020 - [info] Checking SSH publickey authentication settings on the current master..
Mon Aug 24 17:57:27 2020 - [info] HealthCheck: SSH to 10.31.1.120 is reachable.
Mon Aug 24 17:57:27 2020 - [info] Master MHA Node version is 0.56.
Mon Aug 24 17:57:27 2020 - [info] Checking recovery script configurations on 10.31.1.120(10.31.1.120:3306)..
Mon Aug 24 17:57:27 2020 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=10-31-1-120-bin.000001 
Mon Aug 24 17:57:27 2020 - [info]   Connecting to root@10.31.1.120(10.31.1.120:22).. 
  Creating /tmp if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /var/lib/mysql, up to 10-31-1-120-bin.000001
Mon Aug 24 17:57:28 2020 - [info] Binlog setting check done.
Mon Aug 24 17:57:28 2020 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Mon Aug 24 17:57:28 2020 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.31.1.121 --slave_ip=10.31.1.121 --slave_port=3306 --workdir=/tmp --target_version=5.7.31-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Mon Aug 24 17:57:28 2020 - [info]   Connecting to root@10.31.1.121(10.31.1.121:22).. 
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to 10-31-1-121-relay-bin.000005
    Temporary relay log file is /var/lib/mysql/10-31-1-121-relay-bin.000005
    Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Mon Aug 24 17:57:28 2020 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=10.31.1.122 --slave_ip=10.31.1.122 --slave_port=3306 --workdir=/tmp --target_version=5.7.31-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Mon Aug 24 17:57:28 2020 - [info]   Connecting to root@10.31.1.122(10.31.1.122:22).. 
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to 10-31-1-122-relay-bin.000005
    Temporary relay log file is /var/lib/mysql/10-31-1-122-relay-bin.000005
    Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Mon Aug 24 17:57:29 2020 - [info] Slaves settings check done.
Mon Aug 24 17:57:29 2020 - [info] 
10.31.1.120(10.31.1.120:3306) (current master)
 +--10.31.1.121(10.31.1.121:3306)
 +--10.31.1.122(10.31.1.122:3306)

Mon Aug 24 17:57:29 2020 - [info] Checking replication health on 10.31.1.121..
Mon Aug 24 17:57:29 2020 - [info]  ok.
Mon Aug 24 17:57:29 2020 - [info] Checking replication health on 10.31.1.122..
Mon Aug 24 17:57:29 2020 - [info]  ok.
Mon Aug 24 17:57:29 2020 - [info] Checking master_ip_failover_script status:
Mon Aug 24 17:57:29 2020 - [info]   /usr/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=10.31.1.120 --orig_master_ip=10.31.1.120 --orig_master_port=3306 


IN SCRIPT TEST====/sbin/ifconfig ens192:1 down==/sbin/ifconfig ens192:1 10.31.1.241/24===

Checking the Status of the script.. OK 
Mon Aug 24 17:57:29 2020 - [info]  OK.
Mon Aug 24 17:57:29 2020 - [warning] shutdown_script is not defined.
Mon Aug 24 17:57:29 2020 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.
[root@10-31-1-119 ~]#

检查MHA Manager的状态

在hdp1上用root用户操作。

[root@10-31-1-119 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 is stopped(2:NOT_RUNNING).
[root@10-31-1-119 ~]#

显示"NOT_RUNNING"，这代表MHA监控没有开启。执行下面的命令后台启动MHA。

mkdir -p  /var/log/masterha/app1/
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &

启动参数说明：
remove_dead_master_conf：该参数代表当发生主从切换后，老的主库的ip将会从配置文件中移除。
manger_log：日志存放位置。
ignore_last_failover：在缺省情况下，如果MHA检测到连续发生宕机，且两次宕机间隔不足8小时的话，则不会进行Failover，之所以这样限制是为了避免ping-pong效应。该参数代表忽略上次MHA触发切换产生的文件，默认情况下，MHA发生切换后会在日志目录，也就是上面设置的/data产生app1.failover.complete文件，下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件。为了方便，这里设置为–ignore_last_failover。

再次检查MHA Manager的状态：
 

[root@10-31-1-119 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 monitoring program is now on initialization phase(10:INITIALIZING_MONITOR). Wait for a while and try checking again.
[root@10-31-1-119 ~]#

查看启动日志

在hdp1上用root用户操作。

[root@10-31-1-119 ~]# tail -n20 /var/log/masterha/app1/manager.log
    Cleaning up test file(s).. done.
Mon Aug 24 18:06:00 2020 - [info] Slaves settings check done.
Mon Aug 24 18:06:00 2020 - [info] 
10.31.1.120(10.31.1.120:3306) (current master)
 +--10.31.1.121(10.31.1.121:3306)
 +--10.31.1.122(10.31.1.122:3306)

Mon Aug 24 18:06:00 2020 - [info] Checking master_ip_failover_script status:
Mon Aug 24 18:06:00 2020 - [info]   /usr/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=10.31.1.120 --orig_master_ip=10.31.1.120 --orig_master_port=3306 


IN SCRIPT TEST====/sbin/ifconfig eth1:1 down==/sbin/ifconfig eth1:1 10.31.1.241/24===

Checking the Status of the script.. OK 
Mon Aug 24 18:06:00 2020 - [info]  OK.
Mon Aug 24 18:06:00 2020 - [warning] shutdown_script is not defined.
Mon Aug 24 18:06:00 2020 - [info] Set master ping interval 1 seconds.
Mon Aug 24 18:06:00 2020 - [info] Set secondary check script: /usr/bin/masterha_secondary_check -s hdp2 -s hdp3 --user=root --master_host=hdp2 --master_ip=10.31.1.120 --master_port=3306
Mon Aug 24 18:06:00 2020 - [info] Starting ping health check on 10.31.1.120(10.31.1.120:3306)..
Mon Aug 24 18:06:00 2020 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

功能测试

初始绑定VIP

在hdp2 10.31.1.112（master）上用root用户执行：

/sbin/ifconfig ens192:1 10.31.1.241/24

执行记录

[root@10-31-1-120 src]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
[root@10-31-1-120 src]# 
[root@10-31-1-120 src]# 
[root@10-31-1-120 src]# /sbin/ifconfig ens192:1 10.31.1.241/24
[root@10-31-1-120 src]# 
[root@10-31-1-120 src]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 10.31.1.241/24 brd 10.31.1.255 scope global secondary ens192:1
       valid_lft forever preferred_lft forever
    inet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
[root@10-31-1-120 src]#

也可以手工解除绑定

[root@10-31-1-120 keepalived-2.0.15]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 10.31.1.241/24 brd 10.31.1.255 scope global secondary ens192:1
       valid_lft forever preferred_lft forever
    inet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
[root@10-31-1-120 keepalived-2.0.15]# 
[root@10-31-1-120 keepalived-2.0.15]# 
[root@10-31-1-120 keepalived-2.0.15]# ifconfig ens192:10.31.1.241/24 down
SIOCSIFFLAGS: 无法指定被请求的地址
[root@10-31-1-120 keepalived-2.0.15]# 
[root@10-31-1-120 keepalived-2.0.15]# 
[root@10-31-1-120 keepalived-2.0.15]# ifconfig ens192:1 down             
[root@10-31-1-120 keepalived-2.0.15]# 
[root@10-31-1-120 keepalived-2.0.15]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
[root@10-31-1-120 keepalived-2.0.15]#

测试自动切换

通过vip登陆mysql

[root@uattest ~]# mysql -uroot -p -h10.31.1.241
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 27
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

停止主库hp1的mysql服务器

service mysqld stop
ip addr

[root@10-31-1-120 src]# service mysqld stop
Redirecting to /bin/systemctl stop mysqld.service
[root@10-31-1-120 src]# 
[root@10-31-1-120 src]# 
[root@10-31-1-120 src]# 
-- 可以看到  10.31.1.241的vip已经不再hp1(master)上了
[root@10-31-1-120 src]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff

hp2和hp3查看

hp2

-- 可以看到 10.31.1.241这个vip飘到hp2上了
[root@10-31-1-121 src]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:e1:fe:b4 brd ff:ff:ff:ff:ff:ff
    inet 10.31.1.121/24 brd 10.31.1.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 10.31.1.241/24 brd 10.255.255.255 scope global ens192:1
       valid_lft forever preferred_lft forever
    inet6 fe80::9470:f61e:6e0e:48e4/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::1aa1:d23e:e28a:fb62/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::f974:3f03:d1f0:1672/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff

-- hp2已经由从库变为主库了
[root@10-31-1-121 src]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 53
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 
mysql> show slave status\G
Empty set (0.00 sec)

mysql> show master status\G
*************************** 1. row ***************************
             File: 10-31-1-121-bin.000001
         Position: 507
     Binlog_Do_DB: 
 Binlog_Ignore_DB: 
Executed_Gtid_Set: 
1 row in set (0.00 sec)

hp3

hp3变为了hp2的从库
[root@10-31-1-122 bin]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 48
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.31.1.121
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: 10-31-1-121-bin.000001
          Read_Master_Log_Pos: 507
               Relay_Log_File: 10-31-1-122-relay-bin.000002
                Relay_Log_Pos: 679
        Relay_Master_Log_File: 10-31-1-121-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 507
              Relay_Log_Space: 892
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 102
                  Master_UUID: 5ad1f5aa-e2cb-11ea-ba89-000c29e1feb4
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
1 row in set (0.00 sec)

还原环境

还原数据库复制：

-- 在hdp2、hdp3、hdp4上重置master、slave
stop slave;
reset master;
reset slave all;
 
-- 在hdp3、hdp4上重新指向hdp2为master
change master to
master_host='10.31.1.120',
master_port=3306,
master_user='repl',
master_password='abc123',
master_log_file='10-31-1-120-bin.000001',
master_log_pos=154;
 
start slave;
show slave status\G

还原VIP绑定：

# 在hdp3上用root用户执行
/sbin/ifconfig ens192:1 down
 
# 在hdp2上用root用户执行
/sbin/ifconfig ens192:1 10.31.1.241/24

还原配置文件：
编辑在hdp1上/etc/masterha/app1.cnf，将[server1]段添加回去。

启动MHA Manage：

# 在hdp1上用root用户执行
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &

至此环境还原完毕。

测试手工切换

停止MHA Manage

在hp1用root用户操作

masterha_stop --conf=/etc/masterha/app1.cnf

关闭master

在hdp2上用root用户操作。

service mysqld stop

执行手工切换

在hdp1上用root用户操作。

masterha_master_switch --master_state=dead --conf=/etc/masterha/app1.cnf --dead_master_host=10.31.1.120 --dead_master_port=3306 --new_master_host=10.31.1.121 --new_master_port=3306 --ignore_last_failover

执行记录

[root@10-31-1-119 ~]# masterha_master_switch --master_state=dead --conf=/etc/masterha/app1.cnf --dead_master_host=10.31.1.120 --dead_master_port=3306 --new_master_host=10.31.1.121 --new_master_port=3306 --ignore_last_failover
--dead_master_ip=<dead_master_ip> is not set. Using 10.31.1.120.
Tue Aug 25 16:18:55 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Aug 25 16:18:55 2020 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Tue Aug 25 16:18:55 2020 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Tue Aug 25 16:18:55 2020 - [info] MHA::MasterFailover version 0.56.
Tue Aug 25 16:18:55 2020 - [info] Starting master failover.
Tue Aug 25 16:18:55 2020 - [info] 
Tue Aug 25 16:18:55 2020 - [info] * Phase 1: Configuration Check Phase..
Tue Aug 25 16:18:55 2020 - [info] 
Tue Aug 25 16:18:56 2020 - [info] GTID failover mode = 0
Tue Aug 25 16:18:56 2020 - [info] Dead Servers:
Tue Aug 25 16:18:56 2020 - [info]   10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:18:56 2020 - [info] Checking master reachability via MySQL(double check)...
Tue Aug 25 16:18:56 2020 - [info]  ok.
Tue Aug 25 16:18:56 2020 - [info] Alive Servers:
Tue Aug 25 16:18:56 2020 - [info]   10.31.1.121(10.31.1.121:3306)
Tue Aug 25 16:18:56 2020 - [info]   10.31.1.122(10.31.1.122:3306)
Tue Aug 25 16:18:56 2020 - [info] Alive Slaves:
Tue Aug 25 16:18:56 2020 - [info]   10.31.1.121(10.31.1.121:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:18:56 2020 - [info]     Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:18:56 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Aug 25 16:18:56 2020 - [info]   10.31.1.122(10.31.1.122:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:18:56 2020 - [info]     Replicating from 10.31.1.120(10.31.1.120:3306)
Master 10.31.1.120(10.31.1.120:3306) is dead. Proceed? (yes/NO): yes
Tue Aug 25 16:19:01 2020 - [info] Starting Non-GTID based failover.
Tue Aug 25 16:19:01 2020 - [info] 
Tue Aug 25 16:19:01 2020 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Aug 25 16:19:01 2020 - [info] 
Tue Aug 25 16:19:01 2020 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Aug 25 16:19:01 2020 - [info] 
Tue Aug 25 16:19:01 2020 - [info] HealthCheck: SSH to 10.31.1.120 is reachable.
Tue Aug 25 16:19:01 2020 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Aug 25 16:19:01 2020 - [info] Executing master IP deactivation script:
Tue Aug 25 16:19:01 2020 - [info]   /usr/bin/master_ip_failover --orig_master_host=10.31.1.120 --orig_master_ip=10.31.1.120 --orig_master_port=3306 --command=stopssh --ssh_user=root  


IN SCRIPT TEST====/sbin/ifconfig ens192:1 down==/sbin/ifconfig ens192:1 10.31.1.241/24===

Disabling the VIP on old master: 10.31.1.120 
Tue Aug 25 16:19:02 2020 - [info]  done.
Tue Aug 25 16:19:02 2020 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Aug 25 16:19:02 2020 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Aug 25 16:19:02 2020 - [info] 
Tue Aug 25 16:19:02 2020 - [info] * Phase 3: Master Recovery Phase..
Tue Aug 25 16:19:02 2020 - [info] 
Tue Aug 25 16:19:02 2020 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Aug 25 16:19:02 2020 - [info] 
Tue Aug 25 16:19:02 2020 - [info] The latest binary log file/position on all slaves is 10-31-1-120-bin.000001:154
Tue Aug 25 16:19:02 2020 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Aug 25 16:19:02 2020 - [info]   10.31.1.121(10.31.1.121:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:19:02 2020 - [info]     Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:19:02 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Aug 25 16:19:02 2020 - [info]   10.31.1.122(10.31.1.122:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:19:02 2020 - [info]     Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:19:02 2020 - [info] The oldest binary log file/position on all slaves is 10-31-1-120-bin.000001:154
Tue Aug 25 16:19:02 2020 - [info] Oldest slaves:
Tue Aug 25 16:19:02 2020 - [info]   10.31.1.121(10.31.1.121:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:19:02 2020 - [info]     Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:19:02 2020 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Aug 25 16:19:02 2020 - [info]   10.31.1.122(10.31.1.122:3306)  Version=5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:19:02 2020 - [info]     Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:19:02 2020 - [info] 
Tue Aug 25 16:19:02 2020 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Tue Aug 25 16:19:02 2020 - [info] 
Tue Aug 25 16:19:02 2020 - [info] Fetching dead master's binary logs..
Tue Aug 25 16:19:02 2020 - [info] Executing command on the dead master 10.31.1.120(10.31.1.120:3306): save_binary_logs --command=save --start_file=10-31-1-120-bin.000001  --start_pos=154 --binlog_dir=/var/lib/mysql --output_file=/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56
  Creating /tmp if not exists..    ok.
 Concat binary/relay logs from 10-31-1-120-bin.000001 pos 154 to 10-31-1-120-bin.000001 EOF into /tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog ..
 Binlog Checksum enabled
  Dumping binlog format description event, from position 0 to 154.. ok.
  Dumping effective binlog data from /var/lib/mysql/10-31-1-120-bin.000001 position 154 to tail(177).. ok.
 Binlog Checksum enabled
 Concat succeeded.
saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog                                                                                                           100%  177   140.4KB/s   00:00    
Tue Aug 25 16:19:03 2020 - [info] scp from root@10.31.1.120:/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog to local:/var/log/masterha/app1.log/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog succeeded.
Tue Aug 25 16:19:04 2020 - [info] HealthCheck: SSH to 10.31.1.121 is reachable.
Tue Aug 25 16:19:05 2020 - [info] HealthCheck: SSH to 10.31.1.122 is reachable.
Tue Aug 25 16:19:05 2020 - [info] 
Tue Aug 25 16:19:05 2020 - [info] * Phase 3.3: Determining New Master Phase..
Tue Aug 25 16:19:05 2020 - [info] 
Tue Aug 25 16:19:05 2020 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Tue Aug 25 16:19:05 2020 - [info] All slaves received relay logs to the same position. No need to resync each other.
Tue Aug 25 16:19:05 2020 - [info] 10.31.1.121 can be new master.
Tue Aug 25 16:19:05 2020 - [info] New master is 10.31.1.121(10.31.1.121:3306)
Tue Aug 25 16:19:05 2020 - [info] Starting master failover..
Tue Aug 25 16:19:05 2020 - [info] 
From:
10.31.1.120(10.31.1.120:3306) (current master)
 +--10.31.1.121(10.31.1.121:3306)
 +--10.31.1.122(10.31.1.122:3306)

To:
10.31.1.121(10.31.1.121:3306) (new master)
 +--10.31.1.122(10.31.1.122:3306)

Starting master switch from 10.31.1.120(10.31.1.120:3306) to 10.31.1.121(10.31.1.121:3306)? (yes/NO): yes
Tue Aug 25 16:19:09 2020 - [info] New master decided manually is 10.31.1.121(10.31.1.121:3306)
Tue Aug 25 16:19:09 2020 - [info] 
Tue Aug 25 16:19:09 2020 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Tue Aug 25 16:19:09 2020 - [info] 
Tue Aug 25 16:19:09 2020 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Tue Aug 25 16:19:09 2020 - [info] Sending binlog..
saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog                                                                                                           100%  177   176.2KB/s   00:00    
Tue Aug 25 16:19:10 2020 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog to root@10.31.1.121:/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog succeeded.
Tue Aug 25 16:19:10 2020 - [info] 
Tue Aug 25 16:19:10 2020 - [info] * Phase 3.4: Master Log Apply Phase..
Tue Aug 25 16:19:10 2020 - [info] 
Tue Aug 25 16:19:10 2020 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Tue Aug 25 16:19:10 2020 - [info] Starting recovery on 10.31.1.121(10.31.1.121:3306)..
Tue Aug 25 16:19:10 2020 - [info]  Generating diffs succeeded.
Tue Aug 25 16:19:10 2020 - [info] Waiting until all relay logs are applied.
Tue Aug 25 16:19:10 2020 - [info]  done.
Tue Aug 25 16:19:10 2020 - [info] Getting slave status..
Tue Aug 25 16:19:10 2020 - [info] This slave(10.31.1.121)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(10-31-1-120-bin.000001:154). No need to recover from Exec_Master_Log_Pos.
Tue Aug 25 16:19:10 2020 - [info] Connecting to the target slave host 10.31.1.121, running recover script..
Tue Aug 25 16:19:10 2020 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=10.31.1.121 --slave_ip=10.31.1.121  --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog --workdir=/tmp --target_version=5.7.31-log --timestamp=20200825161855 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Tue Aug 25 16:19:10 2020 - [info] 
MySQL client version is 5.7.31. Using --binary-mode.
Applying differential binary/relay log files /tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog on 10.31.1.121:3306. This may take long time...
Applying log files succeeded.
Tue Aug 25 16:19:10 2020 - [info]  All relay logs were successfully applied.
Tue Aug 25 16:19:10 2020 - [info] Getting new master's binlog name and position..
Tue Aug 25 16:19:10 2020 - [info]  10-31-1-121-bin.000001:154
Tue Aug 25 16:19:10 2020 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.31.1.121', MASTER_PORT=3306, MASTER_LOG_FILE='10-31-1-121-bin.000001', MASTER_LOG_POS=154, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Tue Aug 25 16:19:10 2020 - [info] Executing master IP activate script:
Tue Aug 25 16:19:10 2020 - [info]   /usr/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=10.31.1.120 --orig_master_ip=10.31.1.120 --orig_master_port=3306 --new_master_host=10.31.1.121 --new_master_ip=10.31.1.121 --new_master_port=3306 --new_master_user='root' --new_master_password='abc123'  
Unknown option: new_master_user
Unknown option: new_master_password


IN SCRIPT TEST====/sbin/ifconfig ens192:1 down==/sbin/ifconfig ens192:1 10.31.1.241/24===

Enabling the VIP - 10.31.1.241 on the new master - 10.31.1.121 
Tue Aug 25 16:19:10 2020 - [info]  OK.
Tue Aug 25 16:19:10 2020 - [info] ** Finished master recovery successfully.
Tue Aug 25 16:19:10 2020 - [info] * Phase 3: Master Recovery Phase completed.
Tue Aug 25 16:19:10 2020 - [info] 
Tue Aug 25 16:19:10 2020 - [info] * Phase 4: Slaves Recovery Phase..
Tue Aug 25 16:19:10 2020 - [info] 
Tue Aug 25 16:19:10 2020 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Tue Aug 25 16:19:10 2020 - [info] 
Tue Aug 25 16:19:10 2020 - [info] -- Slave diff file generation on host 10.31.1.122(10.31.1.122:3306) started, pid: 14489. Check tmp log /var/log/masterha/app1.log/10.31.1.122_3306_20200825161855.log if it takes time..
Tue Aug 25 16:19:11 2020 - [info] 
Tue Aug 25 16:19:11 2020 - [info] Log messages from 10.31.1.122 ...
Tue Aug 25 16:19:11 2020 - [info] 
Tue Aug 25 16:19:10 2020 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Tue Aug 25 16:19:11 2020 - [info] End of log messages from 10.31.1.122.
Tue Aug 25 16:19:11 2020 - [info] -- 10.31.1.122(10.31.1.122:3306) has the latest relay log events.
Tue Aug 25 16:19:11 2020 - [info] Generating relay diff files from the latest slave succeeded.
Tue Aug 25 16:19:11 2020 - [info] 
Tue Aug 25 16:19:11 2020 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Tue Aug 25 16:19:11 2020 - [info] 
Tue Aug 25 16:19:11 2020 - [info] -- Slave recovery on host 10.31.1.122(10.31.1.122:3306) started, pid: 14491. Check tmp log /var/log/masterha/app1.log/10.31.1.122_3306_20200825161855.log if it takes time..
saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog                                                                                                           100%  177   170.8KB/s   00:00    
Tue Aug 25 16:19:13 2020 - [info] 
Tue Aug 25 16:19:13 2020 - [info] Log messages from 10.31.1.122 ...
Tue Aug 25 16:19:13 2020 - [info] 
Tue Aug 25 16:19:11 2020 - [info] Sending binlog..
Tue Aug 25 16:19:12 2020 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog to root@10.31.1.122:/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog succeeded.
Tue Aug 25 16:19:12 2020 - [info] Starting recovery on 10.31.1.122(10.31.1.122:3306)..
Tue Aug 25 16:19:12 2020 - [info]  Generating diffs succeeded.
Tue Aug 25 16:19:12 2020 - [info] Waiting until all relay logs are applied.
Tue Aug 25 16:19:12 2020 - [info]  done.
Tue Aug 25 16:19:12 2020 - [info] Getting slave status..
Tue Aug 25 16:19:12 2020 - [info] This slave(10.31.1.122)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(10-31-1-120-bin.000001:154). No need to recover from Exec_Master_Log_Pos.
Tue Aug 25 16:19:12 2020 - [info] Connecting to the target slave host 10.31.1.122, running recover script..
Tue Aug 25 16:19:12 2020 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=10.31.1.122 --slave_ip=10.31.1.122  --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog --workdir=/tmp --target_version=5.7.31-log --timestamp=20200825161855 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Tue Aug 25 16:19:13 2020 - [info] 
MySQL client version is 5.7.31. Using --binary-mode.
Applying differential binary/relay log files /tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog on 10.31.1.122:3306. This may take long time...
Applying log files succeeded.
Tue Aug 25 16:19:13 2020 - [info]  All relay logs were successfully applied.
Tue Aug 25 16:19:13 2020 - [info]  Resetting slave 10.31.1.122(10.31.1.122:3306) and starting replication from the new master 10.31.1.121(10.31.1.121:3306)..
Tue Aug 25 16:19:13 2020 - [info]  Executed CHANGE MASTER.
Tue Aug 25 16:19:13 2020 - [info]  Slave started.
Tue Aug 25 16:19:13 2020 - [info] End of log messages from 10.31.1.122.
Tue Aug 25 16:19:13 2020 - [info] -- Slave recovery on host 10.31.1.122(10.31.1.122:3306) succeeded.
Tue Aug 25 16:19:13 2020 - [info] All new slave servers recovered successfully.
Tue Aug 25 16:19:13 2020 - [info] 
Tue Aug 25 16:19:13 2020 - [info] * Phase 5: New master cleanup phase..
Tue Aug 25 16:19:13 2020 - [info] 
Tue Aug 25 16:19:13 2020 - [info] Resetting slave info on the new master..
Tue Aug 25 16:19:13 2020 - [info]  10.31.1.121: Resetting slave info succeeded.
Tue Aug 25 16:19:13 2020 - [info] Master failover to 10.31.1.121(10.31.1.121:3306) completed successfully.
Tue Aug 25 16:19:13 2020 - [info] 

----- Failover Report -----

app1: MySQL Master failover 10.31.1.120(10.31.1.120:3306) to 10.31.1.121(10.31.1.121:3306) succeeded

Master 10.31.1.120(10.31.1.120:3306) is down!

Check MHA Manager logs at 10-31-1-119 for details.

Started manual(interactive) failover.
Invalidated master IP address on 10.31.1.120(10.31.1.120:3306)
The latest slave 10.31.1.121(10.31.1.121:3306) has all relay logs for recovery.
Selected 10.31.1.121(10.31.1.121:3306) as a new master.
10.31.1.121(10.31.1.121:3306): OK: Applying all logs succeeded.
10.31.1.121(10.31.1.121:3306): OK: Activated master IP address.
10.31.1.122(10.31.1.122:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
10.31.1.122(10.31.1.122:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.31.1.121(10.31.1.121:3306)
10.31.1.121(10.31.1.121:3306): Resetting slave info succeeded.
Master failover to 10.31.1.121(10.31.1.121:3306) completed successfully.
[root@10-31-1-119 ~]#

验证VIP漂移到10.31.1.121

[root@10-31-1-121 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:e1:fe:b4 brd ff:ff:ff:ff:ff:ff
    inet 10.31.1.121/24 brd 10.31.1.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 10.31.1.241/24 brd 10.31.1.255 scope global secondary ens192:1
       valid_lft forever preferred_lft forever
    inet6 fe80::9470:f61e:6e0e:48e4/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::1aa1:d23e:e28a:fb62/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::f974:3f03:d1f0:1672/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff

验证复制关系

[root@10-31-1-122 bin]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 81
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.31.1.121
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: 10-31-1-121-bin.000001
          Read_Master_Log_Pos: 154
               Relay_Log_File: 10-31-1-122-relay-bin.000002
                Relay_Log_Pos: 326
        Relay_Master_Log_File: 10-31-1-121-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 154
              Relay_Log_Space: 539
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 102
                  Master_UUID: 5ad1f5aa-e2cb-11ea-ba89-000c29e1feb4
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
1 row in set (0.00 sec)

mysql>

验证客户端VIP访问

[root@uattest ~]# mysql -uroot -p -h10.31.1.241
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 99
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

mysql> 
mysql>

修复宕机的Master

通常情况下自动切换以后，原master可能已经废弃掉，待原master主机修复后，如果数据完整的情况下，可能想把原来master重新作为新主库的slave。这时我们可以借助当时自动切换时刻的MHA日志来完成对原master的修复。下面是提取相关日志的命令：

grep -i "All other slaves should start" /var/log/masterha/app1/manager.log

可以看到类似下面的信息：

All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.31.1.120', MASTER_PORT=3306, MASTER_LOG_FILE='10-31-1-121-bin.000005', MASTER_LOG_POS=120, MASTER_USER='repl', MASTER_PASSWORD='abc123';

意思是说，如果Master主机修复好了，可以在修复好后的Master执行CHANGE MASTER操作，作为新的slave库。

参考

MySQL MHA高可用实战_只是甲的博客-CSDN博客

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/342183.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

2023软考考哪个证书好?

2023软考考哪个证书好?

软考有三个级别&#xff08;初级&#xff0c;中级和高级&#xff09;&#xff0c;这三个级别分别对应5个方向&#xff0c;下面这张图片呢&#xff0c;可以一目了然&#xff0c;一些小小建议&#xff01;&#xff01;&#xff01;遵循一个原则&#xff1a;首先选专业对口的科目&…
阅读更多...
为什么这11道JVM面试题这么重要（附答案）

为什么这11道JVM面试题这么重要（附答案）

本文内容整理自 博学谷狂野架构师 运行时数据区都包含什么 虚拟机的基础面试题 程序计数器Java 虚拟机栈本地方法栈Java 堆方法区 程序计数器 程序计数器是线程私有的&#xff0c;并且是JVM中唯一不会溢出的区域&#xff0c;用来保存线程切换时的执行行数 程序计数器&#xff…
阅读更多...
算法笔记（三）—— 桶排序及排序总结

算法笔记（三）—— 桶排序及排序总结

堆 逻辑上是一棵完全二叉树&#xff08;依次遍满或者全满&#xff09;。 数组可以转为完全二叉树&#xff0c;完全二叉树某结点左孩子(2*i1)&#xff0c;右孩子(i*22)&#xff0c;父结点((i-1/)2)&#xff0c;根节点的父还是自己。 如何将数组转化为堆&#xff08;大根堆&…
阅读更多...
Web自动化测试——selenium篇（一）

Web自动化测试——selenium篇（一）

文章目录一、环境准备二、Web 自动化测试 Demo三、元素定位常用方法四、元素定位失败可能原因五、测试对象操作六、等待操作七、信息打印在学习 Web 自动化测试的过程中&#xff0c;selenium 是其中的常用工具。除了其开源免费&#xff0c;包含丰富的 API 以外&#xff0c;它还…
阅读更多...
基于“PLUS模型+“生态系统服务多情景模拟预测

基于“PLUS模型+“生态系统服务多情景模拟预测

工业革命以来&#xff0c;社会生产力迅速提高&#xff0c;人类活动频繁&#xff0c;此外人口与日俱增对土地的需求与改造更加强烈&#xff0c;人-地关系日益紧张。此外&#xff0c;土地资源的不合理开发利用更是造成了水土流失、植被退化、水资源短缺、区域气候变化、生物多样性…
阅读更多...
windows安装proget实现nuget私有包部署

windows安装proget实现nuget私有包部署

下载proget 官网 下载地址 免费下载 安装proget 下载完成之后双击安装 选择ProGet 默认选择即可 也可以指定数据库&#xff0c;SQL Server数据库 Server服务器名;Database数据库名;User Id用户名;Password密码 Serverlocalhost;DatabaseProGet2;User Idsa;Passwordxxxx…
阅读更多...
浅谈一下前端工作中全流程多层次的四款测试工具

浅谈一下前端工作中全流程多层次的四款测试工具

在应届生找工作的时候&#xff0c;我们经常会见到一条招聘要求&#xff1a;要求实习经历。或者 有实习经历者优先。 为什么大部分公司在招聘时&#xff0c;都要求你必须有实习经历&#xff1f; 商业项目与个人项目不同&#xff0c;一段实习经历&#xff0c;能够熟悉公司中成熟…
阅读更多...
最接近的三数之和-力扣16-java排序+双指针

最接近的三数之和-力扣16-java排序+双指针

一、题目描述给你一个长度为 n 的整数数组 nums 和 一个目标值 target。请你从 nums 中选出三个整数&#xff0c;使它们的和与 target 最接近。返回这三个数的和。假定每组输入只存在恰好一个解。示例 1&#xff1a;输入&#xff1a;nums [-1,2,1,-4], target 1输出&#xff…
阅读更多...
Multisim14 安装包及安装教程

Multisim14 安装包及安装教程

Multisim14 安装教程 Multisim14下载地址&#xff1a;Kevin的学习站–安装包下载地址 Multisim14 简介&#xff1a; Multisim 14 是美国国家仪器有限公司&#xff08;National Instrument&#xff0c;NI&#xff09;推出的以 Windows 为基础、符合工业标准的、具有 SPICE 最佳仿…
阅读更多...
三数之和-力扣15-java排序+双指针

三数之和-力扣15-java排序+双指针

一、题目描述给你一个整数数组 nums &#xff0c;判断是否存在三元组 [nums[i], nums[j], nums[k]] 满足 i ! j、i ! k 且 j ! k &#xff0c;同时还满足 nums[i] nums[j] nums[k] 0 。请你返回所有和为 0 且不重复的三元组。注意&#xff1a;答案中不可以包含重复的三元组。…
阅读更多...
好用的研发管理看板工具有哪些？10款主流看板管理软件盘点

好用的研发管理看板工具有哪些？10款主流看板管理软件盘点

10大企业看板工具软件&#xff1a;1.软件开发项目看板 PingCode&#xff1b;2.通用看板软件 Worktile&#xff1b;3.开源看板软件 Wekan&#xff1b;4.免费看板软件 Trello&#xff1b;5.个人和小团队的看板软件 Todoist &#xff1b;6.开源免费看 Kanboard&#xff1b;7.面向个…
阅读更多...
想考个PMP证书，怎么报考啊？

想考个PMP证书，怎么报考啊？

先说报名条件&#xff1a; PMP报名条件说难也不难&#xff0c;但也是有条件的&#xff0c;看下面的视频&#xff0c;最难满足的就是题主担心的工作经验&#xff0c;其实这个也并不一定要项目管理经验&#xff0c;年纪合适&#xff0c;一般来说也是可以考的。 PMP考试时间 23年…
阅读更多...
微服务实战--高级篇：分布式缓存 Redis

微服务实战--高级篇：分布式缓存 Redis

分布式缓存 – 基于Redis集群解决单机Redis存在的问题 单机的Redis存在四大问题&#xff1a; 1.Redis持久化 Redis有两种持久化方案&#xff1a; RDB持久化AOF持久化 1.1.RDB持久化 RDB全称Redis Database Backup file&#xff08;Redis数据备份文件&#xff09;&#xf…
阅读更多...
数据结构 - 优先级队列（堆）

数据结构 - 优先级队列（堆）

文章目录前言1.介绍优先级队列2. 认识堆3. 实现优先级队列3.1 了解优先级队列的构造方法&#xff1a;3.2 使用优先级队列解决问题&#xff1a;总结前言 本篇PriorityQueue优先级队列的介绍其底层是堆&#xff0c;关于堆的认识&#xff0c;使用优先级队列能解决的一些问题&…
阅读更多...
day12_Django的基本操作

day12_Django的基本操作

文章目录创建django项目项目-各个文件的含义app-各个文件的含义MVT运行&#xff0c;查看是否创建项目框架成功ORM迁移->在数据库中形成对应的 表通过类名 对象的方法完成数据库的增删查改操作通过模型类&#xff08;shell&#xff09;的方法操作数据表后台管理进入后台http:…
阅读更多...
dvwa靶场的搭建过程

dvwa靶场的搭建过程

dvwa靶场的搭建过程 记录一下自己重新开始学习web安全之路①。 首先一个要搭建一个网站需要的准备 服务器 – 操作系统 windows linux&#xff08;可以将电脑当成服务器&#xff09; 数据库 MySQL sql server Oracle access 、、、、作用&#xff1a;存储数据 中间件 apac…
阅读更多...
零基础学MySQL（五）-- 详细讲解数据库中的常用函数

零基础学MySQL（五）-- 详细讲解数据库中的常用函数

目录&#x1f387;一、聚合函数1️⃣count 函数&#xff08;1&#xff09;基本语法&#xff08;2&#xff09;基本练习&#xff08;3&#xff09;注意细节2️⃣sum 函数&#xff08;1&#xff09;基本语法&#xff08;2&#xff09;基本练习&#xff08;3&#xff09;注意细节3…
阅读更多...
opencv常用函数

opencv常用函数

1)读视频 img cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) if vc.isOpened():ret, frame vc.read() else:ret False while ret:#此处省略具体的操作ret, frame vc.read() # 读下一帧 vc.release() 2&#xff09;保存视频 def mk_video_writer(vc, path&#xff0c;frame_…
阅读更多...
MASA Stack 1.0 发布会讲稿——生态篇

MASA Stack 1.0 发布会讲稿——生态篇

2022年运营回顾 贡献者 首先感谢贡献者们为MASA Stack社区所作的积极贡献&#xff0c;这些贡献者给我们提出了很多宝贵的建议&#xff0c;更是积极的提交PR帮助我们一起让产品更健壮&#xff0c;更完善&#xff0c;还在各种场合推广我们的解决方案&#xff0c;非常给力&#x…
阅读更多...
智能工厂以MES系统为基础,实现＂信息化减人,自动化换人＂

智能工厂以MES系统为基础,实现＂信息化减人,自动化换人＂

MES是一种生产信息化的管理系统&#xff0c;它适用于制造业的车间实施层面。MES能够为企业提供生产数据、项目看板、库存、成本、工装、生产计划、计划排程、质量、人力资源、采购、生产过程控制、底层数据集成分析、上层数据集成分解等管理模块&#xff0c;为企业打造一个扎实…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023