此时主节点上的dml和ddl操作都会挂死,由于脑裂问题,无法判断谁是主谁是备,所以节点无法写操作。
此时需要手动介入处理:
mysql> show variables like '%group_replication_member_expel_timeout%';
+----------------------------------------+-------+
| Variable_name | Value |
+----------------------------------------+-------+
| group_replication_member_expel_timeout | 30 |
+----------------------------------------+-------+
1 row in set (0.01 sec)
mysql> show variables like '%group_replication_unreachable_majority_timeout%';
+------------------------------------------------+-------+
| Variable_name | Value |
+------------------------------------------------+-------+
| group_replication_unreachable_majority_timeout | 0 |
+------------------------------------------------+-------+
1 row in set (0.00 sec)
mysql> set GLOBAL group_replication_unreachable_majority_timeout=35;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like '%group_replication_unreachable_majority_timeout%';
+------------------------------------------------+-------+
| Variable_name | Value |
+------------------------------------------------+-------+
| group_replication_unreachable_majority_timeout | 35 |
+------------------------------------------------+-------+
1 row in set (0.01 sec)
mysql> SELECT @@group_replication_local_address;
+-----------------------------------+
| @@group_replication_local_address |
+-----------------------------------+
| 10.1.1.51:33071 |
+-----------------------------------+
1 row in set (0.00 sec)
mysql> SET GLOBAL group_replication_force_members="10.1.1.51:33071";
ERROR 4118 (HY000): The 'SET GLOBAL group_replication_force_members=10.1.1.51:33071' command encountered a failure. Timeout on wait for view after setting group_replication_force_members.
mysql>
mysql>
mysql> show variables like '%group_replication_force_members%';
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| group_replication_force_members | |
+---------------------------------+-------+
1 row in set (0.01 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 773123a4-daad-11ee-930b-3009f923fbf1 | 10.1.1.51 | 3307 | ONLINE | PRIMARY | 8.0.36 | XCom |
| group_replication_applier | b2b77622-daad-11ee-a732-3009f925119f | 10.1.1.52 | 3307 | UNREACHABLE | SECONDARY | 8.0.36 | XCom |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
2 rows in set (0.00 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 773123a4-daad-11ee-930b-3009f923fbf1 | 10.1.1.51 | 3307 | ONLINE | PRIMARY | 8.0.36 | XCom |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
1 row in set (0.00 sec)
mysql> show variables like '%group_replication_force_members%';
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| group_replication_force_members | |
+---------------------------------+-------+
1 row in set (0.01 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 773123a4-daad-11ee-930b-3009f923fbf1 | 10.1.1.51 | 3307 | ONLINE | PRIMARY | 8.0.36 | XCom |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
1 row in set (0.00 sec)
mysql> show variables like '%group_replication_force_members%';
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| group_replication_force_members | |
+---------------------------------+-------+
1 row in set (0.00 sec)
mysql> show processlist;
+--------+-----------------+---------------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| 7 | event_scheduler | localhost | NULL | Daemon | 44294 | Waiting for next activation | NULL |
| 15 | system user | | NULL | Connect | 1452183 | waiting for handler commit | Group replication applier module |
| 17 | system user | | NULL | Query | 1452178 | Replica has read all relay log; waiting for more updates | NULL |
| 18 | system user | | NULL | Connect | 1452183 | Waiting for an event from Coordinator | NULL |
| 19 | system user | | NULL | Connect | 1452183 | Waiting for an event from Coordinator | NULL |
| 20 | system user | | NULL | Connect | 1452183 | Waiting for an event from Coordinator | NULL |
| 21 | system user | | NULL | Connect | 1452183 | Waiting for an event from Coordinator | NULL |
| 483710 | root | 112.74.19.133:33882 | prebill | Sleep | 1469 | | NULL |
| 483860 | root | 112.74.19.133:33934 | prebill | Sleep | 1040 | | NULL |
| 483925 | root | 10.1.1.51:57846 | prebill | Query | 0 | init | show processlist |
+--------+-----------------+---------------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
10 rows in set, 1 warning (0.00 sec)
mysql> kill 483710
-> ;
Query OK, 0 rows affected (0.00 sec)
mysql> show processlist;
+--------+-----------------+---------------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| 7 | event_scheduler | localhost | NULL | Daemon | 44417 | Waiting for next activation | NULL |
| 15 | system user | | NULL | Connect | 1452306 | waiting for handler commit | Group replication applier module |
| 17 | system user | | NULL | Query | 1452301 | Replica has read all relay log; waiting for more updates | NULL |
| 18 | system user | | NULL | Connect | 1452306 | Waiting for an event from Coordinator | NULL |
| 19 | system user | | NULL | Connect | 1452306 | Waiting for an event from Coordinator | NULL |
| 20 | system user | | NULL | Connect | 1452306 | Waiting for an event from Coordinator | NULL |
| 21 | system user | | NULL | Connect | 1452306 | Waiting for an event from Coordinator | NULL |
| 483860 | root | 112.74.19.133:33934 | prebill | Sleep | 1163 | | NULL |
| 483925 | root | 10.1.1.51:57846 | prebill | Query | 0 | init | show processlist |
+--------+-----------------+---------------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
9 rows in set, 1 warning (0.00 sec)
mysql> kill 483860;
Query OK, 0 rows affected (0.00 sec)
mysql> show processlist;
+--------+-----------------+-----------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+-----------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| 7 | event_scheduler | localhost | NULL | Daemon | 44426 | Waiting for next activation | NULL |
| 15 | system user | | NULL | Connect | 1452315 | waiting for handler commit | Group replication applier module |
| 17 | system user | | NULL | Query | 1452310 | Replica has read all relay log; waiting for more updates | NULL |
| 18 | system user | | NULL | Connect | 1452315 | Waiting for an event from Coordinator | NULL |
| 19 | system user | | NULL | Connect | 1452315 | Waiting for an event from Coordinator | NULL |
| 20 | system user | | NULL | Connect | 1452315 | Waiting for an event from Coordinator | NULL |
| 21 | system user | | NULL | Connect | 1452315 | Waiting for an event from Coordinator | NULL |
| 483925 | root | 10.1.1.51:57846 | prebill | Query | 0 | init | show processlist |
+--------+-----------------+-----------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
8 rows in set, 1 warning (0.00 sec)
mysql> show processlist;
+--------+-----------------+-----------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+-----------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| 7 | event_scheduler | localhost | NULL | Daemon | 44513 | Waiting for next activation | NULL |
| 15 | system user | | NULL | Connect | 1452402 | waiting for handler commit | Group replication applier module |
| 17 | system user | | NULL | Query | 1452397 | Replica has read all relay log; waiting for more updates | NULL |
| 18 | system user | | NULL | Connect | 1452402 | Waiting for an event from Coordinator | NULL |
| 19 | system user | | NULL | Connect | 1452402 | Waiting for an event from Coordinator | NULL |
| 20 | system user | | NULL | Connect | 1452402 | Waiting for an event from Coordinator | NULL |
| 21 | system user | | NULL | Connect | 1452402 | Waiting for an event from Coordinator | NULL |
| 483925 | root | 10.1.1.51:57846 | prebill | Query | 0 | init | show processlist |
+--------+-----------------+-----------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
8 rows in set, 1 warning (0.00 sec)
mysql> show processlist;
+--------+-----------------+-----------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+-----------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
| 7 | event_scheduler | localhost | NULL | Daemon | 44516 | Waiting for next activation | NULL |
| 15 | system user | | NULL | Connect | 1452405 | waiting for handler commit | Group replication applier module |
| 17 | system user | | NULL | Query | 1452400 | Replica has read all relay log; waiting for more updates | NULL |
| 18 | system user | | NULL | Connect | 1452405 | Waiting for an event from Coordinator | NULL |
| 19 | system user | | NULL | Connect | 1452405 | Waiting for an event from Coordinator | NULL |
| 20 | system user | | NULL | Connect | 1452405 | Waiting for an event from Coordinator | NULL |
| 21 | system user | | NULL | Connect | 1452405 | Waiting for an event from Coordinator | NULL |
| 483925 | root | 10.1.1.51:57846 | prebill | Query | 0 | init | show processlist |
+--------+-----------------+-----------------+---------+---------+---------+----------------------------------------------------------+----------------------------------+
8 rows in set, 1 warning (0.00 sec)
mysql> show variables like '%group_replication_force_members%';
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| group_replication_force_members | |
+---------------------------------+-------+
1 row in set (0.01 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 773123a4-daad-11ee-930b-3009f923fbf1 | 10.1.1.51 | 3307 | ONLINE | PRIMARY | 8.0.36 | XCom |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
1 row in set (0.00 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 773123a4-daad-11ee-930b-3009f923fbf1 | 10.1.1.51 | 3307 | ONLINE | PRIMARY | 8.0.36 | XCom |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
1 row in set (0.00 sec)
mysql> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 773123a4-daad-11ee-930b-3009f923fbf1 | 10.1.1.51 | 3307 | ONLINE | PRIMARY | 8.0.36 | XCom |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+----------------------------+
1 row in set (0.00 sec)
相关参考:
MGR网络抖动问题分析和group_replication_unreachable_majority_timeout等参数分析_group_replication_member_expel_timeout-CSDN博客
https://www.51cto.com/article/718981.html
MySQL之MGR成员故障导致整个集群不可用的问题排查分析 - 简书