Redis - 集群(Cluster)

一、基本概念

上述的哨兵模式,提⾼了系统的可⽤性.但是真正⽤来存储数据的还是master和slave节点.所有的数据都需要存储在单个master和slave节点中.

如果数据量很⼤,接近超出了master/slave所在机器的物理内存,就可能出现严重问题了.

如何获取更⼤的空间?加机器即可!所谓"⼤数据"的核⼼,其实就是⼀台机器搞不定了,⽤多台机器来搞.

Redis 的集群就是在上述的思路之下,引⼊多组Master/Slave,每⼀组Master/Slave存储数据全集的⼀部分,从⽽构成⼀个更⼤的整体,称为Redis集群(Cluster).

假定整个数据全集是1TB,引⼊三组Master/Slave来存储.那么每⼀组机器只需要存储整个数据全集的1/3即可.

在上述图中：

Master1和Slave11和Slave12保存的是同样的数据.占总数据的1/3
Master2和Slave21和Slave22保存的是同样的数据.占总数据的1/3
Master3和Slave31和Slave32保存的是同样的数据.占总数据的1/3

这三组机器存储的数据都是不同的

每个Slave都是对应Master的备份(当Master挂了,对应的Slave会补位成Master).

每个红框部分都可以称为是⼀个分⽚(Sharding).

如果全量数据进⼀步增加,只要再增加更多的分⽚,即可解决.

二、数据分片算法

Redis cluster 的核⼼思路是⽤多组机器来存数据的每个部分.那么接下来的核⼼问题就是,给定⼀个数据(⼀个具体的key),那么这个数据应该存储在哪个分⽚上?读取的时候⼜应该去哪个分⽚读取?

围绕这个问题,业界有三种⽐较主流的实现⽅式.

2.1、哈希求余

设有N个分⽚,使⽤[0,N-1]这样序号进⾏编号.

针对某个给定的key,先计算hash值,再把得到的结果%N,得到的结果即为分⽚编号.

例如,N为3.给定key为hello,对hello计算hash值(⽐如使⽤md5算法),得到的结果为 bc4b2a76b9719d91 ,再把这个结果%3,结果为0,那么就把hello这个key放到0号分⽚上.

当然,实际⼯作中涉及到的系统,计算hash的⽅式不⼀定是md5,但是思想是⼀致的.

后续如果要取某个key的记录,也是针对key计算hash,再对N求余,就可以找到对应的分⽚编号了.

优点: 简单⾼效,数据分配均匀.

缺点: ⼀旦需要进⾏扩容,N改变了,原有的映射规则被破坏,就需要让节点之间的数据相互传输,重新排列,以满⾜新的映射规则.此时需要搬运的数据量是⽐较多的,开销较⼤.

N为3的时候,[100,120]这21个hash值的分布(此处假定计算出的hash值是⼀个简单的整数,⽅便⾁眼观察)

当引⼊⼀个新的分⽚,N从3=>4时,⼤量的key都需要重新映射.(某个key%3和%4的结果不⼀样, 就映射到不同机器上了).

如上图可以看到,整个扩容⼀共21个key,只有3个key没有经过搬运,其他的key都是搬运过的

2.2、⼀致性哈希算法

为了降低上述的搬运开销,能够更⾼效扩容,业界提出了"⼀致性哈希算法".

key 映射到分⽚序号的过程不再是简单求余了,⽽是改成以下过程:

第⼀步：把0->2^32-1这个数据空间,映射到⼀个圆环上.数据按照顺时针⽅向增⻓.

第⼆步：假设当前存在三个分⽚,就把分⽚放到圆环的某个位置上.

第三步：假定有⼀个key,计算得到hash值H,那么这个key映射到哪个分⽚呢?规则很简单,就是从H 所在位置,顺时针往下找,找到的第⼀个分⽚,即为该key所从属的分⽚.

这就相当于,N个分⽚的位置,把整个圆环分成了N个管辖区间.Key的hash值落在某个区间内,就归对应区间管理.

在这个情况下,如果扩容⼀个分⽚,如何处理呢? 原有分⽚在环上的位置不动,只要在环上新安排⼀个分⽚位置即可.

此时,只需要把0号分⽚上的部分数据,搬运给3号分⽚即可.1号分⽚和2号分⽚管理的区间都是不变的.

优点: ⼤⼤降低了扩容时数据搬运的规模,提⾼了扩容操作的效率.

缺点: 数据分配不均匀(有的多有的少,数据倾斜).

2.3、哈希槽分区算法(Redis使⽤)

为了解决上述问题(搬运成本⾼和数据分配不均匀),Rediscluster引⼊了哈希槽(hashslots)算法

hash_slot = crc16(key) % 16384

其中crc16也是⼀种hash算法.

16384 其实是16*1024,也就是2^14.

相当于是把整个哈希值,映射到16384个槽位上,也就是[0,16383]。

然后再把这些槽位⽐较均匀的分配给每个分⽚.每个分⽚的节点都需要记录⾃⼰持有哪些分⽚.

假设当前有三个分⽚,⼀种可能的分配⽅式：

0号分⽚:[0,5461],共5462个槽位
1号分⽚:[5462,10923],共5462个槽位
2号分⽚:[10924,16383],共5460个槽位

这⾥的分⽚规则是很灵活的.每个分⽚持有的槽位也不⼀定连续. 每个分⽚的节点使⽤位图来表⽰⾃⼰持有哪些槽位.对于16384个槽位来说,需要2048个字节(2KB)⼤⼩的内存空间表⽰.

如果需要进⾏扩容,⽐如新增⼀个3号分⽚,就可以针对原有的槽位进⾏重新分配.

⽐如可以把之前每个分⽚持有的槽位,各拿出⼀点,分给新分⽚.

⼀种可能的分配⽅式:

0号分⽚:[0,4095],共4096个槽位
1号分⽚:[5462,9557],共4096个槽位
2号分⽚:[10924,15019],共4096个槽位
3号分⽚:[4096,5461]+[9558,10923]+[15019,16383],共4096个槽位

我们在实际使⽤Redis集群分⽚的时候,不需要⼿动指定哪些槽位分配给某个分⽚,只需要告诉某个分⽚应该持有多少个槽位即可,Redis会⾃动完成后续的槽位分配,以及对应的key搬运的⼯作.

此处还有两个问题:

问题⼀：Redis集群是最多有16384个分⽚吗?

并⾮如此.如果⼀个分⽚只有⼀个槽位,这对于集群的数据均匀其实是难以保证的

实际上Redis的作者建议集群分⽚数不应该超过1000.

⽽且,16000这么⼤规模的集群,本⾝的可⽤性也是⼀个⼤问题.⼀个系统越复杂,出现故障的概率是越⾼的.

问题⼆：为什么是16384个槽位?

Redis 作者的答案：https://github.com/antirez/redis/issues/2576

节点之间通过⼼跳包通信.⼼跳包中包含了该节点持有哪些slots.这个是使⽤位图这样的数据结构表⽰的.表⽰16384(16k)个slots,需要的位图⼤⼩是2KB.如果给定的slots数更多了,⽐如65536 个了,此时就需要消耗更多的空间,8KB位图表⽰了.8KB,对于内存来说不算什么,但是在频繁的⽹络⼼跳包中,还是⼀个不⼩的开销的.
另⼀⽅⾯,Redis集群⼀般不建议超过1000个分⽚.所以16k对于最⼤1000个分⽚来说是⾜够⽤的,同时也会使对应的槽位配置位图体积不⾄于很⼤.

三、集群搭建(基于docker)

接下来基于docker,搭建⼀个集群.每个节点都是⼀个容器.

拓扑结构如下:

注意! 此处先创建出11个redis节点.其中前9个⽤来演⽰集群的搭建. 后两个⽤来演⽰集群扩容

3.1、创建⽬录和配置

创建redis-cluster ⽬录.内部创建两个⽂件

redis-cluster/
 ├── docker-compose.yml 
 └── generate.sh

generate.sh 内容如下

 for port in $(seq 1 9); \
 do \
 mkdir -p redis${port}/
 touch redis${port}/redis.conf
 cat << EOF > redis${port}/redis.conf
 port 6379
 bind 0.0.0.0
 protected-mode no
 appendonly yes
 cluster-enabled yes
 cluster-config-file nodes.conf
 cluster-node-timeout 5000
 cluster-announce-ip 172.30.0.10${port}
 cluster-announce-port 6379
 cluster-announce-bus-port 16379
 EOF
 done

 # 注意 cluster-announce-ip 的值有变化. 
 for port in $(seq 10 11); \
 do \
 mkdir -p redis${port}/
 touch redis${port}/redis.conf
 cat << EOF > redis${port}/redis.conf
 port 6379
 bind 0.0.0.0
 protected-mode no
 appendonly yes
 cluster-enabled yes
 cluster-config-file nodes.conf
 cluster-node-timeout 5000
 cluster-announce-ip 172.30.0.1${port}
 cluster-announce-port 6379
 cluster-announce-bus-port 16379
 EOF
 done

执⾏命令

 bash generate.sh

⽣成⽬录如下:

redis-cluster/
├── docker-compose.yml 
├── generate.sh 
├── redis1
│   └── redis.conf 
├── redis10 
│   └── redis.conf 
├── redis11 
│   └── redis.conf 
├── redis2 
│   └── redis.conf 
├── redis3 
│   └── redis.conf 
├── redis4 
│   └── redis.conf 
├── redis5 
│   └── redis.conf 
├── redis6 
│   └── redis.conf 
├── redis7 
│   └── redis.conf 
├── redis8 
│   └── redis.conf 
└── redis9 
    └── redis.conf

其中redis.conf每个都不同.以redis1为例：

区别在于每个配置中配置的 cluster-announce-ip 是不同的,其他部分都相同.

 port 6379
 bind 0.0.0.0
 protected-mode no
 appendonly yes
 cluster-enabled yes
 cluster-config-file nodes.conf
 cluster-node-timeout 5000
 cluster-announce-ip 172.30.0.101
 cluster-announce-port 6379
 cluster-announce-bus-port 16379

后续会给每个节点分配不同的ip地址.

配置说明：

cluster-enabled yes 开启集群
cluster-config-file nodes.conf 集群节点⽣成的配置.
cluster-node-timeout 5000 节点失联的超时时间.
cluster-announce-ip 172.30.0.101 节点⾃⾝ip.
cluster-announce-port 6379 节点⾃⾝的业务端⼝.
cluster-announce-bus-port 16379 节点⾃⾝的总线端⼝.集群管理的信息交互是通过这个端⼝进⾏的.

3.2、编写docker-compose.yml

先创建networks，并分配⽹段为 172.30.0.0/24
配置每个节点.注意配置⽂件映射,端⼝映射,以及容器的ip地址.设定成固定ip⽅便后续的观察和操作.

此处的端⼝映射不配置也可以,配置的⽬的是为了可以通过宿主机ip+映射的端⼝进⾏访问.通过容器⾃⾝ip:6379的⽅式也可以访问.

 version: '3.7'
 networks:
   mynet:
     ipam:
       config:
         - subnet: 172.30.0.0/24
 services:
   redis1:
     image: 'redis:5.0.9'
     container_name: redis1
     restart: always
     volumes:
       - ./redis1/:/etc/redis/
     ports:
       - 6371:6379- 16371:16379
     command:
       redis-server /etc/redis/redis.conf
     networks:
       mynet:
         ipv4_address: 172.30.0.101

   redis2:
     image: 'redis:5.0.9'
     container_name: redis2
     restart: always
     volumes:
       - ./redis2/:/etc/redis/
     ports:
       - 6372:6379
       - 16372:16379
     command:
       redis-server /etc/redis/redis.conf
     networks:
       mynet:
         ipv4_address: 172.30.0.102

  redis3:
    image: 'redis:5.0.9'
    container_name: redis3
    restart: always
    volumes:
      - ./redis3/:/etc/redis/
    ports:
      - 6373:6379
      - 16373:16379
    command:
      redis-server /etc/redis/redis.conf
    networks:
      mynet:
        ipv4_address: 172.30.0.103

  redis4:
    image: 'redis:5.0.9'
    container_name: redis4
    restart: always
    volumes:
      - ./redis4/:/etc/redis/
    ports:
      - 6374:6379
      - 16374:16379
    command:
      redis-server /etc/redis/redis.conf
    networks:
      mynet:
        ipv4_address: 172.30.0.104

  redis5:
    image: 'redis:5.0.9'
    container_name: redis5
    restart: always
    volumes:
      - ./redis5/:/etc/redis/
 ports:
      - 6375:6379
      - 16375:16379
    command:
      redis-server /etc/redis/redis.conf
    networks:
      mynet:
        ipv4_address: 172.30.0.105

  redis6:
    image: 'redis:5.0.9'
    container_name: redis6
    restart: always
    volumes:
      - ./redis6/:/etc/redis/
    ports:
      - 6376:6379
      - 16376:16379
    command:
      redis-server /etc/redis/redis.conf
    networks:
      mynet:
        ipv4_address: 172.30.0.106

  redis7:
    image: 'redis:5.0.9'
    container_name: redis7
    restart: always
    volumes:
      - ./redis7/:/etc/redis/
    ports:
      - 6377:6379
      - 16377:16379
    command:
      redis-server /etc/redis/redis.conf
    networks:
      mynet:
        ipv4_address: 172.30.0.107

  redis8:
    image: 'redis:5.0.9'
    container_name: redis8
    restart: always
    volumes:
      - ./redis8/:/etc/redis/
    ports:
      - 6378:6379
  - 16378:16379
    command:
      redis-server /etc/redis/redis.conf
    networks:
      mynet:
        ipv4_address: 172.30.0.108

  redis9:
    image: 'redis:5.0.9'
    container_name: redis9
    restart: always
    volumes:
      - ./redis9/:/etc/redis/
    ports:
      - 6379:6379
      - 16379:16379
    command:
      redis-server /etc/redis/redis.conf
    networks:
      mynet:
        ipv4_address: 172.30.0.109
        
  redis10:
    image: 'redis:5.0.9'
    container_name: redis10
    restart: always
    volumes:
      - ./redis10/:/etc/redis/
    ports:
      - 6380:6379
      - 16380:16379
    command:
      redis-server /etc/redis/redis.conf
    networks:
      mynet:
        ipv4_address: 172.30.0.110

  redis11:
    image: 'redis:5.0.9'
    container_name: redis11
    restart: always
    volumes:
      - ./redis11/:/etc/redis/
    ports:
      - 6381:6379
      - 16381:16379
    command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.111

3.3、启动容器

docker-compose up -d

3.4、构建集群

启动⼀个docker客⼾端

此处是把前9个主机构建成集群,3主6从.后2个主机暂时不⽤.

redis-cli --cluster create 172.30.0.101:6379 172.30.0.102:6379 
172.30.0.103:6379 172.30.0.104:6379 172.30.0.105:6379 172.30.0.106:6379 
172.30.0.107:6379 172.30.0.108:6379 172.30.0.109:6379  --cluster-replicas 2

--cluster create 表⽰建⽴集群.后⾯填写每个节点的ip和地址.
--cluster-replicas 2 表⽰每个主节点需要两个从节点备份.

执⾏之后,容器之间会进⾏加⼊集群操作.

⽇志中会描述哪些是主节点,哪些从节点跟随哪个主节点.

 >>> Performing hash slots allocation on 9 nodes...
 Master[0] -> Slots 0 - 5460
 Master[1] -> Slots 5461 - 10922
 Master[2] -> Slots 10923 - 16383
 Adding replica 172.30.0.105:6379 to 172.30.0.101:6379
 Adding replica 172.30.0.106:6379 to 172.30.0.101:6379
 Adding replica 172.30.0.107:6379 to 172.30.0.102:6379
 Adding replica 172.30.0.108:6379 to 172.30.0.102:6379
 Adding replica 172.30.0.109:6379 to 172.30.0.103:6379
 Adding replica 172.30.0.104:6379 to 172.30.0.103:6379
 M: e4f37f8f0ea0dafc584349999795716613910e51 172.30.0.101:6379
    slots:[0-5460] (5461 slots) master
 M: 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 172.30.0.102:6379
    slots:[5461-10922] (5462 slots) master
 M: b3c0a96f6a206088ecea639147b6fcf903afe872 172.30.0.103:6379
    slots:[10923-16383] (5461 slots) master
 S: 85025819223f12615046c54d89f510e9cd0444a1 172.30.0.104:6379
    replicates b3c0a96f6a206088ecea639147b6fcf903afe872
 S: 2e5dc211288784ba55d554a377b87bfe2b5398db 172.30.0.105:6379
    replicates e4f37f8f0ea0dafc584349999795716613910e51
 S: 29f05d98982bd3df05d0222091e4b8ef9569f424 172.30.0.106:6379
    replicates e4f37f8f0ea0dafc584349999795716613910e51
 S: 3584840ac704c3ee016f3bdcca3f7ebe6f6e8e80 172.30.0.107:6379
    replicates 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c
 S: 0a889103b35db2a6e82e8c09904bbef310cff3b1 172.30.0.108:6379
    replicates 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c
 S: 00ba82bed6abeb015116d51d1af7fcb1609d03ad 172.30.0.109:6379
    replicates b3c0a96f6a206088ecea639147b6fcf903afe872
 Can I set the above configuration? (type 'yes' to accept): yes
 >>> Nodes configuration updated
 >>> Assign a different config epoch to each node
 >>> Sending CLUSTER MEET messages to join the cluster
 Waiting for the cluster to join
 ...
 >>> Performing Cluster Check (using node 172.30.0.101:6379)
 M: e4f37f8f0ea0dafc584349999795716613910e51 172.30.0.101:6379
    slots:[0-5460] (5461 slots) master
    2 additional replica(s)
 M: 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 172.30.0.102:6379
    slots:[5461-10922] (5462 slots) master
    2 additional replica(s)
 S: 2e5dc211288784ba55d554a377b87bfe2b5398db 172.30.0.105:6379
    slots: (0 slots) slave
    replicates e4f37f8f0ea0dafc584349999795716613910e51
 S: 0a889103b35db2a6e82e8c09904bbef310cff3b1 172.30.0.108:6379
    slots: (0 slots) slave
    replicates 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c
 S: 3584840ac704c3ee016f3bdcca3f7ebe6f6e8e80 172.30.0.107:6379
    slots: (0 slots) slave
    replicates 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c
 S: 85025819223f12615046c54d89f510e9cd0444a1 172.30.0.104:6379
    slots: (0 slots) slave
    replicates b3c0a96f6a206088ecea639147b6fcf903afe872
 S: 00ba82bed6abeb015116d51d1af7fcb1609d03ad 172.30.0.109:6379
    slots: (0 slots) slave
    replicates b3c0a96f6a206088ecea639147b6fcf903afe872
 S: 29f05d98982bd3df05d0222091e4b8ef9569f424 172.30.0.106:6379
    slots: (0 slots) slave
    replicates e4f37f8f0ea0dafc584349999795716613910e51
 M: b3c0a96f6a206088ecea639147b6fcf903afe872 172.30.0.103:6379
    slots:[10923-16383] (5461 slots) master
    2 additional replica(s)
 [OK] All nodes agree about slots configuration.
 >>> Check for open slots...
 >>> Check slots coverage...
 [OK] All 16384 slots covered.

⻅到下⽅的[OK]说明集群建⽴完成.

此时,使⽤客⼾端连上集群中的任何⼀个节点,都相当于连上了整个集群.

客⼾端后⾯要加上-c 选项,否则如果key没有落到当前节点上,是不能操作的. -c 会自动吧请求重定向到对应节点.
使⽤ cluster nodes 可以查看到整个集群的情况.

 # redis-cli -h 172.30.0.101 -p 6379 -c
 172.30.0.101:6379> CLUSTER nodes
 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 172.30.0.102:6379@16379 master - 0 
 1682595474000 2 connected 5461-10922
 2e5dc211288784ba55d554a377b87bfe2b5398db 172.30.0.105:6379@16379 slave 
 e4f37f8f0ea0dafc584349999795716613910e51 0 1682595475158 5 connected
 0a889103b35db2a6e82e8c09904bbef310cff3b1 172.30.0.108:6379@16379 slave 
 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682595475560 8 connected
 3584840ac704c3ee016f3bdcca3f7ebe6f6e8e80 172.30.0.107:6379@16379 slave 
 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682595474656 7 connected
 85025819223f12615046c54d89f510e9cd0444a1 172.30.0.104:6379@16379 slave 
 b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682595475560 4 connected
 e4f37f8f0ea0dafc584349999795716613910e51 172.30.0.101:6379@16379 myself,master - 0 
 1682595475000 1 connected 0-5460
 00ba82bed6abeb015116d51d1af7fcb1609d03ad 172.30.0.109:6379@16379 slave 
 b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682595475000 9 connected
 29f05d98982bd3df05d0222091e4b8ef9569f424 172.30.0.106:6379@16379 slave 
 e4f37f8f0ea0dafc584349999795716613910e51 0 1682595474000 6 connected
 b3c0a96f6a206088ecea639147b6fcf903afe872 172.30.0.103:6379@16379 master - 0 
 1682595475659 3 connected 10923-16383

 172.30.0.101:6379> set k1 1
 -> Redirected to slot [12706] located at 172.30.0.103:6379
 OK
 172.30.0.103:6379> get k1
 "1"

四、主节点宕机

4.1、演⽰效果

⼿动停⽌⼀个master节点,观察效果

⽐如上述拓扑结构中,可以看到redis1redis2redis3是主节点,随便挑⼀个停掉.

docker stop redis1

连上redis2,观察结果.

 172.30.0.102:6379> CLUSTER NODES
 e4f37f8f0ea0dafc584349999795716613910e51 172.30.0.101:6379@16379 master,fail - 
 1682599172083 1682599170000 1 connected
 0a889103b35db2a6e82e8c09904bbef310cff3b1 172.30.0.108:6379@16379 slave 
 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682603536508 8 connected
 29f05d98982bd3df05d0222091e4b8ef9569f424 172.30.0.106:6379@16379 slave 
 2e5dc211288784ba55d554a377b87bfe2b5398db 0 1682603537512 10 connected
 00ba82bed6abeb015116d51d1af7fcb1609d03ad 172.30.0.109:6379@16379 slave 
 b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682603536000 9 connected
 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 172.30.0.102:6379@16379 myself,master - 0 
 1682603535000 2 connected 5461-10922
 b3c0a96f6a206088ecea639147b6fcf903afe872 172.30.0.103:6379@16379 master - 0 
 1682603537009 3 connected 10923-16383
 2e5dc211288784ba55d554a377b87bfe2b5398db 172.30.0.105:6379@16379 master - 0 
 1682603537000 10 connected 0-5460
 3584840ac704c3ee016f3bdcca3f7ebe6f6e8e80 172.30.0.107:6379@16379 slave 
 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682603537512 7 connected
 85025819223f12615046c54d89f510e9cd0444a1 172.30.0.104:6379@16379 slave 
 b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682603536708 4 connected

可以看到,101已经提⽰fail,然后原本是slave的105成了新的master.

如果重新启动redis1

docker start redis1

再次观察结果.可以看到101启动了,仍然是slave.

 172.30.0.102:6379> CLUSTER NODES
 e4f37f8f0ea0dafc584349999795716613910e51 172.30.0.101:6379@16379 slave 
 2e5dc211288784ba55d554a377b87bfe2b5398db 0 1682603944000 10 connected
 0a889103b35db2a6e82e8c09904bbef310cff3b1 172.30.0.108:6379@16379 slave 
 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682603944628 8 connected
 29f05d98982bd3df05d0222091e4b8ef9569f424 172.30.0.106:6379@16379 slave 
 2e5dc211288784ba55d554a377b87bfe2b5398db 0 1682603943625 10 connected
 00ba82bed6abeb015116d51d1af7fcb1609d03ad 172.30.0.109:6379@16379 slave 
 b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682603944126 9 connected
 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 172.30.0.102:6379@16379 myself,master - 0 
 1682603944000 2 connected 5461-10922
 b3c0a96f6a206088ecea639147b6fcf903afe872 172.30.0.103:6379@16379 master - 0 
 1682603945030 3 connected 10923-16383
 2e5dc211288784ba55d554a377b87bfe2b5398db 172.30.0.105:6379@16379 master - 0 
 1682603944025 10 connected 0-5460
 3584840ac704c3ee016f3bdcca3f7ebe6f6e8e80 172.30.0.107:6379@16379 slave 
 5f71983ad52cc7077ce8874ae1c4f9c23d9f502c 0 1682603945131 7 connected
 85025819223f12615046c54d89f510e9cd0444a1 172.30.0.104:6379@16379 slave 
 b3c0a96f6a206088ecea639147b6fcf903afe872 0 1682603944628 4 connected

可以使⽤ cluster failover 进⾏集群恢复.也就是把101重新设定成master.(登录到101上执⾏)

4.2、处理流程

1) 故障判定

集群中的所有节点,都会周期性的使⽤⼼跳包进⾏通信.

节点A给节点B发送ping包,B就会给A返回⼀个pong包.ping和pong除了 message type 属性之外,其他部分都是⼀样的.这⾥包含了集群的配置信息(该节点的id,该节点从属于哪个分⽚, 是主节点还是从节点,从属于谁,持有哪些slots的位图...).
每个节点,每秒钟,都会给⼀些随机的节点发起ping包,⽽不是全发⼀遍.这样设定是为了避免在节点很多的时候,⼼跳包也⾮常多(⽐如有9个节点,如果全发,就是9*8有72组⼼跳了,⽽且这是按照N^2这样的级别增⻓的)
当节点A给节点B发起ping包,B不能如期回应的时候,此时A就会尝试重置和B的tcp连接,看能否连接成功.如果仍然连接失败,A就会把B设为PFAIL状态(相当于主观下线).
A判定B为PFAIL之后,会通过redis内置的Gossip协议,和其他节点进⾏沟通,向其他节点确认B 的状态.(每个节点都会维护⼀个⾃⼰的"下线列表",由于视⻆不同,每个节点的下线列表也不⼀定相同).
此时A发现其他很多节点,也认为B为PFAIL,并且数⽬超过总集群个数的⼀半,那么A就会把B标记成FAIL(相当于客观下线),并且把这个消息同步给其他节点(其他节点收到之后,也会把B标记成 FAIL).

⾄此,B就彻底被判定为故障节点了.

某个或者某些节点宕机，有的时候会引起整个集群都宕机(称为fail状态).

以下三种情况会出现集群宕机:

某个分⽚,所有的主节点和从节点都挂了.
某个分⽚,主节点挂了,但是没有从节点.
超过半数的master节点都挂了.

核⼼原则是保证每个slots都能正常⼯作(存取数据).

2) 故障迁移

上述例⼦中,B故障,并且A把BFAIL的消息告知集群中的其他节点.

如果B是从节点,那么不需要进⾏故障迁移.
如果B是主节点,那么就会由B的从节点(⽐如C和D)触发故障迁移了.

所谓故障迁移,就是指把从节点提拔成主节点,继续给整个redis集群提供⽀持.

具体流程如下:

从节点判定⾃⼰是否具有参选资格.如果从节点和主节点已经太久没通信(此时认为从节点的数据和主节点差异太⼤了),时间超过阈值,就失去竞选资格.
具有资格的节点,⽐如C和D,就会先休眠⼀定时间.休眠时间=500ms基础时间+[0,500ms]随机时间+排名*1000ms.offset的值越⼤,则排名越靠前(越⼩).
⽐如C的休眠时间到了,C就会给其他所有集群中的节点,进⾏拉票操作.但是只有主节点才有投票资格.
主节点就会把⾃⼰的票投给C(每个主节点只有1票).当C收到的票数超过主节点数⽬的⼀半,C就会晋升成主节点.(C⾃⼰负责执⾏slaveofnoone,并且让D执⾏slaveofC).
同时,C还会把⾃⼰成为主节点的消息,同步给其他集群的节点.⼤家也都会更新⾃⼰保存的集群结构信息.

上述选举的过程,称为Raft算法,是⼀种在分布式系统中⼴泛使⽤的算法. 在随机休眠时间的加持下,基本上就是谁先唤醒,谁就能竞选成功.

五、集群扩容

扩容是⼀个在开发中⽐较常遇到的场景.

随着业务的发展,现有集群很可能⽆法容纳⽇益增⻓的数据.此时给集群中加⼊更多新的机器,就可以使存储的空间更⼤了.

所谓分布式的本质,就是使⽤更多的机器,引⼊更多的硬件资源.

5.1、把新的主节点加⼊到集群

上⾯已经把redis1-redis9重新构成了集群.接下来把redis10和redis11也加⼊集群.

此处把redis10作为主机,redis11作为从机.

redis-cli --cluster add-node 172.30.0.110:6379 172.30.0.101:6379

add-node后的第⼀组地址是新节点的地址.第⼆组地址是集群中的任意节点地址.

执⾏结果

 >>> Adding node 172.30.0.110:6379 to cluster 172.30.0.101:6379
 >>> Performing Cluster Check (using node 172.30.0.101:6379)
 M: 00d319e23ef76a4d51e74600c42ee2a371ae81f6 172.30.0.101:6379
    slots:[0-5460] (5461 slots) master
    2 additional replica(s)
 S: e34911c57d7605903de84ec05b3deac611aaef7e 172.30.0.105:6379
    slots: (0 slots) slave
    replicates 00d319e23ef76a4d51e74600c42ee2a371ae81f6
 S: 6cf48cc11d0171b6ab1b418808473167acd7986e 172.30.0.106:6379
    slots: (0 slots) slave
    replicates 00d319e23ef76a4d51e74600c42ee2a371ae81f6
 S: fd18c7f164b09ec563f4573ec9d6466e6769221e 172.30.0.108:6379
    slots: (0 slots) slave
    replicates b3f2ba758318f4bd54031c98c01d7a6155ff43d3
 M: 579282abe81b3f20ffd17d5a1956cdca3b0e71b0 172.30.0.103:6379
    slots:[10923-16383] (5461 slots) master
    2 additional replica(s)
 S: e9ea79b1326ea5a75a1701d5c12a0f6081c1d043 172.30.0.109:6379
    slots: (0 slots) slave
    replicates 579282abe81b3f20ffd17d5a1956cdca3b0e71b0
 S: 628d1ec9ceef6760b9038c4fbc83ee92430062ac 172.30.0.107:6379
    slots: (0 slots) slave
    replicates b3f2ba758318f4bd54031c98c01d7a6155ff43d3
 M: b3f2ba758318f4bd54031c98c01d7a6155ff43d3 172.30.0.102:6379
    slots:[5461-10922] (5462 slots) master
   2 additional replica(s)
 S: 2a248acb47f0036655397897f9800c70ea22514f 172.30.0.104:6379
   slots: (0 slots) slave
   replicates 579282abe81b3f20ffd17d5a1956cdca3b0e71b0
 [OK] All nodes agree about slots configuration.
 >>> Check for open slots...
 >>> Check slots coverage...
 [OK] All 16384 slots covered.
 >>> Send CLUSTER MEET to node 172.30.0.110:6379 to make it join the cluster.
 [OK] New node added correctly.

此时的集群状态如下,可以看到 1 2 3 4 5 6 7 172.30.0.110 这个节点已经成为了集群中的主节点.

 127.0.0.1:6379> CLUSTER NODES
 b3f2ba758318f4bd54031c98c01d7a6155ff43d3 172.30.0.102:6379@16379 master - 0 
 1683183720184 2 connected 5461-10922
 e34911c57d7605903de84ec05b3deac611aaef7e 172.30.0.105:6379@16379 slave 
 00d319e23ef76a4d51e74600c42ee2a371ae81f6 0 1683183722189 5 connected
 522a1bd88a1a9084e6919fa88f4bf1c3655ad837 172.30.0.110:6379@16379 master - 0 
 1683183720000 10 connected
 579282abe81b3f20ffd17d5a1956cdca3b0e71b0 172.30.0.103:6379@16379 master - 0 
 1683183721084 3 connected 10923-16383
 00d319e23ef76a4d51e74600c42ee2a371ae81f6 172.30.0.101:6379@16379 master - 0 
 1683183721185 1 connected 0-5460
 6cf48cc11d0171b6ab1b418808473167acd7986e 172.30.0.106:6379@16379 slave 
 00d319e23ef76a4d51e74600c42ee2a371ae81f6 0 1683183720000 6 connected
 e9ea79b1326ea5a75a1701d5c12a0f6081c1d043 172.30.0.109:6379@16379 myself,slave 
 579282abe81b3f20ffd17d5a1956cdca3b0e71b0 0 1683183720000 9 connected
 fd18c7f164b09ec563f4573ec9d6466e6769221e 172.30.0.108:6379@16379 slave 
 b3f2ba758318f4bd54031c98c01d7a6155ff43d3 0 1683183720684 8 connected
 2a248acb47f0036655397897f9800c70ea22514f 172.30.0.104:6379@16379 slave 
 579282abe81b3f20ffd17d5a1956cdca3b0e71b0 0 1683183722088 4 connected
 628d1ec9ceef6760b9038c4fbc83ee92430062ac 172.30.0.107:6379@16379 slave 
 b3f2ba758318f4bd54031c98c01d7a6155ff43d3 0 1683183720000 7 connected

5.2、重新分配slots

 redis-cli --cluster reshard 172.30.0.101:6379

reshard 后的地址是集群中的任意节点地址.

另外,注意单词拼写,是reshard(重新切分),不是reshared(重新分享),不要多写个e

执⾏之后,会进⼊交互式操作,redis会提⽰⽤⼾输⼊以下内容:

多少个slots要进⾏reshard?(此处我们填写4096)
哪个节点来接收这些slots?(此处我们填写 172.30.0.110 这个节点的集群节点id)
这些slots从哪些节点搬运过来?(此处我们填写all,表⽰从其他所有的节点都进⾏搬运)

执⾏结果如下

 How many slots do you want to move (from 1 to 16384)? 4096
 What is the receiving node ID? 522a1bd88a1a9084e6919fa88f4bf1c3655ad837
 Please enter all the source node IDs.
   Type 'all' to use all the nodes as source nodes for the hash slots.
   Type 'done' once you entered all the source nodes IDs.
 Source node #1: all

确定之后,会初步打印出搬运⽅案,让⽤⼾确认.

之后就会进⾏集群的key搬运⼯作.这个过程涉及到数据搬运.可能需要消耗⼀定的时间.

在搬运key的过程中,对于那些不需要搬运的key,访问的时候是没有任何问题的.但是对于需要搬运的key,进⾏访问可能会出现短暂的访问错误(key的位置出现了变化).

随着搬运完成,这样的错误⾃然就恢复了.

5.3、给新的主节点添加从节点

光有主节点了,此时扩容的⽬标已经初步达成.但是为了保证集群可⽤性,还需要给这个新的主节点添加从节点,保证该主节点宕机之后,有从节点能够顶上.

redis-cli --cluster add-node 172.30.0.111:6379 172.30.0.101:6379 --cluster
slave --cluster-master-id [172.30.1.110 节点的 nodeId]

执⾏完毕后,从节点就已经被添加完成了.

六、集群缩容

扩容是⽐较常⻅的,但是缩容其实⾮常少⻅.此处我们简单了解缩容的操作步骤即可.

接下来演⽰把 110 和 111 这两个节点删除.

6.1、删除从节点

此处删除的节点 nodeId 是 111 节点的 id.

 # redis-cli --cluster del-node [集群中任⼀节点ip:port] [要删除的从机节点 nodeId]
 redis-cli --cluster del-node 172.30.0.101:6379 
 03f4a97806a0d3de2299cc16e6a3559f0c832bc1

 >>> Removing node 03f4a97806a0d3de2299cc16e6a3559f0c832bc1 from cluster 
 172.30.0.101:6379
 >>> Sending CLUSTER FORGET messages to the cluster...
 >>> SHUTDOWN the node.

6.2、重新分配slots

 redis-cli --cluster reshard 172.30.0.101:6379

执⾏后仍然进⼊交互式操作.

注意!!此时要删除的主节点,包含4096个slots.我们把 110 这个注解上的这4096个slots 分成三份(1365+1365+1366),分别分给其他三个主节点.

这样可以使reshard之后的集群各个分⽚slots数⽬仍然均匀.

第⼀次重分配：分配给 101 1365 个 slots

接收slots 的 nodeId 填写 101 的 nodeId.Source Node 填写 110 的nodeId

 How many slots do you want to move (from 1 to 16384)? 1365
 What is the receiving node ID? 3397c6364b43dd8a8d49057ad37be57760d3a81f
 Please enter all the source node IDs.
   Type 'all' to use all the nodes as source nodes for the hash slots.
   Type 'done' once you entered all the source nodes IDs.
 Source node #1: 7c343b7e3f82f2e601ac6b9eba9f846b3065c600
 Source node #2: done

第⼆次重分配：分配给接 102 1365个slots

接收slots的nodeId填写 102 的nodeId.SourceNode填写 110 的 nodeld

 How many slots do you want to move (from 1 to 16384)? 1365
 What is the receiving node ID? 98736357a53c85aaebb31fa5ad286ab36b862426
 Please enter all the source node IDs.
   Type 'all' to use all the nodes as source nodes for the hash slots.
   Type 'done' once you entered all the source nodes IDs.
 Source node #1: 7c343b7e3f82f2e601ac6b9eba9f846b3065c600
 Source node #2: done

第三次重分配：分配给接 103 1366个slots

接收 slots的nodeId填写 103 的nodeId.SourceNode填写 110 的 nodeId

 How many slots do you want to move (from 1 to 16384)? 1366
 What is the receiving node ID? 26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c
 Please enter all the source node IDs.
   Type 'all' to use all the nodes as source nodes for the hash slots.
   Type 'done' once you entered all the source nodes IDs.
 Source node #1: 7c343b7e3f82f2e601ac6b9eba9f846b3065c600
 Source node #2: done

此时查看集群状态,可以看到 110 节点已经不再持有slots了.

 127.0.0.1:6379> CLUSTER NODES
 26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 172.30.0.103:6379@16379 master - 0 
 1683187990000 13 connected 6826 10923-16383
 ee283a923b77fda2e370ff4463cbb22047f261f7 172.30.0.108:6379@16379 slave 
 98736357a53c85aaebb31fa5ad286ab36b862426 0 1683187991544 12 connected
 98736357a53c85aaebb31fa5ad286ab36b862426 172.30.0.102:6379@16379 master - 0 
 1683187991544 12 connected 5461-6825 6827-10922
 b951da5c3573eec5db2926e022302188a8b686a6 172.30.0.107:6379@16379 slave 
 98736357a53c85aaebb31fa5ad286ab36b862426 0 1683187991645 12 connected
 df193127f99c59999e14518ff7235faeb7b7e8e7 172.30.0.104:6379@16379 slave 
 26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 0 1683187990000 13 connected
 7c343b7e3f82f2e601ac6b9eba9f846b3065c600 172.30.0.110:6379@16379 master - 0 
 1683187991000 10 connected
 3d148089278f9022a62a5936022e9c086fb7a062 172.30.0.105:6379@16379 slave 
 3397c6364b43dd8a8d49057ad37be57760d3a81f 0 1683187990540 11 connected
 6de544df01337d4bbdb2f4f69f4c408314761392 172.30.0.109:6379@16379 myself,slave 
 26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 0 1683187990000 9 connected
 3397c6364b43dd8a8d49057ad37be57760d3a81f 172.30.0.101:6379@16379 master - 0 
 1683187990640 11 connected 0-5460
 3b47ebc9868bbda018d8a5b926256861bb800cdc 172.30.0.106:6379@16379 slave 
 26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 0 1683187991544 13 connected

6.3、删除主节点

把 110 节点从集群中删除.

 # redis-cli --cluster del-node [集群中任⼀节点ip:port] [要删除的从机节点 nodeId]
 redis-cli --cluster del-node 172.30.0.101:6379 
 7c343b7e3f82f2e601ac6b9eba9f846b3065c600 
 >>> Removing node 7c343b7e3f82f2e601ac6b9eba9f846b3065c600 from cluster 
 172.30.0.101:6379
 >>> Sending CLUSTER FORGET messages to the cluster...
 >>> SHUTDOWN the node.

再次查看集群节点信息, 110 节点已经不在集群中了.

 127.0.0.1:6379> CLUSTER nodes
 26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 172.30.0.103:6379@16379 master - 0 
 1683188151590 13 connected 6826 10923-16383
 ee283a923b77fda2e370ff4463cbb22047f261f7 172.30.0.108:6379@16379 slave 
 98736357a53c85aaebb31fa5ad286ab36b862426 0 1683188152000 12 connected
 98736357a53c85aaebb31fa5ad286ab36b862426 172.30.0.102:6379@16379 master - 0 
 1683188151000 12 connected 5461-6825 6827-10922
 b951da5c3573eec5db2926e022302188a8b686a6 172.30.0.107:6379@16379 slave 
 98736357a53c85aaebb31fa5ad286ab36b862426 0 1683188153195 12 connected
 df193127f99c59999e14518ff7235faeb7b7e8e7 172.30.0.104:6379@16379 slave 
 26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 0 1683188151187 13 connected
 3d148089278f9022a62a5936022e9c086fb7a062 172.30.0.105:6379@16379 slave 
 3397c6364b43dd8a8d49057ad37be57760d3a81f 0 1683188152191 11 connected
 6de544df01337d4bbdb2f4f69f4c408314761392 172.30.0.109:6379@16379 myself,slave 
 26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 0 1683188152000 9 connected
 3397c6364b43dd8a8d49057ad37be57760d3a81f 172.30.0.101:6379@16379 master - 0 
 1683188152593 11 connected 0-5460
 3b47ebc9868bbda018d8a5b926256861bb800cdc 172.30.0.106:6379@16379 slave 
 26e98f947b99b3a2a5da5a7c3ed3875ae9cf366c 0 1683188152694 13 connected

⾄此,缩容操作完成