生产级Redis Cluster部署
- 环境准备
| 主机名 | IP地址 | 端口 | 描述 |
| redis-master | 192.168.1.51 | 7000 | redis-master01 |
| 7001 | redis-master02 | ||
| 7002 | redis-master03 | ||
| redis-slave | 192.168.1.52 | 8000 | redis-slave01 |
| 8001 | redis-slave02 | ||
| 8002 | redis-slave03 |
- 初始化环境
两台主机配置主机名，关闭防火墙和selinux
hostname redis-master #修改主机名字
Bash #刷新
iptables -F #关闭防火墙规则
setenforce 0
systemctl stop firewalld
- Redis环境部署
redis编译部署及基础优化（master）
yum -y install wget gcc gcc-c++ make tar openssl openssl-devel cmake
tar xf redis-4.0.10.tar.gz -C /usr/src/ #需有个redis-4.0.10.tar.gz的安装包
cd /usr/src/redis-4.0.10/
make
make MALLOC=jemalloc #避免碎片和可伸缩的并发支持
make PREFIX=/usr/local/redis install
mkdir -p /usr/local/redis/conf
优化redis程序命令（master）
cp sentinel.conf /usr/local/redis/conf/ #redis哨兵配置文件
cp src/redis-trib.rb /usr/local/redis/bin/ #redis-cluster的集群创建工具
ln -s /usr/local/redis/bin/* /usr/local/bin/
redis配置文件精简（master）
cp redis.conf /usr/local/redis/conf/
cd /usr/local/redis/
cp conf/redis.conf{,.bak}
egrep -v "^$|^#" conf/redis.conf.bak > conf/redis.conf
系统调优配置（master）
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo "* - nofile 10240" >> /etc/security/limits.conf --文件并发数
echo "net.core.somaxconn = 10240" >> /etc/sysctl.conf --监听队列
echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf -- 0内存不够，就会拒绝申请
1物理内存都允许分配给你，只要有内存就给你用，这样可以避免申请内存失败的问题。
sysctl -p
- Redis多实列配置
mkdir -p /data/redis-cluster （#master）
cd /data/redis-cluster
mkdir 7000 7001 7002
cp /usr/local/redis/conf/redis.conf /data/redis-cluster/7000/
cp /usr/local/redis/conf/redis.conf /data/redis-cluster/7001/
cp /usr/local/redis/conf/redis.conf /data/redis-cluster/7002/
在redis-slave从库上操作（slave）
mkdir -p /data/redis-cluster
cd /data/redis-cluster
mkdir 8000 8001 8002
将redis-master的配置文件复制到redis-slave主机上（master） （最好作完下列配置后再远程复制）
scp /data/redis-cluster/7000/redis.conf 192.168.1.52:/data/redis-cluster/8000/
scp /data/redis-cluster/7000/redis.conf 192.168.1.52:/data/redis-cluster/8001/
scp /data/redis-cluster/7000/redis.conf 192.168.1.52:/data/redis-cluster/8002/
修改redis-master和redis-slave的redis.conf配置文件
配置参考模板，修改配置文件的对应端口
vim 7000/redis.conf #红色字体一般都是要修改的 其它基本不要修改
cluster-enabled yes #启动集群这条要自己手动输入进去
bind 0.0.0.0
port 7000
pidfile /data/redis-cluster/7000/redis.pid
logfile "/data/redis-cluster/7000/redis.log"
dir /data/redis-cluster/7000/
tcp-backlog 1024
timeout 0
tcp-keepalive 0
daemonize yes
loglevel notice
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
slave-lazy-flush no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
启动redis-master多实例（master）
redis-server /data/redis-cluster/7001/redis.conf
redis-server /data/redis-cluster/7002/redis.conf
redis-server /data/redis-cluster/7000/redis.conf
netstat -antup | grep redis #查看端口筛选出redis服务的端口
tcp 0 0 0.0.0.0:7000 0.0.0.0:* LISTEN 5412/redis-server 0
tcp 0 0 0.0.0.0:7001 0.0.0.0:* LISTEN 5400/redis-server 0
tcp 0 0 0.0.0.0:7002 0.0.0.0:* LISTEN 5405/redis-server 0
tcp 0 0 0.0.0.0:17000 0.0.0.0:* LISTEN 5412/redis-server 0
tcp 0 0 0.0.0.0:17001 0.0.0.0:* LISTEN 5400/redis-server 0
tcp 0 0 0.0.0.0:17002 0.0.0.0:* LISTEN 5405/redis-server 0
- 构建Redis cluster 集群
编译安装高ruby版本，Ruby版本需要大于等于2.2.2（yum安装的不符合）（master）
tar xf ruby-2.5.1.tar.gz -C /usr/src/
cd /usr/src/ruby-2.5.1/
./configure && make && make install
ruby --version
在线安装ruby的redis扩展（master）
/usr/local/bin/gem install redis （#这里需要等上一会）
Fetching: redis-4.1.4.gem (100%)
Successfully installed redis-4.1.4
Parsing documentation for redis-4.1.4
Installing ri documentation for redis-4.1.4
Done installing documentation for redis after 1 seconds
1 gem installed
在redis-master上操作（master）
redis-trib.rb create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002
>>> Creating cluster
>>> Performing hash slots allocation on 3 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
M: 4e729d539ceb6b5b6bce86f4d6f3dbb6de1cb5ca 127.0.0.1:7000
slots:0-5460 (5461 slots) master
M: 6e3809c102d7cb3013ef8a89bc17f4d6ca1fc93a 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
M: 61b60ec318606a1c86770cee4da74e5b6962076b 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes #输入yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 4e729d539ceb6b5b6bce86f4d6f3dbb6de1cb5ca 127.0.0.1:7000
slots:0-5460 (5461 slots) master
0 additional replica(s)
M: 61b60ec318606a1c86770cee4da74e5b6962076b 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
0 additional replica(s)
M: 6e3809c102d7cb3013ef8a89bc17f4d6ca1fc93a 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
查看cluster集群的构建节点信息（master）
redis-cli -p 7000 cluster nodes
61b60ec318606a1c86770cee4da74e5b6962076b 127.0.0.1:7002@17002 master - 0 1589298445481 3 connected 10923-16383
4e729d539ceb6b5b6bce86f4d6f3dbb6de1cb5ca 127.0.0.1:7000@17000 myself,master - 0 1589298444000 1 connected 0-5460
6e3809c102d7cb3013ef8a89bc17f4d6ca1fc93a 127.0.0.1:7001@17001 master - 0 1589298446490 2 connected 5461-10922
ls /data/redis-cluster/7000/
nodes.conf redis.conf redis.log redis.pid
cat /data/redis-cluster/7000/nodes.conf
61b60ec318606a1c86770cee4da74e5b6962076b 127.0.0.1:7002@17002 master - 0 1589298205185 3 connected 10923-16383
4e729d539ceb6b5b6bce86f4d6f3dbb6de1cb5ca 127.0.0.1:7000@17000 myself,master - 0 0 1 connected 0-5460
6e3809c102d7cb3013ef8a89bc17f4d6ca1fc93a 127.0.0.1:7001@17001 master - 0 1589298205085 2 connected 5461-10922
vars currentEpoch 3 lastVoteEpoch 0
使用redis-cli去操作集群，需要加入-c参数
在redis-master上操作（master）
redis-cli -c -p 7000
127.0.0.1:7000> set name crushlinux
-> Redirected to slot [5798] located at 127.0.0.1:7001 #被分配到了7001端口
OK
127.0.0.1:7001> set age 18
-> Redirected to slot [741] located at 127.0.0.1:7000
OK
127.0.0.1:7000> exit
去中心化，随意一个入口
redis-cli -c -p 7000 get name
"crushlinux"
redis-cli -c -p 7001 get name
"crushlinux"
redis-cli -c -p 7002 get name
"crushlinux"
在redis-slave上操作（slave）
redis-cli -h 192.168.1.51 -c -p 7000 get name
Could not connect to Redis at 127.0.0.1:7001: Connection refused #被拒绝了
Could not connect to Redis at 127.0.0.1:7001: Connection refused
redis-cli -h 192.168.1.51 -c -p 7001 get name
"crushlinux" #只有7001成功了
redis-cli -h 192.168.1.51 -c -p 7002 get name
Could not connect to Redis at 127.0.0.1:7001: Connection refused #被拒绝了
Could not connect to Redis at 127.0.0.1:7001: Connection refused
以远程的方式访问redis-cluster时，被拒绝是因为我们创建集群的时候，是以127.0.0.1的本地IP方式创建的，因此只有本地访问时cluster集群才能发挥作用。
远程访问7001端口时，之所以能成功，是因为数据本来就是存在了监听7001端口的redis上，所以我们不加-c也能访问成功。
redis-cli -h 192.168.1.51 -p 7001 get name
"crushlinux"
- Cluster生产集群部署
在redis-master上操作（master）
ls /data/redis-cluster/7000
nodes.conf redis.conf redis.log redis.pid
ls /data/redis-cluster/7001
nodes.conf redis.conf redis.log redis.pid
ls /data/redis-cluster/7002
nodes.conf redis.conf redis.log redis.pid
删除cluster集群配置文件（master）
redis-cli -p 7000 shutdown
redis-cli -p 7001 shutdown
redis-cli -p 7002 shutdown #先暂停再删除
rm -rf /data/redis-cluster/7000/nodes.conf
rm -rf /data/redis-cluster/7001/nodes.conf
rm -rf /data/redis-cluster/7002/nodes.conf
启动redis-server（master）
redis-server /data/redis-cluster/7000/redis.conf
redis-server /data/redis-cluster/7001/redis.conf
redis-server /data/redis-cluster/7002/redis.conf
重新创建redis-cluster集群（master）
redis-trib.rb create 192.168.1.51:7000 192.168.1.51:7001 192.168.1.51:7002
>>> Creating cluster
>>> Performing hash slots allocation on 3 nodes...
Using 3 masters:
192.168.1.51:7000
192.168.1.51:7001
192.168.1.51:7002
M: a4f23ae3a3fbbfab8c9fcd22f499d96badff9434 192.168.1.51:7000
slots:0-5460 (5461 slots) master
M: d41141bbb9559a0d1a7df036ed0bca190603d786 192.168.1.51:7001
slots:5461-10922 (5462 slots) master
M: e6421050008242d4e98090c0d45df98edcd66d5b 192.168.1.51:7002
slots:10923-16383 (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
>>> Performing Cluster Check (using node 192.168.1.51:7000)
M: a4f23ae3a3fbbfab8c9fcd22f499d96badff9434 192.168.1.51:7000
slots:0-5460 (5461 slots) master
0 additional replica(s)
M: d41141bbb9559a0d1a7df036ed0bca190603d786 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
0 additional replica(s)
M: e6421050008242d4e98090c0d45df98edcd66d5b 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
ps -ef | grep cluster | grep -v grep
root 20005 1 0 23:54 ? 00:00:00 redis-server 0.0.0.0:7000 [cluster]
root 20010 1 0 23:54 ? 00:00:00 redis-server 0.0.0.0:7001 [cluster]
root 20015 1 0 23:54 ? 00:00:00 redis-server 0.0.0.0:7002 [cluster]
在redis-slave上进行远程连接cluster集群测试（slave）
redis-cli -h 192.168.1.51 -c -p 7000 set name crushlinux
OK
redis-cli -h 192.168.1.51 -c -p 7001 get name
"crushlinux"
redis-cli -h 192.168.1.51 -c -p 7002 get name
"crushlinux"
redis-cli -h 192.168.1.51 -c -p 7001
192.168.1.51:7001> get name
"crushlinux"
192.168.1.51:7001> set age 18
-> Redirected to slot [741] located at 192.168.1.51:7000
OK
192.168.1.51:7000> get age
"18"
192.168.1.51:7000> exit
批量导入数据观察key的集群分布情况（slave）（导入量过大需等一会）
for line in `seq -w 10000`;do redis-cli -h 192.168.1.51 -p 7000 -c set key3_${line} value_${line};done
分析cluster集群key的节点分布情况
redis-cli -h 192.168.1.51 -p 7000 info Keyspace
# Keyspace
db0:keys=3338,expires=0,avg_ttl=0
redis-cli -h 192.168.1.51 -p 7001 info Keyspace
# Keyspace
db0:keys=3354,expires=0,avg_ttl=0
redis-cli -h 192.168.1.51 -p 7002 info Keyspace
# Keyspace
db0:keys=3310,expires=0,avg_ttl=0
三个节点个数大概平均（10000）
- Cluster集群故障切换
启动redis-slave上所有的从库（slave）
redis-server /data/redis-cluster/8000/redis.conf
redis-server /data/redis-cluster/8001/redis.conf
redis-server /data/redis-cluster/8002/redis.conf
netstat -lnpt | grep redis
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 30034/redis-server
tcp 0 0 0.0.0.0:8001 0.0.0.0:* LISTEN 30039/redis-server
tcp 0 0 0.0.0.0:8002 0.0.0.0:* LISTEN 30044/redis-server
tcp 0 0 0.0.0.0:18000 0.0.0.0:* LISTEN 30034/redis-server
tcp 0 0 0.0.0.0:18001 0.0.0.0:* LISTEN 30039/redis-server
tcp 0 0 0.0.0.0:18002 0.0.0.0:* LISTEN 30044/redis-server
redis-cluster集群从库的添加，在redis-master上操作
添加第一组从服务器（master）
redis-trib.rb add-node --slave 192.168.1.52:8000 192.168.1.51:7000
添加第二组从服务器（master）
redis-trib.rb add-node --slave 192.168.1.52:8001 192.168.1.51:7001
添加第三组从服务器（master）
redis-trib.rb add-node --slave 192.168.1.52:8002 192.168.1.51:7002
查看集群所有节点的信息（master）
redis-cli -p 7000 cluster nodes
slave从库的读写测试（slave）
redis-cli -h 192.168.1.52 -c -p 8000
192.168.1.52:8000> set address beijing
-> Redirected to slot [3680] located at 192.168.1.51:7000
OK
192.168.1.51:7000> exit
redis-cli -h 192.168.1.52 -c -p 8000 get address
"beijing"
redis-cli -h 192.168.1.52 -c -p 8001 get address
"beijing"
redis-cli -h 192.168.1.52 -c -p 8002 get address
"beijing"
redis-cli -h 192.168.1.51 -c -p 7000 get address
"beijing"
redis-cli -h 192.168.1.51 -c -p 7001 get address
"beijing"
redis-cli -h 192.168.1.51 -c -p 7002 get address
"beijing"
通过测试，我们发现redis的cluster集群不论是主库还是从库都可以进行set和get。因此，在使用中我们就没必要都去主库了。
查看主从cluster集群key的分布情况（slave）
redis-cli -h 192.168.1.52 -c -p 8000 info Keyspace
# Keyspace
db0:keys=3339,expires=0,avg_ttl=0
redis-cli -h 192.168.1.52 -c -p 8001 info Keyspace
# Keyspace
db0:keys=3354,expires=0,avg_ttl=0
redis-cli -h 192.168.1.52 -c -p 8002 info Keyspace
# Keyspace
db0:keys=3310,expires=0,avg_ttl=0
redis-cli -h 192.168.1.51 -c -p 7000 info Keyspace
# Keyspace
db0:keys=3339,expires=0,avg_ttl=0
redis-cli -h 192.168.1.51 -c -p 7001 info Keyspace
# Keyspace
db0:keys=3354,expires=0,avg_ttl=0
redis-cli -h 192.168.1.51 -c -p 7002 info Keyspace
# Keyspace
db0:keys=3310,expires=0,avg_ttl=0
redis集群的主从自动切换，主库挂掉后，从自动变为主
手动切换主从命令cluster failover（slave）
[root@redis-slave ~]# redis-cli -h 192.168.1.51 -p 7000 shutdown
[root@redis-slave ~]# redis-cli -h 192.168.1.52 -p 8000 cluster nodes
手动down掉了7000端口的redis-server。从信息得知，7000挂了，8000被切换成了master
重新启动7000端口的server，再次查看（master）
[root@redis-master ~]# redis-server /data/redis-cluster/7000/redis.conf
[root@redis-slave ~]# redis-cli -h 192.168.1.52 -p 8000 cluster nodes
从信息得知：7000的redis启动以后，变成了slave。
手动将redis-server 7000端口重新切换成主库
[root@redis-slave ~]# redis-cli -h 192.168.1.51 -c -p 7000 cluster failover
OK
[root@redis-slave ~]# redis-cli -h 192.168.1.52 -p 8000 cluster nodes
删除节点
[root@redis-master ~]# redis-cli -h 192.168.1.52 8000 cluster node
[root@redis-master ~]# redis-trib.rb del-node 192.168.1.52:8001
[root@redis-master ~]# redis-cli -h 192.168.1.52 -p 8000 cluster nodes
最后跟着的是这个节点的ID，可在redis-cli终端中使用cluster nodes查看。
修复节点
[root@redis-master ~]# redis-trib.rb add-node --slave 192.168.1.52:8001
清空8001节点现有数据
[root@redis-slave ~]#kill -9 67708
[root@redis-slave ~]# rm -rf /data/redis-cluster/8001/dump.rdb
[root@redis-slave ~]# rm -rf /data/redis-cluster/8001/nodes.conf
[root@redis-slave ~]# rm -rf /data/redis-cluster/8001/redis.pid
[root@redis-slave ~]# redis-server /data/redis-cluster/8001/redis.conf
[root@redis-master ~]#redis-trib.rb add-node --slave 192.168.1.52:8001 192.168.1.51:7001
[root@redis-master ~]# redis-cli -h 192.168.1.52 -p 8000 cluster nodes
不足之处，还请各位大佬担待
