伪集群及Docker集群搭建测试流程
- 1.伪集群搭建
- 1.1 环境
- 1.2 搭建
- 1.2.1 集群配置
- 1.2.2 生成其他5个节点配置
- 1.2.3 启动并验证节点状态
- 1.2.4 创建集群
- 1.2.5 集群信息
- 1.3 测试
- 2.Docker集群
- 2.1 环境
- 2.2 搭建
- 2.2.1 创建专用网络
- 2.2.2 生成配置文件
- 2.2.3 容器启动及验证
- 2.2.4 创建集群
- 2.2.5 删除容器及配置文件
- 2.3.测试
- 3.总结
1.伪集群搭建
1.1 环境
使用的是腾讯的云服务器1核心2G内存50G存储
,系统信息如下:
[root@tcloud ~]# rpm -qi centos-release
Name : centos-release
Version : 7
Release : 9.2009.1.el7.centos
Architecture: x86_64
Install Date: Tue 19 Jan 2021 06:23:16 PM CST
Group : System Environment/Base
Size : 44787
License : GPLv2
Signature : RSA/SHA256, Thu 03 Dec 2020 12:35:28 AM CST, Key ID 24c6a8a7f4a80eb5
Source RPM : centos-release-7-9.2009.1.el7.centos.src.rpm
Build Date : Mon 23 Nov 2020 11:08:41 PM CST
Build Host : x86-01.bsys.centos.org
Relocations : (not relocatable)
Packager : CentOS BuildSystem <http://bugs.centos.org>
Vendor : CentOS
Summary : CentOS Linux release file
Description :
CentOS Linux release files
1.2 搭建
- 每个Redis节点必须要有一个备机,例如搭建3个节点的集群就要有6个Redis实例。
- 数据按照slots分布式存储在不同的Redis节点上,节点中的数据可共享,可以动态调整数据的分布。
- 可扩展性强,可以动态增删节点 ,最多可扩展1000+节点。
1.2.1 集群配置
使用相对性能较好的5.0.14
版本进行搭建:
# 1.复制单机版并修改配置
cp -r ./redis-5.0.14 ./redis-5.0.14-0
# 修改【redis-5.0.14-0】的配置信息如下
################################## NETWORK #####################################
protected-mode no
port 6370
################################# GENERAL #####################################
daemonize yes
################################ REDIS CLUSTER ###############################
cluster-enabled yes
cluster-config-file nodes-6370.conf
cluster-node-timeout 15000
1.2.2 生成其他5个节点配置
# 2.复制5个节点并修改配置
cp -r ./redis-5.0.14-0 ./redis-5.0.14-1
cp -r ./redis-5.0.14-0 ./redis-5.0.14-2
cp -r ./redis-5.0.14-0 ./redis-5.0.14-3
cp -r ./redis-5.0.14-0 ./redis-5.0.14-4
cp -r ./redis-5.0.14-0 ./redis-5.0.14-5
# 修改每个节点的配置
sed 's/6370/6371/g' ./redis-5.0.14-1/redis.conf > ./redis-5.0.14-1/redis-6371.conf
sed 's/6370/6372/g' ./redis-5.0.14-2/redis.conf > ./redis-5.0.14-2/redis-6372.conf
sed 's/6370/6373/g' ./redis-5.0.14-3/redis.conf > ./redis-5.0.14-3/redis-6373.conf
sed 's/6370/6374/g' ./redis-5.0.14-4/redis.conf > ./redis-5.0.14-4/redis-6374.conf
sed 's/6370/6375/g' ./redis-5.0.14-5/redis.conf > ./redis-5.0.14-5/redis-6375.conf
1.2.3 启动并验证节点状态
# 3.启动6个节点并验证启动状态
./redis-5.0.14-0/bin/redis-server ./redis-5.0.14-0/redis.conf
./redis-5.0.14-1/bin/redis-server ./redis-5.0.14-1/redis-6371.conf
./redis-5.0.14-2/bin/redis-server ./redis-5.0.14-2/redis-6372.conf
./redis-5.0.14-3/bin/redis-server ./redis-5.0.14-3/redis-6373.conf
./redis-5.0.14-4/bin/redis-server ./redis-5.0.14-4/redis-6374.conf
./redis-5.0.14-5/bin/redis-server ./redis-5.0.14-5/redis-6375.conf
[root@tcloud local]# ps -ef | grep redis
root 1866 1 0 17:12 ? 00:00:00 ./redis-5.0.14-0/bin/redis-server 127.0.0.1:6370 [cluster]
root 1871 1 0 17:12 ? 00:00:00 ./redis-5.0.14-1/bin/redis-server 127.0.0.1:6371 [cluster]
root 1876 1 0 17:12 ? 00:00:00 ./redis-5.0.14-2/bin/redis-server 127.0.0.1:6372 [cluster]
root 1881 1 0 17:12 ? 00:00:00 ./redis-5.0.14-3/bin/redis-server 127.0.0.1:6373 [cluster]
root 1886 1 0 17:12 ? 00:00:00 ./redis-5.0.14-4/bin/redis-server 127.0.0.1:6374 [cluster]
root 1896 1 0 17:12 ? 00:00:00 ./redis-5.0.14-5/bin/redis-server 127.0.0.1:6375 [cluster]
root 1957 1522 0 17:12 pts/0 00:00:00 grep --color=auto redis
1.2.4 创建集群
# 4.创建集群
[root@tcloud redis-5.0.14-0]# ./bin/redis-cli --cluster create 127.0.0.1:6370 127.0.0.1:6371 127.0.0.1:6372 127.0.0.1:6373 127.0.0.1:6374 127.0.0.1:6375 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:6374 to 127.0.0.1:6370
Adding replica 127.0.0.1:6375 to 127.0.0.1:6371
Adding replica 127.0.0.1:6373 to 127.0.0.1:6372
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 60772cf691608a8ae8e6f519b1a95b52c84b2457 127.0.0.1:6370
slots:[0-5460] (5461 slots) master
M: ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 127.0.0.1:6371
slots:[5461-10922] (5462 slots) master
M: 6bb4e92cec0464c9969397eccaf1de1ce54b3e56 127.0.0.1:6372
slots:[10923-16383] (5461 slots) master
S: e793d2b868dbcd0b701bae10ea4d689b13a1c650 127.0.0.1:6373
replicates ffe9663d3fd1fc55c74c7bea6c70069fb742a69f
S: baae7ed422dc563bbbd22e1ac2d859ba4835dd63 127.0.0.1:6374
replicates 6bb4e92cec0464c9969397eccaf1de1ce54b3e56
S: 2fddd068ad1ae78d816d2932f012fae025e4e1f8 127.0.0.1:6375
replicates 60772cf691608a8ae8e6f519b1a95b52c84b2457
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:6370)
M: 60772cf691608a8ae8e6f519b1a95b52c84b2457 127.0.0.1:6370
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 6bb4e92cec0464c9969397eccaf1de1ce54b3e56 127.0.0.1:6372
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 127.0.0.1:6371
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: baae7ed422dc563bbbd22e1ac2d859ba4835dd63 127.0.0.1:6374
slots: (0 slots) slave
replicates 6bb4e92cec0464c9969397eccaf1de1ce54b3e56
S: 2fddd068ad1ae78d816d2932f012fae025e4e1f8 127.0.0.1:6375
slots: (0 slots) slave
replicates 60772cf691608a8ae8e6f519b1a95b52c84b2457
S: e793d2b868dbcd0b701bae10ea4d689b13a1c650 127.0.0.1:6373
slots: (0 slots) slave
replicates ffe9663d3fd1fc55c74c7bea6c70069fb742a69f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
1.2.5 集群信息
# 5.集群并查看节点数据
[root@tcloud redis-5.0.14-0]# ./bin/redis-cli -c -p 6370
127.0.0.1:6370> cluster nodes
6bb4e92cec0464c9969397eccaf1de1ce54b3e56 127.0.0.1:6372@16372 master - 0 1683192656954 3 connected 10923-16383
ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 127.0.0.1:6371@16371 master - 0 1683192657956 2 connected 5461-10922
60772cf691608a8ae8e6f519b1a95b52c84b2457 127.0.0.1:6370@16370 myself,master - 0 1683192655000 1 connected 0-5460
baae7ed422dc563bbbd22e1ac2d859ba4835dd63 127.0.0.1:6374@16374 slave 6bb4e92cec0464c9969397eccaf1de1ce54b3e56 0 1683192655951 5 connected
2fddd068ad1ae78d816d2932f012fae025e4e1f8 127.0.0.1:6375@16375 slave 60772cf691608a8ae8e6f519b1a95b52c84b2457 0 1683192655000 6 connected
e793d2b868dbcd0b701bae10ea4d689b13a1c650 127.0.0.1:6373@16373 slave ffe9663d3fd1fc55c74c7bea6c70069fb742a69f 0 1683192653000 4 connected
1.3 测试
# 延迟测试
./bin/redis-cli -c -p 6370 --intrinsic-latency 10
# 性能测试
./bin/redis-benchmark -h 127.0.0.1 -p 6370 -q
- 延迟测试结果
# 1
150971628 total runs (avg latency: 0.0662 microseconds / 66.24 nanoseconds per run).
Worst run took 117245x longer than the average latency.
# 2
149243181 total runs (avg latency: 0.0670 microseconds / 67.00 nanoseconds per run).
Worst run took 155735x longer than the average latency.
# 3
150936098 total runs (avg latency: 0.0663 microseconds / 66.25 nanoseconds per run).
Worst run took 87950x longer than the average latency.
- 性能测试结果
# 1
PING_INLINE: 62814.07
PING_BULK: 64143.68
SET: 61087.36
GET: 62344.14
INCR: 59665.87
LPUSH: 54945.05
RPUSH: 56785.91
LPOP: 55991.04
RPOP: 57208.24
SADD: 62932.66
HSET: 59347.18
SPOP: 63091.48
LPUSH (needed to benchmark LRANGE): 56211.35
LRANGE_100 (first 100 elements): 34614.05
LRANGE_300 (first 300 elements): 16131.63
LRANGE_500 (first 450 elements): 11568.72
LRANGE_600 (first 600 elements): 9214.89
MSET (10 keys): 54644.81
# 2
PING_INLINE: 61881.19
PING_BULK: 64267.35
SET: 61881.19
GET: 61804.70
INCR: 61387.36
LPUSH: 55555.56
RPUSH: 55370.98
LPOP: 56369.79
RPOP: 57736.72
SADD: 61614.29
HSET: 61387.36
SPOP: 63613.23
LPUSH (needed to benchmark LRANGE): 53792.36
LRANGE_100 (first 100 elements): 33681.38
LRANGE_300 (first 300 elements): 15637.22
LRANGE_500 (first 450 elements): 11828.72
LRANGE_600 (first 600 elements): 9221.69
MSET (10 keys): 52826.20
# 3
PING_INLINE: 63091.48
PING_BULK: 63572.79
SET: 61012.81
GET: 60569.35
INCR: 60459.49
LPUSH: 55463.12
RPUSH: 56850.48
LPOP: 55617.35
RPOP: 57570.52
SADD: 60790.27
HSET: 60168.47
SPOP: 62695.92
LPUSH (needed to benchmark LRANGE): 55679.29
LRANGE_100 (first 100 elements): 34435.26
LRANGE_300 (first 300 elements): 16064.26
LRANGE_500 (first 450 elements): 11496.90
LRANGE_600 (first 600 elements): 9180.21
MSET (10 keys): 53648.07
2.Docker集群
2.1 环境
[root@tcloud ~]# docker -v
Docker version 20.10.13, build a224086
Redis的官方Docker镜像地址:https://hub.docker.com/_/redis
2.2 搭建
2.2.1 创建专用网络
# 创建桥接网卡
docker network create redis --subnet 172.81.0.0/16
# 查看所有 网卡
docker network ls
# 查看网卡详情
docker network inspect NETWORKID
[
{
"Name": "redis",
"Id": "79a62c1104195f5f077665fcdd8d4e1c92697af7db227c1fd1a92219f22c2501",
"Created": "2023-05-05T10:38:24.604493511+08:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.81.0.0/16",
"Gateway": "172.81.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {},
"Labels": {}
}
]
2.2.2 生成配置文件
# 通过脚本创建6个Redis配置
# 这里一定要注意不要加 daemonize yes 否则无法启动
for port in $(seq 1 6); \
do \
mkdir -p /usr/local/redis-docker/node-${port}/conf
touch /usr/local/redis-docker/node-${port}/conf/redis.conf
cat << EOF >/usr/local/redis-docker/node-${port}/conf/redis.conf
bind 0.0.0.0
protected-mode no
port 637${port}
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip 172.81.0.1${port}
cluster-announce-port 637${port}
cluster-announce-bus-port 1637${port}
appendonly no
EOF
done
2.2.3 容器启动及验证
# 启动容器
for port in $(seq 1 6); \
do \
docker run -p 637${port}:637${port} -p 1637${port}:1637${port} --name redis-${port} \
-v /usr/local/redis-docker/node-${port}/data:/data \
-v /usr/local/redis-docker/node-${port}/conf/redis.conf:/etc/redis/redis.conf \
-d --net redis --ip 172.81.0.1${port} redis:5.0.14 redis-server /etc/redis/redis.conf
done
# 查看启动状态
[root@tcloud local]# docker ps | grep redis
a4e70ff2d840 redis:5.0.14 "docker-entrypoint.s…" 48 seconds ago Up 46 seconds 0.0.0.0:6376->6376/tcp, :::6376->6376/tcp, 0.0.0.0:16376->16376/tcp, :::16376->16376/tcp, 6379/tcp redis-6
854f8e023ce1 redis:5.0.14 "docker-entrypoint.s…" 49 seconds ago Up 47 seconds 0.0.0.0:6375->6375/tcp, :::6375->6375/tcp, 0.0.0.0:16375->16375/tcp, :::16375->16375/tcp, 6379/tcp redis-5
5165d1e61a7f redis:5.0.14 "docker-entrypoint.s…" 50 seconds ago Up 48 seconds 0.0.0.0:6374->6374/tcp, :::6374->6374/tcp, 0.0.0.0:16374->16374/tcp, :::16374->16374/tcp, 6379/tcp redis-4
f08dbda27366 redis:5.0.14 "docker-entrypoint.s…" 51 seconds ago Up 50 seconds 0.0.0.0:6373->6373/tcp, :::6373->6373/tcp, 0.0.0.0:16373->16373/tcp, :::16373->16373/tcp, 6379/tcp redis-3
e5e0feb9546a redis:5.0.14 "docker-entrypoint.s…" 52 seconds ago Up 51 seconds 0.0.0.0:6372->6372/tcp, :::6372->6372/tcp, 0.0.0.0:16372->16372/tcp, :::16372->16372/tcp, 6379/tcp redis-2
d8fae67a40b2 redis:5.0.14 "docker-entrypoint.s…" 54 seconds ago Up 52 seconds 0.0.0.0:6371->6371/tcp, :::6371->6371/tcp, 0.0.0.0:16371->16371/tcp, :::16371->16371/tcp, 6379/tcp redis-1
2.2.4 创建集群
# 创建集群
docker exec -it redis-1 /bin/sh
# 容器内操作
cd /usr/local/bin
redis-cli --cluster create 172.81.0.11:6371 172.81.0.12:6372 172.81.0.13:6373 172.81.0.14:6374 172.81.0.15:6375 172.81.0.16:6376 --cluster-replicas 1
# 打印出来的信息不再贴出
# 连接集群
cd /usr/local/redis-5.0.14
./bin/redis-cli -p 6371 -c
cluster nodes
# 退出容器
exit
2.2.5 删除容器及配置文件
# 停止
for port in $(seq 1 6); \
do \
docker stop redis-${port}
done
# 删除停止的容器
docker rm $(docker ps -qa)
# 删除配置及数据文件
rm -rf ./redis-docker
2.3.测试
# 延迟测试
./bin/redis-cli -c -p 6371 --intrinsic-latency 10
# 性能测试
./bin/redis-benchmark -h 172.81.0.1 -p 6371 -q
- 延迟测试结果
# 1
148915461 total runs (avg latency: 0.0672 microseconds / 67.15 nanoseconds per run).
Worst run took 172533x longer than the average latency.
# 2
148930804 total runs (avg latency: 0.0671 microseconds / 67.15 nanoseconds per run).
Worst run took 147203x longer than the average latency.
# 3
149999312 total runs (avg latency: 0.0667 microseconds / 66.67 nanoseconds per run).
Worst run took 119474x longer than the average latency.
- 性能测试结果
# 1
PING_INLINE: 49212.60
PING_BULK: 50327.12
SET: 48590.86
GET: 47846.89
INCR: 48923.68
LPUSH: 44863.16
RPUSH: 45106.00
LPOP: 45599.63
RPOP: 44682.75
SADD: 50377.83
HSET: 48239.27
SPOP: 50479.56
LPUSH (needed to benchmark LRANGE): 44345.89
LRANGE_100 (first 100 elements): 29691.21
LRANGE_300 (first 300 elements): 14858.84
LRANGE_500 (first 450 elements): 11119.76
LRANGE_600 (first 600 elements): 8946.94
MSET (10 keys): 43859.65
# 2
PING_INLINE: 49975.02
PING_BULK: 51493.30
SET: 47664.44
GET: 49358.34
INCR: 49358.34
LPUSH: 44822.95
RPUSH: 45146.73
LPOP: 44404.97
RPOP: 45703.84
SADD: 50125.31
HSET: 45829.52
SPOP: 49701.79
LPUSH (needed to benchmark LRANGE): 42643.92
LRANGE_100 (first 100 elements): 30349.01
LRANGE_300 (first 300 elements): 14641.29
LRANGE_500 (first 450 elements): 11111.11
LRANGE_600 (first 600 elements): 8964.59
MSET (10 keys): 44642.86
# 3
PING_INLINE: 50075.11
PING_BULK: 49900.20
SET: 48971.59
GET: 49164.21
INCR: 49925.11
LPUSH: 44130.62
RPUSH: 40950.04
LPOP: 40338.84
RPOP: 44111.16
SADD: 48449.61
HSET: 47687.18
SPOP: 49455.98
LPUSH (needed to benchmark LRANGE): 45004.50
LRANGE_100 (first 100 elements): 30637.26
LRANGE_300 (first 300 elements): 14788.52
LRANGE_500 (first 450 elements): 11183.18
LRANGE_600 (first 600 elements): 9005.76
MSET (10 keys): 45167.12
3.总结
我们跟之前的单机版本进行对比,延迟对比结果:
性能比对结果:
- 伪集群和在一台服务器上的Docker集群一样无法提升性能。