RocketMQ读写分离实战

这里是weihubeats,觉得文章不错可以关注公众号小奏技术，文章首发。拒绝营销号，拒绝标题党

背景

继上次分析RocketMQ线上各种system busy

比较合理能提升性能和缓解system busy的方法就是修改broker的配置为transientStorePoolEnable = true

今天我们就来实战下

RocketMQ读写分离

RocketMQ实现了内存级别的读写分离。

想要开启很简单，我们在broker的配置文件中添加如下配置

transientStorePoolEnable = true

默认是关闭的

开启后需要注意transientStorePoolSize的大小，默认是5，mappedFileSizeCommitLog大小默认是1G

private int mappedFileSizeCommitLog = 1024 * 1024 * 1024;

所以如果我们要开启transientStorePoolEnable，然后其他的不配置的话，我们至少需要预留5g的堆外内存给RocketMQ使用

比如我们机器是16G，那么JVM的大小最多能给的是16- 5 = 11 G,实际netty也是使用堆外内存传输的，所以实际最多给到的JVM内存大小可能是8-9G

实现原理

开启 TransientStorePool 后，消息写入时将写入ByteBuffer.allocateDirect 申请的堆外内存中，由异步刷盘线程写入 fileChannel 中（Page Cache），最后进行进行刷盘。消息读取时,只会从 Page Cache 读取

所以如果消息未刷盘到Page Cache，broker宕机了，消息就会丢失。

优缺点

优点：

主要是性能的提升:写数据是完全写内存，速度相较于写Page Cache更快,读写分离，降低锁的占用。

缺点：

主要是broker宕机，内存中的消息没有刷盘到Page Cache消息会丢失。其次是消耗更多的内存

实战

本次我们将使用RocketMQ的基准测试脚本分别测试开启读写分离和不开启读写分离的性能。

jvm参数

-server -Xms8g -Xmx8g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0 -verbose:gc -Xloggc:/dev/shm/rmq_srv_gc_%p_%t.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m -XX:-OmitStackTraceInFastThrow -XX:-UseLargePages

broker机器配置

4核16G

压测脚本

主要是官方提供的基准测试脚本

生产者启动参数

JAVA_HOME=/usr sh producer.sh -n 192.168.1.1:9876 -t xiazou-topic -s 3072

消费者启动参数

JAVA_HOME=/usr sh consumer.sh -n 192.168.1.1:9876 -t xiazou-topic -g gid_xiaozou

压测方式，一个机器同时执行两个脚本，进行生产和消费

不开启读写分离

机器内存使用在8G左右

生产

benchmark git:(master) topic: xiazou-topic, threadCount: 64, messageSize: 3072, keyEnable: false, propertySize: 0, tagCount: 0, traceEnable: false, aclEnable: false, messageQuantity: 0, delayEnable: false, delayLevel: 1, asyncEnable: false
Current Time: 2023-10-17 11:59:31,297 | Send TPS: 35634 | Max RT(ms): 421 | Average RT(ms):   1.796 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 11:59:41,297 | Send TPS: 39868 | Max RT(ms): 421 | Average RT(ms):   1.605 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 11:59:51,297 | Send TPS: 37434 | Max RT(ms): 421 | Average RT(ms):   1.709 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:01,297 | Send TPS: 26596 | Max RT(ms): 421 | Average RT(ms):   2.407 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:11,297 | Send TPS: 35836 | Max RT(ms): 421 | Average RT(ms):   1.786 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:21,297 | Send TPS: 36223 | Max RT(ms): 421 | Average RT(ms):   1.767 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:31,297 | Send TPS: 35910 | Max RT(ms): 421 | Average RT(ms):   1.782 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:41,297 | Send TPS: 34397 | Max RT(ms): 421 | Average RT(ms):   1.860 | Send Failed: 0 | Response Failed: 0

消费

➜  benchmark git:(master) topic: xiazou-topic, threadCount 20, group: gid_xiaozou, suffix: false, filterType: null, expression: null, msgTraceEnable: false, aclEnable: false
Consumer Started.
Current Time: 2023-10-17 11:59:31,297 | Consume TPS: 35822 | AVG(B2C) RT(ms): 2.776 | AVG(S2C) RT(ms): 2.536 | MAX(B2C) RT(ms): 25 | MAX(S2C) RT(ms): 25 | Consume Fail: 0
Current Time: 2023-10-17 11:59:41,297 | Consume TPS: 36315 | AVG(B2C) RT(ms):  2.741 | AVG(S2C) RT(ms):  2.505 | MAX(B2C) RT(ms): 32 | MAX(S2C) RT(ms): 32 | Consume Fail: 0
Current Time: 2023-10-17 11:59:51,297 | Consume TPS: 36315 | AVG(B2C) RT(ms):  2.680 | AVG(S2C) RT(ms):  2.449 | MAX(B2C) RT(ms): 43 | MAX(S2C) RT(ms): 30 | Consume Fail: 0

生产和消费消息QPS在3.5w左右

开启读写分析

机器内存使用在12G左右

生产

benchmark git:(master) topic: xiazou-topic, threadCount: 64, messageSize: 3072, keyEnable: false, propertySize: 0, tagCount: 0, traceEnable: false, aclEnable: false, messageQuantity: 0, delayEnable: false, delayLevel: 1, asyncEnable: false
Current Time: 2023-10-17 13:59:31,297 | Send TPS: 27799 | Max RT(ms): 163 | Average RT(ms):   2.302 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 13:59:41,297 | Send TPS: 34444 | Max RT(ms): 163 | Average RT(ms):   1.858 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 13:59:51,297 | Send TPS: 37879 | Max RT(ms): 163 | Average RT(ms):   1.689 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:01,297 | Send TPS: 37457 | Max RT(ms): 163 | Average RT(ms):   1.709 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:11,297 | Send TPS: 35222 | Max RT(ms): 163 | Average RT(ms):   1.818 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:21,297 | Send TPS: 41495 | Max RT(ms): 163 | Average RT(ms):   1.542 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:31,297 | Send TPS: 41249 | Max RT(ms): 163 | Average RT(ms):   1.550 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:41,297 | Send TPS: 41962 | Max RT(ms): 163 | Average RT(ms):   1.525 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:51,297 | Send TPS: 39707 | Max RT(ms): 523 | Average RT(ms):   1.612 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:01,297 | Send TPS: 43444 | Max RT(ms): 523 | Average RT(ms):   1.473 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:11,297 | Send TPS: 42391 | Max RT(ms): 523 | Average RT(ms):   1.510 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:21,297 | Send TPS: 41915 | Max RT(ms): 523 | Average RT(ms):   1.527 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:31,297 | Send TPS: 42221 | Max RT(ms): 523 | Average RT(ms):   1.516 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:41,297 | Send TPS: 42480 | Max RT(ms): 523 | Average RT(ms):   1.506 | Send Failed: 0 | Response Failed: 0

消费

➜  benchmark git:(master) topic: xiazou-topic, threadCount 20, group: gid_xiaozou, suffix: false, filterType: null, expression: null, msgTraceEnable: false, aclEnable: false
Consumer Started.
Current Time: 2023-10-17 13:59:36,467 | Consume TPS: 45646 | AVG(B2C) RT(ms): 2135.170 | AVG(S2C) RT(ms): 2135.517 | MAX(B2C) RT(ms): 6164 | MAX(S2C) RT(ms): 6160 | Consume Fail: 0
Current Time: 2023-10-17 13:59:46,467 | Consume TPS: 38940 | AVG(B2C) RT(ms):  60.442 | AVG(S2C) RT(ms):  61.163 | MAX(B2C) RT(ms): 239 | MAX(S2C) RT(ms): 240 | Consume Fail: 0
Current Time: 2023-10-17 13:59:56,467 | Consume TPS: 38094 | AVG(B2C) RT(ms):  39.721 | AVG(S2C) RT(ms):  40.450 | MAX(B2C) RT(ms): 239 | MAX(S2C) RT(ms): 239 | Consume Fail: 0
Current Time: 2023-10-17 14:00:06,467 | Consume TPS: 33780 | AVG(B2C) RT(ms):  52.946 | AVG(S2C) RT(ms):  53.575 | MAX(B2C) RT(ms): 285 | MAX(S2C) RT(ms): 286 | Consume Fail: 0
Current Time: 2023-10-17 14:00:16,467 | Consume TPS: 40338 | AVG(B2C) RT(ms):  48.543 | AVG(S2C) RT(ms):  49.286 | MAX(B2C) RT(ms): 248 | MAX(S2C) RT(ms): 248 | Consume Fail: 0
Current Time: 2023-10-17 14:00:26,467 | Consume TPS: 41998 | AVG(B2C) RT(ms):  66.248 | AVG(S2C) RT(ms):  67.008 | MAX(B2C) RT(ms): 238 | MAX(S2C) RT(ms): 238 | Consume Fail: 0
Current Time: 2023-10-17 14:00:36,467 | Consume TPS: 41713 | AVG(B2C) RT(ms):  72.638 | AVG(S2C) RT(ms):  73.397 | MAX(B2C) RT(ms): 231 | MAX(S2C) RT(ms): 232 | Consume Fail: 0
Current Time: 2023-10-17 14:00:46,467 | Consume TPS: 39853 | AVG(B2C) RT(ms):  53.383 | AVG(S2C) RT(ms):  54.047 | MAX(B2C) RT(ms): 524 | MAX(S2C) RT(ms): 234 | Consume Fail: 0
Current Time: 2023-10-17 14:00:56,467 | Consume TPS: 42399 | AVG(B2C) RT(ms):  59.808 | AVG(S2C) RT(ms):  60.542 | MAX(B2C) RT(ms): 242 | MAX(S2C) RT(ms): 243 | Consume Fail: 0
Current Time: 2023-10-17 14:01:06,467 | Consume TPS: 43756 | AVG(B2C) RT(ms):  75.232 | AVG(S2C) RT(ms):  75.986 | MAX(B2C) RT(ms): 252 | MAX(S2C) RT(ms): 253 | Consume Fail: 0
Current Time: 2023-10-17 14:01:16,467 | Consume TPS: 42434 | AVG(B2C) RT(ms):  57.688 | AVG(S2C) RT(ms):  58.413 | MAX(B2C) RT(ms): 245 | MAX(S2C) RT(ms): 245 | Consume Fail: 0
Current Time: 2023-10-17 14:01:26,467 | Consume TPS: 42152 | AVG(B2C) RT(ms):  69.523 | AVG(S2C) RT(ms):  70.260 | MAX(B2C) RT(ms): 237 | MAX(S2C) RT(ms): 237 | Consume Fail: 0
Current Time: 2023-10-17 14:01:36,467 | Consume TPS: 42024 | AVG(B2C) RT(ms):  53.667 | AVG(S2C) RT(ms):  54.385 | MAX(B2C) RT(ms): 246 | MAX(S2C) RT(ms): 246 | Consume Fail: 0
Current Time: 2023-10-17 14:01:46,467 | Consume TPS: 41974 | AVG(B2C) RT(ms):  71.306 | AVG(S2C) RT(ms):  72.068 | MAX(B2C) RT(ms): 250 | MAX(S2C) RT(ms): 251 | Consume Fail: 0

生产和消费消息QPS在4w左右，相比不开启读写分离QPS生产和消费提高了5000QPS(大概提升了15%的性能),但是RT变长了(每条平均耗时)