kafka 3.5.0 raft协议安装

前言

最近做项目，需要使用kafka进行通信，且只能使用kafka，笔者没有测试集群，就自己搭建了kafka集群，实际上笔者在很早之前就搭建了，因为当时还是zookeeper（简称ZK）注册元数据，现在新版kafka（3.0.0开始）已经自带了元数据能力（使用raft协议）减少了kafka对zk的依赖性。笔者在查询资料发现，说jdk至少jdk11，实测jdk8也能运行，且并不需要网上说的3+4节点，3+3即可，当然理论上broker节点越多越好，但是元数据节点建议3、5个最合适，raft的过半一致性和容错性的综合取舍。

准备

准备kafka安装包：Apache Kafka

笔者使用的kafka 3.5.0和scala 2.13，采用3台虚拟机，当然容器也不是不行，注意持久化pv pvc和配置的管理（ip换成域名，dns的切换支持），中间件建议使用虚拟机，可以降低很多容错性。

jdk使用open jdk，配置java_home和path，以Ubuntu为例

sudo apt install openjdk-8-jdk-headless

以macOS为例，创建一个ubuntu-server 最小安装的虚拟机（vmware，毕竟个人使用不要钱），然后安装openssh 和 openjdk，然后shutdown now

网络选择桥接，相当于一台“真实在”网络上的一台物理机

这样就得到了

192.168.0.108

192.168.0.107

192.168.0.106

3台虚拟机

步骤

先看kafka集群的架构图，实际上安装的过程就是架构图的执行过程

从图中可以看出已经没有zk的存在了，从kafka节点自己管理元数据，通过raft协议选主的方式。

1. kafka的准备

上传kafka安装包，必须是二进制安装包，不要源码包，编译比较麻烦，然后解压

tar -zxvf kafka_2.13-3.5.0.gz

查看配置目录会发现

明显多了kraft的配置目录，那么如果使用kafka raft元数据中心，则需要修改kraft目录，启动时指定kraft目录的配置

2. 配置修改

raft协议实际上跟zk差不多，使用raft协议的中间件就太多了，但是本质上每个节点都需要一个唯一id，zk也是如此，所以kafka kraft就相当于集成的zk。

在kraft下的有3个文件文件，其中启动相关的是server.properties中

执行配置修改


# The role of this server. Setting this puts us in KRaft mode
process.roles=broker,controller

# The node id associated with this instance's roles
node.id=1

# The connect string for the controller quorum
controller.quorum.voters=1@localhost:9093

############################# Socket Server Settings #############################

# The address the socket server listens on.
# Combined nodes (i.e. those with `process.roles=broker,controller`) must list the controller listener here at a minimum.
# If the broker listener is not defined, the default listener will use a host name that is equal to the value of java.net.InetAddress.getCanonicalHostName(),
# with PLAINTEXT listener name, and port 9092.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092,CONTROLLER://:9093

# Name of listener used for communication between brokers.
inter.broker.listener.name=PLAINTEXT

# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for "listeners".
advertised.listeners=PLAINTEXT://localhost:9092

# A comma-separated list of the names of the listeners used by the controller.
# If no explicit mapping set in `listener.security.protocol.map`, default will be using PLAINTEXT protocol
# This is required if running in KRaft mode.
controller.listener.names=CONTROLLER

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/tmp/kraft-combined-logs

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

每一行都有注释，重点关注

笔者设定

192.168.0.106 nodeid 1

192.168.0.107 nodeid 2

192.168.0.108 nodeid 3

至此配置基本上完成，同理一个节点可以同时是controller和broker，也可以仅仅是controller或者broker，因为controller的负载比较轻，所以一般是和broker一起。其中有个log.dir这个的路径是下面元数据生成的路径（选主）和数据事务日志，索引日志的存储目录

3. 启动

1. 生成uuid

任意找一个节点执行：

./kafka-storage.sh random-uuid

每次执行uuid会不一样，这个uuid标识是一个集群，所以所有节点公用一个uuid，不要每个节点重新生成，会识别不了

然后执行format，如下标红是我生成的，这个每次不是固定的

./kafka-storage.sh format -t gZzkfRm4T1y8wSAY-ZNG5Q -c ../config/kraft/server.properties

格式化配置文件，同步其他节点

配置文件有什么变化？在日志配置的目录下出现

关键还是meta的文件，有集群id和节点id，版本号，这个对启动至关重要。

即在上面的log.dir的目录生成，所以尽量不能使用临时目录

2. 启动

启动就很简单了，使用刚刚配置的server.properties执行启动即可

./kafka-server-start.sh -daemon ../config/kraft/server.properties

不过为了方便查看启动日志，建议执行日志的console文件输出

先看事务日志和索引

验证

验证很简单，查看bin同级目录下的日志即可

日志带有[2025-02-08 08:34:12,286] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)

如果生成用途可以安装kafka的控制台，kafka-ui，不过我这里就不安装了，因为docker安装比较容易。

总结

kafka从3.0.0开始推出了raft模式的元数据中心，实际上类似zk，kafka自己命名kraft。使用这种方式搭建kafka集群将不再需要zk，同理，kafka的集群的每个节点可以同时是broker和controller（以前zk充当），也可以是单独的broker，controller（负载不重，不建议单独controller，跟zk没区别），官方说明需要jdk11及以上，实测jdk8可以运行，但是生成建议严格按照官方标定的jdk执行，jdk是向下兼容的，但是不确定是否会涉及新api或新特性的使用。

另外实际使用中，可能会涉及使用iptables做nat限制kafka的连接方，比如在kafka节点通过iptables限制发送者或者消费端的ip

iptables -t nat -A PREROUTING -p tcp -m tcp --dport 9093 -j DNAT --to-destination kafkaxxx:9093

kafkaxxx --- 指定的是 Kafka 服务所在的机器地址

如果kafka是对接方提供，则在nat打通时，需要客户端连接的服务器也执行iptables，否则可能出现连接kafka正常，但是不能消费。

iptables -t nat -A POSTROUTING -p tcp -m tcp --dport 9093 -j SNAT --to-source natxxx

natxxx --- 指定的是配置 iptables 的本机的地址