KafKa 集群【docker compose】

news2024/10/19 0:13:32

文章目录

主机准备
部署
- 编辑 docker-compose.yml
- - controller
  - broker
  - 生成cluster_id
- 一篇完整的 docker-compose.yml 文件
- 查看集群状态
- - 使用 kafka-ui 查看
  - - 拉取 kafka-ui
    - 添加集群
    - 查看集群状态
  - 使用命令行查看
配置讲解
- controller
- broker

主机准备

IP	controller id	broker id
192.168.142.157	1	5
192.168.142.156	2	6
192.168.142.155	3	7
192.168.142.158	4	8

部署

创建文件夹

mkdir -p /docker/kafka/{data,data1,data2}

chmod 777 -R {data,data1,data2}

编辑 docker-compose.yml

controller

docker-compose.yml

services:
  controller:
    image: bitnami/kafka:3.8.0
    container_name: controller
    network_mode: host
    restart: always
    volumes:
      - /docker/kafka/data:/data
      - /etc/localtime:/etc/localtime:ro
    environment:
      KAFKA_CFG_NODE_ID: 1
      KAFKA_CFG_PROCESS_ROLES: controller
      KAFKA_CFG_LISTENERS: CONTROLLER://192.168.142.157:9093
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@192.168.142.157:9093,2@192.168.142.156:9093,3@192.168.142.155:9093,4@192.168.142.158:9093
      KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CFG_LOG_DIRS: /data

broker

image: bitnami/kafka:3.8.0
    container_name: broker
    network_mode: host
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /docker/kafka/data1:/data1
      - /docker/kafka/data2:/data2
    environment:
      KAFKA_CFG_NODE_ID: 5
      KAFKA_CFG_PROCESS_ROLES: broker
      KAFKA_CFG_LISTENERS: PLAINTEXT://192.168.142.157:9092
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.142.157:9092
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@192.168.142.157:9093,2@192.168.142.156:9093,3@192.168.142.155:9093,4@192.168.142.158:9093
      KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CFG_LOG_DIRS: "/data1,/data2"

所有的主机这一段基本都是一样的只有 id 不同，id 可以根据上面表格进行修改

生成cluster_id

root@master:/docker/kafka# docker run -it --rm bitnami/kafka:3.8.0 kafka-storage.sh random-uuid
kafka 06:43:36.79 INFO  ==>
kafka 06:43:36.80 INFO  ==> Welcome to the Bitnami kafka container
kafka 06:43:36.80 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 06:43:36.80 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 06:43:36.80 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
kafka 06:43:36.80 INFO  ==>

PavLImfXSqaOOdrMgq1teA

拿到 id

PavLImfXSqaOOdrMgq1teA

新增到 docker-compose.yml 文件中

一篇完整的 docker-compose.yml 文件

services:
  controller:
    image: bitnami/kafka:3.8.0
    container_name: controller
    network_mode: host
    restart: always
    volumes:
      - /docker/kafka/data:/data
      - /etc/localtime:/etc/localtime:ro
    environment:
      KAFKA_CFG_NODE_ID: 1
      KAFKA_CFG_PROCESS_ROLES: controller
      KAFKA_CFG_LISTENERS: CONTROLLER://192.168.142.157:9093
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@192.168.142.157:9093,2@192.168.142.156:9093,3@192.168.142.155:9093,4@192.168.142.158:9093
      KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CFG_LOG_DIRS: /data
      KAFKA_KRAFT_CLUSTER_ID: PavLImfXSqaOOdrMgq1teA
  broker:
    image: bitnami/kafka:3.8.0
    container_name: broker
    network_mode: host
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /docker/kafka/data1:/data1
      - /docker/kafka/data2:/data2
    environment:
      KAFKA_CFG_NODE_ID: 5
      KAFKA_CFG_PROCESS_ROLES: broker
      KAFKA_CFG_LISTENERS: PLAINTEXT://192.168.142.157:9092
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://192.168.142.157:9092
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@192.168.142.157:9093,2@192.168.142.156:9093,3@192.168.142.155:9093,4@192.168.142.158:9093
      KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CFG_LOG_DIRS: "/data1,/data2"
      KAFKA_KRAFT_CLUSTER_ID: PavLImfXSqaOOdrMgq1teA

启动

docker compose up -d

查看启动状态

docker ps -a | grep Up | wc -l
2

查看集群状态

使用 kafka-ui 查看

拉取 kafka-ui

只需要在一台主机上部署 ui 就可以了

docker run -d -p 8080:8080 -e DYNAMIC_CONFIG_ENABLED=true provectuslabs/kafka-ui:v0.7.1

通过 ip + 8080 端口访问

在这里插入图片描述

添加集群

在这里插入图片描述

查看集群状态

在这里插入图片描述

使用命令行查看

root@master:/docker/kafka# docker exec -it broker ./opt/bitnami/kafka/bin/kafka-metadata-quorum.sh --bootstrap-controller 192.168.142.157:9093,192.168.142.156:9093,192.168.142.155:9093,192.168.142.158:9093 describe --status
ClusterId:              PavLImfXSqaOOdrMgq1teA
LeaderId:               1
LeaderEpoch:            22
HighWatermark:          7695
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   0
CurrentVoters:          [1,2,3,4]
CurrentObservers:       [5,6,7,8]

到此集群搭建完成

配置讲解

controller

参数	解释
KAFKA_CFG_NODE_ID	节点 id
KAFKA_CFG_PROCESS_ROLES	节点角色
KAFKA_CFG_LISTENERS	节点监听的 ip 和端口，并且标记为 controller
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP	定义不同监听器使用的加密协议
KAFKA_CFG_CONTROLLER_LISTENER_NAMES	控制器（Controller）用于接收来自其他控制器节点和 broker 节点请求的监听器（Listener）的名称
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS	添加集群节点
KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS	当一个新的消费者加入到一个空的消费组时，协调者（Coordinator）将延迟多长时间再开始重新平衡（Rebalance）的过程
KAFKA_CFG_LOG_DIRS	日志目录
KAFKA_KRAFT_CLUSTER_ID	集群 id

broker

参数	解释
KAFKA_CFG_NODE_ID	节点 id
KAFKA_CFG_PROCESS_ROLES	节点角色
KAFKA_CFG_LISTENERS	节点监听的 ip 和端口，并且标记为 broker
KAFKA_CFG_INTER_BROKER_LISTENER_NAME	集群中 broker 之间通信所使用的监听器名称
KAFKA_CFG_CONTROLLER_LISTENER_NAMES	监听器的名称
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP	定义不同监听器使用的加密协议
KAFKA_CFG_CONTROLLER_QUORUM_VOTERS	添加集群节点
KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS	当一个新的消费者加入到一个空的消费组时，协调者（Coordinator）将延迟多长时间再开始重新平衡（Rebalance）的过程
KAFKA_CFG_LOG_DIRS	日志目录
KAFKA_KRAFT_CLUSTER_ID	集群 id