【k8s中安装rabbitmq】k8s中安装rabbitmq并搭建镜像集群-hostpath版

文章目录

- - 简介
  - 一.条件及环境说明
  - 二.需求说明
  - 三.实现原理及说明
  - 四.详细步骤
  - - 4.1.规划节点标签
    - 4.2.创建configmap配置
    - 4.3.创建三个statefulset和service headless配置
    - 4.4.创建service配置
  - 五.安装完后的配置
  - 六.安装说明

简介

k8s集群中搭建rabbitmq集群服务一般都会用到pvc，但是考虑到有些自建k8s环境下，搭建的共享存储可能会存在稳定性及性能问题，所以这次是通过采用节点亲和性和hostpath来实现，目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将rabbitmq镜像集群的搭建采用hostpath+亲和性的权重+多副本来实现数据持久化和高可用。

一.条件及环境说明

k8s版本k8s-1.29.4，环境搭建在电信机房,六个worker节点，每个节点有一块非系统盘的ssd盘挂载到/data/路径，不采用ceph或nas之类的共享存储，也未采用operator，目前rabbitmq-operator基本都是采用了共享存储。

二.需求说明

搭建rabbitmq镜像模式集群。
高可用：有一个k8s节点死掉或重启pod之后也不影响使用。
高性能：读写本地磁盘实现高性能的io，共享存储如果资源及硬件性能不够的话，io将会是性能瓶颈。
数据安全：在发生切换,节点故障以后，数据要尽可能的保证完整
安装简单、管理维护容易

三.实现原理及说明

四.详细步骤

4.1.规划节点标签

节点名称	角色	标签
k8s-worker-120-81	主	storage-selector=node-a-master
k8s-worker-120-82	主	storage-selector=node-b-master
k8s-worker-120-83	主	storage-selector=node-c-master
k8s-worker-120-84	备	storage-selector=node-a-slve
k8s-worker-120-85	备	storage-selector=node-a-slve
k8s-worker-120-86	备	storage-selector=node-a-slve

注：这里的rabbitmq是三节点的镜像集群，每个redis的pod分别落在三个主的节点上，当主节点发生故障是，就通过权重分配切换到备节点，例如：名称为autotest-rabbitmq-a-0的pod，默认落在标签为storage-selector=node-a-master的节点上，当该节点发生故障是，就会根据权重匹配罗在storage-selector=node-a-slve的节点上。

4.2.创建configmap配置

将如下配置保存到autotest-rabbitmq-config.yaml，配置中配置默认的vhost和用户密码，并初始化好集群节点【安装前规划好节点名称】信息。

apiVersion: v1
kind: ConfigMap
metadata:
  name: autotest-rabbitmq-config
  namespace: zx-app
  labels:
    appname: pcauto-zx
    app: autotest-rabbitmq-config 
data:
  enabled_plugins: |
    [rabbitmq_management,rabbitmq_prometheus].
  rabbitmq.conf: |
    #
    listeners.tcp.default = 5672
    
    default_vhost = /
    default_user = admin
    default_pass = admin

    default_user_tags.administrator = true
    default_user_tags.management = true
    default_user_tags.custom_tag = true

    channel_max = 1024
    tcp_listen_options.backlog = 2048

    cluster_partition_handling = autoheal
    cluster_partition_handling.pause_if_all_down.recover = autoheal

    cluster_formation.peer_discovery_backend = classic_config
    cluster_formation.classic_config.nodes.1 = rabbit@autotest-rabbitmq-a-0.autotest-rabbitmq-a.zx-app.svc.cluster.local
    cluster_formation.classic_config.nodes.2 = rabbit@autotest-rabbitmq-b-0.autotest-rabbitmq-b.zx-app.svc.cluster.local
    cluster_formation.classic_config.nodes.3 = rabbit@autotest-rabbitmq-c-0.autotest-rabbitmq-c.zx-app.svc.cluster.local
   
    # cluster_formation.node_type = disc
    # cluster_keepalive_interval = 10000

    log.file.level = error
    log.console = true

4.3.创建三个statefulset和service headless配置

1.autotest-a.yaml 节点a的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autotest-rabbitmq-a 
  namespace: zx-app 
  labels:
    appname: pcauto-zx
    app: autotest-rabbitmq-a
    appgroup: autotest-rabbitmq
spec:
  serviceName: "autotest-rabbitmq-a"
  replicas: 1
  selector:
    matchLabels:
      app: autotest-rabbitmq-a
      appgroup: autotest-rabbitmq
  template:
    metadata:
      labels:
        app: autotest-rabbitmq-a
        appgroup: autotest-rabbitmq
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-a-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-a-slave
      containers:
      - name: rabbitmq-server
        image: xxx-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management 
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh","-c"]
        args: 
        - mkdir -p /home/rabbitmq/${POD_NAME};chown rabbitmq.rabbitmq -R /home/rabbitmq/${POD_NAME};rabbitmq-server
        env:
        - name: RABBITMQ_ERLANG_COOKIE
          value: "YZSDHWMFSMKEMBDHSGGZ"
        - name: K8S_SERVICE_NAME
          value: autotest-rabbitmq-a
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_MNESIA_BASE
          value: /home/rabbitmq/$(POD_NAME)/mnesia
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
        ports:
        - name: http
          containerPort: 15672 
        - name: amqp
          containerPort: 5672 
        volumeMounts:
        - name: rbmq-data
          mountPath: /home/rabbitmq 
        - name: rabbitmq-config-volume
          mountPath: /etc/rabbitmq/ 
      restartPolicy: Always 
      terminationGracePeriodSeconds: 30
      volumes:
        - name: rbmq-data
          hostPath:
            path: /data/rabbitmq
            type: ""
        - name: rabbitmq-config-volume
          configMap:
            name: autotest-rabbitmq-config
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: autotest-rabbitmq-a
  namespace: zx-app
  labels:
    appname: pcauto-zx
    app: autotest-rabbitmq-a
spec:
  ports:
  - port: 5672
  clusterIP: None
  selector:
    app: autotest-rabbitmq-a

2.autotest-b.yaml 节点b的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autotest-rabbitmq-b 
  namespace: zx-app 
  labels:
    appname: pcauto-zx
    app: autotest-rabbitmq-b
    appgroup: autotest-rabbitmq
spec:
  serviceName: "autotest-rabbitmq-b"
  replicas: 1
  selector:
    matchLabels:
      app: autotest-rabbitmq-b
      appgroup: autotest-rabbitmq
  template:
    metadata:
      labels:
        app: autotest-rabbitmq-b
        appgroup: autotest-rabbitmq
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-b-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-b-slave
      containers:
      - name: rabbitmq-server
        image: xxx-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management 
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh","-c"]
        args: 
        - mkdir -p /home/rabbitmq/${POD_NAME};chown rabbitmq.rabbitmq -R /home/rabbitmq/${POD_NAME};rabbitmq-server
        env:
        - name: RABBITMQ_ERLANG_COOKIE
          value: "YZSDHWMFSMKEMBDHSGGZ"
        - name: K8S_SERVICE_NAME
          value: autotest-rabbitmq-b
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_MNESIA_BASE
          value: /home/rabbitmq/$(POD_NAME)/mnesia
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
        ports:
        - name: http
          containerPort: 15672 
        - name: amqp
          containerPort: 5672 
        volumeMounts:
        - name: rbmq-data
          mountPath: /home/rabbitmq 
        - name: rabbitmq-config-volume
          mountPath: /etc/rabbitmq/ 
      restartPolicy: Always 
      terminationGracePeriodSeconds: 30
      volumes:
        - name: rbmq-data
          hostPath:
            path: /data/rabbitmq
            type: ""
        - name: rabbitmq-config-volume
          configMap:
            name: autotest-rabbitmq-config
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: autotest-rabbitmq-b
  namespace: zx-app
  labels:
    appname: pcauto-zx
    app: autotest-rabbitmq-b
spec:
  ports:
  - port: 5672
  clusterIP: None
  selector:
    app: autotest-rabbitmq-b

3.autotest-c.yaml 节点c的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: autotest-rabbitmq-c 
  namespace: zx-app 
  labels:
    appname: pcauto-zx
    app: autotest-rabbitmq-c
    appgroup: autotest-rabbitmq
spec:
  serviceName: "autotest-rabbitmq-c"
  replicas: 1
  selector:
    matchLabels:
      app: autotest-rabbitmq-c
      appgroup: autotest-rabbitmq
  template:
    metadata:
      labels:
        app: autotest-rabbitmq-c
        appgroup: autotest-rabbitmq
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-c-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-c-slave
      containers:
      - name: rabbitmq-server
        image: xxx-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management 
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh","-c"]
        args: 
        - mkdir -p /home/rabbitmq/${POD_NAME};chown rabbitmq.rabbitmq -R /home/rabbitmq/${POD_NAME};rabbitmq-server
        env:
        - name: RABBITMQ_ERLANG_COOKIE
          value: "YZSDHWMFSMKEMBDHSGGZ"
        - name: K8S_SERVICE_NAME
          value: autotest-rabbitmq-c
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_MNESIA_BASE
          value: /home/rabbitmq/$(POD_NAME)/mnesia
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
        ports:
        - name: http
          containerPort: 15672 
        - name: amqp
          containerPort: 5672 
        volumeMounts:
        - name: rbmq-data
          mountPath: /home/rabbitmq 
        - name: rabbitmq-config-volume
          mountPath: /etc/rabbitmq/ 
      restartPolicy: Always 
      terminationGracePeriodSeconds: 30
      volumes:
        - name: rbmq-data
          hostPath:
            path: /data/rabbitmq
            type: ""
        - name: rabbitmq-config-volume
          configMap:
            name: autotest-rabbitmq-config
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: autotest-rabbitmq-c
  namespace: zx-app
  labels:
    appname: pcauto-zx
    app: autotest-rabbitmq-c
spec:
  ports:
  - port: 5672
  clusterIP: None
  selector:
    app: autotest-rabbitmq-c

4.4.创建service配置

给amqp端口5672和管理端口15672配置一个service的代理服务，注意其中的selector是指向到appgroup,这是将三个statefulset定义了一个appgroup的标签。

apiVersion: v1
kind: Service
metadata:
  name: autotest-rabbitmq-manage
  namespace: zx-app
  labels:
    app: autotest-rabbitmq-manage
    appname: pcauto-zx
spec:
  ports:
  - port: 5672
    name: amqp
  - port: 15672
    name: http
  selector:
    appgroup: autotest-rabbitmq
  sessionAffinity: ClientIP
  type: LoadBalancer

五.安装完后的配置

将rabbitmq的集群设置成3节点的镜像集群，在4.4步骤操作完以后就可以通过loadbalancer的IP进行登录控制台了：http://ip:15672,用configmap中设置的 default_user和default_pass来登录。

镜像模式设置：

设置后：

镜像模式可以通过命令设置：
设置demo的vhost的镜像执行命令：
rabbitmqctl set_policy -p demo ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
设置默认vhost /的镜像模式：
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

六.安装说明

需要先在k8s的节点创建/data/rabbitmq的路径来存放数据。
使用配置前将配置中的autotest替换成自己需要的命名,配置的命名空间为：zx-app，修改成自己的命名空间，appname为pcauto-zx，修改成自己的应用名称，该标签也可以删除。
配置中的镜像地址采用了私有的镜像地址，镜像是dockerhub上下载的rabbitmq:3.12.14-management 镜像到私有镜像仓库的，如果能直通外网的k8s可以直接用rabbitmq:3.12.14-management 。否则配置一个可以通的镜像地址。
默认的用户名和密码是admin。
节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。