Helm部署高可用rabbitmq k8s 镜像集群

前提条件：k8s集群、harbor私服、helm、storage Class

安装部署

添加bitnami仓库并查找redis

helm repo add bitnami https://charts.bitnami.com/bitnami

helm repo update
[kmning@k8s-register-node ~]$ helm search repo rabbitmq
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
bitnami/rabbitmq                        11.13.0         3.11.13         RabbitMQ is an open source general-purpose mess...
bitnami/rabbitmq-cluster-operator       3.2.10          2.2.0           The RabbitMQ Cluster Kubernetes Operator automa...

拉取chat到本地

helm pull bitnami/rabbitmq --version 11.13.0

tar -zxvf rabbitmq-11.13.0.tgz

cp rabbitmq/values.yaml ./values-rabbitmq.yaml

对本地values-rabbitmq.yaml进行修改，配置非常多，根据实际情况进行修改，比如我主要修改了如下内容

通用配置修改

global:
  imageRegistry: "k8s-register-node.com:443"
  imagePullSecrets: []
  storageClass: "managed-nfs-storage"

查找所有用到镜像的配置，修改成私服，storageClass修改成我们定义的storageClass即可。
image:
  registry: k8s-register-node.com:443
  repository: lib-proxy/bitnami/rabbitmq
  tag: 3.11.13-debian-11-r0


persistence:
  enabled: true
  storageClass: "managed-nfs-storage"

rabbitmq配置修改

auth:
  username: kmning
  password: "yourpwd"
  existingPasswordSecret: ""
  erlangCookie: "secretcookie"
  existingErlangSecret: ""

如果不希望把密码配置到配置文件中，可以在安装时通过提供参数的方式设置

--set auth.username=euht,auth.password=yourpwd,auth.erlangCookie=secretcookie

开启clustering.forceBoot

clustering:
  enabled: true
  addressType: hostname
  rebalance: false
  forceBoot: true

指定时区

extraEnvVars: 
  - name: TZ
    value: "Asia/Shanghai"

指定副本数

replicaCount: 3

持久化配置

persistence:
  enabled: true
  storageClass: "managed-nfs-storage"
  selector: {}
  accessMode: ReadWriteOnce
  existingClaim: ""
  size: 8Gi

helm安装redis集群

kubectl create ns rabbitmq-cluster
helm -n rabbitmq-cluster install rabbitmq-cluster  rabbitmq-11.13.0.tgz  -f values-rabbitmq.yaml \
--set useBundledSystemChart=true

安装后打印

kmning@k8s-master-1:~/rabbitmq-k8s-cluster$ helm -n rabbitmq-cluster install rabbitmq-cluster  rabbitmq-11.13.0.tgz  -f values-rabbitmq.yaml \
> --set useBundledSystemChart=true
NAME: rabbitmq-cluster
LAST DEPLOYED: Fri Apr 28 06:38:58 2023
NAMESPACE: rabbitmq-cluster
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: rabbitmq
CHART VERSION: 11.13.0
APP VERSION: 3.11.13** Please be patient while the chart is being deployed **

Credentials:
    echo "Username      : euht"
    echo "Password      : $(kubectl get secret --namespace rabbitmq-cluster rabbitmq-cluster -o jsonpath="{.data.rabbitmq-password}" | base64 -d)"
    echo "ErLang Cookie : $(kubectl get secret --namespace rabbitmq-cluster rabbitmq-cluster -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 -d)"

Note that the credentials are saved in persistent volume claims and will not be changed upon upgrade or reinstallation unless the persistent volume claim has been deleted. If this is not the first installation of this chart, the credentials may not be valid.
This is applicable when no passwords are set and therefore the random password is autogenerated. In case of using a fixed password, you should specify it when upgrading.
More information about the credentials may be found at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases.

RabbitMQ can be accessed within the cluster on port 5672 at rabbitmq-cluster.rabbitmq-cluster.svc.cluster.local

To access for outside the cluster, perform the following steps:

To Access the RabbitMQ AMQP port:

    echo "URL : amqp://127.0.0.1:5672/"
    kubectl port-forward --namespace rabbitmq-cluster svc/rabbitmq-cluster 5672:5672

To Access the RabbitMQ Management interface:

    echo "URL : http://127.0.0.1:15672/"
    kubectl port-forward --namespace rabbitmq-cluster svc/rabbitmq-cluster 15672:15672

查看已安装chat

kmning@k8s-master-1:~/rabbitmq-k8s-cluster$ helm -n rabbitmq-cluster list
NAME                    NAMESPACE               REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
rabbitmq-cluster        rabbitmq-cluster        1               2023-04-28 06:38:58.749511901 +0000 UTC deployed        rabbitmq-11.13.0        3.11.13

如果需要卸载

helm -n rabbitmq-cluster uninstall rabbitmq-cluster

查看服务部署情况

kmning@k8s-master-1:~/rabbitmq-k8s-cluster$ kubectl get pods -n rabbitmq-cluster
NAME                 READY   STATUS              RESTARTS   AGE
rabbitmq-cluster-0   1/1     Running             0          3m53s
rabbitmq-cluster-1   1/1     Running             0          2m18s
rabbitmq-cluster-2   0/1     ContainerCreating   0          73s
kmning@k8s-master-1:~/rabbitmq-k8s-cluster$ kubectl get svc -n rabbitmq-cluster
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                 AGE
rabbitmq-cluster            ClusterIP   10.43.65.148   <none>        5672/TCP,4369/TCP,25672/TCP,15672/TCP   4m14s
rabbitmq-cluster-headless   ClusterIP   None           <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   4m14s
kmning@k8s-master-1:~/rabbitmq-k8s-cluster$ kubectl get pv -n rabbitmq-cluster
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                      STORAGECLASS          REASON   AGE

pvc-7f4b0a27-c370-45d5-8af3-633954ae39ef   8Gi        RWO            Delete           Bound    rabbitmq-cluster/data-rabbitmq-cluster-2   managed-nfs-storage            114s
pvc-83f15f1e-9f16-4eeb-acaa-28c627ad90f3   8Gi        RWO            Delete           Bound    rabbitmq-cluster/data-rabbitmq-cluster-0   managed-nfs-storage            4m33s
pvc-b7245f5d-72d4-45a1-8328-83928bfdd347   8Gi        RWO            Delete           Bound    rabbitmq-cluster/data-rabbitmq-cluster-1   managed-nfs-storage            2m59s

kmning@k8s-master-1:~/rabbitmq-k8s-cluster$ kubectl get pvc -n rabbitmq-cluster
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
data-rabbitmq-cluster-0   Bound    pvc-83f15f1e-9f16-4eeb-acaa-28c627ad90f3   8Gi        RWO            managed-nfs-storage   5m1s
data-rabbitmq-cluster-1   Bound    pvc-b7245f5d-72d4-45a1-8328-83928bfdd347   8Gi        RWO            managed-nfs-storage   3m26s
data-rabbitmq-cluster-2   Bound    pvc-7f4b0a27-c370-45d5-8af3-633954ae39ef   8Gi        RWO            managed-nfs-storage   2m21s

服务正常，此时，我们已经可以使用服务域名rabbitmq-cluster.rabbitmq-cluster.svc.cluster.local去访问这个集群了。

查看集群状态，随便进入一个pod

kubectl exec -it -n rabbitmq-cluster rabbitmq-cluster-0 -- bash

# 查看集群状态
rabbitmqctl cluster_status

# 列出策略（尚未设置镜像模式）
rabbitmqctl list_policies

#设置集群名称
rabbitmqctl set_cluster_name [cluster_name]

服务暴露

5672的服务端口，k8s集群内的应用只需要通过rabbitmq-cluster.rabbitmq-cluster.svc.cluster.local:5672即可连接到集群，而15672端口我们手动创建一个NodePort，让外网可以访问，方便管理。

先获取sts的selector

kmning@k8s-master-1:~/rabbitmq-k8s-cluster$ kubectl get svc -n rabbitmq-cluster -o wide
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                 AGE   SELECTOR
rabbitmq-cluster            ClusterIP   10.43.65.148   <none>        5672/TCP,4369/TCP,25672/TCP,15672/TCP   17m   app.kubernetes.io/instance=rabbitmq-cluster,app.kubernetes.io/name=rabbitmq
rabbitmq-cluster-headless   ClusterIP   None           <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   17m   app.kubernetes.io/instance=rabbitmq-cluster,app.kubernetes.io/name=rabbitmq

创建nodePort服务

rabbitmq-cluster-svc-nodeport.yaml

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-cluster-nodeport
  namespace: rabbitmq-cluster
spec:
  ports:
  - nodePort: 30072
    port: 5672
    name: rab-sv-port
    protocol: TCP
    targetPort: 5672
  - nodePort: 30073
    port: 15672
    name: rab-ad-port
    protocol: TCP
    targetPort: 15672  
  selector:
    app.kubernetes.io/instance: rabbitmq-cluster
    app.kubernetes.io/name: rabbitmq
  type: NodePort

创建后随便使用一个工作节点IP进行访问：http://yourWorker:30072

可见，集群已经正常运行。

镜像模式配置

进入任意一个pod

kubectl exec -it -n rabbitmq-cluster rabbitmq-cluster-0 -- bash

# 设置镜像模式
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all" , "ha-sync-mode":"automatic"}'

# 列出策略
rabbitmqctl list_policies

打印如下

kmning@k8s-master-1:~/rabbitmq-k8s-cluster$ kubectl exec -it -n rabbitmq-cluster rabbitmq-cluster-0 -- bash
I have no name!@rabbitmq-cluster-0:/$ rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all" , "ha-sync-mode":"automatic"}'
Setting policy "ha-all" for pattern "^" to "{"ha-mode":"all" , "ha-sync-mode":"automatic"}" with priority "0" for vhost "/" ...
I have no name!@rabbitmq-cluster-0:/$ rabbitmqctl list_policies
Listing policies for vhost "/" ...
vhost   name    pattern apply-to        definition      priority
/       ha-all  ^       all     {"ha-mode":"all","ha-sync-mode":"automatic"}    0

镜像队列模式设置成功。此时，随意连接一个节点，创建队列和交互机，把数据发送到队列，所有节点将同步队列的数据，避免在消费数据前数据丢失的风险。然后，如果有消费者对数据进行消费，所有节点对应的数据也将被清理。

最后，在k8s集群内部访问这个rabbitmq集群只需要使用Service的域名即可，如下

redis-cluster.redis.svc.cluster.local:6379

这样一来，访问这个节点，k8s自动为我们做了负载均衡（kube-proxy组件），我们不需要再配置nginx负载均衡。

spring-boot配置示例：

spring:
  rabbitmq:
    host: rabbitmq-cluster.rabbitmq-cluster.svc.cluster.local
    port: 5672
    virtual-host: /
    username: kmning
    password: yourpwd