K8S-1.18.20高可用集群之部署集群插件-KUBE-PROMETHEUS插件
一、简介
kube-prometheus 是一整套监控解决方案，它使用 Prometheus 采集集群指标，Grafana 做展示，包含如下组件：
The Prometheus Operator
Highly available Prometheus
Highly available Alertmanager
Prometheus node-exporter
Prometheus Adapter for Kubernetes Metrics APIs （k8s-prometheus-adapter）
kube-state-metrics
Grafana
其中 k8s-prometheus-adapter 使用 Prometheus 实现了 metrics.k8s.io 和 custom.metrics.k8s.io API，所以不需要再部署 metrics-server。
如果没有特殊指明，本文档的所有操作均在 k8s-Mater 节点上执行；
二、下载kube-prometheus部署yaml
2.1、选择kube-prometheus版本
最新版本参考kube-prometheus git仓库：https://github.com/prometheus-operator/kube-prometheus
2.2、下载K8S版本对应的kube-prometheus
这里下载kube-prometheus 0.5.0
下载地址：https://codeload.github.com/prometheus-operator/kube-prometheus/zip/refs/tags/v0.5.0
三、修改yaml文件使用阿里云镜像
3.1、查看部署Yaml所需要的镜像
[root@localhost opt]# grep -r image: ./manifests
./manifests/alertmanager-alertmanager.yaml: image: quay.io/prometheus/alertmanager:v0.20.0
./manifests/grafana-deployment.yaml: image: grafana/grafana:6.6.0
./manifests/kube-state-metrics-deployment.yaml: image: quay.io/coreos/kube-state-metrics:v1.9.5
./manifests/kube-state-metrics-deployment.yaml: image: quay.io/coreos/kube-rbac-proxy:v0.4.1
./manifests/kube-state-metrics-deployment.yaml: image: quay.io/coreos/kube-rbac-proxy:v0.4.1
./manifests/node-exporter-daemonset.yaml: image: quay.io/prometheus/node-exporter:v0.18.1
./manifests/node-exporter-daemonset.yaml: image: quay.io/coreos/kube-rbac-proxy:v0.4.1
./manifests/prometheus-adapter-deployment.yaml: image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0
./manifests/prometheus-prometheus.yaml: image: quay.io/prometheus/prometheus:v2.15.2
./manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml: image:
./manifests/setup/prometheus-operator-deployment.yaml: image: quay.io/coreos/prometheus-operator:v0.38.1
./manifests/setup/prometheus-operator-deployment.yaml: image: quay.io/coreos/kube-rbac-proxy:v0.4.1
[root@localhost opt]# grep -r "image: grafana" ./manifests
./manifests/grafana-deployment.yaml: image: grafana/grafana:6.6.0
3.2、替换yaml中镜像地址为阿里云镜像
3.2.1、替换quay.io镜像
[root@localhost opt]# sed -i "s/quay.io\/prometheus/registry.cn-hangzhou.aliyuncs.com\/wc181/g" `grep "quay.io" -rl ./`
[root@localhost opt]# sed -i "s/quay.io\/coreos/registry.cn-hangzhou.aliyuncs.com\/wc181/g" `grep "quay.io" -rl ./`
[root@localhost opt]# sed -i "s/grafana\/prometheus/registry.cn-hangzhou.aliyuncs.com\/wc181/g" `grep "quay.io" -rl ./`
3.2.3、kube-prometheus需要用到的镜像
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/alertmanager:v0.20.0
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/grafana:6.6.0
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/kube-state-metrics:v1.9.5
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/kube-rbac-proxy:v0.4.1
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/node-exporter:v0.18.1
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/k8s-prometheus-adapter-amd64:v0.5.0
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/prometheus:v2.15.2
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/prometheus-operator:v0.38.1
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/configmap-reload:v0.3.0
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/prometheus-config-reloader:v0.38.1
以上项目部署在monitoring 名称空间下
验证部署情况
[root@dfycb-master1 setup]# kubectl get pod -o wide -n monitoring
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
alertmanager-main-0 2/2 Running 0 7h57m 10.70.7.71 dfycb-node5 <none> <none>
alertmanager-main-1 2/2 Running 0 7h57m 10.70.6.55 dfycb-node4 <none> <none>
alertmanager-main-2 2/2 Running 0 7h57m 10.70.5.66 dfycb-node3 <none> <none>
grafana-7469c5dc7d-bffpr 1/1 Running 0 8h 10.70.7.68 dfycb-node5 <none> <none>
kube-state-metrics-74b48b4bb7-lsnbl 3/3 Running 0 8h 10.70.3.60 dfycb-node1 <none> <none>
node-exporter-4sdbn 2/2 Running 0 8h 10.60.6.147 dfycb-node8 <none> <none>
node-exporter-628bp 2/2 Running 0 8h 10.60.6.142 dfycb-node3 <none> <none>
node-exporter-8jc9p 2/2 Running 0 8h 10.60.6.139 dfycb-master3 <none> <none>
node-exporter-f9zxr 2/2 Running 0 8h 10.60.6.140 dfycb-node1 <none> <none>
node-exporter-gghpn 2/2 Running 0 8h 10.60.6.143 dfycb-node4 <none> <none>
node-exporter-hfg9j 2/2 Running 0 8h 10.60.6.138 dfycb-master2 <none> <none>
node-exporter-nggm2 2/2 Running 0 8h 10.60.6.137 dfycb-master1 <none> <none>
node-exporter-qfnmr 2/2 Running 0 8h 10.60.6.145 dfycb-node6 <none> <none>
node-exporter-vjsx4 2/2 Running 0 8h 10.60.6.146 dfycb-node7 <none> <none>
node-exporter-vm8t4 2/2 Running 0 8h 10.60.6.141 dfycb-node2 <none> <none>
node-exporter-zphp4 2/2 Running 0 8h 10.60.6.144 dfycb-node5 <none> <none>
prometheus-adapter-6fb854c766-22fp5 1/1 Running 0 8h 10.70.6.53 dfycb-node4 <none> <none>
prometheus-k8s-0 3/3 Running 1 7h57m 10.70.5.67 dfycb-node3 <none> <none>
prometheus-k8s-1 3/3 Running 1 7h57m 10.70.9.68 dfycb-node7 <none> <none>
prometheus-operator-65549d9758-jhqz9 2/2 Running 0 7h57m 10.70.7.70 dfycb-node5 <none> <none>
四、部署kube-prometheus
4.1、部署kube-prometheus
[root@k8s01 manifests]# kubectl apply -f setup/*
[root@k8s01 manifests]# kubectl apply -f ./*
4.2、通过ingress暴露prometheus、grafana
prometheus
grafana
4.3、网页验证是否可以正常打开
prometheus UI
grafana
五、grafana展示
5.1、添加数据源
默认已添加
5.2、导入展示面板
通过ID导入：8919
也可以通过连接下载导入json
https://grafana.com/grafana/dashboards/8919-1-node-exporter-for-prometheus-dashboard-cn-0413-consulmanager/?tab=revisions
5.3、导入8919
因为是内网环境无法通过互联网导入，这里选择先下载json再导入
下载地址：https://grafana.com/api/dashboards/8919/revisions/14/download
文件：1-node-exporter-for-prometheus-dashboard-cn-0413-consulmanager_rev14
下载版本：14
最后展示效果：
文章参考：https://www.freesion.com/article/8614604815/
