1、prometheus自定义的dashboard模块
文件路径/etc/prometheus/config_out/prometheus-env.yaml

- job_name: serviceMonitor/monitoring/pfil/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - monitoring
  relabel_configs:
  - source_labels:
    - job
    target_label: __tmp_prometheus_job_name
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    - __meta_kubernetes_service_labelpresent_app
    regex: (pfil);true
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_source
    - __meta_kubernetes_service_labelpresent_source
    regex: (external);true
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_pod_container_name
    target_label: container
  - action: drop
    source_labels:
    - __meta_kubernetes_pod_phase
    regex: (Failed|Succeeded)
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_jobname
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: metrics
  - source_labels:
    - __address__
    target_label: __tmp_hash
    modulus: 1
    action: hashmod
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep

注释

Prometheus 配置文件 (/etc/prometheus/config_out/prometheus-env.yaml) 使用的是 Kubernetes 服务发现（kubernetes_sd_configs），并且指定了 role: endpoints。这意味着 Prometheus 会自动去查找 Kubernetes 集群里的 Endpoints（端点） 对象来发现监控目标。

根据配置，Prometheus 只会抓取满足以下所有条件的目标：

命名空间（Namespace）: 目标必须位于 monitoring 这个命名空间内。
服务标签（Service Labels）: 与目标关联的 Kubernetes Service（服务） 必须 同时 具有 app=pfil 和 source=external 这两个标签。
端口名称（Port Name）: 在 Service 定义中，目标所使用的那个端口，其 名称（name） 必须是 metrics。
Pod 状态（Pod Phase）: 不会抓取状态是 Failed (失败) 或 Succeeded (成功) 的 Pod 对应的目标

2、创建ServiceMonitor
注意标签selector

  selector:
    matchLabels:
      app: pfil
      source: external

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations: {}
  labels:
    app: pfil
    release: kube-prometheus
    source: external
  name: pfil
  namespace: monitoring
  resourceVersion: '308908134'
spec:
  endpoints:
    - port: metrics
  jobLabel: jobname
  namespaceSelector:
    matchNames:
      - monitoring
  selector:
    matchLabels:
      app: pfil
      source: external

3、创建Endpoints
Endpoints1

apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app: pfil
    jobname: pfil
    source: external
  name: pfil
  namespace: monitoring
  resourceVersion: "552235238"
  uid: dbd41ece-caf3-4b7b-8510-3c6eda714f7a
subsets:
- addresses:
  - hostname: test-main
    ip: 1.1.1.1
  - hostname: test-east-1
    ip: 2.2.2.2
  - hostname: test-west-1
    ip: 3.3.3.3
  ports:
  - name: metrics
    port: 9002
    protocol: TCP

Endpoints2

apiVersion: v1
kind: Endpoints
metadata:
  name: parasail-miner
  namespace: monitoring
  labels:
    app: pfil
    source: external
    component: parasail-miner
subsets:
- addresses:
  - ip: 1.1.1.1
    hostname: parasail-miner
  ports:
  - name: metrics 
    port: 8002
    protocol: TCP

4、创建Service
service1

---
apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    app: pfil
    component: parasail-miner
    source: external
  name: parasail-miner
  namespace: monitoring
  resourceVersion: '552542525'
spec:
  clusterIP: 10.100.103.0
  clusterIPs:
    - 10.100.103.0
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - name: metrics
      port: 8002
      protocol: TCP
      targetPort: 8002
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

service2

---
apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    app: pfil
    jobname: pfil
    source: external
  name: pfil
  namespace: monitoring
  resourceVersion: '552219945'
spec:
  clusterIP: 10.100.78.9
  clusterIPs:
    - 10.100.78.9
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - name: metrics
      port: 9002
      protocol: TCP
      targetPort: 9002
    - name: metrics-8002
      port: 8002
      protocol: TCP
      targetPort: 8002
  sessionAffinity: None
  type: ClusterIP