黑盒监控
“白盒监控” 是需要把对应的Exporter程序安装到被监控的目标主机上,从而实现对主机各种资源以及状态的数据采集工作 ”黑盒监控“ 是不需要把Exporter程序部署到被监控的目标主机上,比如全球的网络质量的稳定性,通常用ping操作,对选取的节点进行icmp测试 Prometheus社区提供黑盒解决方案,用户只需要将其安装在与Prometheus和被监控目标互通的环境中,通过HTTP、HTTPS、DNS、TCP、ICMP,证书的监控等 方式对网络进行探测监控。
blackbox_exporter
Prometheus 官方提供的exporter,可以提供http dns tcp icmp的监控数据采集
创建配置文件
mkdir /data/blackbox_exporter -p
cat >/data/blackbox_exporter/config.yml <<'EOF'
modules:
http_2xx:
prober: http
http:
method: GET
http_port_2xx:
prober: http
http:
method: POST
tcp_connect:
prober: tcp
pop3s_banner:
prober: tcp
tcp:
query_response:
- expect: "^+OK"
tls: true
tls_config:
insecure_skip_verify: false
grpc:
prober: grpc
grpc:
tls: true
preferred_ip_protocol: "ip4"
grpc_plain:
prober: grpc
tls: false
service: "service1"
ssh_banner:
prober: tcp
tcp:
query_response:
- expect: "^SSH-2.0-"
- send: "SSH-2.0-blackbox-ssh-check"
irc_banner:
prober: tcp
tcp:
query_response:
- send: "NICK prober"
- send: "USER prober prober prober :prober"
- expect: "PING :([^ ]+)"
send: "PONG ${1}"
- expect: "^:[^ ]+ 001"
icmp:
prober: icmp
icmp_ttl5:
prober: icmp
timeout: 5s
icmp:
ttl: 5
EOF
修改后
root@prometheus220:/data/blackbox_exporter# cat /data/blackbox_exporter/config.yml
modules:
http_2xx:
prober: http
http:
method: GET
http_port_2xx:
prober: http
http:
method: POST
tcp_connect:
prober: tcp
icmp:
prober: icmp
docker 直接运行
docker run -d --restart=always --name blackbox-exporter -p 9115:9115 \
-v /data/blackbox_exporter:/etc/blackbox_exporter prom/blackbox-exporter:v0.16.0 \
--config.file=/ect/blackbox_exporter/config.yml
docker-compose 方式
cd /data/blackbox_exporter/
cat >docker-compose.yml << 'EOF'
version: '3.3'
services:
blackbox_exporter:
image: prom/blackbox-exporter:v0.16.0
container_name: blackbox_exporter
restart: always
volumes:
- /data/blackbox_exporter:/etc/blackbox_exporter
ports:
- 9115:9115
EOF
启动
docker-compose up -d
确认启动
docker ps -a
访问测试
http://10.19.1.220:9115/
添加Prometheus的配置
cd /opt/data/docker-prometheus/prometheus
cat >> prometheus/prometheus.yml<<'EOF'
#http 配置
- job_name: 'blackbox_http'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://www.baidu.com
- https://www.163.com
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 10.19.1.220:9115
#tcp 配置
- job_name: 'blackbox_tcp'
metrics_path: /probe
params:
module: [tcp_connect]
static_configs:
- targets:
- 10.19.1.220:22
- 10.19.1.206:9090
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 10.19.1.220:9115
#icmp 配置 ping
- job_name: 'blackbox_icmp'
metrics_path: /probe
params:
module: [icmp]
static_configs:
- targets:
- 10.19.1.206
- 10.19.1.220
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 10.19.1.220:9115
EOF
热启动
curl -X POST http://localhost:9090/-/reload
Prometheus界面已经显示配置成功
监控项:
probe_
probe_success #是否探测成功(取值1、0,分别表示成功、失败)
probe_duration_seconds #探测的耗时
#关于DNS
probe_dns_lookup_time_seconds #DNS解析的耗时
probe_ip_protocol #IP协议取值为4、6
probe_ip_addr_hash #IP地址的哈希值,用于判断IP是否变化
#关于HTTP
probe_http_status_code #HTTP响应的状态码,如果发生重定向,则取决于最后一次响应
probe_http_content_length #HTTP响应的body长度,单位bytes
probe_http_version #HTTP响应的协议版本,比如1.1
probe_http_ssl #HTTP响应是否采用SSL,取值 1、0
probe_ssl_earliest_cert_expiry #SSL证书过期时间,为Unix时间戳
触发器
Prometheus 配置
#报警(触发器)配置
rule_files:
- "alert.yml"
- "rules/*.yml"
添加blackbox_exporter触发器
cd /opt/data/docker-prometheus/
创建触发器文件
cat > prometheus/rules/blackbox_exporter.yml << 'EOF'
groups:
- name: Blackbox
rules:
- alert: 黑盒子探测失败报警
expr: probe_success == 0
for: 1m
labels:
severity: critical
annotations:
summary: '黑盒子探测失败 {{ $labels.instance }}'
description: "黑盒子探测失败,当前值{{ $value }}"
- alert: 请求慢告警
expr: avg_over_time(probe_duration_seconds[1m]) > 1
for: 1m
labels:
severity: warning
annotations:
summary: '请求慢 {{ $labels.instance }}'
description: "请求时间操过1秒,值{{ $value }}"
- alert: http状态码检测失败
expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
for: 1m
labels:
severity: critical
annotations:
summary: 'http状态码检测失败{{ $labels.instance }}'
description: "HTTP状态码非200-399,当前状态码为{{ $value }}"
- alert: ssl证书即将到期
expr: probe_ssl_earliest_cert_expiry - time() < 86400 *30
for: 1m
labels:
severity: warning
annotations:
summary: '证书即将到期 {{ $labels.instance }}'
description: "SSL 证书在30天后到期,值{{ $value }}"
EOF
重新加载配置
curl -X POST http://localhost:9090/-/reload
alert中生效
Grafana增加dashboard
Blackbox Exporter Dashboard 20220412-StarsL.cn | Grafana Labs
Grafana
admin/password