https://cloud.tencent.com/developer/article/1449258

https://www.cnblogs.com/jason2018524/p/16995927.html

https://developer.aliyun.com/article/1141712

prometheus

• docker安装

  https://prometheus.io/docs/prometheus/latest/installation/

  docker run --name prometheus1 -p 9090:9090 prom/prometheus
  

• 本地安装

  下载解压到指定目录：/usr/local/prometheus https://prometheus.io/download/

  [Unit]
  Description=https://prometheus.io
  
  [Service]
  Restart=on-failure
  ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --web.listen-address=:9090 --web.enable-lifecycle
  
  [Install]
  WantedBy=multi-user.target
  

• 配置文件/usr/local/prometheus/prometheus.yml

  # my global config
  global:
    scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
    evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
    # scrape_timeout is set to the global default (10s).
  
  # Alertmanager configuration
  alerting:
    alertmanagers:
      - static_configs:
          - targets:
            # - alertmanager:9093
  
  # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
  rule_files:
    - "rules/*.yml"
    # - "first_rules.yml"
    # - "second_rules.yml"
  
  # A scrape configuration containing exactly one endpoint to scrape:
  # Here it's Prometheus itself.
  scrape_configs:
    # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
    - job_name: "prometheus"
      # metrics_path defaults to '/metrics'
      # scheme defaults to 'http'.
  
      static_configs:
        - targets: ["192.168.1.253:9090"]
    - job_name: 'master_prometheus'
      static_configs:
        - targets: ["192.168.1.253:9100"]
    - job_name: 'node_exporter'
      file_sd_configs:
      - files:
        - 'node.yml'
    - job_name: 'redis_exporter'
      file_sd_configs:
      - files:
        - 'redis.yml'
  

  热加载： curl -X POST http://localhost:9090/-/reload -u username:password

node.yml

- labels:
  env: linux_env
  idc: linux_idc
  system_type: openEuler
targets: ["192.168.1.171:9100","192.168.1.172:9100",'192.168.1.173:9100']

redis.yml

- labels:
    env: redis_env
    idc: redis_idc
    system_type: openEuler
  targets:
    - 192.168.1.171:9121
    - 192.168.1.172:9121
    - 192.168.1.173:9121

列表中的目标对象，编辑后无需重启prometheus可以识别

启动prometheus

systemctl daemon-reload
systemctl enable prometheus
systemctl start prometheus
systemctl status prometheus

prometheus web 安全

https://blog.51cto.com/u_13236892/5882606

安装工具包

yum install -y httpd-tools

生成加密密码

[root@monitor system]# htpasswd -nBC 12 '' | tr -d ':\n'
New password: 
Re-type new password: 
$2y$12$.a389gPJhlI11k6Ksystemctl daemon-reloadjYn.

填入一个配置文件中 webconf.yml

[root@monitor prometheus]# cat webconf.yml 
basic_auth_users:
  admin: $2y$12$.a389gPJhlI11k6KI00DR.2nSCZizhLDtCvVQixFBqHXM8NErjYn.

修改systemd启动服务 prometheus.service

[Unit]
Description=https://prometheus.io

[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --web.listen-address=:9090 --web.config.file=/usr/local/prometheus/webconf.yml --web.enable-lifecycle

[Install]
WantedBy=multi-user.target

401错误处理

• prometheus界面上发现出现了401错误

  

• prometheus.yml增加认证配置

  

  

• grafana中点击Dashboards时也会出现登陆框

  

• 直接在数据源中配置Authentication即可

  

登陆地址

http://192.168.1.253:9090/

exporter

node_exporter

https://blog.csdn.net/ygq13572549874/article/details/129115350

• 解压并mv到目录/usr/local/bin

    cat > /usr/lib/systemd/system/node_exporter.service <<EOF
    [Unit]
    Description=node_exporter
    Documentation=https://prometheus.io/
    After=network.target
    [Service]
    Type=simple
    User=root
    Group=root
    ExecStart=/usr/local/bin/node_exporter
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF
  

• 启动

  systemctl daemon-reload
  systemctl enable node_exporter
  systemctl start node_exporter
  systemctl status node_exporter
  

• web地址

  

  http://192.168.1.253:9100/

redis_exporter

https://blog.csdn.net/web13293720476/article/details/126494940

• 如果redis设置有密码，启动命令：

  nohup ./redis_exporter -redis.addr 你的redis的ip:6379  -redis.password 密码  -web.listen-address :9121 &
  

• 没有密码

  nohup ./redis_exporter -redis.addr 你的redis的ip:6379  -web.listen-address :9121 &
  

• 配置系统服务

https://github.com/oliver006/redis_exporter/releases

• 解压并mv到目录/usr/local/bin

  cat > /usr/lib/systemd/system/redis_exporter.service <<EOF
  [Unit]
  Description=node_exporter
  Documentation=https://prometheus.io/
  After=network.target
  [Service]
  Type=simple
  User=root
  Group=root
  ExecStart=/usr/local/bin/redis_exporter -redis.addr 192.168.1.173:6379  -redis.password 123456  -web.listen-address :9121 &
  Restart=on-failure
  [Install]
  WantedBy=multi-user.target
  EOF
  

• 重载

  systemctl daemon-reload
  systemctl enable redis_exporter
  systemctl start redis_exporter
  systemctl status redis_exporter
  

• web地址

  

  http://192.168.1.171:9121/

内存指标无法显示，默认maxmemory 内存大小为0，要么设置一个值，要么改变grafana的指标计算公式

https://www.cnblogs.com/caoweixiong/p/12736384.html

• 如下，Memory Usage 无法显示数值

  

• 查看面板的计算，redis_memory_max_bytes （maxmemory）值为0，导致除数为0发生异常

  127.0.0.1:6379> config get maxmemory
  1) "maxmemory"
  2) "0"
  

  

• 调整redis参数maxmemory后正常

  

mysqld_exporter

https://github.com/prometheus/mysqld_exporter/releases/download/v0.15.1/mysqld_exporter-0.15.1.linux-amd64.tar.gz

• mysql授权

  CREATE USER 'exporter'@'%' IDENTIFIED BY '123456' WITH MAX_USER_CONNECTIONS 3;
  GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';
  

• 配置文件

  cat > /usr/local/exporter/config.my-cnf <<EOF
  [client]
  host = 192.168.1.253
  port = 3306
  user = exporter
  password = 123456
  EOF
  

  cat > /usr/local/prometheu/mysql.yml <<EOF
  - labels:
      env: linux_env
      idc: linux_idc
      system_type: openEuler
    targets: ["192.168.1.253:9104"]
  

  - job_name: mysql # To get metrics about the mysql exporter’s targets
      basic_auth:
        username: admin
        password: Mon2024##
      file_sd_configs:
      - files:
        - 'mysql.yml'
  

• 生成web密码

  密码如果使用123456，会出现错误提示，更改为复杂密码即可

  Feb 19 09:20:42 monitor mysqld_exporter: ts=2024-02-19T01:20:42.649Z caller=mysqld_exporter.go:268 level=error msg="Error starting HTTP server" err="crypto/bcrypt: hashedSecret too short to be a bcrypted password"
  

  cat > /usr/local/exporter/webconf.yml <<EOF
  basic_auth_users:
    admin: $2y$12$jPDaatE7sZ13DMEErQgTH.Hp.uhSYAvfU7bY/cEdwq7q1PwPCyEPa
  EOF
  

• 连接mysql

  [client]
  host = 192.168.1.253
  port = 3306
  user = exporter
  password = 123456
  

• 启动服务

  cat > /usr/lib/systemd/system/mysqld_exporter.service <<EOF
  [Unit]
  Description=mysqld_exporter
  Documentation=https://prometheus.io/
  After=network.target
  [Service]
  Type=simple
  User=root
  Group=root
  ExecStart=/usr/local/exporter/mysqld_exporter --config.my-cnf=/etc/my.cnf --web.config.file=/usr/local/exporter/webconf.yml 
  Restart=on-failure
  [Install]
  WantedBy=multi-user.target
  EOF
  

• 启动

  systemctl daemon-reload
  systemctl enable mysqld_exporter --now
  systemctl start mysqld_exporter
  systemctl status mysqld_exporter
  

• web地址

  http://192.168.1.253:9104/

• 简单压测

  mysqlslap -uroot  --concurrency=1 --iterations=1 --create-schema='test' --auto-generate-sql --auto-generate-sql-load-type=mixed --auto-generate-sql-add-autoincrement --engine=innodb --number-char-cols=10 --number-int-cols=5 --number-of-queries=5000 -S/usr/local/mysql/mysql.sock
  

指标不显示问题

原来：
(mysql_global_variables_innodb_buffer_pool_size{instance="$host"} * 100) / on (instance) node_memory_MemTotal_bytes{instance="$host"}
替换为：
(label_replace(mysql_global_variables_innodb_buffer_pool_size{instance="$host"}, "nodename", "$1", "instance", "(.*):.*") * 100) / on(nodename) (label_replace(node_memory_MemTotal_bytes, "nodename", "$1", "instance", "(.*):.*"))

grafana

https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/

docker run -d --name=grafana -p 3000:3000 grafana/grafana:latest

使用到的模板ID

• redis id

  763
  

• node

  11074
  15172
  16098