Centos7 随笔记录记录 docker compose 统一管理 granfana + loki + vector 监控kafka 信息。
当然如果仅仅是想通过 Grafana 监控kafka，推荐使用 Grafana Prometheus 通过JMX监控kafka
目录
1. 目录结构
2. 前提已安装Docker-Compose
3. docker-compose 自定义服务相关配置文件
3.1 配置 docker-compose-grafana-loki-vector.yml 文件
3.2 创建 loki-config.yaml
3.3 配置loki 服务
4. docker-compose 单独启动指定服务
4.1 单独启动 loki
4.1.1 启动loki
4.1.2 查询 log
4.2 单独启动vector
4.2.1 启动 vector
4.2.2 查询vector log
4.2.2.1 错误log 问题定位
4.2.2.2 解决方法
4.2.2.3 重启vector 容器，查看无error log
4.3 单独启动grafana
4.3.1 启动 grafana
4.3.2 查看 grafana log
4.3.2.1 错误log 问题定位
4.3.2.2 解决方法
4.3.2.3 重启grafana 容器，查看无error log
4.3.3 访问grafana
4.4 查看loki metrics数据 和loki 运行状态
5. docker-compose 启停所有服务
5.1 docker-compose 启动所有自定义容器服务
5.2 docker-compose 停止所有容器服务
6. 配置grafana
6.1 添加loki 数据源
6.2 修改时区
6.3 添加query
6.4 过滤展示kafka 消息
6.5 优化后 dashboard, 展示kafka 指定字段
6.5.1 添加添加dashboard
6.5.2 优化dashboard, 显示指定字段
6.5.2.1 筛选kafka 原始消息
6.5.2.2 优化表格显示指定字段
7. 其他异常问题
7.1 grafana 页面有时跳转 网址 http://localhost:3000/XXXX 错误
7.2 跳转错误网址原因定位
7.3 解决方法
7.3.1 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址
7.3.2 为了grafana 有数据，必须修改vector.tml 配置参数
7.3.3 使配置生效
7.4 验证 跳转网址正确
1. 目录结构
# 1 创建 4个文件夹：grafana loki vector wal
# 2 为 wal 赋777 权限
[root@localhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1859 Dec 26 10:58 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# tree
.
├── docker-compose-grafana-loki-vector.yml
└── grafana_loki_vector # 创建 grafana folder
├── grafana
├── loki # 创建 loki folder
│ └── loki-config.yaml
├── vector # 创建 vector folder
│ └── vector.toml
└── wal # 创建 wal folder,并赋 777 权限
8 directories, 5 files
[root@localhost grafana_loki_vector]#
# 为避免docker 中权限问题，为wal folder 赋权限
[root@localhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector/grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# chmod 777 wal
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# ll
total 0
drwxr-xr-x. 4 root root 41 Dec 26 10:04 grafana
drwxr-xr-x. 2 root root 44 Dec 26 10:47 loki
drwxr-xr-x. 2 root root 33 Dec 21 15:03 vector
drwxrwxrwx. 3 root root 59 Dec 26 11:19 wal
[root@localhost grafana_loki_vector]#
2. 前提已安装Docker-Compose
如何安装 Docker-Compose，请参考 安装部署Docker-Compose
3. docker-compose 自定义服务相关配置文件
3.1 配置 docker-compose-grafana-loki-vector.yml 文件
# docker-compose 配置参考：https://github.com/grafana/loki/blob/master/production/docker-compose.yaml
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1859 Dec 26 10:58 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# cat docker-compose-grafana-loki-vector.yml
version: "3" # 这是Docker Compose的版本指定，该配置文件使用版本3的语法
# 网桥loki -> 方便相互通讯
networks: # 定义了一个名为loki的网络，该网络是为了Loki和其他服务之间相互通信而创建的网桥。
loki:
services: # 在这个部分定义了要运行的各个服务
loki: # Loki服务的配置
image: grafana/loki:latest # 使用了grafana/loki:latest镜像
container_name: grafana_loki_vector_demo_loki # 定义 容器名
restart: unless-stopped # 设置了重启策略
ports:
- "3100:3100" # 将主机的3100端口映射到Loki容器的3100端口，以便能够从主机访问Loki的API
volumes: # 将两个主机目录与容器内的目录进行了卷映射
- ./grafana_loki_vector/loki:/etc/loki
- ./grafana_loki_vector/wal:/wal
command: -config.file=/etc/loki/loki-config.yaml
networks:
- loki
# vector 用于对接kafka 消息
vector: # vector 服务的配置
# 指定使用 timberio/vector的Docker 镜像，并使用0.10.0-alpine 版本
image: timberio/vector:0.10.0-alpine
container_name: grafana_loki_vector_demo_vector # 定义 容器名
restart: unless-stopped # 指定容器在退出时重新启动，除非显式停止
ports:
- "80:80" # 容器的 80 端口映射到主机的 80 端口上
volumes:
# - ./grafana_loki_vector/vector/logs:/var/logs
# 本地主机目录./grafana_loki_vector/vector挂载到容器的/etc/vector目录上
- "./grafana_loki_vector/vector:/etc/vector"
# 这是要在容器中运行的命令，通过-c /etc/vector/vector.toml指定了配置文件的路径，并通过 -v 启用了详细日志输出
command: -c /etc/vector/vector.toml -v
networks:
- loki
# 用于UI展示
grafana:
image: grafana/grafana:latest
container_name: grafana_loki_vector_demo_grafana
restart: unless-stopped
ports:
- "3000:3000"
volumes:
- "./grafana_loki_vector/grafana/data:/var/lib/grafana"
# - "./grafana_promtail_loki/grafana/config:/etc/grafana"
- "./grafana_loki_vector/grafana/log:/var/log/grafana"
environment:
# GF_EXPLORE_ENABLED: "true"
# 避免有时Grafana 会将跳转 URL 设置为 "localhost"
- GF_SERVER_ROOT_URL=http://192.168.2.247:3000
networks:
- loki
=====================================================================
配置参数说明：
volumes: # 将两个主机目录与容器内的目录进行了卷映射
- ./grafana_loki_vector/loki:/etc/loki
# ./grafana_loki_vector/loki映射到了/etc/loki
- ./grafana_loki_vector/wal_tmp1:/wal
# ./grafana_loki_vector/wal_tmp1映射到了/wal
command: -config.file=/etc/loki/loki-local-config.yaml
# -config.file=/etc/loki/loki-local-config.yaml 指定了Loki容器启动时使用的配置文件路径
environment:
# 避免有时Grafana 会将跳转 URL 设置为 "localhost"
- GF_SERVER_ROOT_URL=http://192.168.2.247:3000 3.2 创建 loki-config.yaml
创建loki-local-config.yaml 文件，配置loki 服务
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# tree
.
├── docker-compose-grafana-loki-vector.yml
└── grafana_loki_vector
├── grafana
│ ├── data
│ └── log
├── loki
│ └── loki-config.yaml # 创建loki-config.yaml文件，配置loki 服务
├── vector
│ └── vector.toml
└── wal
3.3 配置loki 服务
# 这个是Loki的配置文件示例，其中包含了各个方面的配置，用于自定义和配置Loki日志系统的行为和存储。
auth_enabled: false # 设置认证功能是否启用。设为false表示禁用认证，即任何用户都可以访问Loki服务
server:
http_listen_port: 3100 # 配置Loki服务监听的HTTP端口。在这个示例中，Loki将监听3100端口
ingester: # 配置Loki的ingester（摄取器）相关参数
lifecycler:
address: 127.0.0.1
ring:
kvstore: # 使用内存作为kvstore存储方式
store: inmemory
replication_factor: 1 # 复制因子设置为 1
final_sleep: 0s
chunk_idle_period: 5m # chunk的闲置期 5m
chunk_retain_period: 30s # chunk的保留期 30s
schema_config: # 配置存储模式参数
configs:
- from: 2018-04-15 # 从特定日期开始使用的配置
store: boltdb # 数据存储方式（boltdb，文件系统）
object_store: filesystem # 数据存储方式 (文件系统)
schema: v9 # schema版本
index: # 索引相关参数
prefix: index_
period: 168h
storage_config: # 配置存储配置，指定Loki数据的持久化存储位置
boltdb:
directory: /tmp/loki/index # 索引数据将存储在/tmp/loki/index目录
filesystem:
directory: /tmp/loki/chunks # chunk数据将存储在/tmp/loki/chunks目录
# 配置限制参数，用于强制限制指标名称、拒绝旧样本数据，以及设置拒绝旧样本数据的最大时间范围
limits_config:
enforce_metric_name: false # 强制限制指标名称
reject_old_samples: true # 拒绝旧样本数据
reject_old_samples_max_age: 168h # 设置拒绝旧样本数据的最大时间范围
chunk_store_config: # 配置chunk存储参数，设置最大回溯时间范围
max_look_back_period: 0s # 设置最大回溯时间范围
table_manager: # 配置表管理器相关参数
chunk_tables_provisioning: # 设置chunk表的吞吐量
inactive_read_throughput: 0
inactive_write_throughput: 0
provisioned_read_throughput: 0
provisioned_write_throughput: 0
index_tables_provisioning: # 设置索引表的吞吐量
inactive_read_throughput: 0
inactive_write_throughput: 0
provisioned_read_throughput: 0
provisioned_write_throughput: 0
retention_deletes_enabled: false # 保留删除是否启用
retention_period: 0s # 保留期设置
# retention_deletes_enabled: true
# retention_period: 168h
4. docker-compose 单独启动指定服务
建议docker-compose 自定义多个服务时，先单独启动每个服务调试，没问题后，再统一启动
4.1 单独启动 loki
4.1.1 启动loki
# 启动指定自定义服务
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up <服务名> -d
# 启动loki
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up loki -d
[+] Running 7/7
✔ loki 6 layers [⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 24.0s
✔ c926b61bad3b Pull complete 12.0s
✔ 21a0947e3c21 Pull complete 6.0s
✔ 19779106cfff Pull complete 17.5s
✔ 9c6ee7a30a41 Pull complete 7.4s
✔ 2f9c53357ffd Pull complete 8.7s
✔ d8cb9832c63f Pull complete 11.4s
[+] Running 1/1
✔ Container grafana_loki_vector_demo_loki Started 0.2s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps # 查询启动容器
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
58528f1b4782 grafana/loki:latest "/usr/bin/loki -conf…" 9 seconds ago Up 8 seconds 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp grafana_loki_vector_demo_loki
c4f7b69648a8 wurstmeister/kafka "start-kafka.sh" 18 hours ago Up 18 hours 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 18 hours ago Up 18 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp zookeeper
[root@localhost grafana_loki_vector]#
4.1.2 查询 log
# 查看 loki 容器 log
# docker logs <loki服务容器名>
[root@localhost grafana_loki_vector]# pwd
/home/magx/Docker-Compose-Master/grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_loki
level=warn ts=2023-12-26T03:39:33.891828641Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\")."
''''''
''''''
''''''
level=info ts=2023-12-26T03:44:33.988929326Z caller=checkpoint.go:502 msg="atomic checkpoint finished" old=wal/checkpoint.000006.tmp new=wal/checkpoint.000006
[root@localhost grafana_loki_vector]#
4.2 单独启动vector
4.2.1 启动 vector
# 启动指定自定义服务
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up <服务名> -d
# 启动 vector
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1857 Dec 26 11:31 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up vector -d
[+] Running 6/6
✔ vector 5 layers [⣿⣿⣿⣿⣿] 0B/0B Pulled 30.6s
✔ 21c83c524219 Pull complete 18.0s
✔ 88943f7483ee Pull complete 2.9s
✔ 24da72aaf03b Pull complete 23.4s
✔ 4145aaf02e3f Pull complete 10.7s
✔ b16d307dc057 Pull complete 12.0s
[+] Running 1/1
✔ Container grafana_loki_vector_demo_vector Started 0.3s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps # 查询启动容器
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
776ab907da32 timberio/vector:0.10.0-alpine "/usr/local/bin/vect…" About a minute ago Up About a minute 0.0.0.0:80->80/tcp, :::80->80/tcp grafana_loki_vector_demo_vector
2a5581eea2eb grafana/loki:latest "/usr/bin/loki -conf…" 2 minutes ago Up 2 minutes 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp grafana_loki_vector_demo_loki
c4f7b69648a8 wurstmeister/kafka "start-kafka.sh" 21 hours ago Up 21 hours 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 21 hours ago Up 21 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp zookeeper
[root@localhost grafana_loki_vector]#
4.2.2 查询vector log
# 查看 vector 容器log
# docker logs <vector 容器名>
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_vector
Dec 26 06:02:01.842 INFO vector: Log level "debug" is enabled.
Dec 26 06:02:01.857 INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Dec 26 06:02:01.862 INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-377-g0f0311a" released="Wed, 22 Jul 2020 20:01:29 +0000" arch="x86_64"
Dec 26 06:02:01.876 DEBUG vector::tls::settings: Fetching system root certs.
Dec 26 06:02:01.876 INFO vector::topology: Running healthchecks.
Dec 26 06:02:01.876 INFO vector::topology: Starting source "in"
Dec 26 06:02:01.876 INFO vector::topology: Starting sink "out"
Dec 26 06:02:01.876 DEBUG http: vector::sinks::util::http: sending request. uri=http://loki:3100/ready method=GET
Dec 26 06:02:01.884 DEBUG http: vector::sinks::util::http: response. status=503 version=HTTP/1.1
Dec 26 06:02:01.885 ERROR vector::topology::builder: Healthcheck: Failed Reason: A non-successful status returned: 503 Service Unavailable # Error Log
Dec 26 06:02:05.978 DEBUG http: vector::sinks::util::http: sending request. uri=http://loki:3100/loki/api/v1/push method=POST
Dec 26 06:02:06.060 DEBUG http: vector::sinks::util::http: response. status=204 version=HTTP/1.1
[root@localhost grafana_loki_vector]#
4.2.2.1 错误log 问题定位
# 确认 Vector 可以连接到 Loki 服务
在 Vector 配置文件中配置的 Loki 服务的地址应该是可访问的。
可以尝试使用 curl 或类似的工具连接到 Loki 服务
# curl http://loki:3100/ready
[root@localhost grafana_loki_vector]# curl http://loki:3100/ready
curl: (6) Could not resolve host: loki; Unknown error
[root@localhost grafana_loki_vector]#
总结： curl 请求无法解析 loki 主机名，并出现了 Unknown error 错误，vector 无法连接loki4.2.2.2 解决方法
1） 检查 DNS 配置：确认您的 DNS 配置正确，可以解析主机名
# nslookup loki
[root@localhost grafana_loki_vector]# nslookup loki
Server: 114.114.114.114
Address: 114.114.114.114#53
** server can't find loki: NXDOMAIN # loki无法解析为有效的 IP 地址，返回了 NXDOMAIN 错误。
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
2）添加主机名映射：如果 DNS 解析不起作用，您可以尝试手动添加主机名到 IP 地址的映射
[root@localhost grafana_loki_vector]# cat /etc/hosts # 添加前
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# cat /etc/hosts # 添加后
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.247 loki
[root@localhost grafana_loki_vector]#
3）检查vector 是否可以连接loki
# curl http://loki:3100/ready
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# curl http://loki:3100/ready
ready # 可以连接loki
[root@localhost grafana_loki_vector]#
4.2.2.3 重启vector 容器，查看无error log
# 重启vector 容器 or 重新创建 vector 容器
# 无error log, vector 服务运行正常
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_vector
Dec 26 06:20:58.569 INFO vector: Log level "debug" is enabled.
Dec 26 06:20:58.583 INFO vector: Loading configs. path=["/etc/vector/vector.toml"]
Dec 26 06:20:58.588 INFO vector: Vector is starting. version="0.10.0" git_version="v0.9.0-377-g0f0311a" released="Wed, 22 Jul 2020 20:01:29 +0000" arch="x86_64"
Dec 26 06:20:58.602 DEBUG vector::tls::settings: Fetching system root certs.
Dec 26 06:20:58.602 INFO vector::topology: Running healthchecks.
Dec 26 06:20:58.602 INFO vector::topology: Starting source "in"
Dec 26 06:20:58.602 INFO vector::topology: Starting sink "out"
Dec 26 06:20:58.602 DEBUG http: vector::sinks::util::http: sending request. uri=http://loki:3100/ready method=GET
Dec 26 06:20:58.606 DEBUG http: vector::sinks::util::http: response. status=200 version=HTTP/1.1
Dec 26 06:20:58.607 INFO vector::topology::builder: Healthcheck: Passed.
[root@localhost grafana_loki_vector]#
4.3 单独启动grafana
4.3.1 启动 grafana
# 启动指定自定义服务
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up <服务名> -d
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up grafana -d
[+] Running 11/11
✔ grafana 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 106.8s
✔ 96526aa774ef Pull complete 2.3s
✔ 8107a14e7703 Pull complete 1.4s
✔ 41aaa20cd87e Pull complete 2.7s
✔ e00aef65f013 Pull complete 5.1s
✔ 257ea9b0297f Pull complete 5.5s
✔ 599ebbeda3b8 Pull complete 6.1s
✔ ca6b21901a74 Pull complete 59.8s
✔ b5fb47d1c454 Pull complete 92.5s
✔ 4fc0f14a8243 Pull complete 8.0s
✔ f638e65e9fc0 Pull complete 13.7s
[+] Running 1/1
✔ Container grafana_loki_vector_demo_grafana Started 0.3s
[root@localhost grafana_loki_vector]#
4.3.2 查看 grafana log
# 查看 vector 容器log
# docker logs <vector 容器名>
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_grafana
GF_PATHS_DATA='/var/lib/grafana' is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied
.....
....
[root@localhost grafana_loki_vector]#
4.3.2.1 错误log 问题定位
这些错误通常表示在容器中的某个目录或文件上没有写入权限。
4.3.2.2 解决方法
# 解决此问题的一种可能方法是确保您在运行容器时具有足够的权限
# 确保您以具有适当权限的用户身份运行 Docker 命令
# sudo usermod -aG docker $USER
# 更改目标目录的所有者和权限
sudo chown -R 472:472 ./grafana_loki_vector/grafana/data
sudo chmod -R 755 ./grafana_loki_vector/grafana/data
# 检查您的目标目录权限，并确保它具有足够的写入权限
[root@localhost grafana_loki_vector]# sudo chmod -R 777 ./grafana_loki_vector/grafana/data
[root@localhost grafana_loki_vector]#
4.3.2.3 重启grafana 容器，查看无error log
# 重启grafana 容器 or 重新创建 grafana 容器
# 无error log, grafana服务运行正常
# 此处每次都是：
1） 停止容器 # docker stop <container_ID or container_name>
2） 删除容器 # docker rm <container_ID or container_name>
3） 删除镜像 # docker rmi <image_ID>
4) 创建 grafana 容器
5） 查看 log
[root@localhost grafana_loki_vector]# docker stop 839b53f097ee # 停止容器
839b53f097ee
[root@localhost grafana_loki_vector]# docker rm 839b53f097ee # 删除容器
839b53f097ee
[root@localhost grafana_loki_vector]# docker images # 查询镜像ID
REPOSITORY TAG IMAGE ID CREATED SIZE
grafana/grafana latest 8387f19108f9 7 days ago 399MB
grafana/loki latest c2c89eb81f6e 13 days ago 67.7MB
hello-world latest 9c7a54a9a43c 7 months ago 13.3kB
wurstmeister/kafka latest a692873757c0 19 months ago 468MB
timberio/vector 0.10.0-alpine 2ebd263245dd 3 years ago 80.7MB
wurstmeister/zookeeper latest 3f43f72cb283 4 years ago 510MB
[root@localhost grafana_loki_vector]# docker rmi 8387f19108f9 # 删除镜像
Untagged: grafana/grafana:latest
Untagged: grafana/grafana@sha256:6b5b37eb35bbf30e7f64bd7f0fd41c0a5b7637f65d3bf93223b04a192b8bf3e2
Deleted: sha256:8387f19108f986254ecb7eeb1124445a9aeda16602114d98348ed5cd49151a7a
Deleted: sha256:06619c984d0071cd4b3a8fbfb377c1ca5b991497f4e1f6c2f280b9f33d402cb4
Deleted: sha256:5fe5dac0845a9eb44e2c5cfed4fb2754a4adf3224209f58c1a46cb7bd250f2e7
Deleted: sha256:91a3b38581f407bb488e03f002566138875732ba1da4a5e91c55b3dec08a2756
Deleted: sha256:18c0364cc25d61f2e9c8ba8582944e1fc6528ba44abc25ffc3e3bc5ed6a8dec9
Deleted: sha256:e335659dfae7632b652b2c796e742f11b2a9f0c0c7def8511b4df29158a50ee5
Deleted: sha256:5f7c41d4921ce413887919068764757d42744f4898c3f790953fee2e1b33b79b
Deleted: sha256:e8df864ddb32cefc11b6a549add118948327816e858570f6fcbb4b393643d196
Deleted: sha256:14b7f68d3d9094af53f80ebf90273ec864b731545874ee30cfb09cae5d097935
Deleted: sha256:baf151d12231a621fead4ca244823c83fb4ae6e01c36501e37c48a964138e724
Deleted: sha256:cc2447e1835a40530975ab80bb1f872fbab0f2a0faecf2ab16fbbb89b3589438
[root@localhost grafana_loki_vector]#
# 创建 grafana 容器
[root@localhost grafana_loki_vector]# sudo docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up grafana -d
[+] Running 11/11
✔ grafana 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 113.6s
✔ 96526aa774ef Pull complete 12.7s
✔ 8107a14e7703 Pull complete 2.3s
✔ 41aaa20cd87e Pull complete 2.2s
✔ e00aef65f013 Pull complete 10.4s
✔ 257ea9b0297f Pull complete 5.5s
✔ 599ebbeda3b8 Pull complete 10.5s
✔ ca6b21901a74 Pull complete 83.9s
✔ b5fb47d1c454 Pull complete 97.5s
✔ 4fc0f14a8243 Pull complete 17.4s
✔ f638e65e9fc0 Pull complete 28.6s
[+] Running 1/1
✔ Container grafana_loki_vector_demo_grafana Started 0.5s
[root@localhost grafana_loki_vector]#
# 查看 log
[root@localhost grafana_loki_vector]# docker logs grafana_loki_vector_demo_grafana
logger=settings t=2023-12-26T07:25:30.701353225Z level=info msg="Starting Grafana" version=10.2.3 commit=1e84fede543acc892d2a2515187e545eb047f237 branch=HEAD compiled=2023-12-18T15:46:07Z
logger=settings t=2023-12-26T07:25:30.701604782Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
......
......
......
logger=ngalert.scheduler t=2023-12-26T07:26:18.891340858Z level=info msg="Starting scheduler" tickInterval=10s
logger=ticker t=2023-12-26T07:26:18.891454409Z level=info msg=starting first_tick=2023-12-26T07:26:20Z
logger=http.server t=2023-12-26T07:26:18.897364437Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=plugins.update.checker t=2023-12-26T07:26:19.502508394Z level=info msg="Update check succeeded" duration=611.312832ms
logger=grafana.update.checker t=2023-12-26T07:26:20.504658267Z level=info msg="Update check succeeded" duration=1.613658029s
logger=infra.usagestats t=2023-12-26T07:26:51.904796949Z level=info msg="Usage stats are ready to report"
[root@localhost grafana_loki_vector]#
4.3.3 访问grafana
打开 http://<grafana 容器所在主机IP>:3000
默认 user/pwd: admin/admin
4.4 查看loki metrics数据 和loki 运行状态
# 查看loki metrics数据
# 访问：http://<loki 容器所在主机IP>:3100/metrics
# 查看loki运行状态
# 访问：http://<loki 容器所在主机IP>:3100/ready
到此已经调试完成，可以通过docker-compose 统一启动所有服务
5. docker-compose 启停所有服务
5.1 docker-compose 启动所有自定义容器服务
# 启动 定义服务
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[+] Running 3/3
✔ Container grafana_loki_vector_demo_loki Started 0.0s
✔ Container grafana_loki_vector_demo_vector Started 0.0s
✔ Container grafana_loki_vector_demo_grafana Started 0.0s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps # 查询启动容器
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
252ca4e81d6f grafana/grafana:latest "/run.sh" 2 hours ago Up 2 seconds 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp grafana_loki_vector_demo_grafana
7068d0da2577 timberio/vector:0.10.0-alpine "/usr/local/bin/vect…" 3 hours ago Up 2 seconds 0.0.0.0:80->80/tcp, :::80->80/tcp grafana_loki_vector_demo_vector
2a5581eea2eb grafana/loki:latest "/usr/bin/loki -conf…" 3 hours ago Up 2 seconds 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp grafana_loki_vector_demo_loki
c4f7b69648a8 wurstmeister/kafka "start-kafka.sh" 24 hours ago Up 24 hours 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 24 hours ago Up 24 hours 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp zookeeper
[root@localhost grafana_loki_vector]#
5.2 docker-compose 停止所有容器服务
# docker-compose 停止所有容器服务
# docker-compose -f <docker-compose 自定义容器服务配置 yml 文件> -p <项目名> down
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1909 Dec 28 17:27 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
[+] Running 4/4
✔ Container grafana_loki_vector_demo_vector Removed 0.4s
✔ Container grafana_loki_vector_demo_loki Removed 2.0s
✔ Container grafana_loki_vector_demo_grafana Removed 0.0s
✔ Network grafana_loki_vector_loki Removed 0.2s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c4f7b69648a8 wurstmeister/kafka "start-kafka.sh" 3 days ago Up 3 days 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp kafka
a893cce0d465 wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" 3 days ago Up 3 days 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp zookeeper
b38e9b5b6a2e hello-world "/hello" 3 weeks ago Exited (0) 3 weeks ago infallible_rosalind
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
grafana/grafana latest 8387f19108f9 9 days ago 399MB
grafana/loki latest c2c89eb81f6e 2 weeks ago 67.7MB
hello-world latest 9c7a54a9a43c 7 months ago 13.3kB
wurstmeister/kafka latest a692873757c0 19 months ago 468MB
timberio/vector 0.10.0-alpine 2ebd263245dd 3 years ago 80.7MB
wurstmeister/zookeeper latest 3f43f72cb283 4 years ago 510MB
[root@localhost grafana_loki_vector]#
6. 配置grafana
6.1 添加loki 数据源
如果找不到data Source, 可以直接search 查找。
6.2 修改时区
6.3 添加query
click 'Home' -> Explore -> change 'loki' option
6.4 过滤展示kafka 消息
原始kafka 消息数据：
{
"currentIndex":0,
"header":{
"headerMap":{
"35":"381166"
}
},
"mapList":[
{
"10007":"1"
},
{
"37":"48",
"38":"300000",
"44":"15007900",
"48":"688021",
"1301":"1",
"11001":"国盾量子",
"11436":"20231227",
"28000":"20231227",
"28001":"1703642600915",
"28005":"2",
"28101":"893786271",
"43220":"金额:[3000000,450416980],数量:[300000,300100],价格涨跌幅:[0.02,0.19934],市场成交量占比:[0.3,0.967752],区间内最高卖出成交价: 1799.8,昨收盘价: 1500,下单数量: 300000,当前交易价格: 1500.79,挂单数量: 0,挂单金额: 0,成交数量: 100,成交金额: 179980,市场成交数量: 10100",
"64101":"2403",
"64102":"1101",
"64103":"2403-4096个账户-4096个证券",
"65098":"1",
"66002":"5",
"66003":"1",
"66004":"0",
"66005":"2",
"91001":"200000",
"93020":"2"
}
]
}将以上kafka 原始消息数据，筛选过滤出指定字段，语句如下：
# 将以上kafka 原始消息数据，筛选过滤出指定字段，语句如下：
{key="mag1225"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]",ruleID="mapList[1][\"64101\"]",rcSysId="mapList[1][\"66002\"]",logDate="mapList[1][\"28000\"]",logTime="mapList[1][\"28001\"]",securityExchange="mapList[1][\"1301\"]",threshold="mapList[1][\"43220\"]",stockName="mapList[1][\"11001\"]",securityId="mapList[1][\"48\"]",orderId="mapList[1][\"37\"]",orderBs="mapList[1][\"93020\"]",tradeSysId="mapList[1][\"66003\"]",riskConCaption="mapList[1][\"64103\"]",riskConType="mapList[1][\"64102\"]"|line_format "{{.mapList1}}"| __error__=``
6.5 优化后 dashboard, 展示kafka 指定字段
{key="mag1225"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]"|line_format "{{.mapList1}}"| __error__=`` 6.5.1 添加添加dashboard
Clieck 'Add visualizaion' Button or click 'Add'-> Visualization
6.5.2 优化dashboard, 显示指定字段
6.5.2.1 筛选kafka 原始消息
clieck Query 输入 将 kafka 原始消息数据，筛选过滤出指定字段语句
过滤语句：
{key="mag1225"}|= ""|json| line_format "{{.message}}" | json mapList1="mapList[1]",ruleID="mapList[1][\"64101\"]",rcSysId="mapList[1][\"66002\"]",logDate="mapList[1][\"28000\"]",logTime="mapList[1][\"28001\"]",securityExchange="mapList[1][\"1301\"]",threshold="mapList[1][\"43220\"]",stockName="mapList[1][\"11001\"]",securityId="mapList[1][\"48\"]",orderId="mapList[1][\"37\"]",orderBs="mapList[1][\"93020\"]",tradeSysId="mapList[1][\"66003\"]",riskConCaption="mapList[1][\"64103\"]",riskConType="mapList[1][\"64102\"]"|line_format "{{.mapList1}}"| __error__=``
注意：
key=<vector.toml 中sinks.out中的自定义的labels.key>
[root@localhost vector]# cat vector.toml
# __ __ __
# \ \ / / / /
# \ V / / /
# \_/ \/
#
# V E C T O R
# Configuration
#
# ------------------------------------------------------------------------------
# Website: https://vector.dev
# Docs: https://vector.dev/docs/
# ------------------------------------------------------------------------------
# Note: A full config spec is located at ./vector.spec.toml and examples
# in the ./examples folder.
# Vector config kafka + Loki
[ sources.in ]
type = "kafka"
bootstrap_servers = "192.168.2.247:9092" # kafka 地址
group_id = "kafka_mag_groupId1225" # "<消费组id>"
topics = ["alarm_warning","kafka_consumer"] # kafka_consumer
auto_offset_reset = "smallest" # 从最早未处理的消息开始消费
# topics = ["^(prefix1|prefix2)-.+", "topic-1", "topic-2"] # topic名字，支持正则
#[sources.in.encoding]
#codec = "json"
#timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
#json.fields = ["level", "message"]
[sinks.out]
endpoint = "http://loki:3100"
inputs = ["in"] # source.in
type = "loki"
labels.key = "mag1225" # 自定义key
# labels.key = "{{ event_field }}" # event的动态值
#[sinks.out.encoding]
#codec = "json"
# timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
# json.fields = ["level", "message"]
[root@localhost vector]#
6.5.2.2 优化表格显示指定字段
# 优化已表格显示指定字段
1. Click 'Transform data' --> Add Transformation --> Reformat -> Extract fields
2. 添加Extract files -->Source:Line -->Format:JSON --> 打开 'Replacee all fields' --> 打开'Keep time'
重名列名->调整顺序->显示隐藏指定列，配置如下：
3. click 'Add another transformation' --> 'Reorder and rename' --> 'Organize fields by name'
7. 其他异常问题
7.1 grafana 页面有时跳转 网址 http://localhost:3000/XXXX 错误
grafana 容器所在主机IP:192.168.2.247
grafana web address: http://192.168.2.247:3000
grafana web 页面有时操作跳转 网址：http://localhost:3000/XXXX,导致页面跳转错误
7.2 跳转错误网址原因定位
原因定位：
Grafana Explore 页面上的 "Add to Dashboard" 后，跳转网址为 "http://localhost:3000/..." 的问题可能是因为 Grafana 配置中使用了相对 URL 路径导致的7.3 解决方法
7.3.1 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址
1. 修改 docker-compose-grafana-loki-vector.yml 文件，
将 Grafana 服务的环境变量 GF_SERVER_ROOT_URL 设置为正确的访问地址
environment:
- GF_SERVER_ROOT_URL=http://192.168.2.247:3000
[root@localhost grafana_loki_vector]# cat docker-compose-grafana-loki-vector.yml
# docker-compose配置参考：https://github.com/grafana/loki/blob/master/production/docker-compose.yaml
version: "3"
# 网桥loki -> 方便相互通讯
networks:
loki:
services:
# Loki日志系统
loki:
image: grafana/loki:latest
container_name: grafana_loki_vector_demo_loki
restart: unless-stopped
ports:
- "3100:3100"
volumes:
- ./grafana_loki_vector/loki:/etc/loki
- ./grafana_loki_vector/wal:/wal
command: -config.file=/etc/loki/loki-config.yaml
networks:
- loki
# 用于UI展示
grafana:
image: grafana/grafana:latest
container_name: grafana_loki_vector_demo_grafana
restart: unless-stopped
ports:
- "3000:3000"
volumes:
- "./grafana_loki_vector/grafana/data:/var/lib/grafana"
# - "./grafana_promtail_loki/grafana/config:/etc/grafana"
- "./grafana_loki_vector/grafana/log:/var/log/grafana"
environment:
# GF_EXPLORE_ENABLED: "true"
- GF_SERVER_ROOT_URL=http://192.168.2.247:3000 # 修改正确网址 grafana 容器所在主机
networks:
- loki
# vector 用于对接kafka 消息
vector:
image: timberio/vector:0.10.0-alpine
container_name: grafana_loki_vector_demo_vector
restart: unless-stopped
ports:
- "80:80"
volumes:
# - ./grafana_loki_vector/vector/logs:/var/logs
- "./grafana_loki_vector/vector:/etc/vector"
command: -c /etc/vector/vector.toml -v
networks:
- loki
[root@localhost grafana_loki_vector]#
7.3.2 为了grafana 有数据，必须修改vector.tml 配置参数
# 为了grafana 展示kafka 数据，必须修改vector.tml 文件中2个参数
1. group_id = "kafka_mag_groupId1228" # 任意值 从 *1225 改为 *1228
2. labels.key = "mag1228" # 自定义key 从 *1225 改为 *1228
[root@localhost grafana_loki_vector]# ll
total 4
-rw-r--r--. 1 root root 1909 Dec 28 17:27 docker-compose-grafana-loki-vector.yml
drwxr-xr-x. 6 root root 78 Dec 26 10:52 grafana_loki_vector
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# cat grafana_loki_vector/vector/vector.toml
# __ __ __
# \ \ / / / /
# \ V / / /
# \_/ \/
#
# V E C T O R
# Configuration
#
# ------------------------------------------------------------------------------
# Website: https://vector.dev
# Docs: https://vector.dev/docs/
# ------------------------------------------------------------------------------
# Note: A full config spec is located at ./vector.spec.toml and examples
# in the ./examples folder.
# Vector config kafka + Loki
[ sources.in ]
type = "kafka"
bootstrap_servers = "192.168.2.247:9092" # kafka 地址
group_id = "kafka_mag_groupId1228" # "<消费组id>" 修改group_ID
topics = ["alarm_warning","kafka_consumer"] # kafka_consumer
auto_offset_reset = "smallest" # 从最早未处理的消息开始消费
# topics = ["^(prefix1|prefix2)-.+", "topic-1", "topic-2"] # topic名字，支持正则
#[sources.in.encoding]
#codec = "json"
#timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
#json.fields = ["level", "message"]
[sinks.out]
endpoint = "http://loki:3100"
inputs = ["in"] # source.in
type = "loki"
labels.key = "mag1228" # 自定义key
# labels.key = "{{ event_field }}" # event的动态值
#[sinks.out.encoding]
#codec = "json"
# timestamp_format = "%Y-%m-%dT%H:%M:%S.%fZ"
# json.fields = ["level", "message"]
[root@localhost grafana_loki_vector]#
7.3.3 使配置生效
# 使配置生效
2. docker-compose 关闭项目
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
3. docker-copose 启动项目即可。
# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
# docker-compose 关闭项目
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector down
[+] Running 4/4
✔ Container grafana_loki_vector_demo_vector Removed 0.5s
✔ Container grafana_loki_vector_demo_loki Removed 1.7s
✔ Container grafana_loki_vector_demo_grafana Removed 0.4s
✔ Network grafana_loki_vector_loki Removed 0.1s
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]#
# docker-copose 启动项目
[root@localhost grafana_loki_vector]#
[root@localhost grafana_loki_vector]# docker-compose -f docker-compose-grafana-loki-vector.yml -p grafana_loki_vector up -d
[+] Running 4/4
✔ Network grafana_loki_vector_loki Created 0.1s
✔ Container grafana_loki_vector_demo_vector Started 0.1s
✔ Container grafana_loki_vector_demo_grafana Started 0.1s
✔ Container grafana_loki_vector_demo_loki Started 0.1s
[root@localhost grafana_loki_vector]#
7.4 验证 跳转网址正确
在新页面打开，跳转网址正确 http://192.168.2.247:3000/XXXXX
到此基本完成。
当然如果仅仅是想通过 Grafana 监控kafka，推荐使用 Grafana Prometheus 通过JMX监控kafka
