ELK日志收集之多文件提取文件名和日志时间

news2024/12/27 10:09:11

需求：多个设备的日志同时保存在一台服务器上，日志文件的文件名是设备的ID，需要将多个文件提取文件名作为最终的筛选字段，同时提取日志中的时候日期时间替换系统的@timestamp
在这里插入图片描述

filebeat配置：

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /opt/data/*.log
    tags: ["test-android-log"]
    fields:
      log_source: my_log_source
    fields_under_root: true
    processors:
      - dissect:
          tokenizer: "/opt/data/%{filename}.log"
          field: "log.file.path"
          target_prefix: "file"

output:
  logstash:
    hosts: ["192.168.0.102:5044"]

logstash配置：

input {
  beats {
    port => 5044
  }
}

filter {
  if [file][filename] {
    mutate {
      add_field => { "device_no" => "%{[file][filename]}" }
    }
  }
  grok {
    match => { "message" => "%{MONTHNUM:month}-%{MONTHDAY:day} %{TIME:time} %{GREEDYDATA:log_message}" }
    add_field => { "timestamp" => "%{month}-%{day} %{time}" }
  }

  date {
    match => ["timestamp", "MM-dd HH:mm:ss.SSS"]
    target => "@timestamp"
  }
  mutate {
    remove_field => [ "timestamp", "month", "day", "time" ]
  }
}

output {
  if "test-android-log" in [tags] {
    elasticsearch {
      hosts => ["192.168.0.101:9200"]
      index => "test-android_log_t2014"
    }
  }

  stdout { codec => rubydebug }
}

使用kibana的开发工具获取一下对应index的结果看下是否有想要的字段传过来

GET /test-android_log_t2014/_search
{
  "size": 1,
  "_source": [
    "device_no"
  ]
}

我这边想要的是device_no，查看见过如下表示获取成功：

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5392,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "test-android_log_t2014",
        "_id": "vohlxZABk6v1MxO1ydv2",
        "_score": 1,
        "_source": {
          "device_no": "20240718173333"
        }
      }
    ]
  }
}