收集apache日志输出到es

在真实服务器安装logstash，httpd

systemctl start httpd

echo 666 > /var/www/html/index.html

cat /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/httpd #系统内置变量

cd /usr/local/logstash/config/

cp logstash-sample.conf httpd-access.conf

vim httpd-access.conf

input {
   stdin {}
}
filter {
  grok  {
    match => {
      "message" => "%{HTTPD_COMBINEDLOG}"
    }
    remove_field => ["message","auth","ident"]
  }
  date {
      match => ["timestamp","dd/MM/yyy:HH:mm:ss Z"]
  }
}

output {
   stdout {}
}

cat /var/log/httpd/access_log

127.0.0.1 - - [21/Jun/2023:14:41:22 +0800] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"

logstash -f httpd-access.conf

127.0.0.1 - - [21/Jun/2023:14:41:22 +0800] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"

可以看出apache的日志输出带有\，咱可以改进一下：

cd /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/

grep QS -R .

grep QUOTEDSTRING -R .

此时可以看出定义太复杂，咱自定义一个变量ALL，取代QS，双引号引起来。

vim /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/httpd

ALL .* #空行新增

HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} "%{ALL:referrer}" "%{ALL:agent}"

此时filter的相关配置已经完善!

接下来配置input引用apache日志，output输出到es。

vim /usr/local/logstash/config/httpd-access.conf

input {
   file {
       path => ["/var/log/*.log","/var/log/message*"]
       type => "httpd_access"
       start_position => "beginning"
   }
}
filter {
  grok  {
    match => {
      "message" => "%{HTTPD_COMBINEDLOG}"
    }
    remove_field => ["message","auth","ident"]
  }
  date {
      match => ["timestamp","dd/MM/yyy:HH:mm:ss Z"]
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.148.132:9200"]
    index => "%{type}-%{+YYYY.MM.dd}"
  }
}

logstash -f /usr/local/logstash/config/httpd-access.conf

127.0.0.1 - - [21/Jun/2023:14:41:22 +0800] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"

安装插件ElasticSearch Head可视化工具

（略）

测试：浏览器访问192.168.148.132:9100

http://localhost:9200/ > http://192.168.148.132:9200/ > 连接

收集nginx日志输出到es

方案一

nginx:

修改 nginx server 的配置文件

    log_format  json '{'
                           '"client":"$remote_addr",'
                           '"time":"$time_local",'
                           '"verb":"$request_method",'
                           '"url":"$request_uri",'
                           '"status":"$status",'
                           '"size":$body_bytes_sent,' 
                           '"referer": "$http_referer",'
                           '"agent": "$http_user_agent"'
               '}';
   access_log  /var/log/nginx/access_json.log  json;   

logstash配置文件:

input {
   file {
      path => "/var/log/nginx/access_json.log"
      codec => "json"                           #输入预定义好的 JSON 数据, 可以省略掉 filter/grok 配置, 从而减轻logstash的负载
      start_position => "beginning"
   }
}
output {
    elasticsearch {
           hosts => ["192.168.10.11:9200"]
           index => "nginx-log-%{+YYYY.MM.dd}"
    }
}

方案二

Logstash 对nginx 标准日志的 grok 正则定义是:

MAINNGINXLOG %{COMBINEDAPACHELOG} %{QS:x_forwarded_for}

logstash:直接使用访问日志

input {
    file {
        path => "/var/log/nginx/access.log"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG} %{QS:x_forwarded_for}"}    
    }    
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]    
    }    
    geoip {
        source => "clientip"    
    }
}
output {
    elasticsearch {     
        hosts => "192.168.11.10:9200" 
    }
}