1、作用
1、可以在本机收集日志
2、也可以远程收集日志
3、轻量级的日志收集系统，可以在非java环境运行。
logstash是在jmv环境中运行，资源消耗很大，启动一个logstash要消耗500M左右的内存，filebeat只消耗10M左右的内存。
收集nginx的日志
2、收集nginx日志
systemctl restart nginx
systemctl stop firewalld
setenforce 0
#解压，将filebeat移到/usr/local下
tar -xf filebeat-6.7.2-linux-x86_64.tar.gz
mv filebeat-6.7.2-linux-x86_64 /usr/local/filebeat
#备份
cd /usr/local/filebeat
cp filebeat.yml filebeat.yml.bak
#修改配置文件
vim filebeat.yml
type: log
enabled: true
paths:
- /usr/local/nginx/logs/access.log
- /usr/local/nginx/logs/error.log
#开启日志收集，以及确定日志文本的路径，指定标签和发送到目标主机的logstash
tags: ["nginx"]
fields:
service_name: 192.168.230.21_nginx
log_type_ nginx
from: 192.168.230.21
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
outout elasticsearch
output.logsatsh:
hosts: ["192.168.230.30:5045"]
#5044是logstash默认的端口，只要是logstash主机上没有被占用的端口都可以使用，端口号要大于1024
#修改从21接收的nginx日志文件
vim nginx_21.conf
input {
beats { port => "5045"}
}
output {
if "nginx" in [tags] {
elasticsearch {
hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
}
}
}
#启动filebeat
nohup ./filebeat -e -c filebeat.yml > filebeat.out &
#-e：输出到标准输出
-c：指定配置文件
nohup：在系统的后台运行，不会因为终端的关闭导致程序停机运行；可以把运行的日志保存到指定文件
#后后台运行
logstash -f nginx_61.conf --path.data /opt/test2 &
3、远程收集nginx、http、mysql日志
filebeat远程收集发送到logstash主机
展示的索引：
192.168.230.21_mysql-*
192.168.230.21_nginx-*
192.168.230.21_http-*
systemctl stop firewalld
setenforce 0
#修改配置文件
vim /etc/my.cnf
添加：
general_log=ON
general_log_file=/usr/local/mysql/data/mysql_general.log
#安装httpd、nginx
yum -y install httpd nginx
#重启mysqld、httpd
systemctl restart mysqld
systemctl restart httpd
#修改nginx的端口号
listen 85;
#listen [::]:80;
#重启nginx
systemctl restart nginx
#浏览器访问nginx、httpd
#将filebeat的安装包放到/opt目录下，并解压
tar -xf filebeat-6.7.2-linux-x86_64.tar.gz
#将filebeat-6.7.2-linux-x86_64放到/opt命令下，改名为filebeat
mv filebeat-6.7.2-linux-x86_64 filebeat
#修改filebeat的配置文件
vim filebeat.yml
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
- /var/log/nginx/error.log
tags: ["nginx"]
fields:
service_name: 192.168.230.21_nginx
log_type: nginx
from: 192.168.230.21
- type: log
enabled: true
paths:
- /var/log/httpd/access_log
- /var/log/httpd/error_log
tags: ["httpd"]
fields:
service_name: 192.168.230.21_httpd
log_type: httpd
from: 192.168.230.21
- type: log
enabled: true
paths:
- /usr/local/mysql/data/mysql_general.log
tags: ["mysqld"]
fields:
service_name: 192.168.230.21_mysqld
log_type: mysqld
from: 192.168.230.21
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
output.logstash:
# The Logstash hosts
hosts: ["192.168.230.30:5048"]
vim nmh_21.conf
input {
beats { port => "5048"}
}
output {
if "nginx" in [tags] {
elasticsearch {
hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
}
}
if "httpd" in [tags] {
elasticsearch {
hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
}
}
if "mysqld" in [tags] {
elasticsearch {
hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"
}
}
}
#开启日志收集
nohup ./filebeat -e -c filebeat.yml > filebeat.out &
#后台运行
logstash -f nmh_21.conf --path.data /opt/test3 &
#多了httpd、nginx、mysqld的日志
#到kibana可视化界面创建索引模式
4、收集http日志
#安装httpd
yum -y install httpd
cd /etc/logstash
cd conf.d
#配置文件
vim http.conf
input {
file {
path => "/etc/httpd/logs/access_log"
type => "access"
start_position => "beginning"
}
file {
path => "/etc/httpd/logs/error_log"
type => "error"
start_position => "beginning"
}
}
output {
if [type] == "access" {
elasticsearch {
hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
index => "apache_access-%{+YYYY.MM.dd}"
}
}
if [type] == "error" {
elasticsearch {
hosts => ["192.168.230.10:9200","192.168.230.230:9200"]
index => "apache_error-%{+YYYY.MM.dd}"
}
}
#启动logstash
logstash -f http.conf --path.data /opt/test1 &
显示：Successful started Logstash API endpoint {:port=>9601}
API接口：软件内部代码之间通信的接口，代码的连接点
端口是对外提供访问程序的内容接口
#到kibana可视化界面创建索引模式
