6.6 工具-ELK安装

news2024/11/28 15:43:56

6.6.1 Elasticsearch安装

6.6.1.1 安装

6.6.1.1.1 window

6.6.1.1.2 Linux

6.6.1.2 问题

6.6.1.2.1 问题一

6.6.1.2.2 问题二

6.6.2 Logstash安装

6.6.2.1 安装

6.6.2.1.1 window

6.6.2.1.2 Linux

6.6.2.2 问题

6.6.2.2.1 问题一

6.6.3 Kibana

6.6.3.1 安装

6.6.1 Elasticsearch安装

6.6.1.1 安装

6.6.1.1.1 window

下载地址：下载 Elastic 产品 | Elastic，在下载页面我们需要下载 Elasticsearch和Kibana

下载后解压进入到bin目录下，双击elasticsearch.bat，即可启动。

在浏览器输入http://localhost:9200/ (如果是Linux环境安装，则访问路径http://Linux的IP地址:9200)

6.6.1.1.2 Linux

在opt目录下解压elasticsearch-7.4.2

进入elasticsearch-7.4.2目录下的bin目录

./elasticsearch

注意这里可能会出现用户问题,解决办法看《6.6.1.2.2 问题二》

6.6.1.2 问题

6.6.1.2.1 问题一

 max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]

elasticsearch用户拥有的可创建文件描述的权限太低，至少需要65536

切换到root用户修改

su root

vim /etc/security/limits.conf

在最后面追加下面内容

SHIFT+G 可以直接定位到最后

# * ：表示用户
* hard nofile 65536

* soft nofile 65536

重新su以下，才能使配置生效

su root
或者
# su其他用户是因为es启动不能以root，所以需要添加新的用户来启动es
su 其他用户

查看硬限制

[sandwich@centos-elk ~]$ ulimit -Hn
65536
[sandwich@centos-elk ~]$ ulimit -Sn
65536

6.6.1.2.2 问题二

3405 -- main class information unavailable
[root@master bin]# [2019-06-20T00:14:04,792][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.0.jar:6.8.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root

不能用root用户启动elasticsearch

创建elsearch用户组及elsearch用户：

groupadd elsearch
useradd elsearch -g elsearch
passwd elsearch

更改elasticsearch文件夹及内部文件的所属用户及组为elsearch:elsearch

cd /opt
chown -R elsearch:elsearch  elasticsearch-6.8.0

切换到elsearch用户再启动

su elsearch 
cd /opt/elasticsearch-6.8.0/bin
./elasticsearch

6.6.2 Logstash安装

6.6.2.1 安装

6.6.2.1.1 window

Logstash是一个开源的服务器端数据处理管道，可以同时从多个数据源获取数据，并对其进行转换，然后将其发送到你最喜欢的“存储”。

Logstash的下载地址：Download Logstash Free | Get Started Now | Elastic

下载测试数据集，下载地址：Index of /datasets/movielens，在该网页中下载ml-latest.zip

movies.csv

下载后解压目录，将movies.csv文件拷贝到指定的目录(随便创建一个文件夹如：D:/logstash-datas/)下。

进入到Logstash的解压目录，进入到config目录下，新建logstash.conf，文件内容如下：

input {
  file {
# 注意路径要修改为自己的路径
    path => "D:/logstash-datas/movies.csv"
    start_position => "beginning"
    sincedb_path => "D:/Develop/ELK/logstash-7.4.2/db_path.log"
  }
}
filter {
# 将数据分为三列id，content，genre
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
# 之后将genre用|去分割，并将"path","host"等删除
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {
# content 用(分割 前面为title,后面为year
    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
# year类型转为integer
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
# 将数据存储到指定es中
output {
   elasticsearch {
     hosts => "http://localhost:9200"
# 索引叫movies
     index => "movies"
# 文档id
     document_id => "%{id}"
   }
  stdout {}
}

注意修改路径

input {
file {
path => "D:/logstash-datas/movies.csv" 数据集路径
start_position => "beginning"
sincedb_path => "D:/Develop/ELK/logstash-7.4.2/db_path.log" logstash的路径
}
}

Logstash会转换数据，上面配置的logstash.conf就是转换规则

在命令行进入到logstash的bin目录下，输入如下命令启动Logstash

logstash.bat -f D:\Develop\ELK\logstash-7.4.2\config\logstash.conf

6.6.2.1.2 Linux

将logstash7.4.2解压到opt目录下

进入到Logstash的解压目录，进入到config目录下，新建logstash.conf

logstash.conf

input {
  file {
    path => "/opt/logstash-datas/movies.csv" # 这里指定数据集路径
    start_position => "beginning"
# 这里指定db_path.log路径，起初elasticsearch-7.4.2/目录下是没有db_path.log的，跑完数据集会自动生成db_path.log
    sincedb_path => "/opt/elasticsearch-7.4.2/db_path.log"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
   elasticsearch {
# es的地址
     hosts => "http://localhost:9200"
     index => "movies"
     document_id => "%{id}"
   }
  stdout {}
}

配置完成后进入bin目录,启动logstash，并指定配置文件

./logstash -f config/logstash.conf

开始转换数据

kibana中输入，查看所有索引

GET _cat/indices

6.6.2.2 问题

6.6.2.2.1 问题一

Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.

删除 logstash-7.4.2/data中的.lock 文件