資訊專(zhuān)欄INFORMATION COLUMN
摘要：新增了一個(gè)，它是一個(gè)輕量級(jí)的日志收集處理工具，占用資源少，適合于在各個(gè)服務(wù)器上搜集日志后傳輸給，官方也推薦此工具。具體的結(jié)構(gòu)如下主機(jī)系統(tǒng)部署服務(wù)安裝準(zhǔn)備是需要的，建議安裝。先下載通用安裝包。
ELK簡(jiǎn)介
ELK是三個(gè)開(kāi)源軟件的縮寫(xiě)，分別表示：Elasticsearch , Logstash, Kibana , 它們都是開(kāi)源軟件。新增了一個(gè)FileBeat，它是一個(gè)輕量級(jí)的日志收集處理工具(Agent)，F(xiàn)ilebeat占用資源少，適合于在各個(gè)服務(wù)器上搜集日志后傳輸給Logstash，官方也推薦此工具。
Elasticsearch是個(gè)開(kāi)源分布式搜索引擎，提供搜集、分析、存儲(chǔ)數(shù)據(jù)三大功能。它的特點(diǎn)有：分布式，零配置，自動(dòng)發(fā)現(xiàn)，索引自動(dòng)分片，索引副本機(jī)制，restful風(fēng)格接口，多數(shù)據(jù)源，自動(dòng)搜索負(fù)載等。
Logstash 主要是用來(lái)日志的搜集、分析、過(guò)濾日志的工具，支持大量的數(shù)據(jù)獲取方式。一般工作方式為c/s架構(gòu)，client端安裝在需要收集日志的主機(jī)上，server端負(fù)責(zé)將收到的各節(jié)點(diǎn)日志進(jìn)行過(guò)濾、修改等操作在一并發(fā)往elasticsearch上去。
Kibana 也是一個(gè)開(kāi)源和免費(fèi)的工具，Kibana可以為 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以幫助匯總、分析和搜索重要數(shù)據(jù)日志。
Filebeat隸屬于Beats。目前Beats包含四種工具：
Packetbeat（搜集網(wǎng)絡(luò)流量數(shù)據(jù)）
Topbeat（搜集系統(tǒng)、進(jìn)程和文件系統(tǒng)級(jí)別的 CPU 和內(nèi)存使用情況等數(shù)據(jù)）
Filebeat（搜集文件數(shù)據(jù)）
Winlogbeat（搜集 Windows 事件日志數(shù)據(jù)）
一般而言，ELK主要用在海量零散數(shù)據(jù)的匯總和信息提取分析上。在分布式系統(tǒng)的日志統(tǒng)計(jì)，大數(shù)據(jù)的數(shù)據(jù)分析，業(yè)務(wù)數(shù)據(jù)的快速檢索，服務(wù)器集群上每臺(tái)服務(wù)器的運(yùn)行情況查詢監(jiān)控等方面有很強(qiáng)大的功能。
拿ELK在分布式系統(tǒng)上的日志收集舉例。隨著微服務(wù)的流行，分布式的使用，以往日志文件寫(xiě)在具體的服務(wù)器上的某一位置做法變得不符合需求，首先是服務(wù)器越來(lái)越多而且后端服務(wù)集群橫跨多個(gè)服務(wù)器導(dǎo)致日志越來(lái)越散，不論是開(kāi)發(fā)，測(cè)試還是線上的日志定位越來(lái)越難，準(zhǔn)確的找到有用的信息需要運(yùn)維/開(kāi)發(fā)不段的排查，這時(shí)ELK就派上用場(chǎng)了，它將服務(wù)集群里面的日志收集匯總并建立索引，當(dāng)出現(xiàn)問(wèn)題是定位問(wèn)題就像Google這類(lèi)搜素引擎一樣高效簡(jiǎn)單。
安裝一般單臺(tái)機(jī)器就可以安裝了，我這里為了貼近實(shí)際使用，分為3個(gè)機(jī)器來(lái)部署一個(gè)入門(mén)的ELK。
具體的結(jié)構(gòu)如下
| 主機(jī) | IP | 系統(tǒng) | 部署服務(wù) |
|---|---|---|---|
| thinkvmc01 | 192.168.50.207 | CentOS7 | ElasticSearch |
| thinkvmc02 | 192.168.50.19 | CentOS7 | Logstash |
| thinkvmc03 | 192.168.50.54 | CentOS7 | Kibana |
ELK是需要Java的，建議安裝Java8。這里我就啰嗦了
# 先檢查JDK環(huán)境 [thinktik@thinkvmc01 ~]$ java -version java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)開(kāi)始安裝
安裝ELK不難，按照官方的文檔即可，官網(wǎng)鏈接下
開(kāi)源搜索與分析 · Elasticsearch
我們先安裝 ElasticSearch。先下載 Linux 通用安裝包 elasticsearch-6.7.1.tar.gz
。當(dāng)然為了簡(jiǎn)單，你也可以下載具體Linux發(fā)行版的預(yù)編譯包，這樣安裝更加簡(jiǎn)單，不過(guò)缺少靈活性。
# 下載
[thinktik@thinkvmc01 thinktik]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.7.1.tar.gz
--2019-04-08 22:51:05-- https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.7.1.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 151.101.230.222, 2a04:4e42:1a::734
Connecting to artifacts.elastic.co (artifacts.elastic.co)|151.101.230.222|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 148542786 (142M) [application/x-gzip]
Saving to: ‘elasticsearch-6.7.1.tar.gz.1’
2% [> ] 3,699,945 42.4KB/s eta 25m 36s
....
# 下載完畢，解壓
[thinktik@thinkvmc01 thinktik]# ls
elasticsearch-6.7.1.tar.gz java8 jdk-8u201-linux-x64.tar.gz
[thinktik@thinkvmc01 thinktik]# tar -zxvf elasticsearch-6.7.1.tar.gz
elasticsearch-6.7.1/
elasticsearch-6.7.1/lib/
....
elasticsearch-6.7.1/logs/
elasticsearch-6.7.1/plugins/
# 進(jìn)入安裝文件目錄
[thinktik@thinkvmc01 thinktik]# cd elasticsearch-6.7.1
[thinktik@thinkvmc01 elasticsearch-6.7.1]# ls
bin config lib LICENSE.txt logs modules NOTICE.txt plugins README.textile
[thinktikt@thinkvmc01 elasticsearch-6.7.1]# cd config/
[thinktik@thinkvmc01 config]# ls
elasticsearch.yml jvm.options log4j2.properties role_mapping.yml roles.yml users users_roles
# 修改配置，綁定我們的網(wǎng)卡。不修改默認(rèn)為127.0.0.1，那樣其余的機(jī)器上的Logstash，Kibana就沒(méi)法訪問(wèn)這臺(tái)機(jī)的ES了
[thinktik@thinkvmc01 config]# vim elasticsearch.yml
#修改如下
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
# IP地址
network.host: 192.168.50.207
#
# Set a custom port for HTTP:
# 端口，默認(rèn)9200
http.port: 9200
# 啟動(dòng)
[thinktik@thinkvmc01 bin]$ ./elasticsearch
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
[2019-04-08T23:11:44,120][INFO ][o.e.e.NodeEnvironment ] [ZVfIMzv] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [26.7gb], net total_space [28.9gb], types [rootfs]
[2019-04-08T23:11:44,126][INFO ][o.e.e.NodeEnvironment ] [ZVfIMzv] heap size [1015.6mb], compressed
....
# 這里報(bào)了錯(cuò)，很明顯了
ERROR: [2] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-04-08T23:12:06,558][INFO ][o.e.n.Node ] [ZVfIMzv] stopping ...
[2019-04-08T23:12:06,636][INFO ][o.e.n.Node ] [ZVfIMzv] stopped
[2019-04-08T23:12:06,637][INFO ][o.e.n.Node ] [ZVfIMzv] closing ...
[2019-04-08T23:12:06,673][INFO ][o.e.n.Node ] [ZVfIMzv] closed
# 我們按它的提示該系統(tǒng)配置
[thinktik@thinkvmc01 bin]$ vim /etc/security/limits.conf
[thinktik@thinkvmc01 bin]$ su
Password:
#添加如下配置
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
# 繼續(xù)啟動(dòng)
[thinktik@thinkvmc01 bin]$ ./elasticsearch
# 報(bào)錯(cuò)，那么繼續(xù)修改
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-04-08T23:19:38,825][INFO ][o.e.n.Node ] [ZVfIMzv] stopping ...
[2019-04-08T23:19:38,844][INFO ][o.e.n.Node ] [ZVfIMzv] stopped
[2019-04-08T23:19:38,845][INFO ][o.e.n.Node ] [ZVfIMzv] closing ...
[2019-04-08T23:19:38,887][INFO ][o.e.n.Node ] [ZVfIMzv] closed
[2019-04-08T23:19:38,889][INFO ][o.e.x.m.p.NativeController] [ZVfIMzv] Native controller process has stopped - no new native processes can be started
# 繼續(xù)修改
[thinktik@thinkvmc01 bin]$ su
Password:
[root@thinkvmc01 bin]# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
# 繼續(xù)啟動(dòng)
[thinktik@thinkvmc01 bin]$ ./elasticsearch
[2019-04-08T23:22:37,612][INFO ][o.e.c.s.ClusterApplierService] [ZVfIMzv] new_master {ZVfIMzv}{ZVfIMzviR5ie4WVCaO9CZA}{B3vTE3wKSriPc-LwHC8J-A}{192.168.50.207}{192.168.50.207:9300}{ml.machine_memory=1927471104, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {ZVfIMzv}{ZVfIMzviR5ie4WVCaO9CZA}{B3vTE3wKSriPc-LwHC8J-A}{192.168.50.207}{192.168.50.207:9300}{ml.machine_memory=1927471104, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-04-08T23:22:37,792][INFO ][o.e.h.n.Netty4HttpServerTransport] [ZVfIMzv] publish_address {192.168.50.207:9200}, bound_addresses {192.168.50.207:9200}
[2019-04-08T23:22:37,792][INFO ][o.e.n.Node ] [ZVfIMzv] started
[2019-04-08T23:22:38,740][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [ZVfIMzv] Failed to clear cache for realms [[]]
[2019-04-08T23:22:38,850][INFO ][o.e.l.LicenseService ] [ZVfIMzv] license [41e1ad3d-893b-48c6-98b1-71e02ab1a367] mode [basic] - valid
[2019-04-08T23:22:38,873][INFO ][o.e.g.GatewayService ] [ZVfIMzv] recovered [0] indices into cluster_state
# 成功
驗(yàn)證
[thinktik@thinkvmc01 ~]$ netstat -nlp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
# 9200,9300 被ES監(jiān)聽(tīng)
tcp6 0 0 192.168.50.207:9200 :::* LISTEN 12829/java
tcp6 0 0 192.168.50.207:9300 :::* LISTEN 12829/java
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 ::1:25 :::* LISTEN -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
raw6 0 0 :::58 :::* 7 -
raw6 0 0 :::58 :::* 7 -
# 防火墻開(kāi)放端口
[root@thinkvmc01 thinktik]# firewall-cmd --zone=public --add-port=9200/tcp --permanent
success
[root@thinkvmc01 thinktik]# firewall-cmd --zone=public --add-port=9300/tcp --permanent
success
[root@thinkvmc01 thinktik]# firewall-cmd --reload
success
# thinkvmc02 主機(jī)驗(yàn)證 thinkvmc01 的ES效果。你用瀏覽器驗(yàn)證下面的地址也可以
[thinktik@thinkvmc02 ~]$ curl -i http://192.168.50.207:9200/
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 493
{
"name" : "ZVfIMzv",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "mhuFY2EcRl6Bt9xqKiyY7Q",
"version" : {
"number" : "6.7.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "2f32220",
"build_date" : "2019-04-02T15:59:27.961366Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
你用瀏覽器驗(yàn)證下面的地址也可以
到這里ES就安裝好了
thinkvmc02 先安裝Logstash# 驗(yàn)證java
[thinktik@thinkvmc02 java8]$ java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
# 下載
[thinktik@thinkvmc02 java8]$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.7.1.tar.gz
--2019-04-08 23:32:55-- https://artifacts.elastic.co/downloads/logstash/logstash-6.7.1.tar.gz
Resolving artifacts.elastic.co (artifacts.elastic.co)... 151.101.110.222, 2a04:4e42:36::734
Connecting to artifacts.elastic.co (artifacts.elastic.co)|151.101.110.222|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 175824421 (168M) [application/x-gzip]
Saving to: ‘logstash-6.7.1.tar.gz’
6% [==> ] 10,605,295 113KB/s eta 9m 57s
...
[thinktik@thinkvmc02 ~]$ ls
java8 jdk-8u201-linux-x64.tar.gz logstash-6.7.1.tar.gz
[thinktik@thinkvmc02 ~]$ tar -zxvf logstash-6.7.1.tar.gz
...
logstash-6.7.1/x-pack/src/test/java/org
logstash-6.7.1/x-pack/src/test/java/org/logstash
logstash-6.7.1/x-pack/src/test/java/org/logstash/xpack
logstash-6.7.1/x-pack/src/test/java/org/logstash/xpack/test
logstash-6.7.1/x-pack/src/test/java/org/logstash/xpack/test/RSpecIntegrationTests.java
logstash-6.7.1/x-pack/src/test/java/org/logstash/xpack/test/RSpecTests.java
logstash-6.7.1/LICENSE.txt
logstash-6.7.1/logstash-core/lib/logstash/build.rb
[thinktik@thinkvmc02 ~]$ cd logstash-6.7.1
[thinktik@thinkvmc02 logstash-6.7.1]$ ls
bin CONTRIBUTORS Gemfile lib logstash-core modules tools x-pack
config data Gemfile.lock LICENSE.txt logstash-core-plugin-api NOTICE.TXT vendor
[thinktik@thinkvmc02 logstash-6.7.1]$ cd config/
[thinktik@thinkvmc02 config]$ ls
jvm.options log4j2.properties logstash-sample.conf logstash.yml pipelines.yml startup.options
[thinktik@thinkvmc02 config]$ cp logstash-sample.conf logstash.conf
[thinktik@thinkvmc02 config]$ vim logstash.conf
# 這里將ES地址寫(xiě)對(duì)就可以了
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://192.168.50.207:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
[thinktik@thinkvmc02 config]$ vim logstash.yml
# 這里寫(xiě)正確自己的IP
# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
http.host: "192.168.50.19"
# 啟動(dòng)
[thinktik@thinkvmc02 bin]$ ./logstash -f ../config/logstash.conf
Sending Logstash logs to /home/thinktik/logstash-6.7.1/logs which is now configured via log4j2.properties
[2019-04-08T23:47:53,295][WARN ][logstash.config.source.multilocal] Ignoring the "pipelines.yml" file because modules or command line options are specified
[2019-04-08T23:47:53,324][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.7.1"}
[2019-04-08T23:48:08,245][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-04-08T23:48:09,323][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.50.207:9200/]}}
# 日志顯示ES地址對(duì)了
[2019-04-08T23:48:09,919][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://192.168.50.207:9200/"}
[2019-04-08T23:48:10,080][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-04-08T23:48:10,096][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won"t be used to determine the document _type {:es_version=>6}
[2019-04-08T23:48:10,174][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://192.168.50.207:9200"]}
[2019-04-08T23:48:10,250][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-04-08T23:48:10,318][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-04-08T23:48:11,308][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-04-08T23:48:11,360][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"}
[2019-04-08T23:48:11,499][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
# 日志顯示5044，9600被監(jiān)聽(tīng)
[2019-04-08T23:48:11,589][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-04-08T23:48:12,194][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
# 檢查下端口監(jiān)聽(tīng)
[thinktik@thinkvmc02 ~]$ netstat -nlp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
tcp6 0 0 :::5044 :::* LISTEN 27467/java
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 ::1:25 :::* LISTEN -
tcp6 0 0 192.168.50.19:9600 :::* LISTEN 27467/java
udp 0 0 0.0.0.0:68 0.0.0.0:* -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
raw6 0 0 :::58 :::* 7 -
raw6 0 0 :::58 :::* 7 -
# 防火墻打開(kāi)
[root@thinkvmc02 thinktik]# firewall-cmd --zone=public --add-port=9600/tcp --permanent
success
[root@thinkvmc02 thinktik]# firewall-cmd --zone=public --add-port=5044/tcp --permanent
success
[root@thinkvmc02 thinktik]# firewall-cmd --reload
success
到這里logstash安裝完畢
thinkvmc03 先安裝Kibana[thinktik@thinkvmc03 ~]$ java -version java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) # 修改配置 [thinktik@thinkvmc03 config]$ pwd /home/thinktik/kibana-6.7.1-linux-x86_64/config [thinktik@thinkvmc03 config]$ vim kibana.yml # 這里修改為自己的IP，端口默認(rèn)5601 # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "192.168.50.54" # 這里修改ES服務(wù)的地址 # The URLs of the Elasticsearch instances to use for all your queries. elasticsearch.hosts: ["http://192.168.50.54:9200"] # 啟動(dòng) [thinktik@thinkvmc03 bin]$ ./kibana log [16:04:24.455] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready log [16:04:24.507] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch log [16:04:24.510] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch log [16:04:24.523] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch # 檢查 [thinktik@thinkvmc03 config]$ netstat -nlp (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN - # 監(jiān)聽(tīng)正常 tcp 0 0 192.168.50.54:5601 0.0.0.0:* LISTEN 27474/./../node/bin tcp6 0 0 :::22 :::* LISTEN - tcp6 0 0 ::1:25 :::* LISTEN - udp 0 0 0.0.0.0:68 0.0.0.0:* - udp 0 0 0.0.0.0:68 0.0.0.0:* - raw6 0 0 :::58 :::* 7 - raw6 0 0 :::58 :::* 7 - # 防火墻開(kāi)放端口 [root@thinkvmc03 config]# firewall-cmd --zone=public --add-port=5601/tcp --permanent success [root@thinkvmc03 config]# firewall-cmd --reload success
kibana效果
到這里就完成了ELK基礎(chǔ)搭建
thinkvmc02 先安裝Filebeat接下來(lái)我們安裝 Filebeat，使用ELKF架構(gòu)來(lái)實(shí)現(xiàn)log4j的日志收集。
為了方便Filebeat安裝在thinkvmc03上與thinkvmc02的Logstash形成分布式結(jié)構(gòu)來(lái)模擬日志數(shù)據(jù)的收集與傳輸
官方的安裝教程也很簡(jiǎn)單，屬于基本操作了
[thinktik@thinkvmc03 ~]$ tar -zxvf filebeat-6.7.1-linux-x86_64.tar.gz
filebeat-6.7.1-linux-x86_64/.build_hash.txt
filebeat-6.7.1-linux-x86_64/fields.yml
filebeat-6.7.1-linux-x86_64/LICENSE.txt
filebeat-6.7.1-linux-x86_64/NOTICE.txt
filebeat-6.7.1-linux-x86_64/kibana/
filebeat-6.7.1-linux-x86_64/kibana/5/
filebeat-6.7.1-linux-x86_64/
...
filebeat-6.7.1-linux-x86_64/module/traefik/access/machine_learning/visitor_rate.json
filebeat-6.7.1-linux-x86_64/module/traefik/access/manifest.yml
filebeat-6.7.1-linux-x86_64/module/traefik/module.yml
filebeat-6.7.1-linux-x86_64/filebeat.reference.yml
filebeat-6.7.1-linux-x86_64/filebeat
# 修改配置使filebeat對(duì)準(zhǔn)我們的輸出
[thinktik@thinkvmc03 filebeat-6.7.1-linux-x86_64]$ vim filebeat.yml
#=========================== Filebeat inputs =============================
# 設(shè)置Filebeat讀取/home/thinktik/ELKF_TEST.log日志
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
# 這里設(shè)為T(mén)rue開(kāi)啟日志讀入
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /home/thinktik/ELKF_TEST.log
#- /var/log/*.log
#- c:programdataelasticsearchlogs*
#-------------------------- Elasticsearch output ------------------------------
# 直接輸出到Elasticsearch 這里我們不建議直接輸出
#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["192.168.50.207:9200"]
# Enabled ilm (beta) to use index lifecycle management instead daily indices.
#ilm.enabled: false
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
# 這里才是Logstash，直接輸出到logstash 這里我們建議直接輸出，地址配對(duì)就可以
output.logstash:
# The Logstash hosts
hosts: ["192.168.50.19:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
# 保存后啟動(dòng)
[thinktik@thinkvmc03 filebeat-6.7.1-linux-x86_64]$ ./filebeat
# 后續(xù)的你直接修改/home/thinktik/ELKF_TEST.log，寫(xiě)入一些數(shù)據(jù)到這個(gè)文件里面等待Kibana顯示出來(lái)效果
# 我們先理下思路，流程是：filebeat -> Logstash -> ES -> Kibana
# 如果沒(méi)有問(wèn)題，那么我們?cè)貹ibana檢查效果
檢查ELKF效果
這里看到日志被正確讀取
我們?cè)偌?xì)節(jié)設(shè)置下
我們搜索下看看
先搜索主機(jī) host.name=thinkvmc03 的日志
再搜索主機(jī) host.name=thinkvmc03 的日志 而且日志源文件是 source =/home/thinktik/ELKF_TEST.log的
看到匹配正確
我們繼續(xù)收集下其他類(lèi)型的日志，那log4j來(lái)試一下
本文原創(chuàng)鏈接
Linux ELK 安裝（服務(wù)器架設(shè)篇）
參考鏈接
https://juejin.im/entry/59e6b...
文章版權(quán)歸作者所有，未經(jīng)允許請(qǐng)勿轉(zhuǎn)載,若此文章存在違規(guī)行為，您可以聯(lián)系管理員刪除。
轉(zhuǎn)載請(qǐng)注明本文地址：http://systransis.cn/yun/77528.html
閱讀 1992·2021-11-24 09:39
閱讀 2161·2021-09-22 15:50
閱讀 2084·2021-09-22 14:57
閱讀 738·2021-07-28 00:13
閱讀 1101·2019-08-30 15:54
閱讀 2385·2019-08-30 15:52
閱讀 2715·2019-08-30 13:07
閱讀 3815·2019-08-30 11:27
