ELK Stack企业级日志平台搭建从0到1
ELK需求背景
（1）开发人员经常需要去服务器后端查看日志，为了排除问题。而对于开发人员后来说，不可能拥有服务器的绝对权限。而且服务器数量很可能激增。
（2）服务器数量激增，项目数量激增，日志类型众多。
对于Linux运维人员来说，一般采用awk, sed,grep 三种命令来对日志进行stdout（csv）.但是这种纯文本输出不易于阅读，需要采用开源画图JS进行日志展示（比如：百度开源的Echarts: https://echarts.baidu.com/）
Logstash: 开源的服务器端数据处理管道，能够同时从多个来源采集数据、转换数据，然后将数据存储到数据库中。（抽取日志，制定规则）
Elasticsearch：搜索、分析和存储数据。(分布式数据库，相当于一个管道)
Kibana：数据可视化。
Beats ：轻量型采集器的平台，从边缘机器向Logstash 和Elasticsearch 发送数据。
Filebeat：轻量型日志采集器。
https://www.elastic.co/cn/
https://www.elastic.co/subscriptions
ELK Stack架构
Input：输入，输出数据可以是Stdin、File、TCP、Redis、Syslog等。（数据源，redis, mysql ）
https://www.elastic.co/guide/en/logstash/current/input-plugins.html
Filter：过滤，将日志格式化。有丰富的过滤插件：Grok正则捕获、Date时间处理、Json编解码、Mutate数据修改等。(将数据结构化).
https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
Output：输出，输出目标可以是Stdout、File、TCP、Redis、ES等。
https://www.elastic.co/guide/en/logstash/current/output-plugins.html
ElasticSearch
基本概念
Node：运行单个ES实例的服务器.
Cluster：一个或多个节点构成集群
Index：索引是多个文档的集合
Document：Index里每条记录称为Document，若干文档构建一个Index
Type：一个Index可以定义一种或多种类型，将Document逻辑分组
Field：ES存储的最小单元
Shards：ES将Index分为若干份，每一份就是一个分片
Replicas：Index的一份或多份副本。
| ElasticSearch | 关系型数据库（比如MySQL） |
|---|---|
| Index | Database |
| Type | Table |
| Document | Row |
| Field | Column |
环境配置
| 组件名称 | IP设定 |
|---|---|
| elk-web | 172.16.4.22 |
| elk-lk | 172.16.4.21 |
| elk-es3 | 172.16.4.20 |
| elk-es2 | 172.16.4.19 |
| elk-es1 | 172.16.4.18 |
安装要求
以下操作需要在所有组件上执行：
# 安装网络插件
yum install -y net-tools
# 安装vim文本编辑器
yum install -y vim
# 确认各组件之间网络是互通的，以确保之间能相互通信。
iptables -F
iptables -I INPUT -s 172.16.4.0/24 -j ACCEPT
# 当然也可以设置开启启动生效,在/etc/rc.loal下加入iptables -I INPUT -s 172.16.4.0/24 -j ACCEPT
vim /etc/rc.local
chmod +x /etc/rc.local
# 关闭selinux，修改SELINUX=disabled 并重启生效。
vim /etc/selinux/config
执行getenforce，确保得到的值为Disabled。
# 修改时区信息，使得各节点之间时间同步
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
yum install -y ntpdate -y; ntpdate time.windows.com
# 安装上传下载拖拽工具
yum -y install lrzsz
ElasticSearch集群部署
在elk-es1，elk-es2，elk-es3三台机器上安装环境
yum install java-1.8.0-openjdk -y
安装Logstash的方式：https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#package-repositories
此处采用yum安***r> （1）首先导入公钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
（2）添加其yum源
# elastic.repo不存在则创建
[root@elk-es1 ~]# cat /etc/yum.repos.d/elastic.repo
[elastic-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
（3）安装elasticsearch
yum install -y elasticsearch
- 注意，安装时间挺长的，取决于网速。
（4）修改 /etc/elasticsearch/elasticsearch.yml，配置如下参数
# 此处以第一个节点为例，其余节点请修改相应的node.name和network.host值
cluster.name: elk-cluster
node.name: node-1 #此处以第一个es节点为例，后续请修改此值。
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# 注意：172.16.4.18是第一个es节点的IP，其余两个请自行更换掉。
network.host: 172.16.4.18
http.port: 9200
discovery.seed_hosts: ["172.16.4.18", "172.16.4.19", "172.16.4.20"]
cluster.initial_master_nodes: ["172.16.4.18", "172.16.4.19"]
或cluster.initial_master_nodes: ["node-1", "node-2"]
# 你可以通过为 cluster.initial_master_nodes 参数设置一系列符合主节点条件的节点的主机名或 IP 地址来引导启动集群。
（5）启动elasticsearch
systemctl restart elasticsearch
# 以第一个节点为例：查看启动日志，发现node-1已加入集群中。
[root@elk-es1 ~]# tail -f /var/log/elasticsearch/elk-cluster.log
[2019-06-20T10:20:03,555][INFO ][o.e.n.Node ] [node-1] initialized
[2019-06-20T10:20:03,555][INFO ][o.e.n.Node ] [node-1] starting ...
[2019-06-20T10:20:03,778][INFO ][o.e.t.TransportService ] [node-1] publish_address {172.16.4.18:9300}, bound_addresses {172.16.4.18:9300}
[2019-06-20T10:20:03,786][INFO ][o.e.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-06-20T10:20:03,813][INFO ][o.e.c.c.Coordinator ] [node-1] cluster UUID [NH-xRtJlSxedjkkTTU6W6w]
[2019-06-20T10:20:04,322][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous [], current [{node-2}{FMT_mamATbaSJN4jTO0HgA}{TxDTa1-pQ_mAV5eAeYqaoA}{172.16.4.19}{172.16.4.19:9300}{ml.machine_memory=16629563392, ml.max_open_jobs=20, xpack.installed=true}]}, added {{node-2}{FMT_mamATbaSJN4jTO0HgA}{TxDTa1-pQ_mAV5eAeYqaoA}{172.16.4.19}{172.16.4.19:9300}{ml.machine_memory=16629563392, ml.max_open_jobs=20, xpack.installed=true},{node-3}{Q1gJOBGLQLa_tT47v_wAAA}{gH9EbgR6QU2dozCVIqFCbw}{172.16.4.20}{172.16.4.20:9300}{ml.machine_memory=16629559296, ml.max_open_jobs=20, xpack.installed=true},}, term: 40, version: 21, reason: ApplyCommitRequest{term=40, version=21, sourceNode={node-2}{FMT_mamATbaSJN4jTO0HgA}{TxDTa1-pQ_mAV5eAeYqaoA}{172.16.4.19}{172.16.4.19:9300}{ml.machine_memory=16629563392, ml.max_open_jobs=20, xpack.installed=true}}
[2019-06-20T10:20:04,355][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {172.16.4.18:9200}, bound_addresses {172.16.4.18:9200}
[2019-06-20T10:20:04,356][INFO ][o.e.n.Node ] [node-1] started
[2019-06-20T10:20:04,567][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [node-1] Failed to clear cache for realms [[]]
[2019-06-20T10:20:04,594][INFO ][o.e.l.LicenseService ] [node-1] license [9452dc51-8874-4ac1-82f3-bb9f6002dc7e] mode [basic] - valid
[2019-06-20T10:20:04,854][INFO ][o.e.x.s.a.TokenService ] [node-1] refresh keys
[2019-06-20T10:20:05,039][INFO ][o.e.x.s.a.TokenService ] [node-1] refreshed keys
（6）校验集群健康性：https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-cluster-health.html
- 点击上述链接中的COPY AS CURL，复制粘贴到某个节点，注意修改IP值。
[root@elk-es3 ~]# curl -X GET "172.16.4.20:9200/_cat/health?v"
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1560997781 02:29:41 elk-cluster green 3 3 0 0 0 0 0 0 - 100.0%
# 集群状态显示“Green”，表示集群健康。还有其他“Yellow和Red”两种状态。
- 查看集群节点
<https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-cluster-health.html>点击链接中的下图红框，并粘贴到终端：
[root@elk-es1 ~]# curl -X GET "172.16.4.20:9200/_cat/nodes?v"
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.16.4.18 25 16 0 0.03 0.02 0.05 mdi - node-1
172.16.4.20 26 16 0 0.00 0.01 0.05 mdi - node-3
172.16.4.19 15 16 0 0.00 0.01 0.05 mdi * node-2
# heap.percent是堆内存的百分比
## ElasticSearch-数据操作
RestFul API格式
curl -X<verb> ‘<protocol>://<host>:<port>/<path>?<query_string>’-d ‘<body>’
-X 指定动作：（get post, put等）
| 参数 | 描述 |
| ------------ | ------------------------------------------------------- |
| verb | HTTP方法，比如GET、POST、PUT、HEAD、DELETE |
| host | ES集群中的任意节点主机名 |
| port | ES HTTP服务端口，默认9200 |
| path | 索引路径 |
| query_string | 可选的查询请求参数。例如?pretty参数将格式化输出JSON数据 |
| -d | 里面放一个GET的JSON格式请求主体 |
| body | 自己写的 JSON格式的请求主体 |
### 增删改
（1）列出索引：
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
# 当下没有创建索引，所以默认显示上述结果。
（2）创建索引：
[root@elk-es1 ~]# curl -X PUT "172.16.4.18:9200/logs-test-2019.06.20?pretty"
{
"error" : {
"root_cause" : [
{
"type" : "resource_already_exists_exception",
"reason" : "index [logs-test-2019.06.20/TbyeRd2pQZCciyJ4O5mRog] already exists",
"index_uuid" : "TbyeRd2pQZCciyJ4O5mRog",
"index" : "logs-test-2019.06.20"
}
],
"type" : "resource_already_exists_exception",
"reason" : "index [logs-test-2019.06.20/TbyeRd2pQZCciyJ4O5mRog] already exists",
"index_uuid" : "TbyeRd2pQZCciyJ4O5mRog",
"index" : "logs-test-2019.06.20"
},
"status" : 400
}
# 其中?pretty表示是采用json格式化显示结果，再次查看索引，发现已经发生变化。
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logs-test-2019.06.20 TbyeRd2pQZCciyJ4O5mRog 1 1 0 0 460b 230b
（3）查询文档
# 向创建的索引中添加内容。
[root@elk-es1 ~]# curl -X PUT "172.16.4.18:9200/logs-test-2019.06.20/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
"name": "Ge Zirong"
}
'
# 返回如下结果
{
"_index" : "logs-test-2019.06.20",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}
# 获取索引内容
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/logs-test-2019.06.20/_doc/1?pretty"
{
"_index" : "logs-test-2019.06.20",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"_seq_no" : 0,
"_primary_term" : 1,
"found" : true,
"_source" : {
"name" : "Ge Zirong"
}
}
（4）删除索引
[root@elk-es1 ~]# curl -X DELETE "172.16.4.18:9200/logs-test-2019.06.20?pretty"
{
"acknowledged" : true
}
# 如果再次查看索引内容，会返回404错误。
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/logs-test-2019.06.20/_doc/1?pretty"
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index [logs-test-2019.06.20]",
"resource.type" : "index_expression",
"resource.id" : "logs-test-2019.06.20",
"index_uuid" : "_na_",
"index" : "logs-test-2019.06.20"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index [logs-test-2019.06.20]",
"resource.type" : "index_expression",
"resource.id" : "logs-test-2019.06.20",
"index_uuid" : "_na_",
"index" : "logs-test-2019.06.20"
},
"status" : 404
}
（5）修改数据
# 重新创建
[root@elk-es1 ~]# curl -X PUT "172.16.4.18:9200/logs-test-2019.06.20/_doc/1?pretty" -H 'Content-Type: application/json' -d'{
"name": "Zhangsan"
}
'
# 再次查看，确保索引创建成功
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/logs-test-2019.06.20/_doc/1?pretty"
# 直接修改
[root@elk-es1 ~]# curl -X PUT "172.16.4.18:9200/logs-test-2019.06.20/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
"name": "Lisi"
}
'
{
"_index" : "logs-test-2019.06.20",
"_type" : "_doc",
"_id" : "1",
"_version" : 2,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 1,
"_primary_term" : 1
}
# 发现上述结果是更新操作，Zhangsan 变成了Lisi
# 通过POST进行修改
[root@elk-es1 ~]# curl -X POST "172.16.4.18:9200/logs-test-2019.06.20/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
"name": "Wangwu"
}
'
{
"_index" : "logs-test-2019.06.20",
"_type" : "_doc",
"_id" : "1",
"_version" : 3,
"result" : "updated",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 2,
"_primary_term" : 1
}
# 修改Document的字段，增加age字段
[root@elk-es1 ~]# curl -X POST "172.16.4.18:9200/logs-test-2019.06.20/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
"name": "Wangwu","age": 20
}
'
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/logs-test-2019.06.20/_doc/1?pretty"
{
"_index" : "logs-test-2019.06.20",
"_type" : "_doc",
"_id" : "1",
"_version" : 4,
"_seq_no" : 3,
"_primary_term" : 1,
"found" : true,
"_source" : {
"name" : "Wangwu",
"age" : 20
}
}
多种查询方式
# 下载官方提供的数据进行
[root@elk-es1 ~]# wget https://raw.githubusercontent.com/elastic/elasticsearch/master/docs/src/test/resources/accounts.json
# 导入数据
[root@elk-es1 ~]# curl -H "Content-Type: application/json" -XPOST "172.16.4.18:9200/bank/_bulk?pretty&refresh" --data-binary "@accounts.json"
# 查看数据
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open bank XHBHpyl-SxiLlaWaRzta1A 1 1 1000 0 841.2kb 414.3kb
green open logs-test-2019.06.20 1sV4KMz6SdKapiUoOkmgrA 1 1 1 2 13.1kb 9.3kb
（1）多类型查询
https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-search-API.html
- match_all查询
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?q=*&sort=account_number:asc&pretty"
# 将查询结果返回，通过search接口查询，q=*表示所有文档 asc升序排序。
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search" -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"sort": [
{ "account_number": "asc" }
]
}
'
- from-size查询方式
# 1. 规定记录条数，size=1
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"size": 1
}
'
# 返回结果如下
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
> {
> "query": { "match_all": {} },
> "size": 1
> }
> '
{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1000,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "bank",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"account_number" : 1,
"balance" : 39225,
"firstname" : "Amber",
"lastname" : "Duke",
"age" : 32,
"gender" : "M",
"address" : "880 Holmes Lane",
"employer" : "Pyrami",
"email" : "amberduke@pyrami.com",
"city" : "Brogan",
"state" : "IL"
}
}
]
}
}
# 2. 规定记录的范围
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search" -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"from": 10,
"size": 10
}
'
# 易于用于分页显示
# 3. 指定字段排序
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"_source": ["account_number", "balance"]
}
'
{
"took" : 10,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1000,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "bank",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"account_number" : 1,
"balance" : 39225
}
},
......
]
}
}
- match（根据字段输出结果）
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
# 显示结果如下
{
"query": { "match_all": {} },
"_source": ["account_number", "balance"]
}
'
{
"took" : 10,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1000,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "bank",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"account_number" : 1,
"balance" : 39225
}
},
......
]
}
}
# match模糊查询，不区分大小写
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": { "match": { "address": "mill" } }
}
'
# 匹配或关系查询
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": { "match": { "address": "mill lane" } }
}
'
- bool值查询
# 1. 与查询，查询包含mill和lane的记录
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'
# 2. 或查询，查询包含
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"should": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'
# 3. “非”查询
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must_not": [
{ "match": { "address": "mill" } },
{ "match": { "address": "lane" } }
]
}
}
}
'
# 4. 混合查询，精确查询
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must": [
{ "match": { "age": "40" } }
],
"must_not": [
{ "match": { "state": "ID" } }
]
}
}
}
'
- filter（范围查询）
# 1. 数值范围查询(20000<=balance<=30000),也可以采用时间之类的范围查询
[root@elk-es1 ~]# curl -X GET "172.16.4.18:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"must": { "match_all": {} },
"filter": {
"range": {
"balance": {
"gte": 20000,
"lte": 30000
}
}
}
}
}
}
'
Head插件图形管理ElasticSearch
以下操作只在es1:172.16.4.18上操作：
[root@elk-es1 ~]# wget https://npm.taobao.org/mirrors/node/latest/node-v12.4.0-linux-x64.tar.gz
[root@elk-es1 ~]# tar zxvf node-v12.4.0-linux-x64.tar.gz
# 将解压的文件移动
[root@elk-es1 ~]# mv node-v12.4.0-linux-x64 /usr/local/node-v12.4
# 配置环境变量
vim /etc/profile
# 并加入以下内容
NODE_NAME=/usr/local/node-v12.4
PATH=$NODE_NAME/bin:$PATH
export NODE_HOME PATH
source /etc/profile # 使其生效
# 验证安装成功与否
[root@elk-es1 ~]# npm -v
6.9.0
[root@elk-es1 ~]# node -v
v12.4.0
# 安装git客户端并拉取
yum install -y git
git clone https://github.com/mobz/elasticsearch-head.git
# 进入elasticsearch-head并安装npm
cd elasticsearch-head/ && npm install
# 编辑Gruntfile.js，在95行后加入hostname: '*',以监听所有地址。
vim Gruntfile.js
......
port: 9100,
base: '.',
keepalive: true,
hostname: '*'
# 启动npm
[root@elk-es1 elasticsearch-head]# npm run start
> elasticsearch-head@0.0.0 start /root/elasticsearch-head
> grunt server
>> Local Npm module "grunt-contrib-jasmine" not found. Is it installed?
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100
# 此时需要修改/etc/elasticsearch/elasticsearch.yml，在文档最后加入下面两行内容
vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: "*"
# 重启elasticsearch使其生效
[root@elk-es1 ~]# systemctl restart elasticsearch
# 查看日志，看到“started”字样表明启动成功
[root@elk-es1 ~]# tail /var/log/elasticsearch/elk-cluster.log -f
......
[2019-06-22T11:30:25,163][INFO ][o.e.n.Node ] [node-1] started
....
- 在浏览器中输入172.16.4.18:9100即可访问
- 可以根据字段来查询
- 复合查询
- 查看集群信息（点击右上角的“信息”按钮）
Logstash
安装
以下操作在elk-lk：172.16.4.21上执行。
# 先从es1上传入yum源过来。
[root@elk-es1 ~]# scp /etc/yum.repos.d/elastic.repo root@172.16.4.21:/etc/yum.repos.d/
# 安装jdk
[root@elk-lk ~]# yum install java-1.8.0-openjdk -y
# 在elk节点上安装
[root@elk-lk ~]# yum install -y logstash
# 查看命令执行路径
[root@elk-lk ~]# rpm -ql logstash |more
....
/usr/share/logstash/bin/logstash
......
# 将执行路径写入环境变量，写入/etc/profile
PATH=$PATH:/usr/share/logstash/bin
export PATH
source /etc/profile
# 查看安装版本信息
[root@elk-lk ~]# logstash -V
logstash 7.1.1
Logstash-条件判断
比较操作符：
相等: ==, !=, <, >, <=, >=
正则: =~(匹配正则), !~(不匹配正则)
包含: in(包含), not in(不包含)
布尔操作符：
and(与), or(或), nand(非与), xor(非或)
一元运算符：
!(取反)
()(复合表达式), !()(对复合表达式结果取反)
Input插件
（1）Stdin：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-stdin.html
[root@elk-lk ~]# vim /etc/logstash/conf.d/test.conf
# 输入以下内容。
input {
stdin {
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
}
# 测试配置文件正确性，如下：-t表示测试，-f 指定文件位置
[root@elk-lk ~]# logstash -t -f /etc/logstash/conf.d/test.conf
......
Configuration OK
[INFO ] 2019-06-22 16:46:25.010 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
.......
# 直接运行，加-r参数
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-22 16:50:27.933 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
345
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"@version" => "1",
"@timestamp" => 2019-06-22T08:50:53.993Z,
"message" => "345",
"host" => "elk-lk"
}
124
{
"@version" => "1",
"@timestamp" => 2019-06-22T08:50:56.786Z,
"message" => "124",
"host" => "elk-lk"
}
# 如上键入123 345,都能形成timestamp日志，系统帮你加入时间戳
（2）Flie示例：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
# 重新开一个终端：elk-lk-1:172.16.4.21 更改/etc/logstash/conf.d/test.conf内容如下：
input {
file {
path =>"/var/log/messages"
tags =>"123"
type =>"syslog"
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
}
# 重新执行
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-22 16:57:47.013 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2019-06-22 16:57:47.094 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-06-22 16:57:47.103 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2019-06-22 16:57:47.376 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"path" => "/var/log/messages",
"type" => "syslog",
"@timestamp" => 2019-06-22T09:01:02.150Z,
"@version" => "1",
"message" => "Jun 22 17:01:01 elk-lk systemd: Started Session 85 of user root.",
"tags" => [
[0] "123"
],
"host" => "elk-lk"
}
# 输出内容如上
# 可以重开一个终端，安装httpd,使其产生日志
[root@elk-lk ~]# yum install -y httpd
......
---> Package httpd-tools.x86_64 0:2.4.6-89.el7.centos will be installed
......
# 返回另外一个终端，查看到/var/log/messages下出现很多日志,其中有一条与上述对应：
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-22 17:08:50.190 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
......
{
"path" => "/var/log/messages",
"@version" => "1",
"host" => "elk-lk",
"tags" => [
[0] "123"
],
"type" => "syslog",
"@timestamp" => 2019-06-22T09:13:50.914Z,
"message" => "Jun 22 17:13:50 elk-lk yum[10167]: Installed: httpd-tools-2.4.6-89.el7.centos.x86_64"
}
......
}
（3）TCP示例：https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html
# 更改/etc/logstash/conf.d/test.conf内容如下：
input {
tcp {
port =>12345
type =>"nc"
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
}
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
# 查看是否监听到12345
[root@elk-lk ~]# netstat -antp | grep 12345
tcp6 0 0 :::12345 :::* LISTEN 10436/java
# 采用nc发送数据，为此在elk-web:172.16.4.22上安装nc
[root@elk-web ~]# yum install -y nc
# 同时关闭两端（elk-web和elk-lk）的iptabels
iptables -F
# 在elk-web采用nc发送TCP数据；
[root@elk-web ~]# nc 172.16.4.21 12345
1
2
3
gezirong
# 在elk-lk对端能够监听并接受到
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-22 20:51:32.979 [[main]<tcp] tcp - Starting tcp input listener {:address=>"0.0.0.0:12345", :ssl_enable=>"false"}
[INFO ] 2019-06-22 20:51:33.054 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-06-22 20:51:33.286 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"type" => "nc",
"@timestamp" => 2019-06-22T12:53:17.524Z,
"host" => "elk-web",
"@version" => "1",
"message" => "1",
"port" => 55440
}
{
"type" => "nc",
"@timestamp" => 2019-06-22T12:53:23.484Z,
"host" => "elk-web",
"@version" => "1",
"message" => "2",
"port" => 55440
}
{
"type" => "nc",
"@timestamp" => 2019-06-22T12:53:24.977Z,
"host" => "elk-web",
"@version" => "1",
"message" => "3",
"port" => 55440
}
{
"type" => "nc",
"@timestamp" => 2019-06-22T12:53:28.152Z,
"host" => "elk-web",
"@version" => "1",
"message" => "gezirong",
"port" => 55440
}
# 上述键入1 2 3 gezirong都被作为一个nc数据监听到记录下来了。
Logstash Codec常用插件
官方参考地址：https://www.elastic.co/guide/en/logstash/current/codec-plugins.html
（1）Json/Json_lines示例：
# 更改/etc/logstash/conf.d/test.conf为以下内容：
input {
stdin {
codec =>json {
charset => ["UTF-8"]
}
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
}
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
.........
[INFO ] 2019-06-22 21:06:04.902 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 粘贴如下内容，json必须为一行。
{"ip":"172.16.4.18","hostname":"elk-es1","method":"GET"}
# 按回车就能识别出来json数据
{
"@version" => "1",
"@timestamp" => 2019-06-22T13:21:33.539Z,
"method" => "GET",
"ip" => "172.16.4.18",
"host" => "elk-lk",
"hostname" => "elk-es1"
}
（2）Multiline示例：https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html
日志可以按照特征进行聚合，避免报重复错误。
# 更改/etc/logstash/conf.d/test.conf为以下内容，并保存退出：
# pattern => "^\s"表示匹配以字符开头的行，what => "previous" 表示将不是以字符串开头的行聚合到上一行。
input {
stdin {
codec => multiline {
pattern => "^\s"
what => "previous"
}
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
}
# 执行文件
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
.....
[INFO ] 2019-06-22 21:35:50.908 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 键入以下内容
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
# 显示内容如下，把日志聚合了：
{
"host" => "elk-lk",
"@timestamp" => 2019-06-22T13:42:52.708Z,
"tags" => [
[0] "multiline"
],
"@version" => "1",
"message" => "Caused by: java.lang.reflect.InvocationTargetException\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)"
}
Logstash Filter常用插件
主要对日志进行结构化处理。
官方文档：https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
json插件
https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html
# 更改/etc/logstash/conf.d/test.conf为以下内容，并保存退出：
# 其中source是指定待转化的字段，
input {
stdin {
}
}
filter {
json {
source => "message"
target => "content"
}
}
output {
stdout {
codec => rubydebug
}
}
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-22 21:52:14.165 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 键入以下内容，并回车
{"ip":"172.16.4.18","hostname":"elk-es1","method":"GET"}
# 显示结果如下，把message中的数据结构化到content中了,因为上述test.conf明确指定了message结构化指定的字段名称。
{
"content" => {
"ip" => "172.16.4.18",
"hostname" => "elk-es1",
"method" => "GET"
},
"message" => "{\"ip\":\"172.16.4.18\",\"hostname\":\"elk-es1\",\"method\":\"GET\"}",
"@timestamp" => 2019-06-22T13:54:32.910Z,
"host" => "elk-lk",
"@version" => "1"
}
kv插件
https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html
主要用于拆分一个字符串。
# 更改/etc/logstash/conf.d/test.conf为以下内容，并保存退出
input {
stdin {
}
}
filter {
kv {
field_split => "&?"
}
}
output {
stdout {
codec => rubydebug
}
}
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-22 22:05:44.802 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 输入官方提供的以下示例内容按回车
pin=12345~0&d=123&e=foo@bar.com&oq=bobo&ss=12345?a=123
# 显示内容如下：其中用“&”和“?”将字符分割了，采用的是key=value进行解析的。
{
"@timestamp" => 2019-06-22T14:14:58.026Z,
"host" => "elk-lk",
"pin" => "12345~0",
"e" => "foo@bar.com",
"ss" => "12345",
"a" => "123",
"message" => "pin=12345~0&d=123&e=foo@bar.com&oq=bobo&ss=12345?a=123",
"@version" => "1",
"d" => "123",
"oq" => "bobo"
}
# 当然也可以使用正则匹配，更改/etc/logstash/conf.d/test.conf中kv里的值。
input {
stdin {
}
}
filter {
kv {
field_split_pattern => ":+"
}
}
output {
stdout {
codec => rubydebug
}
}
# 重新启动
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-22 22:18:57.477 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 键入以下内容
k1=v1:k2=v2::k3=v3:::k4=v4
# 显示如下：可以自动忽略多个:
{
"k4" => "v4",
"k3" => "v3",
"message" => "k1=v1:k2=v2::k3=v3:::k4=v4",
"k2" => "v2",
"host" => "elk-lk",
"k1" => "v1",
"@timestamp" => 2019-06-22T14:22:58.473Z,
"@version" => "1"
}
GeoIP示例
是获取IP地理位置信息的插件
官方文档：https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html
下载GeoIP的官方城市数据库：https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz放在elk-lk服务器上。
[root@elk-lk ~]# wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
[root@elk-lk ~]# tar zxvf GeoLite2-City.tar.gz
GeoLite2-City_20190618/
GeoLite2-City_20190618/LICENSE.txt
GeoLite2-City_20190618/GeoLite2-City.mmdb
GeoLite2-City_20190618/COPYRIGHT.txt
GeoLite2-City_20190618/README.txt
# 将最新更新的城市数据拷贝到/opt下。
[root@elk-lk ~]# cp GeoLite2-City_20190618/GeoLite2-City.mmdb /opt/
# 更改/etc/logstash/conf.d/test.conf为以下内容，并保存退出
input {
stdin {
}
}
filter {
grok {
match => {
"message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"
}
}
geoip {
source => "client"
database => "/opt/GeoLite2-City.mmdb"
}
}
output {
stdout {
codec => rubydebug
}
}
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-22 22:40:44.731 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 输入以下内容
202.201.1.20 GET /lab_team.html 500 0.1
# 返回如下内容，如显示地理位置为兰州。
{
"request" => "/lab_team.html",
"message" => "202.201.1.20 GET /lab_team.html 500 0.1",
"geoip" => {
"region_name" => "Gansu",
"country_name" => "China",
"location" => {
"lat" => 36.057,
"lon" => 103.8399
},
"timezone" => "Asia/Shanghai",
"country_code3" => "CN",
"country_code2" => "CN",
"continent_code" => "AS",
"ip" => "202.201.1.20",
"region_code" => "GS",
"city_name" => "Lanzhou",
"longitude" => 103.8399,
"latitude" => 36.057
},
"duration" => "0.1",
"client" => "202.201.1.20",
"@timestamp" => 2019-06-23T01:56:12.786Z,
"@version" => "1",
"host" => "elk-lk",
"bytes" => "500",
"method" => "GET"
}
date示例
主要用于数据的时间进行修改。即匹配日志中时间戳，而不是处理日志的时间戳。
官方地址：https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html
Grok插件
正则匹配非结构化的数据进行结构化的输出，方便前台展示。
官方地址：https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
grok有很多的正则匹配模式，详见：https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
# 更改/etc/logstash/conf.d/test.conf为以下内容，并保存退出
input {
stdin {
}
}
filter {
grok {
match => {
"message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"
}
}
}
output {
stdout {
codec => rubydebug
}
}
# 启动logstash
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-23 10:29:59.401 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 输入以下内容，回车
202.201.1.20 GET /lab_team.html 500 0.1
# 显示内容如下：
{
"@version" => "1",
"bytes" => "500",
"@timestamp" => 2019-06-23T02:59:10.150Z,
"request" => "/lab_team.html",
"message" => "202.201.1.20 GET /lab_team.html 500 0.1",
"duration" => "0.1",
"method" => "GET",
"client" => "202.201.1.20",
"host" => "elk-lk"
}
[root@elk-lk ~]# rpm -ql logstash | grep grok-patterns
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
[root@elk-lk ~]# cat /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
# 上述结果的输出与官网中的grok-patterns内容一致
# 更改/etc/logstash/conf.d/test.conf为以下内容，并保存退出
input {
stdin {
}
}
filter {
grok {
patterns_dir =>"/opt/patterns"
match => {
"message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} %{ID:id}"
}
}
}
output {
stdout {
codec => rubydebug
}
}
# 自定义模式
vim /opt/patterns
# 写入以下内容，匹配10-11位上的数字和大写字母
ID [0-9A-Z]{10,11}
# 启动
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-23 19:04:57.425 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 输入以下内容
202.201.1.20 GET /lab_team.html 500 0.1 123456
# 显示内容如下，由于123456不足10位，所以显示匹配不成功：_grokparsefailure
{
"tags" => [
[0] "_grokparsefailure"
],
"host" => "elk-lk",
"message" => "202.201.1.20 GET /lab_team.html 500 0.1 123456",
"@version" => "1",
"@timestamp" => 2019-06-23T11:07:53.062Z
}
# 虽然位数够了，但是第10位上不是数字或者大写字母，所以照样失败
202.201.1.20 GET /lab_team.html 500 0.1 123456789a
{
"tags" => [
[0] "_grokparsefailure"
],
"host" => "elk-lk",
"message" => "202.201.1.20 GET /lab_team.html 500 0.1 123456789a",
"@version" => "1",
"@timestamp" => 2019-06-23T11:11:15.303Z
}
# 匹配成功
202.201.1.20 GET /lab_team.html 500 0.1 123456789A
{
"message" => "202.201.1.20 GET /lab_team.html 500 0.1 123456789A",
"client" => "202.201.1.20",
"@version" => "1",
"@timestamp" => 2019-06-23T11:11:25.405Z,
"id" => "123456789A",
"host" => "elk-lk",
"request" => "/lab_team.html",
"method" => "GET",
"bytes" => "500",
"duration" => "0.1"
}
- 如果自己自定义的模式，需要调试，可以在grok工具下调试：http://grokdebug.herokuapp.com/
（5）多模式匹配：
# 更改/etc/logstash/conf.d/test.conf为以下内容，并保存退出。
input {
stdin {
}
}
filter {
grok {
patterns_dir =>"/opt/patterns"
match => [
"message","%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} %{ID:id}",
"message","%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} %{TAG:tag}"
]
}
}
output {
stdout {
codec => rubydebug
}
}
# 修改/opt/patterns内容为如下：
ID [0-9A-Z]{10-11}
TAG SYSLOG1
# 此时grok应该能匹配两种日志格式：SYSLOG1 和 ID的
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/test.conf
......
[INFO ] 2019-06-23 19:51:30.492 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 输入以下内容回车：
202.201.1.20 GET /lab_team.html 500 0.1 123456789A
# 显示内容如下：
{
"host" => "elk-lk",
"client" => "202.201.1.20",
"@timestamp" => 2019-06-23T11:52:43.098Z,
"id" => "123456789A",
"@version" => "1",
"message" => "202.201.1.20 GET /lab_team.html 500 0.1 123456789A",
"bytes" => "500",
"duration" => "0.1",
"method" => "GET",
"request" => "/lab_team.html"
}
# 输入以下内容回车
202.201.1.20 GET /lab_team.html 500 0.1 SYSLOG1
# 显示内容如下：
{
"host" => "elk-lk",
"client" => "202.201.1.20",
"tag" => "SYSLOG1",
"@timestamp" => 2019-06-23T11:53:23.848Z,
"@version" => "1",
"message" => "202.201.1.20 GET /lab_team.html 500 0.1 SYSLOG1",
"bytes" => "500",
"duration" => "0.1",
"method" => "GET",
"request" => "/lab_team.html"
}
Logstash-Output插件
官方地址：https://www.elastic.co/guide/en/logstash/current/output-plugins.html
在elk-es1上启动下head,必须在elasticsearch-head目录下执行
[root@elk-es1 ~]# cd elasticsearch-head/
# 将head放入后台执行
[root@elk-es1 elasticsearch-head]# nohup npm run start &
[1] 11957
[root@elk-es1 elasticsearch-head]# nohup: ignoring input and appending output to ‘nohup.out’
# 按下回车即可放入后台，并查看是否在后台启动成功。
[root@elk-es1 elasticsearch-head]# tail nohup.out
Started connect web server on http://localhost:9100
> elasticsearch-head@0.0.0 start /root/elasticsearch-head
> grunt server
>> Local Npm module "grunt-contrib-jasmine" not found. Is it installed?
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100
# 在elk-lk下安装上传下载工具：
[root@elk-lk ~]# yum install -y lrzsz
# 编写一个测试文档logstash-to-es.conf,内容如下，注意修改hosts的IP：
input {
file {
path => ["/var/log/messages"]
type => "system"
tags => ["syslog","test"]
start_position => "beginning"
}
file {
path => ["/var/log/audit/audit.log"]
type => "system"
tags => ["auth","test"]
start_position => "beginning"
}
}
filter {
}
output {
if [type] == "system" {
if [tags][0] == "syslog" {
elasticsearch {
hosts => ["http://172.16.4.18:9200","http://172.16.4.19:9200","http://172.16.4.20:9200"]
index => "logstash-system-syslog-%{+YYYY.MM.dd}"
}
stdout { codec=> rubydebug }
}
else if [tags][0] == "auth" {
elasticsearch {
hosts => ["http://172.16.4.18:9200","http://172.16.4.19:9200","http://172.16.4.20:9200"]
index => "logstash-system-auth-%{+YYYY.MM.dd}"
}
stdout { codec=> rubydebug }
}
}
}
# 上述文件表明收集了系统日志/var/log/messages(安装软件，debug日志)和 /var/log/audit/audit.log（ssh登录时，或者一些其他的审计日志）
# 由于上次已经配置过了yum源，所以可以直接安装kibana
[root@elk-lk ~]# yum install -y kibana
output中elasticsearch插件：https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
Kibana
ELK Stack 常用案例
# 在elk-lk上更改/etc/kibana/kibana.yml 配置主节点
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://172.16.4.19:9200"]
-
打开浏览器输入172.16.4.21:5601即可访问kibana, 如果出现：“Kibana server is not ready yet”, 请多刷几次页面，或者查看kibana和elasticsearch的版本是否一致。
[root@elk-es2 ~]# rpm -qa elasticsearch elasticsearch-7.1.1-1.x86_64 [root@elk-lk ~]# rpm -qa kibana kibana-7.1.1-1.x86_64 -
可以查看到kibana中管理的我们的索引
[root@elk-lk ~]# mv logstash-to-es.conf /etc/logstash/conf.d/
[root@elk-lk ~]# logstash -r -f /etc/logstash/conf.d/logstash-to-es.conf
# 然后会看到生成很多个记录文件。
# 再到es1的elasticsearch-head目录下，查看生成的索引，发现logstash-system-auth和logstash-system-syslog中有了。
[root@elk-es1 elasticsearch-head]# curl -X GET "172.16.4.18:9200/_cat/indices?v"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open bank XHBHpyl-SxiLlaWaRzta1A 1 1 1000 0 841.2kb 414.3kb
green open logs-test-2019.06.20 1sV4KMz6SdKapiUoOkmgrA 1 1 1 2 13.1kb 9.3kb
green open .kibana_task_manager KPSZy8i0RPmLcb9wctH0aA 1 1 2 0 75kb 45.5kb
green open logstash-system-auth-2019.06.25 AUWPnGytSKyOTaZvZlK67A 1 1 2141 0 795.5kb 388.9kb
green open .kibana_1 9EaFiVF_TyOxnX2-_nk9Bg 1 1 4 0 35.3kb 17.6kb
green open logstash-system-syslog-2019.06.25 YbV3UavDQXuBSFsJcfXDng 1 1 913 0 418.4kb 205.9kb
- 再次刷新页面，发现出现了新的索引
- 创建logstash-system-auth
- 同样的操作创建logstash-system-syslog
- 点击发现日志，选择刚才的“logstash-system-syslog”或者“logstash-system-auth”即可查看到日志
- 可以在“Available fields”中选择需要显示的字段，如hosts,message等
Kibana常用查询表达式
- 注意如果要查询的字段是包含空格需要加双引号，使其成为一个整体字符串。
Find requests that contain the number 200, in any field 200
Find 200 in the status field status:200
Find all status codes between 400-499 status:[400 TO 499]
Find status codes 400-499 with the extension php status:[400 TO 499] AND extension:PHP
Find status codes 400-499 with the extension php or html
status:[400 TO 499] AND (extension:php OR extension:html)
基于Nginx实现Kibana访问认证
增加kibana的安全性，elastic提供了安全组件X-Pack（付费服务）:
网上提供了破解的方法（试试看）：
# 在elk-web上安装NGINX，地址：http://nginx.org/download/
wget http://nginx.org/download/nginx-1.17.0.tar.gz
# 进行源码编译安装
[root@elk-web ~]# tar zxvf nginx-1.17.0.tar.gz
[root@elk-web ~]# cd nginx-1.17.0
# 安装依赖库文件
[root@elk-web nginx-1.17.0]# yum install -y openssl-devel zlib-devel pcre-devel
[root@elk-web nginx-1.17.0]# yum install -y gcc
[root@elk-web nginx-1.17.0]# ./configure --prefix=/usr/local/nginx
[root@elk-web nginx-1.17.0]# make && make install
[root@elk-web nginx-1.17.0]# pkill nginx
# 进入安装目录
[root@elk-web nginx-1.17.0]# cd /usr/local/nginx/sbin
[root@elk-web sbin]# ./nginx
- 在浏览器测试能不能访问。输入elk-web的IP
# 可以查看到访问日志
[root@elk-web sbin]# tail ../logs/access.log -f
210.26.55.133 - - [25/Jun/2019:11:33:17 +0800] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
210.26.55.133 - - [25/Jun/2019:11:33:17 +0800] "GET /favicon.ico HTTP/1.1" 404 555 "http://172.16.4.22/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
# 接下来配置虚拟主机，来代理Kibana
[root@elk-web sbin]# cd ../conf/
# 在nginx.conf中配置代理的地址，在43行location下增加以下字段
[root@elk-web conf]# vim nginx.conf
......
43 location / {
44 proxy_pass http://172.16.4.21:5601;
.......
# 重载下nginx
[root@elk-web conf]# ../sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@elk-web conf]# ../sbin/nginx -s reload
- 此时elk-lk(172.16.4.21)已经被代理到了elk-web(172.16.4.22)上，浏览器输入172.16.4.22就可以访问kibana了。
- 现在我们可以在nginx内加入请求认证
# 再次修改/usr/local/nginx/conf文件，加入auth_basic字段，要求用户输入用户名和密码，加入auth_basic_user_file字段，用户名和密码的存放位置.
vim /usr/local/nginx/conf/nginx.conf
......
proxy_pass http://172.16.4.21:5601;
auth_basic "Please input user and password";
auth_basic_user_file /usr/local/nginx/conf/passwd.db;
......
# 注意在原先加入proxy_pass下加入上述两行文件
[root@elk-web conf]# ../sbin/nginx -s reload
- 刷新浏览器，此时就要求输入用户名和密码了
# 创建用户名和密码，上述已经指定了文件路径为/usr/local/nginx/conf/passwd.db
# 采用openssl生成密文密码
[root@elk-web conf]# openssl passwd -crypt 123456
Fvh88eAdf3TWI
# vim /usr/local/nginx/conf/passwd.db
username:Fvh88eAdf3TWI
- 然后在浏览器中输入“username”和“123456”就可以访问了。
引入Redis
Logstash：过滤和收集，此处分开功能。
- 在服务器比较多的情况下，日志数据较多的时候，可以采用redis（缓存），防止elasticsearch采集日志较慢，前台kibana展示数据有时间差。
- 统一日志的管理
（1）编写logstash-from-redis.conf文件至elk-lk的/etc/logstash/conf.d/目录中，该文件是从redis中去读数据。
[root@elk-lk conf.d]# cat logstash-from-redis.conf
input {
redis {
host => "172.16.4.21"
port => 6379
password => "123456"
db => "0"
data_type => "list"
key => "logstash"
}
}
filter {
}
output {
if [type] == "system" {
if [tags][0] == "syslog" {
elasticsearch {
hosts => ["http://172.16.4.18:9200","http://172.16.4.19:9200","http://172.16.4.20:9200"]
index => "logstash-system-syslog-%{+YYYY.MM.dd}"
}
stdout { codec=> rubydebug }
}
else if [tags][0] == "auth" {
elasticsearch {
hosts => ["http://172.16.4.18:9200","http://172.16.4.19:9200","http://172.16.4.20:9200"]
index => "logstash-system-auth-%{+YYYY.MM.dd}"
}
stdout { codec=> rubydebug }
}
}
}
# Input redis插件中的host只能为一个特定的IP，不是数组类型
# 安装redis的源，默认是没有的
[root@elk-lk ~]# yum install -y epel-release
[root@elk-lk conf.d]# yum install -y redis
# 配置redis
vim /etc/redis.conf
# 在第61行加入监听地址，此处为了方便，默认监听所有地址
......
61 bind 0.0.0.0
.....
# 在第480行，输入redis密码，此处为方便，设置为123456，并保存退出
480 requirepass 123456
# 启动redis
[root@elk-lk conf.d]# systemctl start redis
[root@elk-lk conf.d]# ps -ef | grep redis
redis 21456 1 0 15:07 ? 00:00:00 /usr/bin/redis-server 0.0.0.0:6379
（2）在elk-web(172.16.4.22)上安装logstash，因为需要
# 复制elk-es1的yum源至elk-web
[root@elk-es1 ~]# scp /etc/yum.repos.d/elastic.repo root@172.16.4.22:/etc/yum.repos.d/
# 在elk-web上安装logstash
[root@elk-web conf.d]# yum install -y java-1.8.0-openjdk
[root@elk-web ~]# yum install -y logstash
# 在elk-web中将logstash加入环境变量
PATH=$PATH:/usr/share/logstash/bin
export PATH
# 强制使其生效
source /etc/profile
[root@elk-web ~]# cd /etc/logstash/conf.d/
[root@elk-web conf.d]# vim logstash-to-redis.conf
# 编写logstash写入redis的文件，内容如下，注意修改为自己的集群IP
input {
file {
path => ["/var/log/messages"]
type => "system"
tags => ["syslog","test"]
start_position => "beginning"
}
file {
path => ["/var/log/audit/audit.log"]
type => "system"
tags => ["auth","test"]
start_position => "beginning"
}
}
filter {
}
output {
redis {
host => ["172.16.4.21:6379"]
password => "123456"
db => "0"
data_type => "list"
key => "logstash"
}
}
# 注意：output的redis的host一定是指向安装了redis的IP。可以是多个IP，这样可以在多台里写入数据，提高冗余。elastic output plugin的redis插件用法详见官网：https://www.elastic.co/guide/en/logstash/current/plugins-outputs-redis.html
# 查看redis中是否有数据
[root@elk-lk ~]# redis-cli -a 123456
127.0.0.1:6379> keys *
(empty list or set)
# 启动logstash,看到Successfully started表示启动成功。
[root@elk-web conf.d]# logstash -f /etc/logstash/conf.d/logstash-to-redis.conf
......
[INFO ] 2019-06-25 15:34:31.289 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
# 再次登录redis,查看key,发现多了一个key
[root@elk-lk ~]# redis-cli -a 123456
127.0.0.1:6379> keys *
1) "logstash"
# 查看key的记录数
127.0.0.1:6379> llen logstash
(integer) 2244
# 在elk-es1的elasticsearch-head下启动npm
[root@elk-es1 ~]# cd elasticsearch-head/
# 之前的操作，将npm放入后台启动过，如果没关，无需下述操作。
[root@elk-es1 ~]# npm run start
- 在浏览器输入elk-es的IP，端口号9100,访问es集群
- 在elk-lk开始启动logstash-from-redis.conf，开始从redis读入日志数据，为了方便演示，可以将上述无关的索引进行删除
[root@elk-lk conf.d]# logstash -f /etc/logstash/conf.d/logstash-from-redis.conf
# 会发现有大量日志记录刷新在屏幕上，重开一个elk-lk终端，再次查看redis中的记录数，发现被消费完了。
127.0.0.1:6379> llen logstash
(integer) 0
- 再次在浏览器中访问Kibana，也能查看到日志情况
引入Filebeat
官方文档：https://www.elastic.co/products/beats/filebeat
Filebeat实际上比较省资源，为了解决logstash比较占资源的弊端。
[root@elk-web conf.d]# yum install -y filebeat
[root@elk-web conf.d]# vim /etc/filebeat/filebeat.yml
# 将以下内容生效，其余暂时注释掉。
filebeat.inputs:
- type: log
paths:
- /var/log/messages
tags: ["syslog","test"]
fileds:
type: system
fileds_under_root: true
- type: log
paths:
- /var/log/audit/audit.log
tags: ["auth","test"]
fields:
type: system
fields_under_root: true
# 输出到redis中
output.redis:
hosts: ["172.16.4.21"]
password: "123456"
key: "filebeat"
db: 0
datatype: list
# 启动filebeat
[root@elk-web conf.d]# systemctl start filebeat
# 查看redis中是否收集到信息
[root@elk-lk ~]# redis-cli -a 123456
127.0.0.1:6379> keys *
1) "filebeat"
127.0.0.1:6379> llen filebeat
(integer) 3277
- 可以查看filebeat支持的输入类型：https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html，此处输出到redis中：https://www.elastic.co/guide/en/beats/filebeat/current/redis-output.html
- 测试Kibana是否能展示出filebeat收集的数据，同样，打开浏览器输入es集群的IP。
# 更改输入源为filebeat,其余数据保持不变
[root@elk-lk conf.d]# cat /etc/logstash/conf.d/logstash-from-redis.conf
input {
redis {
host => "172.16.4.21"
port => 6379
password => "123456"
db => "0"
data_type => "list"
key => "filebeat" # 此处修改
}
}
filter {
}
output {
if [type] == "system" {
if [tags][0] == "syslog" {
elasticsearch {
hosts => ["http://172.16.4.18:9200","http://172.16.4.19:9200","http://172.16.4.20:9200"]
index => "logstash-system-syslog-%{+YYYY.MM.dd}"
}
stdout { codec=> rubydebug }
}
else if [tags][0] == "auth" {
elasticsearch {
hosts => ["http://172.16.4.18:9200","http://172.16.4.19:9200","http://172.16.4.20:9200"]
index => "logstash-system-auth-%{+YYYY.MM.dd}"
}
stdout { codec=> rubydebug }
}
}
}
# 启动logstash，从redis读入数据
[root@elk-lk conf.d]# logstash -f /etc/logstash/conf.d/logstash-from-redis.conf
# 重开一个elk-lk, 进入redis，发现日志记录为0，已被logstash消费完
127.0.0.1:6379> llen filebeat
(integer) 0
127.0.0.1:6379> get filebeat
(nil)
# 在elk-es1中ssh到elk-web，然后会发现elk-lk会收集到登录日志信息，如下：
......
{
"log" => {
"offset" => 466114,
"file" => {
"path" => "/var/log/audit/audit.log"
}
},
"tags" => [
[0] "auth",
[1] "test"
],
"ecs" => {
"version" => "1.0.0"
},
"@version" => "1",
"type" => "system",
"message" => "type=CRYPTO_SESSION msg=audit(1561470434.558:1651): pid=34870 uid=0 auid=4294967295 ses=4294967295 msg='op=start direction=from-client cipher=chacha20-poly1305@openssh.com ksize=512 mac=<implicit> pfs=curve25519-sha256 spid=34871 suid=74 rport=33384 laddr=172.16.4.22 lport=22 exe=\"/usr/sbin/sshd\" hostname=? addr=172.16.4.18 terminal=? res=success'",
"agent" => {
"version" => "7.1.1",
"hostname" => "elk-web",
"type" => "filebeat",
"ephemeral_id" => "ed5247b8-5bc1-48ea-a57c-4d18d474b023",
"id" => "bb3b79b8-a0f1-48e6-ba52-1d6c70a77833"
},
"input" => {
"type" => "log"
},
"host" => {
"name" => "elk-web"
},
"@timestamp" => 2019-06-25T13:47:17.441Z
}
......
# 说明收集日志成功
- 此时刷新浏览器，访问es集群，发现出现了logstash-system-auth的信息，同样地，打开kibana也能查看到刷新的日志信息。
生产应用案例
收集Nginx访问日志
采用filebeat写入日志数据到Redis
# 确保以下服务正常运行
[root@elk-lk ~]# ps -ef | grep redis
redis 21456 1 0 15:07 ? 00:01:03 /usr/bin/redis-server 0.0.0.0:6379
# 修改/usr/local/nginx/conf的nginx.conf，取消如下三行
......
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
......
# 将日志格式修改为json格式，内容如下并在上述的nginx.conf文件中的log_format main内容后：
log_format json '{ "@timestamp": "$time_iso8601", '
'"remote_addr": "$remote_addr", '
'"remote_user": "$remote_user", '
'"body_bytes_sent": "$body_bytes_sent", '
'"request_time": "$request_time", '
'"status": "$status", '
'"request_uri": "$request_uri", '
'"request_method": "$request_method", '
'"http_referrer": "$http_referer", '
'"http_x_forwarded_for": "$http_x_forwarded_for", '
'"http_user_agent": "$http_user_agent"}';
# 将以下行生效，注意，之前是调用main.
access_log logs/access.log json;
# 将nginx命令加入环境变量/etc/profile中
PATH=$PATH:/usr/share/logstash/bin:/usr/share/filebeat/bin:/usr/local/nginx/sbin
export PATH
# 使其生效
[root@elk-web conf]# source /etc/profile
# 重载nginx配置
[root@elk-web conf]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@elk-web conf]# nginx -s reload
- 更改elk-web上的filebeat配置文件内容为如下：
[root@elk-web ~]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
paths:
- /usr/local/nginx/logs/access.log
fields:
app: www
type: nginx-access
fields_under_root: true
- type: log
paths:
- /usr/local/nginx/logs/error.log
fields:
app: www
type: nginx-error
fields_under_root: true
output.redis:
hosts: ["172.16.4.21"]
password: "123456"
key: "filebeat"
db: 0
datatype: list
- 重启filebeat，并查看是否是收集上述两个目录的日志。
[root@elk-web ~]# systemctl restart filebeat
[root@elk-web ~]# tail /var/log/messages
Jun 26 15:19:17 elk-web filebeat: 2019-06-26T15:19:17.406+0800#011INFO#011log/input.go:148#011Configured paths: [/usr/local/nginx/logs/access.log]
Jun 26 15:19:17 elk-web filebeat: 2019-06-26T15:19:17.406+0800#011INFO#011input/input.go:114#011Starting input of type: log; ID: 13046343894051577886
Jun 26 15:19:17 elk-web filebeat: 2019-06-26T15:19:17.406+0800#011INFO#011log/input.go:148#011Configured paths: [/usr/local/nginx/logs/error.log]
Jun 26 15:19:17 elk-web filebeat: 2019-06-26T15:19:17.406+0800#011INFO#011input/input.go:114#011Starting input of type: log; ID: 3665717057268333493
Jun 26 15:19:17 elk-web filebeat: 2019-06-26T15:19:17.406+0800#011INFO#011crawler/crawler.go:106#011Loading and starting Inputs completed. Enabled inputs: 2
Jun 26 15:19:17 elk-web filebeat: 2019-06-26T15:19:17.407+0800#011INFO#011log/harvester.go:254#011Harvester started for file: /usr/local/nginx/logs/access.log
Jun 26 15:19:17 elk-web filebeat: 2019-06-26T15:19:17.407+0800#011INFO#011log/harvester.go:254#011Harvester started for file: /usr/local/nginx/logs/error.log
Jun 26 15:19:18 elk-web filebeat: 2019-06-26T15:19:18.407+0800#011INFO#011pipeline/output.go:95#011Connecting to redis(tcp://172.16.4.21:6379)
Jun 26 15:19:18 elk-web filebeat: 2019-06-26T15:19:18.411+0800#011INFO#011pipeline/output.go:105#011Connection to redis(tcp://172.16.4.21:6379) established
# 查看redis是否写入数据
[root@elk-lk ~]# redis-cli -a 123456
127.0.0.1:6379> keys *
1) "filebeat"
127.0.0.1:6379> llen filebeat
(integer) 2978
# 刷新下浏览器，发现日志数确实有所增加
127.0.0.1:6379> llen filebeat
(integer) 3073
采用Logstash读取日志到elasticsearch中
（1）更改编写日志读取配置文件logstash-to-es-nginxjson.conf
[root@elk-lk ~]# cat /etc/logstash/conf.d/logstash-to-es-nginxjson.conf
input {
redis {
host => "172.16.4.21"
port => 6379
password => "123456"
db => "0"
data_type => "list"
key => "filebeat"
}
}
filter {
if [app] == "www" {
if [type] == "nginx-access" {
json {
source => "message"
remove_field => ["message"]
}
geoip {
source => "remote_addr"
target => "geoip"
database => "/opt/GeoLite2-City.mmdb"
add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"]
add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"]
}
mutate {
convert => ["[geoip][coordinates]", "float"]
}
}
}
}
output {
elasticsearch {
hosts => ["http://172.16.4.18:9200","http://172.16.4.19:9200","http://172.16.4.20:9200"]
index => "logstash-%{type}-%{+YYYY.MM.dd}"
}
stdout{codec => rubydebug }
}
- mutate是filter里的插件
启动logstash，消费redis中的日志记录，在屏幕中会看到，很多条记录刷新
[root@elk-lk ~]# logstash -f /etc/logstash/conf.d/logstash-to-es-nginxjson.conf
{
"log" => {
"offset" => 49770,
"file" => {
"path" => "/usr/local/nginx/logs/access.log"
}
},
"status" => "200",
"agent" => {
"version" => "7.1.1",
"type" => "filebeat",
"ephemeral_id" => "d0ac533d-e71a-4a39-b3f6-b3a6c906d9ec",
"hostname" => "elk-web",
"id" => "bb3b79b8-a0f1-48e6-ba52-1d6c70a77833"
},
"type" => "nginx-access",
"app" => "www",
"body_bytes_sent" => "898",
"http_x_forwarded_for" => "-",
"request_time" => "0.033",
"ecs" => {
"version" => "1.0.0"
},
"request_method" => "POST",
"host" => {
"name" => "elk-web"
},
"@version" => "1",
"input" => {
"type" => "log"
},
"remote_user" => "username",
"geoip" => {
"coordinates" => [
[0] 103.8399,
[1] 36.057
],
"country_name" => "China",
"continent_code" => "AS",
"country_code3" => "CN",
"longitude" => 103.8399,
"region_name" => "Gansu",
"location" => {
"lon" => 103.8399,
"lat" => 36.057
},
"city_name" => "Lanzhou",
"ip" => "210.26.55.133",
"region_code" => "GS",
"latitude" => 36.057,
"country_code2" => "CN",
"timezone" => "Asia/Shanghai"
},
"http_user_agent" => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36",
"request_uri" => "/api/index_management/indices/reload",
"http_referrer" => "http://172.16.4.22/app/kibana",
"remote_addr" => "210.26.55.133",
"@timestamp" => 2019-06-26T08:33:56.000Z
}
.....
# 此时在另一个elk-lk终端再次连接到redis查看记录，发现已被消耗完
127.0.0.1:6379> llen filebeat
(integer) 0
- 打开kibana的“Index management”，会发现多出了两个索引logstash-nginx-access-XXX和logstash-nginx-error-XXX.
- 我们可以点击下方的Index Patterns创建后，在Discovery中观察到日志。
[root@elk-web ~]# tail /usr/local/nginx/logs/access.log
......
{ "@timestamp": "2019-06-26T16:47:32+08:00", "remote_addr": "210.26.55.133", "remote_user": "username", "body_bytes_sent": "208", "request_time": "0.017", "status": "200", "request_uri": "/elasticsearch/_msearch?rest_total_hits_as_int=true&ignore_throttled=true", "request_method": "POST", "http_referrer": "http://172.16.4.22/app/kibana", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"}
# 仔细对比kibana中和文件日志中的最后 一条记录，发现两者的时间一致。
（2）同样地，打开elasticsearch，也能发现日志已经收集到es中了
- 可以按IP查询访问日志
收集Java堆栈日志
安装tomcat
此处采用tomcat收集日志，官方软件下载地址：http://tomcat.apache.org/
[root@elk-web ~]# wget https://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.42/bin/apache-tomcat-8.5.42.tar.gz
[root@elk-web ~]# tar zxvf apache-tomcat-8.5.42.tar.gz
[root@elk-web ~]# mv apache-tomcat-8.5.42/ /usr/local/tomcat && cd /usr/local/tomcat/; ls
bin BUILDING.txt conf CONTRIBUTING.md lib LICENSE logs NOTICE README.md RELEASE-NOTES RUNNING.txt temp webapps work
[root@elk-web tomcat]# cd bin/
[root@elk-web bin]# ./startup.sh
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@elk-web bin]# tail ../logs/catalina.out -f
......
26-Jun-2019 21:11:13.397 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 584 ms
- 浏览器输入服务器端IP和端口号（8080）即可访问
使用Filebeat收集日志
（1）修改filebeat.yml，监听tomcat
# 在/etc/filebeat/filebeat.yml的filebeat.inputs添加日志类型如下，其余保持不变
- type: log
paths:
- /usr/local/tomcat/logs/catalina.out
# tags: ["tomcat"]
fields:
app: www
type: tomcat-catalina
fields_under_root: true
multiline:
pattern: '^\['
negate: true
match: after
（2）重启filebeat，使配置生效
[root@elk-web ROOT]# systemctl restart filebeat
[root@elk-web ROOT]# tail /var/log/messages
......
Jun 26 21:28:19 elk-web filebeat: 2019-06-26T21:28:19.400+0800#011INFO#011log/harvester.go:254#011Harvester started for file: /usr/local/tomcat/logs/catalina.out
Jun 26 21:28:25 elk-web filebeat: 2019-06-26T21:28:25.402+0800#011INFO#011pipeline/output.go:95#011Connecting to redis(tcp://172.16.4.21:6379)
Jun 26 21:28:25 elk-web filebeat: 2019-06-26T21:28:25.406+0800#011INFO#011pipeline/output.go:105#011Connection to redis(tcp://172.16.4.21:6379) established
（3）打开kibana发现多出了logstash-tomcat-catalina-XXX的索引日志。
定制日志格式收集
（1）以nginx为例，之前采用json格式化日志输出形式，此处先改回来
# 修改/usr/local/nginx/conf/nginx.conf的内容，将
access_log logs/access.log json;
# 改为如下：
access_log logs/access.log main;
# 重载配置
[root@elk-web bin]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@elk-web bin]# nginx -s reload
（2）采用grok插件进行模式匹配
input {
redis {
host => "172.16.4.21"
port => 6379
password => "123456"
db => "0"
data_type => "list"
key => "filebeat"
}
}
filter {
if [app] == "www" {
if [type] == "nginx-access" {
grok {
match => {
"message" => "%{IPV4:remote_addr} - (%{USERNAME:remote_user}|-) \[%{HTTPDATE:time_local}\] \"%{WORD:request_method} %{URIPATHPARAM:request_uri} HTTP/%{NUMBER:http_protocol}\" %{NUMBER:http_status} %{NUMBER:body_bytes_sent} \"%{GREEDYDATA:http_referer}\" \"%{GREEDYDATA:http_user_agent}\" \"(%{IPV4:http_x_forwarded_for}|-)\""
}
overwrite => ["message"]
}
geoip {
source => "remote_addr"
target => "geoip"
database => "/opt/GeoLite2-City.mmdb"
add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"]
add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"]
}
date {
locale => "en"
match => ["time_local", "dd/MMM/yyyy:HH:mm:ss Z"]
}
mutate {
convert => ["[geoip][coordinates]", "float"]
}
}
}
}
output {
elasticsearch {
hosts => ["http://172.16.4.18:9200","http://172.16.4.19:9200","http://172.16.4.20:9200"]
index => "logstash-%{type}-%{+YYYY.MM.dd}"
}
stdout{codec => rubydebug }
}
# 先取消代理，注释一下3行内容
[root@elk-web ~]# vim /usr/local/nginx/conf/nginx.conf
#proxy_pass http://172.16.4.21:5601;
#auth_basic "Please input user and password";
#auth_basic_user_file /usr/local/nginx/conf/passwd.db;
# 重载nginx
[root@elk-web ~]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@elk-web ~]# nginx -s reload
- 默认访问nginx首页
（3）启动logstash
[root@elk-lk ~]# logstash -f /etc/logstash/conf.d/logstash-to-es-custom.conf
Kibana仪表盘与可视化
手动添加日志记录，方便演示
[root@elk-web logs]# echo '210.26.55.132 - username [26/Jun/2019:22:34:34 +0800] "GET /app/kibana HTTP/1.1" 404 555 "http://172.16.4.22/app/kibana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-"' >> access.log
[root@elk-web logs]# echo '210.26.55.131 - username [26/Jun/2019:22:34:34 +0800] "GET /app/kibana HTTP/1.1" 404 555 "http://172.16.4.22/app/kibana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-"' >> access.log
[root@elk-web logs]# echo '210.26.55.130 - username [26/Jun/2019:22:34:34 +0800] "GET /app/kibana HTTP/1.1" 404 555 "http://172.16.4.22/app/kibana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-"' >> access.log
[root@elk-web logs]# echo '210.16.55.110 - username [26/Jun/2019:22:34:34 +0800] "GET /app/kibana HTTP/1.1" 404 555 "http://172.16.4.22/app/kibana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-"' >> access.log
[root@elk-web logs]# echo '210.16.54.119 - username [26/Jun/2019:22:34:34 +0800] "GET /app/kibana HTTP/1.1" 404 555 "http://172.16.4.22/app/kibana" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-"' >> access.log
- 刷新浏览器，发现kibana中确实多了上述几个IP的日志记录
PV/UV
（1）可以通过Kibana左侧的“Visualize”来可视化日志记录数，首先选择需要统计的日志，此处以“logstash-nginx-access-*”为例，选择X轴以时间轴为单位，纵坐标以日志数（Count）为值.
（2）对UV进行统计用户数
用户地理位置分布
- 按照之前的操作，创建Coordinate Map.
- 按照如下字段选择横纵左边所要显示的的内容。
- 点击上述的加号，缩小地理范围，可以查看到IP来自的城市
URL/HTTP Status/IP TOP10
（1）统计URI数量，建立新的可视化视图，选择“Data Table”
（2）统计HTTP状态码Top10
（3）统计源IP访问TOP10
- 至此，我们完成了6张图的绘制，现在我们采用dashboard仪表盘来添加刚才的图表，从而可视化数据
备注：本内容来源于阿良老师的博客和课程，如有侵权，请联系删除，无意冒犯。只是记录学习。
京公网安备 11010502036488号