泉州网站设计找哪家,赵朴初网站建设,上海市工程建设,自助服务平台一、部署ELK
上文把采集端filebeat如何使用介绍完#xff0c;现在随着数据的链路#xff0c;继续~~
同样#xff0c;使用docker-compose部署#xff1a;
version: 3
services:elasticsearch:container_name: elasticsearchimage: elastic/elasticsearch:7.9…一、部署ELK
上文把采集端filebeat如何使用介绍完现在随着数据的链路继续~~
同样使用docker-compose部署
version: 3
services:elasticsearch:container_name: elasticsearchimage: elastic/elasticsearch:7.9.3restart: alwaysuser: rootports:- 9200:9200- 9300:9300volumes:- ./elasticsearch/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml- ./elasticsearch/data:/usr/share/elasticsearch/data- ./elasticsearch/logs:/usr/share/elasticsearch/logsenvironment:- discovery.typesingle-node- TAKE_FILE_OWNERSHIPtrue- ES_JAVA_OPTS-Xms1500m -Xmx1500m- TZAsia/Shanghaikibana:container_name: kibanaimage: elastic/kibana:7.9.3restart: alwaysports:- 5601:5601volumes:- ./kibana/conf/kibana.yml:/usr/share/kibana/config/kibana.ymlenvironment:- elasticsearch.hostselasticsearch:9200- TZAsia/Shanghaidepends_on:- elasticsearchlogstash:image: elastic/logstash:7.9.3restart: alwayscontainer_name: logstashvolumes:- ./logstash/conf/logstash.conf:/usr/share/logstash/pipeline/logstash.conf- ./logstash/template.json:/etc/logstash/template.jsonports:- 5044:5044- 9600:9600environment:- LS_JAVA_OPTS-Xms1024m -Xmx1024m- elasticsearch.hostselasticsearch:9200- TZAsia/Shanghaidepends_on:- elasticsearch可以看到logstash和kibana都依赖于ElasticSearch填写es的地址使用容器名“elasticsearch:9200”省去分配内网IP的过程。
es存储需要持久化 volumes:- ./elasticsearch/data:/usr/share/elasticsearch/data三个组件的配置文件都开放便于在宿主机上修改。
├── elasticsearch
│ ├── conf
│ │ └── elasticsearch.yml
│ ├── data
│ └── logs
│ ├── gc.log
│ ├── gc.log.00
│ ├── gc.log.01
│ ├── gc.log.02
│ ├── gc.log.03
│ ├── gc.log.04
│ ├── gc.log.05
│ └── gc.log.06
├── kibana
│ └── conf
│ └── kibana.yml
└── logstash├── conf│ └── logstash.conf└── template.json由于es和kibana在后文将另外讲述所以本文只进一步介绍logstash的使用。
二、logstash的配置
1、template.json
定义索引的mapping信息
{template: jvm-*,settings: {number_of_shards: 1,number_of_replicas: 0},mappings: {properties: {logclass: {type: text},appname: {type: keyword}, traceid: {type: keyword},spanid: {type: keyword},export: {type: boolean},logpid: {type: keyword},logdate: {type: date,format: yyyy-MM-dd HH:mm:ss.SSS},loglevel: {type: keyword},threadname: {type: keyword},logmsg: {type: text}}}
}2、logstash.conf
input {beats {port 5044}
}filter {grok {pattern_definitions {QUALIFIED [a-zA-Z0-9$_.]}match {message %{TIMESTAMP_ISO8601:logdate}%{SPACE}%{WORD:loglevel}%{SPACE}\[%{DATA:appname},%{DATA:traceid},%{DATA:spanid},%{DATA:export}\]%{SPACE}%{NUMBER:logpid} --- \[%{USERNAME:threadname}\] %{DATA:logclass} - %{GREEDYDATA:logmsg}}}}output {elasticsearch {hosts [elasticsearch:9200]#索引的正则表达式比如jvm-20231227index jvm-%{yyyy.MM.dd}template /etc/logstash/template.jsontemplate_name logstash}}三、注意事项
1、logstash.conf中的注释#开头不能加空格
下面是错误的注释
# 索引的正则表达式比如jvm-20231227正确的注释是
#索引的正则表达式比如jvm-202312272、grok语法
已有在线的grok表达式这里推荐一款kibana的开发工具
具体的语法见其github官网
https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/ecs-v1/grok-patterns
