filebeat收集单日志到本地文件
1）配置
|
1 2 3 4 5 6 7 8 9 10 11 |
[root@web01 ~]# vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log output.file: path: "/tmp/" filename: "nginx.log" |
2) 启动
|
1 2 |
[root@web01 ~]# systemctl start filebeat.service |
filebeat收集单日志到ES
1) 配置
|
1 2 3 4 5 6 7 8 9 10 |
[root@web01 ~]# vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log output.elasticsearch: hosts: ["http://10.0.0.71:9200"] |
2) 启动
|
1 2 |
[root@web01 ~]# systemctl restart filebeat.service |
filebeat收集单日志json格式到ES
1) 配置nginx的json格式日志
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
[root@web01 ~]# cat /etc/nginx/nginx.conf http { ... ... log_format json '{ "time_local": "$time_local", ' '"remote_addr": "$remote_addr", ' '"referer": "$http_referer", ' '"request": "$request", ' '"status": $status, ' '"bytes": $body_bytes_sent, ' '"agent": "$http_user_agent", ' '"x_forwarded": "$http_x_forwarded_for", ' '"up_addr": "$upstream_addr",' '"up_host": "$upstream_http_host",' '"upstream_time": "$upstream_response_time",' '"request_time": "$request_time" }'; access_log /var/log/nginx/access.log json; ... ... |
2) 配置收集日志
|
1 2 3 4 5 6 7 8 9 10 11 12 |
[root@web01 ~]# vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log json.keys_under_root: true json.overwrite_keys: true output.elasticsearch: hosts: ["http://10.0.0.71:9200"] |
3) 启动
|
1 2 |
[root@web01 ~]# systemctl restart nginx |
filebeat收集单日志到redis
1) 配置
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
[root@web01 ~]# vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log json.keys_under_root: true json.overwrite_keys: true output.redis: hosts: ["10.0.0.81:6379"] key: "nginx_log" db: 0 |
2) 启动
3) redis查看数据
|
1 2 3 4 5 |
127.0.0.1:6379> keys * 1) "nginx_log" 127.0.0.1:6379> LLEN nginx_log (integer) 33 |
filebeat收集单日志到logstash
1) 配置
|
1 2 3 4 5 6 7 8 9 10 11 |
[root@web01 ~]# vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log json.keys_under_root: true json.overwrite_keys: true output.logstash: hosts: ["10.0.0.81:7890"] |
2) 启动
|
1 2 |
[root@web01 ~]# systemctl restart filebeat.service |
3) 配置logstash
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
[root@redis01 ~]# vim /etc/logstash/conf.d/filebeat_logstash_es.conf input { beats { port => "7890" } } output { elasticsearch { hosts => ["10.0.0.71:9200"] index => "filebeat_logstash_%{+YYYY-MM-dd}" } } [root@redis01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/filebeat_logstash_es.conf & |
filebeat收集多日志到ES
方法一：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
[root@web01 ~]# vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log - /var/log/nginx/error.log json.keys_under_root: true json.overwrite_keys: true output.elasticsearch: hosts: ["http://10.0.0.71:9200"] index: "nginx_json_%{+yyyy-MM-dd}" setup.template.name: "filebeat-*" setup.template.pattern: "filebeat-*" |
方法二：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
[root@web01 ~]# cat /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log json.keys_under_root: true json.overwrite_keys: true - type: log enabled: true paths: - /var/log/nginx/error.log output.elasticsearch: hosts: ["http://10.0.0.71:9200"] index: "nginx_json_%{+yyyy-MM-dd}" setup.template.name: "filebeat-*" setup.template.pattern: "filebeat-*" |
filebeat收集多日志到多个ES索引
方法一
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
[root@web01 ~]# cat !$ cat /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log json.keys_under_root: true json.overwrite_keys: true - type: log enabled: true paths: - /var/log/nginx/error.log output.elasticsearch: hosts: ["http://10.0.0.71:9200"] indices: - index: "nginx_access_%{+yyyy-MM-dd}" when.contains: source: "/var/log/nginx/access.log" - index: "nginx_error_%{+yyyy-MM-dd}" when.contains: source: "/var/log/nginx/error.log" setup.template.name: "filebeat-*" setup.template.pattern: "filebeat-*" |
方法二
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
[root@web01 ~]# cat /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log json.keys_under_root: true json.overwrite_keys: true tags: ["access"] - type: log enabled: true paths: - /var/log/nginx/error.log tags: ["error"] output.elasticsearch: hosts: ["http://10.0.0.71:9200"] indices: - index: "nginx_access_%{+yyyy-MM-dd}" when.contains: tags: "access" - index: "nginx_error_%{+yyyy-MM-dd}" when.contains: tags: "error" setup.template.name: "filebeat-*" setup.template.pattern: "filebeat-*" |
filebeat收集java的报错日志
1) 配置收集tomcat日志
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
[root@web01 ~]# vim /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /usr/local/tomcat/logs/tomcat_access_json.*.log json.keys_under_root: true json.overwrite_keys: true output.elasticsearch: hosts: ["http://10.0.0.71:9200"] index: "tomcat_access_%{+yyyy-MM-dd}" setup.template.name: "filebeat-*" setup.template.pattern: "filebeat-*" |
配置收集java报错日志
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
[root@web01 ~]# cat /etc/filebeat/filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /usr/local/tomcat/logs/localhost_access_log.*.txt multiline.pattern: '^\[' multiline.negate: true multiline.match: after json.keys_under_root: true json.overwrite_keys: true json.message_key: log output.elasticsearch: hosts: ["http://10.0.0.71:9200"] index: "tomcat_access_%{+yyyy-MM-dd}" setup.template.name: "filebeat-*" setup.template.pattern: "filebeat-*" |