摘要：
首先为了更适合ELK采集输出，把Nginx日志格式改成json格式输出：log_format json '{ &quo... 首先为了更适合ELK采集输出，把Nginx日志格式改成json格式输出：
log_format json '{ "@timestamp": "$time_iso8601","remote_addr": "$remote_addr","method":"$request_method",'
'"scheme":"$scheme","host":"$host","server_port","$server_port","request":"$request_uri","status":"$status", "body_size":"$body_bytes_sent",'
'"referer":"$http_referer","ua":"$http_user_agent","xff":"$http_x_forwarded_for","up_addr":"$upstream_addr",'
'"up_resp_time":"$upstream_response_time","up_code":"$upstream_status","rt":"$request_time"}';关于Nginx的两种时间格式介绍：
$time_iso8601
"2020-04-15T01:32:20+08:00"
这种格式下对应的Logstash配置处理：
注意这里的时间格式可以选择$time_iso8601，刚好符合Logstash中date模块处理格式。输出的格式为："2020-04-15T01:32:20+08:00"，Logstash配置里需要对此格式进行匹配：
date {
match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ssZZ"]
target => "@timestamp"
}$time_local
"08/Apr/2020:03:52:37 +0800"
这种格式下对应的Logstash配置处理：
date {
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
target => "@timestamp"
}经过这样处理后kibana中显示的时间才能与日志的时间保持一致。对比两种格式使用iso8601更友好些.
/var/log/nginx/access.log：
{ "@timestamp": "2020-04-15T21:44:00+08:00","remote_addr": "192.168.10.27","method":"GET","scheme":"http","host":"dev01","server_port":80,"request":"/img/header-background.png","status":200, "body_size":82896,"referer":"-","ua":"curl/7.29.0","xff":"-","up_addr":"-","up_resp_time":"-","up_code":"-","rt":0.000}
{ "@timestamp": "2020-04-15T21:44:01+08:00","remote_addr": "192.168.10.27","method":"GET","scheme":"http","host":"dev01","server_port":80,"request":"/index.html","status":200, "body_size":4833,"referer":"-","ua":"curl/7.29.0","xff":"-","up_addr":"-","up_resp_time":"-","up_code":"-","rt":0.000}
{ "@timestamp": "2020-04-15T21:44:02+08:00","remote_addr": "192.168.10.27","method":"GET","scheme":"http","host":"dev01","server_port":80,"request":"/img/header-background.png","status":200, "body_size":82896,"referer":"-","ua":"curl/7.29.0","xff":"-","up_addr":"-","up_resp_time":"-","up_code":"-","rt":0.000}
{ "@timestamp": "2020-04-15T21:44:03+08:00","remote_addr": "192.168.10.27","method":"GET","scheme":"http","host":"dev01","server_port":80,"request":"/index.html","status":200, "body_size":4833,"referer":"-","ua":"curl/7.29.0","xff":"-","up_addr":"-","up_resp_time":"-","up_code":"-","rt":0.000}安装filebeat后增加配置：inputs.d/nginx.yml:
- type: log enabled: true paths: - /var/log/nginx/access.log json.keys_under_root: true json.add_error_key: true json.overwrite_keys: true tags: ["nginx_access"] fields: svcname: nginx logtype: nginx_access - type: log enabled: true paths: - /var/log/nginx/error.log tags: ["nginx_error"] fields: svcname: nginx logtype: nginx_error
Logstash配置：这里贴的是全部的配置，重点关注这里跟Nginx相关的部分 [fields][logtype] == "nginx_access"：
主配置：日志流处理，匹配时间格式
[root@logstash01 ~]# cat /etc/logstash/conf.d/main.conf
input {
beats {
port => 5000
}
}
filter {
ruby {
code => "event.set('myindex_date', event.get('@timestamp').time.localtime + 8*60*60)"
}
if [fields][logtype] == "nginx_access" {
date {
match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ssZZ"]
target => "@timestamp"
}
}
if [fields][svcname] == "syslog" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:syslogtime}" }
}
date {
match => ["syslogtime", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
}
}[root@logstash01 ~]# cat /etc/logstash/conf.d/nginx.conf
output {
if [fields][logtype] == "nginx_access" {
elasticsearch {
hosts => ["es01:9200","es02:9200","es03:9200"]
index => "nginx-access-%{myindex_date}"
}
}
if [fields][logtype] == "nginx_error" {
elasticsearch {
hosts => ["es01:9200","es02:9200","es03:9200"]
index => "nginx-error-%{myindex_date}"
}
}
}kibana采集日志的展示：
通过截图显示可以看到Nginx中定义的key在这里也作为独立的fields。这里的@timestamp时间通过Logstash处理后跟日志里的时间保持一致了。但是截图里的Time和@timestamp的时间格式并不是Nginx日志中的格式，是因为kibana显示的原因，kibana还可以重新定义一次用于展示的时间格式。Nginx中的@timestamp实际是什么呢，我们来看下：
这个时间则是采集到的Nginx日志里的真实数据，不过为什么这里不是Nginx日志的原始数据，目前也还没弄明白。初步查了下，kibana的日期格式与Logstash有点区别，特别是时区。Logstash中date模块里[+08:00]用"ZZ"表示，而kibana则是用一个大写字母Z表示. 不知道是否跟这个有关系
还没有评论，来说两句吧...