官网:https://www.elastic.co


Logash+Elasticsearch+Kibana 日志系统安装部署

环境

类型

操作系统

IP

软件包

服务端

Centos 6.5 X64

192.168.0.100

JDK1.8

Elasticsearch-2.4.1

Kibana-4.6.1

客户端

Centos 6.5 X64

192.168.0.101

JDK1.8

Logash-2.4.0

 

注：Logash、Elasticsearch、Kibana运行需要JDK。


JDK安装

#yum install java-1.8.0-openjdk


Logash配置

#cat /opt/logstash-2.4.0/conf/nginx_log.conf

内容如下:


input {
        file {
                type => "nginx_log"
                path => "/opt/nginx/logs/access.log"
        }
}
output {
        stdout { codec => rubydebug }
        elasticsearch {
                host => "192.168.0.100：9200"
        }
}



创建nginx日志格式配置文件

注: path => "/opt/nginx/logs/access.log"  #Nginx日志文件


match => { "message" => "%{NGINXACCESS}" }  #Nginx日志格式，变量%{NGINXACCESS}会自动在/opt/logstash-1.4.2/patterns 目录下查找。
host => "192.168.0.100"  # elasticsearch 服务端IP
port => "9200"           # elasticsearch 服务端端口


#cat /opt/logstash-1.4.2/patterns/nginx

NGUSERNAME [a-zA-Z\.\@\-\+_%]+

NGUSER %{NGUSERNAME}

NGINXACCESS %{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}


Logash启动

#/opt/logstash-2.4.0/bin/logstash -f /opt/logstash-2.4.0/conf/nginx_log.conf > /dev/null 2>&1 &


Logash 添加IP查询


#wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
#gunzip GeoLiteCity.dat.gz
#mv GeoLiteCity.dat /opt/logstash-2.4.0/conf


在/opt/logstash-2.4.0/conf/nginx_log.conf 配置文件filter{} 加入以下内容:


geoip {
      source => "remote_addr"
      target => "geoip"
      database => "/opt/logstash-2.4.0/conf/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }
  }
}