GNet Solutions » FreeBSD Administration

Support #414

Updated by Daniel Curtis about 9 years ago

h1. Prepare the system 

 * Update the system 
 <pre> 
 pkg update && pkg upgrade 
 portsnap fetch extract 
 </pre> 

 * Install portmaster: 
 <pre> 
 cd /usr/ports/ports-mgmt/portmaster 
 make install clean 
 pkg2ng 
 </pre> 

 --- 

 h1. Installing ElasticSearch 

 * Install elasticsearch: 
 <pre> 
 portmaster textproc/elasticsearch 
 </pre> 

 * Start and enable ElasticSearch at boot 
 <pre> 
 echo 'elasticsearch_enable="YES"' >> /etc/rc.conf 
 service elasticsearch start 
 </pre> 

 h2. Securing Elasticsearch 

 * Up to version 1.2, Elasticsearch's dynamic scripting capability was enabled by default. Since this tutorial sets up the Kibana dashboard to be accessed from the public Internet, let's disable dynamic scripting by appending the following line at the end of the ElasticSearch configuration file: 
 <pre> 
 echo "script.disable_dynamic: true" >> /usr/local/etc/elasticsearch/elasticsearch.yml vi /etc/elasticsearch/elasticsearch.yml 
 </pre> 
 > script.disable_dynamic: true 

 * Restart Elasticsearch: 
 <pre> 
 service elasticsearch restart 
 </pre> 

 Logstash includes an embedded ElasticSearch instance you can use for standalone installations (see my previous post for an introductory view on Logstash operation modes). The required configuration to bootstrap the embedded ElasticSearch instance and to have Logstash use it as its outputs is described in the following sections. 

 Although simpler from the standpoint of the configuration, Logstash installations using separate ElasticSearch instances are out of the scope of this post. 

 --- 

 h1. Install Logstash 

 * Install logstash: 
 <pre> 
 portmaster sysutils/logstash 
 </pre> 

 * Create a basic configuration: 
 <pre> 
 vi /usr/local/etc/logstash/logstash.conf /usr/local/etc/logstash14/logstash14.conf 
 </pre> 
 #* Then modify add the following: 
 <pre> 
 input { 
 file { 
 type => "syslog" 
 # # Wildcards work, here :) 
 # path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ] 
 path => "/var/log/messages" 
 start_position => "beginning" 
 } 
 } 
 filter { 
 if [type] == "syslog" { 
 grok { 
 match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} (%{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}|%{GREEDYDATA:syslog_message})" } 
 add_field => [ "received_at", "%{@timestamp}" ] 
 add_field => [ "received_from", "%{@source_host}" ] 
 } 
 if !("_grokparsefailure" in [tags]) { 
 mutate { 
 replace => [ "@source_host", "%{syslog_hostname}" ] 
 replace => [ "@message", "%{syslog_message}" ] 
 } 
 } 
 mutate { 
 remove_field => [ "syslog_hostname", "syslog_message" ] 
 } 
 date { 
 match => [ "syslog_timestamp","MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] 
 } 
 syslog_pri { } 
 } 
 } 
 output { 
 # Emit events to stdout for easy debugging of what is going through 
 # logstash. 
 #stdout { debug => "true" } 
 # This will use elasticsearch to store your logs. 
 # The 'embedded' option will cause logstash to run the elasticsearch 
 # server in the same process, so you don't have to worry about 
 # how to download, configure, or run elasticsearch! 
 elasticsearch { 
 embedded => false 
 #embedded_http_port => 9200 
 cluster => elasticsearch 
 host => log.example.com 
 port => 9200 
 } 
 } 
 </pre> 

 * Start and enable logstash at boot: 
 <pre> 
 echo 'logstash_enable="YES"' 'logstash14_enable="YES"' >> /etc/rc.conf 
 service logstash logstash14 start 
 </pre> 

 h1. Install Kibana 

 * Install kibana: 
 <pre> 
 portmaster textproc/kibana 
 </pre> 

 h2. h3. Configuring Kibana 

 Kibana will use the default port 9200 to talk to Elasticsearch, Kibana's config.js will not need to be updated. 

 h1. h2. Installing Nginx 

 We will use Nginx as a proxy server to allow access to the dashboard from the Public Internet (with basic authentication). 

 * Install Nginx: 
 <pre> 
 portmaster www/nginx 
 </pre> 

 * Start and enable nginx at boot 
 <pre> 
 echo 'nginx_enable="YES"' >> /etc/rc.conf 
 service nginx start 
 </pre> 

 * Edit the nginx configuration file and change the primary server block as follows: 
 <pre> 
 vi /usr/local/etc/nginx/nginx.conf 
 </pre> 
 #* And add the following: 
 <pre> 
 # 
 # Nginx proxy for Elasticsearch + Kibana 
 # 
 # In this setup, we are password protecting the saving of dashboards. You may 
 # wish to extend the password protection to all paths. 
 # 
 # Even though these paths are being called as the result of an ajax request, the 
 # browser will prompt for a username/password on the first request 
 # 
 # If you use this, you'll want to point config.js at http://FQDN:80/ instead of 
 # http://FQDN:9200 
 # 
 server { 
  listen                  80; *:80 ; 
  server_name             localhost; 
  access_log              /var/log/nginx-logstash.log; 

  location / { 
    root    /usr/local/www/kibana; /usr/local/kibana; 
    index    index.html    index.htm; 
  } 
 </pre> 

 


 * Finally, restart nginx as follows: 
 <pre> 
 service nginx restart 
 </pre>

Back