Project

General

Profile

Support #414

Install an ElasticSearch, Logstash, Kibana (ELK) Stack on FreeBSD

Added by Daniel Curtis about 6 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Logging Server
Target version:
Start date:
07/10/2014
Due date:
% Done:

100%

Estimated time:
2.00 h
Spent time:

Description

This is a guide on setting up an ElasticSearch, Logstash, Kibana stack with Nginx on FreeBSD 9.3-RELEASE.

Prepare the system

  • Update the system
    pkg update && pkg upgrade
    portsnap fetch extract
    
  • Install portmaster:
    cd /usr/ports/ports-mgmt/portmaster
    make install clean
    pkg2ng
    

Installing ElasticSearch

  • Install elasticsearch:
    portmaster textproc/elasticsearch
    
  • Start and enable ElasticSearch at boot
    echo 'elasticsearch_enable="YES"' >> /etc/rc.conf
    service elasticsearch start
    

Securing Elasticsearch

  • Up to version 1.2, Elasticsearch's dynamic scripting capability was enabled by default. Since this tutorial sets up the Kibana dashboard to be accessed from the public Internet, let's disable dynamic scripting by appending the following line at the end of the ElasticSearch configuration file:
    echo "script.disable_dynamic: true" >> /usr/local/etc/elasticsearch/elasticsearch.yml
    
  • Also enable cross origin access:
    echo 'http.cors.allow-origin: "/.*/"' >> /usr/local/etc/elasticsearch/elasticsearch.yml
    echo 'http.cors.enabled: true' >> /usr/local/etc/elasticsearch/elasticsearch.yml
    
  • Restart Elasticsearch:
    service elasticsearch restart
    

Install Logstash

  • Install logstash:
    portmaster sysutils/logstash
    
  • Create a basic configuration:
    vi /usr/local/etc/logstash/logstash.conf
    
    • Then modify the following:
      output {
      # Emit events to stdout for easy debugging of what is going through
      # logstash.
      #stdout { debug => "true" }
      
        elasticsearch {
          host => localhost
      }
      
  • Start and enable logstash at boot:
    echo 'logstash_enable="YES"' >> /etc/rc.conf
    service logstash start
    

Install Kibana

  • Install kibana:
    portmaster textproc/kibana
    

Configuring Kibana

Since Kibana will use port 80 to talk to Elasticsearch as opposed to the default port 9200, Kibana's config.js must be updated.

  • Open Kibana configuration file and look for the following line:
    vi /usr/local/www/kibana/config.js
    
    • And change the elasticsearch: "http://"+window.location.hostname+":9200", parameter to the following:
      elasticsearch: "http://"+window.location.hostname+":80",
      

Installing Nginx

We will use Nginx as a proxy server to allow access to the dashboard from the Public Internet (with basic authentication).

  • Install Nginx:
    portmaster www/nginx security/py-htpasswd
    
  • Start and enable nginx at boot
    echo 'nginx_enable="YES"' >> /etc/rc.conf
    service nginx start
    
  • Edit the nginx configuration file and change the primary server block as follows:
    vi /usr/local/etc/nginx/nginx.conf
    
    • And add the following:
      # Nginx proxy for Elasticsearch + Kibana
      #
      server {
       listen                80;
       server_name           localhost;
       access_log            /var/log/nginx-logstash.log;
      
       location / {
         root  /usr/local/www/kibana;
         index  index.html  index.htm;
       }
      
       location ~ ^/_aliases$ {
         proxy_pass http://127.0.0.1:9200;
         proxy_read_timeout 90;
       }
       location ~ ^/.*/_aliases$ {
         proxy_pass http://127.0.0.1:9200;
         proxy_read_timeout 90;
       }
       location ~ ^/_nodes$ {
         proxy_pass http://127.0.0.1:9200;
         proxy_read_timeout 90;
       }
       location ~ ^/.*/_search$ {
         proxy_pass http://127.0.0.1:9200;
         proxy_read_timeout 90;
       }
       location ~ ^/.*/_mapping {
         proxy_pass http://127.0.0.1:9200;
         proxy_read_timeout 90;
       }
      
       # Password protected end points
       location ~ ^/kibana-int/dashboard/.*$ {
         proxy_pass http://127.0.0.1:9200;
         proxy_read_timeout 90;
         limit_except GET {
           proxy_pass http://127.0.0.1:9200;
           auth_basic "Restricted";
           auth_basic_user_file /usr/local/etc/nginx/log.example.com.htpasswd;
         }
       }
       location ~ ^/kibana-int/temp.*$ {
         proxy_pass http://127.0.0.1:9200;
         proxy_read_timeout 90;
         limit_except GET {
           proxy_pass http://127.0.0.1:9200;
           auth_basic "Restricted";
           auth_basic_user_file /usr/local/etc/nginx/log.example.com.htpasswd;
         }
       }
      
  • And generate a htpasswd file:
    python2.7 /usr/local/bin/htpasswd.py -c -b /usr/local/etc/nginx/log.example.com.htpasswd username SuperSecretPassword
    
  • Finally, restart nginx as follows:
    service nginx restart
    

Related issues

Copied to FreeBSD Administration - Support #437: Install an ElasticSearch, Fluentd, Kibana Stack on FreeBSD Closed 07/10/2014

History

#1 Updated by Daniel Curtis about 6 years ago

  • Description updated (diff)

#2 Updated by Daniel Curtis about 6 years ago

  • Description updated (diff)

#3 Updated by Daniel Curtis about 6 years ago

  • Status changed from In Progress to Feedback
  • % Done changed from 20 to 90

#4 Updated by Daniel Curtis about 6 years ago

  • Copied to Support #437: Install an ElasticSearch, Fluentd, Kibana Stack on FreeBSD added

#5 Updated by Daniel Curtis about 6 years ago

  • Description updated (diff)
  • Status changed from Feedback to In Progress

#6 Updated by Daniel Curtis about 6 years ago

  • Description updated (diff)

#7 Updated by Daniel Curtis about 6 years ago

  • Description updated (diff)

#8 Updated by Daniel Curtis about 6 years ago

  • Description updated (diff)

#9 Updated by Daniel Curtis about 6 years ago

  • Subject changed from Installing Logstash on FreeBSD to Installing an ElasticSearch, Logstash, Kibana Stack on FreeBSD
  • Description updated (diff)

#10 Updated by Daniel Curtis over 5 years ago

  • Project changed from 90 to FreeBSD Administration
  • Category set to Logging Server
  • Target version set to FreeBSD 9

#11 Updated by Daniel Curtis over 5 years ago

  • Description updated (diff)

#12 Updated by Daniel Curtis over 5 years ago

  • Subject changed from Installing an ElasticSearch, Logstash, Kibana Stack on FreeBSD to Install an ElasticSearch, Logstash, Kibana (ELK) Stack on FreeBSD
  • Description updated (diff)

#13 Updated by Daniel Curtis over 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100

#14 Updated by Daniel Curtis over 5 years ago

  • Description updated (diff)

#15 Updated by Daniel Curtis over 5 years ago

  • Subject changed from Install an ElasticSearch, Logstash, Kibana (ELK) Stack on FreeBSD to Install an ElasticSearch, Logstash, Kibana (ELK) Stack on FreeBSD

#16 Updated by Daniel Curtis over 5 years ago

  • Description updated (diff)

#17 Updated by Daniel Curtis over 5 years ago

  • Description updated (diff)

#18 Updated by Daniel Curtis over 5 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF