Support #437
Updated by Daniel Curtis about 9 years ago
{{>toc}}
I've decided to centralize all the logs generated by a client's production systems to a syslog server and after assessing a bunch of products, I am now working with Fluentd.
The platform chosen to run Fluentd is FreeBSD inside a Jail (9.3-RELEASE (9.2-RELEASE at the time), a rock-solid and very well documented UNIX-like operating system. Besides, it also ships a production-ready ZFS implementation which always comes handy in the data center. FreeBSD-9.3 FreeBSD-9.2 currently has the _sysutils/rubygem-fluentd_ is available in the ports tree. This tree, -however the package has been marked as BROKEN- and is a guide on available using pkg. So I will document how to install Fluentd with ElasticSearch and Kibana.
h1. Setting up the Environment h2. Prerequisites
* Start by making sure everything is up to date: Install rubygem-fluentd
<pre>
pkg update && pkg upgrade install rubygem-fluentd
portsnap fetch extract
</pre>
* Set the default Ruby version Now enable fluentd to 2.2: start at boot:
<pre>
echo 'DEFAULT_VERSIONS= ruby=2.2' 'fluentd_enable="YES"' >> /etc/make.conf /etc/rc.conf
</pre>
h3. (*Optional*) Installation of Fluentd from source
* Install portmaster: In order to build Fluentd, there are are a few packages needed:
<pre>
cd /usr/ports/ports-mgmt/portmaster
make pkg install clean git ruby ruby19-gems rubygem-bundler
pkg2ng
</pre>
h1. Install * Fetch the Fluentd
* Install rubygem-fluentd source code:
<pre>
portmaster sysutils/rubygem-fluentd cd /usr/local
git clone https://github.com/fluent/fluentd.git
cd fluentd
</pre>
* Start Build and enable fluentd at boot: Install
<pre>
echo 'fluentd_enable="YES"' >> /etc/rc.conf bundle install
service fluentd start
</pre>
h2. Installing Fluentd Plugins
We need a couple of plugins:
# *out_elasticsearch*: this plugin lets Fluentd > Fetching gem metadata from https://rubygems.org/.........
> ...
> Your bundle is complete!
> Use `bundle show [gemname]` to stream data to Elasticsearch.
# *outrecordreformer*: this plugin lets us process data into see where a more useful format.
* The following commands install both plugins: bundled gem is installed.
<pre>
fluent-gem install fluent-plugin-elasticsearch bundle exec rake build
fluent-gem </pre>
> fluentd xxx built to pkg/fluentd-xxx.gem.
<pre>
gem install fluent-plugin-record-reformer pkg/fluentd-xxx.gem
</pre>
h2. Add h3. Test Run
* Run the Syslog configuration following commands to to confirm that Fluentd
Next, we configure Fluentd to listen to syslog messages and send them to Elasticsearch.
* Edit the fluentd config file: was installed successfully:
<pre>
vi /usr/local/etc/fluentd/fluent.conf fluentd --setup ./fluent
fluentd -c ./fluent/fluent.conf -vv &
echo '{"json":"message"}' | fluent-cat debug.test
</pre>
#* And add If the installation was successful, Fluentd will output the following lines at the top of the file: message:
<pre>
## Syslog input 2011-07-10 16:49:50 +0900 debug.test: {"json":"message"}
<source>
type syslog
port 5140
tag system
</source>
<match system.*.*>
type record_reformer
tag elasticsearch
facility ${tag_parts[1]}
severity ${tag_parts[2]}
</match>
<match elasticsearch>
type copy
<store>
type stdout
</store>
<store>
type elasticsearch
logstash_format true
flush_interval 5s #debug
</store>
</match>
</pre>
* Restart fluentd:
<pre>
service fluentd restart
</pre>
#* (Optional) Start The last command sends Fluentd a message ‘{“json”:”message”}’ with verbose debugging, run a “debug.test” tag.
#* Kill the following command: test run:
<pre>
fluentd -c /usr/local/fluentd/fluent.conf -vv & fg 1
[ctrl]+[c]
</pre>
---
h1. h2. Install ElasticSearch
* Install ElasticSearch:
<pre>
portmaster textproc/elasticsearch pkg install elasticsearch
</pre>
* Start and enable Enable ElasticSearch at boot
<pre>
echo 'elasticsearch_enable="YES"' >> /etc/rc.conf
</pre>
* Start ElasticSearch
<pre>
service elasticsearch start
</pre>
h2. h3. Securing Elasticsearch
* Up to version 1.2, Elasticsearch's dynamic scripting capability was enabled by default. Since this tutorial sets up the Kibana dashboard to be accessed from the public Internet, let's disable dynamic scripting by running the following line to append the parameter at the end of the ElasticSearch configuration file:
<pre>
echo 'script.disable_dynamic: true' >> /usr/local/etc/elasticsearch/elasticsearch.yml
</pre>
* Restart Elasticsearch:
<pre>
service elasticsearch restart
</pre>
---
h1. Install h2. Installing and Configuring Kibana
* Install kibana:
<pre>
portmaster textproc/kibana pkg install kibana
</pre>
h3. (*Optional*) Installing Kibana from source
* Change to the fluentd user, move to your home directory and download Kibana as follows:
<pre>
cd /usr/local
wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip
unzip kibana-latest.zip
mv kibana-latest kibana
cd kibana
</pre>
h3. Configuring Kibana
Since Kibana will use port 80 to talk to Elasticsearch as opposed to the default port 9200, Kibana's config.js must be updated.
* Open Kibana configuration file and look for the following line:
<pre>
vi /usr/local/www/kibana/config.js
</pre>
#* And change the @elasticsearch: "http://"+window.location.hostname+":9200",@ parameter to the following:
<pre>
elasticsearch: "http://"+window.location.hostname+":80",
</pre>
h1. Install h2. Installing and Configuring Nginx (Proxy Server)
We will use Nginx as a proxy server to allow access to the dashboard from the Public Internet (with basic authentication).
* Install Nginx as follows:
<pre>
portmaster www/nginx security/py-htpasswd pkg install nginx py27-htpasswd
</pre>
* Enable nginx to start at boot
<pre>
echo 'nginx_enable="YES"' >> /etc/rc.conf
</pre>
* Start nginx
<pre>
service nginx start
</pre>
* Edit /usr/local/etc/nginx/nginx.conf
<pre>
vi /usr/local/etc/nginx/nginx.conf
</pre>
#* And change the primary server block as follows:
<pre>
#
# Nginx proxy for Elasticsearch + Kibana
#
# In this setup, we are password protecting the saving of dashboards. You may
# wish to extend the password protection to all paths.
#
# Even though these paths are being called as the result of an ajax request, the
# browser will prompt for a username/password on the first request
#
# If you use this, you'll want to point config.js at http://FQDN:80/ instead of
# http://FQDN:9200
#
server {
listen *:80 ;
server_name localhost;
access_log /var/log/nginx-kibana.log;
location / {
root /usr/local/www/kibana;
index index.html index.htm;
}
location ~ ^/_aliases$ {
proxy_pass http://127.0.0.1:9200;
proxy_read_timeout 90;
}
location ~ ^/.*/_aliases$ {
proxy_pass http://127.0.0.1:9200;
proxy_read_timeout 90;
}
location ~ ^/_nodes$ {
proxy_pass http://127.0.0.1:9200;
proxy_read_timeout 90;
}
location ~ ^/.*/_search$ {
proxy_pass http://127.0.0.1:9200;
proxy_read_timeout 90;
}
location ~ ^/.*/_mapping {
proxy_pass http://127.0.0.1:9200;
proxy_read_timeout 90;
}
# Password protected end points
location ~ ^/kibana-int/dashboard/.*$ {
proxy_pass http://127.0.0.1:9200;
proxy_read_timeout 90;
limit_except GET {
proxy_pass http://127.0.0.1:9200;
auth_basic "Restricted";
auth_basic_user_file /usr/local/etc/nginx/log.example.com.htpasswd;
}
}
location ~ ^/kibana-int/temp.*$ {
proxy_pass http://127.0.0.1:9200;
proxy_read_timeout 90;
limit_except GET {
proxy_pass http://127.0.0.1:9200;
auth_basic "Restricted";
auth_basic_user_file /usr/local/etc/nginx/log.example.com.htpasswd;
}
}
}
</pre>
* And generate a htpasswd file:
<pre>
python2.7 /usr/local/bin/htpasswd.py -c -b /usr/local/etc/nginx/log.example.com.htpasswd username SuperSecretPassword
</pre>
*NOTE*: Make sure to change the username and SuperSecretPassword to your needs
* Finally, restart nginx as follows:
<pre>
service nginx restart
</pre>
Now, you should be able to see the generic Kibana dashboard at your server's IP address or domain, using your favorite browser.
--- h2. Installing Fluentd Plugins
h1. Forwarding We need a couple of plugins:
# *out_elasticsearch*: this plugin lets Fluentd to stream data to Elasticsearch.
# *outrecordreformer*: this plugin lets us process data into a more useful format.
* The following commands install both plugins, it requires libcurl
<pre>
pkg install curl fpc-libcurl
fluent-gem install fluent-plugin-elasticsearch
fluent-gem install fluent-plugin-record-reformer
</pre>
h2. Add the Syslog configuration to Fluentd
Next, we configure Fluentd to listen to syslog messages and send them to Elasticsearch.
* Edit @/usr/local/etc/fluentd/fluent.conf@ and add the following lines at the top of the file:
<pre>
## Syslog input
<source>
type syslog
port 5140
tag system
</source>
<match system.*.*>
type record_reformer
tag elasticsearch
facility ${tag_parts[1]}
severity ${tag_parts[2]}
</match>
<match elasticsearch>
type copy
<store>
type stdout
</store>
<store>
type elasticsearch
logstash_format true
flush_interval 5s #debug
</store>
</match>
</pre>
h2. Starting Fluentd
* Start fluentd with the following command:
<pre>
service fluentd start
</pre>
#* To start Fluentd with verbose debugging, run the following command:
<pre>
fluentd -c /usr/local/fluentd/fluent.conf -vv &
</pre>
---
h2. Forwarding Debian rsyslog Traffic to Fluentd
I use Debian an many production systems, and one one the packages is rsyslogd. It needs to be reconfigured to forward syslog events to the port Fluentd listens to (port 5140 in this example).
* Open the rsyslog configuration file and add the following line at the top
<pre>
sudo vi/etc/rsyslog.conf
</pre>
#* And add the following line
<pre>
*.* @127.0.0.1:5140
</pre>
* After saving and exiting the editor, restart rsyslogd as follows:
<pre>
sudo service rsyslog restart
</pre>
h2. Forwarding FreeBSD syslogd Traffic to Fluentd
I also have been switching many of my production systems to FreeBSD, and the default logging mechanism is syslogd. It needs to be reconfigured to forward syslog events to the port Fluentd listens to (port 5140 in this example).
* Open the rsyslog configuration file and add the following line at the top
<pre>
sudo vi/etc/syslog.conf
</pre>
#* And add the following line
<pre>
*.* @127.0.0.1:5140
</pre>
---
h2. Setting Up Kibana Dashboard Panels
Kibana's default panels are very generic, so it's recommended to customize them. Here, we show two methods.
h3. *Method 1*: Using a Template
* The Fluentd team offers an alternative Kibana configuration that works with this setup better than the default one. To use this alternative configuration, run the following command:
<pre>
sudo cp default.json /usr/local/kibana/app/dashboards/default.json
</pre>
Note: The original configuration file is from the author's GitHub gist.
If you refresh your Kibana dashboard home page at your server's URL, Kibana should now be configured to show histograms by syslog severity and facility, as well as recent log lines in a table.
h3. *Method 2*: Manually Configuring
Go to your server's IP address or domain to view the Kibana dashboard.
There are a couple of starter templates, but let's choose the blank one called *Blank Dashboard*.
Next, click on the *+ ADD A ROW* button on the right side of the dashboard. A configuration screen for a new row (a row consists of one or more panels) should show up. Enter a title, press the *Create Row* button, followed by *Save*. This creates a row.
When an empty row is created, Kibana shows the prompt Add panel to empty row on the left. Click this button. It takes you to the configuration screen to add a new panel. Choose histogram from the dropdown menu.
There are many parameters to configure for a new histogram, but you can just scroll down and press the Save button. This creates a new panel.
h2. Resources
* http://docs.fluentd.org/articles/install-from-source
* https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization
* https://raw.githubusercontent.com/elasticsearch/kibana/kibana3/sample/nginx.conf