Support #414
Updated by Daniel Curtis almost 10 years ago
I've decided to centralize all the logs generated by a client's production systems to a syslog server and after assessing a bunch of products, I chose Logstash (now part of the ElasticSearch family) as the tool to organise the unstructured logs into meaningful data structures which can then be searched, filtered and exploited.
The platform chosen to run Logstash is FreeBSD inside a Jail (9.2-RELEASE at the time), a rock-solid and very well documented UNIX-like operating system. Besides, it also ships a production-ready ZFS implementation which always comes handy in the data center.
h2. Installation
* Installing Logstash in FreeBSD is very easy because a good Logstash port exists and the binary package can be installed with a one-liner:
<pre>
pkg install logstash
</pre>
The FreeBSD package manager will take care of installing Logstash and all its dependencies.
h2. Configuration
The Logstash port performs a very good initial installation of Logstash and very few customizations are required in most cases.
h3. Operation Mode
The Logstash service rc script is installed in @/usr/local/etc/rc.d/logstash@ and supports three modes of operation (further information can be found in the official Logstash documentation):
* standalone
* agent
* web
* The operation mode can be set setting the logstash_mode variable in the /etc/rc.conf file:
> logstash_mode="standalone"
The default operation mode is +standalone+ does the following:
* It launches a local ElasticSearch instance.
* It launches the Logstash agent.
* It launches the bundled Kibana web interface.
A standalone Logstash service is the easiest way to bootstrap a fully functional Logstash server with an embedded ElasticSearch instance. A more complex setup, for example, could contemplate a separate ElasticSearch server.
h3. Embedded ElasticSearch
If the standalone operation mode is used, then an embedded ElasticSearch instance is launched. This instance stores its index files in the directory specified by the logstash_elastic_datadir configuration variable which, by default, is @/var/db/logstash@.
If you use the embedded ElasticSearch instance, you may want to mount a separate disk on @/var/db/logstash@ for easier management in case you wanted to dedicate more space as the time passes. A ZFS dataset, in this case, is possibly the most flexible option available on FreeBSD (and other systems).
If you don't plan to store ElasticSearch indexes indefinitely, you're likely looking for a way to optimise them and remove older index entries.
h3. Updating a Stale Logstash JAR
The Logstash port is not updated as often as Logstash is and you are likely going to get a pretty old version. Fortunately, the port lets you very easily run an updated Logstash binary. Beware the following:
* Changing a ports' file may cause problems while upgrading a port.
* Index files generated (or updated) by newer ElasticSearch instance may not be backwards compatible and rolling back a manual JAR upgrade may not be easy or possible altogether.
The Logstash service rc script uses the logstash_jar configuration variable to set the path of the Logstash JAR archive, the default value being
* @/usr/local/logstash/logstash-${version}-flatjar.jar@
where @${version}@ is the version of the Logstash port.
An updated Logstash JAR archive can be downloaded and the value of the logstash_jar variable can be overridden in the @/etc/rc.conf@ configuration file:
> logstash_jar="/usr/local/logstash/logstash-1.3.3-flatjar.jar"
Always test a newer Logstash instance in a test environment.
Edited on June, 20th: Since version 1.4 Logstash is not distributed as a single JAR file any longer.
h3. Adding Custom Java VM Options
Very often it's desirable to pass additional options to the Java VM but unfortunately the Logstash port service rc script doesn't allow you to do so. Unless, of course, you modify it.
I usually define a new empty configuration variable, called logstash_java_opts: in the service rc file @/usr/local/etc/rc.d/logstash@ (added line in bold):
> [...snip...]
> : ${logstash_java_home="/usr/local/openjdk6"}
> *: ${logstash_java_opts=""}*
> : ${logstash_log="NO"}
> [...snip...]
and then update the launch command (added fragment in bold):
[...snip...]
> command_args="-f -p ${pidfile} ${java_cmd} *${logstash_java_opts}*
> required_files="${java_cmd} ${logstash_config}"
>
> run_rc_command "$1"
Now, you can override the value of the @logstash_java_opts@ variable in @/etc/rc.conf@:
> logstash_java_opts="-Xmx2048M"
Once again, beware the consequence of changing the files of an installed binary package. In the meantime, I've asked the current port maintainer to consider a modification to support this use case.
h2. Configuring Logstash
Logstash should now be configured according to your needs. The configuration file used by this package rc script is specified by the logstash_config variable whose default value is set by the rc script (only the relevant lines are shown):
> name=logstash
> : ${logstash_config="/usr/local/etc/${name}/${name}.conf"}
This value can be overridden setting the @logstash_config@ variable in the @/etc/rc.conf@ configuration file.
Detailed information and working examples can be found in the official Logstash documentation.
h3. Testing Logstash
* To start the Logstash service to test its configuration, the following command can be used:
<pre>
service logstash onestart
</pre>
* To stop it, use:
<pre>
service logstash onestop
</pre>
To enable the Logstash log, the @logstash_log@ should be set to *YES* in the @/etc/rc.conf@ file:
> logstash_log="YES"
The log file location is specified by the logstash_log_file variable, whose default value is set by the service rc file (only the relevant lines are shown):
> name=logstash
> logdir="/var/log"
> : ${logstash_log_file="${logdir}/${name}.log"}
The log file location can be overridden setting the @logstash_log_file@ variable in the @/etc/rc.conf@ file.
h3. Enabling the Logstash Service
* When the configuration is correct, it can be enabled so that it's started during the system startup. To enable the Logstash service, add the following line to @/etc/rc.conf@:
> logstash_enable="YES"
h1. Installing Logstash (v1.4 and higher)
As seen in the previous post, a Logstash FreeBSD port exists, but it is currently outdated since it bundles Logstash v1.2. But while this could be used as a starting point for JAR-based Logstash installations (as we have seen, the update process only required updating the Logstash JAR), this is not possible with the new Logstash distribution because the included rc script will fail to work.
Skimming through the original post is recommended because it provides general information about Logstash and FreeBSD which is required to properly plan and execute a Logstash setup.
h2. Prerequisites
The essential prerequisites required to execute Logstash are:
* A working Java runtime environment
* An ElasticSearch instance
The former is required because LogStash is a JRuby application while the latter, although not technically a requirement, is the recommended output for Logstash.
h3. Installing Java
To install OpenJDK on FreeBSD you can use pkg to install a ready-to-use binary package:
<pre>
pkg install openjdk
</pre>
Currently, this command will install an instance of OpenJDK v.7 in both FreeBSD 9 and 10. If you'd rather install a different version, you can search the available packages and pick the one you prefer.
h3. Installing ElasticSearch
Logstash includes an embedded ElasticSearch instance you can use for standalone installations (see my previous post for an introductory view on Logstash operation modes). The required configuration to bootstrap the embedded ElasticSearch instance and to have Logstash use it as its outputs is described in the following sections.
Although simpler from the standpoint of the configuration, Logstash installations using separate ElasticSearch instances are out of the scope of this post.
* Install elasticsearch
<pre>
pkg install elasticsearch
</pre>
* Enable elasticsearch to start at boot
<pre>
echo 'elasticsearch_enable="YES"' >> /etc/rc.conf
</pre>
* Start elasticsearch
<pre>
service elasticsearch
</pre>
h2. Installing Logstash
Logstash installation procedure is fairly simple since it is distributed as a tarball:
* Download Logstash from the official website:
<pre>
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
</pre>
* Extract the tarball in the designated installation directory (my personal suggestion is to avoid /usr/local because it is used by ports and to use /opt instead):
<pre>
mkdir /opt
mv logstash-1.4.2.tar.gz /opt
tar xzf logstash-1.4.2.tar.gz
</pre>
h3. Creating an rc.d Script
An rc.d script is required in a BSD system to register a service, define its configuration and have the rc framework manage its lifetime. The following script can be used as is or as a starting point to customize your own. If used as is, be aware that the script uses the following default values:
# Installation directory: ${logstash14_home="/opt/logstash-1.4.2"}
# Configuration file path: ${logstash14_config="/usr/local/etc/${name}/${name}.conf"}
# ElasticSearch data directory: ${logstash14_elastic_datadir="/var/db/logstash14"}
# Java home: ${logstash14_java_home="/usr/local/openjdk7"}
* /usr/local/etc/rc.d/logstash14
<pre>
#!/bin/sh
# Configuration settings for Logstash in /etc/rc.conf:
#
# logstash14_enable (bool):
# Default value: "NO"
# Flag that determines whether Logstash is enabled.
#
# logstash14_home (string):
# Default value: "/opt/logstash-1.4.1"
# Logstash installation directory.
#
# logstash14_config (string):
# Default value: /usr/local/etc/${name}/${name}.conf
# Logstash configuration file path.
#
# logstash14_mode (string):
# Default value: "standalone"
# Valid options:
# "standalone": agent, web & elasticsearch
# "web": Starts logstash as a web ui
# "agent": Justs works as a log shipper
#
# logstash14_port (int):
# Default value: 9292
# Port of the Kibana web interface.
#
# logstash14_log (bool):
# Set to "NO" by default.
# Set it to "YES" to enable logstash logging to file
# Default output to /var/log/logstash.log
#
# logstash14_log_file (string):
# Default value: "${logdir}/${name}.log"
# Log file path.
#
# logstash14_java_home (string):
# Default value: "/usr/local/openjdk6"
# Root directory of the desired Java SDK.
# The JAVA_HOME environment variable is set with the contents of this
# variable.
#
# logstash14_java_opts (string):
# Default value: ""
# Options to pass to the Java Virtual Machine.
# The JAVA_OPTS environment variable is set with the contents of this
# variable.
#
# logstash14_elastic_datadir (string):
# Default value: "/var/db/logstash14".
# Data directory of the embedded ElasticSearch instance.
#
. /etc/rc.subr
name=logstash14
rcvar=logstash14_enable
load_rc_config ${name}
logdir="/var/log"
: ${logstash14_enable="NO"}
: ${logstash14_home="/opt/logstash-1.4.2"}
: ${logstash14_config="/usr/local/etc/${name}/${name}.conf"}
: ${logstash14_log="NO"}
: ${logstash14_mode="standalone"}
: ${logstash14_port="9292"}
: ${logstash14_log_file="${logdir}/${name}.log"}
: ${logstash14_elastic_datadir="/var/db/logstash14"}
: ${logstash14_java_home="/usr/local/openjdk7"}
: ${logstash14_java_opts=""}
piddir=/var/run/${name}
pidfile=${piddir}/${name}.pid
if [ -d $piddir ]; then
mkdir -p $piddir
fi
logstash14_cmd="${logstash14_home}/bin/logstash"
procname="${logstash14_java_home}/bin/java"
logstash14_chdir=${logstash14_home}
logstash14_log_options=""
logstash14_elastic_options=""
if checkyesno logstash14_log; then
logstash14_log_options=" --log ${logstash14_log_file}"
fi
if [ ${logstash14_mode} = "standalone" ]; then
logstash14_args="agent -f ${logstash14_config} ${logstash14_log_options} -- web --port ${logstash14_port}"
logstash14_elastic_options="-Des.path.data=${logstash14_elastic_datadir}"
elif [ ${logstash14_mode} = "agent" ]; then
logstash14_args="agent -f ${logstash14_config} ${logstash14_log_options}"
elif [ ${logstash14_mode} = "web" ]; then
logstash14_args="web --port ${logstash14_port} ${logstash14_log_options}"
fi
JAVA_OPTS="${logstash14_java_opts} ${logstash14_elastic_options}"
JAVA_HOME="${logstash14_java_home}"
export JAVA_OPTS
export JAVA_HOME
command="/usr/sbin/daemon"
command_args="-f -p ${pidfile} ${logstash14_cmd} ${logstash14_args}"
required_files="${logstash14_home} ${logstash14_java_home} ${logstash14_cmd} ${logstash14_config}"
run_rc_command "$1"
</pre>
You can override any of the supported configuration values in the @/etc/rc.conf@ file. If, for example, you want to use an alternate Java home path, just add the following line to @/etc/rc.conf@ setting the desired value:
<pre>
logstash14_java_home="/usr/local/openjdk7"
</pre>
h3. Testing the Service
* To test the Logstash service, the following command can be used:
<pre>
service logstash14 onestart
</pre>
* To stop it, use:
<pre>
service logstash14 onestop
</pre>
To help troubleshooting any problem you might find you can enable the Logstash log, setting the @logstash14_log@ variable to +YES+ in the @/etc/rc.conf@ file:
<pre>
logstash14_log="YES"
</pre>
The log file location is specified by the logstash14_log_file variable, whose default value is set by the service rc file (only the relevant lines are shown):
> name=logstash14
> logdir="/var/log"
> : ${logstash14_log_file="${logdir}/${name}.log"}
The log file location can be overridden setting the @logstash14_log_file@ variable in the @/etc/rc.conf@ file.
h3. Enabling the Service
Note that the rc script described above does not enable the Logstash service:
<pre>
: ${logstash14_enable="NO"}
</pre>
If everything works, you can enable the Logstash service just adding the following line to @/etc/rc.conf@:
<pre>
logstash14_enable="YES"
</pre>
h2. Resources
* http://thegreyblog.blogspot.com/2014/01/installing-logstash-on-freebsd.html
* http://logstash.net/docs/1.2.0/