Support #414
Updated by Daniel Curtis over 9 years ago
h1. Prepare the system h3. Prerequisites * Update the system The essential prerequisites required to execute Logstash are: <pre> # A working Java runtime environment. pkg update && pkg upgrade portsnap fetch extract </pre> # An ElasticSearch instance. * Install portmaster: <pre> cd /usr/ports/ports-mgmt/portmaster make install clean pkg2ng </pre> The former is required because LogStash is a JRuby application while the latter, although not technically a requirement, is the recommended output for Logstash. --- h1. h2. Installing ElasticSearch Java * Install elasticsearch: To install OpenJDK on FreeBSD you can use pkg to install a ready-to-use binary package: <pre> portmaster textproc/elasticsearch pkg install openjdk </pre> * Start and enable ElasticSearch at boot <pre> echo 'elasticsearch_enable="YES"' >> /etc/rc.conf service elasticsearch start </pre> h2. Securing Elasticsearch * Up to version 1.2, Elasticsearch's dynamic scripting capability was enabled by default. Since #* Currently, this tutorial sets up command will install an instance of OpenJDK v. 7 in both FreeBSD 9 and 10. If you'd rather install a different version, you can search the Kibana dashboard to be accessed from available packages and pick the public Internet, let's disable dynamic scripting by appending the following line at the end of the ElasticSearch configuration file: one you prefer (command output has been filtered for brevity): <pre> vi /etc/elasticsearch/elasticsearch.yml pkg search openjdk </pre> > script.disable_dynamic: true * Restart Elasticsearch: openjdk-7.60.19,1 > openjdk6-b31_3,1 > openjdk8-8.5.13_7 <pre> service elasticsearch restart pkg install openjdk8-8.5.13_7 </pre> h2. Installing ElasticSearch Logstash includes an embedded ElasticSearch instance you can use for standalone installations (see my previous post for an introductory view on Logstash operation modes). The required configuration to bootstrap the embedded ElasticSearch instance and to have Logstash use it as its outputs is described in the following sections. Although simpler from the standpoint of the configuration, Logstash installations using separate ElasticSearch instances are out of the scope of this post. --- h2. Installing Logstash h1. Install h3. Logstash installation procedure is fairly simple since it is distributed as a tarball * Install logstash: Download Logstash from the official website: cd /usr/local wget --no-check-certificate https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz * Extract the tarball in the designated installation directory: <pre> portmaster sysutils/logstash tar xzf logstash-1.4.2.tar.gz mv logstash-1.4.2 logstash cd logstash </pre> * Create the Logstash directories: <pre> mkdir /usr/local/etc/logstash14 mkdir /var/db/logstash14 mkdir /var/run/logstash14 </pre> * Create a basic configuration: <pre> vi /usr/local/etc/logstash14/logstash14.conf </pre> #* Then add the following: <pre> input { file { type => "syslog" # # Wildcards work, here :) # path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ] path => "/var/log/messages" start_position => "beginning" } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} (%{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}|%{GREEDYDATA:syslog_message})" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{@source_host}" ] } if !("_grokparsefailure" in [tags]) { mutate { replace => [ "@source_host", "%{syslog_hostname}" ] replace => [ "@message", "%{syslog_message}" ] } } mutate { remove_field => [ "syslog_hostname", "syslog_message" ] } date { match => [ "syslog_timestamp","MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ] } syslog_pri { } } } output { # Emit events to stdout for easy debugging of what is going through # logstash. #stdout { debug => "true" } # This will use elasticsearch to store your logs. # The 'embedded' option will cause logstash to run the elasticsearch # server in the same process, so you don't have to worry about # how to download, configure, or run elasticsearch! elasticsearch { embedded => false #embedded_http_port => 9200 cluster => elasticsearch host => log.example.com port => 9200 } } </pre> h3. Creating an rc.d Script An rc.d script is required in a BSD system to register a service, define its configuration and have the rc framework manage its lifetime. The following script can be used as is or as a starting point to customise your own. If used as is, be aware that the script uses the following default values: # *Installation directory*: ${logstash14_home="/usr/local/logstash"} # *Configuration file path*: ${logstash14_config="/usr/local/etc/${name}/${name}.conf"} # *ElasticSearch data directory*: ${logstash14_elastic_datadir="/var/db/logstash14"} # *Java home*: ${logstash14_java_home="/usr/local/openjdk7"} * Start and Create the Logstash rc.d script <pre> vi /usr/local/etc/rc.d/logstash14 </pre> #* Add the following: <pre> #!/bin/sh # Configuration settings for Logstash in /etc/rc.conf: # # logstash14_enable (bool): # Default value: "NO" # Flag that determines whether Logstash is enabled. # # logstash14_home (string): # Default value: "/usr/local/logstash" # Logstash installation directory. # # logstash14_config (string): # Default value: /usr/local/etc/${name}/${name}.conf # Logstash configuration file path. # # logstash14_mode (string): # Default value: "standalone" # Valid options: # "standalone": agent, web & elasticsearch # "web": Starts logstash as a web ui # "agent": Justs works as a log shipper # # logstash14_port (int): # Default value: 9292 # Port of the Kibana web interface. # # logstash14_log (bool): # Set to "NO" by default. # Set it to "YES" to enable logstash at boot: logging to file # Default output to /var/log/logstash.log # # logstash14_log_file (string): # Default value: "${logdir}/${name}.log" # Log file path. # # logstash14_java_home (string): # Default value: "/usr/local/openjdk7" # Root directory of the desired Java SDK. # The JAVA_HOME environment variable is set with the contents of this # variable. # # logstash14_java_opts (string): # Default value: "" # Options to pass to the Java Virtual Machine. # The JAVA_OPTS environment variable is set with the contents of this # variable. # # logstash14_elastic_datadir (string): # Default value: "/var/db/logstash14". # Data directory of the embedded ElasticSearch instance. # . /etc/rc.subr name=logstash14 rcvar=logstash14_enable load_rc_config ${name} logdir="/var/log" : ${logstash14_enable="NO"} : ${logstash14_home="/usr/local/logstash"} : ${logstash14_config="/usr/local/etc/${name}/${name}.conf"} : ${logstash14_log="NO"} : ${logstash14_mode="standalone"} : ${logstash14_port="9292"} : ${logstash14_log_file="${logdir}/${name}.log"} : ${logstash14_elastic_datadir="/var/db/logstash14"} : ${logstash14_java_home="/usr/local/openjdk7"} : ${logstash14_java_opts=""} piddir=/var/run/${name} pidfile=${piddir}/${name}.pid if [ -d $piddir ]; then mkdir -p $piddir fi logstash14_cmd="${logstash14_home}/bin/logstash" procname="${logstash14_java_home}/bin/java" logstash14_chdir=${logstash14_home} logstash14_log_options="" logstash14_elastic_options="" if checkyesno logstash14_log; then logstash14_log_options=" --log ${logstash14_log_file}" fi if [ ${logstash14_mode} = "standalone" ]; then logstash14_args="agent -f ${logstash14_config} ${logstash14_log_options} -- web --port ${logstash14_port}" logstash14_elastic_options="-Des.path.data=${logstash14_elastic_datadir}" elif [ ${logstash14_mode} = "agent" ]; then logstash14_args="agent -f ${logstash14_config} ${logstash14_log_options}" elif [ ${logstash14_mode} = "web" ]; then logstash14_args="web --port ${logstash14_port} ${logstash14_log_options}" fi JAVA_OPTS="${logstash14_java_opts} ${logstash14_elastic_options}" JAVA_HOME="${logstash14_java_home}" export JAVA_OPTS export JAVA_HOME command="/usr/sbin/daemon" command_args="-f -p ${pidfile} ${logstash14_cmd} ${logstash14_args}" required_files="${logstash14_home} ${logstash14_java_home} ${logstash14_cmd} ${logstash14_config}" run_rc_command "$1" </pre> * Now make the rc.d script executable: <pre> chmod 755 /usr/local/etc/rc.d/logstash14 </pre> NOTE: You can override any of the supported configuration values in the /etc/rc.conf file. If, for example, you want to use an alternate Java home path, just add the following line to /etc/rc.conf setting the desired value: > logstash14_java_home="/usr/local/openjdk8" h3. Testing the Logstash service * To test the Logstash service, the following command can be used: <pre> service logstash14 onestart </pre> * To stop it, use: <pre> service logstash14 onestop </pre> * To help troubleshooting any problem you might find you can enable the Logstash log, setting the logstash14_log variable to YES in the /etc/rc.conf file: <pre> echo 'logstash14_log="YES"' >> /etc/rc.conf </pre> * The log file location is specified by the logstash14_log_file variable, whose default value is set by the service rc file (only the relevant lines are shown): <pre> name=logstash14 logdir="/var/log" : ${logstash14_log_file="${logdir}/${name}.log"} </pre> The log file location can be overridden setting the logstash14_log_file variable in the /etc/rc.conf file. h3. Enabling the Logstash service Note that the rc script described above does not enable the Logstash service: > : ${logstash14_enable="NO"} * If everything works, you can enable the Logstash service just adding the following line to /etc/rc.conf: <pre> echo 'logstash14_enable="YES"' >> /etc/rc.conf </pre> h2. Install ElasticSearch * Install ElasticSearch: <pre> pkg install elasticsearch </pre> #* Enable ElasticSearch at boot <pre> echo 'elasticsearch_enable="YES"' >> /etc/rc.conf </pre> #* Start ElasticSearch <pre> service logstash14 elasticsearch start </pre> h1. Install h3. Securing Elasticsearch * Up to version 1.2, Elasticsearch's dynamic scripting capability was enabled by default. Since this tutorial sets up the Kibana dashboard to be accessed from the public Internet, let's disable dynamic scripting by appending the following line at the end of the ElasticSearch configuration file: <pre> vi /etc/elasticsearch/elasticsearch.yml </pre> > script.disable_dynamic: true * Install kibana: Restart Elasticsearch: <pre> portmaster textproc/kibana service elasticsearch restart </pre> h2. Installing and Configuring Kibana h3. Getting Kibana * Change to the fluentd user, move to your home directory and download Kibana as follows: <pre> cd /usr/local wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip unzip kibana-latest.zip mv kibana-latest kibana </pre> h3. Configuring Kibana Kibana will use the default port 9200 to talk to Elasticsearch, Kibana's config.js will not need to be updated. h2. Installing and Configuring Nginx (Proxy Server) We will use Nginx as a proxy server to allow access to the dashboard from the Public Internet (with basic authentication). * Install Nginx: Nginx as follows: <pre> portmaster www/nginx pkg install nginx </pre> * Start and enable #* Enable nginx to start at boot <pre> echo 'nginx_enable="YES"' >> /etc/rc.conf </pre> #* Start nginx <pre> service nginx start </pre> * Edit the nginx configuration file and change the primary server block as follows: <pre> vi /usr/local/etc/nginx/nginx.conf </pre> #* And add the following: <pre> # # Nginx proxy for Elasticsearch + Kibana # # In this setup, we are password protecting the saving of dashboards. You may # wish to extend the password protection to all paths. # # Even though these paths are being called as the result of an ajax request, the # browser will prompt for a username/password on the first request # # If you use this, you'll want to point config.js at http://FQDN:80/ instead of # http://FQDN:9200 # server { listen *:80 ; server_name localhost; access_log /var/log/nginx-logstash.log; location / { root /usr/local/kibana; index index.html index.htm; } </pre> * Finally, restart nginx as follows: <pre> service nginx restart </pre>