Support #543
Updated by Daniel Curtis almost 10 years ago
{{>toc}} Puppet and Nagios have been two of the most useful tools for my small business; puppet allowing me to manage many servers and services using Ruby to write my configs, and nagios allowing me to monitor the servers and services. This is a simple guide to setup something similar to the system I have on FreeBSD 9.2. h1. Prepare the Server * Once the server baseline has been installed and root access has been obtained make sure the server is up to date: <pre> pkg update && upgrade portsnap fetch extract </pre> h2. Install and configure MariaDB server * Start by installing the mariadb-server and mariadb-client packages: <pre> pkg install mariadb55-{server,client} </pre> * Copy a base MariaDB configuration to use <pre> cp /usr/local/share/mysql/my-small.cnf /var/db/mysql/my.cnf </pre> * Tuning: Copy one of the default config files and change the max packet size to allow for the 17 MB data rows that Dashboard can occasionally send: <pre> vi /var/db/mysql/my.cnf </pre> #* and modify @max_allowed_packet@ to 32M <pre> max_allowed_packet = 32M </pre> * Enable and start MariaDB <pre> echo 'mysql_enable="YES"' >> /etc/rc.conf service mysql-server start </pre> * Prepare Database for use by running the secure installation. +Choose a root password+ and answer +yes+ to all questions. <pre> mysql_secure_installation </pre> * Login to MariaDB and create appropriate tables for Dashboard. <pre> mysql -u root -p </pre> #* and run the following SQL queries to create the *puppetmaster* user: <pre> CREATE DATABASE dashboard_production CHARACTER SET utf8; CREATE USER 'puppetmaster'@'127.0.0.1' IDENTIFIED BY 'SecretPuppetMasterPassword'; GRANT ALL PRIVILEGES ON puppetmaster.* TO 'puppetmaster'@'127.0.0.1'; flush privileges; </pre> #* and run the following SQL queries to create the *puppet-dashboard* user: <pre> CREATE DATABASE dashboard_production CHARACTER SET utf8; CREATE USER 'dashboard'@'127.0.0.1' IDENTIFIED BY 'SuperSecretPassword'; GRANT ALL PRIVILEGES ON dashboard_production.* TO 'dashboard'@'127.0.0.1'; flush privileges; </pre> #* and run the following SQL queries to create the *nagios* user: <pre> CREATE DATABASE nagios CHARACTER SET utf8; CREATE USER 'nagios'@'127.0.0.1' IDENTIFIED BY 'SecretNagiosPassword'; GRANT ALL PRIVILEGES ON nagios.* TO 'nagios'@'127.0.0.1'; flush privileges; </pre> h1. Install Puppet Master * Install the puppet package <pre> pkg install puppet rubygem-puppet-lint rubygem-rake rubygem-bundler rubygem-activerecord rubygem-sqlite3 libxslt node git </pre> * Enable puppet and/or puppetmaster in @/etc/rc.conf@: <pre> echo 'puppet_enable="YES"' >> /etc/rc.conf echo 'puppetmaster_enable="YES"' >> /etc/rc.conf </pre> h2. Puppet Initial Testing * At this point, Puppet needs to be started so that all its SSL keys can be generated. This gives the chance to test that Puppet does work before anything else gets stacked on as well as ensures the SSL keys referenced by Nginx's config file are in place before that step. <pre> service puppetmaster onestart </pre> * On *client.example.com* - start Puppet on the client system <pre> service puppet onestart </pre> #* Or <pre> puppet agent -vt --waitforcert 60 </pre> * On *puppet.example.com* - sign client.example.com's SSL key on the Puppetmaster <pre> puppet cert sign client.example.com </pre> * On *client.example.com* - Run a test on the client to ensure it works and do a onestop afterwards <pre> puppet agent --test service puppet onestop </pre> *NOTE*: I encountered a problem while migrating the Admin node, the new server uses a newer version of Puppet, and broke the fileserver feature. To work around this: # In *@/usr/local/etc/puppet/fileserver.conf@* put the name of your mount point, the path, and an allow * directive.: <pre> [files] path /usr/local/etc/puppet/files allow * </pre> # In *@/usr/local/etc/puppet/auth.conf@*: Use a regular expression path to match both the file_metadata and file_content endpoints followed by the name of your custom mount point. Then, use any combination of allow and allow_ip directives to control access. <pre> path ~ ^/file_(metadata|content)/files/ auth yes allow /^(.+\.)?example.com$/ allow_ip 192.168.100.0/24 </pre> Configure your @[puppetmasterd]@ section to reflect these settings: <pre> [puppetmasterd] storeconfigs = true dbadapter = mysql dbuser = puppetmaster dbpassword = SecretPuppetMasterPassword dbserver = localhost dbsocket = /var/run/mysqld/mysqld.sock </pre> * To optimize some often run Puppet queries on your MySQL database, use the following index: <pre> create index exported_restype_title on resources (exported, restype, title(50)); </pre> h1. Install Puppet Dashboard Now its time to install Puppet Dashboard, a web frontend to display puppet reports. * Install puppet-dashboard from git <pre> cd /usr/local/www git clone git://github.com/sodabrew/puppet-dashboard.git </pre> * Manually create the 'puppet-dashboard' user and provide required permissions: <pre> pw groupadd -n puppet-dashboard -g 800 pw useradd -n puppet-dashboard -c "Puppet Dashboard,,," -u 800 -g puppet-dashboard -s /usr/sbin/nologin chown -R puppet-dashboard:puppet-dashboard /usr/local/www/puppet-dashboard </pre> h2. Configure Puppet Dashboard * Copy the example database YAML file and update with database information: <pre> cd /usr/local/www/puppet-dashboard/config cp database.yml.example database.yml </pre> * Then patch the database.yml with the corrected information; make sure to replace the +password+ and +host+: <pre> vi database.yml </pre> #* and modify the following parameter accordingly: <pre> production: database: dashboard_production username: dashboard password: SuperSecretPassword encoding: utf8 adapter: mysql2 </pre> * Change the ownership and harden the permissions: <pre> chown puppet-dashboard:puppet-dashboard database.yml chmod 660 database.yml </pre> * Copy the example settings YAML file, no changes needed: <pre> cd /usr/local/www/puppet-dashboard/config cp settings.yml.example settings.yml chown puppet-dashboard:puppet-dashboard settings.yml chmod 660 settings.yml </pre> * Fix shebang line in External Node Classifier Script. <pre> sed -i '' -e 's/#! \/usr\/bin\/ruby/#!\/usr\/local\/bin\/ruby/' /usr/local/share/puppet-dashboard/bin/external_node </pre> * Install gems required in the 'Gemfile' via the Rubygem Bundler. If the postgresql gem bundles is required additional dependencies are needed: <pre> cd /usr/local/www/puppet-dashboard bundle install --path vendor/bundle --without postgresql </pre> * Generate secret_token. Cleanup any errors and the default token after generating the new one. <pre> echo "secret_token: `bundle exec rake secret`" >> config/settings.yml vi config/settings.yml </pre> h3. Creating and configuring the MariaDB database for Puppet Dashboard * At this point the database was already installed with some blank tables. We need to run rake to finish the process with the database structure needed: <pre> cd /usr/local/share/puppet-dashboard env RAILS_ENV=production bundle exec rake db:setup </pre> h3. Testing That Dashboard is Working * Run Dashboard using Ruby's built-in WEBrick server to validate functionality. It will be available at http://dashboard:3000 <pre> cd /usr/local/share/puppet-dashboard su -m puppet-dashboard -c 'bundle exec rails server' </pre> * Before going into a production environment, Dashboard 2.0 must precompile assets for production: <pre> env RAILS_ENV=production bundle exec rake assets:precompile </pre> * Chown any files created up until now to the right owner: <pre> chown -R puppet-dashboard:puppet-dashboard /usr/local/share/puppet-dashboard </pre> h1. Configuring Puppet All agent nodes have to be configured to submit reports to the master. The master has to be configured to send reports to Dashboard. If you already have a working Puppet installation you can configure it to distribute the updated puppet.conf to your hosts. * puppet.conf (on each agent) <pre> [agent] report = true </pre> * puppet.conf (on the Puppetmaster) <pre> [master] reports = store, http reporturl = http://dashboard.example.com:3000/reports/upload node_terminus = exec external_nodes = /usr/bin/env PUPPET_DASHBOARD_URL=http://dashboard.example.com:3000 /usr/local/share/puppet-dashboard/bin/external_node </pre> * Testing Puppet's Connection to Dashboard From a Puppet agent, run @puppet agent --test@. A new background task should show in the Dashboard UI at http://dashboard:3000 h2. Starting and Managing Delayed Job Workers * Dashboard ships a worker process manager under script/delayed_job. It can manually start delayed jobs via the following command: <pre> su -m puppet-dashboard -c 'env RAILS_ENV=production bundle exec script/delayed_job -p dashboard -n 2 -m start' </pre> h2. Delayed Job Worker Init Script However, rather than manually triggering background workers, this rc script will accomplish the same thing and ensure the background jobs get started on the next reboot. * Create puppet dashboard FreeBSD init script: <pre> vi /usr/local/etc/rc.d/dashboard_workers </pre> #* and add the following <pre> #!/bin/sh # PROVIDE: dashboard_workers # REQUIRE: LOGIN # KEYWORD: shutdown # By default dashboard_workers uses flags '-n 1' for 1 worker. This should be # adjusted to the number of CPU cores. dashboard_workers_enable=${dashboard_workers_enable:-"NO"} dashboard_workers_flags=${dashboard_workers_flags:-"-n 1"} # The default rails environment is set to production dashboard_workers_env=${dashboard_workers_env:-"/usr/bin/env PATH=${PATH}:/usr/local/bin RAILS_ENV=production"} # The default user is set to puppet-dashboard and install location is set to # /usr/local/share/puppet-dashboard. dashboard_workers_user=${dashboard_workers_user:-"puppet-dashboard"} dashboard_workers_chdir=${dashboard_workers_chdir:-"/usr/local/share/puppet-dashboard"} . /etc/rc.subr name="dashboard_workers" rcvar="dashboard_workers_enable" load_rc_config $name extra_commands="reload run zap status" # All commands call the same function and strip the fast|one|quiet prefix # to deliver to the bundler. reload_cmd="f_dashboard_workers reload" restart_cmd="f_dashboard_workers restart" run_cmd="f_dashboard_workers run" start_cmd="f_dashboard_workers start" status_cmd="f_dashboard_workers status" stop_cmd="f_dashboard_workers stop" zap_cmd="f_dashboard_workers zap" # Use the function's ARVG $1 as the bundler program's '-m' flag f_dashboard_workers() { cd $dashboard_workers_chdir && \ su -m "$dashboard_workers_user" \ -c "${dashboard_workers_env} bundle exec script/delayed_job ${rc_flags} -p dashboard -m $1" || \ echo "Failed to $1 dashboard_workers" } run_rc_command "$1" </pre> #* Ane make it executable: <pre> chmod +x /usr/local/etc/rc.d/dashboard_workers </pre> * With that in place, we need to override the defaults and enable the script along with setting '-n 4' workers to match the number of processor cores and ensure it's ready for a production workload. <pre> echo 'dashboard_workers_enable="YES"' >> /etc/rc.conf echo 'dashboard_workers_flags="-n 4"' >> /etc/rc.conf service dashboard_workers start </pre> h1. Running Dashboard in a Production-Quality Server with Nginx and Passenger * Compile and install Nginx and Passenger <pre> portsnap fetch extract cd /usr/ports/www/nginx make config make install clean rehash mkdir /usr/local/etc/nginx/conf.d </pre> *NOTE*: Make sure to enable [X]PASSENGER when running @make config@ * Configuring Nginx and Passenger, edit the @/usr/local/etc/nginx/nginx.conf@ file: <pre> vi /usr/local/etc/nginx/conf.d/nagios.conf And add/modify the following user www www; worker_processes 4; error_log /var/log/nginx/error.log notice; pid /var/run/nginx.pid; events { worker_connections 1024; } http { passenger_root /usr/local/lib/ruby/gems/1.9/gems/passenger-4.0.29; passenger_ruby /usr/local/bin/ruby; passenger_max_pool_size 15; passenger_pool_idle_time 300; #passenger_spawn_method direct; # Uncomment on Ruby 1.8 for ENC to work include mime.types; default_type application/octet-stream; sendfile on; tcp_nopush on; keepalive_timeout 65; tcp_nodelay on; # Load config files from the /etc/nginx/conf.d directory include /usr/local/etc/nginx/conf.d/*.conf; } </pre> * And create a server block for the puppet master <pre> vi /usr/local/etc/nagios/conf.d/puppetmaster.conf </pre> #* And add the following: <pre> server { listen 8140 ssl; server_name puppet.example.com; passenger_enabled on; passenger_set_cgi_param HTTP_X_CLIENT_DN $ssl_client_s_dn; passenger_set_cgi_param HTTP_X_CLIENT_VERIFY $ssl_client_verify; passenger_user puppet; passenger_group puppet; access_log /var/log/nginx/puppet_access.log; root /usr/local/etc/puppet/rack/public; ssl_certificate /var/puppet/ssl/certs/puppet.example.com.pem; ssl_certificate_key /var/puppet/ssl/private_keys/puppet.example.com.pem; ssl_crl /var/puppet/ssl/ca/ca_crl.pem; ssl_client_certificate /var/puppet/ssl/certs/ca.pem; ssl_ciphers SSLv2:-LOW:-EXPORT:RC4+RSA; ssl_prefer_server_ciphers on; ssl_verify_client optional; ssl_verify_depth 1; ssl_session_cache shared:SSL:128m; ssl_session_timeout 5m; } </pre> * And create a server block for the puppet dashboard <pre> vi /usr/local/etc/nagios/conf.d/puppet-dashboard.conf </pre> #* And add the following: server { listen 3000; server_name puppet.example.com; passenger_enabled on; passenger_user puppet-dashboard; passenger_group puppet-dashboard; access_log /var/log/nginx/dashboard_access.log; root /usr/local/share/puppet-dashboard/public; } * And add a default site configuration in /usr/local/etc/nginx/conf.d/default.conf: <pre> server { listen 80 default; server_name _; index index.html index.php; root /usr/local/www; # IP and IP ranges which should get access allow 10.0.0.0/24; allow 10.1.0.1; # all else will be denied deny all; # basic HTTP auth auth_basic "Restricted"; auth_basic_user_file htpasswd; location ~ \.cgi$ { try_files $uri =404; include fastcgi_params; fastcgi_pass unix:/var/run/fcgiwrap/fcgiwrap.sock; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param REMOTE_USER $remote_user; } location ~ \.php$ { try_files $uri =404; include fastcgi_params; fastcgi_pass 127.0.0.1:9000; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; } } </pre> * Create the log directory to prevent issues on startup: <pre> mkdir /var/log/nginx </pre> * Enable a daily log file rotation via @newsyslog.conf@: <pre> printf "/var/log/nginx/*.log\t\t\t644 7\t * @T00 JG /var/run/nginx.pid 30\n" >> /etc/newsyslog.conf </pre> * If the puppetmaster service is still running from earlier testing, stop it now: <pre> service puppetmaster onestop </pre> * With initial setup of the Puppetmaster done, a RACK file that Nginx will use to start the Ruby application will be needed: <pre> mkdir -p /usr/local/etc/puppet/rack/public </pre> * Create the config.ru file <pre> vi /usr/local/etc/puppet/rack/config.ru </pre> #* And add the following <pre> # Trimmed back FreeBSD Version of https://github.com/puppetlabs/puppet/blob/master/ext/rack/files/config.ru $0 = "master" ARGV << "--rack" ARGV << "--confdir" << "/usr/local/etc/puppet" ARGV << "--vardir" << "/var/puppet" require 'puppet/util/command_line' run Puppet::Util::CommandLine.new.execute </pre> * Make the script executable <pre> chown -R puppet:puppet /usr/local/etc/puppet/rack </pre> * Enable nginx service and start it. At this point basic functionality is online: <pre> echo 'nginx_enable="YES"' >> /etc/rc.conf service nginx start </pre> *NOTE*: I had a problem connecting to the puppet master, getting 403 errors, after I had setup Nginx/Passenger. The problem was with the @/usr/local/etc/puppet/puppet.conf@ and there being a couple of parameters that needed to be taken out. If the following two lines are present, remove them. They are are provided in the nginx.conf: > ssl_client_header = SSL_CLIENT_S_DN > ssl_client_verify_header = SSL_CLIENT_VERIFY h2. Configuring Dashboard - Advanced Features * Generating Certs and Connecting to the Puppet Master With separate Puppet/Dashboard systems the puppet cert sign dashboard will be on the Puppetmaster: <pre> cd /usr/local/share/puppet-dashboard su -m puppet-dashboard -c 'bundle exec rake cert:create_key_pair' su -m puppet-dashboard -c 'bundle exec rake cert:request' puppet cert sign dashboard su -m puppet-dashboard -c 'bundle exec rake cert:retrieve' </pre> * Enabling Inventory Support *# Example @auth.conf@ (on Puppet master) <pre> path /facts auth yes method find, search allow dashboard </pre> * Enabling the Filebucket Viewer *# Example site.pp (on Puppet master): <pre> filebucket { "main": server => "{your puppet master}", path => false, } </pre> *# In either site.pp, in an individual init.pp, or in a specific manifest. <pre> File { backup => "main" } </pre> * Go back and add the line for Inventory Support <pre> cd /usr/local/share/puppet-dashboard/config vi settings.yml </pre> #*And change the following parameters: <pre> enable_inventory_service: true use_file_bucket_diffs: true </pre> * With all the updates made, restart so that it takes effect: <pre> service nginx restart </pre> * For future maintenance, periodic jobs to prune old reports and run DB optimization. <pre> mkdir -p /usr/local/etc/periodic/monthly vi /usr/local/etc/periodic/monthly/clean_dashboard_database.sh </pre> #* And add the following: <pre> #!/bin/sh cd /usr/local/share/puppet-dashboard && \ echo "Pruning Old Reports from Puppet Dashboard Database" && \ /usr/bin/su -m puppet-dashboard -c '/usr/local/bin/bundle exec rake RAILS_ENV=production reports:prune upto=3 unit=mon' && \ echo "Optimizing Database" && \ /usr/bin/su -m puppet-dashboard -c '/usr/local/bin/bundle exec rake RAILS_ENV=production db:raw:optimize' </pre> #* And make it executable: <pre> chmod 755 /usr/local/etc/periodic/monthly/clean_dashboard_database.sh </pre> * And create a weekly script: mkdir -p /usr/local/etc/periodic/weekly vi /usr/local/etc/periodic/weekly/clean_puppet_reports.sh </pre> #* And add the following: <pre> #!/bin/sh echo "Pruning Puppetmaster Reports greater than 7 days old" echo -n " Reports Removed:" find /var/puppet/reports -mtime 7 | xargs rm -v | wc -l </pre> #* And make it executable: <pre> chmod 755 /usr/local/etc/periodic/weekly/clean_puppet_reports.sh </pre> h1. Install Nagios * Start by installing nagios, php, and fcgi: <pre> pkg install nagios nagios-plugins nrpe php5 spawn-fcgi fcgiwrap samba36-smbclient p5-DBI p5-DBD-mysql </pre> * And then enable the service to start at boot: <pre> echo 'nagios_enable="YES"' >> /etc/rc.conf echo 'spawn_fcgi_enable="YES"' >> /etc/rc.conf echo 'fcgiwrap_enable="YES"' >> /etc/rc.conf echo 'fcgiwrap_user="www"' >> /etc/rc.conf echo 'nrpe2_enable="YES"' >> /etc/rc.conf </pre> * Now copy the sample config files to real config files <pre> cd /usr/local/etc/nagios/ cp cgi.cfg-sample cgi.cfg cp nagios.cfg-sample nagios.cfg cp resource.cfg-sample resource.cfg cd /usr/local/etc/nagios/objects/ cp commands.cfg-sample commands.cfg cp contacts.cfg-sample contacts.cfg cp localhost.cfg-sample localhost.cfg cp printer.cfg-sample printer.cfg cp switch.cfg-sample switch.cfg cp templates.cfg-sample templates.cfg cp timeperiods.cfg-sample timeperiods.cfg </pre> * Now check you nagios configurations errors <pre> nagios -v /usr/local/etc/nagios/nagios.cfg </pre> * Now set a password for the web interface <pre> htpasswd -c /usr/local/etc/nagios/htpasswd.users nagiosadmin </pre> * Next, create the @/usr/local/etc/nginx/conf.d/nagios.conf@ file: <pre> vi /usr/local/etc/nginx/conf.d/nagios.conf </pre> #* And add the following to @/usr/local/etc/nginx/nginx.conf@: <pre> server { listen 8000 default; server nagios.example.com; index index.html index.php; root /usr/local/www/nagios; # IP and IP ranges which should get access allow 10.0.0.0/24; allow 10.1.0.1; # all else will be denied deny all; # basic HTTP auth auth_basic "Restricted"; auth_basic_user_file htpasswd; location ~ \.cgi$ { try_files $uri =404; include fastcgi_params; fastcgi_pass unix:/var/run/fcgiwrap/fcgiwrap.sock; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param REMOTE_USER $remote_user; } location ~ \.php$ { try_files $uri =404; include fastcgi_params; fastcgi_pass 127.0.0.1:9000; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; } } </pre> * Load the schema <pre> cd /usr/local/share/doc/ndoutils perl ./installdb -u nagios-p SecretNagiosPassword -h localhost -d nagios </pre> #* Example output: <pre> DBD::mysql::db do failed: Table ‘nagios.nagios_dbversion’ doesn’t exist at ./installdb line 51. ** Creating tables for version 1.4b9 Using mysql.sql for installation… ** Updating table nagios_dbversion Done! </pre> * Configure the @/usr/local/etc/nagios/nagios.cfg@ file and change the following parameter: <pre> event_broker_options=-1 </pre> * Add the following in “EVENT BROKER MODULE(S)” section <pre> broker_module=/usr/local/bin/ndomod.o config_file=/usr/local/etc/nagios/ndomod.cfg </pre> * Configure the @/usr/local/etc/nagios/ndo2db.cfg@ file, and change the below three lines: <pre> db_user=nagios db_pass=SecretNagiosPassword debug_level=-1 </pre> * Start ndo2db <pre> service ndo2db start </pre> * Start Nagios <pre> service nagios restart </pre> * Start nginx, and check to see if nagios is working by opening a web browser and going to http://example.com/nagios: <pre> service nginx start </pre> NOTE: I kept getting a permission problem when logging in to the Nagios interface. It turned out I needed to add the correct admin user in @/usr/local/etc/nagios/cgi.cfg@: <pre> use_authentication=1 #edit username authorized_for_all_host_commands=username authorized_for_all_hosts=username authorized_for_all_service_commands=username authorized_for_all_services=username authorized_for_configuration_information=username authorized_for_system_commands=username authorized_for_system_information=username </pre> h2. Resources * http://docs.puppetlabs.com/dashboard/manual/1.2/bootstrapping.html#configuring-dashboard * http://docs.puppetlabs.com/dashboard/manual/1.2/bootstrapping.html#installing-puppet-dashboard * http://docs.puppetlabs.com/dashboard/manual/1.2/bootstrapping.html#creating-and-configuring-a-mysql-database * http://docs.puppetlabs.com/dashboard/manual/1.2/bootstrapping.html#testing-that-dashboard-is-working http://docs.puppetlabs.com/dashboard/manual/1.2/bootstrapping.html#configuring-puppet * http://docs.puppetlabs.com/dashboard/manual/1.2/bootstrapping.html#starting-and-managing-delayed-job-workers * http://docs.puppetlabs.com/dashboard/manual/1.2/bootstrapping.html#running-dashboard-in-a-production-quality-server * http://z0mbix.github.io/blog/2012/03/01/use-nginx-and-passenger-to-power-your-puppet-master/ * http://www.watters.ws/mediawiki/index.php/Configure_puppet_master_using_nginx_and_mod_passenger * http://docs.puppetlabs.com/dashboard/manual/1.2/configuring.html * https://forums.freebsd.org/viewtopic.php?&t=42071 * http://projects.puppetlabs.com/issues/16686 * http://rlaskey.org/words/897/nagios-nginx-freebsd/ * http://www.unixmen.com/it-appears-as-though-you-do-not-have-permission-to-view-information-for-any-of-the-hosts-you-requested/