Feature #453
Updated by Daniel Curtis over 10 years ago
One of the features offered by Puppet is the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to the web server environment don’t get applied to machines that don’t need that configuration. h2. Prepare the Local Computer environments * Create an environment to hold the configurations: <pre> mkdir -p ~/git/web-service-puppetry/environments/ </pre> Mapping the Puppet code base against the environments shows the power of this method. People often use a version control system to manage the code, and create a set of branches that each map to an environment. h2. Prepare the Environments on the Puppet Master Server * Adopting the development, testing and production workflow, we can have a puppet.conf that looks something like this: <pre> vi /usr/local/etc/puppet.conf </pre> #* And should look something like the following <pre> [main] server = puppet.example.com environment = production confdir = /usr/local/etc/puppet /etc/puppet [agent] report = true show_diff = true [production] manifest = /usr/local/etc/puppet/environments/production/manifests/site.pp /etc/puppet/environments/production/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/production/modules /etc/puppet/environments/production/modules [testing] manifest = /usr/local/etc/puppet/environments/testing/manifests/site.pp /etc/puppet/environments/testing/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/testing/modules /etc/puppet/environments/testing/modules [development] manifest = /usr/local/etc/puppet/environments/development/manifests/site.pp /etc/puppet/environments/development/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/development/modules /etc/puppet/environments/development/modules </pre> This will set 3 different environment: # *production*: production*: This is for production ready systems, all bugs and changes should be made in testing or development for environments before being applied to production. # *testing*: testing*: This is for testing certain features that may be broken or needs more work before entering the production environment. # *development*: # development*: This is for developing features, fixing bugs, and/or modifying themes before entering the testing environment. With this configuration, we could map three Git branches for these environments and set up a central Git repository with post receive hooks. When changes were pushed to this repository, they would be automatically deployed to the puppet master. The example post-receive hook later in this post will work with this kind of environment setup. h1. Git Hooks h2. Git Update Hook Dynamic Environments While there are benefits to having a set of branches and environments like the previously outlined configuration, there are also drawbacks with a static set of environments. This will create model is somewhat constrained by enforcing a script single workflow. For instance, if multiple people are working on experimental features in the puppet master server that will be executed development branch, they'll have to constantly merge their code through the development process. If multiple people are developing different features in the same branch, they’ll all have to work with unrelated changes in the code. In addition, migrating a single feature to testing or production gets more complicated if multiple features are in the same branch. Modern distributed version control systems like Git handle these constraints by making branch creation and merging lightweight operations, allowing us to generate Puppet environments on the GitLab servers Web Hook upon each push update. * Log fly. Puppet can set up environments with an explicitly defined section in the configuration file, but we can exploit the fact Puppet will set the @$environment@ variable to the git server and create name of environment it is currently running under. * With this in mind, we can write a post-receive hook puppet.conf to resemble this: <pre> ssh vi /usr/local/etc/puppet/puppet.conf </pre> #* And modify it similar to the following: <pre> [main] server = puppet.example.com environment = production confdir = /etc/puppet sudo -s [master] environment = production manifest = $confdir/environments/$environment/manifests/site.pp modulepath = $confdir/environments/$environment/modules [agent] report = true show_diff = true environment = production </pre> h3. Puppet Post Receive Update Hook * Create This handles the dynamic environment aspect; all we have to do is create a post-receive hook file: directory with manifests in @$confdir/environments@ and we have created that environment: <pre> vi /usr/local/share/puppet-dashboard/public/post-receive mkdir -p /usr/local/etc/puppet/environments mkdir /usr/local/etc/puppet/environments/production mkdir /usr/local/etc/puppet/environments/testing mkdir /usr/local/etc/puppet/environments/development </pre> #* To update puppet configurations * Generating new environments using Git is similarly easy. We can create a central Git repository on the puppet master server on each git push: GitLab instance, with a @post-receive@ hook that looks something like this: <pre> <pre #!/usr/local/bin/env #!/usr/bin/env ruby # Puppet Labs is a ruby shop, so why not do the post-receive hook in ruby? require 'fileutils' # Set this to where you want to keep your environments ENVIRONMENT_BASEDIR = "/usr/local/etc/puppet/environments" # post-receive hooks set GIT_DIR to the current repository. If you want to # clone from a non-local repository, set this to the URL of the repository, # such as git@git.host:puppet.git SOURCE_REPOSITORY = File.expand_path(ENV['GIT_DIR']) # The git_dir environment variable will override the --git-dir, so we remove it # to allow us to create new repositories cleanly. ENV.delete('GIT_DIR') # Ensure that we have the underlying directories, otherwise the later commands # may fail in somewhat cryptic manners. unless File.directory? ENVIRONMENT_BASEDIR puts %Q{#{ENVIRONMENT_BASEDIR} does not exist, cannot create environment directories.} exit 1 end # You can push multiple refspecs at once, like 'git push origin branch1 branch2', # so we need to handle each one. $stdin.each_line do |line| oldrev, newrev, refname = line.split(" ") # Determine the branch name from the refspec we're received, which is in the # format refs/heads/, and make sure that it doesn't have any possibly # dangerous characters branchname = refname.sub(%r{^refs/heads/(.*$)}) { $1 } if branchname =~ /[\W-]/ puts %Q{Branch "#{branchname}" contains non-word characters, ignoring it.} next end environment_path = "#{ENVIRONMENT_BASEDIR}/#{branchname}" if newrev =~ /^0+$/ # We've received a push with a null revision, something like 000000000000, # which means that we should delete the given branch. puts "Deleting existing environment #{branchname}" if File.directory? environment_path FileUtils.rm_rf environment_path, :secure => true end else # We have been given a branch that needs to be created or updated. If the # environment exists, update it. Else, create it. if File.directory? environment_path # Update an existing environment. We do a fetch and then reset in the # case that someone did a force push to a branch. puts "Updating existing environment #{branchname}" Dir.chdir environment_path %x{git fetch --all} %x{git reset --hard "origin/#{branchname}"} else # Instantiate a new environment from the current repository. puts "Creating new environment #{branchname}" %x{git clone #{SOURCE_REPOSITORY} #{environment_path} --branch #{branchname}} end end end </pre> h3. Puppet Syntax Checking Hook * Create a update hook file: <pre> vi /usr/local/share/puppet-dashboard/public/update </pre> * To catch syntax errors and other basic problems, you can use a *+server-side+* Git update hook like this: <pre> #!/bin/bash NOBOLD="\033[0m" BOLD="\033[1m" BLACK="\033[30m" GREY="\033[0m" RED="\033[31m" GREEN="\033[32m" YELLOW="\033[33m" BLUE="\033[34m" MAGENTA="\033[35m" CYAN="\033[36m" WHITE="\033[37m" # V +1007 # Peff helped: # http://thread.gmane.org/gmane.comp.version-control.git/118626 # For Puppet 0.25.x: # syntax_check="puppet --color=false --confdir=/tmp --vardir=/tmp --parseonly --ignoreimport" # # For Puppet 2.7.x: # syntax_check="puppet parser validate --ignoreimport" # # NOTE: There is an outstanding bug against `puppet parser` which causes # the --ignoreimport option to turn the syntax check into a no-op. Until # the bug is resolved, the syntax check hook should not include the # --ignoreimport option and will only work correctly on manifests which # do not contain "import" lines. # See http://projects.puppetlabs.com/issues/9670 # syntax_check="puppet parser validate" tmp=$(mktemp /tmp/git.update.XXXXXX) log=$(mktemp /tmp/git.update.log.XXXXXX) tree=$(mktemp /tmp/git.diff-tree.XXXXXX) git diff-tree -r "$2" "$3" > $tree echo echo diff-tree: cat $tree exit_status=0 while read old_mode new_mode old_sha1 new_sha1 status name do # skip lines showing parent commit test -z "$new_sha1" && continue # skip deletions [ "$new_sha1" = "0000000000000000000000000000000000000000" ] && continue # Only test .pp files if [[ $name =~ [.]pp$ ]] then git cat-file blob $new_sha1 > $tmp set -o pipefail $syntax_check $tmp 2>&1 | sed "s|$tmp|$name|"> $log if [[ $? != 0 ]] then echo echo -e "$(cat $log | sed 's|JOJOMOJO|'\\${RED}${name}\\${NOBOLD}'|')" >&2 echo -e "For more details run this: ${CYAN} git diff $old_sha1 $new_sha1 ${NOBOLD}" >&2 echo exit_status=1 fi fi done < $tree rm -f $log $tmp $tree exit $exit_status </pre> * Make the post-receive hook executable <pre> chmod +x /usr/local/share/puppet-dashboard/public/post-receive </pre> * Make the update hook executable <pre> chmod +x /usr/local/share/puppet-dashboard/public/update </pre> * Now restart nginx: <pre> service nginx restart </pre> h3. Git Pre-Commit Hook (Puppet Master Side) NOTE: This is not used in this guide, but included for posterity. * To catch syntax errors and other basic problems, you can use a *+client-side+* Git pre-commit hook like this (useful for cron scripts): <pre> #!/bin/sh syntax_errors=0 error_msg=$(mktemp /tmp/error_msg.XXXXXX) if git rev-parse --quiet --verify HEAD > /dev/null then against=HEAD else # Initial commit: diff against an empty tree object against=4b825dc642cb6eb9a060e54bf8d69288fbee4904 fi # Get list of new/modified manifest and template files to check (in git index) for indexfile in `git diff-index --diff-filter=AM --name-only --cached $against | egrep '\.(pp|erb)'` do # Don't check empty files if [ `git cat-file -s :0:$indexfile` -gt 0 ] then case $indexfile in *.pp ) # Check puppet manifest syntax #git cat-file blob :0:$indexfile | puppet --color=false --parseonly --ignoreimport > $error_msg ;; # Updated for 2.7.x puppet parser validate $indexfile > $error_msg ;; *.erb ) # Check ERB template syntax # -P : ignore lines which start with "%" git cat-file blob :0:$indexfile | erb -P -x -T - | ruby -c 2> $error_msg > /dev/null ;; esac if [ "$?" -ne 0 ] then echo -n "$indexfile: " cat $error_msg syntax_errors=`expr $syntax_errors + 1` fi fi done rm -f $error_msg if [ "$syntax_errors" -ne 0 ] then echo "Error: $syntax_errors syntax errors found, aborting commit." exit 1 fi </pre> *NOTE*: Make sure that the post receive hook is executable! h3. Creating and using the environments a new environment <pre> git branch </pre> > * production <pre> git checkout -b development new_feature </pre> > Switched to a new branch 'development' 'new_feature' <pre> mvim site.pp git add site.pp git commit -m 'Implemented a newly developed my feature' </pre> > [new_feature 25a9e1b] Implemented a newly developed my feature > 1 files changed, 1 insertions(+), 0 deletions(-) <pre> git push origin development new_feature </pre> > Counting objects: 5, done. > Writing objects: 100% (3/3), 255 bytes, done. > Total 3 (delta 0), reused 0 (delta 0) > Unpacking objects: 100% (3/3), done. > remote: Creating new environment development new_feature > To git@git.host:deploy.git > * [new branch] development new_feature -> development new_feature And from here on out, you can use the @development@ @new_feature@ environment on your hosts, and use git like you would with any code base. This development model gives us some simple access control. Utilizing access control with a tool like GitLab, we can allow people to generate new environments to test their own code, but deny them access to change the production environment. This allows us to institute some sort of change control, by requiring all code to be reviewed by a merge master before inclusion into production, and allows code to be tested and verified before the request for submission is made. h2. Adding the Web Hook To GitLab This part is where the magic happens. # Log into GitLab and go to the +_Project_+ created to manage the puppet configurations. # Next go to the projects +_Settings -> Web Hooks_+. # Enter in the puppet master server and location of the post-receive script, example: *http://puppet.example.com/post-receive* h2. Resources * http://puppetlabs.com/blog/git-workflow-and-puppet-environments * https://groups.google.com/forum/#!topic/gitlabhq/WWUox4MLe2U * https://www.digitalocean.com/community/tutorials/how-to-set-up-automatic-deployment-with-git-with-a-vps * http://dreamingwell.com/articles/archives/2014/02/using-git-and-puppet-for-agile-server-management.php * http://krisjordan.com/essays/setting-up-push-to-deploy-with-git * http://projects.puppetlabs.com/projects/1/wiki/puppet_version_control