Feature #453
Updated by Daniel Curtis over 10 years ago
One of the features offered by Puppet is the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to the web server environment don’t get applied to machines that don’t need that configuration. There are two servers that will be used in this guide: # puppet.example.com: the puppet master server # git.example.com: the GitLab server NOTE: This will not manage the puppet master configurations directly, but rather the environments associated with each branch of the mail git repository. h1. Prepare the Local Computer environments * Log into the GitLab web interface and create a new +_Project_+ for the web service puppetry to be stored #* This example uses +_Web Service Puppetry_+ * Create an repository to hold the configurations: <pre> mkdir -p ~/git/web-service-puppetry && cd ~/git/web-service-puppetry </pre> * Initialize the repository: <pre> git init </pre> * Create a README, add it to the local repository, then commit the local changes to be pushed to the remote git server <pre> touch README git add README git commit -m 'first commit' </pre> * Add the remote git server location to the local repository <pre> git remote add origin ssh://git@git.altservice.com/SecretUser/web-service-puppetry.git </pre> * Push the first commit to the remote git server <pre> git push -u origin master </pre> Mapping the Puppet code base against the environments shows the power of this method. People often use a version control system to manage the code, and create a set of branches that each map to an environment. h2. Creating and using the environments * Show the current branch <pre> git branch </pre> > * master h3. Create the environment branches +PRODUCTION BRANCH+ * Create the *production* environment branch: <pre> git checkout -b production </pre> #* Will result in: <pre> Switched to a new branch 'production' </pre> * Populate a bare @site.pp@ for the *production* environment <pre> mvim site.pp git add site.pp git commit -m 'Created the initial site.pp for the production environment' </pre> #* Will result in: <pre> [production 25a9e1b] Created the initial site.pp for the production environment 1 files changed, 1 insertions(+), 0 deletions(-) </pre> NOTE: I migrated my existing puppet master configurations into to *production* branch, and then used these files for the *testing* and *development* later: <pre> git add --all git commit -m 'Migrated existing puppet master configuration' </pre> * Push the changes to the git server <pre> git push origin production </pre> #* Will result in: <pre> Counting objects: 5, done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. remote: Creating new environment production To git@git.host:deploy.git * [new branch] production -> production </pre> +DEVELOPMENT BRANCH+ * Create the *development* environment branch: <pre> git checkout -b development </pre> #* Will result in: <pre> Switched to a new branch 'development' </pre> * Populate a bare @site.pp@ for the *development* environment <pre> mvim site.pp git add site.pp git commit -m 'Created the initial site.pp for the development environment' </pre> #* Will result in: <pre> [development 25a9e1b] Created the initial site.pp for the development environment 1 files changed, 1 insertions(+), 0 deletions(-) </pre> * Push the changes to the git server <pre> git push origin development </pre> #* Will result in: <pre> Counting objects: 5, done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. remote: Creating new environment development To git@git.host:deploy.git * [new branch] development -> development </pre> +TESTING BRANCH+ * Create the *testing* environment branch: <pre> git checkout -b testing </pre> #* Will result in: <pre> Switched to a new branch 'testing' </pre> * Populate a bare @site.pp@ for the *testing* environment <pre> mvim site.pp git add site.pp git commit -m 'Created the initial site.pp for the testing environment' </pre> #* Will result in: <pre> [testing 25a9e1b] Created the initial site.pp for the testing environment 1 files changed, 1 insertions(+), 0 deletions(-) </pre> * Push the changes to the git server <pre> git push origin testing </pre> #* Will result in: <pre> Counting objects: 5, done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. remote: Creating new environment development To git@git.host:deploy.git * [new branch] testing -> testing </pre> * Now check to see the environment branches: <pre> git branch </pre> #* Will result in: <pre> development master production * testing </pre> * To switch between each branch just run, adjusting accordingly: <pre> git checkout production </pre> And from here on out, you can use the @production@ environment on your hosts, and use git like you would with any code base. This development model gives us some simple access control. Utilizing access control with a tool like GitLab, we can allow people to generate new environments to test their own code, but deny them access to change the production environment. This allows us to institute some sort of change control, by requiring all code to be reviewed by a merge master before inclusion into production, and allows code to be tested and verified before the request for submission is made. h2. Prepare the Environments on the Puppet Master Server * Create the environments directory on the puppet master server: mkdir /usr/local/etc/puppet/environments * Adopting the development, testing and production workflow, we can have a puppet.conf that looks something like this: <pre> vi /usr/local/etc/puppet.conf </pre> #* And should look something like the following <pre> [main] server = puppet.example.com environment = production confdir = /usr/local/etc/puppet [agent] report = true show_diff = true [production] manifest = /usr/local/etc/puppet/environments/production/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/production/modules [testing] manifest = /usr/local/etc/puppet/environments/testing/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/testing/modules [development] manifest = /usr/local/etc/puppet/environments/development/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/development/modules </pre> This will set 3 different environment: # *production*: This is for production ready systems, all bugs and changes should be made in testing or development for environments before being applied to production. # *testing*: This is for testing certain features that may be broken or needs more work before entering the production environment. # *development*: This is for developing features, fixing bugs, and/or modifying themes before entering the testing environment. With this configuration, we could map three Git branches for these environments and set up a central Git repository with post receive hooks. When changes were pushed to this repository, they would be automatically deployed to the puppet master. The example post-receive hook later in this post will work with this kind of environment setup. An example will help clarify. Suppose I’m working on a new set of functionality, and don’t want to touch the current set of Puppet modules and inadvertently cause change in production. I can create a testing branch like so: git clone git@git.example.com/SecretUser/web-services-puppetry.git <pre> cd puppet git branch testing git checkout testing </pre> #* ... make some edits <pre> git add * git commit -m "made some edits" git push origin testing </pre> * and then run my Puppet agent against this new testing code: <pre> puppet agent --test --environment=testing </pre> > Info: Retrieving plugin > Info: Caching catalog for sun.local > Info: Applying configuration version '1371740640' As you can see, I set a default environment of production, and then specify paths to the manifest and modulepath directories, using the $environment variable to dynamically populate the path. Production manifest and modulepath paths will end up being $confdir/environments/production/manifests/site.pp and $confdir/environments/production/modules respectively. As new environments are dynamically created, the $environment variable will be substituted as appropriate. * Next, I moved my existing Puppet module and manifest structure around to suit the new configuration: <pre> mkdir -p /usr/local/etc/puppet/environments/production mv /usr/local/etc/puppet/manifests /usr/local/etc/puppet/modules /usr/local/etc/puppet/environments/production </pre> * And restart nginx (as I run my puppetmaster under Nginx/Passenger): <pre> service nginx restart </pre> * I then ran a couple of agents to ensure everything was still working: <pre> puppet agent --test </pre> They defaulted, as expected, to the Production environment. * Next, install git on the puppet master: <pre> pkg install git </pre> * Next, create a local git repository from the existing Puppet configuration: <pre> cd /usr/local/etc/puppet/environments/production git init git add * git commit -m "Initial import of Production Puppet repository" </pre> * Now add the remote GitLab repository: <pre> git remote add ssh://git@git.example.com/SecretUser/web-services-puppetry.git </pre> * And push the committed initial configuration to the remote GitLab repository: <pre> git push origin production </pre> h3. (Optional) Create Web Server SSH Key Pair Since I am accessing a private repository, my puppet master web server needs to have an ssh key pair. * Generate the key pair <pre> ssh-keygen -t ecdsa /usr/local/www/.ssh/id_ecdsa </pre> * Change the private key owner to the web server: <pre> chown www /usr/local/www/.ssh/id_ecdsa </pre> * Add the following to the GitLab ssh keys allowed access to the private repository: <pre> cat /usr/local/www/.ssh/id_ecdsa.pub </pre> h1. Git Hooks h2. Git Update Hook This will create a script on the puppet master server that will be executed by the GitLab servers Web Hook upon each push update. * Log in to the git server and create a post-receive hook <pre> ssh puppet.example.com sudo -s </pre> h3. Puppet Post Receive Update Hook * First install bash <pre> pkg install bash </pre> * Create a post-receive hook file: <pre> vi /usr/local/share/puppet-dashboard/public/post-receive </pre> #* To update puppet configurations on the puppet master server on each git push: <pre> #!/usr/local/bin/bash read oldrev newrev refname REPOSITORY="ssh://git@git.example.com/SecretUser/web-services-puppetry.git #!/usr/local/bin/env ruby BRANCH=$( echo "${refname}" | sed -n 's!^refs/heads/!!p' ) ENVIRONMENT_BASE="/usr/local/etc/puppet/environments" # master branch, as defined by git, Puppet Labs is production a ruby shop, so why not do the post-receive hook in ruby? if [[ "${BRANCH}" == "master" ]]; then ACTUAL_BRANCH="production" require 'fileutils' # Set this to where you want to keep your environments else ACTUAL_BRANCH=${BRANCH} ENVIRONMENT_BASEDIR = "/usr/local/etc/puppet/environments" # post-receive hooks set GIT_DIR to the current repository. If you want to fi # newrev is clone from a bunch non-local repository, set this to the URL of 0s the repository, echo "${newrev}" | grep -qs '^0*$' # such as git@git.host:puppet.git if [ "$?" -eq "0" ]; then #SOURCE_REPOSITORY = File.expand_path(ENV['GIT_DIR']) SOURCE_REPOSITORY = File.expand_path(ENV['git.example.com/SecretUser/web-service-puppetry.git']) # The git_dir environment variable will override the --git-dir, so we remove it # to allow us to create new repositories cleanly. ENV.delete('GIT_DIR') # Ensure that we have the underlying directories, otherwise the later commands # may fail in somewhat cryptic manners. unless File.directory? ENVIRONMENT_BASEDIR puts %Q{#{ENVIRONMENT_BASEDIR} does not exist, cannot create environment directories.} exit 1 end # You can push multiple refspecs at once, like 'git push origin branch1 branch2', # so we need to handle each one. $stdin.each_line do |line| oldrev, newrev, refname = line.split(" ") # Determine the branch name from the refspec we're received, which is marked for deletion in the # format refs/heads/, and make sure that it doesn't have any possibly # dangerous characters branchname = refname.sub(%r{^refs/heads/(.*$)}) { $1 } if branchname =~ /[\W-]/ puts %Q{Branch "#{branchname}" contains non-word characters, ignoring it.} next end environment_path = "#{ENVIRONMENT_BASEDIR}/#{branchname}" if [ "${ACTUAL_BRANCH}" = "production" ]; then echo "No way!" exit 1 newrev =~ /^0+$/ fi # We've received a push with a null revision, something like 000000000000, echo # which means that we should delete the given branch. puts "Deleting remote branch ${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}" existing environment #{branchname}" ssh-agent bash -c "ssh-add /usr/local/www/.ssh/id_ecdsa; cd ${ENVIRONMENT_BASE}; rm -rf ${ACTUAL_BRANCH}" if File.directory? environment_path FileUtils.rm_rf environment_path, :secure => true end else echo "Updating remote # We have been given a branch ${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}" that needs to be created or updated. If the # environment exists, update it. Else, create it. if [ -d "${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}" ]; File.directory? environment_path # Update an existing environment. We do a fetch and then ssh-agent bash -c "ssh-add /usr/local/www/.ssh/id_ecdsa; cd ${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}; git reset in the # case that someone did a force push to a branch. puts "Updating existing environment #{branchname}" Dir.chdir environment_path %x{git fetch --all; git --all} %x{git reset --hard origin/${BRANCH}" "origin/#{branchname}"} else ssh-agent bash -c "ssh-add /usr/local/www/.ssh/id_ecdsa; cd ${ENVIRONMENT_BASE}; git # Instantiate a new environment from the current repository. puts "Creating new environment #{branchname}" %x{git clone ${REPOSITORY} ${ACTUAL_BRANCH} #{SOURCE_REPOSITORY} #{environment_path} --branch ${BRANCH}" #{branchname}} fi end end fi exit 0 end </pre> #* NOTE: A simpler bash variation can be accomplished by using the following for the post-receive script: <pre> #!/bin/sh git --work-tree=/usr/local/etc/puppet --git-dir=/var/repo/web-service-puppetry.git checkout -f </pre> h3. Puppet Syntax Checking Hook * Create a update hook file: <pre> vi /usr/local/share/puppet-dashboard/public/update </pre> * To catch syntax errors and other basic problems, you can use a *+server-side+* Git update hook like this: <pre> #!/usr/local/bin/bash #!/bin/bash NOBOLD="\033[0m" BOLD="\033[1m" BLACK="\033[30m" GREY="\033[0m" RED="\033[31m" GREEN="\033[32m" YELLOW="\033[33m" BLUE="\033[34m" MAGENTA="\033[35m" CYAN="\033[36m" WHITE="\033[37m" # V +1007 # Peff helped: # http://thread.gmane.org/gmane.comp.version-control.git/118626 # For Puppet 0.25.x: # syntax_check="puppet --color=false --confdir=/tmp --vardir=/tmp --parseonly --ignoreimport" # # For Puppet 2.7.x: # syntax_check="puppet parser validate --ignoreimport" # # NOTE: There is an outstanding bug against `puppet parser` which causes # the --ignoreimport option to turn the syntax check into a no-op. Until # the bug is resolved, the syntax check hook should not include the # --ignoreimport option and will only work correctly on manifests which # do not contain "import" lines. # See http://projects.puppetlabs.com/issues/9670 # syntax_check="puppet parser validate" tmp=$(mktemp /tmp/git.update.XXXXXX) log=$(mktemp /tmp/git.update.log.XXXXXX) tree=$(mktemp /tmp/git.diff-tree.XXXXXX) git diff-tree -r "$2" "$3" > $tree echo echo diff-tree: cat $tree exit_status=0 while read old_mode new_mode old_sha1 new_sha1 status name do # skip lines showing parent commit test -z "$new_sha1" && continue # skip deletions [ "$new_sha1" = "0000000000000000000000000000000000000000" ] && continue # Only test .pp files if [[ $name =~ [.]pp$ ]] then git cat-file blob $new_sha1 > $tmp set -o pipefail $syntax_check $tmp 2>&1 | sed "s|$tmp|$name|"> $log if [[ $? != 0 ]] then echo echo -e "$(cat $log | sed 's|JOJOMOJO|'\\${RED}${name}\\${NOBOLD}'|')" >&2 echo -e "For more details run this: ${CYAN} git diff $old_sha1 $new_sha1 ${NOBOLD}" >&2 echo exit_status=1 fi fi done < $tree rm -f $log $tmp $tree exit $exit_status </pre> * Make the post-receive hook executable <pre> chmod +x /usr/local/share/puppet-dashboard/public/post-receive </pre> * Make the update hook executable <pre> chmod +x /usr/local/share/puppet-dashboard/public/update </pre> * Now restart nginx: <pre> service nginx restart </pre> h3. Git Pre-Commit Hook (Puppet Master Side) NOTE: This is not used in this guide, but included for posterity. * To catch syntax errors and other basic problems, you can use a *+client-side+* Git pre-commit hook like this (useful for cron scripts): <pre> #!/bin/sh syntax_errors=0 error_msg=$(mktemp /tmp/error_msg.XXXXXX) if git rev-parse --quiet --verify HEAD > /dev/null then against=HEAD else # Initial commit: diff against an empty tree object against=4b825dc642cb6eb9a060e54bf8d69288fbee4904 fi # Get list of new/modified manifest and template files to check (in git index) for indexfile in `git diff-index --diff-filter=AM --name-only --cached $against | egrep '\.(pp|erb)'` do # Don't check empty files if [ `git cat-file -s :0:$indexfile` -gt 0 ] then case $indexfile in *.pp ) # Check puppet manifest syntax #git cat-file blob :0:$indexfile | puppet --color=false --parseonly --ignoreimport > $error_msg ;; # Updated for 2.7.x puppet parser validate $indexfile > $error_msg ;; *.erb ) # Check ERB template syntax # -P : ignore lines which start with "%" git cat-file blob :0:$indexfile | erb -P -x -T - | ruby -c 2> $error_msg > /dev/null ;; esac if [ "$?" -ne 0 ] then echo -n "$indexfile: " cat $error_msg syntax_errors=`expr $syntax_errors + 1` fi fi done rm -f $error_msg if [ "$syntax_errors" -ne 0 ] then echo "Error: $syntax_errors syntax errors found, aborting commit." exit 1 fi </pre> *NOTE*: Make sure that the post receive hook is executable! h2. Adding the Web Hook To GitLab This part is where the magic happens. # Log into GitLab and go to the +_Project_+ created to manage the puppet configurations. # Next go to the projects +_Settings -> Web Hooks_+. # Enter in the puppet master server and location of the post-receive script, example: *http://puppet.example.com/post-receive* h2. Resources * http://puppetlabs.com/blog/git-workflow-and-puppet-environments * https://groups.google.com/forum/#!topic/gitlabhq/WWUox4MLe2U * https://www.digitalocean.com/community/tutorials/how-to-set-up-automatic-deployment-with-git-with-a-vps * http://dreamingwell.com/articles/archives/2014/02/using-git-and-puppet-for-agile-server-management.php * http://krisjordan.com/essays/setting-up-push-to-deploy-with-git * http://projects.puppetlabs.com/projects/1/wiki/puppet_version_control