Feature #453
Updated by Daniel Curtis over 9 years ago
One of the features offered by Puppet is the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to the web server environment don’t get applied to machines that don’t need that configuration. There are two servers that will be used in this guide: # puppet.example.com: the puppet master server # git.example.com: the GitLab server NOTE: This will not manage the puppet master configurations directly, but rather the environments associated with each branch of the mail git repository. h1. Prepare the Local Computer environments * Log into the GitLab web interface and create a new +_Project_+ for the web service puppetry to be stored #* This example uses +_Web Service Puppetry_+ * Create an repository to hold the configurations: <pre> mkdir -p ~/git/web-service-puppetry && cd ~/git/web-service-puppetry </pre> * Initialize the repository: <pre> git init </pre> * Create a README, add it to the local repository, then commit the local changes to be pushed to the remote git server <pre> touch README git add README git commit -m 'first commit' </pre> * Add the remote git server location to the local repository <pre> git remote add origin ssh://git@git.altservice.com/SecretUser/web-service-puppetry.git </pre> * Push the first commit to the remote git server <pre> git push -u origin master </pre> Mapping the Puppet code base against the environments shows the power of this method. People often use a version control system to manage the code, and create a set of branches that each map to an environment. h2. Creating and using the environments * Show the current branch <pre> git branch </pre> > * master h3. Create the environment branches +PRODUCTION BRANCH+ * Create the *production* environment branch: <pre> git checkout -b production </pre> #* Will result in: <pre> Switched to a new branch 'production' </pre> * Populate a bare @site.pp@ for the *production* environment <pre> mvim site.pp git add site.pp git commit -m 'Created the initial site.pp for the production environment' </pre> #* Will result in: <pre> [production 25a9e1b] Created the initial site.pp for the production environment 1 files changed, 1 insertions(+), 0 deletions(-) </pre> NOTE: I migrated my existing puppet master configurations into to *production* branch, and then used these files for the *testing* and *development* later: <pre> git add --all git commit -m 'Migrated existing puppet master configuration' </pre> * Push the changes to the git server <pre> git push origin production </pre> #* Will result in: <pre> Counting objects: 5, done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. remote: Creating new environment production To git@git.host:deploy.git * [new branch] production -> production </pre> +DEVELOPMENT BRANCH+ * Create the *development* environment branch: <pre> git checkout -b development </pre> #* Will result in: <pre> Switched to a new branch 'development' </pre> * Populate a bare @site.pp@ for the *development* environment <pre> mvim site.pp git add site.pp git commit -m 'Created the initial site.pp for the development environment' </pre> #* Will result in: <pre> [development 25a9e1b] Created the initial site.pp for the development environment 1 files changed, 1 insertions(+), 0 deletions(-) </pre> * Push the changes to the git server <pre> git push origin development </pre> #* Will result in: <pre> Counting objects: 5, done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. remote: Creating new environment development To git@git.host:deploy.git * [new branch] development -> development </pre> +TESTING BRANCH+ * Create the *testing* environment branch: <pre> git checkout -b testing </pre> #* Will result in: <pre> Switched to a new branch 'testing' </pre> * Populate a bare @site.pp@ for the *testing* environment <pre> mvim site.pp git add site.pp git commit -m 'Created the initial site.pp for the testing environment' </pre> #* Will result in: <pre> [testing 25a9e1b] Created the initial site.pp for the testing environment 1 files changed, 1 insertions(+), 0 deletions(-) </pre> * Push the changes to the git server <pre> git push origin testing </pre> #* Will result in: <pre> Counting objects: 5, done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. remote: Creating new environment development To git@git.host:deploy.git * [new branch] testing -> testing </pre> * Now check to see the environment branches: <pre> git branch </pre> #* Will result in: <pre> development master production * testing </pre> * To switch between each branch just run, adjusting accordingly: <pre> git checkout production </pre> And from here on out, you can use the @production@ environment on your hosts, and use git like you would with any code base. This development model gives us some simple access control. Utilizing access control with a tool like GitLab, we can allow people to generate new environments to test their own code, but deny them access to change the production environment. This allows us to institute some sort of change control, by requiring all code to be reviewed by a merge master before inclusion into production, and allows code to be tested and verified before the request for submission is made. h2. Prepare the Environments on the Puppet Master Server * Create the environments directory on the puppet master server: mkdir /usr/local/etc/puppet/environments * Adopting the development, testing and production workflow, we can have a puppet.conf that looks something like this: <pre> vi /usr/local/etc/puppet.conf </pre> #* And should look something like the following <pre> [main] server = puppet.example.com environment = production confdir = /usr/local/etc/puppet [agent] report = true show_diff = true [production] manifest = /usr/local/etc/puppet/environments/production/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/production/modules [testing] manifest = /usr/local/etc/puppet/environments/testing/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/testing/modules [development] manifest = /usr/local/etc/puppet/environments/development/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/development/modules </pre> This will set 3 different environment: # *production*: This is for production ready systems, all bugs and changes should be made in testing or development for environments before being applied to production. # *testing*: This is for testing certain features that may be broken or needs more work before entering the production environment. # *development*: This is for developing features, fixing bugs, and/or modifying themes before entering the testing environment. With this configuration, we could map three Git branches for these environments and set up a central Git repository with post receive hooks. When changes were pushed to this repository, they would be automatically deployed to the puppet master. The example post-receive hook later in this post will work with this kind of environment setup. An example will help clarify. Suppose I’m working on a new set of functionality, and don’t want to touch the current set of Puppet modules and inadvertently cause change in production. I can create a testing branch like so: git clone git@git.example.com/SecretUser/web-services-puppetry.git <pre> cd puppet git branch testing git checkout testing </pre> #* ... make some edits <pre> git add * git commit -m "made some edits" git push origin testing </pre> * and then run my Puppet agent against this new testing code: <pre> puppet agent --test --environment=testing </pre> > Info: Retrieving plugin > Info: Caching catalog for sun.local > Info: Applying configuration version '1371740640' As you can see, I set a default environment of production, and then specify paths to the manifest and modulepath directories, using the $environment variable to dynamically populate the path. Production manifest and modulepath paths will end up being $confdir/environments/production/manifests/site.pp and $confdir/environments/production/modules respectively. As new environments are dynamically created, the $environment variable will be substituted as appropriate. * Next, I moved my existing Puppet module and manifest structure around to suit the new configuration: <pre> mkdir -p /usr/local/etc/puppet/environments/production mv /usr/local/etc/puppet/manifests /usr/local/etc/puppet/modules /usr/local/etc/puppet/environments/production </pre> * And restart nginx (as I run my puppetmaster under Nginx/Passenger): <pre> service nginx restart </pre> * I then ran a couple of agents to ensure everything was still working: <pre> puppet agent --test </pre> They defaulted, as expected, to the Production environment. * Next, install git on the puppet master: <pre> pkg install git </pre> * Next, create a local git repository from the existing Puppet configuration: <pre> cd /usr/local/etc/puppet/environments/production git init git add * git commit -m "Initial import of Production Puppet repository" </pre> * Now add the remote GitLab repository: <pre> git remote add ssh://git@git.example.com/SecretUser/web-services-puppetry.git </pre> * And push the committed initial configuration to the remote GitLab repository: <pre> git push origin production </pre> h2. Continuous Integration of GitLab and Puppet Master h3. (Optional) Create Web Server SSH Key Pair After many hours of trial and error, and still not being able to get Since I am accessing a functioning post-receive script private repository, my puppet master web server needs to work, I decided have an ssh key pair. * Generate the key pair <pre> ssh-keygen -t ecdsa /usr/local/www/.ssh/id_ecdsa </pre> * Change the private key owner to modify the above steps web server: <pre> chown www /usr/local/www/.ssh/id_ecdsa </pre> * Add the following to have the GitLab ssh keys allowed access to the private repository: <pre> cat /usr/local/www/.ssh/id_ecdsa.pub </pre> h1. Git Hooks h2. Git Update Hook This will create a script on the puppet master server that will be executed by the GitLab servers Web Hook upon each environment have its own branch push update. * Log in to the web services repo. git server and create a post-receive hook <pre> ssh puppet.example.com sudo -s </pre> h3. Puppet Post Receive Update Hook * With First install bash <pre> pkg install bash </pre> * Create a post-receive hook file: <pre> vi /usr/local/share/puppet-dashboard/public/post-receive </pre> #* To update puppet configurations on the puppet master server on each git push: <pre> #!/usr/local/bin/bash read oldrev newrev refname REPOSITORY="ssh://git@git.example.com/SecretUser/web-services-puppetry.git BRANCH=$( echo "${refname}" | sed -n 's!^refs/heads/!!p' ) ENVIRONMENT_BASE="/usr/local/etc/puppet/environments" # master branch, as defined by git, is production environment being continually updated using if [[ "${BRANCH}" == "master" ]]; then ACTUAL_BRANCH="production" else ACTUAL_BRANCH=${BRANCH} fi # newrev is a cron script: bunch of 0s echo "${newrev}" | grep -qs '^0*$' if [ "$?" -eq "0" ]; then # branch is marked for deletion if [ "${ACTUAL_BRANCH}" = "production" ]; then echo "No way!" exit 1 fi echo "Deleting remote branch ${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}" ssh-agent bash -c "ssh-add /usr/local/www/.ssh/id_ecdsa; cd ${ENVIRONMENT_BASE}; rm -rf ${ACTUAL_BRANCH}" else echo "Updating remote branch ${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}" if [ -d "${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}" ]; then ssh-agent bash -c "ssh-add /usr/local/www/.ssh/id_ecdsa; cd ${ENVIRONMENT_BASE}/${ACTUAL_BRANCH}; git fetch --all; git reset --hard origin/${BRANCH}" else ssh-agent bash -c "ssh-add /usr/local/www/.ssh/id_ecdsa; cd ${ENVIRONMENT_BASE}; git clone ${REPOSITORY} ${ACTUAL_BRANCH} --branch ${BRANCH}" fi fi exit 0 </pre> h3. Puppet Syntax Checking Hook * Create a update hook file: <pre> vi /etc/crontab /usr/local/share/puppet-dashboard/public/update </pre> * And then added To catch syntax errors and other basic problems, you can use a *+server-side+* Git update hook like this: <pre> #!/usr/local/bin/bash NOBOLD="\033[0m" BOLD="\033[1m" BLACK="\033[30m" GREY="\033[0m" RED="\033[31m" GREEN="\033[32m" YELLOW="\033[33m" BLUE="\033[34m" MAGENTA="\033[35m" CYAN="\033[36m" WHITE="\033[37m" # V +1007 # Peff helped: # http://thread.gmane.org/gmane.comp.version-control.git/118626 # For Puppet 0.25.x: # syntax_check="puppet --color=false --confdir=/tmp --vardir=/tmp --parseonly --ignoreimport" # # For Puppet 2.7.x: # syntax_check="puppet parser validate --ignoreimport" # # NOTE: There is an outstanding bug against `puppet parser` which causes # the following --ignoreimport option to turn the end: syntax check into a no-op. Until <pre> # the bug is resolved, the syntax check hook should not include the ## Pull latest puppetry from GitLab # --ignoreimport option and will only work correctly on manifests which */2 * # * do not contain "import" lines. # * * root cd /usr/local/etc/puppet/environments/production See http://projects.puppetlabs.com/issues/9670 # syntax_check="puppet parser validate" tmp=$(mktemp /tmp/git.update.XXXXXX) log=$(mktemp /tmp/git.update.log.XXXXXX) tree=$(mktemp /tmp/git.diff-tree.XXXXXX) git diff-tree -r "$2" "$3" > $tree echo echo diff-tree: cat $tree exit_status=0 while read old_mode new_mode old_sha1 new_sha1 status name do # skip lines showing parent commit test -z "$new_sha1" && continue # skip deletions [ "$new_sha1" = "0000000000000000000000000000000000000000" ] && continue # Only test .pp files if [[ $name =~ [.]pp$ ]] then git pull origin master && service nginx restart cat-file blob $new_sha1 > $tmp set -o pipefail $syntax_check $tmp 2>&1 | sed "s|$tmp|$name|"> $log if [[ $? != 0 ]] then echo echo -e "$(cat $log | sed 's|JOJOMOJO|'\\${RED}${name}\\${NOBOLD}'|')" >&2 echo -e "For more details run this: ${CYAN} git diff $old_sha1 $new_sha1 ${NOBOLD}" >&2 echo exit_status=1 fi fi done < $tree rm -f $log $tmp $tree exit $exit_status </pre> * Make the post-receive hook executable <pre> chmod +x /usr/local/share/puppet-dashboard/public/post-receive </pre> * Then restart Make the cron service to enable the funcationality immediately: update hook executable <pre> chmod +x /usr/local/share/puppet-dashboard/public/update </pre> * Now restart nginx: <pre> service cron nginx restart </pre> h3. (Optional) Adding Continuous Integration to Puppet Git Pre-Commit Hook (Puppet Master Files Side) I manage many NOTE: This is not used in this guide, but included for posterity. * To catch syntax errors and other basic problems, you can use a *+client-side+* Git pre-commit hook like this (useful for cron scripts): <pre> #!/bin/sh syntax_errors=0 error_msg=$(mktemp /tmp/error_msg.XXXXXX) if git rev-parse --quiet --verify HEAD > /dev/null then against=HEAD else # Initial commit: diff against an empty tree object against=4b825dc642cb6eb9a060e54bf8d69288fbee4904 fi # Get list of my services with raw configuration new/modified manifest and template files (until I can learn to check (in git index) for indexfile in `git diff-index --diff-filter=AM --name-only --cached $against | egrep '\.(pp|erb)'` do # Don't check empty files if [ `git cat-file -s :0:$indexfile` -gt 0 ] then case $indexfile in *.pp ) # Check puppet manifest syntax #git cat-file blob :0:$indexfile | puppet --color=false --parseonly --ignoreimport > $error_msg ;; # Updated for 2.7.x puppet parser validate $indexfile > $error_msg ;; *.erb ) # Check ERB template syntax # -P : ignore lines which start with "%" git cat-file blob :0:$indexfile | erb -P -x -T - | ruby -c 2> $error_msg > /dev/null ;; esac if [ "$?" -ne 0 ] then echo -n "$indexfile: " cat $error_msg syntax_errors=`expr $syntax_errors + 1` fi fi done rm -f $error_msg if [ "$syntax_errors" -ne 0 ] then echo "Error: $syntax_errors syntax errors found, aborting commit." exit 1 fi </pre> *NOTE*: Make sure that the post receive hook is executable! h2. Adding the Web Hook To GitLab This part is where the magic happens. # Log into GitLab and go to the +_Project_+ created to manage templates via @.erb@ files), so adding those configurations the puppet configurations. # Next go to either their own branch or even their own repository entirely helps me out. the projects +_Settings -> Web Hooks_+. # Enter in the puppet master server and location of the post-receive script, example: *http://puppet.example.com/post-receive* h2. Resources * http://puppetlabs.com/blog/git-workflow-and-puppet-environments * https://groups.google.com/forum/#!topic/gitlabhq/WWUox4MLe2U * https://www.digitalocean.com/community/tutorials/how-to-set-up-automatic-deployment-with-git-with-a-vps * http://dreamingwell.com/articles/archives/2014/02/using-git-and-puppet-for-agile-server-management.php * http://krisjordan.com/essays/setting-up-push-to-deploy-with-git * http://projects.puppetlabs.com/projects/1/wiki/puppet_version_control