Feature #453
Updated by Daniel Curtis over 9 years ago
One of h2. Preparing the features offered Repositories * Keep things straightforward by Puppet is placing the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles git repository, remote git repository, and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to deploy directory under the web server environment don’t get applied to machines that don’t need that configuration. same local directory: <pre> mkdir push-to-deploy cd push-to-deploy mkdir {development,remote,deploy} </pre> Mapping * Now setup the Puppet code base against the environments shows the power of remote git repository. With a real project, this method. People often use would be a version control system to manage the code, and create directory on your production server. There is a set of branches that each map special setup for git repositories whose purpose is to an environment. * Adopting the development, testing and production workflow, we receive pushes from developers, they're called "bare" repositories. You can have a puppet.conf that looks something like this: read more about "bare" repositories, but for our purposes their purpose is just to receive pushes: <pre> vi /usr/local/etc/puppet.conf cd remote git init --bare cd .. </pre> #* And should look something like * Now setup the following development directory with git. (You could also copy one of your git project's contents into the development folder.) <pre> [main] server = puppet.example.com environment = production confdir = /etc/puppet cd development [agent] report = true show_diff = true git init [production] manifest = /etc/puppet/environments/production/manifests/site.pp modulepath = /etc/puppet/environments/production/modules echo "Hello, world." >> file.txt [testing] manifest = /etc/puppet/environments/testing/manifests/site.pp modulepath = /etc/puppet/environments/testing/modules git add file.txt [development] manifest = /etc/puppet/environments/development/manifests/site.pp modulepath = /etc/puppet/environments/development/modules git commit -m 'First commit.' </pre> This will set 3 different environment: # production*: This Now there is for production ready systems, all bugs and changes should be made in testing or a development for environments before being applied repository with its first commit. The last preparation step is to production. register the "remote" repository. If you're fuzzy on git remotes, the official git site has you covered: # testing*: This is for testing certain features that may be broken or needs more work before entering the <pre> git remote add production environment. ../remote # # development*: This is for developing features, fixing bugs, and/or modifying themes before entering the testing environment. git push production master </pre> With this configuration, we could map three Git branches for these environments and set up a central Git repository with post receive hooks. When changes were Boom. You've just pushed your commit from development to this repository, they would be automatically deployed your bare, 'remote' repository. We're ready to the puppet master. The example post-receive hook later in this post will work with this kind of environment setup. setup push-to-deploy. h2. Dynamic Environments Set up Push-to-Deploy While there are benefits to having a set of branches and environments like Now that the previously outlined configuration, there are also drawbacks with a static set of environments. This model remote repository is somewhat constrained by enforcing a single workflow. For instance, if multiple people are working on experimental features in the development branch, they'll have setup, we're ready to constantly merge their code through the development process. If multiple people are developing different features in the same branch, they’ll all have to work with unrelated changes in the code. In addition, migrating write a single feature to testing or production gets more complicated if multiple features are in the same branch. Modern distributed version control systems like Git handle these constraints by making branch creation and merging lightweight operations, allowing us to generate Puppet environments on the fly. Puppet can set up environments with an explicitly defined section in the configuration file, but we can exploit the fact Puppet will set the @$environment@ variable to the name of environment script for what it'll do when it is currently running under. receives a push. * With this in mind, we can write a puppet.conf Navigate to resemble this: <pre> vi /usr/local/etc/puppet/puppet.conf </pre> #* And modify it similar to the following: hooks folder of the remote repository. Hooks are scripts that git runs when certain events happen: <pre> [main] server = puppet.example.com environment = production confdir = /etc/puppet cd ../remote/hooks [master] environment = production manifest = $confdir/environments/$environment/manifests/site.pp modulepath = $confdir/environments/$environment/modules touch post-receive [agent] report = true show_diff = true environment = production chmod +x post-receive </pre> * This handles the dynamic environment aspect; all The hook we have to do care about for push-to-deploy is create post-receive. It is run after receiving and accepting a directory push of commits. In the commands above, we're creating the post-receive script file with manifests in @$confdir/environments@ `touch`, and we have created that environment: <pre> mkdir -p /usr/local/etc/puppet/environments mkdir /usr/local/etc/puppet/environments/production mkdir /usr/local/etc/puppet/environments/testing mkdir /usr/local/etc/puppet/environments/development </pre> making sure it's an executable file. * Generating new environments using Git is similarly easy. We can create a central Git repository on Next, open the GitLab instance, with a @post-receive@ hook that looks something like this: post-receive script file in any preferred text editor. Copy the contents below: <pre <pre> #!/usr/bin/env ruby # Puppet Labs is a ruby shop, so why not do the post-receive hook in ruby? require 'fileutils' # Set this to where you want to keep your environments 1. Read STDIN (Format: "from_commit to_commit branch_name") ENVIRONMENT_BASEDIR from, to, branch = "/usr/local/etc/puppet/environments" ARGF.read.split " " # post-receive hooks set GIT_DIR to the current repository. If you want to 2. Only deploy if master branch was pushed if (branch =~ /master$/) == nil puts "Received branch #{branch}, not deploying." exit end # clone from a non-local repository, set this 3. Copy files to the URL of the repository, deploy directory # such as git@git.host:puppet.git SOURCE_REPOSITORY deploy_to_dir = File.expand_path(ENV['GIT_DIR']) # The git_dir environment variable will override the --git-dir, so we remove it File.expand_path('../deploy') # `GIT_WORK_TREE="#{deploy_to_dir}" git checkout -f master` puts "DEPLOY: master(#{to}) copied to allow us to create new repositories cleanly. ENV.delete('GIT_DIR') '#{deploy_to_dir}'" # Ensure that we have the underlying directories, otherwise the later commands 4.TODO: Deployment Tasks # may fail in somewhat cryptic manners. i.e.: Run Puppet Apply, Restart Daemons, etc unless File.directory? ENVIRONMENT_BASEDIR puts %Q{#{ENVIRONMENT_BASEDIR} does not exist, cannot create environment directories.} exit 1 end </pre> # You can push multiple refspecs at once, like 'git push origin branch1 branch2', # so we need to handle * A walk through each one. of the steps: $stdin.each_line do |line| oldrev, newrev, refname = line.split(" ") # Determine *# When git runs post-receive, the branch data about what was received is provided to the script via STDIN. It contains three arguments separated by spaces: the previous HEAD commit ID, the new HEAD commit ID, and the name from of the refspec we're received, which branch being pushed to. We're reading these values and assigning them to from, to, and branch variables, respectively. *# Our purpose here is in the # format refs/heads/, and make sure that it doesn't have any possibly # dangerous characters branchname = refname.sub(%r{^refs/heads/(.*$)}) { $1 } if branchname =~ /[\W-]/ puts %Q{Branch "#{branchname}" contains non-word characters, ignoring it.} next end environment_path = "#{ENVIRONMENT_BASEDIR}/#{branchname}" if newrev =~ /^0+$/ # We've received to automate push-to-deploy. Assuming a push with a null revision, something like 000000000000, # which means workflow that keeps production on the master branch, we should delete want to exit this script prior to deploying if the given branch. puts "Deleting existing environment #{branchname}" if File.directory? environment_path FileUtils.rm_rf environment_path, :secure => true end else # We have been given a branch that needs being pushed is not master. *# The first deploy step is to be created "checkout", basically export or updated. If copy, files from the # environment exists, update it. Else, create it. if File.directory? environment_path # Update an existing environment. We do a fetch and then reset master branch to the directory where our project is deployed to in production. (Remember, in this demo it's the # case fake "deploy" directory, in the real world this might be /var/www, or wherever your project expects to be in production.) *# Now that someone did a force push our deploy directory is up-to-date, we can run whatever deployment tasks we need to run. This could be applying Puppet scripts (I'll write a branch. puts "Updating existing environment #{branchname}" Dir.chdir environment_path %x{git fetch --all} %x{git reset --hard "origin/#{branchname}"} else # Instantiate post on this scenario soon), restarting a new environment from web or application server, clearing cache files, recompiling static assets, etc. Whatever steps you'd normally need to do manually after updating your project's files, automate them here! Save your post-receive hook, and test it out! h2. Testing with Pushing * Test the current repository. puts "Creating script manually, by creating a new environment #{branchname}" %x{git clone #{SOURCE_REPOSITORY} #{environment_path} --branch #{branchname}} end end end </pre> *NOTE*: Make sure that commit in the post receive hook is executable! h3. Creating development directory and using a new environment <pre> pushing: git branch </pre> > * production <pre> git checkout -b new_feature cd ../../development </pre> echo "New line." >> file.txt > Switched to a new branch 'new_feature' <pre> mvim site.pp git add site.pp file.txt git commit -m 'Implemented my feature' 'Testing push-to-deploy' git push production master </pre> > [new_feature 25a9e1b] Implemented my feature > 1 files changed, 1 insertions(+), 0 deletions(-) <pre> In the output of the git push origin new_feature command, you should see lines starting with "remote:". These lines are the output of our post-receive script: </pre> > Counting objects: 5, done. Already on 'master' > Writing objects: 100% (3/3), 255 bytes, done. > Total DEPLOY: master($TO_ID) copied to 'push-to-deploy/deploy' The first line is noisy output from the git checkout command in step 3, we can ignore it. The second line, is from the puts command, also from step 3 (delta 0), reused 0 (delta 0) in our post-receive script. * The directory we're deploying to should now be populated and up-to-date: > Unpacking objects: 100% (3/3), done. <pre> > remote: Creating new environment new_feature ls ../deploy > To git@git.host:deploy.git diff file.txt ../deploy/file.txt > </pre> Pretty awesome, right? h2. Testing without Pushing * [new branch] new_feature -> new_feature And from here When you're working on out, a post-receive hook, it's annoying to muck up your project's commit history and push each time you make a change. Luckily, because it's just a script, we can use fake it from the @new_feature@ environment on your hosts, and use command-line. <pre> cd ../remote git like you would with any code base. log -2 --format=oneline --reverse </pre> This development model gives * First, get the IDs of our most recent 2 commits. The git log command, above, will give us some simple access control. Utilizing access control with a tool like GitLab, we can allow people these two IDs in the order you'll want to generate new environments to test their own code, but deny them access to change replace the production environment. $FROM_ID and $TO_ID variables with, respectively. <pre> echo "$FROM_ID $TO_ID master" | ./hooks/post-receive </pre> This allows us method makes setting up your post-receive hooks enjoyable, enabling you to institute some sort of change control, by requiring all code quickly iterate on your script and execute it repeatedly. h3. Next Steps In this post, we've walked through how to be reviewed by setup the push-to-deploy with git. For a merge master before inclusion into production, real world project, your 'remote' and allows code to 'deploy' folders would usually be tested setup on a server, not locally. The details of doing that and verified before properly configuring SSH is beyond the request for submission is made. post of this scope (note to self: I should write on SSH configuration, too!). h2. Resources * http://puppetlabs.com/blog/git-workflow-and-puppet-environments http://krisjordan.com/essays/setting-up-push-to-deploy-with-git