Feature #453
Updated by Daniel Curtis over 9 years ago
One of the features offered by Puppet is the ability to break up infrastructure configuration into environments. With environments, you can use a single Puppet master to serve multiple isolated configurations. For instance, you can adopt the development, testing and production series of environments embraced by a number of software development life cycles and by application frameworks such as Ruby on Rails, so that new functionality can be added incrementally without interfering with production systems. Environments can also be used to isolate different sets of machines. A good example of this functionality would be using one environment for web servers and another for databases, so that changes made to the web server environment don’t get applied to machines that don’t need that configuration. There are two servers that will be used in this guide: # puppet.example.com: the puppet master server # git.example.com: the GitLab server NOTE: This will not manage the puppet master configurations directly, but rather the environments associated with each branch of the mail git repository. h1. Prepare the Local Computer environments * Log into the GitLab web interface and create a new +_Project_+ for the web service puppetry to be stored #* This example uses +_Web Service Puppetry_+ * Create an repository to hold the configurations: <pre> mkdir -p ~/git/web-service-puppetry && cd ~/git/web-service-puppetry </pre> * Initialize the repository: <pre> git init </pre> * Create a README, add it to the local repository, then commit the local changes to be pushed to the remote git server <pre> touch README git add README git commit -m 'first commit' </pre> * Add the remote git server location to the local repository <pre> git remote add origin ssh://git@git.altservice.com/SecretUser/web-service-puppetry.git </pre> * Push the first commit to the remote git server <pre> git push -u origin master </pre> Mapping the Puppet code base against the environments shows the power of this method. People often use a version control system to manage the code, and create a set of branches that each map to an environment. h2. Creating and using the environments * Show the current branch <pre> git branch </pre> > * master h3. Create the environment branches +PRODUCTION BRANCH+ * Create the *production* environment branch: <pre> git checkout -b production </pre> #* Will result in: <pre> Switched to a new branch 'production' </pre> * Populate a bare @site.pp@ for the *production* environment <pre> mvim site.pp git add site.pp git commit -m 'Created the initial site.pp for the production environment' </pre> #* Will result in: <pre> [production 25a9e1b] Created the initial site.pp for the production environment 1 files changed, 1 insertions(+), 0 deletions(-) </pre> NOTE: I migrated my existing puppet master configurations into to *production* branch, and then used these files for the *testing* and *development* later: <pre> git add --all git commit -m 'Migrated existing puppet master configuration' </pre> * Push the changes to the git server <pre> git push origin production </pre> #* Will result in: <pre> Counting objects: 5, done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. remote: Creating new environment production To git@git.host:deploy.git * [new branch] production -> production </pre> +DEVELOPMENT BRANCH+ * Create the *development* environment branch: <pre> git checkout -b development </pre> #* Will result in: <pre> Switched to a new branch 'development' </pre> * Populate a bare @site.pp@ for the *development* environment <pre> mvim site.pp git add site.pp git commit -m 'Created the initial site.pp for the development environment' </pre> #* Will result in: <pre> [development 25a9e1b] Created the initial site.pp for the development environment 1 files changed, 1 insertions(+), 0 deletions(-) </pre> * Push the changes to the git server <pre> git push origin development </pre> #* Will result in: <pre> Counting objects: 5, done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. remote: Creating new environment development To git@git.host:deploy.git * [new branch] development -> development </pre> +TESTING BRANCH+ * Create the *testing* environment branch: <pre> git checkout -b testing </pre> #* Will result in: <pre> Switched to a new branch 'testing' </pre> * Populate a bare @site.pp@ for the *testing* environment <pre> mvim site.pp git add site.pp git commit -m 'Created the initial site.pp for the testing environment' </pre> #* Will result in: <pre> [testing 25a9e1b] Created the initial site.pp for the testing environment 1 files changed, 1 insertions(+), 0 deletions(-) </pre> * Push the changes to the git server <pre> git push origin testing </pre> #* Will result in: <pre> Counting objects: 5, done. Writing objects: 100% (3/3), 255 bytes, done. Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. remote: Creating new environment development To git@git.host:deploy.git * [new branch] testing -> testing </pre> * Now check to see the environment branches: <pre> git branch </pre> #* Will result in: <pre> development master production * testing </pre> * To switch between each branch just run, adjusting accordingly: <pre> git checkout production </pre> And from here on out, you can use the @production@ environment on your hosts, and use git like you would with any code base. This development model gives us some simple access control. Utilizing access control with a tool like GitLab, we can allow people to generate new environments to test their own code, but deny them access to change the production environment. This allows us to institute some sort of change control, by requiring all code to be reviewed by a merge master before inclusion into production, and allows code to be tested and verified before the request for submission is made. h2. Prepare the Environments on the Puppet Master Server * Create the environments directory on the puppet master server: mkdir /usr/local/etc/puppet/environments * Adopting the development, testing and production workflow, we can have a puppet.conf that looks something like this: <pre> vi /usr/local/etc/puppet.conf </pre> #* And should look something like the following <pre> [main] server = puppet.example.com environment = production confdir = /usr/local/etc/puppet [agent] report = true show_diff = true [production] manifest = /usr/local/etc/puppet/environments/production/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/production/modules [testing] manifest = /usr/local/etc/puppet/environments/testing/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/testing/modules [development] manifest = /usr/local/etc/puppet/environments/development/manifests/site.pp modulepath = /usr/local/etc/puppet/environments/development/modules </pre> This will set 3 different environment: # *production*: This is for production ready systems, all bugs and changes should be made in testing or development for environments before being applied to production. # *testing*: This is for testing certain features that may be broken or needs more work before entering the production environment. # *development*: This is for developing features, fixing bugs, and/or modifying themes before entering the testing environment. With this configuration, we could map three Git branches for these environments and set up a central Git repository with post receive hooks. When changes were pushed to this repository, they would be automatically deployed to the puppet master. The example post-receive hook later in this post will work with this kind of environment setup. An example will help clarify. Suppose I’m working on a new set of functionality, and don’t want to touch the current set of Puppet modules and inadvertently cause change in production. I can create a testing branch like so: git clone git@git.example.com/SecretUser/web-services-puppetry.git <pre> cd puppet git branch testing git checkout testing </pre> #* ... make some edits <pre> git add * git commit -m "made some edits" git push origin testing </pre> * and then run my Puppet agent against this new testing code: <pre> puppet agent --test --environment=testing </pre> > Info: Retrieving plugin > Info: Caching catalog for sun.local > Info: Applying configuration version '1371740640' As you can see, I set a default environment of production, and then specify paths to the manifest and modulepath directories, using the $environment variable to dynamically populate the path. Production manifest and modulepath paths will end up being $confdir/environments/production/manifests/site.pp and $confdir/environments/production/modules respectively. As new environments are dynamically created, the $environment variable will be substituted as appropriate. * Next, I moved my existing Puppet module and manifest structure around to suit the new configuration: <pre> mkdir -p /usr/local/etc/puppet/environments/production mv /usr/local/etc/puppet/manifests /usr/local/etc/puppet/modules /usr/local/etc/puppet/environments/production </pre> * And restart nginx (as I run my puppetmaster under Nginx/Passenger): <pre> service nginx restart </pre> * I then ran a couple of agents to ensure everything was still working: <pre> puppet agent --test </pre> They defaulted, as expected, to the Production environment. * Next, install git on the puppet master: <pre> pkg install git </pre> * Next, create a local git repository from the existing Puppet configuration: <pre> cd /usr/local/etc/puppet/environments/production git init git add * git commit -m "Initial import of Production Puppet repository" </pre> * Now add the remote GitLab repository: <pre> git remote add ssh://git@git.example.com/SecretUser/web-services-puppetry.git </pre> * And push the committed initial configuration to the remote GitLab repository: <pre> git push origin production </pre> h2. Continuous Integration of GitLab and Puppet Master After many hours of trial and error, and still not being able to get a functioning post-receive script to work, I decided to modify the above steps to have each environment have its own branch in the web services repo. * With the production environment being continually updated using a cron script: <pre> vi /etc/crontab </pre> * And then added the following to the end: <pre> ## Pull latest puppetry from GitLab */2 * * * * root cd /usr/local/etc/puppet/environments/production && git fetch --all && git reset --hard production pull origin master && service nginx restart </pre> * Then restart the cron service to enable the funcationality immediately: <pre> service cron restart </pre> h3. (Optional) Adding Continuous Integration to Puppet Master Files I manage many of my services with raw configuration files (until I can learn to manage templates via @.erb@ files), so adding those configurations to either their own branch or even their own repository entirely helps me out. h2. Resources * http://puppetlabs.com/blog/git-workflow-and-puppet-environments * https://groups.google.com/forum/#!topic/gitlabhq/WWUox4MLe2U * https://www.digitalocean.com/community/tutorials/how-to-set-up-automatic-deployment-with-git-with-a-vps * http://dreamingwell.com/articles/archives/2014/02/using-git-and-puppet-for-agile-server-management.php * http://krisjordan.com/essays/setting-up-push-to-deploy-with-git * http://projects.puppetlabs.com/projects/1/wiki/puppet_version_control