SRC:CLR Deploy - Merge-to-Master -> Jenkins -> Puppet
At SRC:CLR we used to use Opsworks to deploy our code pipeline. Opsworks is great but is overloaded with a lot of extra things that we don’t need or want. It also doesn’t allow us to fully open our code pipeline to outside applications in the way we wanted.
In order to give our devs the easist way to deployment, we setup a pipeline from merge-to-master to deploy that encompasses the peer review process, java unit tests, and of course Puppet.
If you don’t want to read the post, we’ve open sourced the code for this webhook.
The 10,000 Mile View
- A developer submits a PR to merge to master from a dev branch
- Her peers review this PR and accept the merge to master
- Jenkins watches this repo, and starts a build on that branch
- If all tests pass, the jar file is sent to s3:
- A post build shell script runs that sends a POST with data about the buid to our puppet master
- The webhook on the master accepts this POST and uses the data to:
- Update hte version of the service in Hiera
- Update the git repo where the data file that tracks versions exists (in this case the puppet control repo with hiera data)
- Run mCollective to update all nodes matching the role which the service runs on
- The node(s) check in with the master, and diff the versino in the title of the S3 resource for the service jar file
- See the version has changed (since we updated it in hiera) the agent pulls down hte new jar file with version $version
- The agent restarts the services
The More Intimate View
This sinatra hook sits on a Puppet Master and listens for POSTs on :1015, when hit with a payload it updates the version number of a service in Hiera data and executes a mCollective call to run puppet on the node with a matching $::role value.
This is great when used in conjunction with puppet-s3 provider or something similar where you can classify the resrouce like
Then, using our webhook or a modified version of it, you can have a one click deploy in Jenkins since this hook will update the version in Hiera data that the
s3 resource is diff'ing from and execute mCollective to run puppet on the node matching a specific role - i.e., the role that classifies the node running the service being built by jenkins and pulled down by the
How it’s done…
Lets say you wanted to use the open sourced webhook and integrate it with your environment, here’s what you need to know:
A few assumptions:
- The payload from Jenkins, or whatever tool you’re using to hit this hook, will pass the following parameters:
1 2 3 4 5 6
- The Hiera data key matches the following pattern:
1 2 3 4 5 6 7 8
You can override this in the options you pass via the JSON POST with the key
- You’ll fork this repo, and update the
data_fileand maybe the
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
- You’re using a
$::rolefact. I roll in AWS, so everything is classified based on
$::role. This webhook won’t be able to run puppet on the node running your service you just updated the version for until you modify this code or get yourself a role face.
- Clone the repo to your pupetmaster and make the above suggested changes to make it work with your deployment
- Turn it on:
- Have a post-run stage in your jenkins build for a given service that executes something akin to the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Kick off a build….
This should implement the following chain:
- Build is executed on jenkins
- If successful the build drops the new versioned jar or zip file or whatever in S3:
- If successful the shell post-run is executed, running the above script that gets the new $version and POSTs to our webhook on the puppet master
- The webhook updates hiera data with the correct value for the updated version
- The webhook updates git to ensure that jenkins owns our versioning
- The Webhook executes an MCO call to run puppet on the node running this service based on the
- The nodes matching the
$::roleget the updated version in hiera data and match that against the
s3resource in that:
1 2 3 4 5 6 7 8 9
$version is derived from
1 2 3
Some Closing Remarks… or why this is rad
Your devs should never have to worry about depoying code, they have enough to worry about in writing it. The pipeline that is built around deployment should be mind numbingly simple for them to implement. Merge-to-master is scary for a lot of reasons. Automating the updating of values in Hiera is scary for a lot of reasons. At the end of the day however, those are my problems and not the developers. My job is to make efficient pipelines for our product, and the release pipeline is the purest incarnation of that. We think this is pretty rad, and if you want to play along and fork our public repo and submit some PR’s we’d love to hear from you.