Most companies don’t have a single monolithic application, today most applications evolve out of lots of atomic services: thus, service oriented architecture.
What this means for your development environment is that you end up having many different applications, which is many cases are atomic services and in other cases are a conglomeration of services. Either way, you end up with a need for many dev environments.
At Connect Solutions we have several environments that require infrastructure development. The main ones are our custom load balancer (we call it MTLB), our Connect Cluster, CQ, and our Connect Experience applications.
In order to streamline our dev process with Puppet we created a repo called ‘puppet-dev-enviro’ that deploys VM’s with Vagrant and manages what modules you want to test via Rake & r10k.
For any given envio we like to mirror what it looks like in prod as best we can. Therefore, any enviro always uses the version of puppet we run in prod on a separate Puppet Master, then the nodes that make of the rest of the enviro.
A simple example, theMTLB enviro deploys a Puppet Master and an MTLB load balancer node.
Since we have so many enviros we branch the ‘puppet-dev-enviro’ repo for each one. So if you’re testing MTLB code the workflow looks like this:
1 2 3
Those first two commands are pretty straight forward. However, the deploy command probably needs some explanation. I wrote this task to streamline our development pipeline. It’s main functionality is to read from a subdirectory called puppet and a file called the Puppetfile.
You might be familiar with the Puppetfile concept if you’ve used Puppet Librarian or r10k. In this task, I actually invoke r10k locally on the Puppetfile to pull down all the modules listed in it to puppet/modules. In the case of Connect Solutions, we’re currently in the process of moving our legacy puppet module structure from a monolithic one to their own atomic repos. This isn’t complete yet, so the MONO=true env variable envokes a subcommand in deploy to rip out all the sub-modules of this monolithic repo and place them in puppet/modules.
‘Deploy’ also supports hiera data management by checking for our
puppet-configuration repo in the puppet/modules directory. If it’s listed in the Puppetfile this repo will be present here. If so, it moves all the data from puppet-configuration into puppet/data.
puppet/data are shared with the puppet master VM deploed with Vagrant. As a Vagrant post-provisioning step I blow away the
/etc/puppetlabs/puppet/data directories and sym link the shared modules and data directories out of
The Deploy Task
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
This structure allows us to play around with the module code on the fly from the host VM, with our own dotfiles and local editing or preferred pipelines. For example, I am a heavy Vim user, however, a lot of my colleagues are Sublime text users. You can choose what method best suites you and not be beholden to who provisioned the VM .box file beforehand.
For large code-base changes, i.e., developing an entireley new topic branch, this pipeline allows seamless integration with git and enables better version control management. An example workflow would be:
- Open two screen or tmux sessions
- In one session cd into
- In the other session cd into
- git checkout my-enviro
1 2 3
MONO=true rake deploy
This will now pull down and configure the environment for the VM’s. When finished, all the puppet data and modules will be live on the master VM.
One step I’ve skipped here however is making a git-branch for my-enviro. That’s a completely different step but mainly involves updateing the Vagrantfile with the neccessary VM information. I won’t cover that here since that is specific to every enviro the same way Puppet code is speciic to each module. I will however share what stays the same between most Vagrantfiles in each enviro and that is the Master VM configuration block:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Code Test -> Develop Pipeline
Once the VM’s are up and running you can ssh into the VM under test and run
puppet agent -t. If everything went as planned the agent will get a catlog from the master and start configuring itself. If you’re like me, it didn’t work on the first try. Now we need to update our code-base.
In the screen or tmux session that is CD'ed to your
puppet-modules directory go into the manifests dir of the code in question. Edit that code or make whatever changes are needed for further testing.
Keep in mind we are working on our
my-topic-branch branch of the
puppet-modules repo. So when we’re done making the local changes:
1 2 3
Then in the other screen window that’s in the
puppet-dev-enviro directory run:
This runs only r10k and the workflow neccessary to pull down the new Puppet code. It is a heavy operation if you have a LOT of code in that Puppetfile. We haven’t ran into an issue with that yet however for our individual test enviros. The :pull task looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
Now you have new code in your master VM, run
puppet agent -t again on your agent.
Wait a minute!
Some will say, “This is more work than before. Now I have this extra git step!”. Yup, that’s right, you’re forced to commit your changes to your own topic branch before you test on your local machine. Yes, this isn’t good for doing work on the airplane. Granted, this test enviro as it currently stands isn’t designed for the latter, it’s designed towards seamless collaboration.
What do I mean by that? I mean, I don’t want to have the conversation with a colleage at 4PM on a Thursday night that goes like this: “Hey, I need to collaborate on your code. Can you push it to a topic branch in git for me?” “Sure, I’ll do that as soon as I get home tonight after picking up the kids and making dinner.” “Great!” … The following day: “So hey, I REALLY wanted to test out that code we need in prod by this afternoon last night, but I never saw a push for your branch.” “Yeah, sorry, I got caught up in stuff. You know, life!”.
That ‘extra’ git step ensures all the development steps are being tracked, and anyone at anytime can spin up this environmnet with the exact same Puppetfile and site.pp configuration and get similar testing results.
Some day, I’ll put together a local-only development environment but for now this is not the purpose of what we’ve created here.