PuppetConf 2014 - Part 2

Day 1

This is our summary of PuppetConf 2014. In our previous post we gave an overview of the contributor summit. This post will provide an overview
of the first day of PuppetConf.

As you might expect there were great keynotes with plenty of announcements and too many talks for us to attend. We have provided an outline for all the talks
we did attend and links to those we didn’t.


Nearly a Decade of Puppet: What We’ve Learned and Where We’re Going Next - Luke Kanies, PuppetLabs - Slides

The big keynote of the event to kick off the first day from the author of Puppet himself. This was obviously going to be a tweet worthy affair full of photos
and big announcements and it did not disappoint.

[Read More]

PuppetConf 2014 - Part 1

The start of PuppetConf 2014

It has been one week since our attendance at this years PuppetConf and we have just now caught up on all the great talks that were
given and the projects demonstrated over the 3 day period. Here’s our summary of the event (split into 3 parts), hopefully you will
find as much inspiration in the content as we have.

Day 0 - Contributor Summit

For the first time, this years Puppet Contributor Summit was held the day prior to the conference itself and I think this was a great idea.
Most of the Puppetlabs staff and many of the high profile community members were in town for PuppetConf anyway so it made sense. There was
roughly 60-70 people in attendance both senior contributors and people new to the community so it was a great mix that led to some
fantastic discussions.

The day itself had two tracks: a module track for forge modules and a core track for people contributing to puppet and factor.

Those of you who have seen our forge module page will understand why we chose to stay in the module track.
Although I heard there were many great discussions to be had with regards to Puppet 4 in the core track.

[Read More]

Introducing Hobknob: Feature toggling with etcd

The ability to dynamically turn features on/off in software without the need to redeploy code is extremely beneficial. Whether you are trialing a new feature or using branch by abstraction to avoid creating feature branches, the use of feature toggles can aid continuous delivery and provide a mechanism to reduce mean time to resolution when an issue occurs.

With a relatively large engineering department with multiple teams spread across the US and UK the need to manage feature toggles has evolved to the point whereby individual teams have developed their own implementations. Most of these are simple config files.

We decided to unify this effort by providing a central place to store feature toggles, provide a dashboard to be able to turn these toggles on/off and provide language specific clients to integrate into our software components.

The results of this was Hobknob.

Why etcd?

We made the decision to use etcd. Etcd is “a highly-available key value store for shared configuration” (https://github.com/coreos/etcd#etcd). It provides a HTTP API to store and retrieve data. This is what makes it perfect for a feature toggling solution used by multiple components. It means that we didn’t have to write an intermediate API on top of a data store for consumers.

[Read More]

Testing Puppet with Beaker pt.3 - Testing Roles

In the first two parts of this blog series we have focusing on testing puppet modules with beaker. As an open source contributor there is always
a large test matrix so this makes absolute sense. But what about the other large use-case for beaker - what about our day-to-day internal code base?
Not all of this is modules, in fact a large portion of it is other puppet code - roles, profiles, facts, hiera data etc. All of this needs testing
as well.

In this blog post I will be showing how we have started using beaker to test our puppet roles and profiles for both Linux and Windows.


Prior to this post all our beaker testing has been master-less i.e. using using puppet agent apply. This is perfectly adequate for most use cases when
testing modules in isolation but doesn’t always work when testing an internal code base (unless you are masterless there as well then please skip to the next section).

At OpenTable we do use a central puppet master to compile our catalogs. So when testing our puppet roles we wanted to make sure that we were also testing
with a master-agent configuration. It is worth mentioning here that if (like us) you are testing windows agents then you are going to need to test with master-agent
approach due to the lack of a windows master.

Testing the master-agent configuration means configuring multi-node sets in beaker. There are not many examples of this but the principle is very much
the same as the single-node nodeset. Here is an example:

[Read More]

Testing Puppet with Beaker pt.2 - The Windows story

In part one we discussed our first steps into the world of acceptance testing our Puppet manifests.
By using Beaker we able to test managing local users on our Linux boxes. This was a positive experience for us. It allowed us to get to grips with the basics of configuring
Beaker to run tests and configuring our node sets to run those tests against. In this post, we will be discussing how we went about getting Beaker working with Windows.

As many of your reading this will be aware, OpenTable currently has quite a large Windows infrastructure and we are using Puppet extensively to maintain that environment.
We are also moving forward with releasing as many of our modules open source onto the Puppet Forge as possible (11 out of 18 of which are Windows
exclusive). What this means is that there was no way that we could ignore trying to use Beaker to test our manifests against Windows. We knew that we would have to support
many different versions and editions of Windows out there in the community as well that the ones we have to support internally.

This was going to be a challenge (configuration management with Windows usually is) but we were up for it.

##The Preliminaries

The first step was looking at Serverspec. Serverspec is a Ruby gem that provides extensions to RSpec that allow you to test the actual state of your servers, either locally
or from the outside in via SSH. What we needed to know was did it support Windows? The answer was thankfully a resounding “Yes!”. All the resource types that you might want to test including file, service and user are available and supported on Windows. There are also a couple of Windows specific ones such as iis_website, Windows_feature and Windows_registry_key. We even added our own to support Windows_scheduled_task. Interestingly Serverspec also supports WinRM as an alternative to SSH when you are testing from the outside-in but we will go back into that later. As long as your using Serverspec > 1.6 you will have all the Windows support you might need.

[Read More]

What can I do?

I’ve noticed recently a lot of content in my social media network is based on the current escalating problems in the Middle East.

Whenever I see such content from a friend or follower, I’m reminded of Stephen Covey’s The 7 Habits of Highly Effective People - Circle of Influence, Circle of Concern.

Covey says that in order to remain truly effective, we should focus our time and energy on situations we can influence. Don’t worry about things we cannot control, such as the weather, the economy or indeed foreign conflicts.

Covey’s work implies that if we identify the areas that are inside our Circle of Influence we will be more productive and make more of an impact;


[Read More]

Acceptance Now

When is acceptance-only testing a good idea, and how can its problems be overcome?

In a recent post, I espoused some of the benefits my team enjoyed by reducing our test-base to a single layer of acceptance tests, with no separate unit or integration tests. It caused some minor controversy, which was not to be unexpected. At the time, I knew I had left out some details for brevity’s sake. In this post—spurred on by some interesting questions and commentary—I’d like to offer a more constructive view on the subject, and dig a little deeper into the nitty gritty of how we made it work.

I’ll also point out some other hidden benefits of moving to acceptance-only testing, and suggest synergistic practices that can help decide if this is the right approach for your project.


After-the-fact unit testing requires us to find the seams along which code can be isolated and tested. Where those seams don’t exist, the temptation is to refactor code until they do, using patterns like dependency injection, and following SRP and other SOLID patterns.

Test-first unit testing aka TDD naturally tends to maximise seams, and results in highly decoupled code with small, specific tests. TDD must result in 100% unit test coverage, if practiced according to the gospel. In other words, TDD results in the best kind of code for later modification without risk, and the best tests for detailed, granular feedback. Unit tests also tend to run very quickly, sometimes fast enough to run every time you hit “Save”.

[Read More]

Continuous Delivery: Automating Deployment Visibility

In our continued effort to drive towards a service oriented architecture each of our teams are continuously improving their deployment processes. Recently our team has focussed on automating as much as possible, putting as much into chat as we can and improving our logging/metrics.

The image below shows at a high level what our teams current deployment pipeline looks like and this post will attempt to summarise some recent changes that have allowed us to automate visibility.

##Kicking off a deployment

I wrote previously that we started using chatops to increase visibility operationally. Hubot is central to this and we wrote a small script to kick off deployments within Hipchat

[Read More]

Managing Windows Features with Puppet

Back in June 2013, I wrote about Windows Feature Management with PowerShell. We have since released a Puppet module that will do this for us. We originally wrote PowerShell:

Import-Module ServerManager
Add-WindowsFeature Web-Asp-Net

The Puppet module now wraps this code as follows:

windowsfeature { 'Web-Asp-Net': }

The declaration windowsfeature is a specific Puppet type called a define. In developer terms, this is the equivalent of a helper method that can be reused. We can also make sure that Windows Features are not installed on the server as follows:

windowsfeature { 'Telnet-Server': 
  ensure => absent 
[Read More]

Managing Windows Web Applications with Puppet

As part of our move towards a configuration management tool, we really wanted to start automating as much of our infrastructure as possible. This included our application configuration stack. IIS management is pretty easy with PowerShell. It would look something like this

Import-Module WebAdministration
New-WebSite -Name "DemoSite" -Port 80 -IP * -PhysicalPath "c:\inetpub\wwwroot" -ApplicationPool "MyAppPool"

This would of course set up a website called ‘DemoSite’ running on port 80 on the local machine. The cmdlets that come with PowerShell make this pretty easy. This is great if it is a one-off job to set up a site. We run our websites from a number of webservers, therefore, it would be silly to have to RDP into each webserver and run a script on it. This is why tools like Puppet, Chef, Ansible etc. exist. We needed a configuration management tool to do this work for us. It has a number of benefits:

  • Orchestration
  • Idempotency
  • Makes sure that each server is configured in ‘exactly’ the same way as no human intervention is needed
  • Developers can help the operations team by creating the scripts needed. This is great for collaboration between teams

On investigating how we would do this with Puppet, we noticed that there were not many other people managing their site in this way. Therefore, we would have to turn our PowerShell scripts into Puppet modules to manage our system.

We have since created a Puppet module to manage IIS. To manage IIS with Puppet, we can now write the following code:

[Read More]