Hapi.js and SIGTERM

When we first stood up our hapi.js APIs, we wrote init scripts to start/stop them. Stopping the server, was simply a case of sending SIGKILL (causing the app to immediately exit).

Whilst this is fine for most cases, if we want our apps to be good Linux citizens, then they should terminate gracefully. Hapi.js has the handy server.stop(...) command (see docs here) which will terminate the server gracefully. It will cause the server to respond to new connections with a 503 (server unavailable), and wait for existing connections to terminate (up to some specified timeout), before stopping the server and allowing the node.js process to exit. Perfect.

This makes our graceful shutdown code really simple:

1
2
3
4
5
process.on('SIGTERM', function(){
server.stop({ timeout: 5 * 1000}, function(){
process.exit(0);
});
});

When we see a SIGTERM, call server.stop(), then once the server has stopped, call process.exit(0). Easy peasy.

[Read More]

Dismantling the monolith - Microsites at Opentable

A couple of years ago we started to break-up the code-base behind our consumer site opentable.com, to smaller units of code, in order to improve our productivity. New teams were created with the goal of splitting up the logic that was powering the back-end and then bring to life new small services. Then, we started working on what we call Microsites.

Microsites

A microsite is a very small set of web-pages, or even a single one, that takes care of handling a very specific part of the system’s domain logic. Examples are the Search Results page or the Restaurant’s Profile page. Every microsite is an independently deployable unit of code, so it is easier to test, to deploy, and in consequence more resilient. Microsites are then all connected by a front-door service that handles the routing.

Not a free ride

When we deployed some microsites to production we immediately discovered a lot of pros:

  • Bi-weekly deployments of the monolith became hundreds of deployments every week.
  • Not anymore a shared codebase for hundreds of engineers. Pull requests accepted, merged, and often deployed on the same day.
  • Teams experimenting and reiterating faster: product was happy.
  • Diversity on tech stacks: teams were finally able to pick their own favourite web-stack, as soon as they were capable of deploying their code and taking care of it in terms of reliability and performance.
  • Robustness: when something was wrong with a microsite, everything else was fine.
[Read More]

A Beginner's guide to REST services

Why this post?

As a junior, I always find it easier to just sit and write code than actually stop to think about the theoretical basis that lie under the applications I work on. REST is one of those terms I heard a lot about, so I decided to try to sum up what it means and how it affects the choices we make everyday as software engineers.

Introduction to REST

REST stands for Representational State Transfer, and it can be defined as an architectural style used to build Web Services that are lightweight, maintainable, and scalable. A service that is designed by REST principles can be called a RESTful service.

It has been described first in 2000 by Roy Fielding, in a dissertation called “Architectural Styles and the Design of Network-based Software Architectures”. The basic idea was to describe the interactions between the components of a distributed system, putting constraints on them and emphasizing the importance of an uniform interface, that is abstracted from the single components.

REST is often applied to the design and development of web services, which is the scenario I’ll try to address in this post.

[Read More]

On Strongly Typed Logging

Logging is a crucial element of monitoring highly available systems. It allows not only to find out about errors but also quickly identify their cause. Logs are often used to generate metrics that help business and engineering make informative decisions on future development directions.

At OpenTable we have a central logging infrastructure, that means all logs are stored in the same shared database (ElasticSearch for us). And everybody can access any logs they want without having very specialized knowledge (thanks Kibana!).

ElasticSearch, though living in a NoSQL world, is not actually a schema-free database. Sure, you do not need to provide schema to it but instead ES will infer schema for you from documents you send to it. This is very similar to type inference you can find in many programming languages. You do not need to specify type of field, but if you later on try to assign inappropriate value to it you will get an exception.

This trait of our database goes all the way to the root of our logging system design. Let me explain why I say that we have ‘strongly typed logs’.

In The Beginning There Was String

Before centralization we just logged a single message along with its importance. In code it looked something like:

1
logger.ERROR(“Kaboom!”)

[Read More]

Building a living styleguide at OpenTable

If you’re reading this you’ve probably built yourself a website. A site - large or small - that’s thrown together or crafted over many months. And if you have, you’ve probably kept all your CSS class names in your head, or at least been able to go straight to the relevant stylesheets to retrieve them.

Well OpenTable is unsurprisingly built by many engineering teams across multiple continents, and was completely redesigned last year. And as soon as you have more than a handful of people working on your front-end you will quickly find a well-intentioned developer causing one or both of these problems:

  • Well-intentioned developer adds a new submission form but, like the design Philistine he is, his buttons are 18px Verdana #E40000, not the correct 16px Arial #DA3743
  • Your good old developer knows which font size and colour it should be, but bungs a duplicate class into a random stylesheet (or worse still, inline)

Despite these risks, a single front-end dev (or a team of them) cannot check every new piece of code or they will quickly become a bottleneck.

You need some guidelines

Offline designers regularly create ‘brand guidelines’ or ‘design standards’ to document the precise way their brand or product should be recreated when outside of their control. Online, such guidelines are similarly invaluable for maintaining brand and code consistency with multiple engineers and designers, but it is blindingly obvious that a printed or ‘static’ set of guidelines is completely unsuitable for a constantly changing website.

[Read More]

Explaining Flux architecture with macgyver.js

What is Flux?

Flux is an application architectural pattern developed by Facebook. It was developed to solve some of the complexities of the MVC pattern when used at scale by favouring a uni-directional approach. It is a pattern and not a technology or framework.

MVC scale issue

When applications that use the model-view-controller (MVC) pattern at any scale it becomes difficult to maintain consistent data across multiple views. In particular the case whereby flow between models and views is not uni-directional and may require increasing logic to maintain parity between views when model data is updated. Facebook hit this issue several times and in particular with their unseen count (an incremented value of unseen messages which is updated by several UI chat components). It wasn’t until they realised that the MVC pattern accomodated the complexity that they stepped back from the problem and addressed the architecture.

Flux is intentionally unidirectional.

flux

[Read More]

Supporting IE8 in the OpenTable redesign

We’re really proud to have released last week our redesigned OpenTable site, the culmination of months of hard work from many talented people here in London and in San Francisco.

However despite killing off our old site and its 2004 design, 2.8% of our visitors could have been crying into their keyboard as a far worse opentable.co.uk filled their screen.

That version of OpenTable was our new responsive site viewed in Internet Explorer 8.

Our redesign before we optimised for IE8

The fundamental issue is that IE8 doesn’t support media queries so the age-old browser would try to stretch our mobile-first responsive design as wide as it could go - not great across a 27” Thunderbolt.

[Read More]

Proxying Services With Hapi.js

I’ve raved in the past about how awesome hapi.js is, but I’m going to talk about just a specific case today.

We started off with just a couple of hapi.js apis. This was at a time when standing up new infrastructure was still a bit painful, so inevitably those apis ended up having more functionality in them than they should have. Now it’s easy for us to get infrastructure, so we want to do more of it.

Our goal is to have lots of small(er) apis that just look after one specific piece (skillfully avoiding using the buzzword ‘microservices’).

When you want to split out functionality from one api to another, it can be a pain, especially if you have a lot of consumers who aren’t particularly fast-moving or communicative. Or maybe you don’t know all your consumers up front.

You’ve got a couple of options here:

[Read More]

Hobknob v2.0: A new dimension

Sometimes there is the requirement for more granularity when toggling a feature switch.
Version 2.0 of Hobknob hopes to address this with feature categories.

TL;DR.

Hobknob now allows you to define categories of features that have multiple toggles per feature.

For example, you can define the ‘Domain Features’ category which allows you to toggle a feature OFF in your-website.com, but ON in your-website.co.uk.

Domain Features

Categories

[Read More]

Interacting with ElasticSearch using Hubot

At OpenTable, we use a few ElasticSearch clusters. Our aim was to be able to interact with our ElasticSearch clusters via HipChat so that we could troubleshoot easily and without having to log into our VPN. We already use Hubot as part of our systems workflow, so it made sense to be able to interact with ElasticSearch with it.

Setting a cluster alias

When a pager wakes me at 3am, I really do not want to have to try and type the cluster URL into my mobile hipchat client. So the first thing that was added to the script was the ability to give a cluster an alias.

1
elasticsearch add alias my-test-alias http://my-cluster.com:9200

add-alias

This allows us to use that alias for all commands going forward. Please note that you can remove and query aliases as well:

[Read More]