Using Consul Key-Value Store for Service Configuration

How do you usually configure an app? Over the decades our industry came up with multiple approaches, like providing command line arguments, various config files, registry settings and environment variables. Even hardcoding certain options into the app itself also works sometimes. Whenever we need to reconfigure the app we’d just go to its host, change a setting or two, and the job is done.

And now imagine a challenge: you have a cluster of services that come and go, they have different versions and locations, and you need to reconfigure all of them. How would you do that?

Serving configuration with Consul

Consul is scalable and highly available tool developed by HashiCorp that helps distributed applications in many ways. It helps services to discover each other, performs health checks on them, acts as DNS service and even provides a reliable key-value storage. Such storage is a convenient place for storing all sorts of configuration options that individual services can use. Moreover, Consul comes with set of tools that can generate and update config files on the fly, which makes it perfect for reconfiguring large amount of services from single place.

Installation

Consul works on all popular platforms and comes as standalone executable. Using Docker image is also an option. After downloading an archive, unpack it and start with the following arguments:

This will start local Consul agent in server mode. -dev flag puts it into development mode, so when it exits, no traces will be left.

Consul agent also comes with UI, which we can observe at port 8500:

Consul Key-Value Store

As I mentioned before, one of the features that consul provides is a key-value store. Keys can be organized into some sort of a tree by separating its “path” components with forward slash:

Manipulating key-value data

Obviously, we can get and set the data into the store via web UI. But from automation perspective manipulating data programmatically is much better choice. We can access the data via RESTful API, consul kv utility or numerous API clients.

Reading a value is extremely simple. It’s a regular GET request to /v1/kv endpoint followed by the key:

Response’s Value property is Base64 encoded, but it holds the same value as seen in web UI, I checked.

It’s also possible to list all key-value pairs recursively:

Putting new key-value pair just requires another HTTP verb:

The command above will create web/config/experimental key with enabled value.

We also could send another PUT request to update the value, or DELETE request to nuke it.

Similarly, consul kv will do the same get/put/delete operations, but with shorter syntax and without Base64 encoding:

Long polling for receiving values updates

Let’s assume we decided to configure an app with an option downloaded from Consul KV via HTTP request. If that option can change, and the app can handle reconfiguration without restarting itself, it would make sense to continue making HTTP requests with some interval. However, choosing the right interval might be tricky. Too short, and we’d be hammering the network and the server for nothing. Too long, and most likely updated setting will get to us later than sooner.

There’s a third option: we can make a call and wait until new value comes.

Let’s make another call to key-value store, but this time examine HTTP response headers. Along with standard ones, Consul will add something on its own, including X-Consul-Index header:

If we include the last value of X-Consul-Index header into the next KV GET request, the server will pause the request until it has something new to respond with, or timeout happens. Default timeout is five minutes, but you can provide your own with wait= query string parameter, up to 10 minutes.

With the long polling we can keep a number of requests to Consul agent low, yet still receive updated values with little to no delay.

Linking config files to Key-Value store

Imagine we have config.json file with keys like this:

If you remember, we have the same keys in our key-value store. Wouldn’t it be great if this config file was generated using the keys and values from the store and kept in sync with it? Apparently, consul-template can do that.

consul-template is another downloadable tool from HashiCorp. It comes as a zip archive and works out of the box. What it does is it takes a template file with placeholders pointing to KV entries, processes it and saves in a new file with values populated. It can run in a loop, thus providing near real time sync with the source of configuration truth – Consul KV store.

Simple substitutions

Coming back to config.json file. Let’s create a template file – config.tpl – and put the following content in it:

It’s basically the same config.json file with values replaced with handlebars-like placeholders.

Now let’s run consul-template against it:

Isn’t that cool?! It connected to local Consul instance, picked the values, saved the file and exited. Just a side note, we also could’ve point it to remote Consul server.

If we remove -once from the arguments, consul-template will keep running, waiting for new values and applying them as soon as they come. Simple and brilliant.

Complex substitutions

We also could do something more complex than simple values substitutions. What if we don’t know how many options there’re in KV store, but we need to put all of them into the config file? Well, we could list all sub keys in db/config, iterate through them and write whatever we found into the output:

We also could use if statement to remove trailing comma at last element, but that would be too much for this example.

Apart from just rendering the file, consul-template can be responsible for launching an executable after config for it is ready. Moreover, it can send configurable ‘reload’ signal when configuration changes.

Summary

Configuring a swarm of services somewhere in a cloud can be very tricky. It’s not a problem to provide an initial configuration, but keeping it up to date or making a change simultaneously in many services is really hard. Tools like Consul and consul-template can make this task much more manageable. We can use Consul Key-Value store as a storage for configuration elements and allow services to query them via HTTP or API clients. Or even better, we could outsource this task to consul-template and let it keep service configuration file up to date with the single source of truth – Key-Value store.