Quick intro to etcd

It feels like last months I focused way too much on debugging and .NET Core  and stopped paying attention to topic I enjoyed to blog about over the last year – DevOps and distributed applications. That doesn’t feel right and in order to fix that I’ll play with something new today. For instance, with etcd.

During my romance with distributed apps, etcd always was somewhere near. It came up as alternative to Consul when I was experimenting with service discovery and configuration management. At some point etcd also comes up as a storage where Kubernetes stores its cluster data. Etcd is everywhere! So I think it worth understanding what that is and how it looks like.

What’s etcd

etcd is primitive but highly available key-value storage. “Primitive” means that key-value is the only thing it can store, and “highly available” implies that even if one or few of etcd cluster nodes dies, the data will live. No more, no less. Comparing to Consul it has lesser amount of features, but sometimes in order to crack a nut you don’t need a sledgehammer and regular hammer will get the job done.

However, being a key-value storage doesn’t mean there’re just three etcd commands – get/set/remove – and the story is over. Far from that. It supports transactional assignment of values, ability to watch for value change, assign user roles, use synchronization primitives and even set Time To Live duration.

So let’s have a closer look.

Installation

The best of installation approaches – download to install – works for etcd too. As usual, because I’m lazy and don’t want to pollute my laptop with temporary stuff, I’ll create a Vagrantfile with download and unpacking instructions, so creating disposable etcd machine can happen in one command.

Here’s the Vagrantfile I ended up with.

The script is reasonably simple. The first highlighted block declares target version with download and install paths, and the second one simply downloads and extracts etcd itself and adds path to it to PATH. vagrant up  will do its usual stuff and after a minute or so of downloading and preparing new virtual machine we should be able to vagrant ssh and start doing something. For instance, checking etcd version number:

Starting one node etcd cluster

Official documentation provides ridiculously long command for starting even one node cluster. However, if we just want to play on local machine, starting etcd without any parameter at all works just as fine:

We can confirm that cluster is operational by asking for its members list with etcdctl command:

If I was going to use more than one machine or connect to remote etcd cluster I’d have to provide etcdctl with cluster URL (for instance -c http://localhost:2379). But again, it’s localhost, so who needs that.

Setting, getting and removing values

We can use very same etcdctl for manipulating key-value pairs. However, it supports more than one API version so ideally we should be specific about which one we’re going to use. For instance, in order to use the latest version (3) we need to set ETCDCTL_API environmental variable (export ETCDCTL_API=3).

That affects quite a lot. For instance, legacy API uses set command to set key value, whereas v3 uses put for that. In fact, the whole commands lists are totally different.

Speaking about put, here’s how we can add, query and remove key-value pairs:

Using HTTP endpoints

etcdctl obviously is not the only way to manipulate key-values. Despite under the hood etcd speaks gRPC, it’s still able to accept HTTP API calls via builtin grpc-gateway. Sometimes it feels a little bit weird and definitely HTTP API stuck somewhere at the bottom of Richardson Maturity Model, but it works. Even if it needs keys and values to be base64 encoded.

Here’s how adding mykey:myvalue key-value pair would looks like:

And now getting it back:

Looks… exotic. But again, it works.

Some other nice features

Watch for value change

What if we want to be notified when key’s value changes? No problem. We can even wait for value change of a key that doesn’t exist yet:

I remember Consul was doing something like this. Request will block until somebody updates shutdown-requested value..

…just to keep blocking again.

Transactions

Imagine you need to update a key only if some other conditions are met. For instance, a host will put its name in a leader key, but only of there’s no other leader yet. We can do that with txn command in interactive mode by explicitly saying what key needs to be tested, what to do if its value is ‘correct’, and what to do if it doesn’t.

Let’s start with scenario, when there leader key is already set.

When transaction hits the node, it obviously responds with FAILURE, as key’s value is different from what we expected – none. However, setting leader value to none will let the next transaction to succeed:

Conclusion

That’s etcd. Obviously, not all of it, but you’ve got the idea. I’m still going through its documentation and samples and it’s “Demos” section is probably the best one that I saw recently. The animation part is mostly useless and somewhat distracting, but having short code samples for every major feature really helps to get the idea of what this tool is about. I like that. Well, and etcd itself – clearly can see how I can use it in production.