Building RabbitMQ Cluster

As I promised last time, it’s time to check out RabbitMQ feature we can consider advanced – clustering. RabbitMQ cluster is a set of individual nodes that share the same users, queues, exchanges and runtime parameters. New nodes can come and go, be located at different continents, yet for the connected client they will look like one entity.

Clustering is not the same as replication or high availability. Yes, users and whatever is usually necessary for node to work will be copied across all nodes. Queues, however, will reside on the node they were initially created, though they will be accessible from any node. If one node goes down, its queues go with him.

How to set up RabbitMQ cluster

Setting up a cluster is surprisingly simple. Basically, you’ll need to do two things:

1. Share the same Erlang cookie between all nodes, so they know they are buddies.
2. Call rabbitmqctl join_cluster command at all nodes except for the first one.

Erlang cookie is just a regular string typically stored at /var/lib/rabbitmq/.erlang.cookie. As for command  join_cluster , it takes only one argument – name of the node to connect to. Unless you run some experiments on config file, node name will be something like rabbit@hostname .

Prepare RabbitMQ nodes in Docker

Because we need at least two nodes to make up a cluster, the easiest way to get them is to start few Docker containers with RabbitMQ in them. There’re two families of images to choose from – rabbitmq and rabbitmq:management. I’ll pick the second one, as it comes with management plugin and web UI – an easy way to monitor individual nodes state and settings.

Another thing is, because now we’re talking about at least two containers, which have to reside in the same network, see each other and accept erlang cookie as a parameter – the easiest way to configure and to start them is through docker-compose . It’ll create containers and networks, connect ones to the other, assign names, parameters, etc. Clean and simple.

docker-compose configuration

So let’s start with docker-compose config file – docker-compose.yml.

It’s a little longer than ‘hello-world’, but I swear, it’s simple. Let’s see what it has:

1. Two services (containers) that will become RabbitMQ nodes called rabbit and hamster. Why hamster? It’s almost midnight, I’m too tired to google other members of rabbit family.
2. Host names are explicitly set to be the same as container names. Nodes in the cluster identify each other by host name, but container name is used to access it over the network. Make them equal and the problem is solved.
3. Admin web UI listens at port 15672 and I want to make it accessible from outside of the container. Because there’re two nodes and two UIs, the second one gets port 15673.
4. Official rabbitmq image accepts RABBITMQ_ERLANG_COOKIE parameter which will go directly to .erlang.cookie file, so we won’t have to do that manually. My cookie is ‘mysecret’, but that could’ve been any other string.

Starting nodes

After docker-compose.yml is ready, it’s time to start it:

Command will produce lots of output, but it’ll also start two independent nodes that share the same Erlang cookie. Before we move any further, let’s just make sure we can access their web management UI.

IP address will depend on operating system and version of Docker, but it will be either localhost, or whatever docker-machine ip  or boot2docker ip  tells you. Default login and password for web UI (as well as RabbitMQ itself) is guest:guest.

It seems like management page is working for the first node and I’m sure it’s also fine with the second one.

Setting up a cluster

It’s time to make a cluster. The second step in cluster formation is executing rabbitmqctl join_cluster command. For that we’ll get into, e.g. hamster, join it with rabbit and let interspecies breeding to do its magic. Unfortunately, docker-compose prefixes container names, so we needed to find hamster‘s exact name first.

It’s cluster_hamster_1. Now, joining the cluster:

Behold! The cluster has came to existence. Going back to admin page:

Now there’re two nodes visible at rabbit’s admin page. It’s officially a cluster.

Playing with the cluster

Let’s head to rabbit’s management page and create one queue in it, as well as one new user called ‘rabbit’:

Now if we go to hamster’s admin page, 127.0.0.1:15673, both new queue and user will also be there:

Queues page admits that ‘demoqueue’ is actually hosted at node called rabbit, but user account has no special marks. It’s true local user.

Not, let’s try out one more thing – stop rabbit:

If we go back to hamster’s users page, nothing will change and newly created user will still be there, but demoqueue, which was hosted at rabbit, is down.

This correlates with what I’ve said in the beginning: users are essential for node to run, so they are copied across all nodes in the cluster. Queues are not essential and stay wherever they were declared.

There’s one more observation we can make – there’s no ‘master’ node. All nodes are equal and can be brought up and down in any order. We stopped the first and presumably ‘main’ node, but cluster is still fine. The only exception is bringing the last node down – it also should the first one to come back.

Summary

Clustering in general is very tricky piece of functionality to implement, but it’s essential when you’re trying to scale your app horizontally. However, building a cluster of nodes in RabbitMQ is trivial. It’s just a matter of specifying a token and calling a single command. I’ve seen “hello-worlds” harder than that.

There’re few more details about configuration and features in the official RabbitMQ clustering guide, like disk and ram nodes, other cluster formation techniques and some limitations. Some folks also came up with Docker images which are cluster-ready, e.g. docker-rabbitmq-cluster. Along with Erlang cookie parameter, they accept name of the node to join with, so single command can start the whole cluster.