How to use Vagrant to create Consul cluster

Last two articles about Consul service discovery involved one simple but extremely boring manual task: creating and configuring a cluster. In fact, I had to do it twice. I had to create three virtual machines, download and unpack Consul on them, find out their IP addresses, add configuration files and finally launch the binaries.

It’s dull. It’s boring. Humans shouldn’t do that kinds of things by hand. Seeing how easily we can automate creation of Docker containers with Dockerfile and docker-compose makes me wonder if we can do the same for hosts.

In fact, we can. Vagrant is a tool to do exactly that. It has it’s own Vagrantfile which can store configuration for one or more hosts and then bring them up to life with simple vagrant up.

What’s Vagrant

As official documentation says, Vagrant is a tool for building and managing virtual environments. If we ever need to configure a virtual machine either for development, for tests, or for some production needs, we can do that manually (and forget in a week how exactly we did that), or we can put VM configuration steps into a Vagrantfile. Not only we’ll be able to create a VM from that as many times as we need, we also could add it to version control system, share with the teammates, or test a VM locally before bringing identical machine to production. What’s cool, the same Vagrantfile used for creating a VM in, let’s say, VirtualBox on local machine, with little to no modifications can be used for bringing up AWS or Azure host.

The plan for today

Knowing how that I already had to create three almost identical VMs twice, it’s safe to expect that it might happen again, so.. let’s automate that! Let’s create a Vagrantfile to bring up three VirtualBox VMs with one Consul server and two regular agents on them. Obviously, server and agents will work as a cluster. Should we begin?

Prerequisites

As usual, before we can start we need to do some installation first. Mighty Google can point to download pages for both Vagrant and VirtualBox, and installation process itself is totally painless. I’m running both of the tools on Mac, but Windows and main flavors of Linux are also supported.

Step 0. Creating blank virtual machine

Creating new virtual machine with Vargrant starts with creating new Vagrantfile. It’s fairly easy to do:

Everything is pretty straightforward. init obviously creates the file. ubuntu/xenial64 is a box to use, and the box itself is like a base image in Docker. In our case the box is “64 bit Ubuntu 16.04 LTS”, but there’re many, many others. Finally, as init command tends to produce huge Vagrantfile with three meaningful lines and tons of comments, --minimal is the way to keep the file compact.

Now, type vagrant up and few minutes later you’ll have fully functional Ubuntu VM:

Vagrant used VirtualBox as default VM provider, but we could choose something else. E.g. Google Compute Engine would require --provider=google flag.

After VM is created we can get into it with vagrant ssh:

Step 1. Provisioning the VM

Now it’s time for installing and configuring Consul agents, and starting with Consul server seems like a logical choice. Here’s what we need to do:

  1. Give the VM a meaningful name, e.g. consul-server
  2. Download and unzip Consul binaries
  3. Assign static IP address (otherwise how other cluster members are going to find it?)
  4. Register and start consul as Ubuntu service

Step 1.1. Setting up a VM name

For this task we’ll need to make some changes in Vagrantfile. This is what init --minimal command created for us:

Setting up a VM name is just one more line in Ruby language:

Now let’s reload the VM with vagrant reload, SSH back into it and confirm that hostname indeed has changed:

Yup, it all worked.

Step 1.2. Shell provisioner for installing Consul

This task will be a little bit harder. The process of configuring the server is called provisioning, and Vagrant supports all sorts of ways to perform that. For instance, it supports shell provisioner for executing shell files, file for copying a file or folder, or even ansible, chef or puppet to do virtually anything.

For our task shell provisioner will do:

path points to the file with provisioning steps. We need to create that one as well:

However, if you try to call vagrant reload this time, the changes won’t be applied. The thing is Vagrant does provisioning only during VM creation. Or when it’s told to do so. So type vagrant reload --provision (or vagrant provision if VM was already reloaded), and enjoy newly deployed Consul:

Step 1.2.1. Optional: make provision script idempotent

There’s old Jedi trick allowing to apply provisioning script multiple times without nasty side effects or errors. It comes in handy when we add new features to provisioning stage and we want to make sure that existing ones don’t break anything during reprovisioning.

At the moment our provisioning script does only two things:

  • installs unzip
  • installs consul

If provision.sh will try to install unzip or consul only if they aren’t installed already, it will become idempotent.

Now reprosivioning VM will skip the steps that already has been taken thus avoiding ‘already exists’ sort of errors.

Step 1.3. Assigning static IP

Piece of cake. Two more lines in Vagrantfile and it’s ready:

Step 1.4. Starting Consul as Ubuntu service

This is actually hard. Not a rocket science, but also not entirely trivial.

In order to make Consul a service we have to declare it as systemd service. We can download sample service definition file from here and then copy it to /etc/systemd/system/ directory during provisioning. consul executable location itself also has to be changed from home folder to something more appropriate. I’ll skip most of the details (sources are available at github), but here’re some main points from this step.

  1. Vagrant mounts the whole directory with Vagrantfile to /vagrant path at guest OS, so we can use this mount for copying configuration files during provisioning. E.g.
  2. Consul agent as a service reads config files from /etc/systemd/system/consul.d/. It’s convenient, as now we can configure server and agents by putting some sort of init.json in there.
  3. Finally, service needs to be started. That’s one more provisioning line:

Step 1.5. Test run

If you haven’t fallen asleep so far, this is the right time to type vagrant reload --provision and go to 192.168.99.100:8500 in your browser of choice. There’s something new in there:

consul-from-vagrant

It’s true Consul server. You can stop the VM any time. You can even delete it. But as long as you have Vagrantfile, restoring ready to use Consul server is as easy as running vagrant up.

Step 2. Adding more VMs to the cluster

Vagrantfile is not limited to only one VM. By using define function we can declare as many VMs as we like:

As Vagrantfile itself is written in Ruby, we can use loops and functions to make multiple hosts creation easier. But before that happens, destroy existing consul-server with vagrant destroy -f, as we’re totally going to change the configuration file.

I think some refactoring also won’t do any harm. After all, configuring Consul server and regular agents is almost identical, so why not try to reuse the code:

This will allow to create Consul server with one function call:

And now just take a look what it takes to create two more virtual machines with fully functional Consul agents in them:

Isn’t that cool? We’re creating and configuring virtual machines in a loop!

But let’s check that new Vagrantfile indeed worked:

Vagrant create Consul cluster

The whole cluster was created from configuration file. I don’t have to do that manually anymore.

Here should come the conclusion…

But I can’t come up with any.

Well, maybe one thought. Somehow moving from manual to automatic creation and provisioning of VMs and hosts is hard. Not from technology point of view but from some sort of psychological inertia. “VMs are big and heavy, how can we automate them?”. Or even my favorite: “I know it’s one time job, I won’t need to do that again”. But there’s always the second time. At some point handcrafted environments start pushing us back, as they are not easily reproducible, often outdated and honestly, most of the time we have no idea what exactly is installed on them and how they manage to work.

Putting VM and host configuration to a file and creating new instances from it changes everything. Those hosts are predictable, always up to date and it’s not a big deal to create a new one. Vagrant is one of the tools that makes this magic possible. You can find all the source code from this post at github.

7 thoughts on “How to use Vagrant to create Consul cluster

  1. Hello pav, Thanks for the details tutorial
    I followed the same steps, vagrant reload –provision returns success but when i try to see result on UI using IP address http://192.168.99.100:8500/ I am always getting ERR_CONNECTION_REFUSED Am i missing something?

    1. Hey Vrushali,
      It’s probably something specific to your machine. I just git-cloned the source code for this article (https://github.com/pavel-klimiankou/vagrant-consul, haven’t touched it 2 years), did vagrant up in it, and, after it’s finished, 192.168.99.100:8500 was there. You can compare that code with what you have locally – maybe there’s some tiny difference.

Leave a Reply

Your email address will not be published. Required fields are marked *