Using private registry in Docker Swarm

containersIn one of my previous posts about Docker health checks closer to the end of the post I managed to build a Dockerfile and run it as a service in Docker in Swarm mode. To be honest, I’m a little bit surprised that Docker allowed me to do that. That Swarm cluster could’ve had more than one host. What if the service went somewhere, where underlying image didn’t exist? Swarm node wouldn’t copy the image to the node that needs it, right? Or would it?

Let’s try replicating our service based on custom image across all hosts of multi-host Swarm cluster and see how that goes (spoiler: we’ll need private registry in order for that to work).

Attempt 1. Simply use locally built image

Create Swarm

As usual, I have docker-machine and VirtualBox installed, so creating a multi-host Swarm cluster becomes a trivial task. First, create one VM for manager node and other two for worker nodes:

Piece of cake. Then, let’s turn them to Swarm cluster by running docker swarm init in master node and executing ‘join’ command it produces on other two hosts:

Swarm is ready. Now let’s install helper viz service and get to business.

Install visualization service

When dealing with multiple hosts and services on them it’s useful to have some sort of visual feedback. That’s why I think having visualization service in a cluster is a good idea. First, I’ll connect local Docker client to Docker Engine in manager host, and then we can create viz service directly in cluster.

Checking master‘s port 8080 produces this nice picture, which means we’re safe to go.

visualization service

Building a custom image

Even though any Dockerfile would suffice, I picked the one with JS file from Docker health checks post. It’s just simpler for me and I know it worked at least once. So here’s the Dockerfile I used back then:

And server.js that accompanied it:

Because our local Docker client is still connected to master Docker Engine, we can simply build the image and expect that it’ll be available at least at one host in the cluster – master:

And now to the main part: creating and scaling the service based on custom image.

Starting the service in Swarm

Creating a service from custom image on single node that knows about the image is easy. I did it once, there’s no reason it shouldn’t work again.

By the way, service create output did warn us that the image was not taken from the registry.

Checking it in visualization service confirms it did work:

One instance of node server

And now the moment of truth: will it scale?

Hm, interesting. It did scale. But how? Let’s check what service’s tasks have to say:

Dirty cheater, it put all replicas onto the same host: master. But it’s really interesting to see a number of failed tasks, which did try deploying the service on worker nodes, but they all failed with quite predictable error No such image: server:latest. What a surprise. But at least we confirmed that Swarm won’t copy-paste custom images between its host and we need to find another approach.

Let’s remove the service and try something else.

Attempt 2. Use private registry in Swarm

The obvious something else is using some sort of in-swarm Docker Hub: private registry service. If it’s available within the cluster, we could push our custom image into it and refer to it during node-server creation. Creating private registry in general is fairly easy to do and something like docker run -d -p5000:5000 registry:latest  would do the trick. However, there’s a catch: HTTP based registry is only available from localhost, so if we need to make it available within the whole cluster, we need HTTPS and therefore – a trusted SSL certificate (I don’t want to use  insecure-registries Docker Engine flag or whatever it’s called now). (Update: as smart people mentioned, if registry is going to be pushed to and pulled from only from within the cluster, we still can go with HTTP – localhost is shared within the cluster). On the other hand, there’s couple of tricks we could use to get away with self-signed certificate, so here’s the plan:

  1. Generate self-signed SSL certificate for, let’s say, myregistry.com, which will point to our in-swarm registry.
  2. Force Docker to trust that certificate.
  3. Add myregistry.com to /etc/hosts of master and worker-* nodes, so they know where the registry is.
  4. Create registry service in HTTPS mode.
  5. Push server:latest image into the registry
  6. Create a service from the image from private registry.

That’s a lot of things to do, so let’s get started.

Generate self signed certificate

Fortunately, this part is trivial. At least on *nix systems:

It asked several questions along the way which I safely ignored, but one of them – Common Name – is important. That should be the name of our registry service: myregistry.com.

Force Docker to trust self-signed certificate

That’s one would be also simple to do, if it wasn’t 1 AM in my time zone. The secret is to place registry.crt file to Docker Engine’s certificates store. I did that in three steps:

  • Copy registry.crt file to Swarm host,
  • create a folder for the certificate,
  • move registry.crt into that folder.

This will make master node to trust the certificate, but we need to repeat this command for worker-1 and worker-2 as well. I’m almost certain that there’s much simpler way to do that, but after midnight this is as far as I can go.

Add myregistry.com to /etc/hosts

master node happened to have 192.168.99.100 IP address and this is the node that will host the registry, so myregistry.com should point to that IP.

This also needs to be done to worker nodes.

Create registry in HTTPS mode

registry service will need to have an access to certificate files and I didn’t want to use docker secret to store them (with no particular reason), so first we need to copy them to master node:

And now the time for our own Docker registry has come:

Behold! One more colored rectangle on the cluster map:

registry service

Push server image into private registry

We could’ve just added updated the tag of existing server image, but rebuilding the image with the right tag takes about the same amount of keystrokes, so why bother:

Because our image tag starts with registry URL, it’ll find the right way when we push it:

Now our image supposed to be in the registry hosted in Swarm. We’ll confirm that in a moment.

Create and scale Swarm service

The moment of truth:

And visualizer:

replicated node server

It worked! Scriptures didn’t lie and using private repository does solve the problem of custom images.

Summary

Using custom Docker images in Swarm is not that trivial as using them in standalone Docker host. Unless you deploy your image on every host in the cluster, which sounds quite unpractical, you’ll have to use private registry for it. Fortunately, with legit SSL certificate it’s fairly easy to do. Even with self-signed one, if you Swarm hosts are provisioned automatically (as they should), configuring them to trust to self-signed certificate is quite straightforward.

Leave a Reply

Your email address will not be published. Required fields are marked *