Using Consul for Service Discovery

Consul logoImagine your distributed app has two kinds of services: web and db. Both of them are replicated for higher availability, live on different hosts, go online and offline whenever they like. So, here’s a question: how do web‘s find db‘s?

Obvious solution would be to come up with some sort of reliable key-value storage, and whenever service comes online, it would register itself with the address in the store. But what happens when service goes offline? It probably could notify the store just before that, but c’mon, it’s internet: things can go offline without any warning. OK, then we could implement some sort of service health checks to ensure that they are still available… By the way, did you notice how quickly the simple idea of using external store for service discovery started to become a reasonably large infrastructure project?

Service discovery is something very hard to do. But we don’t have to – there’re tools for that, and Consul is one of them. Continue reading “Using Consul for Service Discovery”

Using Consul Key-Value Store for Service Configuration

Consul logoHow do you usually configure an app? Over the decades our industry came up with multiple approaches, like providing command line arguments, various config files, registry settings and environment variables. Even hardcoding certain options into the app itself also works sometimes. Whenever we need to reconfigure the app we’d just go to its host, change a setting or two, and the job is done.

And now imagine a challenge: you have a cluster of services that come and go, they have different versions and locations, and you need to reconfigure all of them. How would you do that?

Continue reading “Using Consul Key-Value Store for Service Configuration”

“Hello world” with Apache Kafka

Single node cluster

So it’s time to send some data bits through Apache Kafka. But first, as usual, we need to install it first.

Installing Kafka is so trivial, so I’ll change my rule and will actually explain the process. Here goes the manual:

  1. Install Java Development Kit (you probably have it already)
  2. Download Kafka tarball
  3. Uncompress it ( tar -xzf kafka_2.11- in *nix systems)
  4. Done. You installed Kafka.

Continue reading ““Hello world” with Apache Kafka”

Quick intro to Apache Kafka

What is Apache Kafka

Official definition of Apache Kafka is distributed streaming platform, which starts to make sense only after reading at least few chapters of its documentation. However, idea behind it is relatively simple. In large distributed apps we have many services that produce messages: logs, monitoring events, audit entries – any type of records. On the other hand there’s similar amount of services that consume that data. Kafka brings these parties together: it accepts data from producers, reliably stores it in topics and allows consumers to subscribe to them. In other words, Kafka is a love child of distributed storage and messaging system.

Apache Kafka

Continue reading “Quick intro to Apache Kafka”

Building RabbitMQ Cluster

Cluster with RabbitMQ

As I promised last time, it’s time to check out RabbitMQ feature we can consider advanced – clustering. RabbitMQ cluster is a set of individual nodes that share the same users, queues, exchanges and runtime parameters. New nodes can come and go, be located at different continents, yet for the connected client they will look like one entity.

Clustering is not the same as replication or high availability. Yes, users and whatever is usually necessary for node to work will be copied across all nodes. Queues, however, will reside on the node they were initially created, though they will be accessible from any node. If one node goes down, its queues go with him.

Continue reading “Building RabbitMQ Cluster”

Quick intro to RabbitMQ

Quick intro to RabbitMQ

RabbitMQ is an example of full blown Message Queue that somehow remained simple to use. Unlike ZeroMQ, which is embeddable into the services that use it, RabbitMQ is a broker. It’s an intermediary messaging service with own users, permissions, encryption, configurable durability and delivery acknowledgements, clustering, high availability, and bazillion of other features you might never need. RabbitMQ is built on top of Erlang and inherits its known resilience with compatibility to virtually any OS.

In the following article we’ll try to get a sense of how messaging with RabbitMQ feels like. I’ve chosen Ubuntu (in a Docker container) as a platform, but it could’ve been anything else. Continue reading “Quick intro to RabbitMQ”

Using ZeroMQ with Docker

ZeroMQ, Node.js and Docker

Last time we built three client-server Node.js apps that were talking to each other using ZeroMQ. However, running both client and server on localhost is a little bit lame. Let’s put them into containers! They’ll still be lame, but now with Docker around.

So, here’s the plan: let’s see what we need to do to last week’s fire-and-forget ZeroMQ app, so its client and server can work and communicate from within Docker containers. Continue reading “Using ZeroMQ with Docker”

Inter-service messaging with ZeroMQ and Node.js


ZeroMQ is small, fast and very easy to use messaging library, which works equally well both within the same process and over the network. Despite being written in C++, it has bindings for most of the languages you can come up with. And it’s free. Hurrah!

Working with ZeroMQ resembles working with TCP/UDP sockets. In fact, ZeroMQ endpoints are called sockets. You create one, bind it or connect to certain address, and then magic begins… But let’s see some code first and dive into the details as we go. Continue reading “Inter-service messaging with ZeroMQ and Node.js”

Asynchronous communication with message queue

In previous posts I mentioned ‘microservice communication’ quite often, but never actually explained how exactly that happens.

Because of services isolation, choices of transport are limited and most likely you’ll be using TCP or UDP to talk over the network. But how exactly this will happen depends on what patterns or tools you choose.

Patterns of inter-service communication

There’re two of them: synchronous and asynchronous communication. Continue reading “Asynchronous communication with message queue”

Quick intro to docker-compose

What is docker-compose

docker-compose logoLike docker itself allows managing single container, docker-compose makes it easy to control not just one, but all containers that make distributed app. This includes containers, networks, volumes and all related settings.

If you think about it, starting an app that has more than one container is less than trivial task and it gets exponentially harder as you add more or them. Let’s check out simple example: distributed web-application that consists of two containers – one with web content and one with database.

Continue reading “Quick intro to docker-compose”