Highly available Kafka cluster in Docker

Apache Kafka cluster in Docker

Up until now we’ve been experimenting with Apache Kafka, a tool build with cluster and high availability in mind, but using exactly one host and availability settings which only few very optimistic people would call high.

Not today.

Today we’re going to spin up multi-host Kafka cluster and we’ll replicate topic in it, so if one host goes down, data and its availability won’t suffer.

Continue reading “Highly available Kafka cluster in Docker”

“Hello world” with Apache Kafka

Single node cluster

So it’s time to send some data bits through Apache Kafka. But first, as usual, we need to install it first.

Installing Kafka is so trivial, so I’ll change my rule and will actually explain the process. Here goes the manual:

  1. Install Java Development Kit (you probably have it already)
  2. Download Kafka tarball
  3. Uncompress it ( tar -xzf kafka_2.11- in *nix systems)
  4. Done. You installed Kafka.

Continue reading ““Hello world” with Apache Kafka”

Quick intro to Apache Kafka

What is Apache Kafka

Official definition of Apache Kafka is distributed streaming platform, which starts to make sense only after reading at least few chapters of its documentation. However, idea behind it is relatively simple. In large distributed apps we have many services that produce messages: logs, monitoring events, audit entries – any type of records. On the other hand there’s similar amount of services that consume that data. Kafka brings these parties together: it accepts data from producers, reliably stores it in topics and allows consumers to subscribe to them. In other words, Kafka is a love child of distributed storage and messaging system.

Apache Kafka

Continue reading “Quick intro to Apache Kafka”