Visualize Elasticsearch data with Kibana

KibanaToday we’ll take a look at the last component of Elastic’s ELK stack – Kibana. Even though Logstash does a great job of processing logs and other data streams, and Elasticsearch is a powerful hybrid of a search index and a storage for them, these tools do not provide graphical user interface for analyzing the data. For some tasks otherwise convenient command line interface is just not enough. This is where Kibana steps in.

Continue reading “Visualize Elasticsearch data with Kibana”

Quick intro to Elasticsearch

ElasticsearchSo far we’ve been dealing with name-value kind of monitoring data. However, what works well for numeric readings isn’t necessarily useful for textual data. In fact, Grafana, Graphite and Prometheus are useless for other kind of monitoring records – logs and traces.

There’re many, many tools for dealing with those, but I decided to take a look at Elastic’s ELK stack: Elasticsearch, Logstash and Kibana – storage, data processor and visualization tool. And today we’ll naturally start with the first letter of the stack: “E”.

What’s Elasticsearch

Elasticsearch is fast, horizontally scalable open source search engine. It provides HTTP API for storing and indexing JSON documents and with default configuration it behaves a little bit like searchable NoSQL database.

Continue reading “Quick intro to Elasticsearch”

Scraping application metrics with Prometheus

Prometheus logoThere’re two conceptually different approaches in collecting application metrics. There’s PUSH approach, when metrics storage sits somewhere and waits until metrics source pushes some data into it. For instance, Graphite doesn’t do any collection on its own, it waits until somebody like collectd does the delivery.

There’s second approach – PULL. In this approach metrics sources don’t try to be smart and just provide their readings on demand. Whoever needs those metrics can make a call, e.g. HTTP request, in order to get some.

Prometheus collects metrics using the second approach. Continue reading “Scraping application metrics with Prometheus”

Building dashboards with Grafana

Even though Graphite does very decent job in displaying individual metrics graphs, its dashboards support is quite limited. Of cause, we could take its powerful Render URL API and build anything we like in good old HTML, but on the other hand, there’s Grafana.

Grafana dashboard

Continue reading “Building dashboards with Grafana”

Quick intro to rrdtool

I mentioned in previous post that collectd uses rrdtool for saving its data by default. It results .rrd  file for each metric, which later can be rendered using very same rrdtool. RRD files are not something most of the people are familiar with and the tool itself isn’t particularly easy to use, so why such an easy to use tool as collectd would choose it?

For a number of reasons. Continue reading “Quick intro to rrdtool”

Quick intro to Apache Kafka

What is Apache Kafka

Official definition of Apache Kafka is distributed streaming platform, which starts to make sense only after reading at least few chapters of its documentation. However, idea behind it is relatively simple. In large distributed apps we have many services that produce messages: logs, monitoring events, audit entries – any type of records. On the other hand there’s similar amount of services that consume that data. Kafka brings these parties together: it accepts data from producers, reliably stores it in topics and allows consumers to subscribe to them. In other words, Kafka is a love child of distributed storage and messaging system.

Apache Kafka

Continue reading “Quick intro to Apache Kafka”

Quick intro to RabbitMQ

Quick intro to RabbitMQ

RabbitMQ is an example of full blown Message Queue that somehow remained simple to use. Unlike ZeroMQ, which is embeddable into the services that use it, RabbitMQ is a broker. It’s an intermediary messaging service with own users, permissions, encryption, configurable durability and delivery acknowledgements, clustering, high availability, and bazillion of other features you might never need. RabbitMQ is built on top of Erlang and inherits its known resilience with compatibility to virtually any OS.

In the following article we’ll try to get a sense of how messaging with RabbitMQ feels like. I’ve chosen Ubuntu (in a Docker container) as a platform, but it could’ve been anything else. Continue reading “Quick intro to RabbitMQ”

Quick intro to Windows containers

Windows containers

It finally happened. With release of Windows Server 2016 you can run Docker containers with Windows inside. There’s no Virtual Machine hiding somewhere in order for that to happen, or some sort of Windows emulation built on top of Linux core. It’s true Windows in true Docker, which supports Dockerfiles, docker-compose and other docker-goodies. Continue reading “Quick intro to Windows containers”

Quick intro to docker-compose

What is docker-compose

docker-compose logoLike docker itself allows managing single container, docker-compose makes it easy to control not just one, but all containers that make distributed app. This includes containers, networks, volumes and all related settings.

If you think about it, starting an app that has more than one container is less than trivial task and it gets exponentially harder as you add more or them. Let’s check out simple example: distributed web-application that consists of two containers – one with web content and one with database.

Continue reading “Quick intro to docker-compose”