Tracking application events in Graphite

I don’t know if that’s a coincidence or not, but drastic changes in application metrics usually happen soon after a product upgrade was made. In fact, whenever I have to deal with new issue on production server, the first thing I do is checking if it was recently updated. No wonder it makes sense to record such events along with other monitoring data.

But assuming our monitoring data is in Graphite, how would we do that?

Events in Graphite

One of less known features in Graphite is support of events. Event is one time record about something, and as upgrade is definitely a something, we can store it directly in Graphite.

There’re two ways to create new event:

  • via UI,
  • programmatically, through JSON API.

Adding event through UI

The simplest way to create new event is through web UI. Graphite-web has little link in the top-right corner called ‘events’, which leads to the page of questionable aesthetics:

Graphite: home page

Graphite: events

Graphite: add event

It has several fields, but probably the most important ones are the date and event tags. Tags are the primary way for searching for events, so it’s better not to skip it.

After you filled in the form, hit ‘save’ and enjoy newly created event in the list (we’ll put it on graphs later).

Graphite events list

Adding event through API

The most practical way to create new event is through JSON API. After all, you’re not going to hire someone to enter new event every time the release is made. It’s a future, we have machines for that.

Graphite’s JSON API is clean and simple. Every field we saw in UI can be sent as JSON property via curl:

Btw, I grabbed the closest Graphite version I had, which also happened to be quite outdated one. In new versions tags property is an array, not whitespace separated string.

Now we can make sure that event is really there by querying it back:

Displaying events via graph designer

It’s not particularly obvious how to render events, so here’s the secret. Back at Graphite home page, click at ‘graph data’ button just below the graph itself and add drawAsInfinite(events('*')) line to the list. That should do the trick:

Graphite: render events

I used drawAsInfinite function to draw event as vertical line, because otherwise it would be just a point with value 1. events selects exactly what it sounds – events series. It accepts comma separated tags as a filter, or * to select everything.

We also can give a more reasonable name to displayed events by applying alias function: alias(drawAsInfinite(events('upgrade')), 'Upgrade'). And maybe add some other metrics to make the graph look important.

Graphite: render events with CPU metrics

Yeah, that’s much better.

Displaying Graphite events in Grafana.

Grafana makes it easier to build dashboards for Graphite and other data sources. It also understands the concept of events. If you added Graphite as Grafana’s data source, head directly to current dashboard settings and choose ‘Annotations’ menu item.

Grafana annotations

Adding Graphite events here is really simple. You’ll only need to enter event tags to select (‘update’), choose meaningful name (‘Service Update’) and spend 5 to 15 minutes playing with the color.

Grafana: add annotation

But result totally worth that time:

Gafana: view annotations

Conclusion

Tracking application metrics is good, but tracking them along with events that can affect those metrics is even better. Releases, patches, hardware upgrades, troubleshooting sessions on live servers – all of these can and probably will affect metrics, so putting them into the same place will save you lots of efforts when debugging or troubleshooting comes.

Leave a Reply

Your email address will not be published. Required fields are marked *