by I don’t know if that’s a coincidence or not, but drastic changes in application metrics usually happen soon after a product upgrade was made. In fact, whenever I have to deal with new issue on production server, the first thing I do is checking if it was recently updated. No wonder it makes sense to record such events along with other monitoring data.
But assuming our monitoring data is in Graphite, how would we do that?
Events in Graphite
One of less known features in Graphite is support of events. Event is one time record about something, and as upgrade is definitely a something, we can store it directly in Graphite.
There’re two ways to create new event:
- via UI,
- programmatically, through JSON API.
Adding event through UI
The simplest way to create new event is through web UI. Graphite-web has little link in the top-right corner called ‘events’, which leads to the page of questionable aesthetics:
It has several fields, but probably the most important ones are the date and event tags. Tags are the primary way for searching for events, so it’s better not to skip it.
After you filled in the form, hit ‘save’ and enjoy newly created event in the list (we’ll put it on graphs later).
Adding event through API
The most practical way to create new event is through JSON API. After all, you’re not going to hire someone to enter new event every time the release is made. It’s a future, we have machines for that.
Graphite’s JSON API is clean and simple. Every field we saw in UI can be sent as JSON property via curl:
|
1 2 |
curl -X POST http://localhost/events/ \ -d '{"what": "Service patch", "tags": "update patch", "date": '`date +%s`', "data": "Yet another service update. v2.1"}' |
Btw, I grabbed the closest Graphite version I had, which also happened to be quite outdated one. In new versions tags property is an array, not whitespace separated string.
Now we can make sure that event is really there by querying it back:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
curl -s http://localhost/events/get_data | json_pp #[ # { # "tags" : "update", # "what" : "Service update", # "id" : 14, # "when" : 1485839100, # "data" : "From v1.0 to v2.0" # }, # { # "data" : "Yet another service update. v2.1", # "when" : 1485839795, # "id" : 15, # "what" : "Service patch", # "tags" : "update" # } #] |
Displaying events via graph designer
It’s not particularly obvious how to render events, so here’s the secret. Back at Graphite home page, click at ‘graph data’ button just below the graph itself and add drawAsInfinite(events('*')) line to the list. That should do the trick:
I used drawAsInfinite function to draw event as vertical line, because otherwise it would be just a point with value 1. events selects exactly what it sounds – events series. It accepts comma separated tags as a filter, or * to select everything.
We also can give a more reasonable name to displayed events by applying alias function: alias(drawAsInfinite(events('upgrade')), 'Upgrade'). And maybe add some other metrics to make the graph look important.
Yeah, that’s much better.
Displaying Graphite events in Grafana.
Grafana makes it easier to build dashboards for Graphite and other data sources. It also understands the concept of events. If you added Graphite as Grafana’s data source, head directly to current dashboard settings and choose ‘Annotations’ menu item.
Adding Graphite events here is really simple. You’ll only need to enter event tags to select (‘update’), choose meaningful name (‘Service Update’) and spend 5 to 15 minutes playing with the color.
But result totally worth that time:
Conclusion
Tracking application metrics is good, but tracking them along with events that can affect those metrics is even better. Releases, patches, hardware upgrades, troubleshooting sessions on live servers – all of these can and probably will affect metrics, so putting them into the same place will save you lots of efforts when debugging or troubleshooting comes.
