Sending .NET Core app metrics to Graphite with StatsD

It’s been more than a year since I connected a small piece of JavaScript to collectd plugin and started to gather our CI’s monitoring data and store it in Graphite. Surprisingly, the whole thing worked like a charm. Even the JavaScript component.

However, the time comes when I need to collect even more data coming from inside of long running apps, so JavaScript + collectd pair is no longer an option. What might work is those apps sending their metrics directly to Graphite server. After all, it can accept data in plain string format via TCP, so that shouldn’t be hard.

And here comes statsd

But there’s even simpler way. If you check out the latest Graphite Docker image, you’ll notice that it comes with preinstalled tool called StatsD. It’s a little buffer between the app and Graphite, which accepts metrics via UDP, aggregates them, and then puts to Graphite on certain intervals. Because its UDP, applications can fire-and-forget their metrics extremely fast. Because StatsD does buffering and aggregation, we lessen the burden on Graphite itself and therefore can deal with more data sources.

Finally, StatsD is dead simple and can work with different backends, so applications can use it as a unified metrics receiver and stop caring about where the data is going to be saved.

StatsD and .NET Core

There’s nothing special in relationship of StatsD and .NET Core. But because I have to deal with .NET Core a lot now, I’m particularly interested how those two fit together.

The list of StatsD packages on NuGet is a little bit messy. Some of them work with .NET Framework only, the other’s hasn’t been maintained for quite a while. But I found one which is relatively alive and does the job: StatsdClient.

The best way to see how something works is to try to make it work by yourself, so let’s build something.

The grand example

Sample program

I have a simple .NET Core program that runs in two threads: one that does random amount of meaningless job (generating strings) in a loop, and the other one that monitors the rate and the number of garbage collections happening, and how much of the memory we use in total:

It would be better to monitor GC and memory from outside (e.g. via EventTrace), but it also would make example more complex, so probably the next time.

Ok, let’s start the program to see something like this:

Nice numbers, probably mean something. But they would mean much more if we could make a graph of it. Graphite could do that, so let’s install it.

Installing Graphite

Since I installed Graphite the last time, it’s got official Docker image, so instead of several download and configure steps this one-liner will do everything:

For this demo, however, I don’t need container name, restart policy and ports other than 80 (Graphite UI) and 8125/udp (StatsD), so we can simplify the command to something like this:

Now we can open the browser and voilà, here’s Graphite and its glorious emptiness:

Clean Graphite

Time to feed some data to it.

Feeding the data via StatsdClient

Installing StatsdClient package introduces new Metrics class, which I simply used to replace Console.WriteLine‘s (and StopWatch). Resulting code became even simpler than it was before:

Changes are pretty straightforward. Firstly, we need to tell StatsdClient where Graphite server is. Secondly, Graphite metrics names usually look somewhat like this: myserver.myapp.mysensor.mySensorComponent, which then resolves into nice folder-like structure in Graphite UI. Metrics‘s Prefix property simply sets the first part of the name, and the latter will come when we actually start feeding the data.

Finally, Time, GaugeAbsoluteValue and Counter are methods for sending the data to StatsD daemon, and we’ll look at these three a little bit closer.

GaugeAbsoluteValue

This methods accepts gauge-like type of readings, which basically is raw sensor value: temperature, free disk space, total amount of memory, etc. Very same value that you just sent to the storage, you’ll see as is at a graph later. The name of our gauge is totalMemory (line 21), and assuming the newer version of demo program has been running for a while, we’ll see a nice memory graph:

"Gauge" metrics

Counter

Counters are a little bit trickier, because they display not the value itself, but rather the rate it changes. E.g. X units per second/minute/etc. That’s much more informative for some metrics like number if incoming requests or triggered garbage collections.

At line 23 we record how many GCs happened since the last check and we do the check every 100ms (line 24). Recorded values varied between 5 and 17, so I think the rate per second that will be around 100-120 GCs. Checking the graph.. and it seems like I’m almost right.

"Counter" metrics

Time

Using “Time” for metric name is a bit misleading, because it actually doesn’t have to be about time. It would also work for request bytes, or shopping card dollars. StatsD aggregates “Time” readings within its buffering interval (10s by default) and then sends some calculated statistics to Graphite, like:

  1. total count of samples collected during 10s interval,
  2. min sample value,
  3. max sample value,
  4. median,
  5. etc.

In our case we measured job duration, so at the graph we can see the longest, shortest, “usual” and several other durations:

"Time" metrics

Neat.

One more data type

There’s also Set data type, which we haven’t used here. It’s actually very simple: if we record user ids visiting our web site, Set would calculate number of unique visitors during aggregation interval (again, 10s by default). Nothing more, nothing less.

Conclusion

There’s no doubt I could send my metrics to Graphite directly. But as StatsD already comes preinstalled with Graphite’s Docker image and it’s so easy to use, it’s really looks like a default choice. Fan fact. When I was experimenting with it in .NET Core, there were not much of an experiment at all. I just added NuGet package to the project, replaced function calls and it just worked. And I know for sure I’d definitely make a mistake or two if I started with TcpClient and NetworkStreams for talking directly to Graphite.

Leave a Reply

Your email address will not be published. Required fields are marked *