Debugging .NET Core app from a command line on Linux

command line debugging

Million years ago, way before the ice age, I was preparing small C++ project for “Unix Programming” university course and at some point had to debug it via command line. That was mind blowing. And surprisingly productive. Apparently, when nothing stands in the way, especially UI, debugging can become incredibly focused.

Since .NET Framework got his cross platform twin brother .NET Core, I was looking forward to repeat the trick and debug .NET Core app on Ubuntu via command line. Few days ago it finally happened and even though it wasn’t a smooth ride, that was quite an interesting experience. So, let’s have look.

Continue reading “Debugging .NET Core app from a command line on Linux”

The mystery of “Debug adapter process has terminated unexpectedly”

VS Code

Interesting story happened to me the other day. Visual Studio Code, my primary C# editor on Linux, suddenly stopped working. Well, it’s debugger did. Whenever I put a breakpoint and started debugging, nice friendly message would appear in the top of the editor, saying:

Debug adapter process has terminated unexpectedly

And it all worked just fine before. As recently I installed a whole bunch of new tools on my Ubuntu, such as lldb, perf and lttng, I played with removing and re-adding them again, reinstalling VS Code itself, but nothing seemed to help. Surprisingly, another C# IDE that supports Linux – JetBrains Rider – also failed to debug the projects I needed (worked fine with few others, though), so I had no other choice but try to get to the bottom of it, or at least make VS Code work again. Continue reading “The mystery of “Debug adapter process has terminated unexpectedly””

Sending .NET Core app metrics to Graphite with StatsD

time valueIt’s been more than a year since I connected a small piece of JavaScript to collectd plugin and started to gather our CI’s monitoring data and store it in Graphite. Surprisingly, the whole thing worked like a charm. Even the JavaScript component.

However, the time comes when I need to collect even more data coming from inside of long running apps, so JavaScript + collectd pair is no longer an option. What might work is those apps sending their metrics directly to Graphite server. After all, it can accept data in plain string format via TCP, so that shouldn’t be hard. Continue reading “Sending .NET Core app metrics to Graphite with StatsD”

Profiling .NET Core app on Linux

flame graphI’m still busy learning how to troubleshoot .NET Core apps on Linux. Now I more or less understand where to start if I need to analyze .NET Core memory usage, but that’s not the most frequent problem we usually have. Most of the time the main problem is high CPU usage, which obviously leads to the question “What this thing’s doing now??”.

On Windows I’d usually start with analyzing application logs, trace files or performance reports. If that wasn’t enough I’d download Perfview tool and profiled running executable itself. How many threads are there? What functions they execute the most? What their call stacks are? And so forth. The only downside of that tool is that I never read its documentation, so whole troubleshooting experience sometimes resembled a meditation on the monitor, worshiping the holy Google and wild guessing.

While logs and traces are obviously there on Linux as well, I was wondering, can I profile the app in the similar way I would do on Windows? The answer is “Oh, yes”. Monitor, worshiping, guessing – it’s all there.

There’re multiple tools to use out there, but the basic toolkit for profiling .NET Core app on Linux seems to be perf utility along with lttng and perfcollect. Let’s have a look at all of them. Continue reading “Profiling .NET Core app on Linux”

Analyzing .NET Core memory on Linux with LLDB

net-linux-memory

Most of the last week I’ve been experimenting with our .NET Windows project running on Linux in Kubernetes. It’s not as crazy as it sounds. We already migrated from .NET Framework to .NET Core, I fixed whatever was incompatible with Linux, tweaked here and there so it can run in k8s and it really does now. In theory.

In practice, there’re still occasional StackOverflow exceptions (zero segfaults, however) and most of troubleshooting experience I had on Windows is useless here on Linux. For instance, very quickly we noticed that memory consumption of our executable is higher than we’d expect. Physical memory varied between 300 MiB and 2 GiB and virtual memory was tens and tens of gigabytes. I know in production we could use much higher than that, but here, in container on Linux, is that OK? How do I even analyze that?

On Windows I’d took a process dump, feed it to Visual Studio or WinDBG and tried to google what’s to do next. Apparently, googling works for Linux as well, so after a few hours I managed learn several things about debugging on Linux and I’d like to share some of them today. Continue reading “Analyzing .NET Core memory on Linux with LLDB”