It feels like last months I focused way too much on debugging and .NET Core and stopped paying attention to topic I enjoyed to blog about over the last year – DevOps and distributed applications. That doesn’t feel right and in order to fix that I’ll play with something new today. For instance, with etcd.
During my romance with distributed apps, etcd always was somewhere near. It came up as alternative to Consul when I was experimenting with service discovery and configuration management. At some point etcd also comes up as a storage where Kubernetes stores its cluster data. Etcd is everywhere! So I think it worth understanding what that is and how it looks like.
Continue reading “Quick intro to etcd”
llnode plugin which does exactly that. Today I’m not going to go deeply into how it works – it’s not in the focus of my daily job now. But it’s still interesting to get a feeling of it. So, let’s dive right in. Continue reading “Examining NodeJS core dump with llnode and lldb”
Interesting story happened to me the other day. Visual Studio Code, my primary C# editor on Linux, suddenly stopped working. Well, it’s debugger did. Whenever I put a breakpoint and started debugging, nice friendly message would appear in the top of the editor, saying:
And it all worked just fine before. As recently I installed a whole bunch of new tools on my Ubuntu, such as lldb, perf and lttng, I played with removing and re-adding them again, reinstalling VS Code itself, but nothing seemed to help. Surprisingly, another C# IDE that supports Linux – JetBrains Rider – also failed to debug the projects I needed (worked fine with few others, though), so I had no other choice but try to get to the bottom of it, or at least make VS Code work again. Continue reading “The mystery of “Debug adapter process has terminated unexpectedly””
I’ve been looking through the latest Technology Radar issue and here’s what I found in its new Techniques section: “TDD’ing containers”. Wow. Mentally, I’m not yet ready to connect TDD to containers, but I took a look at the tools used for that, and those are quite interesting.
The first one is serverspec, which allows running BDD-like tests against local or remote servers or containers. It looks pretty solid, supports multiple OSs and its only downside (for me) is that serverspec is written in Ruby and therefore doesn’t really fit in the stack I normally work with.
The other one –
goss – leaves an impression of a multitool, which usually worries me, but here… I’m kind of curious, so let’s have a look. Continue reading “How to unit test.. a server with goss”
I’m still busy learning how to troubleshoot .NET Core apps on Linux. Now I more or less understand where to start if I need to analyze .NET Core memory usage, but that’s not the most frequent problem we usually have. Most of the time the main problem is high CPU usage, which obviously leads to the question “What this thing’s doing now??”.
On Windows I’d usually start with analyzing application logs, trace files or performance reports. If that wasn’t enough I’d download Perfview tool and profiled running executable itself. How many threads are there? What functions they execute the most? What their call stacks are? And so forth. The only downside of that tool is that I never read its documentation, so whole troubleshooting experience sometimes resembled a meditation on the monitor, worshiping the holy Google and wild guessing.
While logs and traces are obviously there on Linux as well, I was wondering, can I profile the app in the similar way I would do on Windows? The answer is “Oh, yes”. Monitor, worshiping, guessing – it’s all there.
There’re multiple tools to use out there, but the basic toolkit for profiling .NET Core app on Linux seems to be
perf utility along with
perfcollect. Let’s have a look at all of them. Continue reading “Profiling .NET Core app on Linux”
Most of the last week I’ve been experimenting with our .NET Windows project running on Linux in Kubernetes. It’s not as crazy as it sounds. We already migrated from .NET Framework to .NET Core, I fixed whatever was incompatible with Linux, tweaked here and there so it can run in k8s and it really does now. In theory.
In practice, there’re still occasional StackOverflow exceptions (zero segfaults, however) and most of troubleshooting experience I had on Windows is useless here on Linux. For instance, very quickly we noticed that memory consumption of our executable is higher than we’d expect. Physical memory varied between 300 MiB and 2 GiB and virtual memory was tens and tens of gigabytes. I know in production we could use much higher than that, but here, in container on Linux, is that OK? How do I even analyze that?
On Windows I’d took a process dump, feed it to Visual Studio or WinDBG and tried to google what’s to do next. Apparently, googling works for Linux as well, so after a few hours I managed learn several things about debugging on Linux and I’d like to share some of them today. Continue reading “Analyzing .NET Core memory on Linux with LLDB”
So far all examples I made for Docker in Swarm Mode or Kubernetes blog posts were built around some sort of a service: web server, message queue, message bus. After all, “service” is a main concept in Swarm Mode, and even the whole micro-service application thing has, well, a “service” in it. But what about one-off jobs: maintenance tasks, scheduled events, or anything else, that we need to run just sometimes, not as a service?
Continue reading “One-off Kubernetes jobs”
Much to my surprise, starting from the last week Kubernetes became the part of my job description. It’s no longer something just interesting to try, I actually have to understand it now. And as you probably could tell from my older k8s post, I’m not quite there. The post sort of builds a logical example (containerized web server) but something just doesn’t click.
I was trying to understand what’s missing, and it seems like the problem is in the tooling. You see, there’re two and a half ways to run something in Kubernetes. One is through ad-hoc commands, like
kubectl run or
kubectl expose. They are simple, but they also skip few important concepts happening in the background, so the whole picture stays unclear. Continue reading “Dissecting Kubernetes example”
I’ve been using Gitlab CI for a while now and until certain point it worked really well. We had three build servers (GitLab runners) in the beginning, and when number of teammates or build steps and therefore commits and build jobs increased, I’d just add one more server to handle an extra load and felt that problem was solved.
Not for long. When number of servers climbed to more than ten, it became obvious that simply adding servers one by one doesn’t work anymore. It was both expensive to have all of them running all the time and it still wasn’t enough to handle occasional spikes of commits. Not to mention that during the nights and weekends those servers were doing absolutely nothing.
The whole thing needs to be dynamic and fortunately GitLab CI supports autoscaling out of the box. Documentation is a little bit confusing but in reality it’s very easy to get started. So here’s the plan: let’s try it!
Continue reading “Autoscaling build servers with Gitlab CI”