by Million years ago, way before the ice age, I was preparing small C++ project for “Unix Programming” university course and at some point had to debug it via command line. That was mind blowing. And surprisingly productive. Apparently, when nothing stands in the way, especially UI, debugging can become incredibly focused.
Since .NET Framework got his cross platform twin brother .NET Core, I was looking forward to repeat the trick and debug .NET Core app on Ubuntu via command line. Few days ago it finally happened and even though it wasn’t a smooth ride, that was quite an interesting experience. So, let’s have look.
Setup
We’ll need Ubuntu, .NET Core SDK, lldb debugger and a sample app. Late April, 2018 was the month of updates, so now we have shiny new Ubuntu 18.04 and .NET Core SDK 2.1 RC1, which finally got its libsosplugin.so compiled against lldb-3.9, so v3.6 we had to use for previous .NETs finally can rest in peace. As for demo project, any .NET Core hello-world wannabe with local variables and call stacks will do.
The tools
I’ll install all of that in a VM and here’s Vagrantfile with its provision.sh file to do so:
|
1 2 3 4 5 6 7 8 9 |
Vagrant.configure("2") do |config| config.vm.box = "ubuntu/bionic64" config.vm.provider "virtualbox" do |v| v.memory = 1024 end config.vm.provision "shell", path: "provision.sh" end |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
#!/bin/bash function install_lldb39 { echo "deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.9 main" | tee /etc/apt/sources.list.d/llvm.list wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key | apt-key add - apt-get update -y apt-get install -y lldb-3.9 } function install_ms_package_source { wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/ wget -q https://packages.microsoft.com/config/ubuntu/18.04/prod.list mv prod.list /etc/apt/sources.list.d/microsoft-prod.list } function install_netsdk21rc1 { install_ms_package_source apt-get install -y apt-transport-https apt-get update -y apt-get install -y dotnet-sdk-2.1.300-rc1-008673 } function main { install_netsdk21rc1 install_lldb39 } main |
The project
It’s very simple. Let’s have a function that returns a number of ticks passed since last measurement. It’s absolutely useless except for the fact that it will have local variables and some arguments to examine later.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
using System; namespace console { class Program { static void Main(string[] args) { while (true) { var lastTick = DateTime.Now.Ticks; System.Threading.Thread.Sleep(2000); var ticksElapsed = GetTicksElapsed(lastTick); } } static long GetTicksElapsed(long lastTicks) { var currentTicks = DateTime.Now.Ticks; var delta = lastTicks - currentTicks; return delta; } } } |
Attaching debugger
OK, so let’s start the whole thing and try to connect debugger to it:
|
1 2 3 4 |
dotnet build # ... dotnet bin/Debug/netcoreapp2.1/console.dll & # [1] 6124 |
The program is producing no output and running in a background thread, so knowing the process id (6124) I can start lldb, load SOS plugin and connect to dotnet process:
|
1 2 3 4 5 6 7 8 |
find /usr -name libsosplugin.so # find SOS plugin # /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.0-rc1/libsosplugin.so sudo lldb-3.9 # start LLDB (lldb) plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.0-rc1/libsosplugin.so (lldb) process attach -p 6124 # Process 6124 stopped # * thread #1: tid = 6124, 0x00007fef21a92ed9 libpthread.so.0`__pthread_cond_timedwait + 649, name = 'dotnet', stop reason = signal SIGSTOP # ... |
Nothing difficult. We can make sure that SOS has kicked in by examining e.g. managed threads before moving on to setting up an actual breakpoint:
|
1 2 3 4 5 6 7 8 9 10 11 |
(lldb) clrthreads # ThreadCount: 2 # UnstartedThread: 0 # BackgroundThread: 1 # PendingThread: 0 # DeadThread: 0 # Hosted Runtime: no # Lock # ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception # 1 1 17ec 00000000022884A0 2020020 Preemptive 00007FEE8002DE50:00007FEE8002FB30 000000000226F620 0 Ukn # 7 2 17f2 00000000022AB0D0 21220 Preemptive 0000000000000000:0000000000000000 000000000226F620 0 Ukn (Finalizer) |
Setting up a breakpoint
For that we can use bpmd command introduced by SOS plugin. The only parameter it needs is a method name or its descriptor address, so in order to set up a breakpoint in a method called GetTicksElapsed, which is a member of Program class inside of console.dll assembly, here’s what I’d do:
|
1 2 3 4 |
(lldb) bpmd console.dll Program.GetTicksElapsed MethodDesc = 00007FEEA63F57E8 Setting breakpoint: breakpoint set --address 0x00007FEEA7061777 [console.Program.GetTicksElapsed(Int64)] Adding pending breakpoints... |
As a side note, sometimes it might be actually easier to add a breakpoint by method descriptor address instead. Those are all over the place – in call stacks, instruction pointers, in class method tables. For instance, this is the call stack of currently selected thread:
|
1 2 3 4 5 6 7 8 |
(lldb) clrstack # OS Thread Id: 0x17ec (1) # Child SP IP Call Site # 00007FFE0D679AA8 00007fef21a92ed9 [HelperMethodFrame: 00007ffe0d679aa8] System.Threading.Thread.SleepInternal(Int32) # 00007FFE0D679BF0 00007FEEA7165ABB System.Threading.Thread.Sleep(Int32) # 00007FFE0D679C00 00007FEEA70616FC console.Program.Main(System.String[]) [/home/vagrant/console/Program.cs @ 12] # 00007FFE0D679F10 00007fef2023de1f [GCFrame: 00007ffe0d679f10] # 00007FFE0D67A310 00007fef2023de1f [GCFrame: 00007ffe0d67a310] |
And this how setting up a breakpoint in Main method might look like:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
(lldb) ip2md 00007FEEA70616FC # MethodDesc: 00007feea63f57d8 # Method Name: console.Program.Main(System.String[]) # Class: 00007feea7131088 # MethodTable: 00007feea63f5800 # mdToken: 0000000006000001 # Module: 00007feea63f43e0 # IsJitted: yes # Current CodeAddr: 00007feea7061690 # Code Version History: # CodeAddr: 00007feea7061690 (Non-Tiered) # NativeCodeVersion: 0000000000000000 # Source file: /home/vagrant/console/Program.cs @ 12 (lldb) bpmd -md 00007feea63f57d8 # MethodDesc = 00007FEEA63F57D8 # Setting breakpoint: breakpoint set --address 0x00007FEEA7061690 [console.Program.Main(System.String[])] |
Easy peasy. Now, let’s resume the program with program continue command and see how it almost immediately stops at GetLastTicks:
|
1 2 3 4 5 6 |
(lldb) process continue # Process 6124 resuming (lldb) Process 6124 stopped # * thread #1: tid = 6124, 0x00007feea7061777, name = 'dotnet', stop reason = breakpoint 1.1 # frame #0: 0x00007feea7061777 # ... |
Good. Now it’s time to look around.
Examining call stacks
clrstack (or sos ClrStack) does exactly that:
|
1 2 3 4 5 6 |
(lldb) clrstack # OS Thread Id: 0x17ec (1) # Child SP IP Call Site # 00007FFE0D679BB0 00007FEEA7061777 console.Program.GetTicksElapsed(Int64) [/home/vagrant/console/Program.cs @ 18] # 00007FFE0D679C00 00007FEEA7061706 console.Program.Main(System.String[]) [/home/vagrant/console/Program.cs @ 13] # ... |
The output is pretty straightforward. Even line numbers and file names are there. But clrstack can do more than that. For instance, it also has -p parameter, which will include function arguments into the output, and that’s something really, really useful:
|
1 2 3 4 5 6 7 8 9 10 11 |
(lldb) clrstack -p # OS Thread Id: 0x17ec (1) # Child SP IP Call Site # 00007FFE0D679BB0 00007FEEA7061777 console.Program.GetTicksElapsed(Int64) [/home/vagrant/console/Program.cs @ 18] # PARAMETERS: # lastTicks (0x00007FFE0D679BE0) = 0x08d5bacfae88e0b3 # # 00007FFE0D679C00 00007FEEA7061706 console.Program.Main(System.String[]) [/home/vagrant/console/Program.cs @ 13] # PARAMETERS: # args (0x00007FFE0D679C40) = 0x00007fee8001e5c0 # ... |
Let’s examine what we’ve got. lastTicks value is 0x8d5bacfae88e0b3, which is 636620323491995827 in decimal, which indeed corresponds to current UTC date:
As for args argument, most likely that’s program’s command line arguments and this is something easy to confirm:
|
1 2 3 4 5 6 7 8 |
(lldb) dumpobj 0x00007fee8001e5c0 # Name: System.String[] # MethodTable: 00007feea6e26308 # EEClass: 00007feea65c64a8 # Size: 24(0x18) bytes # Array: Rank 1, Number of elements 0, Type CLASS # Fields: # None |
Yup, that’s empty array of strings allright.
Examining local variables
Bad news here. clrstack -i is the command to see local variables and their values. However, in lldb-3.9 and libsosplugin.so, which comes with .NET Core 2.1 RC1, this command immediately causes segmentation fault and crash of lldb process. Haven’t checked it in earlier versions, but here it happens 100% of the time.
Stepping in/over/out
Unlike with WinDBG, it doesn’t look like there are SOS commands for stepping in, out or over the next statement or function. However, there’re still native commands, which will step over assembly instructions, but it’s better than nothing. Especially when we can call clrstack and check where in managed realm we currently are:
|
1 2 3 4 5 6 7 |
(lldb) next (lldb) Process 6124 stopped # * thread #1: tid = 6124, 0x00007feea7061778, name = 'dotnet', stop reason = instruction step into # frame #0: 0x00007feea7061778 # -> 0x7feea7061778: callq 0x7feea6af3cd0 # 0x7feea706177d: movq %rax, -0x38(%rbp) # 0x7feea7061781: movq -0x38(%rbp), %rdi |
callq, if I’m not mistaken, is the instruction to execute procedure at given address, so step will probably jump us somewhere:
|
1 2 3 4 5 6 7 8 9 10 11 |
(lldb) step (lldb) Process 6124 stopped # * thread #1: tid = 6124, 0x00007feea6af3cd0, name = 'dotnet', stop reason = instruction step into # frame #0: 0x00007feea6af3cd0 # -> 0x7feea6af3cd0: pushq %rbp # ... (lldb) clrstack # OS Thread Id: 0x17ec (1) # Child SP IP Call Site # 00007FFE0D679BA8 00007FEEA6AF3CD0 System.DateTime.get_Now() # 00007FFE0D679BB0 00007FEEA706177D console.Program.GetTicksElapsed(Int64) [/home/vagrant/console/Program.cs @ 19] |
Yup, we’re in DateTime.Now‘s getter now. I actually could test where callq leads without stepping into the method itself. Checking callq‘s argument would tell the same story:
|
1 2 3 4 |
(lldb) ip2md 0x7feea6af3cd0 # MethodDesc: 00007feea66448c0 # Method Name: System.DateTime.get_Now() # ... |
Stepping out from current procedure could’ve been easy, if it didn’t jump 2 levels up instead of one every other time I used it. Maybe it has something to do with the fact that real call stack is actually a mixture of managed and unmanaged entries, whereas clrstack command shows only managed ones (-f argument would show all of them). Then, I couldn’t find an alias for step-out command and had to use its full form instead: thread step-out.
Conclusion
So this is how debugging a .NET Core app from a command line on Linux feels like. It’s surprisingly hard to find any documentation for it, so probably I missed a command or to. Faulting clrstack -i also doesn’t make the debugging easier. But still it’s really cool to be able to add a breakpoint, see how execution goes and examine call stack parameters – all from a command line, on any machine and for any project. It’s also surprising to see how thin the layer between a managed code and assembly language is. If I was able to convert callq instruction argument to managed method description, maybe I’ll be able to convert CPU register values to managed objects as well. Who knows.
