Examining NodeJS core dump with llnode and lldb

Learning how to analyze .net core dumps on Linux with lldb and SOS plugin was really rewarding experience. It just feels good knowing that I can see managed stacks, objects and threads in something that looks very native and unmanaged. I saw there are also lldb plugins for Python and Java, but… can I do the same for NodeJS and JavaScript? Well, of course!

There’s llnode plugin which does exactly that. Today I’m not going to go deeply into how it works – it’s not in the focus of my daily job now. But it’s still interesting to get a feeling of it. So, let’s dive right in.

Getting the tools installed

Today for a change (and for lack of better hardware), I’m going to experiment directly on MacOS, so there won’t be any Docker or apt-get‘s. I already have NodeJS v8.9.4, so the only remaining components to get are llnode and lldb themselves. If MacOS wasn’t on its way of becoming Windows, the following two lines would do:

1. brew update && brew install --with-lldb --with-toolchain llvm – get lldb,
2. sudo npm install -g llnode – get llnode.

However, in my case everything that could go wrong did went wrong. “Please install XCode”, “Please configure code signing certificate”, “Access denied”. Eventually I managed to get all the pieces together except for llnode command line tool itself. However, as that’s just a shell file that starts lldb with llnode plugin enabled and I do have the plugin, we don’t really need that.

So the remaining step is to enable automatic dumps generation and we’re good to go: ulimit -c unlimited.

Getting a core dump

This JavaScript should fail,

while this shell command will launch it and instruct nodejs to fail with a core dump, if it has to:

It takes node a little while to crush, because even for such simple program it produces 1.7 GB core dump file. Not the largest one I’ve seen, but not the smallest one either.

Good. The dump is there, it’s time to look inside of it.

Looking inside of nodejs core dump

It actually feels very same as when I was working with .NET Core dumps.

llnode plugin adds several new lldb commands. All of them start with v8  – easy to remember. For instance, just v8 will output the help about whole 8 subcommands that it supports:

1. bt
2. findjsinstances
3. findobjects
4. findrefs
5. inspect
6. nodeinfo
7. print
8. source

Back trace

bt is a default command to try even in non-nodejs dumps, so it seems logical to start with it first.

The command immediately proves itself worthy,  as we’ve got the whole faulting call stack with our delayedFailure JavaScript function (line 8) being right in the middle of it. But we can get more.

Viewing the source

While looking through the faulting call stack, we can request the source code for its individual frames. For instance, delayedFailure function was in the 5th one:

We can switch to that frame and view its source:

The command for that is called v8 source list, but it doesn’t look that it always works:

However, if we pass additionally frame address, it’ll finally do its thing:

Examining the memory

We didn’t have any interesting objects in the memory, but we still can check if there’s anything there created by the environment:

There’s a console object and we can look for its instances with findjsinstances:

Or go even further: inspect what that console is made of:

Apparently, it’s made of properties. What a reveal. Other than that, there’s not much to see here.

Conclusion

Dealing with .NET Core and NodeJS core dumps actually feels very similar. Respective plugins provide different commands set, but underlying principles are the same: view call stacks and examine memory. .NET Core’s SOS plugin does, however, provide some features that llnode doesn’t. E.g. GC analysis. Given that JavaScript runtime also has Garbage Collection, being able to track GC heaps and object roots might’ve been handy in some scenarios. But for basic troubleshooting it seems good enough.