This week, I have the unbridled displeasure of debugging a C++ project. Given that I’ve never written any C++, you can imagine this fills me with a generalised existential dread. I consider picking up Sartre, to cheer me up a bit, and then I remember a series of excellent posts from Moritz Systems about debugger enhancements in LLVM, specifically for FreeBSD.

The blog posts are excellent, and contain a great deal of information. But the key piece that’s missing, is whether these very useful changes made it into FreeBSD 13.0-RELEASE, which bundles LLVM in the base as a compiler, and if not, LLVM release contains these.

As it’s Friday, I stand no chance of completing this debugging anyway, so I gird up my loins, install all the devel/llvm* versions that look useful from FreeBSD’s port tree, and away I go.

LLVM12 is the Bee’s Knees

After a few minutes stumbling around in the dark, I find this pithy guide on the LLVM website, LLDB remote debugging. You, dear reader, can skip the wrong turns and false starts that I made, and skip directly to the juicy bits. As of mid April, devel/llvm12 installs v12.0.0.r2 of the LLVM toolchain. You’ll need to install this on both your desktop, and the remote system.

remote system

I tried running the debugger as a non-root user, and while launching the lldb-server appeared to work, once I tried to run the program, I got a weird error: 'A' packet returned an error: 8. I guess we need some sort of additional privileges to enable this, but you’d not know that from the error message unless you are somehow divinely inspired.

If you do try it as a normal user, and the debugger won’t work, make sure you have this sysctl set: security.bsd.unprivileged_proc_debug=1, just remember to disable if after debugging.

$ sudo sysctl security.bsd.unprivileged_proc_debug=1
$ sudo pkg install -yr FreeBSD devel/llvm12
$ export CC=clang12
$ export CXX=clang++12
$ lldb-server platform --listen "*:1234" --server
Connection established.

local system

On my local system, I install llvm12, switch to the newer compiler, clean and compile with debug symbols included, and launch lldb.

$ sudo pkg install -yr FreeBSD devel/llvm12
$ export CC=clang12
$ export CXX=clang++12
$ cd /repos/zt
$ gmake clean debug
$ ...

$ lldb12
(lldb) platform select remote-freebsd
  Platform: remote-freebsd
 Connected: no
(lldb) platform connect connect://continuity:1234
  Platform: remote-freebsd
    Triple: x86_64-portbld-freebsd13.0
OS Version: 13.0 (00199506)
  Hostname: continuity
 Connected: yes
WorkingDir: /repos/zt
    Kernel: FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f ...
(lldb) file zerotier-one
Current executable set to '/repos/zt/zerotier-one' (x86_64).
(lldb) br set --name main
Breakpoint 1: where = zerotier-one`main + 25 at one.cpp:2082:2, address = 0x0000000000544429
(lldb) run
Process 82889 launched: '/repos/zt/zerotier-one' (x86_64)
Process 82889 stopped
* thread #1, name = 'zerotier-one', stop reason = breakpoint 1.1
    frame #0: 0x0000000000544429 zerotier-one`main(argc=1, argv=0x00007fffffffed10) at one.cpp:2082:2
   2079	#endif
   2080	
   2081	#ifdef __UNIX_LIKE__
-> 2082		signal(SIGHUP,&_sighandlerHup);
   2083		signal(SIGPIPE,SIG_IGN);
   2084		signal(SIGIO,SIG_IGN);
   2085		signal(SIGUSR1,SIG_IGN);
(lldb) q
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y

That looks pretty fantastic actually. Thanks FreeBSD Foundation and Moritz Systems, respectively, for funding and coding this.

I will leave the actual debugging til next week, but so far this is a really pleasant experience, far more than ferreting around on continuity, which is an underpowered Intel Atom Rangeley CPU acting as my NAS & backup server.