FreeBSD KGDB support in LLDB

The Moritz Systems team has started a new contract on FreeBSD KGDB support in LLDB.

Since its inception FreeBSD has relied on the GNU toolchain both for building the base system, and to provide a number of tools in the base system. Updates to the base system toolchain eventually stalled for several reasons, notably the GNU projects' migration to GPLv3 in 2007.

FreeBSD began the process of migrating to a modern toolchain by importing the Clang/LLVM compiler in 2010. FreeBSD 10.0 featured Clang as the system compiler, no longer including an outdated GCC version in the release. FreeBSD 11.0 features LLDB, the debugger from LLVM family of projects, on amd64 and arm64 targets.

Our recent work has updated the FreeBSD plugin for LLDB to use a new process plugin model using client-server design. We have also improved support for core dumps and added support for follow-fork/follow-vfork functions.

However, the LLDB debugger still lacks a drop-in replacement for kgdb(1), which wraps a modified version of GDB and depends on the libkvm interface for debugging post-mortem and live BSD kernel memory.

Additionally, the GDB Remote Protocol, which is used internally by LLDB, is not fully compatible with GDB and thus not interchangeable between the debuggers and the LLDB frontend is not compatible with preexisting gdb-server implementations, most notably the qemu one that is useful for debugging the kernels.

These limitations mean that there are circumstances that the default system debugger (LLDB) doesn’t cope with, and that the users are required to additionally install and use GDB. Our goal is to make this unnecessary and let users use a single tool for all their debugging needs.

The protocol incompatibilities mostly stem from the fact that the protocol was originally implemented to facilitate internal communications between LLDB and LLGS (gdb-remote server). In some cases the incompatibilities were mere oversights, in other cases they meant to avoid adding complexity that was not perceived necessary at the time.

In some cases, the oversights are not easy to find as they affect very specific debugging scenarios. However, we consider it important to address them before working on further items as they could cause bugs affecting our work. A focused effort on finding and fixing these issues will likely be more efficient and avoid unnecessarily distracting ourselves from the other work in order to debug and fix bugs.

An example of an oversight is using decimal values in the parameters to vFile packets where GDB uses hexadecimal values. Unfortunately, this issue is non-trivial to fix as changing the parameter encoding retroactively would break compatibility with prior versions of lldb-server. A proper fix would require being able to distinguish between old versions of lldb-server (prior to the change), new versions of lldb-server (with the change) and other implementations of gdbserver.

An example of deliberate simplification is register reading support. The traditional gdbserver sends the listing of registers along with their locations inside the register packets as an XML document. LLDB uses an incompatible description in JSON.

Besides the difference in container format, there are also differences in how the data is structured. In particular, the YMM registers overlap with the earlier XMM registers. GDB sends XMM registers and the higher halves of YMM registers, following the same concept as the s XSAVE instruction. LLDB sends both XMM and YMM registers as a whole, resulting in some redundancy.

Resolving these incompatibilities would require making major changes to the internal logic related to storing and processing register information, and most likely their interaction with process plugins. Right now, the abstraction used in LLDB largely assumes that the register packets will use the same or very similar layout as ptrace(2) calls, while GDB/gdbserver provides for arbitrary register layouts. This is why we have decided to address register packet compatibility separately from general tasks.

Our focus is specifically on making LLDB compatible with GDB. We do not believe that making GDB more compatible with the old LLDB protocol is a goal worth pursuing.

GDB and LLDB both support remote debugging via TCP/IP. However, GDB additionally supports using a variant of gdb-remote protocol over the serial port. This is useful for remote debugging of kernel problems without relying on the in-kernel TCP/IP stack.

libkvm provides a uniform interface for accessing kernel virtual memory images. libkvm is part of FreeBSD base system, and provides support for both the live kernel and kernel core dumps. In order to facilitate debugging the latter on non-FreeBSD platforms, libkvm needs to support these platforms. Our goal is to provide a libkvm-portable variant that integrates the original FreeBSD sources with a portable build system, following other *-portable packages (e.g. openssh-portable).

libkvm-portable will open the way towards implementing support for working on FreeBSD kernel core dumps inside LLDB, on all platforms supported by LLDB including both different operating systems and different architectures. We will also use it to implement live kernel debugging on FreeBSD. We are going to start with the amd64 platform, and proceed with other platforms (e.g. arm64, i386) as time permits.

The project does not include work on the existing kgdb(1) tool. For the time being, it focuses on adding the functions necessary for kernel debugging to bare LLDB.

We expect that once our work is complete, the new features in LLDB will become part of the FreeBSD base system after the next update.

Key Deliverables:

• Improvement of GDB protocol compatibility in LLDB, aiming at making it possible to use it with the original gdbserver, as well as third-party servers (e.g. in qemu).
• Support for remote debugging via serial port.
• libkvm-portable – version of libkvm suitable for being built on other operating systems.
• Support for debugging the kernel core files in LLDB, on amd64 + arm64 and other platforms as time permits.
• Support for debugging the running kernel in LLDB, on amd64 + arm64 and other platforms as time permits. processordoesn

This work is sponsored by The FreeBSD Foundation.