Introduction of a new FreeBSD Remote Process Plugin in LLDB
By Michał Górny, Kamil Rytarowski- 9 minutes read - 1836 words
Moritz Systems have been contracted by the FreeBSD Foundation to modernize the LLDB debugger’s support for FreeBSD. We are writing a new plugin utilizing the more modern client-server layout that is already used by Darwin, Linux, NetBSD and (unofficially) OpenBSD. The new plugin is going to gradually replace the legacy one.
The LLVM project provides a modern, modular, permissively licensed compiler infrastructure. A toolchain including Clang compiler, LLD linker and LLDB debugger is being developed as a part of it. FreeBSD has already replaced GCC with Clang as the primary system compiler. However, LLDB is still work-in-progress and it is being installed along with GDB.
The original FreeBSD plugin for LLDB used a legacy monolithic architecture. In this model the debugged program is being run inside the same process space as the debugger’s UI. The new plugin uses a client-server architecture that runs the debugged program as part of lldb-server, while LLDB’s UI runs remotely. This makes the LLDB code easier (or even possible) to reuse as a library with third party code as the debugger relies on monitoring signals and this functionality interferes with other code like GUI toolkits.
The Project Schedule is divided into three milestones, each taking approximately one month:
- M1 Introduce new FreeBSD Remote Process Plugin for x86_64 with basic support and upstream to LLVM.
- M2 Ensure and add the mandated features in the project (process launch, process attach (pid), process attach (name), userland core files, breakpoints, watchpoints, threads, remote debugging) for FreeBSD/amd64 and FreeBSD/i386.
- M3 Iterate over the LLDB tests. Detect and as time permits fix bugs. Ensure bug reports for each non-fixed and known problem. Add missing man pages and update the FreeBSD Handbook.
Handling transition to the new plugin
Since we want to be able to push our work upstream without causing even
temporary regressions in FreeBSD support, we are developing it as a new
plugin, with a technical name of
FreeBSDRemote that coexists with
At the moment, toggling between the two plugins is possible via
FREEBSD_REMOTE_PLUGIN environment variable. If it is unset,
the legacy FreeBSD plugin is being used. If it is set, LLDB switches
to using the new plugin. This is the approach suggested upstream,
and it is consistent with how the new plugin for Windows platform
is being currently developed.
Furthermore, the new plugin is also enabled automatically if lldb-server is started directly. This is because the old plugin simply does not support the client-server model.
Eventually, as the new plugin becomes par with the legacy one on features, we are going to perform the switch depending on the architecture. In other words, we are going to use the new plugin on all architectures it was ported to, and the legacy plugin on these that do not support the new one yet.
Basic platform support
We have decided to base the new plugin on our earlier work for NetBSD. This allowed us to reduce the amount of duplicate work, and instead focus on covering the differences between the two BSD platforms and improving its code. This made it possible to reach a reasonably functional debugger in a month’s time, and it should also make the remaining work easier.
With reasonably small changes, we were able to support controlling the process (resume, stop, single step), handle basic event signals, read and write general-purpose and debug registers, read and write traced process' address space, insert and handle software assisted breakpoints. We also have partial code for multithreaded programs, watchpoint support and 32-bit application debugging (multilib) support.
It is also important to note that the work results in improvements and bugfixes to other platforms (especially NetBSD) that either share similar code or provide a reference for our implementation. Ideally, we should be able to deduplicate more code from existing plugins (e.g. watchpoint support) as it should be suitable for different platforms.
Working on FPU register support
One of the more interesting concepts we have been working on are FPU (x87) registers. Unlike both the general-purpose registers and the registers used by MMX/SSE/AVX…, they can’t be trivially used directly by the program. The form (and completeness) of their dump differs depending on whether we’re running 32-bit or 64-bit, and what exact instruction is being used to generate it.
A great source of information on programming x87 FPU is Raymond Filiatreault’s Simply FPU tutorial.
FPU registers and general operation
The visible FPU registers can be classified in three groups:
eight data registers of 80 bits accessible as a stack
st(0)..st(7), each capable of storing an extended precision x87 floating point number, a 64-bit integer or up to 17-digit packed Binary-Coded Decimal. They overlap with MMX
three control registers: control word (
fctrl) used to program the FPU’s behavior, status word (
fstat) reporting the current FPU state and tag word (
ftag) reporting the state of all
three registers used to report exceptions when software exception handling is used:
fopindicating the opcode of the instruction causing the exception, FIP (or
fiseg:fioff) pointing to the actual instruction and FDP (or
foseg:fooff) pointing to the memory operand of that instruction (if any).
The really confusing part is that FPU uses a static internal numbering
for its registers, while the user-visible
st(*) registers shift
as values are pushed and popped. In other words, as you load values
into the FPU they are loaded into internal registers 7, 6, 5… but from the
user’s perspective the last value loaded is always in
and the indices of previous values grow.
This isn’t really important during regular programming since only
st(*) indexing is used. However, the
ftag register is using
internal register indexing, so if you wish to inspect it, you need
to be able to map between the two systems. This is done using
TOP bitfield of
fstat that indicates which internal register
is currently at
FPU has two modes of handling exceptions. By default, exceptions are handled by hardware (i.e. the FPU itself). This means that invalid operations result in invalid or special values, such as Not-A-Number or infinity. If software exception handling is enabled, the FPU interrupts the instruction and expects the program to handle it.
ftag value in register dumps and the debugger
ftag register was a 16-bit register that described
each of the FPU data registers using one of four states: empty, zero,
normalized value, special value (denormal, infinity, NaN). This full
form is used by the
FRSTOR instructions. However,
the newer instructions (
XSAVE…) instead return
an abridged 8-bit value that only indicates whether the register
is empty or not.
It seems that this difference was omitted while writing LLDB, and all platform plugins return the abridged value on the modern processors, zero-padded to 16 bits. This is not technically a problem as long as it’s consistent but it could be a little confusing to the user and it is inconsistent with what GDB does.
The new FreeBSD plugin also exhibits this arguably buggy behavior for consistency with other platforms. However, ideally LLDB would be taught to convert from the abridged ftag to full version and back. This is generally done by combining the abridged value with inspection of actual data register contents.
FIP/FDP value in register dumps and the debugger
FIP and FDP registers are used to convey respectively the memory
locations of the instruction that caused the FPU exception, and its
operand. Similarly to other memory registers, they technically consist
of a 16-bit segment register and a 32/64-bit offset register. However,
the presence of these registers in
FXSAVE etc. dumps and their
format depends on how the (saving) instruction is executed.
If the instruction is executed from 32-bit code, or from 64-bit code
REX.W prefix (which is normally the case), the dump contains
16-bit segment register and 32-bit offset register. This means that
the address is truncated in 64-bit programs. If the instruction
is executed with
REX.W prefix (i.e.
is called), the dump contains 64-bit address but no segment registers.
The current GDB and LLDB behavior on 64-bit Linux is to issue the 64-bit
variant and split the resulting pointer, so that the higher 32 bits are
foseg while the lower 32 bits are exposed as
We have lined up the FreeBSD plugin with this behavior. However,
ideally we will introduce separate
fdp register names
in the future, to expose the full pointer without the need to recombine.
It should be noted that both FreeBSD and NetBSD have issued the 32-bit
variant of saving instructions, effectively truncating
fdp to 32 bits. We have reported this as a bug:
ptrace() GETFPREGS/SETFPREGS uses 32-bit version of XSAVE/XRSTOR truncating FIP/FDP, and it has
been fixed by Konstantin Belousov.
Accomplishment of the first project milestone
We have reached the first milestone of our work, that is delivering the initial functional version of the new plugin and pushing it upstream. To demonstrate our work, we have prepared two asciicasts.
The first one demonstrates running part of LLDB test suite using both plugins. The legacy plugin is used on the top pane (the default), while the new plugin is enabled via envvar on the bottom pane.
As you can see, all register tests time out using the legacy plugin, while most of them pass (with the exception of YMM register tests since YMM support is not yet implemented).
The second asciicast demonstrates various debugger functions during a sample debugging session. This time, lldb-server is spawned explicitly on the top pane, and the debugger connects to it on the bottom pane. The gdb-remote packet logging is enabled to demonstrate how the client-server model works.
It is worthwhile to notice that when using the client-server model, LLDB UI runs asynchronously. A functional command prompt is visible while the program is running.
Changes merged upstream
All of the forementioned changes were promptly tested, reviewed and upstreamed to the mainline LLVM repository. The list of the committed changes is as follows:
- [lldb] [Process/NetBSD] Fix operating on ftag register
- [lldb] [Process/NetBSD] Fix reading FIP/FDP registers
- [lldb] [Process/NetBSD] Fix crash on unsupported i386 regs
- [lldb] [test/Register] Add read/write tests for x87 regs
- [lldb] [test/Register] Mark new FP reg tests XFAIL on Windows
- [lldb] [test/Register] Attempt to fix x86-fp-read.test on Darwin
- [lldb] [Platform] Move common ::DebugProcess() to PlatformPOSIX
- [lldb] Initial version of FreeBSD remote process plugin
Plan for the next milestone
Our next goal is to reach the second milestone of the contracted work and implement the missing functions in the amd64 debugger plugin. Most notably, this includes proper threading support, remaining register types (using XSAVE) and watchpoints. Once the new plugin reaches feature parity with the old one, we are going to switch it to be used by default on amd64.
Afterwards, during the third month we would like to look into the state of LLDB test suite on FreeBSD. Ideally, we would like to reach as many passing tests as possible, and mark the remaining failures as expected, in order to be able to easily detect regressions in the future.