Moritz Systems have been contracted
by the FreeBSD Foundation to modernize the
LLDB debugger’s support for
FreeBSD. We are writing a new plugin
utilizing the more modern client-server layout that is already used
by Darwin, Linux, NetBSD and (unofficially) OpenBSD. The new plugin is
going to gradually replace the legacy one.
The LLVM project provides a modern, modular,
permissively licensed compiler infrastructure. A toolchain including
Clang compiler, LLD linker and LLDB debugger is being developed
as a part of it. FreeBSD has already replaced GCC with Clang
as the primary system compiler. However, LLDB is still work-in-progress
and it is being installed along with GDB.
The original FreeBSD plugin for LLDB used a legacy monolithic
architecture. In this model the debugged program is being run inside
the same process space as the debugger’s UI. The new plugin uses
a client-server architecture that runs the debugged program as part
of lldb-server, while LLDB’s UI runs remotely. This makes the LLDB
code easier (or even possible) to reuse as a library with third
party code as the debugger relies on monitoring signals and this
functionality interferes with other code like GUI toolkits.
The Project Schedule is divided into three milestones, each taking
approximately one month:
- M1 Introduce new FreeBSD Remote Process Plugin for x86_64 with
basic support and upstream to LLVM.
- M2 Ensure and add the mandated features in the project (process
launch, process attach (pid), process attach (name), userland
core files, breakpoints, watchpoints, threads, remote debugging)
for FreeBSD/amd64 and FreeBSD/i386.
- M3 Iterate over the LLDB tests. Detect and as time permits fix bugs.
Ensure bug reports for each non-fixed and known problem. Add missing
man pages and update the FreeBSD Handbook.
Handling transition to the new plugin
Since we want to be able to push our work upstream without causing even
temporary regressions in FreeBSD support, we are developing it as a new
plugin, with a technical name of
FreeBSDRemote that coexists with
At the moment, toggling between the two plugins is possible via
FREEBSD_REMOTE_PLUGIN environment variable. If it is unset,
the legacy FreeBSD plugin is being used. If it is set, LLDB switches
to using the new plugin. This is the approach suggested upstream,
and it is consistent with how the new plugin for Windows platform
is being currently developed.
Furthermore, the new plugin is also enabled automatically if lldb-server
is started directly. This is because the old plugin simply does not
support the client-server model.
Eventually, as the new plugin becomes par with the legacy one
on features, we are going to perform the switch depending
on the architecture. In other words, we are going to use the new plugin
on all architectures it was ported to, and the legacy plugin on these
that do not support the new one yet.
Basic platform support
We have decided to base the new plugin on our earlier work for NetBSD.
This allowed us to reduce the amount of duplicate work, and instead
focus on covering the differences between the two BSD platforms
and improving its code. This made it possible to reach a reasonably
functional debugger in a month’s time, and it should also make
the remaining work easier.
With reasonably small changes, we were able to support controlling the process
(resume, stop, single step), handle basic event signals, read and write general-purpose
and debug registers, read and write traced process' address space, insert and handle
software assisted breakpoints. We also have partial code for multithreaded
programs, watchpoint support and 32-bit application debugging (multilib)
It is also important to note that the work results in improvements and bugfixes to
other platforms (especially NetBSD) that either share similar code or provide a reference for our
implementation. Ideally, we should be able to deduplicate more code from existing
plugins (e.g. watchpoint support) as it should be suitable for different platforms.
Working on FPU register support
One of the more interesting concepts we have been working on are FPU
(x87) registers. Unlike both the general-purpose registers
and the registers used by MMX/SSE/AVX…, they can’t be trivially used
directly by the program. The form (and completeness) of their dump
differs depending on whether we’re running 32-bit or 64-bit, and what
exact instruction is being used to generate it.
A great source of information on programming x87 FPU is Raymond
Filiatreault’s Simply FPU
FPU registers and general operation
The visible FPU registers can be classified in three groups:
eight data registers of 80 bits accessible as a stack
each capable of storing an extended precision x87 floating point
a 64-bit integer or up to 17-digit packed Binary-Coded Decimal.
They overlap with MMX
three control registers: control word (
fctrl) used to program
the FPU’s behavior, status word (
fstat) reporting the current
FPU state and tag word (
ftag) reporting the state of all
three registers used to report exceptions when software exception
handling is used:
fop indicating the opcode of the instruction
causing the exception, FIP (or
fiseg:fioff) pointing to the actual
instruction and FDP (or
foseg:fooff) pointing to the memory operand
of that instruction (if any).
The really confusing part is that FPU uses a static internal numbering
for its registers, while the user-visible
st(*) registers shift
as values are pushed and popped. In other words, as you load values
into the FPU they are loaded into internal registers 7, 6, 5… but from the
user’s perspective the last value loaded is always in
and the indices of previous values grow.
This isn’t really important during regular programming since only
st(*) indexing is used. However, the
ftag register is using
internal register indexing, so if you wish to inspect it, you need
to be able to map between the two systems. This is done using
TOP bitfield of
fstat that indicates which internal register
is currently at
FPU has two modes of handling exceptions. By default, exceptions
are handled by hardware (i.e. the FPU itself). This means that
invalid operations result in invalid or special values, such as
Not-A-Number or infinity. If software exception handling is enabled,
the FPU interrupts the instruction and expects the program to handle it.
ftag value in register dumps and the debugger
ftag register was a 16-bit register that described
each of the FPU data registers using one of four states: empty, zero,
normalized value, special value (denormal, infinity, NaN). This full
form is used by the
FRSTOR instructions. However,
the newer instructions (
XSAVE…) instead return
an abridged 8-bit value that only indicates whether the register
is empty or not.
It seems that this difference was omitted while writing LLDB, and all
platform plugins return the abridged value on the modern processors,
zero-padded to 16 bits. This is not technically a problem as long
as it’s consistent but it could be a little confusing to the user
and it is inconsistent with what GDB does.
The new FreeBSD plugin also exhibits this arguably buggy behavior
for consistency with other platforms. However, ideally LLDB would
be taught to convert from the abridged ftag to full version and back.
This is generally done by combining the abridged value with inspection
of actual data register contents.
FIP/FDP value in register dumps and the debugger
FIP and FDP registers are used to convey respectively the memory
locations of the instruction that caused the FPU exception, and its
operand. Similarly to other memory registers, they technically consist
of a 16-bit segment register and a 32/64-bit offset register. However,
the presence of these registers in
FXSAVE etc. dumps and their
format depends on how the (saving) instruction is executed.
If the instruction is executed from 32-bit code, or from 64-bit code
REX.W prefix (which is normally the case), the dump contains
16-bit segment register and 32-bit offset register. This means that
the address is truncated in 64-bit programs. If the instruction
is executed with
REX.W prefix (i.e.
is called), the dump contains 64-bit address but no segment registers.
The current GDB and LLDB behavior on 64-bit Linux is to issue the 64-bit
variant and split the resulting pointer, so that the higher 32 bits are
foseg while the lower 32 bits are exposed as
We have lined up the FreeBSD plugin with this behavior. However,
ideally we will introduce separate
fdp register names
in the future, to expose the full pointer without the need to recombine.
It should be noted that both FreeBSD and NetBSD have issued the 32-bit
variant of saving instructions, effectively truncating
fdp to 32 bits. We have reported this as a bug:
ptrace() GETFPREGS/SETFPREGS uses 32-bit version of XSAVE/XRSTOR truncating FIP/FDP, and it has
been fixed by Konstantin Belousov.
Accomplishment of the first project milestone
We have reached the first milestone of our work, that is delivering
the initial functional version of the new plugin and pushing it
upstream. To demonstrate our work, we have prepared two asciicasts.
The first one demonstrates running part of LLDB test suite using both
plugins. The legacy plugin is used on the top pane (the default),
while the new plugin is enabled via envvar on the bottom pane.
As you can see, all register tests time out using the legacy plugin,
while most of them pass (with the exception of YMM register tests
since YMM support is not yet implemented).
The second asciicast demonstrates various debugger functions during
a sample debugging session. This time, lldb-server is spawned explicitly
on the top pane, and the debugger connects to it on the bottom pane.
The gdb-remote packet logging is enabled to demonstrate how
the client-server model works.
It is worthwhile to notice that when using the client-server model,
LLDB UI runs asynchronously. A functional command prompt is visible
while the program is running.
Changes merged upstream
All of the forementioned changes were promptly tested, reviewed and upstreamed
to the mainline LLVM repository. The list of the committed changes is as follows:
Plan for the next milestone
Our next goal is to reach the second milestone of the contracted work
and implement the missing functions in the amd64 debugger
plugin. Most notably, this includes proper threading support,
remaining register types (using XSAVE) and watchpoints. Once the new
plugin reaches feature parity with the old one, we are going to switch
it to be used by default on amd64.
Afterwards, during the third month we would like to look into the state of LLDB test suite
on FreeBSD. Ideally, we would like to reach as many passing tests
as possible, and mark the remaining failures as expected, in order
to be able to easily detect regressions in the future.