Multiprocess support for LLDB

The Moritz Systems team has started a new contract on Multiprocess support for LLDB.

LLDB has replaced GDB as the default debugger on FreeBSD. While LLDB has a fairly wide functionality, it has not yet reached full feature parity with GDB. One of its limitations is a limited support for debugging multiple processes. This project aims to address this issue in order to bring LLDB closer to being a fully featured replacement for GDB, and therefore for FreeBSD to feature a modern and complete debugger for software developers.

Moritz Systems Technology Company Sp. z o.o. is experienced with working with the BSD projects and the LLVM toolchain and already worked on improving FreeBSD support as well as generic features in LLDB.

Project Background Description

At the moment, LLDB features some preliminary support for debugging multiple processes. Notably, it has two fundamental features:

lldb-server is capable of tracing forks. However, it requires that either the parent process or the child process is detached after the fork before further debugging. LLDB client supports multitarget debugging, i.e. maintaining connections to multiple lldb-server instances. This permits debugging multiple independent processes.

The part that is currently lacking is the ability to combine both of these functions in order to support properly tracing both the parent and the child immediately after a fork. This functionality would extend the usefulness of LLDB in debugging programs using process-based parallel processing (often called “multiprocessing”), as well as applications spawning helper processes, including pipelines.

The existing multitarget debugging support in LLDB requires using a separate connection for every process being debugged. On the other hand, the GDB Remote Serial Protocol features a multiprocess extension that permits multiplexing requests and responses corresponding to multiple debugged processes, primarily through extending the syntax already used to multiplex packets regarding different threads. A partial support for this extension has already been added to LLDB as part of our prior work.

After initially discussing this with upstream, we believe that the best way forward is to continue implementing the support for gdb-remote multiprocess extension. Unlike the other solution, it does not require establishing additional communication channels. This makes debugging over firewalled networks easier, and it permits using transports that do not support multiplexing such as the serial port or pipes. Furthermore, through preserving protocol-level compatibility with GDB, it makes it possible to use LLDB as a client for other implementations of GDB server that do support multiprocess extensions.

However, the baseline variant of the GDB Remote Serial Protocol has a limitation that it can only report a single “stop reason” at a time. Whenever a single thread stops, the server reports this event and has to suspend all the other threads until the client processes the stop event and issues another request. When debugging multiple processes, this limitation extends to all the processes being debugged — that is, when a single thread of a single process stops, all threads of all processes need to be stopped as well.

The “non-stop” protocol extension removes this limitation. In this mode, the server does not stop the remaining threads or processes when reporting an event. The server queues additional events and the client can await further events without having to resume the stopped thread. Additionally, GDB supports a full “non-stop mode” where the CLI permits handling multiple running threads asynchronously. We do not plan to implement this at the time in LLDB as it would require significant design changes. Instead, the gdb-remote plugin will always ensure that all threads from the reporting process will be stopped (but not other processes). However, the protocol support implemented as part of this contract will make implementing an optional full non-stop mode in the future easier.

The additional debugged processes will be presented to the client as new targets, in order to reuse the existing code and avoid making major changes to the user interface. In particular, changing the behavior of existing commands such as “process continue” to apply to multiple processes would be confusing to users. If necessary, more convenient methods of working with multiple processes (targets) will be implemented as new commands.

Key Deliverables

• Support for the non-stop variant of GDB Remote Serial Protocol in lldb-server and gdb-remote plugin in LLDB client.
• Full support for multiprocess GDB Remote Serial Protocol extension in lldb-server.
• Support for multiprocess debugging in LLDB client through multiplexing multiple LLDB targets via a single GDB Remote Serial Protocol connection.

This work is sponsored by The FreeBSD Foundation.