LLDB serial port communication support

Moritz Systems have been contracted by the FreeBSD Foundation to continue our work on modernizing the LLDB debugger’s support for FreeBSD.

The primary goal of our contract is to bring kernel debugging into LLDB. The complete Project Schedule is divided into six milestones, each taking approximately one month:

1. Improve LLDB compatibility with the GDB protocol: fix LLDB implementation errors, implement missing packets, except registers.

2. Improve LLDB compatibility with the GDB protocol: support gdb-style flexible register API.

3. Support for debugging via serial port.

4. libkvm-portable and support for debugging kernel core files in LLDB, on amd64 + arm64 platform. Support for other platforms as time permits.

5. Support for debugging the running kernel on amd64 + arm64 platform. Support for other platforms as time permits.

6. Extra month for kgdb work, processing patches on LLDB reviews or miscellaneous tasks – as time permits. Examples of misc tasks: access to extended system and process information, starting processes via shell, $_siginfo support.

This month we have been working on improving serial port support in LLDB. Serial port support is important for FreeBSD users and developers because it permits debugging remote devices on top of minimal kernel support, e.g. without the necessity for a working network stack. This can be used both with embedded devices (especially ARM64, RISC-V) and while debugging the kernel of a regular computer system.

Prior to our work, the LLDB client had a minimal serial port support that used hardwired configuration, and the LLDB server had no serial port support at all. Our goal was to provide a functional serial port support with easy configuration abilities for both LLDB client and server.

Before we start discussing our changes and how to use the serial port in LLDB, let’s start by explaining what the COM port is and roughly how it works. This is necessary to make it clear what the port configuration parameters are and why they need to be set.

Serial ports explained

Serial port communications in general

In the widest sense, ‘serial ports’ refer to the communication ports that transmit data one bit at a time. This term is coined in contrast to ‘parallel ports’ that use parallel wires to transmit multiple bits simultaneously. However, the term ‘serial port’ is also used to refer to a specific PC port that is also called an ‘RS-232 port’ or a ‘COM port’. Other examples of serial ports are FireWire and USB ports, while the most commonly known kind of parallel port is the Centronics or printer port.

The RS-232 ports can exist in a variety of connector variants. Originally, 25-pin DB-25 connectors were used (the same as for parallel ports). Newer PCs used smaller 9-pin DE-9 connectors. Some devices also used MMJ (modified 6P6C), 8P8C (RJ45, the same as used for Ethernet cables) or 10P10C connectors.

Historically, PC serial ports have been used to connect various peripherals, such as modems, mice or various embedded devices. While nowadays physical serial ports are becoming rarer, some devices still emulate serial port interface when connected via USB. Serial ports can also be used to connect two computers. In the Unix world, the most common use of such connection is to provide a remote console. However, it can also be used to transmit arbitrary data including GDB Remote Serial Protocol data.

The traditional serial port model assumes fixed roles of a terminal (i.e. the computer) and a modem. This means that e.g. the pin used to receive data on the computer end is used to send data on the other end. When two computers are to be connected via a serial port, a null-modem cable needs to be used that crosses out the wires to match the PC pinouts.

The serial port communications can be either asynchronous or synchronous. In asynchronous communications, the signal lines are normally idle and the endpoints need to explicitly signal when they are about to start transmitting data. In synchronous communications, the data is transmitted continuously in reference to separate timing signals. The commonly used hardware does not support synchronous communication, and the necessary timing pins are not present on the DE-9 connector (the wider DB-25 is required instead). For our purposes, we will be considering asynchronous communications only.

Asynchronous RS-232 communications

Serial port DE-9 pinout

Pin	Short	Full name	Signal direction
1	DCD	Data Carrier Detect	computer ← modem
2	RXD	Receive Data	computer ← modem
3	TXD	Transmit Data	computer → modem
4	DTR	Data Terminal Ready	computer → modem
5	GND	Ground	n/a
6	DSR	Data Set Ready	computer ← modem
7	RTS	Request To Send	computer → modem
8	CTS	Clear To Send	computer ← modem
9	RI	Ring Indicator	computer ← modem

The table above lists the pinout used in DE-9 serial port connector.

The three wires absolutely necessary to transmit data are RXD, TXD and GND. The ground wire provides the reference voltage for electrical signals, the Receive Data line is used to transmit from the remote end to the local end, while the Transmit Data line is used for data transfer in the opposite direction.

The RTS/CTS lines are normally used for hardware flow control (explained below). The DTR/DTS lines are normally used to indicate that the respective devices are powered up and ready to communicate, though in some scenarios they could be used in place of RTS/CTS for flow control.

The DCD line is used by modems to indicate whether the modem is connected to a remote device. With PC-to-PC connections, Windows hosts use DCD to indicate whether the serial port is open by any program.

The RI line is used by modems to indicate that the phone line is ringing.

For data lines, the negative voltage (-3 V to -15 V relative to ground) is used to indicate ones, while positive (+3 V to +15 V) zeros. For control lines, negative voltage indicates OFF states, while positive ON state.

In asynchronous communications, the idle data line is at negative voltage (i.e. the state corresponding to binary one). A single data frame consists of a start bit (binary zero), followed by 5 to 8 data bits, followed by an optional parity bit, followed by 1, 1.5 or 2 stop bits (ones). After the stop bits, the next character can be transmitted or the line can return to the idle state.

The receiving end detects a data frame through the transition from negative voltage to positive (i.e. from idle state or a stop bit to a start bit). At this point, it waits a short delay and starts sampling the expected number of frame bits. The subsequent start bit transitions are used to synchronize the communications.

It is important to understand here that the exact protocol used to transmit data varies from device to device, and to effect serial port communications both endpoints have to be configured to use the same parameters.

Basic serial port parameters

The basic parameters controlling the serial port transmission are:

• the baud rate, i.e. the transmission speed
• the number of data bits
• the number of stop bits
• the presence and kind of parity bit
• the flow control kind

The baud rate defines transmission speed, and therefore the length of every transmitted bit. Since the protocol does not account for bit-level synchronization, both endpoints have to match baud rate in order to communicate correctly. Otherwise, the receiving endpoint could miss some of the transmitted bits or read the same bit multiple times.

The serial port protocol permits transmitting 5 to 8 data bits per frame. However, for practical purposes either 7-bit (ASCII) or 8-bit transmission needs to be used. If a lower number of bits is used, the bytes written through the system API are truncated.

The stop bits are primarily used to ensure that the receiver can be synchronized reliably when receiving multiple frames in succession. The protocol can use 1, 1.5 or 2 stop bits. Usually a single stop bit is used; the larger values were historically used to provide devices with additional frame processing time.

Optionally, a parity bit can be used to verify the correctness of transmission. In fact, the parity bit can usually be configured in one of the five ways:

• no parity: parity bit is not transmitted

• even parity: parity bit is zero if the data bits contain even number of ones

• odd parity: parity bit is zero if the data bits contain odd number of ones

• space parity: parity bit is always one

• mark parity: parity bit is always zero

The serial port has a very limited ability to detect misconfiguration or transmission errors. If parity checking is not used, it can only detect framing errors — e.g. when zero is received in place of an expected stop bit. If parity checking is enabled, then the parity bit can be used to detect malformed data and parity bit to some degree, including some cases of misconfiguration.

The flow control options are used to determine when the transmitter is permitted to send data to the receiver. This is primarily used to prevent buffer overflows. There are three common options here: no flow control, software flow control or hardware flow control.

If no flow control is used, then both endpoints are permitted to transmit at any time.

In software flow control, the other endpoint can transmit special characters in-band to request stopping (usually XOFF, produced by Ctrl+S) and resuming (XON, Ctrl+Q) transmitting data. As in other cases of in-band signaling, this prevents these characters from appearing in regular data.

When hardware flow control is used, the RTS/CTS lines are used to control data flow. When the transmitter is about to send data, it sets the RTS line. The receiver sets CTS when it’s ready to receive it.

POSIX serial port API

Serial port devices

On common POSIX systems serial ports are exposed as teletype devices (ttys), i.e. as terminals. This is particularly convenient when serial ports are used to provide remote terminal access to a machine. Other examples of ttys are Linux kernel virtual terminals or pseudo-teletypes (ptys) that are used to programmatically create terminals e.g. in terminal emulators (graphical consoles).

Teletypes are exposed as character device files. The exact filenames depend on the hardware providing the serial port. For example, UART serial ports on Linux are named /dev/ttyS*. Character devices can be open, read from and written to like regular files, although they support only a subset of file operations. In addition to that, teletype devices have some special behavior and properties.

The additional properties of teletypes can be accessed using ioctl(2) interface. The termios(3) API provides high-level routines to teletype-specific ioctls. Most importantly, this means getting and setting attributes that control the teletype behavior, including the exact protocol used for serial transmission.

The majority of basic serial port parameters discussed above can be configured via c_cflag (control flag) field of struct termios. However, not all of the aforementioned options are available on POSIX systems.

Setting baud rate

The input and output baud rate can be set using the cf_setispeed(3) and cf_setospeed(3) functions respectively. The functions accept enumeration values for the baud rates. These values are defined as preprocessor macros named B followed by the baud rate, e.g. B38400 for 38400 bits per second. The special value of B0 indicates a hangup. The exact set of enumeration values vary per system and #ifdef needs to be used to determine whether a particular value is available.

There are also corresponding cf_getispeed(3) and cf_getospeed(3) functions to obtain the baud rate from struct termios. BSDs and GNU/Linux also provide a cf_setspeed(3) function to set both input and output baud rate to the same value.

Setting the number of data bits and stop bits

The number of data bits can be set using CS5 through CS8 constants on the c_cflag field. There is also a CSIZE mask that needs to be used to filter all the constants.

Controlling the parity bit

BSD and GNU/Linux support using either one or two stop bits (but not 1.5). The CSTOPB flag on c_cflag is used to toggle between them. If the flag is set, two stop bits are used; otherwise, one bit is used.

POSIX specifies support for no, odd or even parity. GNU/Linux additionally supports mark and space parity. Specific parity variant can be selected via manipulating c_cflag flags as outlined in the following table:

Parity bit settings

Parity	PARENB	PARODD	CMSPAR
No parity	0	n/a
Even	1	0	0
Odd	1	1	0
Space	1	0	1
Mark	1	1	1

Note that the CMSPAR flag is specific to GNU/Linux. On other systems, it is possible to emulate space/mark parity via manually computing the parity bit for each character sent and switching between odd and even parity to obtain the expected parity bit value.

Enabling parity checking

The above options control the presence of parity bit in the transmitted data. Parity verification on input can be controlled via three c_iflag flags as outlined in the following table:

Input flags

INPCK	IGNPAR	PARMRK	On parity errors
0	0	0	ignore (pass data through)
1	0	0	replace w/ NUL
1	0	1	mark (prepend 0xFF 0x00)
1	1	n/a	skip character

Note that PARMRK additionally enables escaping raw 0xFF as 0xFF 0xFF.

Enabling flow control

Software flow control on output and input can be enabled via IXON and IXOFF flags on c_iflag respectively. The characters used to stop and start output can be customized via c_cc[VSTOP] and c_cc[VSTART] members respectively.

BSD and GNU/Linux systems also support enabling hardware flow control via the CRTSCTS flag of c_cflag. FreeBSD additionally supports switching input and output hardware flow control separately via CRTS_IFLOW and CRTS_OFLOW flags respectively.

Option summary

The following table summarizes the serial line properties and their respective settings discussed above:

Serial line settings

Property	Set via	Constant	Portability
Baud rate (input)	cfsetispeed(3)	Bi (e.g. B115200)	POSIX + ext.
Baud rate (output)	cfsetospeed(3)	Bi (e.g. B115200)	POSIX + ext.
Data bits	c_cflag	~CSIZE | {CS5..CS8}	POSIX
Stop bits (1 or 2)	c_cflag	CSTOPB	POSIX
Stop bits (1.5)	(unsupported)
Parity (enable)	c_cflag	PARENB	POSIX
Parity (even/odd)	c_cflag	PARODD	POSIX
Parity (mark/space)	c_cflag	CMSPAR	GNU/Linux
soft. flow control (input)	c_iflag	IXOFF	POSIX
soft. flow control (output)	c_iflag	IXON	POSIX
hardware flow control	c_cflag	CRTSCTS	BSD, GNU/Linux

Terminal vs raw use

The termios(3) API provides quite detailed control over teletype behavior. However, for our purposes it is enough to summarize various flags related to two main uses of serial ports: terminal use and raw use.

When a teletype is intended to be used as a terminal, a number of flags related to input and output processing should be enabled in order to provide better experience for the user using the console. This includes flags such as echo (ECHO) that cause all input to be automatically echoed back to the teletype or canonical mode (ICANON) that enables special handling of control characters corresponding to keystrokes and line editing.

In other words, in this mode no actual data is transmitted over the line until the user finishes typing a line (or sends an EOF character). This makes some degree of editing possible, and the erased characters are not transmitted to the other end. However, this also implies that special characters trigger specific behavior rather than being transmitted and therefore it is not suitable for transmitting binary data.

If the terminal usage is undesired, the teletype should be switched to raw mode. Some operating systems (e.g. FreeBSD and GNU/Linux) provide a convenient cfmakeraw(3) function; on others this could be achieved via disabling the flags related to input and output processing. In the raw mode, binary data is transmitted reliably and teletype-level line buffering is disabled.

Serial port support in LLDB

Transports for the GDB Remote Protocol

So far we have implicitly assumed that the communication between GDB Remote Protocol client and server are done over TCP/IP. However, both GDB and LLDB support more kinds of underlying transports. The modern way of selecting the transport is via a specific URI scheme passed to the platform connect command in LLDB client, or as a command-line argument to the LLDB server.

For example, in order to connect to a remote server over TCP/IP running on local port 1234, you’d issue:

platform connect connect://127.0.0.1:1234

The supported schemes can be classified as symmetric or asymmetric. When using a symmetric scheme (such as serial://), both endpoints use the same scheme. On the other hand, asymmetric schemes require using complementary schemes. For example, when using TCP, one endpoint needs to use the listen:// scheme to listen for incoming connections, while the other uses connect:// to establish a connection.

It should be noted that there is no strict requirement that the server would use a ‘listener’ scheme, and the client would use a ‘connect’ scheme. For example, it is possible to establish a reverse connect model where the server connects to the client. To do that, first establish the listening socket on the client:

platform connect listen://*:1234

Then start the server in reverse connect mode, e.g.:

lldb-server g --reverse-connect 127.0.0.1:1234

The table below summarizes currently available protocols, symmetric plus asymmetric with their counterparts.

Transport schemes supported by LLDB.

Client	Listener	Description	Example
connect (tcp-connect)	listen	TCP/IP connection	connect://127.0.0.1:1234
unix-connect	unix-accept (accept)	Named UNIX socket	unix-connect:///tmp/socket
unix-abstract-connect	unix-abstract-accept	Abstract namespace UNIX socket	unix-abstract-connect://lldb-socket
fd		Open file descriptor	fd://10
file		Filesystem path (e.g. a pty)	file:///tmp/pty1
serial		A serial port	serial:///dev/cuau0

The new serial:// protocol

Prior to our changes, the LLDB client had minimal support for serial port. The file:// scheme could be used to open a serial port, and LLDB set some hardcoded defaults. In order to provide a better serial port support, we have decided to add a new serial:// scheme that provides the ability to control serial port parameters better.

The most basic form of the new scheme is:

serial:///dev/cuau0

In this form, the port is set to the raw mode and no other parameters are altered. Furthermore, upon disconnecting LLDB restores the original serial port attributes.

Additional attributes can be set by providing them in URI query string form, e.g.:

serial:///dev/cuau0?baud=115200&parity=even&parity-check=replace

The supported parameters are:

• baud=<number> to set the baud rate

• parity=(off|even|odd|mark|space) to set the kind of parity bit for transmitted packets, and indicate whether the received packets contain a parity bit

• parity-check=(off|replace|ignore) to specify whether the parity bit (if present) on input should not be checked, or whether parity errors should result in bytes being replaced by NULs (replace) or be skipped entirely (ignore)

• stop-bits=(1|2) to specify whether one or two stop bits should be used

The same baud rate and parity mode should be used on both endpoints. Enabling parity checking increases the chance of detecting transmission errors, though the protocol does not provide any error recovery or retransmission options. One stop bit is sufficient for all modern systems.

Support for more transports in the LLDB server

Prior to our work, lldb-gdbserver supported a few transport modes, via somewhat inconsistent argument set:

• tcp://`, unix://orunix-abstract://`` scheme could be passed to listen on a TCP socket, a named UNIX socket or an abstract namespace UNIX socket

• hostname:port or :port argument could be passed to listen on a TCP socket

• otherwise, the argument was interpreted as path for a named UNIX socket

Furthermore, --fd option could be used to communicate over an open file descriptor, or --reverse-connect could be used to have the server connect to the listening TCP socket set by the client.

We have performed a major refactoring of the code to reduce the duplication of code between the client and server. Now the server uses the same logic as the client to establish connections, and therefore accepts the same schemes. For example, the following pairs of commands are now equivalent:

# listen on localhost:1234
lldb-server g :1234
lldb-server g listen://localhost:1234

# reverse connect to 127.0.0.1:1234
lldb-server g --reverse-connect 127.0.0.1:1234
lldb-server g connect://127.0.0.1:1234

It is also now possible to communicate over the serial port using the serial:// scheme:

lldb-server g 'serial:///dev/ttyu0?baud=115200&parity=no'

As a curiosity, it is worth noting that lldb-server technically claimed to support connections over UDP. However, this code has never worked as it assumed a streaming transport protocol (while UDP is a datagram-based protocol).

Testing serial port support

Ideally, all new code in LLDB needs to be covered by tests. However, proper testing of serial port would require either some kind of logical loopback interface that would accurately emulate a serial port, or a physical loop made using two serial ports and a null modem cable.

Relying on the presence of a physical serial port would be impractical and would mean that the code is tested rarely. While we have not been able to find a maintained accurate virtual serial port implementation, the system pseudoterminal (pty) interface is close enough for basic regression testing.

While ptys do not simulate parameters such as baud rate accurately, they can verify that the basic I/O operations work, as well as that various attributes are actually set (via querying the terminal attributes independently).

In addition to the new functionality tests, we have manually tested the serial port support using two computers connected via a null modem cable. In addition to successfully testing a basic connection between LLDB client and lldb-gdbserver, we have verified the effect of baud rate and parity setting changes on the connectivity.

Changes merged upstream

• [lldb] [Host] Refactor Socket::DecodeHostAndPort() to use LLVM API
• [lldb] [Host] Remove TerminalStateSwitcher
• [lldb] [Host] Refactor TerminalState
• [lldb] Add a gdb_remote_client test for connecting to pty
• [lldb] [test] Delay pty/tty imports to fix Windows builds
• [lldb] [Host] Fix flipped logic in TerminalState::Save()
• [lldb] [Host] Sync TerminalState::Data to struct type
• [lldb] [test] Terminate “process connect” connections via kill
• [lldb] Add unit tests for Terminal API
• [lldb] [ConnectionFileDescriptorPosix] Use a single NativeFile
• [lldb] [ConnectionFileDescriptorPosix] Refactor scheme matching
• [lldb] [test] Use secondary pty end for testing Terminal
• [lldb] [ConnectionFileDescriptorPosix] Combine m_read_sp & m_write_sp
• [lldb] [Utility] Remove Status::WasInterrupted() along with its only use
• [lldb] [lldb-server] Refactor ConnectToRemote()
• [lldb] [Host] Make Terminal methods return llvm::Error
• [lldb] [Host] Add setters for common teletype properties to Terminal
• [lldb] Add serial:// protocol for connecting to serial port
• [lldb] [unittest] Disable SetParity() tests on Linux entirely
• [lldb] [Host/Terminal] Add missing #ifdef for baudRateToConst()
• [lldb] [Host/SerialPort] Add std::moves for better compatibility
• [lldb] [Utility/UriParser] Replace port==-1 with llvm::None
• [lldb] Support serial port parity checking
• [lldb] [Utility/UriParser] Return results as ‘struct URI’
• [lldb] [Host/ConnectionFileDescriptor] Do not use non-blocking mode
• [lldb] [Communication] Add a WriteAll() method that resumes writing
• [lldb] [lldb-gdbserver] Unify listen/connect code to use ConnectionFileDescriptor
• [lldb] [Host] Move port predicate-related logic to gdb-remote

Summary

Over the last month, our primary focus was serial port support in LLDB. Prior to our work, LLDB client and LLDB server were using largely missynced transport implementations. The client was capable of communicating over a serial port with hardwired parameters, the server was not.

Throughout our work, LLDB has undergone connection-related refactoring. We have introduced an abstraction layer for serial port settings that currently provides a thin wrapper over the POSIX termios interface but that could easily be extended to support Windows serial port API in the future. We have built a new ``serial://` scheme upon it that could be used to configure the serial port and establish connection over it. Finally, we have refactored lldb-gdbserver to reuse the same transport API and therefore gain support for our new transport.

At this point, the transport can set baud rate, parity bit generation, parity checking and stop bit count. The URL query string-based layout provides a readable way to alter the parameters, as well as to leave them unchanged if desired. Upon disconnection, LLDB restores the serial port parameters.

We have confirmed that our new code works via a physical serial port and can be used to successfully remotely debug a program on another computer. We have also used pseudoterminals (ptys) to cover our code with best-effort regression tests.

This work brings us one step closer towards feature parity between LLDB and GDB. As a result, the people gathered around the FreeBSD project are no longer required to use GNU GPL-licensed debugger to ease embedded and kernel-mode debugging, in favor of a permissively licensed modern replacement from the LLVM project.

Future plans

The next step in our work is to implement support for debugging FreeBSD kernel core dumps in LLDB.

Currently, GDB supports inspecting kernel coredumps and live kernel memory via libkvm on BSD-derived systems. However, this library is present only on BSD systems and normally supports only the particular BSD flavor.

Our first goal is to create a new portable library for processing FreeBSD core dumps. This library could be built on any system in order to inspect core dumps of the FreeBSD kernel running on any architecture.

With the help of the new library, we are going to implement support for inspecting FreeBSD kernel dumps in LLDB. Our primary development and testing focus are amd64 and arm64 platforms but we are going to implement support for other architectures as time permits.