Generally, during fuzz testing (regardless of the tool used to perform it: American Fuzzy Lop, libFuzzer, or any other), we have to remember to keep the number of iterations per second high. This means that a good fuzzer is a fast fuzzer.

This is mostly facilitated by minimizing the structures and operations needed to prepare the context. We do not reinitialize the mechanisms of the fuzzed library for every iteration. We use the stack instead of the heap and globals. For example, according to Max Moroz’s fuzzing tutorial, slide 62, handling 1 MB of memory on the heap slows down the fuzzer two times, compared to a buffer of the same size on the stack. A second example from the same tutorial is the use of memset(3) function for buffers on the heap that causes the fuzzer performance degradation up to five times. Reducing the size of buffers used temporarily is also worth considering. Allocating a 256 kB buffer on the stack takes three times less time compared to a 1 MB allocation. The last “trick” is to use global variables instead of local. The observed efficiency gain from using this method is about two times. Our experience shows that this is paid for with a slightly higher memory usage at the start.

On the other hand, it is essential to remember to release all the resources used during a fuzzer iteration - this ensures that the tested program does not consume all the system memory.

Tuning the fuzzer usually gives measurable effects, but under certain circumstances we will hit a performance barrier. Despite following the best practices, we will not achieve a significant improvement of the iteration rate. This is especially true for parsers of binary formats such as executables or multimedia files. Fast fuzzing targets include regular expression engines, network stacks, and text formats.

Code coverage testing and maximization

Both AFL++ and libFuzzer use SanitizerCoverage as the default code coverage testing tool. A built-in LLVM tool can be used to generate reports telling us what part of the code is being reached by our test corpora.

We are going to show how to work with SanitizerCoverage, libFuzzer and the Yara project example. Yara is a tool used by malware researchers and helps to detect and analyse malicious code. Yara is designed around textual and binary patterns and integrates well with the libFuzzer project.

In order to instrument Yara with necessary code coverage, perform the following steps:

git clone https://github.com/Moritz-Systems/libfuzzer-coverage-yara
cd libfuzzer-coverage-yara/yara-codecov

./bootstrap.sh

CC=clang CXX=clang++ \
  CFLAGS="-g -O1 -fsanitize=fuzzer-no-link -fprofile-instr-generate \
  -fcoverage-mapping" \
  ./configure

CC=clang CXX=clang++ \
  CFLAGS="-g -O1 -fsanitize=address,fuzzer-no-link \
  -fprofile-instr-generate -fcoverage-mapping" \
  make -j4

libtool --mode=compile --tag=CXX clang++ \
  -fsanitize=address,fuzzer -fprofile-instr-generate \
  -fcoverage-mapping -std=c++11 -I./libyara/include/ \
  -pthread -o yara_rules_lfuzzer_cov.o -c \
  tests/oss-fuzz/rules_fuzzer.cc

libtool --mode=link --tag=CXX clang++ \
  -fsanitize=address,fuzzer -fprofile-instr-generate \
  -fcoverage-mapping \
  -lcrypto -lssl -pthread \
  yara_rules_lfuzzer_cov.o libyara/libyara.la \
  -o yara_rules_lfuzzer_cov

We have to run the fuzzer together with additional switches and variables:

LLVM_PROFILE_FILE="yara.profraw" \
  libtool --mode=execute \
  ./yara_rules_lfuzzer_cov -runs=1 ../yara-fuzzing-corpus
INFO: Seed: 3494707497
INFO: Loaded 2 modules   (10070 inline 8-bit counters): 10063 [0x7f3b18be361f, 0x7f3b18be5d6e), 7 [0x5ad045, 0x5ad04c),
INFO: Loaded 2 PC tables (10070 PCs): 10063 [0x7f3b18be5d70,0x7f3b18c0d260), 7 [0x56f4b0,0x56f520),
INFO:     1115 files found in ../yara-fuzzing-corpus/
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 19232 bytes
INFO: seed corpus: files: 1115 min: 4b max: 19232b total: 1797733b rss: 36Mb
#1024   pulse  cov: 2719 ft: 10489 corp: 735/695Kb exec/s: 341 rss: 264Mb
#1117   INITED cov: 2738 ft: 10987 corp: 781/1017Kb exec/s: 372 rss: 264Mb
#1117   DONE   cov: 2738 ft: 10987 corp: 781/1017Kb lim: 19232 exec/s: 372 rss: 264Mb
Done 1117 runs in 3 second(s)

The result is a file with the .profraw extension, which must be indexed before generating coverage report with the command:

llvm-profdata merge -sparse yara.profraw -o yara.profdata

The result of the last operation is a file that can be passed to the SanitizerCoverage report generator (of course you can debug your harness with coverage!):

llvm-cov show ./yara_rules_lfuzzer_cov.o -instr-profile=yara.profdata
<snipped>                                                  
   30|       |#include <stdint.h>
   31|       |#include <stddef.h>
   32|       |#include <string.h>
   33|       |
   34|       |#include <yara.h>
   35|       |
   36|       |
   37|       |extern "C" int LLVMFuzzerInitialize(int* argc, char*** argv)
   38|      1|{
   39|      1|   yr_initialize();
   40|      1|  return 0;
   41|      1|}
   42|       |
   43|       |
   44|       |extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
   45|  1.11k|{
   46|  1.11k|  YR_RULES* rules;
   47|  1.11k|  YR_COMPILER* compiler;
   48|  1.11k|
   49|  1.11k|  char* buffer = (char*) malloc(size + 1);
   50|  1.11k|
   51|  1.11k|  if (!buffer)
   52|      0|    return 0;
   53|  1.11k|
   54|  1.11k|  strncpy(buffer, (const char *) data, size);
   55|  1.11k|  buffer[size] = 0;
   56|  1.11k|
   57|  1.11k|  if (yr_compiler_create(&compiler) != ERROR_SUCCESS)
   58|  1.11k|  {
   59|      0|    free(buffer);
   60|      0|    return 0;
   61|      0|  }
   62|  1.11k|
   63|  1.11k|  if (yr_compiler_add_string(compiler, (const char*) buffer, NULL) == 0)
   64|    119|  {
   65|    119|    if (yr_compiler_get_rules(compiler, &rules) == ERROR_SUCCESS)
   66|    119|      yr_rules_destroy(rules);
   67|    119|  }
   68|  1.11k|
   69|  1.11k|  yr_compiler_destroy(compiler);
   70|  1.11k|  free(buffer);
   71|  1.11k|
   72|  1.11k|  return 0;
   73|  1.11k|}

The amount of executions of the specified functions in our harness for the prepared corpora.

Additionally, you can print a summary of coverage data for modules or for individual files:

llvm-cov report ./libyara/hex_grammar.o -instr-profile=foo.profdata
Filename                      Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
hex_grammar.c                     489               120    75.46%           4                 0   100.00%         961               165    82.83%
 
Files which contain no functions:
include/yara/hex_lexer.h            0                 0         -           0                 0         -           0                 0         -
include/yara/limits.h               0                 0         -           0                 0         -           0                 0         -
include/yara/re.h                   0                 0         -           0                 0         -           0                 0         -
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                             489               120    75.46%           4                 0   100.00%         961               165    82.83%

Maximizing code coverage

It is worth mentioning that many open-source projects have a variety of test corpora, sometimes even including file formats no longer in broad use. Additionally, unit tests provide a very good source of test cases. After writing a proper parser, they can significantly improve code coverage. Searching for the file extension with Google also brings very good results.

If the project does not provide tests or corpora, the only remaining option is to generate a useful set of files manually: in the case of text formats, it is easy to find relevant information in the code and use it in your files. Binary files can often be obtained using conversion tools included in the project. All you need to do is to find a file in a format supported by the converter and script it to generate the output. Finally, I would like to remind you once again to minimize the file sizes.

One of the key elements causing increased code coverage are dictionaries - text files containing constants for a given file format. This saves the CPU time that would otherwise be needed to perform the initial validation of key elements of the tested format. Dictionaries from AFL and libFuzzer are compatible with each other - the initial “corpora” of the dictionaries can be found in https://github.com/google/fuzzing/tree/master/dictionaries.

Fuzz-Driven Development and Google OSS-Fuzz

libFuzzer’s originator and Google employee, Kostya Serebryany, proposed to extend the classic continuous integration approach to fuzzing. Due to the fact that libFuzzer fuzzers are very similar to unit tests, and unit tests alone are not able to saturate the security tests, this approach is worth considering in the project testing cycle.

Based on libFuzzer, Google launched in December 2016 an open-source project fuzzing service called OSS-Fuzz. Each open-source project developer can apply for testing their own application. The only requirement is to write your own fuzzer and create a pull-request to the Google repository.

At the time of writing the article there were 25,000 VMs available for OSS-Fuzz. From its start, the project helped to find more than 11,000 different problems in the following projects: OpenSSL, ffmpeg, LibreOffice, sqlite3 and freetype2.

Integrating an efficient fuzzer into your project has never been so easy and cheap. In the era of (almost) cost-free computing power and easy access to cloud servers, it is worthwhile to extend the testing of your project with tools such as libFuzzer or AFL++. However, it should be remembered that fuzzers, even though they are very effective, are not able to find all the bugs, and long-term fuzzing without finding a mistake does not mean that there are no problems - with a wink of the eye one can say that it only proves a small portion of the code as used by the corpora.