Ground Truth

Fuzzing and Sanitizers for Every Input Path

Mostafa DhouibMostafa Dhouib··5 min read
The short answer

A coverage guided fuzzer mutates inputs to drive new paths, and a sanitizer catches the out of bounds read that a functional test never notices. Neither half is sufficient alone, and together they cover the adversarial input class where Heartbleed lived.

Fuzzing and Sanitizers for Every Input Path

The short answer. A coverage guided fuzzer mutates inputs to reach paths nobody wrote a test for, and a sanitizer instruments the binary so an out of bounds read fails loudly instead of silently returning adjacent memory. Neither half is sufficient alone, and together they cover the class Heartbleed belonged to.

Why your functional test would not have noticed

Start here, because it is the part that surprises people and it changes what you do next.

An out of bounds read frequently succeeds. It returns whatever bytes happen to be adjacent in memory, the function proceeds normally, and the response is well formed.

So a test that asserts the server responded, or that the response parses, or that no exception was raised, passes. The failure is not a crash. It is a correct-looking answer containing somebody else's private key.

That is the general shape of the most expensive failures in this whole pillar, and it recurs across every case. A units error returns a plausible number. A wildcarded field returns a successful deployment. A stuck sensor returns a valid reading. In none of these does anything object, because there is nothing in the output for an assertion to object to.

Which means detection has to come from instrumentation, not from assertion. That is what a sanitizer is.

The sanitizers

AddressSanitizer for memory errors, UndefinedBehaviorSanitizer for undefined behaviour, MemorySanitizer for uninitialised reads. All via the Clang and GCC -fsanitize= flags.

They instrument the binary so that the out of bounds read a green functional test never notices becomes a loud, immediate failure with a stack trace.

Here is the recommendation most teams skip: run a sanitizer build against your existing suite before writing a single new test. You have not changed what your tests cover. You have changed what they are capable of detecting, which for this bug class is the entire difference between a pass and a finding. It is usually an afternoon of build configuration and it frequently produces results on the first run.

The fuzzers

AFL++, libFuzzer and honggfuzz are coverage guided fuzzers. They mutate inputs and use coverage as a search signal to reach paths that nothing in your test suite reaches.

A fuzzer mutates the input
AFL++, libFuzzer, honggfuzz
Coverage guides it toward new paths
the one legitimate use of coverage in this pillar
The bad path executes
and returns a well formed, plausible response
A sanitizer catches what the assertion cannot
ASan, UBSan, MSan
A sanitizer build is worth running on your existing suite before writing a single new test, because it changes what those tests can detect.
FigureAn out of bounds read usually succeeds. It returns adjacent memory and the function proceeds, so assertions on the output can all pass. Detection has to come from instrumentation rather than from assertion.

Worth pausing on that: coverage guided. This pillar spends a lot of words explaining that coverage is a poor quality target, and here is the one place it earns its keep. As a search signal for a machine hunting new program states, coverage is exactly the right instrument. As a report to a manager about how good a suite is, it is not. Same measurement, and its usefulness depends entirely on who is consuming it and why.

Firmware and embedded

The obstacle in embedded is that fuzzing wants to run thousands of executions per second and the target is a microcontroller.

Renode simulates MCUs, peripherals and sensors, and has been integrated with AFL++ to fuzz Zephyr and Contiki-NG firmware. That puts protocol stack fuzzing and corrupted sensor input inside CI, on simulated hardware.

QEMU supports emulation-based fuzzing, with a caveat worth knowing in advance: ASan and MSan are generally incompatible with QEMU user mode, and QASan is the known workaround.

The rule

Every parser, protocol and sensor input path gets a fuzz test and a negative test.

Those three categories share the property that makes them dangerous: the input arrives from outside your model. A parser accepts what somebody else emits, a protocol handler accepts what a peer sends, a sensor path accepts what physics produces. In the interesting cases what arrives is adversarial rather than merely unusual, and your own imagination is the wrong generator for it.

Pair the two techniques rather than choosing. The property states what must hold. The fuzzer searches for inputs that break it. The sanitizer catches the damage that does not surface as a wrong answer. Heartbleed is caught by any one of the three, and a system with all three is hard to hurt in this particular way.

This is also a diligence red flag in its own right, phrased almost exactly this way: no negative tests on input parsing, protocol or sensor paths.

The general argument is in a test that has never failed is a decoration, the case in Heartbleed, and the complementary technique in property based testing.

FAQ

What is coverage guided fuzzing? A fuzzer that mutates inputs and uses coverage as a search signal to reach program paths nothing else reaches. It is the one legitimate use of coverage in this pillar, because the consumer is a search algorithm rather than a report.

What does AddressSanitizer catch that tests do not? Out of bounds reads and memory errors that succeed and return plausible data. Assertions on output cannot object to a well formed response, so detection has to come from instrumentation.

Should I run a sanitizer even without a fuzzer? Yes, and it is the cheapest first step. A sanitizer build against your existing suite changes what those tests can detect without changing what they cover.

Can I fuzz firmware without hardware? Renode simulates MCUs and peripherals and has been integrated with AFL++ for Zephyr and Contiki-NG. QEMU supports emulation-based fuzzing, though ASan and MSan are generally incompatible with QEMU user mode.

Which code paths need fuzzing? Every parser, protocol and sensor input path, because in all three the input arrives from outside your model and may be adversarial rather than merely unusual.

Free checklist
The Verification Diligence Checklist

Nine controls, each tied to a public failure that cost somebody nine figures or a life, with the four artifacts a verification data room holds and the one question an investor should ask in the room. Built to be printed and ticked only where somebody has actually watched the check go red.

One email, the resource, and nothing else unless you reply.
Carrying a program like this one?

Tell us the system, the stakes, and the date that matters. You get a straight technical reply from the person who would lead the work, within 24 hours.

Bring us the program