Property Based Testing: Stop Hand Picking the Inputs That Pass

Mostafa Dhouib·September 1, 2026·5 min read

Property Based Testing: Stop Hand Picking the Inputs That Pass

The short answer. An implementer writes the example inputs they had in mind, which is why example tests systematically miss the inputs they did not. State a property instead, let the framework generate hundreds of cases and shrink any failure to a minimal one. Heartbleed's declared-much-greater-than-actual is exactly this class.

The structural problem with examples

Example tests encode the inputs the implementer thought of.

Those are the same inputs the implementation already handles, because both came from the same mental model. This is not a criticism of anybody's diligence. It is a property of writing the test and the code from one head: the gap in the tests is congruent with the gap in the code.

Heartbleed is the clean demonstration. OpenSSL's tests exercised well formed heartbeats, where the declared length equals the actual payload length. That is a correct model of a heartbeat, faithfully tested. The failure needed a message where those two numbers disagree, which is not a heartbeat as anybody had conceived of it, and so it was never generated.

More tests from the same model produce more of the same coverage. You need a different generator, and the point of property based testing is that the generator is not you.

What a property is

A statement that should hold across all inputs rather than for one, which the framework then attacks with hundreds of generated cases.

The useful thing is that the properties worth writing are usually already sitting in your design documents as English sentences. Taken straight from the incidents in this pillar:

Output units are always newton seconds. That is the Mars Climate Orbiter property, and it was written down, in the interface specification, as prose that nothing executed.

Declared length never exceeds actual length. Heartbleed.

The template's field count always equals the count of inputs supplied. CrowdStrike.

The decoder never reads past the buffer.

Every one of those reads like a line from a specification, and none of them appeared in a test suite. That is the recurring pattern: an invariant gets written as a comment or a spec clause, which feels like recording it, and nothing ever checks it.

Shrinking is the underrated half

When a property fails, the framework does not hand you the random 400-byte input that broke it. It shrinks the failure toward a minimal case: the smallest, simplest input that still fails.

That is what converts generated testing from noise into something debuggable. Without shrinking you get a failure you cannot reason about; with it you get "declared 2, actual 1" and the bug is obvious.

It is also why the technique survives contact with real teams. A tool that produces incomprehensible failures gets switched off within a month.

Tools

Hypothesis for Python. proptest and quickcheck for Rust. fast-check for JavaScript and TypeScript. jqwik for Java. And the original QuickCheck for Haskell, where the idea started.

As with any tool list, check the current state before relying on one.

Where to apply it first

Every parser, protocol and sensor input path. This is the same rule the fuzzing argument arrives at from a different direction, and the overlap is not a coincidence: both are answers to the problem that the input comes from outside your model.

Anywhere two components exchange a value under a contract. Units, ranges, lengths, counts, ordering guarantees. These are properties by nature, they are exactly where the forensic cases in this pillar cluster, and they are almost never tested because each side tests its own behaviour rather than the agreement.

Anywhere the code has a comment describing an invariant. That comment is a property somebody declined to make executable. Grep for "must", "always", "never" and "assumes" in your comments and you will have a starting list by lunchtime.

Where it does not help

Worth saying, because property based testing gets oversold and then abandoned.

It needs a property you can state. For code whose correct behaviour is a large table of specific cases with no unifying rule, examples are the honest representation and generated inputs will not help.

It also does not tell you the property is the right one. A property expresses your model, and if the model is wrong the tests will confirm it enthusiastically across ten thousand inputs. It widens the input space; it does not audit your understanding.

The general argument is in a test that has never failed is a decoration, the case that motivates it in Heartbleed, and the complementary technique in fuzzing and sanitizers.

FAQ

What is property based testing? Stating a property that should hold for all inputs and letting a framework generate hundreds of cases to break it, then shrink any failure to a minimal counterexample.

How is it different from example based testing? Examples encode the inputs the implementer thought of, which are the inputs the code already handles. A property covers the class, and the generator is not the person who wrote the code.

Would property based testing have caught Heartbleed? Yes. A property over the relationship between declared length and actual length finds that class of input immediately, where the existing well-formed-heartbeat examples never generated it.

What property based testing tools exist? Hypothesis for Python, proptest and quickcheck for Rust, fast-check for JavaScript and TypeScript, jqwik for Java, and QuickCheck for Haskell.

When is it not the right tool? When correct behaviour is a large table of specific cases with no unifying rule, examples are the honest representation. And a property only encodes your model, so it will not tell you the model is wrong.