Fuzz testing or fuzzing, is an automated software testing technique that introduces large amounts of random input data into a system to expose flaws and vulnerabilities in the software. Inputs are introduced by a fuzzing tool which monitors, identifies, and logs errors like crashes and data leaks. Input data can be random characters, malformed data, or extreme values.
The purpose of fuzz testing is to identify potential security, performance, or quality issues by observing how the system reacts to inputs.
Fuzz testing in traditional application development focuses on intentionally causing the applications to crash or behave unexpectedly by exposing vulnerabilities. In smart contract development, fuzz testing is designed to do this, but additionally to test a contract’s logic and verify it maintains predefined properties (invariants). Traditional applications also have invariant tests, but these invariant tests are critical to smart contracts, because violating one could have catastrophic consequences to the financial wellbeing of users.
Fuzz testing helps developers and security researchers strengthen the security of systems before vulnerabilities can be exploited.
Benefits of fuzz testing include:
“Fuzz testing” involves sending random inputs to a smart contract multiple times and checking these properties. It involves:
Performing these three steps is considered one “fuzz run.” Typically, fuzz testing involves performing many fuzz runs. In most fuzzing systems, developers choose the number of “fuzz runs” to perform before they feel confident an invariant holds. Find more information below in the “how many fuzz runs are enough” section.
Stateless and stateful fuzz testing are the two most common approaches used to assess how smart contracts handle inputs and maintain reliability. They differ in how a blockchain’s state is tracked and handled during testing.
Stateless fuzz testing sends random data to a function or contract and discards the results of previous tests. Each fuzz run ignores previous inputs and outputs. You can think of it as such:
It’s called “stateless” fuzz testing, because the contract’s state is reset after each transaction.
Stateless fuzzing can reveal straightforward, shallow issues like input validation errors or incorrect function behavior in isolation. However, it may miss complex issues that depend on state changes.
Stateful fuzz testing is where a fuzz run continues through multiple rounds of random data being sent and the contract’s state persists. The number of times random data is sent to a smart contract is called the “depth” of the fuzz run. If 200 separate transactions are called in a single fuzz run, that stateful fuzz test suite is said to have a “depth” of 200 runs.
A stateful fuzz run with a depth of 3 might look like:
And of course, this will be repeated for as many fuzz runs as the testers like.
Stateful fuzz testing helps identify problems resulting from actions that happen in a particular order. For example, a contract not maintaining balances correctly after multiple transfers, issues caused by having too many or too few tokens, or attacks where a contract function is called before it finishes its previous execution.
Several tools help security researchers perform fuzz tests.
Fuzzers are constantly looking for “new fuzz states” they haven’t seen before (see below). When using echidna or Medusa as a fuzzer, they will indicate the number of states explored. For best results, one should run the fuzzer until the number of states explored stops going up, then set a 24 hour timer. If the number of states increases again, reset the 24 hour timer. This ensures a contract’s state has been explored in as many ways as humanly possible.
A "state" in fuzzing typically refers to a unique, observable condition of the program being tested, characterized by:
Consider a simple conditional:
if (a) {
// State A
do_stuff()
} else {
// State B
something_else()
}
Here, there are two “branches” or “code paths” the computation can reach. Each of these in execution would be considered a new “state.” This is different from “storage” or “contract state.”
uint256 a = 1;
a = 2;
In the above, there is a storage variable whose value changes. This is not what’s referred to when talking about different fuzzer states. States in a fuzzer, are about a different set of code being executed or code path being followed. “Contract states,” on the other hand, are about storage values in a contract.
It's worth noting that the exact definition of a "state" can vary somewhat depending on the specific fuzzing tool or technique being used, as different fuzzers may track and categorize program behaviors in slightly different ways.
Fuzz testing also has challenges:
