Use MiMalloc for in-process fuzzers (#439)

* MiMalloc

* docu

* other fuzzers

* mention asan
This commit is contained in:
Dongjia Zhang 2021-12-31 00:33:23 +09:00 committed by GitHub
parent b5153cc525
commit b537539b54
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
24 changed files with 85 additions and 44 deletions

View File

@ -14,3 +14,44 @@ In Rust, we bind this concept to the [`Executor`](https://docs.rs/libafl/0/libaf
By default, we implement some commonly used Executors such as [`InProcessExecutor`](https://docs.rs/libafl/0/libafl/executors/inprocess/struct.InProcessExecutor.html) is which the target is a harness function providing in-process crash detection. Another Executor is the [`ForkserverExecutor`](https://docs.rs/libafl/0/libafl/executors/forkserver/struct.ForkserverExecutor.html) that implements an AFL-like mechanism to spawn child processes to fuzz. By default, we implement some commonly used Executors such as [`InProcessExecutor`](https://docs.rs/libafl/0/libafl/executors/inprocess/struct.InProcessExecutor.html) is which the target is a harness function providing in-process crash detection. Another Executor is the [`ForkserverExecutor`](https://docs.rs/libafl/0/libafl/executors/forkserver/struct.ForkserverExecutor.html) that implements an AFL-like mechanism to spawn child processes to fuzz.
A common pattern when creating an Executor is wrapping an existing one, for instance [`TimeoutExecutor`](https://docs.rs/libafl/0.6.1/libafl/executors/timeout/struct.TimeoutExecutor.html) wraps an executor and install a timeout callback before calling the original run function of the wrapped executor. A common pattern when creating an Executor is wrapping an existing one, for instance [`TimeoutExecutor`](https://docs.rs/libafl/0.6.1/libafl/executors/timeout/struct.TimeoutExecutor.html) wraps an executor and install a timeout callback before calling the original run function of the wrapped executor.
## InProcessExecutor
Let's begin with the base case; `InProcessExecutor`.
This executor uses [_SanitizerCoverage_](https://clang.llvm.org/docs/SanitizerCoverage.html) as its backend, as you can find the related code in `libafl_targets/src/sancov_pcguards`. Here we allocate a map called `EDGES_MAP` and then our compiler wrapper compiles the harness to write the coverage into this map.
When you want to execute the harness as fast as possible, you will most probably want to use this `InprocessExecutor`.
One thing to note here is, when your harness is likely to have heap corruption bugs, you want to use another allocator so that corrupted heap does not affect the fuzzer itself. (For example, we adopt MiMalloc in some of our fuzzers.). Alternatively you can compile your harness with address sanitizer to make sure you can catch these heap bugs.
## ForkserverExecutor
Next, we'll take a look at the `ForkserverExecutor`. In this case, it is `afl-cc` (from AFLplusplus/AFLplusplus) that compiles the harness code, and therefore, we can't use `EDGES_MAP` anymore. Hopefully, we have [_a way_](https://github.com/AFLplusplus/AFLplusplus/blob/2e15661f184c77ac1fbb6f868c894e946cbb7f17/instrumentation/afl-compiler-rt.o.c#L270) to tell the forkserver which map to record the coverage.
As you can see from the forkserver example,
```rust,ignore
//Coverage map shared between observer and executor
let mut shmem = StdShMemProvider::new().unwrap().new_map(MAP_SIZE).unwrap();
//let the forkserver know the shmid
shmem.write_to_env("__AFL_SHM_ID").unwrap();
let mut shmem_map = shmem.map_mut();
```
Here we make a shared memory region; `shmem`, and write this to environmental variable `__AFL_SHM_ID`. Then the instrumented binary, or the forkserver, finds this shared memory region (from the aforementioned env var) to record its coverage. On your fuzzer side, you can pass this shmem map to your `Observer` to obtain coverage feedbacks combined with any `Feedback`.
Another feature of the `ForkserverExecutor` to mention is the shared memory testcases. In normal cases, the mutated input is passed between the forkserver and the instrumented binary via `.cur_input` file. You can improve your forkserver fuzzer's performance by passing the input with shared memory.
See AFL++'s [_documentation_](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#5-shared-memory-fuzzing) or the fuzzer example in `forkserver_simple/src/program.c` for reference.
It is very simple, when you call `ForkserverExecutor::new()` with `use_shmem_testcase` true, the `ForkserverExecutor` sets things up and your harness can just fetch the input from `__AFL_FUZZ_TESTCASE_BUF`
## InprocessForkExecutor
Finally, we'll talk about the `InProcessForkExecutor`.
`InProcessForkExecutor` has only one difference from `InprocessExecutor`; It forks before running the harness and that's it.
But why do we want to do so? well, under some circumstances, you may find your harness pretty unstable or your harness wreaks havoc on the global states. In this case, you want to fork it before executing the harness runs in the child process so that it doesn't break things.
However, we have to take care of the shared memory, it's the child process that runs the harness code and writes the coverage to the map.
We have to make the map shared between the parent process and the child process, so we'll use shared memory again. You should compile your harness with `pointer_maps` (for `libafl_targes`) features enabled, this way, we can have a pointer; `EDGES_MAP_PTR` that can point to any coverage map.
On your fuzzer side, you can allocate a shared memory region and make the `EDGES_MAP_PTR` point to your shared memory.
```rust,ignore
let mut shmem;
unsafe{
shmem = StdShMemProvider::new().unwrap().new_map(MAX_EDGES_NUM).unwrap();
}
let shmem_map = shmem.map_mut();
unsafe{
EDGES_PTR = shmem_map.as_ptr();
}
```
Again, you can pass this shmem map to your `Observer` and `Feedback` to obtain coverage feedbacks.

View File

@ -1,44 +0,0 @@
# ForkserverExecutor and InprocessForkExecutor
## Introduction
We have `ForkserverExecutor` and `InprocessForkExecutor` in libafl crate.
On this page, we'll quickly explain how they work and see how they compare to normal `InProcessExecutor`
## InprocessExecutor
Let's begin with the base case; `InProcessExecutor`.
This executor uses [_SanitizerCoverage_](https://clang.llvm.org/docs/SanitizerCoverage.html) as its backend, as you can find the related code in `libafl_targets/src/sancov_pcguards`. Here we allocate a map called `EDGES_MAP` and then our compiler wrapper compiles the harness to write the coverage into this map.
## ForkserverExecutor
Next, we'll look at the `ForkserverExecutor`. In this case, it is `afl-cc` (from AFLplusplus/AFLplusplus) that compiles the harness code, and therefore, we can't use `EDGES_MAP` anymore. Hopefully, we have [_a way_](https://github.com/AFLplusplus/AFLplusplus/blob/2e15661f184c77ac1fbb6f868c894e946cbb7f17/instrumentation/afl-compiler-rt.o.c#L270) to tell the forkserver which map to record the coverage.
As you can see from the forkserver example,
```rust,ignore
//Coverage map shared between observer and executor
let mut shmem = StdShMemProvider::new().unwrap().new_map(MAP_SIZE).unwrap();
//let the forkserver know the shmid
shmem.write_to_env("__AFL_SHM_ID").unwrap();
let mut shmem_map = shmem.map_mut();
```
Here we make a shared memory region; `shmem`, and write this to environmental variable `__AFL_SHM_ID`. Then the instrumented binary, or the forkserver, finds this shared memory region (from the aforementioned env var) to record its coverage. On your fuzzer side, you can pass this shmem map to your `Observer` to obtain coverage feedbacks combined with any `Feedback`.
Another feature of the `ForkserverExecutor` to mention is the shared memory testcases. In normal cases, the mutated input is passed between the forkserver and the instrumented binary via `.cur_input` file. You can improve your forkserver fuzzer's performance by passing the input with shared memory.
See AFL++'s [_documentation_](https://github.com/AFLplusplus/AFLplusplus/blob/stable/instrumentation/README.persistent_mode.md#5-shared-memory-fuzzing) or the fuzzer example in `forkserver_simple/src/program.c` for reference.
It is very simple, when you call `ForkserverExecutor::new()` with `use_shmem_testcase` true, the `ForkserverExecutor` sets things up and your harness can just fetch the input from `__AFL_FUZZ_TESTCASE_BUF`
## InprocessForkExecutor
Finally, we'll talk about the `InProcessForkExecutor`.
`InProcessForkExecutor` has only one difference from `InprocessExecutor`; It forks before running the harness and that's it.
But why do we want to do so? well, under some circumstances, you may find your harness pretty unstable or your harness wreaks havoc on the global states. In this case, you want to fork it before executing the harness runs in the child process so that it doesn't break things.
However, we have to take care of the shared memory, it's the child process that runs the harness code and writes the coverage to the map.
We have to make the map shared between the parent process and the child process, so we'll use shared memory again. You should compile your harness with `pointer_maps` (for `libafl_targes`) features enabled, this way, we can have a pointer; `EDGES_MAP_PTR` that can point to any coverage map.
On your fuzzer side, you can allocate a shared memory region and make the `EDGES_MAP_PTR` point to your shared memory.
```rust,ignore
let mut shmem;
unsafe{
shmem = StdShMemProvider::new().unwrap().new_map(MAX_EDGES_NUM).unwrap();
}
let shmem_map = shmem.map_mut();
unsafe{
EDGES_PTR = shmem_map.as_ptr();
}
```
Again, you can pass this shmem map to your `Observer` and `Feedback` to obtain coverage feedbacks.

View File

@ -40,6 +40,7 @@ num-traits = "0.2.14"
rangemap = "0.1.10" rangemap = "0.1.10"
structopt = "0.3.25" structopt = "0.3.25"
serde = "1.0" serde = "1.0"
mimalloc = { version = "*", default-features = false }
backtrace = "0.3" backtrace = "0.3"
color-backtrace = "0.5" color-backtrace = "0.5"

View File

@ -1,5 +1,8 @@
//! A libfuzzer-like fuzzer with llmp-multithreading support and restarts //! A libfuzzer-like fuzzer with llmp-multithreading support and restarts
//! The example harness is built for libpng. //! The example harness is built for libpng.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use frida_gum::Gum; use frida_gum::Gum;
use std::{ use std::{

View File

@ -26,6 +26,7 @@ libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_h
libafl_cc = { path = "../../libafl_cc/" } libafl_cc = { path = "../../libafl_cc/" }
clap = { version = "3.0.0-rc.4", features = ["default"] } clap = { version = "3.0.0-rc.4", features = ["default"] }
nix = "0.23.0" nix = "0.23.0"
mimalloc = { version = "*", default-features = false }
[lib] [lib]
name = "fuzzbench" name = "fuzzbench"

View File

@ -1,4 +1,7 @@
//! A singlethreaded libfuzzer-like fuzzer that can auto-restart. //! A singlethreaded libfuzzer-like fuzzer that can auto-restart.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use clap::{App, Arg}; use clap::{App, Arg};
use core::{cell::RefCell, time::Duration}; use core::{cell::RefCell, time::Duration};

View File

@ -25,6 +25,7 @@ libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_h
# TODO Include it only when building cc # TODO Include it only when building cc
libafl_cc = { path = "../../libafl_cc/" } libafl_cc = { path = "../../libafl_cc/" }
structopt = "0.3.25" structopt = "0.3.25"
mimalloc = { version = "*", default-features = false }
[lib] [lib]
name = "generic_inmemory" name = "generic_inmemory"

View File

@ -1,5 +1,8 @@
//! A libfuzzer-like fuzzer with llmp-multithreading support and restarts //! A libfuzzer-like fuzzer with llmp-multithreading support and restarts
//! The `launcher` will spawn new processes for each cpu core. //! The `launcher` will spawn new processes for each cpu core.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use core::time::Duration; use core::time::Duration;
use std::{env, net::SocketAddr, path::PathBuf}; use std::{env, net::SocketAddr, path::PathBuf};

View File

@ -19,6 +19,7 @@ libafl = { path = "../../libafl/" }
libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_edges", "sancov_value_profile", "libfuzzer"] } libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_edges", "sancov_value_profile", "libfuzzer"] }
# TODO Include it only when building cc # TODO Include it only when building cc
libafl_cc = { path = "../../libafl_cc/" } libafl_cc = { path = "../../libafl_cc/" }
mimalloc = { version = "*", default-features = false }
[build-dependencies] [build-dependencies]
cc = { version = "1.0", features = ["parallel"] } cc = { version = "1.0", features = ["parallel"] }

View File

@ -1,5 +1,8 @@
//! A libfuzzer-like fuzzer with llmp-multithreading support and restarts //! A libfuzzer-like fuzzer with llmp-multithreading support and restarts
//! The example harness is built for libmozjpeg. //! The example harness is built for libmozjpeg.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use std::{env, path::PathBuf}; use std::{env, path::PathBuf};

View File

@ -25,6 +25,7 @@ libafl = { path = "../../libafl/", features = ["default"] }
libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_hitcounts", "libfuzzer", "sancov_cmplog"] } libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_hitcounts", "libfuzzer", "sancov_cmplog"] }
# TODO Include it only when building cc # TODO Include it only when building cc
libafl_cc = { path = "../../libafl_cc/" } libafl_cc = { path = "../../libafl_cc/" }
mimalloc = { version = "*", default-features = false }
[lib] [lib]
name = "libfuzzer_libpng" name = "libfuzzer_libpng"

View File

@ -1,5 +1,8 @@
//! A libfuzzer-like fuzzer with llmp-multithreading support and restarts //! A libfuzzer-like fuzzer with llmp-multithreading support and restarts
//! The example harness is built for libpng. //! The example harness is built for libpng.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use core::time::Duration; use core::time::Duration;
use std::{env, path::PathBuf}; use std::{env, path::PathBuf};

View File

@ -25,6 +25,7 @@ libafl_targets = { path = "../../libafl_targets/", features = ["libfuzzer"] }
# TODO Include it only when building cc # TODO Include it only when building cc
libafl_cc = { path = "../../libafl_cc/" } libafl_cc = { path = "../../libafl_cc/" }
structopt = "0.3.25" structopt = "0.3.25"
mimalloc = { version = "*", default-features = false }
[lib] [lib]
name = "libfuzzer_libpng" name = "libfuzzer_libpng"

View File

@ -2,6 +2,9 @@
//! The example harness is built for libpng. //! The example harness is built for libpng.
//! In this example, you will see the use of the `launcher` feature. //! In this example, you will see the use of the `launcher` feature.
//! The `launcher` will spawn new processes for each cpu core. //! The `launcher` will spawn new processes for each cpu core.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use core::time::Duration; use core::time::Duration;
use std::{env, net::SocketAddr, path::PathBuf}; use std::{env, net::SocketAddr, path::PathBuf};

View File

@ -25,6 +25,7 @@ libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_h
# TODO Include it only when building cc # TODO Include it only when building cc
libafl_cc = { path = "../../libafl_cc/" } libafl_cc = { path = "../../libafl_cc/" }
structopt = "0.3.25" structopt = "0.3.25"
mimalloc = { version = "*", default-features = false }
[lib] [lib]
name = "libfuzzer_libpng" name = "libfuzzer_libpng"

View File

@ -2,6 +2,9 @@
//! The example harness is built for libpng. //! The example harness is built for libpng.
//! In this example, you will see the use of the `launcher` feature. //! In this example, you will see the use of the `launcher` feature.
//! The `launcher` will spawn new processes for each cpu core. //! The `launcher` will spawn new processes for each cpu core.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use core::time::Duration; use core::time::Duration;
use std::{env, net::SocketAddr, path::PathBuf}; use std::{env, net::SocketAddr, path::PathBuf};

View File

@ -24,6 +24,7 @@ libafl = { path = "../../libafl/" }
libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_hitcounts", "libfuzzer"] } libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_hitcounts", "libfuzzer"] }
# TODO Include it only when building cc # TODO Include it only when building cc
libafl_cc = { path = "../../libafl_cc/" } libafl_cc = { path = "../../libafl_cc/" }
mimalloc = { version = "*", default-features = false }
[lib] [lib]
name = "libfuzzer_libpng" name = "libfuzzer_libpng"

View File

@ -1,5 +1,8 @@
//! A libfuzzer-like fuzzer with llmp-multithreading support and restarts //! A libfuzzer-like fuzzer with llmp-multithreading support and restarts
//! The example harness is built for libpng. //! The example harness is built for libpng.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use std::{env, path::PathBuf}; use std::{env, path::PathBuf};

View File

@ -18,6 +18,7 @@ debug = true
[dependencies] [dependencies]
libafl = { path = "../../libafl/" } libafl = { path = "../../libafl/" }
libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_edges", "sancov_cmplog", "libfuzzer"] } libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_edges", "sancov_cmplog", "libfuzzer"] }
mimalloc = { version = "*", default-features = false }
[build-dependencies] [build-dependencies]
cc = { version = "1.0", features = ["parallel"] } cc = { version = "1.0", features = ["parallel"] }

View File

@ -1,5 +1,8 @@
//! A libfuzzer-like fuzzer with llmp-multithreading support and restarts //! A libfuzzer-like fuzzer with llmp-multithreading support and restarts
//! The example harness is built for `stb_image`. //! The example harness is built for `stb_image`.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use std::{env, path::PathBuf}; use std::{env, path::PathBuf};

View File

@ -19,6 +19,7 @@ debug = true
libafl = { path = "../../../libafl/", features = ["concolic_mutation"] } libafl = { path = "../../../libafl/", features = ["concolic_mutation"] }
libafl_targets = { path = "../../../libafl_targets/", features = ["sancov_pcguard_edges", "sancov_cmplog", "libfuzzer"] } libafl_targets = { path = "../../../libafl_targets/", features = ["sancov_pcguard_edges", "sancov_cmplog", "libfuzzer"] }
structopt = "0.3.21" structopt = "0.3.21"
mimalloc = { version = "*", default-features = false }
[build-dependencies] [build-dependencies]
cc = { version = "1.0", features = ["parallel"] } cc = { version = "1.0", features = ["parallel"] }

View File

@ -1,5 +1,8 @@
//! A libfuzzer-like fuzzer with llmp-multithreading support and restarts //! A libfuzzer-like fuzzer with llmp-multithreading support and restarts
//! The example harness is built for `stb_image`. //! The example harness is built for `stb_image`.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use std::{env, path::PathBuf}; use std::{env, path::PathBuf};

View File

@ -19,6 +19,7 @@ debug = true
libafl = { path = "../../libafl/" } libafl = { path = "../../libafl/" }
libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_edges", "sancov_cmplog", "libfuzzer"] } libafl_targets = { path = "../../libafl_targets/", features = ["sancov_pcguard_edges", "sancov_cmplog", "libfuzzer"] }
libafl_sugar = { path = "../../libafl_sugar/" } libafl_sugar = { path = "../../libafl_sugar/" }
mimalloc = { version = "*", default-features = false }
[build-dependencies] [build-dependencies]
cc = { version = "1.0", features = ["parallel"] } cc = { version = "1.0", features = ["parallel"] }

View File

@ -1,5 +1,8 @@
//! A libfuzzer-like fuzzer with llmp-multithreading support and restarts //! A libfuzzer-like fuzzer with llmp-multithreading support and restarts
//! The example harness is built for `stb_image`. //! The example harness is built for `stb_image`.
use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;
use std::{env, path::PathBuf}; use std::{env, path::PathBuf};