From 42b319980794895c0782953a128fe73c88ab12bc Mon Sep 17 00:00:00 2001 From: Romain Malmain Date: Thu, 26 Sep 2024 16:29:32 +0200 Subject: [PATCH] Change action for MD link checks (#2563) * change MD link action checker * fix md files --- .github/workflows/.linkspector.yml | 14 ++++++++++++++ .github/workflows/build_and_test.yml | 20 ++++++++++++++------ .github/workflows/md-config.json | 11 ----------- docs/src/advanced_features/concolic.md | 10 +++++----- docs/src/advanced_features/frida.md | 4 ++-- docs/src/advanced_features/nyx.md | 6 +++--- docs/src/getting_started/crates.md | 2 +- docs/src/tutorial/intro.md | 2 +- scripts/check_md_links.sh | 17 +++++++++++++++++ 9 files changed, 57 insertions(+), 29 deletions(-) create mode 100644 .github/workflows/.linkspector.yml delete mode 100644 .github/workflows/md-config.json create mode 100755 scripts/check_md_links.sh diff --git a/.github/workflows/.linkspector.yml b/.github/workflows/.linkspector.yml new file mode 100644 index 0000000000..e563b9145d --- /dev/null +++ b/.github/workflows/.linkspector.yml @@ -0,0 +1,14 @@ +dirs: + - . + +useGitIgnore: true + +ignorePatterns: + - pattern: "^https://crates.io" + - pattern: "^https://github.com/AFLplusplus/linux-qemu-image-builder" + - pattern: "https://www.romu-random.org/" + +aliveStatusCodes: + - 0 + - 200 + - 403 \ No newline at end of file diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index a4a63f0f2f..ff79b57716 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -221,14 +221,22 @@ jobs: - name: Format Check run: ./scripts/fmt_all.sh check - md-links-check: - runs-on: ubuntu-24.04 + check-md-links: + runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - name: Run Markdown Link checks - uses: gaurav-nelson/github-action-markdown-link-check@v1 - with: - config-file: '.github/workflows/md-config.json' + - name: Install linkspector + shell: bash + run: sudo apt-get install -y npm && npm install -g @umbrelladocs/linkspector + - name: Run linkspector + shell: bash + run: ./scripts/check_md_links.sh + # TODO: Use github action once it's fixed (https://github.com/UmbrellaDocs/action-linkspector/issues/20) + # - name: Run linkspector + # uses: umbrelladocs/action-linkspector@v1 + # with: + # fail_on_error: 'true' + # config_file: '.github/workflows/.linkspector.yml' fuzzers-preflight: runs-on: ubuntu-24.04 diff --git a/.github/workflows/md-config.json b/.github/workflows/md-config.json deleted file mode 100644 index ae7e387a29..0000000000 --- a/.github/workflows/md-config.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "ignorePatterns": [ - { - "pattern": "^https://crates.io" - }, - { - "pattern": "^https://github.com/AFLplusplus/linux-qemu-image-builder" - } - ], - "aliveStatusCodes": [0, 200, 403] -} \ No newline at end of file diff --git a/docs/src/advanced_features/concolic.md b/docs/src/advanced_features/concolic.md index c597c51b9b..aa636df2c1 100644 --- a/docs/src/advanced_features/concolic.md +++ b/docs/src/advanced_features/concolic.md @@ -115,7 +115,7 @@ The `symcc_runtime` crate supports this use case and runtimes built with `symcc_ ## Hybrid Fuzzing in LibAFL -The LibAFL repository contains an [example hybrid fuzzer](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/stb/libfuzzer_stb_image_concolic). +The LibAFL repository contains an [example hybrid fuzzer](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/inprocess/libfuzzer_stb_image_concolic). There are three main steps involved with building a hybrid fuzzer using LibAFL: @@ -130,7 +130,7 @@ For example, we need to have a runtime ready before we can do instrumentation wi Building a custom runtime can be done easily using the `symcc_runtime` crate. Note, that a custom runtime is a separate shared object file, which means that we need a separate crate for our runtime. -Check out the [example hybrid fuzzer's runtime](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/stb/libfuzzer_stb_image_concolic/runtime) and the [`symcc_runtime` docs](https://docs.rs/symcc_runtime/0.1/symcc_runtime) for inspiration. +Check out the [example hybrid fuzzer's runtime](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/inprocess/libfuzzer_stb_image_concolic/runtime) and the [`symcc_runtime` docs](https://docs.rs/symcc_runtime/0.1/symcc_runtime) for inspiration. ### Instrumentation @@ -151,7 +151,7 @@ How exactly this is done does not matter. However, the SymCC compiler needs to be made aware of the location of the runtime that it should instrument against. This is done by setting the `SYMCC_RUNTIME_DIR` environment variable to the directory which contains the runtime (typically the `target/(debug|release)` folder of your runtime crate). -The example hybrid fuzzer instruments the target in its [`build.rs` build script](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/stb/libfuzzer_stb_image_concolic/fuzzer/build.rs#L50). +The example hybrid fuzzer instruments the target in its [`build.rs` build script](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/inprocess/libfuzzer_stb_image_concolic/fuzzer/build.rs#L50). It does this by cloning and building a copy of SymCC and then using this version to instrument the target. The [`symcc_libafl` crate](https://docs.rs/symcc_libafl) contains helper functions for cloning and building SymCC. @@ -169,7 +169,7 @@ No matter the instrumentation method, the interface between the fuzzer and the i The only difference between using SymCC and SymQEMU should be the binary that represents the target: In the case of SymCC it will be the binary that was build with instrumentation and with SymQEMU it will be the emulator binary (eg. `x86_64-linux-user/symqemu-x86_64`), followed by your uninstrumented target binary and its arguments. -You can use the [`CommandExecutor`](https://docs.rs/libafl/latest/libafl/executors/command/struct.CommandExecutor.html) to execute your target ([example](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/stb/libfuzzer_stb_image_concolic/fuzzer/src/main.rs#L244)). +You can use the [`CommandExecutor`](https://docs.rs/libafl/latest/libafl/executors/command/struct.CommandExecutor.html) to execute your target ([example](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/inprocess/libfuzzer_stb_image_concolic/fuzzer/src/main.rs#L244)). When configuring the command, make sure you pass the `SYMCC_INPUT_FILE` environment variable (set to the input file path), if your target reads input from a file (instead of standard input). #### Serialization and Solving @@ -184,4 +184,4 @@ It will attempt to solve all branches, like the original simple backend from Sym ### Example -The example fuzzer shows how to use the [`ConcolicTracingStage` together with the `SimpleConcolicMutationalStage`](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/stb/libfuzzer_stb_image_concolic/fuzzer/src/main.rs#L222) to build a basic hybrid fuzzer. +The example fuzzer shows how to use the [`ConcolicTracingStage` together with the `SimpleConcolicMutationalStage`](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/inprocess/libfuzzer_stb_image_concolic/fuzzer/src/main.rs#L222) to build a basic hybrid fuzzer. diff --git a/docs/src/advanced_features/frida.md b/docs/src/advanced_features/frida.md index b7ba567adb..97e0ab2324 100644 --- a/docs/src/advanced_features/frida.md +++ b/docs/src/advanced_features/frida.md @@ -4,7 +4,7 @@ LibAFL supports different instrumentation engines for binary-only fuzzing. A potent cross-platform (Windows, MacOS, Android, Linux, iOS) option for binary-only fuzzing is Frida; the dynamic instrumentation tool. In this section, we will talk about the components in fuzzing with `libafl_frida`. -You can take a look at a working example in our [`fuzzers/frida/frida_libpng`](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/frida/frida_libpng) folder for Linux, and [`fuzzers/frida/frida_gdiplus`](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/frida/frida_gdiplus) for Windows. +You can take a look at a working example in our [`fuzzers/binary-only/frida_libpng`](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/binary-only/frida_libpng) folder for Linux, and [`fuzzers/binary-only/frida_gdiplus`](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/binary-only/frida_gdiplus) for Windows. ## Dependencies @@ -84,4 +84,4 @@ You can then link this observer to `FridaInProcessExecutor` as follows: ``` And finally you can run the fuzzer. -See the `frida_` examples in [`./fuzzers/frida`](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/frida/) for more information and, for linux or full-system, play around with `libafl_qemu`, another binary-only tracer. +See the `frida_` examples in [`./fuzzers/binary-only`](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/binary-only/) for more information and, for linux or full-system, play around with `libafl_qemu`, another binary-only tracer. diff --git a/docs/src/advanced_features/nyx.md b/docs/src/advanced_features/nyx.md index a48b4e2dbd..40dfeb8b6b 100644 --- a/docs/src/advanced_features/nyx.md +++ b/docs/src/advanced_features/nyx.md @@ -24,7 +24,7 @@ For binary-only fuzzing, Nyx uses intel-PT(Intel® Processor Trace). You can fin ## Preparing the Nyx working directory -This step is used to pack the target into Nyx's kernel. Don't worry, we have a template shell script in our [example](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/nyx/nyx_libxml2_parallel/setup_libxml2.sh): +This step is used to pack the target into Nyx's kernel. Don't worry, we have a template shell script in our [example](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/full-system/nyx_libxml2_parallel/setup_libxml2.sh): the parameter's meaning is listed below: @@ -49,7 +49,7 @@ python3 ./packer/packer/nyx_config_gen.py /tmp/nyx_libxml2/ Kernel || exit ## Standalone fuzzing -In the [example fuzzer](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/nyx/nyx_libxml2_standalone/src/main.rs) you first need to run `./setup_libxml2.sh`. It will prepare your target and create your nyx work directory in `/tmp/libxml2`. After that, you can start to write your code. +In the [example fuzzer](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/full-system/nyx_libxml2_standalone/src/main.rs) you first need to run `./setup_libxml2.sh`. It will prepare your target and create your nyx work directory in `/tmp/libxml2`. After that, you can start to write your code. First, to create `Nyxhelper`: @@ -71,7 +71,7 @@ Finally, use them normally and pass them into `fuzzer.fuzz_loop(&mut stages, &mu ## Parallel fuzzing -In the [example fuzzer](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/nyx/nyx_libxml2_parallel/src/main.rs) you first need to run `./setup_libxml2.sh` as described before. +In the [example fuzzer](https://github.com/AFLplusplus/LibAFL/blob/main/fuzzers/full-system/nyx_libxml2_parallel/src/main.rs) you first need to run `./setup_libxml2.sh` as described before. Parallel fuzzing relies on [`Launcher`](../message_passing/spawn_instances.md), so spawn logic should be written in the scoop of anonymous function `run_client`: diff --git a/docs/src/getting_started/crates.md b/docs/src/getting_started/crates.md index f6e29ecfd7..e90ad3c269 100644 --- a/docs/src/getting_started/crates.md +++ b/docs/src/getting_started/crates.md @@ -51,7 +51,7 @@ In it, you'll find highlights like: The sugar crate abstracts away most of the complexity of LibAFL's API. Instead of high flexibility, it aims to be high-level and easy-to-use. It is not as flexible as stitching your fuzzer together from each individual component, but allows you to build a fuzzer with minimal lines of code. -To see it in action, take a look at the [`libfuzzer_stb_image_sugar` example fuzzer](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/stb/libfuzzer_stb_image_sugar). +To see it in action, take a look at the [`libfuzzer_stb_image_sugar` example fuzzer](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/inprocess/libfuzzer_stb_image_sugar). ### [`libafl_derive`](https://github.com/AFLplusplus/LibAFL/tree/main/libafl_derive) diff --git a/docs/src/tutorial/intro.md b/docs/src/tutorial/intro.md index a0b46df45d..375e31125d 100644 --- a/docs/src/tutorial/intro.md +++ b/docs/src/tutorial/intro.md @@ -5,4 +5,4 @@ > This section is under construction. > Please check back later (or open a PR) > -> In the meantime, find the final Lain-based fuzzer in [the fuzzers folder](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/others/tutorial) +> In the meantime, find the final Lain-based fuzzer in [the fuzzers folder](https://github.com/AFLplusplus/LibAFL/tree/main/fuzzers/inprocess/tutorial) diff --git a/scripts/check_md_links.sh b/scripts/check_md_links.sh new file mode 100755 index 0000000000..7352da59c4 --- /dev/null +++ b/scripts/check_md_links.sh @@ -0,0 +1,17 @@ +#!/bin/bash + +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" +LIBAFL_DIR=$(realpath "$SCRIPT_DIR/..") + +echo "[*] Checking MD links..." + +cd "$LIBAFL" || exit 1 + +if ! command -v linkspector > /dev/null; then + echo "Error: install linkspector to check MD file links." + exit 1 +fi + +linkspector check -c "${LIBAFL_DIR}/.github/workflows/.linkspector.yml" || exit 1 + +echo "[*] Done :)"