GNU libextractor -- in-process fuzzing harnesses
================================================

This directory replaces the old `src/plugins/fuzz_default.sh`, which ran
`zzuf` over the test corpus and invoked the `extract` *binary* once per
mutation.  That approach had four problems, and each of them is what one
of the design decisions below is answering:

  * it forked a process per input, so it managed a few hundred
    executions per second where an in-process harness manages hundreds of
    thousands;
  * `extract` runs plugins **out of process**, so a plugin that crashed
    produced a dead child that the library dutifully restarted -- the
    script saw a clean exit status and reported success;
  * it had no sanitizer, so only a crash the kernel delivered was
    noticed.  A read of 200 bytes past the end of a 16 KiB shared memory
    window is invisible without one, and that is precisely the shape of
    almost every bug in this library;
  * it had no coverage feedback, so the hundredth mutation of a file was
    no more likely to reach new code than the first.

Every harness here is *dual mode*:

  * it exports the libFuzzer entry point

        int LLVMFuzzerTestOneInput (const uint8_t *data, size_t size);

    so the same source links against clang/libFuzzer, AFL++ or
    honggfuzz (see `../../contrib/oss-fuzz/`), and

  * it ships a **built-in standalone driver** (`fuzz_common.h`) with a
    deterministic, seeded generator + mutator loop, so it is useful with
    nothing but gcc and `-fsanitize=address,undefined`.

The standalone driver is compiled unless `FUZZ_NO_MAIN` is defined.


-------------------------------------------------------------------
1. The harnesses
-------------------------------------------------------------------

1.1 Per-plugin targets
----------------------

`fuzz_plugin.c` is compiled once per plugin, producing `fuzz_gif`,
`fuzz_png`, `fuzz_rtf` and so on -- 21 in-tree parsers plus one per
plugin that wraps a third-party library, when that library is present.
Each target links the plugin's own `.c` files, so every line of the
parser is instrumented and neither `dlopen()` nor `fork()` nor the IPC
layer is in the way.

One target per format rather than one target for all of them, because
the corpus is what makes a format fuzzer work: `fuzz_png` starts from
PNG files and mutates PNGs, and its coverage feedback is not diluted by
inputs that only ever exercise the RTF tokeniser.

1.2 Core targets
----------------

fuzz_extract        end-to-end: `EXTRACTOR_plugin_add_defaults()` +
                    `EXTRACTOR_extract()`, plugins in-process.  This is
                    the only target that covers plugin discovery, the
                    dispatch loop, and the plugins that wrap a
                    third-party library, all against the real plugin set.

fuzz_datasource     `src/main/extractor_datasource.c`: the first code in
                    the library to touch attacker bytes, and the only one
                    that runs *before* any plugin is consulted.  The gzip
                    header walker (FEXTRA / FNAME / FCOMMENT / FHCRC) and
                    the "seek backwards through a decompressed stream"
                    path are the interesting parts.

fuzz_unzip          `src/common/unzip.c`, the in-tree ZIP reader shared
                    by the odf, msoffice and zip plugins -- hand-written
                    code descended from unzip 1.00, reachable from three
                    formats at once.  The highest-value single target
                    here.

fuzz_ipc            `EXTRACTOR_IPC_process_reply_()`, the parser in the
                    *trusted parent* for a byte stream produced by an
                    *untrusted child*.  That is the security boundary of
                    the whole out-of-process design, so it is fuzzed on
                    its own even though the function is short.

fuzz_convert        `EXTRACTOR_common_convert_to_utf8()` and the metatype
                    tables.  The helper is called by nsfe, rtf, msoffice
                    and png with a length *and* a charset name that both
                    come out of the file being parsed.


-------------------------------------------------------------------
2. Why the harness finds things `extract` does not
-------------------------------------------------------------------

The substance is in `fuzz_ec.h`, which models
`struct EXTRACTOR_ExtractContext` the way `extractor_plugin_main.c`
implements it, and then makes two of its properties enforceable.

2.1 The read window is exact
----------------------------

`plugin_env_read()` hands the plugin a pointer *into a shared memory
window* and returns how many bytes are valid there.  That is at most
`shm_map_size` -- 16 KiB by default -- and at most what is left of the
file.  A plugin that asks for 100 KiB, gets 16 KiB, and then walks all
100 KiB is reading memory it was never given.  In production that memory
is a live mmap, so nothing crashes, no test fails, and the bug is
invisible.

Here every window is a fresh `malloc()` of *exactly* the returned byte
count, so ASAN's redzone starts at the first byte the plugin was not
promised.  The window size is fuzzer-controlled (byte 0 of the input),
because "the read returned less than I asked for" is the single most
productive precondition in this library, and the 16 KiB default hides it
for every file smaller than that.

2.2 The window slides
---------------------

A pointer from `read()` stays valid only while the window still covers
that part of the file.  A `read()` or `seek()` that needs data outside
the window makes the core refill it, and the plugin's old pointer then
addresses *different file bytes* than it believes it holds.

The model tracks the same window as `plugin_env_read()` and
`plugin_env_seek()` do, so a `seek (0, SEEK_CUR)` -- which never leaves
the window -- invalidates nothing, exactly as in production.  When the
window really does slide, every slice handed out from it is freed, so a
retained pointer becomes an ASAN use-after-free.

Getting this boundary right is the difference between a report worth
reading and noise: an earlier version invalidated on *every* seek and
immediately "found" a bug in png_extractor.c that does not exist under
either shipped implementation.  `LE_FUZZ_STRICT_WINDOW=1` restores that
stricter behaviour on purpose, since it models the in-process
implementation, whose single `ctx->buf` really is overwritten by every
read.

2.3 The metadata processor is an oracle
---------------------------------------

`transmit_reply()` writes `data_len` bytes starting at `data` to a pipe
and calls `strlen()` on the mime type.  The harness touches exactly those
bytes, so a plugin that reports a length longer than its buffer is an
ASAN report here and a real out-of-bounds read in production.

`LE_FUZZ_STRICT_PROC=1` additionally flags string metadata that is not
0-terminated.  Off by default: that is a contract violation rather than a
memory error, and it is common enough in the older plugins to drown
everything else.

2.4 Fault injection
-------------------

Byte 1 of the input arms the failures that really can happen and that
plugins rarely handle: `get_size()` returning `UINT64_MAX` after a failed
IPC round, `read()` or `seek()` returning -1, the application asking to
stop after N items, and running the extract method twice against the same
context (plugins must not carry state from one file to the next -- the
same process handles every file in a directory walk).


-------------------------------------------------------------------
3. Input format
-------------------------------------------------------------------

Every `fuzz_<plugin>` target and `fuzz_unzip` take:

    byte 0    read window size selector; 0 selects the production 16 KiB
    byte 1    fault-injection bitmask (LE_FUZZ_FAULT_* in fuzz_ec.h)
    byte 2    call index at which the injected fault fires
    byte 3    auxiliary parameter
    byte 4..  the file image

An all-zero prefix is exactly what production does, so a corpus entry is
four zero bytes followed by a real file.  That is what
`contrib/oss-fuzz/make_seed_corpus.sh` builds out of
`src/plugins/testdata/`.

`fuzz_extract` uses a two-byte prefix, `fuzz_convert` a two-byte prefix,
`fuzz_datasource` four bytes, and `fuzz_ipc` none (its input is the raw
message stream).  Each is documented at the top of its own source file.


-------------------------------------------------------------------
4. Building and running
-------------------------------------------------------------------

    ./bootstrap
    ./configure --enable-fuzzing --enable-static \
                CC=clang \
                CFLAGS="-g -O1 -fno-omit-frame-pointer \
                        -fsanitize=address,undefined" \
                LDFLAGS="-fsanitize=address,undefined"
    make
    make -C src/fuzz check

`make check` runs every harness under its built-in driver for
`LE_FUZZ_ITERATIONS` iterations (20000 by default).

Note that on a tree that does not carry the fixes from `issues.txt` this
`make check` *fails*, by design: the harnesses find those bugs within a
few thousand iterations.  It cannot affect an ordinary build, because
`--enable-fuzzing` defaults to no and `src/fuzz/` is not even configured
without it.

A longer session:

    make -C src/fuzz check LE_FUZZ_ITERATIONS=5000000 LE_FUZZ_SEED=$RANDOM

Every run is fully reproducible from (harness, seed).  A failing input is
dumped to `$LE_FUZZ_CRASH_DIR` and replayed with:

    ./fuzz_png --file=crashes/crash-fuzz_png-seed1-iter28.bin

Replay the whole checked-in corpus, which is what CI should do after a
fix:

    make -C src/fuzz check-corpus

For a real campaign use libFuzzer via `contrib/oss-fuzz/build.sh`; see
`../../contrib/oss-fuzz/README`.

`CAMPAIGN.md` answers the two questions that come up as soon as the
campaign is longer than a coffee break -- how to split the budget across
the targets, and how long it is worth running -- from a measured
12-core hour rather than from intuition.  Both answers are unobvious:
half the targets reach 99% of their final coverage within 93 seconds,
and a flat allocation spends most of its budget on targets that finished
in the first minute.  `contrib/oss-fuzz/run_campaign.sh` implements the
conclusion:

    contrib/oss-fuzz/run_campaign.sh -b /path/to/build.sh-output -p nightly

Environment knobs, all read once at startup:

    LE_FUZZ_ITERATIONS    iterations of the built-in driver
    LE_FUZZ_SEED          PRNG seed
    LE_FUZZ_TIMEOUT       per-iteration watchdog, seconds (0 disables)
    LE_FUZZ_CRASH_DIR     where reproducers are written
    LE_FUZZ_SKIP_SEEDS    skip the built-in seed corpus at the start
    LE_FUZZ_VERBOSE       be chatty
    LE_FUZZ_STRICT_WINDOW invalidate the read window on every call
    LE_FUZZ_STRICT_PROC   flag non-0-terminated string metadata

`fuzz_extract` additionally needs `LIBEXTRACTOR_PREFIX` pointing at the
directory holding the built plugin modules, normally
`src/plugins/.libs`; `make check` sets it.


-------------------------------------------------------------------
5. The corpus
-------------------------------------------------------------------

`corpus/<target>/` holds the harnesses' own built-in seeds and is
regenerated by

    make -C src/fuzz refresh-corpus

It deliberately does *not* contain copies of `src/plugins/testdata/`:
those files are already in the tree, and
`contrib/oss-fuzz/make_seed_corpus.sh` prepends the configuration prefix
to them at build time instead.  Run that script by hand to materialise
the full corpus for a local campaign.

`corpus/known-findings/` holds the reproducers for the entries in
`issues.txt`.  `make check-corpus` replays it, so each one stays a
permanent regression test.


-------------------------------------------------------------------
6. Adding a plugin target
-------------------------------------------------------------------

1. Give the plugin an `LE_ID_*` number in `fuzz_plugin_name.h` and add
   the `#elif` block with its magic bytes and body shape.  This only
   feeds the *generator*; getting it wrong costs coverage, never
   soundness.
2. Add the target to `Makefile.am`, following one of the existing
   blocks: `_SOURCES = fuzz_plugin.c`, `_CPPFLAGS` with the two `-D`
   flags, `nodist_..._SOURCES` with the plugin's own sources, and the
   libraries it needs.
3. Add it to `PLUGIN_FUZZERS` in `../../contrib/oss-fuzz/build.sh` and
   to the `testdata_glob` table in
   `../../contrib/oss-fuzz/make_seed_corpus.sh`.
