Seed corpus for the src/fuzz harnesses
======================================

`<harness>-NN.bin` is the built-in seed corpus of that harness, dumped
to disk.  Regenerate at any time with

    ./fuzz_<harness> --write-corpus=<this directory>

or, from the build tree,

    make -C src/fuzz refresh-corpus

Replay everything (this is what a CI regression run should do):

    make -C src/fuzz check-corpus
    # or, per harness:
    ./fuzz_request --corpus-dir=src/fuzz/corpus

Files belonging to another harness are simply uninteresting inputs for
the harness that reads them, so pointing every harness at the whole
directory is fine and gives some extra cross-pollination.

The set is kept minimal: every seed here adds coverage that no other
seed provides.  After changing a seed table, re-check that with

    <target> -merge=1 <empty dir> <this directory>

from an OSS-Fuzz style build (contrib/oss-fuzz/build.sh); a seed the
merge does not copy across is redundant and should be deleted from the
table rather than left in the corpus.  Note that fuzz_request is
mildly non-deterministic -- MHD timestamps its nonces and its
connection timeouts -- so a single merge decision worth ~0.1% of the
regions is noise; only drop a seed that several independent merges
agree on.


fuzz_request seeds
------------------

  00  content-length-body              body oracle, Content-Length
  01  chunked-with-extensions          body oracle, chunk extensions
                                       -> chunk-extension CRLF bug
  02  chunked-split                    body oracle, chunk boundary split
                                       across two send() calls
  03  small-pool-trailing-query-arg    128..512 byte connection pool, no
                                       header lines, "?novalue"
                                       -> read-buffer shift-back bug
  04  small-pool-trailing-query-arg-2  same with "?a=1&b"
  05  digest-unknown-algorithm         algorithm=BOGUS
                                       -> MHD_DIGEST_AUTH_ALGO3_INVALID
                                          MHD_PANIC()
  06  digest-overlong-response         401 challenge, then a replay on a
                                       fresh connection with the
                                       harvested nonce and a 128 hex
                                       digit response=
                                       -> stack buffer overflow
  07  digest-userhash                  userhash=true with a 128 char
                                       username
  08  basic-auth                       Authorization: Basic
  09  multipart-post                   chunked multipart/form-data
  10  urlencoded-post                  application/x-www-form-urlencoded
  11  folded-header                    obs-fold continuation line
  12  pipelined                        two requests in one segment

Seeds 00-12 leave the API-selection block (bytes 4-9, see section 2.2.1
of ../README) zero and only drive the request parser.  Seeds 13-39 each
switch on one area of it; without them those entry points are reachable
only by guessing six configuration bytes.

  13  ext-callback-chunked             MHD_SIZE_UNKNOWN content reader,
                                       response headers and a trailer
  14  ext-callback-error               known-length reader that fails
                                       part way through
  15  ext-fd-response                  MHD_create_response_from_fd()
  16  ext-fd-at-offset-response        ..._from_fd_at_offset()
  17  ext-pipe-response                ..._from_pipe()
  18  ext-iovec-response               ..._from_iovec()
  19  ext-empty-response-hdrapi        ..._empty() plus get/del header
                                       and MHD_set_response_options()
  20  ext-digest-check-digest3         pre-computed userdigest path
  21  ext-digest-v1-wrappers           MHD_digest_auth_check()
  22  ext-digest-request-info          MHD_digest_auth_get_request_info3()
  23  ext-external-event-loop          MHD_get_fdset2()/run_from_select2()
                                       plus the timeout accessors
  24  ext-external-event-loop-v1       the v1 fdset/run_from_select pair
  25  ext-connection-api               every introspection accessor
  26  ext-suspend-resume               suspend, resume from the pump loop
  27  ext-suspend-two-connections      two connections parked at once
  28  ext-upgrade                      101 + MHD_upgrade_action()
  29  ext-buffer-persistent            buffer/MHD_RESPMEM_PERSISTENT
  30  ext-buffer-free-callback         ..._with_free_callback()
  31  ext-from-data                    MHD_create_response_from_data()
  32  ext-fd64-response                ..._from_fd64()
  33  ext-basic-auth-v1                MHD_basic_auth_get_username_password()
  34  ext-basic-auth-challenge         MHD_queue_basic_auth_fail_response()
  35  ext-basic-auth-challenge-utf8    ..._response3() with UTF-8 charset
  36  ext-digest-check-digest-v1       MHD_digest_auth_check_digest()
  37  ext-digest-check-digest2         ..._digest2()  (never with
                                       MHD_DIGEST_ALG_AUTO, see K8)
  38  ext-digest-get-username-v1       MHD_digest_auth_get_username()
  39  ext-digest-get-username3         ..._username3()

fuzz_str seeds
--------------

  Cover MHD_hex_to_bin (including the 128 character input that
  digestauth.c used to allow into a 32 byte buffer), MHD_bin_to_hex[_z],
  the percent-decoders (strict, lenient, in place), MHD_str_unquote,
  MHD_str_quote, MHD_base64_to_bin_n, MHD_str[x]_to_uint64_n_ and the
  token helpers.

fuzz_auth_header seeds
----------------------

  Well-formed and broken Digest parameter lists (unknown algorithm,
  extended `username*` notation, unterminated quoted strings, empty
  parameter list, over-long values) plus Basic `token68` variants, and
  -- with bit 0x02 of byte 0 -- the connection-less digest helpers at
  several algorithms and several deliberately undersized output
  buffers.

fuzz_postprocessor seeds
------------------------

  urlencoded bodies with broken percent escapes, multipart bodies with
  ordinary, degenerate ("-") and quote-containing boundaries, a
  multipart Content-Type without a boundary, and a non-POST
  Content-Type.


distilled/
----------

The edge-minimal residue of the fuzzing campaigns: 2562 inputs, 240 KB,
named `<harness>-dNNN.bin` and routed by that prefix exactly like the
seeds above.  Files are numbered in the order the cover picked them, so
`-d000` is the highest-gain input of its harness.

Unlike `<harness>-NN.bin` these are *not* generated from a built-in
table, so `refresh-corpus` neither writes nor clobbers them; and unlike
`known-findings/` they are not hand-edited and carry no individual
meaning.  Do not document them one by one -- the set is only ever
regenerated wholesale.

Provenance: an 8 hour, 12 core, 1.62 billion execution campaign
(ASan+UBSan, libFuzzer, hourly restarts) starting from the seed corpus
above.  The campaign corpus was 18 226 inputs / 12 MB, which is too much
to carry in a source tree, so it was reduced by greedy set cover over
*edges* rather than by `-merge=1`, which minimises for features and
keeps roughly seven times as many files:

    <target> -merge=1 -merge_control_file=mcf.txt <empty dir> <corpus>

writes a `COV` line per input, and the cover is computed from those
offline.  Because these harnesses are mildly non-deterministic, the
cover is built only from edges that two independent merges agree an
input reproduces -- the same "several independent merges" rule stated at
the top of this file.

The `fuzz_eventloop` and `fuzz_tls` inputs come from a second, 1 hour /
4 core / 17.0 million execution campaign (same sanitizers, same engine,
`--enable-https` so that `fuzz_tls` could be built at all -- note that
`contrib/oss-fuzz/build.sh` configures `--disable-https`, which is why
`fuzz_tls` had no distilled inputs before and started that campaign from
its seed table alone).  It ran after the PSK scenario and the
queue-and-stop scenario were added to those two harnesses, and its
starting corpus was this directory plus the first campaign's residue, so
what it contributed is beyond what 8 core-hours per harness had already
reached: `fuzz_tls` 6355 -> 6844 edges, `fuzz_eventloop` 7406 -> 7655.
The other six harnesses were not re-run; their inputs are unchanged.

A `COV` line holds only the edges its input was *first* to reach in that
merge, so what a cover has to pay for depends on what it is credited
with already having.  Credit it with everything else that stays in the
tree -- both seed sets and the distilled inputs of the harnesses not
being regenerated -- because `check-corpus` replays all three
directories through every harness, and an input is worth carrying only
if it adds an edge none of them reach.  Doing that costs nothing and is
worth roughly a third of the files: the `fuzz_tls` cover needed 490
edges and 167 inputs against the whole tree, against 1580 edges and 282
inputs when credited with the seed corpus alone -- identical coverage
either way, because the other harnesses' inputs were already reaching
those 1090 edges.  Compute every harness's cover against the same base
so that no two of them depend on each other.

**An engine-produced corpus is not pristine, and this directory has to
be filtered before it can be committed.**  `fuzz_request` inputs carry a
ground-truth body declaration in their `op == 1` segment, and the body
oracle is armed only for a `fuzz_pristine` input precisely because
"random mutations invalidate such declarations" (`fuzz_common.h`).
libFuzzer mutates that segment independently of the wire segments, so a
campaign corpus is full of inputs whose declaration no longer describes
their own request -- 383 of 6884 here, 5.6%.  Under libFuzzer that is
harmless: `-DFUZZ_NO_MAIN` compiles out every assignment to
`fuzz_pristine`, so the oracle never arms and the campaign never
notices.  But `--corpus-dir` sets `fuzz_pristine = 1` for *every* file it
reads, so replaying such an input reports a "request body desync"
against a declaration that was never true -- a false positive, and one
that looks exactly like a request-smuggling finding.

Committing the raw cover therefore broke `make check-corpus` with 42
such reports.  The fix is to sweep every candidate with
`--file=` first and exclude the ones that fire *before* computing the
cover, which is what produced the set in this directory; all 2562 files
replay clean through all eight harnesses.  Anyone replaying a downloaded
ClusterFuzz corpus locally will hit the same thing, and it is not a bug
in MHD.

Sweep every candidate anyway, but expect this particular exclusion only
for `fuzz_request`: it is the sole reader of `fuzz_pristine`, so it is
the only harness that can report a desync against its own stale
declaration.  The second campaign's 15 554 candidates produced no
rejects at all for that reason.

The set reaches ~99% of the first campaign's edges, not 100%: 3763/3798
for fuzz_request, and exactly 100% for the four direct-API harnesses,
which are deterministic.  Two thirds of that shortfall is the exclusion
above -- 64 edges were reached only by inputs with a stale declaration
-- and the rest is edges no single input reproduces reliably.  Forcing
the latter in by adding every input that covers a rare edge was tried
and moved the number by one, so that part is inherent to minimising
rather than a fixable omission.  Trading 1% of edges for 86% fewer files
is the intended bargain.  Do not chase it.

The second campaign lost nothing: `fuzz_eventloop` ends at 7655 of the
7656 edges its campaign corpus reaches, and `fuzz_tls` at 6844 against
that corpus' 6770 -- above it, because a `fuzz_tls` cover measured
inside the tree also gets the other harnesses' inputs, which are
uninteresting to it but not inert.  Both harnesses are deterministic
enough here that all three `-runs=0` measurements agreed to within two
edges.

To regenerate after a long campaign: keep the campaign corpus, sweep it
for oracle-firing inputs, run two independent merges over what is left,
recompute the cover against everything that stays in the tree, and
replace the affected harnesses' files wholesale.  Check the result the
same way it was checked here -- replay both the campaign corpus and the
distilled set with `-runs=0` three times each and compare the
`INITED cov:` figures, because a single measurement of either is worth a
few edges of noise -- and finish with `make check-corpus`.

Measure *every* harness before and after, not just the ones being
regenerated: dropping an input removes it from the other seven as well.
Replacing the 320 first-campaign `fuzz_eventloop` inputs with 248 new
ones moved the other six harnesses by at most three edges, which is
noise, but that is a thing to confirm rather than assume.  Note also
that `make check-corpus` is a weaker gate than it looks on a tree
configured without `--enable-asserts`, where `mhd_assert()` compiles
away entirely; the campaign build has them, so inputs that survived the
merges have already been through the stronger one.


known-findings/
---------------

Reproducers for the issues that this suite found in MHD itself; see
section 6 of ../README, which records the status of each.  All of them
are fixed on master, so all of them replay clean, and `check-corpus`
replays this directory too.

A file is named `K<n>-<harness>-<what>.bin` for the harness that found
it; the ones without a harness in the name predate that convention and
are all fuzz_request inputs.  `contrib/oss-fuzz/make_seed_corpus.sh`
routes each into the seed corpus of its own harness on that basis.

They are kept as a separate, explicitly named set rather than being
folded into the main corpus because the files there are *generated*: a
hand-written input would be clobbered or renumbered by the next
`refresh-corpus`.  Replay one with

    ./fuzz_request --file=src/fuzz/corpus/known-findings/K1-digest-empty-realm.bin

  K1-digest-empty-realm.bin          digestauth.c is_param_equal()
                                     mhd_assert (0 != param->value.len)
  K2-chunkext-no-space.bin           connection.c handle_recv_no_space()
  K3-chunkext-stop-with-error.bin    connection.c
                                     transmit_error_response_len()
  K4a-wsp-first-header.bin           connection.c get_req_header(),
                                     CLIENT_DISCIPLINE_LVL <= -1
  K4b-empty-header-name.bin          connection.c get_req_header(),
                                     CLIENT_DISCIPLINE_LVL <= -2
  K5-bare-cr-keep.bin                connection.c get_req_header(),
                                     CLIENT_DISCIPLINE_LVL == -3
  K6-nonce-length-collision.bin      digestauth.c check_nonce_nc()
                                     mhd_assert (0 == nn->nonce[noncelen])
  K7-upgrade-after-must-close.bin    connection.c build_header_response()
                                     mhd_assert (upgrade -> MUST_UPGRADE);
                                     a single request carrying both
                                     Content-Length and chunked
                                     Transfer-Encoding, then an upgrade
  K9-fuzz_postprocessor-             postprocessor.c post_process_multipart()
    nested-boundary-leak.bin         leaked the previous nested boundary
                                     on every extra nested
                                     multipart/mixed part
  K14-fuzz_eventloop-                daemon.c call_handlers() asserted a
    force-close-not-closed.bin       post-condition that
                                     MHD_connection_handle_read() does
                                     not guarantee when the read buffer
                                     is full
                                     *** still open, see ../../../patches ***
                                     Only fires on an --enable-asserts
                                     *ASan* build: the redzones change
                                     the pool arithmetic that reaches it.
  K15-fuzz_eventloop-              daemon.c close_all_connections()
    stop-with-queued-connection.bin  asserted that only a daemon with an
                                     internal polling thread can have
                                     connections queued by
                                     MHD_add_connection(), while the code
                                     that fills that list queues for any
                                     thread-safe daemon.  An external
                                     event loop that adds a connection and
                                     stops before the next MHD_run()
                                     aborts on an --enable-asserts build.

These files are *not* regenerated by `--write-corpus`; they are edited by
hand.  When the input format changes they have to be migrated, and the
migration has to be checked -- replay each one with `--verbose` before
and after and confirm the daemon, handler, body and challenge counts are
unchanged.  The last such change made the API-selection block
unconditional, which is six zero bytes inserted at offset 4 for K1-K6.

K8 has no reproducer here: its trigger is an argument the application
chooses, not network input, so the harness never performs it.  See
section 6 of ../README for how to reach it by hand.

A reproducer added while its finding is still open will make
`make check-corpus` fail, which is intended -- that is what a regression
test for an unfixed bug does.  `contrib/oss-fuzz/make_seed_corpus.sh`
skips such a file so that ClusterFuzz does not spend every run
rediscovering it; the rule it uses is whether `patches/$ID.diff` exists.
