e103d7b4a6
Fixes: Enable RLE blocks for superblock mode Fix the limitation that the literals block must shrink. Instead, when we're within 200 bytes of the next header byte size, we will just use the next one up. That way we should (almost?) always have space for the table. Remove the limitation that the first sub-block MUST have compressed literals and be compressed. Now one sub-block MUST be compressed (otherwise we fall back to raw block which is okay, since that is streamable). If no block has compressed literals that is okay, we will fix up the next Huffman table. Handle the case where the last sub-block is uncompressed (maybe it is very small). Before it would skip superblock in this case, now we allow the last sub-block to be uncompressed. To do this we need to regenerate the correct repcodes. Respect disableLiteralsCompression in superblock mode Fix superblock mode to handle a block consisting of only compressed literals Fix a off by 1 error in superblock mode that disabled it whenever there were last literals Fix superblock mode with long literals/matches (> 0xFFFF) Allow superblock mode to repeat Huffman tables Respect ZSTD_minGain(). Tests: Simple check for the condition in #2096. When the simple_round_trip fuzzer enables superblock mode, it checks that the compressed size isn't expanded too much. Remaining limitations: O(targetCBlockSize^2) because we recompute statistics every sequence Unable to split literals of length > targetCBlockSize into multiple sequences Refuses to generate sub-blocks that don't shrink the compressed data, so we could end up with large sub-blocks. We should emit those sections as uncompressed blocks instead. ... Fixes #2096
Fuzzing
Each fuzzing target can be built with multiple engines. Zstd provides a fuzz corpus for each target that can be downloaded with the command:
make corpora
It will download each corpus into ./corpora/TARGET.
fuzz.py
fuzz.py is a helper script for building and running fuzzers.
Run ./fuzz.py -h for the commands and run ./fuzz.py COMMAND -h for
command specific help.
Generating Data
fuzz.py provides a utility to generate seed data for each fuzzer.
make -C ../tests decodecorpus
./fuzz.py gen TARGET
By default it outputs 100 samples, each at most 8KB into corpora/TARGET-seed,
but that can be configured with the --number, --max-size-log and --seed
flags.
Build
It respects the usual build environment variables CC, CFLAGS, etc.
The environment variables can be overridden with the corresponding flags
--cc, --cflags, etc.
The specific fuzzing engine is selected with LIB_FUZZING_ENGINE or
--lib-fuzzing-engine, the default is libregression.a.
Alternatively, you can use Clang's built in fuzzing engine with
--enable-fuzzer.
It has flags that can easily set up sanitizers --enable-{a,ub,m}san, and
coverage instrumentation --enable-coverage.
It sets sane defaults which can be overridden with flags --debug,
--enable-ubsan-pointer-overflow, etc.
Run ./fuzz.py build -h for help.
Running Fuzzers
./fuzz.py can run libfuzzer, afl, and regression tests.
See the help of the relevant command for options.
Flags not parsed by fuzz.py are passed to the fuzzing engine.
The command used to run the fuzzer is printed for debugging.
LibFuzzer
# Build the fuzz targets
./fuzz.py build all --enable-fuzzer --enable-asan --enable-ubsan --cc clang --cxx clang++
# OR equivalently
CC=clang CXX=clang++ ./fuzz.py build all --enable-fuzzer --enable-asan --enable-ubsan
# Run the fuzzer
./fuzz.py libfuzzer TARGET <libfuzzer args like -jobs=4>
where TARGET could be simple_decompress, stream_round_trip, etc.
MSAN
Fuzzing with libFuzzer and MSAN is as easy as:
CC=clang CXX=clang++ ./fuzz.py build all --enable-fuzzer --enable-msan
./fuzz.py libfuzzer TARGET <libfuzzer args>
fuzz.py respects the environment variables / flags MSAN_EXTRA_CPPFLAGS,
MSAN_EXTRA_CFLAGS, MSAN_EXTRA_CXXFLAGS, MSAN_EXTRA_LDFLAGS to easily pass
the extra parameters only for MSAN.
AFL
The default LIB_FUZZING_ENGINE is libregression.a, which produces a binary
that AFL can use.
# Build the fuzz targets
CC=afl-clang CXX=afl-clang++ ./fuzz.py build all --enable-asan --enable-ubsan
# Run the fuzzer without a memory limit because of ASAN
./fuzz.py afl TARGET -m none
Regression Testing
The regression test supports the all target to run all the fuzzers in one
command.
CC=clang CXX=clang++ ./fuzz.py build all --enable-asan --enable-ubsan
./fuzz.py regression all
CC=clang CXX=clang++ ./fuzz.py build all --enable-msan
./fuzz.py regression all