9379 Commits

Author SHA1 Message Date
Ma Lin
cc22042da0 Fix a C89 error in msvc
Variables (r) must be declared at the beginning of a code block.
This causes msvc2012 to fail to compile 64-bit build.
2021-09-25 16:32:06 +08:00
Nick Terrell
b10085d97d [contrib][linux-kernel] Add standard warnings and -Werror to CI
Test the kernel build with the standard warnings enabled so that we
don't miss issues like fixed in PR #2802.

Stacked on top of PR #2802 so CI passes, so it includes the fix. But, I
won't merge until it is merged, so @solbjorn gets the credit for the
fix.
2021-09-24 15:08:11 -07:00
Nick Terrell
32a8443b5c
Merge pull request #2790 from solbjorn/huf-asm-kernel
[contrib][linux] Fix build after introducing ASM HUF implementation
2021-09-24 12:42:03 -07:00
Nick Terrell
14772d97be
Merge pull request #2796 from terrelln/linux-fixes
[lib] Make lib compatible with `-Wfall-through` excepting legacy
2021-09-23 16:11:53 -07:00
Nick Terrell
01976ce4cd
Merge pull request #2799 from terrelln/oss-fuzz-build
[build] Fix oss-fuzz build with the dataflow sanitizer
2021-09-23 15:55:10 -07:00
Nick Terrell
1903d6a5a8
Merge pull request #2798 from abxhr/typo-fix
Fix typo
2021-09-23 13:11:45 -07:00
Nick Terrell
d7ef97a013 [build] Fix oss-fuzz build with the dataflow sanitizer
The dataflow sanitizer requires all code to be instrumented. We can't
instrument the ASM function, so we have to disable it.
2021-09-23 11:48:39 -07:00
Yann Collet
9cb5d35cc5
Merge pull request #2797 from facebook/btconst
minor: constify MatchState* parameter when possible
2021-09-23 11:37:15 -07:00
Abshar Mohammed Aslam
54a888b57b
Fix typo 2021-09-23 21:54:38 +04:00
Nick Terrell
189e87bcbe [lib] Make lib compatible with -Wfall-through excepting legacy
Switch to a macro `ZSTD_FALLTHROUGH;` instead of a comment. On supported
compilers this uses an attribute, otherwise it becomes a comment.

This is necessary to be compatible with clang's `-Wfall-through`, and
gcc's `-Wfall-through=2` which don't support comments. Without this the
linux build emits a bunch of warnings.

Also add a test to CI to ensure that we don't regress.
2021-09-23 10:51:18 -07:00
Yann Collet
fa2a4d77c7 constify MatchState* parameter when possible
turns out, it's possible to constify MatchState* parameter
in some parts of the binary tree algorithm,
making it a pure read-only parameter,
as opposed to a mutable state.

This is supposed to be helpful for both maintenance and the compiler.
2021-09-23 08:27:44 -07:00
Alexander Lobakin
d8b7fc51c8 [contrib][linux] Add contrib/linux-kernel/linux to .gitignore
The mentioned path is being created/used by the 'import' rule for
generating source files for Linux kernel.

Signed-off-by: Alexander Lobakin <alobakin@pm.me>
2021-09-23 16:39:39 +02:00
Alexander Lobakin
c45b27abe7 [contrib][linux] Add huf_decompress_amd64.o target to Makefile
Commit a5f2c4552803 ("Huffman ASM") added a new ASM source file,
but it wasn't added to the kernel Makefile despite that it received
support for Huffman ASM according to the internal definitions. This
leads to undefined references, as huf_decompress.o now calls those
ASM functions.
Add it to the list of sources when building inside the kernel tree.
Kbuild can handle .S files just fine, so none additional rules
needed.

Fixes: a5f2c4552803 ("Huffman ASM")
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
2021-09-23 16:39:31 +02:00
Alexander Lobakin
e6d62bbfe1 [contrib][linux] Fix build with CONFIG_WERROR
Linux 5.15 introduces a new Kconfig option, CONFIG_WERROR, which
forces -Werror for the entire kernel.
Current in-kernel ZSTD implementation uses functions deprecated
in 1.5.0, and thus fails on -Wdeprecated-declarations.
Turn this particular error into warning to be able to build the
kernel with CONFIG_WERROR. I'm not disabling them completely to
make sure they'll be visible and [hopefully] fixed sooner or later.

Signed-off-by: Alexander Lobakin <alobakin@pm.me>
2021-09-23 16:39:10 +02:00
sen
c7afbec4c1
Merge pull request #2780 from senhuang42/blocksplit_stack_reduce
Reduce stack usage of block splitter
2021-09-23 09:48:04 -04:00
senhuang42
1d8143c84f Move block splitter from stack to CCtx 2021-09-23 00:02:31 -04:00
Nick Terrell
e3feec74d3
Merge pull request #2795 from terrelln/linux-fixes
[contrib][linux] Reduce stack usage by 80 bytes
2021-09-22 19:56:53 -07:00
Nick Terrell
162491f601 [contrib][linux] Reduce stack usage by 80 bytes
Instead of calling `ZSTD_compress_advanced()` and
`ZSTD_initCStream_advanced()`, which each take a `ZSTD_parameters` by
value, use the new advanced API.

Stack usage went from 2024 -> 1944.
2021-09-22 18:18:47 -07:00
Nick Terrell
b25d57461e
Merge pull request #2794 from terrelln/spdx-license
[contrib][linux] Fix up SPDX license identifiers
2021-09-22 17:33:31 -07:00
Yann Collet
441eb2654b
Merge pull request #2793 from facebook/rdoc
Minor documentation update
2021-09-22 17:07:49 -07:00
Nick Terrell
4d347a902b [contrib][linux] Fix up SPDX license identifiers
Correctly identify that we are GPL v2+ or BSD 3 clause, as pointed out
in issue #2663.
2021-09-22 15:09:42 -07:00
Yann Collet
8150891939 regenerated zstdless.1 2021-09-22 14:48:51 -07:00
Yann Collet
3addf2f277 updated zstdgrep man page 2021-09-22 14:30:59 -07:00
Yann Collet
70b36c2308 update zstdgrep doc
to mention ripgrep alternative which transparently supports zstd-compressed files.
2021-09-22 14:30:06 -07:00
Yann Collet
999f8778af updated man pages
using ronn-ng
2021-09-22 14:18:24 -07:00
Yann Collet
2832bbbbdc emphasize usage of -r in documentation
notably as a way to overcome shell expansion limitations,
notably in a scenario of dictionary training.
2021-09-22 14:00:20 -07:00
sen
044c8b4722
Merge pull request #2779 from senhuang42/fse_fix
Fix NCountWriteBound
2021-09-22 13:51:21 -04:00
sen
1e99d36361
Merge pull request #2788 from senhuang42/param_switch
Use new paramSwitch enum for row matchfinder and block splitter
2021-09-22 13:27:55 -04:00
Nick Terrell
5208e14ae0
Merge pull request #2791 from terrelln/huf-asm-fixes
[huf] Fix compilation when DYNAMIC_BMI2=0 && BMI2 is supported
2021-09-22 09:59:02 -07:00
senhuang42
99b5e7b8c2 Add test case for FSE over-write 2021-09-22 12:03:46 -04:00
Nick Terrell
9450876a9d [huf] Fix compilation when DYNAMIC_BMI2=0 && BMI2 is supported
* Fix compilation issues pointed out in PR #2790.
* Add test cases to GitHub actions that test all combinations of
  `DYNAMIC_BMI2` BMI2 support.
2021-09-21 16:49:13 -07:00
senhuang42
06f42c3bfd Use new paramSwitch enum for LDM 2021-09-21 14:22:09 -04:00
senhuang42
b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
Nick Terrell
8385355fac
Merge pull request #2722 from terrelln/huf-asm
Assembly implementation of 4X1 & 4X2 Huffman
2021-09-20 17:32:30 -07:00
Nick Terrell
a5f2c45528 Huffman ASM 2021-09-20 14:46:43 -07:00
Nick Terrell
51b123d5f7
Merge pull request #2784 from terrelln/huf-asm-fuzzer
[fuzzer] Add Huffman decompression fuzzer
2021-09-20 11:57:27 -07:00
Yann Collet
847dcb65bf
Merge pull request #2787 from animalize/msvc2019
add msvc2019 to build.generic.cmd
2021-09-20 11:14:51 -07:00
Ma Lin
4eef208889 add msvc2019 to build.generic.cmd 2021-09-19 10:28:16 +08:00
Nick Terrell
d7542aacd9 [fuzzer] Add huf_decompress fuzzer
Add a fuzzer for Huffman decompression. Fix several bugs in Huffman
decompression, mostly related to `op == NULL` and pointer underflow.
2021-09-17 15:00:49 -07:00
Nick Terrell
6392c382bf
Merge pull request #2783 from terrelln/huf-asm-makefiles
[build] Add support for ASM files in Make + CMake
2021-09-17 14:56:47 -07:00
Nick Terrell
8bf699aa59 [build] Add support for ASM files in Make + CMake
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
  Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
  same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
  `make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.

The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.

The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
2021-09-17 14:13:53 -07:00
sen
9d2a45a705
Merge pull request #2778 from senhuang42/opt_inlining_revert
Revert opt outlining change
2021-09-15 14:22:10 -04:00
Sen Huang
a7aa2c5df6 Fix NCountWriteBound 2021-09-15 09:51:42 -07:00
Sen Huang
bd84e4a9d3 Revert opt outlining change 2021-09-15 09:08:41 -07:00
Nick Terrell
2fabd370bb
Merge pull request #2777 from terrelln/oss-fuzz-fix
[rsyncable] Fix test failures
2021-09-14 13:20:22 -07:00
Nick Terrell
9d9e2ed00b [rsyncable] Fix test failures
Test failures showed up on the daily cron job. They didn't show up
in CI because the condition is somewhat rare, and didn't trigger
during the CI tests.

This PR fixes up the logic in `findSynchronizationPoint()` to correctly
handle the edge case. It also un-comments an assert that helps catch the
issue, and verify that rsyncable mode is calculating the correct hash.

After the fix, the test that failed passes:

```
./zstreamtest --newapi -t1 --no-big-tests -s9680
```
2021-09-14 12:28:53 -07:00
Yann Collet
2e6f5bc0d8
Merge pull request #2771 from facebook/opt_investigation
Improve optimal parser performance on small data
2021-09-14 10:36:34 -07:00
Nick Terrell
d22bbed5db
Merge pull request #2776 from terrelln/oss-fuzz-fix
[rsyncable] Ensure ZSTD_compressBound() is respected
2021-09-14 09:37:43 -07:00
Yann Collet
fd94b9d1c9 Merge branch 'dev' into opt_investigation 2021-09-14 01:15:51 -07:00
sen
29f595ef76
Merge pull request #2773 from senhuang42/huffman_bugix
Fix dictionary training huffman segfault and small speed improvement
2021-09-13 21:02:42 -04:00