1432 Commits

Author SHA1 Message Date
senhuang42
44c54a3e31 Addressing comments: more comments, cleanup, remove extra function, checksum logic 2020-08-24 16:14:19 -04:00
Nick Terrell
8def0e5fd3 Fix up code after reading through 2020-08-24 12:24:45 -07:00
senhuang42
ffaa0df76d Document change in CLI for --no-check during decompression in --help menu 2020-08-24 09:49:12 -04:00
senhuang42
e3f5f9658a Added CLI tests for --no-check, fixed ignore checksum logic 2020-08-22 16:05:40 -04:00
senhuang42
20eb095882 Added unit test to fuzzer.c, changed definition param name 2020-08-22 13:26:33 -04:00
senhuang42
1b34b15e6b Adding CLI capability to invoke decompression with no checksum 2020-08-21 17:49:30 -04:00
senhuang42
6a8dbdcd1f Modify decompression loop to gnore checksums if flag is enabled 2020-08-21 16:46:46 -04:00
Nick Terrell
8f8bd2d1ac [regression] Update results.csv 2020-08-20 12:41:35 -07:00
Nick Terrell
575731b6db Use ncount=1 when < 4096 symbols 2020-08-18 16:47:53 -07:00
Nick Terrell
612e947c5e wire up bmi2 support 2020-08-17 16:35:28 -07:00
Nick Terrell
a8006264cf small blocks benchmark 2020-08-14 18:57:20 -07:00
Yann Collet
23941eec04 added tests for newly enabled syntax
for --patch-from origin
and --filelist list

Also : removed some constrained syntax tests,
as the new argument parsing syntax is more permissive.

For example :
    zstd file -of dest
used to be disallowed.

It's now allowed, and understood as:
    zstd file -o dest -f
2020-07-17 13:31:15 -07:00
Xin Xie
9a8ccd4ba3 Add output-dir-mirror option 2020-06-24 22:12:11 -07:00
Bimba Shrestha
de48f35306 adding --patch-from --stream-size test 2020-06-18 10:28:37 -07:00
Nick Terrell
08981d2638 [lib] Allow compression dictionaries with missing symbols
Allow compression to use dictionaries with missing symbols in their
entropy tables. We set the FSE repeat mode to check when there are
missing symbols, and set the FSE repeat mode to valid when all symbols
are present.

Note that when not all symbols are present, the heuristics which favor
dictionary tables for lower compression levels won't activate.

Tested by manually creating a dictionary with missing symbols of every
type, and validing that the compressor rejects it before this change,
and accepts it after this change. Also, I ran the `dictionary_loader`
fuzzer for >1 hour of CPU time without running into cases where
compression succeeds, but decompression fails.

Fixes #2174.
2020-06-12 17:57:19 -07:00
Bimba Shrestha
e2838d9eb9 Spelling mistakes 2020-06-05 05:11:26 -05:00
Shaojing Li
847349195f fix the if statements in posix sh env 2020-06-03 11:36:38 -07:00
Shaojing Li
3a3da1712b check env variables and add default values 2020-06-03 10:49:21 -07:00
Bimba Shrestha
b0f851675a [shellcheck] setting if unset 2020-06-02 09:12:50 -07:00
Bimba Shrestha
151deaf143 [shellcheck] adding quotes to expansion 2020-06-02 09:12:13 -07:00
Yann Collet
26b21e481f fix meson playTests.sh 2020-05-21 15:17:22 -07:00
Nick Terrell
651d3d73e0 [test] Update the ldm loadedDictEnd test to cover zstdmt 2020-05-19 16:14:14 -07:00
Nick Terrell
0dcd3eec43
Merge pull request #2152 from terrelln/simple-rt-bound
[fuzz] Expand the allowedExpansion
2020-05-19 12:56:11 -07:00
Nick Terrell
b82bf711fc [fuzz] Expand the allowedExpansion 2020-05-19 11:42:53 -07:00
Yann Collet
fdc56baa42
fix 22294 (#2151) 2020-05-18 21:05:10 -07:00
Nick Terrell
9778f46014
Merge pull request #2150 from terrelln/ldm-dict-reset
[ldm] Reset loadedDictEnd when the context is reset
2020-05-18 18:33:01 -07:00
Nick Terrell
7b317b4876 [test] Test that the ldm dictionary gets invalidated on reset 2020-05-18 16:00:28 -07:00
Nick Terrell
87dbd6d4bf [test] Improve LDM forceMaxWindow test 2020-05-18 15:11:18 -07:00
W. Felix Handte
d37fcf36eb Don't Use [[ in Shell Scripts 2020-05-18 15:06:56 -04:00
Bimba Shrestha
255e5e3f56
[fuzz] Adding dictionary_stream_round_trip fuzzer (#2140)
* Adding dictionary_stream_round_trip

* fixing memory leak
2020-05-15 13:33:31 -07:00
Nick Terrell
608075abb2 [test][regression] Update results.csv 2020-05-14 17:06:39 -07:00
Nick Terrell
bf0591e1e2 [test] Expose the LDM+MT+dict bug in a unit test 2020-05-14 12:06:55 -07:00
Bimba Shrestha
12071467d3 reverting docs and test 2020-05-13 15:22:07 -05:00
Nick Terrell
c3e921c639
Merge pull request #2131 from terrelln/raw-dict-fuzzer
Fix rare scenario with lazy parser, dictionary, and repcodes
2020-05-12 17:44:31 -07:00
Bimba Shrestha
0453cfa8f5 removing -f test (grep usage not supported on mac) 2020-05-12 15:18:43 -05:00
Nick Terrell
4b88bd3ee0 [lib][fuzz] Assert sequences are valid in round trip tests 2020-05-11 20:38:49 -07:00
Yann Collet
e001715b3d fixed asan test 2020-05-11 20:35:47 -07:00
Yann Collet
20bd246045 blindfix for VS macro redefinition 2020-05-11 19:29:36 -07:00
Nick Terrell
1185dfb8d1 [fuzz] Add raw dictionary content fuzzer 2020-05-11 19:03:33 -07:00
Nick Terrell
301a62fe08 [fuzz] Fix compress bound for dictionary_round_trip 2020-05-11 19:00:52 -07:00
Yann Collet
91ad01218e updated initStatic tests
differentiate small CCtx for small inputs
from full CCtx
from CStream contexts.

Ensure allocation & resize tests are more precise.
2020-05-11 18:50:10 -07:00
Yann Collet
608f1bfc4c fixed context downsize with initStatic
When context is created using initStatic,
no resize is possible.

fix : only bump oversizeDuration when !initStatic
2020-05-11 18:16:38 -07:00
Yann Collet
dd026ca505 re-inforced tests for initStaticCCtx
ensure that `estimateCCtxSize()` works as intended.
2020-05-09 11:30:45 -07:00
Nick Terrell
5717bd39ee [lib] Fix NULL pointer dereference
When the output buffer is `NULL` with size 0, but the frame content size
is non-zero, we will write to the NULL pointer because our bounds check
underflowed.

This was exposed by a recent PR that allowed an empty frame into the
single-pass shortcut in streaming mode.

* Fix the bug.
* Fix another NULL dereference in zstd-v1.
* Overflow checks in 32-bit mode.
* Add a dedicated test.
* Expose the bug in the dedicated simple_decompress fuzzer.
* Switch all mallocs in fuzzers to return NULL for size=0.
* Fix a new timeout in a fuzzer.

Neither clang nor gcc show a decompression speed regression on x86-64.
On x86-32 clang is slightly positive and gcc loses 2.5% of speed.

Credit to OSS-Fuzz.
2020-05-06 12:09:02 -07:00
Bimba Shrestha
e7df0d41bb
Merge pull request #2095 from bimbashrestha/grep
[bugs] zstdgrep/grep inconsistencies
2020-05-06 11:18:15 -05:00
Bimba Shrestha
250184adf6 adding tests back 2020-05-05 16:51:06 -07:00
Felix Handte
7e9aabd652
Merge pull request #2099 from felixhandte/compile-under-pedantic
Compile Under `-pedantic -Werror` and `-std=c90`
2020-05-04 10:07:13 -07:00
Felix Handte
816ed80774
Merge pull request #1984 from MeghnaM/1636-Reduce-stack-usage-of-HUF_sort
Reduce stack usage of HUF_sort()
2020-05-04 08:15:31 -07:00
W. Felix Handte
2cf72d56a6 Try to Fix MSVC Error
It's complaining about the `memcpy`s, saying:

"warning C4090: 'function': different 'const' qualifiers"

Let's try explicitly casting to the argument types...
2020-05-04 10:59:15 -04:00
W. Felix Handte
dacbcd2cc1 Fix Up Some Pointer Handling in Tests 2020-05-04 10:59:15 -04:00