d6dae2000b
Merge pull request #2365 from senhuang42/move_opt_parser_test_to_long_tests
...
Move ldm + opt parser no regression test to long tests
2020-10-20 11:34:36 -04:00
81a2c02d8f
Move ldm no regression test to fuzzer longtests
2020-10-19 15:28:46 -04:00
df470e176b
Add unit test for no cctx requested params change
2020-10-19 10:52:41 -04:00
42d037bdba
Add libregression build target, also fix make clean and .gitignore
2020-10-15 10:34:50 -04:00
f5d5cd3b40
Merge pull request #2341 from senhuang42/ldm_optimized_for_opt_parser
...
Integrate long distance matches into optimal parser
2020-10-13 13:09:07 -07:00
ede4f97153
[zstdmt] Fix bug where extra empty blocks are emitted
...
When zstdmt cannot get a buffer and `ZSTD_e_end` is passed an empty
compression job can be created. Additionally, `mtctx->frameEnded` can be
set to 1, which could potentially cause problems like unterminated blocks.
The fix is to adjust to `ZSTD_e_flush` even when we can't get a buffer.
2020-10-12 12:55:17 -07:00
9ab9229e11
[zstreamtest] Add compression determinism tests
...
* Run compression twice and check the compressed data is byte-identical.
The compression loop had to be rewritten to ensure deteriminism. It is
guaranteed by always making maximal forward progress.
* When nbWorkers > 0, change the number of workers 1/8 of the time.
* Run in single-pass mode 1/4 of the time.
I've run a few hundred thousand iterations of zstreamtest and have seen
no deteriminism issues so far. Before the zstdmt fix that skips the
single-pass shortcut non-determinism showed up in a few hundred
iterations.
2020-10-12 12:55:17 -07:00
c51a9e79b9
[zstdmt] Rip out the zstdmt API
...
This commit leaves only the functions used by zstd_compress.c. All other
functions have been removed from the API. The ZSTDMT unit tests in
fuzzer.c and zstreamtest.c have been rewritten to use the ZSTD API. And
the --mt zstreamtest tests have been ripped out.
2020-10-12 12:55:16 -07:00
d5c688e8ae
Fix ZSTD_adjustCParams_internal() to handle dictionary logic
...
Pass in the `ZSTD_cParamMode_e` to select how we define our cparams.
Based on the mode we either take the `dictSize` into account or we set
it to `0`. See the documentation for `ZSTD_cParamMode_e`.
Some of the modes currently share the same behavior. But they have
distinct modes because they are drastically different cases. E.g.
compression + reprocessing the dictionary and creating a cdict.
Additionally, when downsizing the hashLog and chainLog take the
(adjusted) dictionary size into account, since the size of the
dictionary gets added onto the window size.
Adds a simple test to ensure that we aren't downsizing too far.
2020-10-12 12:50:04 -07:00
7083f79008
[bug] Fix dictContentType when reprocessing cdict
...
Conditions to trigger:
* CDict is loaded as raw content.
* CDict starts with the zstd dictionary magic number.
* The CDict is reprocessed (not attached or copied).
* The new API is used (streaming or `ZSTD_compress2()`).
Bug: The dictionary is loaded as a zstd dictionary, not a raw content
dictionary, because the dict content type is set to `ZSTD_dct_auto`.
Fix: Pass in the dictionary content type from cdict creation to the call
to `ZSTD_compress_insertDictionary()`.
Test: Added a test case that exposes the bug, and fixed the raw
content tests to not modify the `dictBuffer`, which makes all future
tests with the `dictBuffer` raw content, which doesn't seem intentional.
2020-10-12 12:46:10 -07:00
b951ad20a2
Merge pull request #2329 from senhuang42/prevent_summary_updates_when_using_stdout
...
Prevent summary updates when using stdout
2020-10-09 01:01:36 -07:00
c3ee284ca2
Merge pull request #2319 from facebook/fullbench_stream2
...
update fullbench for compressStream2()
2020-10-09 00:40:59 -07:00
e96ea5d147
Fix static analyze fuzzer.c error
2020-10-07 13:56:25 -04:00
b8bfc4e63d
Add cSize regression test to fuzzer.c
2020-10-07 13:56:25 -04:00
429dec4f42
Add DEBUGLOG() calls in ldm helpers
2020-10-07 13:56:25 -04:00
cfd2aec1b7
Add unit tests into playTests.sh
2020-10-07 13:56:25 -04:00
7259b258d1
Add callsites to zstdcli.c and tests to playTests.sh
2020-10-07 13:47:38 -04:00
0057c4acf7
Merge pull request #2333 from terrelln/stable-dst
...
Reset all decompression parameters in ZSTD_DCtx_reset()
2020-10-01 18:56:11 -07:00
2e7d174130
Reset all decompression parameters in ZSTD_DCtx_reset()
...
* Reset all decompression parameters in `ZSTD_DCtx_reset()` when
resetting parameters.
* Add a test case.
2020-10-01 14:19:21 -07:00
83461ce963
Merge pull request #2322 from senhuang42/guard_against_stdin_for_warning_prompts
...
Don't let warning messages consume input from stdin
2020-09-30 08:26:50 -07:00
9f7212a48b
Update unit tests
2020-09-24 16:44:33 -04:00
c6c0a57c53
Merge pull request #2315 from senhuang42/allow_zstd_suffix
...
Support .zstd suffix only for decompression
2020-09-24 09:44:48 -07:00
21cd640b93
Add unit tests to guard against bad stdin
2020-09-22 14:55:41 -04:00
7aa3da1cd7
Use IS_CONSOLE macro to detect that we're indeed using a console
2020-09-22 14:15:52 -04:00
973f2adeec
[tests] Don't write to stdout
2020-09-22 00:40:27 -07:00
5618e000bd
update fullbench for compressStream2()
...
makes it possible to measure scenarios such as #2314
2020-09-21 07:19:20 -07:00
200c960f1d
Merge pull request #2311 from felixhandte/ddss-fix-cparam-derivation
...
Fix Compression Parameter Derivation Bugs Introduced by DDSS Changes
2020-09-18 14:02:14 -04:00
07034952df
Add -f to .zstd decompression CLI test
2020-09-18 13:01:45 -04:00
6b6cc80196
Support .zstd suffix only for decompression
2020-09-18 12:49:51 -04:00
9398acb245
Move Last Two Long Tests in fuzzer.c into Separate --long-tests Section
2020-09-17 13:31:10 -04:00
f23a321781
Update Regression Test Results
2020-09-17 12:23:05 -04:00
e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads
...
Allow environment variable to specify number of threads for compression
2020-09-14 14:04:19 -07:00
dec1a78d3e
minor fix casting for Visual
2020-09-14 11:46:23 -07:00
c91a0855f8
check endDirective in ZSTD_compressStream2()
...
fix #2297
also :
- `assert()` `endDirective` in `ZSTD_compressStream_internal()`, for debug mode
- add relevant tests
2020-09-14 10:56:08 -07:00
d6246d4a0f
Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout
...
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
2020-09-10 23:35:42 -04:00
6d3f816b3e
Test Fewer Dictionary Sizes
2020-09-10 22:30:52 -04:00
b6df3fd438
Fix Debug Logging in 32-bit Build
2020-09-10 22:10:02 -04:00
2cc2b40a1b
Test DDSS A Little More Thoroughly
2020-09-10 22:10:02 -04:00
b81f3a37f9
Easy: Fix Test
2020-09-10 18:51:52 -04:00
2cf6cfc55f
Add Fuzzer Test for the Various Dict Attachment Strategies
2020-09-10 18:51:52 -04:00
a90779397a
[lib] Reduce zstd stack usage by 1KB
2020-09-09 14:35:39 -07:00
a71963c7b8
nbThreads instead of numThreads
2020-09-09 12:40:00 -04:00
0a170b20a8
Add ZSTD_NUMTHREADS tests to playTests.sh
2020-09-08 10:34:50 -04:00
3aec385a10
Fix merge conflicts
2020-08-26 15:43:38 -04:00
a8c66881e5
Merge pull request #2283 from senhuang42/progress_bars_for_multiple_files
...
Refreshing progress bar for processing multiple files
2020-08-26 11:54:50 -07:00
cf83aceaf3
Merge pull request #2282 from terrelln/ncount-fix
...
[bug] Fix FSE_readNCount()
2020-08-26 10:31:07 -07:00
a73e131f10
Adjust playTests.sh refuse overwrite test to include -q
2020-08-26 11:40:05 -04:00
ae163015b1
[fuzz] Fix stream_decompress timeouts
2020-08-25 17:13:09 -07:00
49eeb2d1fc
[fuzz] Disable superblock expansion test
2020-08-25 17:13:06 -07:00
4193638996
[bug] Fix FSE_readNCount()
...
* Fix bug introduced in PR #2271
* Fix long-standing bug that is impossible to trigger inside of zstd
* Add a fuzzer that makes sure the normalized count always round trips
correctly
2020-08-25 15:42:41 -07:00
f82d9865b9
Merge pull request #2278 from senhuang42/ignore_checksum_advanced_param
...
New advanced decompression param to ignore checksums
2020-08-25 12:08:53 -07:00
614e446000
Merge pull request #2271 from terrelln/small-blocks
...
Small block optimizations
2020-08-24 18:54:33 -07:00
dde97de6c4
Only ask to proceed if using --rm, otherwise just display warning. -f bypasses it all. More robust tests
2020-08-24 20:20:39 -04:00
1acf243540
Add a warning whenever (de)compressing multiple files into one source, or into stdout
2020-08-24 19:10:03 -04:00
52f33a1da5
Fix compiler warnings
2020-08-24 16:09:45 -07:00
a030560d62
Add new DCtx param: validateChecksum and update unit tests
2020-08-24 17:28:00 -04:00
1302f8d676
[fix] Always return dstSize_tooSmall when it is the case
2020-08-24 13:38:13 -07:00
44c54a3e31
Addressing comments: more comments, cleanup, remove extra function, checksum logic
2020-08-24 16:14:19 -04:00
8def0e5fd3
Fix up code after reading through
2020-08-24 12:24:45 -07:00
ffaa0df76d
Document change in CLI for --no-check during decompression in --help menu
2020-08-24 09:49:12 -04:00
e3f5f9658a
Added CLI tests for --no-check, fixed ignore checksum logic
2020-08-22 16:05:40 -04:00
20eb095882
Added unit test to fuzzer.c, changed definition param name
2020-08-22 13:26:33 -04:00
1b34b15e6b
Adding CLI capability to invoke decompression with no checksum
2020-08-21 17:49:30 -04:00
6a8dbdcd1f
Modify decompression loop to gnore checksums if flag is enabled
2020-08-21 16:46:46 -04:00
8f8bd2d1ac
[regression] Update results.csv
2020-08-20 12:41:35 -07:00
575731b6db
Use ncount=1 when < 4096 symbols
2020-08-18 16:47:53 -07:00
612e947c5e
wire up bmi2 support
2020-08-17 16:35:28 -07:00
a8006264cf
small blocks benchmark
2020-08-14 18:57:20 -07:00
23941eec04
added tests for newly enabled syntax
...
for --patch-from origin
and --filelist list
Also : removed some constrained syntax tests,
as the new argument parsing syntax is more permissive.
For example :
zstd file -of dest
used to be disallowed.
It's now allowed, and understood as:
zstd file -o dest -f
2020-07-17 13:31:15 -07:00
9a8ccd4ba3
Add output-dir-mirror option
2020-06-24 22:12:11 -07:00
de48f35306
adding --patch-from --stream-size test
2020-06-18 10:28:37 -07:00
08981d2638
[lib] Allow compression dictionaries with missing symbols
...
Allow compression to use dictionaries with missing symbols in their
entropy tables. We set the FSE repeat mode to check when there are
missing symbols, and set the FSE repeat mode to valid when all symbols
are present.
Note that when not all symbols are present, the heuristics which favor
dictionary tables for lower compression levels won't activate.
Tested by manually creating a dictionary with missing symbols of every
type, and validing that the compressor rejects it before this change,
and accepts it after this change. Also, I ran the `dictionary_loader`
fuzzer for >1 hour of CPU time without running into cases where
compression succeeds, but decompression fails.
Fixes #2174 .
2020-06-12 17:57:19 -07:00
e2838d9eb9
Spelling mistakes
2020-06-05 05:11:26 -05:00
847349195f
fix the if statements in posix sh env
2020-06-03 11:36:38 -07:00
3a3da1712b
check env variables and add default values
2020-06-03 10:49:21 -07:00
b0f851675a
[shellcheck] setting if unset
2020-06-02 09:12:50 -07:00
151deaf143
[shellcheck] adding quotes to expansion
2020-06-02 09:12:13 -07:00
26b21e481f
fix meson playTests.sh
2020-05-21 15:17:22 -07:00
651d3d73e0
[test] Update the ldm loadedDictEnd test to cover zstdmt
2020-05-19 16:14:14 -07:00
0dcd3eec43
Merge pull request #2152 from terrelln/simple-rt-bound
...
[fuzz] Expand the allowedExpansion
2020-05-19 12:56:11 -07:00
b82bf711fc
[fuzz] Expand the allowedExpansion
2020-05-19 11:42:53 -07:00
fdc56baa42
fix 22294 ( #2151 )
2020-05-18 21:05:10 -07:00
9778f46014
Merge pull request #2150 from terrelln/ldm-dict-reset
...
[ldm] Reset loadedDictEnd when the context is reset
2020-05-18 18:33:01 -07:00
7b317b4876
[test] Test that the ldm dictionary gets invalidated on reset
2020-05-18 16:00:28 -07:00
87dbd6d4bf
[test] Improve LDM forceMaxWindow test
2020-05-18 15:11:18 -07:00
d37fcf36eb
Don't Use `[[` in Shell Scripts
2020-05-18 15:06:56 -04:00
255e5e3f56
[fuzz] Adding dictionary_stream_round_trip fuzzer ( #2140 )
...
* Adding dictionary_stream_round_trip
* fixing memory leak
2020-05-15 13:33:31 -07:00
608075abb2
[test][regression] Update results.csv
2020-05-14 17:06:39 -07:00
bf0591e1e2
[test] Expose the LDM+MT+dict bug in a unit test
2020-05-14 12:06:55 -07:00
12071467d3
reverting docs and test
2020-05-13 15:22:07 -05:00
c3e921c639
Merge pull request #2131 from terrelln/raw-dict-fuzzer
...
Fix rare scenario with lazy parser, dictionary, and repcodes
2020-05-12 17:44:31 -07:00
0453cfa8f5
removing -f test (grep usage not supported on mac)
2020-05-12 15:18:43 -05:00
4b88bd3ee0
[lib][fuzz] Assert sequences are valid in round trip tests
2020-05-11 20:38:49 -07:00
e001715b3d
fixed asan test
2020-05-11 20:35:47 -07:00
20bd246045
blindfix for VS macro redefinition
2020-05-11 19:29:36 -07:00
1185dfb8d1
[fuzz] Add raw dictionary content fuzzer
2020-05-11 19:03:33 -07:00
301a62fe08
[fuzz] Fix compress bound for dictionary_round_trip
2020-05-11 19:00:52 -07:00
91ad01218e
updated initStatic tests
...
differentiate small CCtx for small inputs
from full CCtx
from CStream contexts.
Ensure allocation & resize tests are more precise.
2020-05-11 18:50:10 -07:00
608f1bfc4c
fixed context downsize with initStatic
...
When context is created using initStatic,
no resize is possible.
fix : only bump oversizeDuration when !initStatic
2020-05-11 18:16:38 -07:00
dd026ca505
re-inforced tests for initStaticCCtx
...
ensure that `estimateCCtxSize()` works as intended.
2020-05-09 11:30:45 -07:00
Nick Terrell
senhuang42
Yann Collet
Yann Collet
Felix Handte
Yann Collet
Xin Xie
Bimba Shrestha
Shaojing Li