200c960f1d
Merge pull request #2311 from felixhandte/ddss-fix-cparam-derivation
...
Fix Compression Parameter Derivation Bugs Introduced by DDSS Changes
2020-09-18 14:02:14 -04:00
5870c47bc0
Merge pull request #2313 from felixhandte/fuzzer-separate-long-tests
...
Separate Long Tests in Fuzzer
2020-09-18 12:03:00 -04:00
e991f9994d
Run Long Tests in GitHub CI
2020-09-17 13:31:10 -04:00
9398acb245
Move Last Two Long Tests in fuzzer.c into Separate --long-tests Section
2020-09-17 13:31:10 -04:00
f23a321781
Update Regression Test Results
2020-09-17 12:23:05 -04:00
8930c6e551
Use ZSTD_CCtxParams_init() to Init CCtxParams, not memset()
...
Even if the discrepancies are at the moment benign, it's probably better to
standardize on using the one true initializer, rather than trying (and failing)
to correctly duplicate its behavior.
2020-09-17 12:15:33 -04:00
e8a44326fa
Avoid Redundancy in ZSTD_initCDict_internal() Args; Don't Take CParams + CCtxParams
2020-09-17 12:08:36 -04:00
eee51a664a
Fall Back if Derived CParams are Incompatible with DDSS; Refactor CDict Creation
...
Rewrite ZSTD_createCDict_advanced() as a wrapper around
ZSTD_createCDict_advanced2(). Evaluate whether to use DDSS mode *after* fully
resolving cparams. If not, fall back.
2020-09-15 18:01:08 -04:00
bc6521a6f6
Make ZSTD_createCDict_advanced2() cctxParams Arg Const
2020-09-15 14:06:10 -04:00
26a96a5b35
Do More Complete CParams Deduction in Non-DDSS Path of ZSTD_createCDict_advanced2
...
Call ZSTD_getCParamsFromCCtxParams() instead of ZSTD_getCParams_internal().
2020-09-15 13:57:43 -04:00
a2af804129
Pull CParam Override Logic into Helper
2020-09-15 13:38:05 -04:00
d96e98cfde
Merge pull request #2308 from terrelln/zstd-kernel
...
[contrib][linux-kernel] Add decompress_sources.h
2020-09-14 14:56:30 -07:00
05622992d5
Merge pull request #2292 from senhuang42/multifile_status_update_fio_refactor
...
Make multifile (de)compression print out a summary
2020-09-14 14:06:26 -07:00
e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads
...
Allow environment variable to specify number of threads for compression
2020-09-14 14:04:19 -07:00
517956e67d
Merge pull request #2306 from facebook/check_endDirective
...
check endDirective in ZSTD_compressStream2()
2020-09-14 13:57:55 -07:00
5651fe9b49
[contrib][linux-kernel] Add decompress_sources.h
...
Add decompress_sources.h, which includes all the decompression .c files.
This is used for kernel decompression.
Also, add a test which checks that including decompress_sources.h works.
2020-09-14 12:49:23 -07:00
dec1a78d3e
minor fix casting for Visual
2020-09-14 11:46:23 -07:00
c91a0855f8
check endDirective in ZSTD_compressStream2()
...
fix #2297
also :
- `assert()` `endDirective` in `ZSTD_compressStream_internal()`, for debug mode
- add relevant tests
2020-09-14 10:56:08 -07:00
d903b552c8
Merge pull request #2295 from felixhandte/dedicated-dict-search-structure-chain
...
DDSS for Lazy: Implement a Dedicated Dictionary Chain Table
2020-09-11 01:39:23 -04:00
d6246d4a0f
Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout
...
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
2020-09-10 23:35:42 -04:00
6d3f816b3e
Test Fewer Dictionary Sizes
2020-09-10 22:30:52 -04:00
b6df3fd438
Fix Debug Logging in 32-bit Build
2020-09-10 22:10:02 -04:00
c5fab8848a
Document searchFuncs Table
2020-09-10 22:10:02 -04:00
85a95840e4
Further Consolidate Dict Mode Checks
2020-09-10 22:10:02 -04:00
032010fcc1
Improve Documentation Slightly
2020-09-10 22:10:02 -04:00
0faefbf1b3
Make DDSS Selection Override ForceCopy Directive
2020-09-10 22:10:02 -04:00
2cc2b40a1b
Test DDSS A Little More Thoroughly
2020-09-10 22:10:02 -04:00
efa33861f2
Attempt to Fix MSVC Warnings
2020-09-10 22:10:02 -04:00
ed43832770
Simplify Match Limit Checks
...
Seems like a ~1.25% speedup.
2020-09-10 22:10:02 -04:00
06d240b8a7
Use All Available Space in the Hash Table to Extent Chain Table Reach
...
Rather than restrict our temp chain table to 2 ** chainLog entries, this
commit uses all available space to reach further back to gather longer
chains to pack into the DDSS chain table.
2020-09-10 22:10:02 -04:00
b2b0641ea0
Rewrite Table Fill to Retain Cache Entries Beyond Chain Window
2020-09-10 22:10:02 -04:00
916238d9dc
Avoid Malloc in Table Fill; Pack Tmp Structure into Hash Table
2020-09-10 22:10:02 -04:00
f42c5bddd9
Truncate Chain at Last Possible Attempt
...
Make the chain table denser?
2020-09-10 22:10:02 -04:00
20a020edbc
Prefetch Chain Table Matches
2020-09-10 22:10:02 -04:00
9b9feb84f2
Lay Out Chain Table Chains Contiguously
...
Rather than interleave all of the chain table entries, tying each entry's
position to the corresponding position in the input, this commit changes the
layout so that all the entries in a single chain are laid out next to each
other. The last entry in the hash table's bucket for this hash is now a packed
pointer of position + length of this chain.
This cannot be merged as written, since it allocates temporary memory inside
ZSTD_dedicatedDictSearch_lazy_loadDictionary().
2020-09-10 22:10:02 -04:00
66509c7bf4
Only Insert Positions Inside the Chain Window
2020-09-10 22:10:02 -04:00
13c5ec3e41
Only Allow Dedicated Dict Search for Dicts Loaded in 1 Chunk
...
The load algorithm requires we do it all in one go.
2020-09-10 22:10:02 -04:00
005ceaa052
Merge pull request #2276 from felixhandte/dedicated-dict-search-structure
...
DDSS For Lazy: Implement a Dedicated Dictionary Hash Table
2020-09-10 22:09:25 -04:00
07793547e6
Fix Bug: Only Use DDSS Insertion on CDict MatchStates
...
Previously, if DDSS was enabled on a CCtx and a dictionary was inserted into
the CCtx, the CCtx MatchState would be filled as a DDSS struct, causing
segfaults etc. This changes the check to use whether the MatchState is marked
as using the DDSS (which is only ever set for CDict MatchStates), rather than
looking at the CCtxParams.
2020-09-10 18:51:52 -04:00
d214d8c859
Shorten Dict Mode Conditionals in Order to Improve Readability
2020-09-10 18:51:52 -04:00
f49c1563ff
Force-Inline ZSTD_insertAndFindFirstIndex_internal()
...
Without this, gcc was declining to inline the function in `ZSTD_noDict` mode,
resulting in a ~10% slowdown.
2020-09-10 18:51:52 -04:00
cab86b074f
Clean Up Search Function Selection
2020-09-10 18:51:52 -04:00
2ffbde0d95
Fix `-Wshorten-64-to-32` Error
2020-09-10 18:51:52 -04:00
7b5d2f72ea
Adjust Working Context Table Sizes Back Down
2020-09-10 18:51:52 -04:00
c09454e28f
Add Warning Comment to ZSTD_createCDict_advanced2() Declaration
2020-09-10 18:51:52 -04:00
d332f57897
Permit Matching Against Lowest Valid Position
...
This comparison was previously faulty: the lowest valid position is itself
valid, and we should therefore be allowed to match against it.
2020-09-10 18:51:52 -04:00
a3659fe1ef
Make ZSTD_dedicatedDictSearch_getCParams Wrap ZSTD_getCParams
...
Fixes up bounds-checking, and lets us clean up what is at the moment an
unnecessary duplication of the default cparams tables.
2020-09-10 18:51:52 -04:00
b81f3a37f9
Easy: Fix Test
2020-09-10 18:51:52 -04:00
7b9a755ac9
Remove Chain Limit on Hash Cache Entries; Slightly Improve Compression
...
Entries in the hashTable chain cache aren't subject to the same aliasing that
the circular chain table is subject to. As such, we don't need to stop when we
cross the chain limit. We can delve deeper. :)
2020-09-10 18:51:52 -04:00
e8b4011b52
Split Lookups in Hash Cache and Chain Table into Two Loops
...
Sliiiight speedup.
2020-09-10 18:51:52 -04:00
Felix Handte
Nick Terrell
Yann Collet
Yann Collet