05622992d5
Merge pull request #2292 from senhuang42/multifile_status_update_fio_refactor
...
Make multifile (de)compression print out a summary
2020-09-14 14:06:26 -07:00
e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads
...
Allow environment variable to specify number of threads for compression
2020-09-14 14:04:19 -07:00
517956e67d
Merge pull request #2306 from facebook/check_endDirective
...
check endDirective in ZSTD_compressStream2()
2020-09-14 13:57:55 -07:00
5651fe9b49
[contrib][linux-kernel] Add decompress_sources.h
...
Add decompress_sources.h, which includes all the decompression .c files.
This is used for kernel decompression.
Also, add a test which checks that including decompress_sources.h works.
2020-09-14 12:49:23 -07:00
dec1a78d3e
minor fix casting for Visual
2020-09-14 11:46:23 -07:00
c91a0855f8
check endDirective in ZSTD_compressStream2()
...
fix #2297
also :
- `assert()` `endDirective` in `ZSTD_compressStream_internal()`, for debug mode
- add relevant tests
2020-09-14 10:56:08 -07:00
17b56f934e
Coding style cleanup
2020-09-11 11:42:12 -04:00
801513b5e7
Modify params rather than cctx->requestedParams
2020-09-11 11:41:10 -04:00
d903b552c8
Merge pull request #2295 from felixhandte/dedicated-dict-search-structure-chain
...
DDSS for Lazy: Implement a Dedicated Dictionary Chain Table
2020-09-11 01:39:23 -04:00
d6246d4a0f
Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout
...
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
2020-09-10 23:35:42 -04:00
6d3f816b3e
Test Fewer Dictionary Sizes
2020-09-10 22:30:52 -04:00
b6df3fd438
Fix Debug Logging in 32-bit Build
2020-09-10 22:10:02 -04:00
c5fab8848a
Document searchFuncs Table
2020-09-10 22:10:02 -04:00
85a95840e4
Further Consolidate Dict Mode Checks
2020-09-10 22:10:02 -04:00
032010fcc1
Improve Documentation Slightly
2020-09-10 22:10:02 -04:00
0faefbf1b3
Make DDSS Selection Override ForceCopy Directive
2020-09-10 22:10:02 -04:00
2cc2b40a1b
Test DDSS A Little More Thoroughly
2020-09-10 22:10:02 -04:00
efa33861f2
Attempt to Fix MSVC Warnings
2020-09-10 22:10:02 -04:00
ed43832770
Simplify Match Limit Checks
...
Seems like a ~1.25% speedup.
2020-09-10 22:10:02 -04:00
06d240b8a7
Use All Available Space in the Hash Table to Extent Chain Table Reach
...
Rather than restrict our temp chain table to 2 ** chainLog entries, this
commit uses all available space to reach further back to gather longer
chains to pack into the DDSS chain table.
2020-09-10 22:10:02 -04:00
b2b0641ea0
Rewrite Table Fill to Retain Cache Entries Beyond Chain Window
2020-09-10 22:10:02 -04:00
916238d9dc
Avoid Malloc in Table Fill; Pack Tmp Structure into Hash Table
2020-09-10 22:10:02 -04:00
f42c5bddd9
Truncate Chain at Last Possible Attempt
...
Make the chain table denser?
2020-09-10 22:10:02 -04:00
20a020edbc
Prefetch Chain Table Matches
2020-09-10 22:10:02 -04:00
9b9feb84f2
Lay Out Chain Table Chains Contiguously
...
Rather than interleave all of the chain table entries, tying each entry's
position to the corresponding position in the input, this commit changes the
layout so that all the entries in a single chain are laid out next to each
other. The last entry in the hash table's bucket for this hash is now a packed
pointer of position + length of this chain.
This cannot be merged as written, since it allocates temporary memory inside
ZSTD_dedicatedDictSearch_lazy_loadDictionary().
2020-09-10 22:10:02 -04:00
66509c7bf4
Only Insert Positions Inside the Chain Window
2020-09-10 22:10:02 -04:00
13c5ec3e41
Only Allow Dedicated Dict Search for Dicts Loaded in 1 Chunk
...
The load algorithm requires we do it all in one go.
2020-09-10 22:10:02 -04:00
005ceaa052
Merge pull request #2276 from felixhandte/dedicated-dict-search-structure
...
DDSS For Lazy: Implement a Dedicated Dictionary Hash Table
2020-09-10 22:09:25 -04:00
07793547e6
Fix Bug: Only Use DDSS Insertion on CDict MatchStates
...
Previously, if DDSS was enabled on a CCtx and a dictionary was inserted into
the CCtx, the CCtx MatchState would be filled as a DDSS struct, causing
segfaults etc. This changes the check to use whether the MatchState is marked
as using the DDSS (which is only ever set for CDict MatchStates), rather than
looking at the CCtxParams.
2020-09-10 18:51:52 -04:00
d214d8c859
Shorten Dict Mode Conditionals in Order to Improve Readability
2020-09-10 18:51:52 -04:00
f49c1563ff
Force-Inline ZSTD_insertAndFindFirstIndex_internal()
...
Without this, gcc was declining to inline the function in `ZSTD_noDict` mode,
resulting in a ~10% slowdown.
2020-09-10 18:51:52 -04:00
cab86b074f
Clean Up Search Function Selection
2020-09-10 18:51:52 -04:00
2ffbde0d95
Fix `-Wshorten-64-to-32` Error
2020-09-10 18:51:52 -04:00
7b5d2f72ea
Adjust Working Context Table Sizes Back Down
2020-09-10 18:51:52 -04:00
c09454e28f
Add Warning Comment to ZSTD_createCDict_advanced2() Declaration
2020-09-10 18:51:52 -04:00
d332f57897
Permit Matching Against Lowest Valid Position
...
This comparison was previously faulty: the lowest valid position is itself
valid, and we should therefore be allowed to match against it.
2020-09-10 18:51:52 -04:00
a3659fe1ef
Make ZSTD_dedicatedDictSearch_getCParams Wrap ZSTD_getCParams
...
Fixes up bounds-checking, and lets us clean up what is at the moment an
unnecessary duplication of the default cparams tables.
2020-09-10 18:51:52 -04:00
b81f3a37f9
Easy: Fix Test
2020-09-10 18:51:52 -04:00
7b9a755ac9
Remove Chain Limit on Hash Cache Entries; Slightly Improve Compression
...
Entries in the hashTable chain cache aren't subject to the same aliasing that
the circular chain table is subject to. As such, we don't need to stop when we
cross the chain limit. We can delve deeper. :)
2020-09-10 18:51:52 -04:00
e8b4011b52
Split Lookups in Hash Cache and Chain Table into Two Loops
...
Sliiiight speedup.
2020-09-10 18:51:52 -04:00
9e83c782f8
Simplify DDS Hash Table Construction
...
No need to walk the chainTable; we can just keep shifting the entries in the
hashTable.
2020-09-10 18:51:52 -04:00
ad9f98ac3f
Document the ZSTD_c_enableDedicatedDictSearch Parameter
2020-09-10 18:51:52 -04:00
5390fee4f7
Rename and Move DD_BLOG Constant to ZSTD_LAZY_DDSS_BUCKET_LOG
2020-09-10 18:51:52 -04:00
5e91ae27eb
Prefetch First Batch of Match Positions; +11% Speed in Level 5 w/ 1 Dict
2020-09-10 18:51:52 -04:00
df386b3d8d
Fix Off-By-One Error in Counting DDS Search Attempts
...
This caused us to double-search the first position and fail to search the
last position in the chain, slowing down search and making it less effective.
2020-09-10 18:51:52 -04:00
d46306087b
Enable Dedicated Dict Search in the CLI
2020-09-10 18:51:52 -04:00
914bfe7ee4
Init CCtx's Local Dict with CCtxParams
2020-09-10 18:51:52 -04:00
db2aa25252
Decision for Whether to Attach Should be Based on CDict Config, not CCtx
2020-09-10 18:51:52 -04:00
a494111385
Move Prefetch Before Insertion; Speed Up ~6%
2020-09-10 18:51:52 -04:00
eede46a47e
Misc Refactor of DDS Search Code
2020-09-10 18:51:52 -04:00
Yann Collet
Nick Terrell
Yann Collet
senhuang42
Felix Handte