facebook
/
zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,188 Commits
2 Branches
0 Tags
28 MiB
Commit Graph

8188 Commits (91a8cb9559342835aa7506d90354e03dfe91f4f0)

Author SHA1 Message Date
Yann Collet 5618e000bd update fullbench for compressStream2() 
makes it possible to measure scenarios such as #2314
 2020-09-21 07:19:20 -07:00
animalize 2e5d73dd72 Use `MEM_STATIC FORCE_INLINE_ATTR` instead of `FORCE_INLINE_TEMPLATE` 
It adds `__attribute__((unused))` for __GNUC__, to eliminate `-Werror=unused-function` error.
 2020-09-21 13:26:38 +08:00
animalize 0a69a6b1ca Let MSVC force inline ZSTD_hashPtr() function 
ZSTD_hashPtr() function was not expanded by MSVC, led to low performance compared to GCC.
 2020-09-21 10:38:55 +08:00
Felix Handte 200c960f1d
Merge pull request #2311 from felixhandte/ddss-fix-cparam-derivation 
Fix Compression Parameter Derivation Bugs Introduced by DDSS Changes
 2020-09-18 14:02:14 -04:00
senhuang42 07034952df Add -f to .zstd decompression CLI test 2020-09-18 13:01:45 -04:00
senhuang42 6b6cc80196 Support .zstd suffix only for decompression 2020-09-18 12:49:51 -04:00
Felix Handte 5870c47bc0
Merge pull request #2313 from felixhandte/fuzzer-separate-long-tests 
Separate Long Tests in Fuzzer
 2020-09-18 12:03:00 -04:00
W. Felix Handte e991f9994d Run Long Tests in GitHub CI 2020-09-17 13:31:10 -04:00
W. Felix Handte 9398acb245 Move Last Two Long Tests in fuzzer.c into Separate --long-tests Section 2020-09-17 13:31:10 -04:00
W. Felix Handte f23a321781 Update Regression Test Results 2020-09-17 12:23:05 -04:00
W. Felix Handte 8930c6e551 Use ZSTD_CCtxParams_init() to Init CCtxParams, not memset() 
Even if the discrepancies are at the moment benign, it's probably better to
standardize on using the one true initializer, rather than trying (and failing)
to correctly duplicate its behavior.
 2020-09-17 12:15:33 -04:00
W. Felix Handte e8a44326fa Avoid Redundancy in ZSTD_initCDict_internal() Args; Don't Take CParams + CCtxParams 2020-09-17 12:08:36 -04:00
senhuang42 29b8f2eb08 Fix multifile decompression summary 2020-09-16 10:38:13 -04:00
senhuang42 1d5c6fda9f Add filename truncations to decompression 2020-09-16 10:28:45 -04:00
W. Felix Handte eee51a664a Fall Back if Derived CParams are Incompatible with DDSS; Refactor CDict Creation 
Rewrite ZSTD_createCDict_advanced() as a wrapper around
ZSTD_createCDict_advanced2(). Evaluate whether to use DDSS mode *after* fully
resolving cparams. If not, fall back.
 2020-09-15 18:01:08 -04:00
senhuang42 ab0d332563 Add display clears 2020-09-15 15:53:32 -04:00
W. Felix Handte bc6521a6f6 Make ZSTD_createCDict_advanced2() cctxParams Arg Const 2020-09-15 14:06:10 -04:00
W. Felix Handte 26a96a5b35 Do More Complete CParams Deduction in Non-DDSS Path of ZSTD_createCDict_advanced2 
Call ZSTD_getCParamsFromCCtxParams() instead of ZSTD_getCParams_internal().
 2020-09-15 13:57:43 -04:00
W. Felix Handte a2af804129 Pull CParam Override Logic into Helper 2020-09-15 13:38:05 -04:00
senhuang42 cad6bf99af Add padding to remove previous line's leftovers, and keep the printed line to around the same size 2020-09-15 13:01:46 -04:00
Nick Terrell d96e98cfde
Merge pull request #2308 from terrelln/zstd-kernel 
[contrib][linux-kernel] Add decompress_sources.h
 2020-09-14 14:56:30 -07:00
Yann Collet 05622992d5
Merge pull request #2292 from senhuang42/multifile_status_update_fio_refactor 
Make multifile (de)compression print out a summary
 2020-09-14 14:06:26 -07:00
Yann Collet e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads 
Allow environment variable to specify number of threads for compression
 2020-09-14 14:04:19 -07:00
Yann Collet 517956e67d
Merge pull request #2306 from facebook/check_endDirective 
check endDirective in ZSTD_compressStream2()
 2020-09-14 13:57:55 -07:00
Nick Terrell 5651fe9b49 [contrib][linux-kernel] Add decompress_sources.h 
Add decompress_sources.h, which includes all the decompression .c files.
This is used for kernel decompression.

Also, add a test which checks that including decompress_sources.h works.
 2020-09-14 12:49:23 -07:00
Yann Collet dec1a78d3e minor fix casting for Visual 2020-09-14 11:46:23 -07:00
Yann Collet c91a0855f8 check endDirective in ZSTD_compressStream2() 
fix #2297
also :
- `assert()` `endDirective` in `ZSTD_compressStream_internal()`, for debug mode
- add relevant tests
 2020-09-14 10:56:08 -07:00
senhuang42 17b56f934e Coding style cleanup 2020-09-11 11:42:12 -04:00
senhuang42 801513b5e7 Modify params rather than cctx->requestedParams 2020-09-11 11:41:10 -04:00
Felix Handte d903b552c8
Merge pull request #2295 from felixhandte/dedicated-dict-search-structure-chain 
DDSS for Lazy: Implement a Dedicated Dictionary Chain Table
 2020-09-11 01:39:23 -04:00
W. Felix Handte d6246d4a0f Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout 
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
 2020-09-10 23:35:42 -04:00
W. Felix Handte 6d3f816b3e Test Fewer Dictionary Sizes 2020-09-10 22:30:52 -04:00
W. Felix Handte b6df3fd438 Fix Debug Logging in 32-bit Build 2020-09-10 22:10:02 -04:00
W. Felix Handte c5fab8848a Document searchFuncs Table 2020-09-10 22:10:02 -04:00
W. Felix Handte 85a95840e4 Further Consolidate Dict Mode Checks 2020-09-10 22:10:02 -04:00
W. Felix Handte 032010fcc1 Improve Documentation Slightly 2020-09-10 22:10:02 -04:00
W. Felix Handte 0faefbf1b3 Make DDSS Selection Override ForceCopy Directive 2020-09-10 22:10:02 -04:00
W. Felix Handte 2cc2b40a1b Test DDSS A Little More Thoroughly 2020-09-10 22:10:02 -04:00
W. Felix Handte efa33861f2 Attempt to Fix MSVC Warnings 2020-09-10 22:10:02 -04:00
W. Felix Handte ed43832770 Simplify Match Limit Checks 
Seems like a ~1.25% speedup.
 2020-09-10 22:10:02 -04:00
W. Felix Handte 06d240b8a7 Use All Available Space in the Hash Table to Extent Chain Table Reach 
Rather than restrict our temp chain table to 2 ** chainLog entries, this
commit uses all available space to reach further back to gather longer
chains to pack into the DDSS chain table.
 2020-09-10 22:10:02 -04:00
W. Felix Handte b2b0641ea0 Rewrite Table Fill to Retain Cache Entries Beyond Chain Window 2020-09-10 22:10:02 -04:00
W. Felix Handte 916238d9dc Avoid Malloc in Table Fill; Pack Tmp Structure into Hash Table 2020-09-10 22:10:02 -04:00
W. Felix Handte f42c5bddd9 Truncate Chain at Last Possible Attempt 
Make the chain table denser?
 2020-09-10 22:10:02 -04:00
W. Felix Handte 20a020edbc Prefetch Chain Table Matches 2020-09-10 22:10:02 -04:00
W. Felix Handte 9b9feb84f2 Lay Out Chain Table Chains Contiguously 
Rather than interleave all of the chain table entries, tying each entry's
position to the corresponding position in the input, this commit changes the
layout so that all the entries in a single chain are laid out next to each
other. The last entry in the hash table's bucket for this hash is now a packed
pointer of position + length of this chain.

This cannot be merged as written, since it allocates temporary memory inside
ZSTD_dedicatedDictSearch_lazy_loadDictionary().
 2020-09-10 22:10:02 -04:00
W. Felix Handte 66509c7bf4 Only Insert Positions Inside the Chain Window 2020-09-10 22:10:02 -04:00
W. Felix Handte 13c5ec3e41 Only Allow Dedicated Dict Search for Dicts Loaded in 1 Chunk 
The load algorithm requires we do it all in one go.
 2020-09-10 22:10:02 -04:00
Felix Handte 005ceaa052
Merge pull request #2276 from felixhandte/dedicated-dict-search-structure 
DDSS For Lazy: Implement a Dedicated Dictionary Hash Table
 2020-09-10 22:09:25 -04:00
W. Felix Handte 07793547e6 Fix Bug: Only Use DDSS Insertion on CDict MatchStates 
Previously, if DDSS was enabled on a CCtx and a dictionary was inserted into
the CCtx, the CCtx MatchState would be filled as a DDSS struct, causing
segfaults etc. This changes the check to use whether the MatchState is marked
as using the DDSS (which is only ever set for CDict MatchStates), rather than
looking at the CCtxParams.
 2020-09-10 18:51:52 -04:00