facebook
/
zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,192 Commits
2 Branches
0 Tags
28 MiB
Commit Graph

8192 Commits (98787a301826b8ecb7f445d199accc56d0455dd0)

Author SHA1 Message Date
Nick Terrell 48ef15fb47 [minor improvement] Pass dictSize when selecting parameters 
When selecting parameters in streaming compression with a dictionary use
the dictionary size to select the parameters.
 2020-10-12 12:47:19 -07:00
Nick Terrell 012818df99 [refactor] Remove ZSTD_resetCStream_internal() 
This function is only called in one place. It isn't a logical separation
of duties, and it was only obsfucating the code now, so inline it.
 2020-10-12 12:46:10 -07:00
Nick Terrell 7083f79008 [bug] Fix dictContentType when reprocessing cdict 
Conditions to trigger:
* CDict is loaded as raw content.
* CDict starts with the zstd dictionary magic number.
* The CDict is reprocessed (not attached or copied).
* The new API is used (streaming or `ZSTD_compress2()`).

Bug: The dictionary is loaded as a zstd dictionary, not a raw content
dictionary, because the dict content type is set to `ZSTD_dct_auto`.

Fix: Pass in the dictionary content type from cdict creation to the call
to `ZSTD_compress_insertDictionary()`.

Test: Added a test case that exposes the bug, and fixed the raw
content tests to not modify the `dictBuffer`, which makes all future
tests with the `dictBuffer` raw content, which doesn't seem intentional.
 2020-10-12 12:46:10 -07:00
senhuang42 d6911b86be Require LDM matches to be strictly greater in length 2020-10-09 12:56:18 -04:00
Like Ma cc907770bd Fix building on AIX 5.1 2020-10-09 18:34:00 +08:00
Yann Collet b951ad20a2
Merge pull request #2329 from senhuang42/prevent_summary_updates_when_using_stdout 
Prevent summary updates when using stdout
 2020-10-09 01:01:36 -07:00
Yann Collet 12541931fa
Merge pull request #2328 from marxin/zstd-pool-api 
Allow external creation of POOLs that can be shared.
 2020-10-09 01:00:50 -07:00
Yann Collet 6fdb0cb8d9
Merge pull request #2303 from senhuang42/let_cdict_take_clevel_priority 
For ZSTD_compressStream2(), let cdict take compression level priority
 2020-10-09 00:48:30 -07:00
Yann Collet c3ee284ca2
Merge pull request #2319 from facebook/fullbench_stream2 
update fullbench for compressStream2()
 2020-10-09 00:40:59 -07:00
senhuang42 b9c8033cde Define kNullRawSeqStore for every file 2020-10-07 19:02:41 -04:00
senhuang42 a6165c1b28 Change matchState_t::ldmSeqStore to pointer 2020-10-07 14:13:57 -04:00
senhuang42 abce708a56 Move posInSequence correction to correct location 2020-10-07 13:56:25 -04:00
senhuang42 0c515590d8 Replace offCode of largest match if ldm's offCode is superior 2020-10-07 13:56:25 -04:00
senhuang42 0fac8e07e1 Refactor usage of ms->ldmSeqStore so that it is not modified during compressBlock(), and simplify skipRawSeqStoreBytes 2020-10-07 13:56:25 -04:00
senhuang42 a5500cf2af Refactor separate ldm variables all into one struct 2020-10-07 13:56:25 -04:00
senhuang42 0731b94e7c Use kNullRawSeqStore constant in zstdmt_compress.c 2020-10-07 13:56:25 -04:00
senhuang42 0325d878f2 Remove bubbling down matches with longer offCode and same matchLen 2020-10-07 13:56:25 -04:00
senhuang42 031b7ec15f Disable LDM minMatch adjustment when using opt parser 2020-10-07 13:56:25 -04:00
senhuang42 ddf8a3f1b9 Enable inclusion of mid-flight LDMs in opt parser 2020-10-07 13:56:25 -04:00
senhuang42 88f72ed942 Correct incorrect offcode calculation 2020-10-07 13:56:25 -04:00
senhuang42 e96ea5d147 Fix static analyze fuzzer.c error 2020-10-07 13:56:25 -04:00
senhuang42 d8b43a4202 Add explicit conversion of size_t to U32 2020-10-07 13:56:25 -04:00
senhuang42 b8bfc4e63d Add cSize regression test to fuzzer.c 2020-10-07 13:56:25 -04:00
senhuang42 c87d2e5866 Prefix new static ldm helpers with ZSTD_opt 2020-10-07 13:56:25 -04:00
senhuang42 429dec4f42 Add DEBUGLOG() calls in ldm helpers 2020-10-07 13:56:25 -04:00
senhuang42 10647924f1 Make function descriptions more accurate 2020-10-07 13:56:25 -04:00
senhuang42 1a687b3fcb Improve documentation of relevant structs 2020-10-07 13:56:25 -04:00
senhuang42 37617e23d7 Correct matchLength calculation and remove unnecessary functions 2020-10-07 13:56:25 -04:00
senhuang42 7dee62c287 Reset ldmSeqStore after initStats_ultra() pass for btultra2 2020-10-07 13:56:25 -04:00
senhuang42 0718aa70df Refactor existing functions to use posInSequence 2020-10-07 13:56:25 -04:00
senhuang42 7348b40a87 Adjustments to ldm_calculateMatchRange() to calculate bounds correctly 2020-10-07 13:56:25 -04:00
senhuang42 a1ef2db5b2 Add ldm_calculateMatchRange() function 2020-10-07 13:56:25 -04:00
senhuang42 ef823e0299 Remove rawSeqStore.base and add rawSeqStore.posInSequence 2020-10-07 13:56:25 -04:00
senhuang42 cfd2aec1b7 Add unit tests into playTests.sh 2020-10-07 13:56:25 -04:00
senhuang42 4793ae3b84 Prevent duplicate LDMs from being inserted 2020-10-07 13:56:25 -04:00
senhuang42 65f9cfeeec Add extra bounds check to prevent heap access after free ASAN error 2020-10-07 13:56:25 -04:00
senhuang42 bff5785fd5 Address mixed variables C90 warning 2020-10-07 13:56:25 -04:00
senhuang42 724b94ed18 ldm_getNextMatch fixed return values 2020-10-07 13:56:25 -04:00
senhuang42 ea92fb3a68 Cleanups, add comments and explanations 2020-10-07 13:56:25 -04:00
senhuang42 78da2e1808 Fixed sifting algorithm 2020-10-07 13:56:25 -04:00
senhuang42 6ccd97fc96 Fixed end of match boundary update issues 2020-10-07 13:56:25 -04:00
senhuang42 28394b64f2 Add proper bounds check on adding ldms 2020-10-07 13:56:25 -04:00
senhuang42 a2f2b58d04 Add a function ldm_voidSequences() 2020-10-07 13:56:25 -04:00
senhuang42 9c3c7cd20e Fix function argument to getNextMatch() 2020-10-07 13:56:25 -04:00
senhuang42 c8b8572b38 Adjustments to no longer segfault on nci 2020-10-07 13:56:25 -04:00
senhuang42 f57c7e6bbf Add base adjustment correction 2020-10-07 13:56:25 -04:00
senhuang42 5df9b5e05f Add initial getNextMatch() in opt parser 2020-10-07 13:56:25 -04:00
senhuang42 f8ce7cabc3 Added more debugging 2020-10-07 13:56:25 -04:00
senhuang42 84009a076a Add re-copying of ldmSeqStore after processing 2020-10-07 13:56:25 -04:00
senhuang42 42395a70c2 Add debug statements, flesh out functions 2020-10-07 13:56:25 -04:00