4238 Commits

Author SHA1 Message Date
Sen Huang
3c595a4a79 Add literalCompressionMode to stable cParams 2021-04-26 09:55:06 -04:00
felixhandte
efa6dfa729 Apply DDS adjustments to avoid assert failures 2021-04-23 16:41:00 -04:00
senhuang42
3b98987496 Remove building of ZBUFF/deprecated folder by default 2021-04-19 17:12:00 -04:00
Sen Huang
c5869677d9 Moved ZSTD_defaultCLevel() into stable API 2021-04-16 10:15:40 -07:00
Sen Huang
9c1ca3c00b Moved ZSTD_getDictID_fromCDict() into stable API 2021-04-16 10:14:29 -07:00
sen
12c045f74d
Merge pull request #2574 from senhuang42/repcode_mismatch_detector_fix
Correct the block splitter mismatched repcodes detection.
2021-04-12 23:27:43 -04:00
Sen Huang
8844f93957 Adjust nb elements to prefetch in ZSTD_row_fillHashCache() 2021-04-12 14:24:58 -04:00
Sen Huang
550f76f131 Correct the detection of mismatched repcodes 2021-04-09 09:08:51 -07:00
Sen Huang
4d63d6e8aa Update results.csv, add Row hash to regression test 2021-04-07 10:31:41 -07:00
Nick Terrell
4694423c4f Add and integrate lazy row hash strategy 2021-04-07 09:53:34 -07:00
sen
f71aabb5b5
Move clevel override to after initLocalDict() (#2571) 2021-04-06 21:05:37 -04:00
sen
f1e8b565c2
Maintain two repcode histories for block splitting, replace invalid repcodes (#2569) 2021-04-06 17:25:55 -04:00
sen
e38124555e
Fix dictionary force reloading clevel selection (#2570)
* Move cdict clevel override to before localdict init

* Update results.csv after dict load changes
2021-04-06 15:35:09 -04:00
Nick Terrell
8383fc828d
Merge pull request #2541 from ihsinme/patch-1
simple fix for using bit operator.
2021-04-02 13:01:09 -07:00
sen
980f3bbf83
[cwksp] Align all allocated "tables" and "aligneds" to 64 bytes (#2546)
* Perform 64-byte alignment of wksp tables and aligneds internally

* Clean up cwskp_finalize() function to only do two allocs

* Refactor aligned/buffer reservation code, remove ASAN req for alignment reservations

* Change from allocating 128 bytes always to allocating only buffer space as needed for tables/aligned

* Back out aligned/table reservation order restriction

* Add stricter bounds for new/resized wksps, fix comment in zstd_cwksp.h
2021-04-01 20:07:19 -04:00
sen
255925c231
Fix repcode-related OSS-fuzz issues in block splitter (#2560)
* Do not emit last partitions of blocks as RLE/uncompressed

* Fix repcode updates within block splitter

* Add a entropytables confirm function, redo ZSTD_confirmRepcodesAndEntropyTables() for better function signature

* Add a repcode updater to block splitter, no longer need to force emit compressed blocks
2021-03-31 15:14:59 -04:00
Nick Terrell
a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
sen
84ccb81e7c
Merge pull request #2561 from senhuang42/longlength_enum
Add enum for representing long length ID
2021-03-26 15:55:12 -04:00
Sen Huang
b1a43455f8 Add enum for representing long length ID 2021-03-26 10:41:09 -07:00
sen
4fe2e7ae14
Merge pull request #2558 from senhuang42/msan_block_splitter_fix
Fix block splitter minor MSAN warning.
2021-03-25 13:51:43 -04:00
sen
b0407b9f0e
Merge pull request #2555 from senhuang42/default_clevel_func
Add ZSTD_defaultCLevel() function to public API
2021-03-25 13:07:28 -04:00
Sen Huang
2a907bf4aa Move lastCountSize into a returned struct, fix MSAN error 2021-03-25 09:11:15 -07:00
Sen Huang
e398744a35 Add ZSTD_defaultCLevel() function to public API 2021-03-25 08:04:00 -07:00
Nick Terrell
f8ac0ea7ef
Merge pull request #2539 from terrelln/linux-kernel-fixes
Fixes for the next linux kernel patch version
2021-03-24 10:34:29 -07:00
sen
bf542c8a8d
Merge pull request #2447 from senhuang42/block_splitter_v2
Recursive block splitting
2021-03-24 12:27:22 -04:00
Sen Huang
5b566ebe08 Rename *compressSequences*() functions for clarity 2021-03-24 08:21:29 -07:00
Sen Huang
0ef1f935b7 Add a fallback in case the total blocksize of split blocks exceeds raw block size 2021-03-24 08:21:29 -07:00
Sen Huang
c90e81a692 Enable block splitter by default when applicable 2021-03-24 08:21:29 -07:00
Sen Huang
e34332834a Clean up various functions, add debuglogging for estimate vs. actual sizes 2021-03-24 08:21:29 -07:00
Sen Huang
41c3eae6d9 Fix various fuzzer failures: repcode history, superblocks 2021-03-24 08:21:29 -07:00
senhuang42
0633bf17c3 Change 1.3.4 bugfix to be cross-compatible with superblocks and normal compression 2021-03-24 08:21:29 -07:00
senhuang42
eb1ee8686d Refactor buildSequencesStatistics() to avoid pointer increment for superblocks 2021-03-24 08:21:29 -07:00
senhuang42
e2bb215117 Add unit tests and fuzzer param 2021-03-24 08:21:09 -07:00
senhuang42
de52de1347 Add recursive block split algorithm 2021-03-24 08:21:09 -07:00
senhuang42
f06f6626ed Update function names for consistency 2021-03-24 08:20:54 -07:00
senhuang42
c56d6e49e8 Add block splitter to experimental params 2021-03-24 08:20:54 -07:00
senhuang42
2949a95224 Refactor block compression logic into single function 2021-03-24 08:20:54 -07:00
senhuang42
c05c090cc2 Centralize entropy statistics calculations to zstd_compress.c 2021-03-24 08:20:29 -07:00
sen
c48889f097
Merge pull request #2538 from senhuang42/monotonicity_test
Add memory monotonicity test over srcSize
2021-03-22 16:54:34 -04:00
Nick Terrell
ebc2dfa821
Merge pull request #2524 from terrelln/huf-stack-reduction
[huf] Reduce stack usage of HUF_readDTableX2 by ~972 bytes
2021-03-22 12:37:54 -07:00
Nick Terrell
634bfd339f [FSE] Clean up workspace using dynamically sized struct 2021-03-22 11:07:07 -07:00
Sen Huang
dff4a0e867 Make ZSTD_estimateCCtxSize_internal() loop through all srcSize parameter sets as well 2021-03-21 16:15:31 -07:00
Yann Collet
f5434663ea Refactor prefetching for the decoding loop
Following #2545,
I noticed that one field in `seq_t` is optional,
and only used in combination with prefetching.
(This may have contributed to static analyzer failure to detect correct initialization).

I then wondered if it would be possible to rewrite the code
so that this optional part is handled directly by the prefetching code
rather than delegated as an option into `ZSTD_decodeSequence()`.

This resulted into this refactoring exercise
where the prefetching responsibility is better isolated into its own function
and `ZSTD_decodeSequence()` is streamlined to contain strictly Sequence decoding operations.
Incidently, due to better code locality,
it reduces the need to send information around,
leading to simplified interface, and smaller state structures.
2021-03-19 15:48:17 -07:00
Nick Terrell
756bd59322 [huf][fse] Clean up workspaces
* Move `counting` to a struct in `FSE_decompress_wksp_body()`
* Fix error code in `FSE_decompress_wksp_body()`
* Rename a variable in `HUF_ReadDTableX2_Workspace`
2021-03-17 16:50:37 -07:00
ihsinme
a5bf09d764
simple fix for using bit operator.
good day.
It seems to me that the developer intended to use a logical operator.
so I suggest a simple fix.
2021-03-17 11:37:42 +03:00
Sen Huang
77ae664ba6 Fix ZSTD_dedicatedDictSearch_isSupported() requirements 2021-03-16 17:36:05 -07:00
senhuang42
386111adec Add a nbSeq argument to compressSequences()
Refactor ZSTD_compressBlock_internal() to do the block header write within and add nbSeq argument to compressSequences()
2021-03-16 14:04:22 -07:00
senhuang42
98764493cf Move block header write into compressBlock_internal() 2021-03-16 14:04:22 -07:00
Nick Terrell
ea288e0d8e [lib] Bump zstd version number 2021-03-16 11:47:27 -07:00
Nick Terrell
cd1551d261 [lib][tracing] Add ZSTD_NO_TRACE macro
When defined, it disables tracing, and avoids including the header.
2021-03-16 11:47:27 -07:00