Bimba Shrestha
1f681d8592
Merge branch 'oss' of https://github.com/bimbashrestha/zstd into oss
2019-11-27 10:56:54 -08:00
Bimba Shrestha
a3a3c62b81
[fuzz] Only set HUF_repeat_valid if loaded table has all non-zero weights ( #1898 )
...
Fixes a fuzz issue where dictionary_round_trip failed because the compressor was generating corrupt files thanks to zero weights in the table.
* Only setting loaded dict huf table to valid on non-zero
* Adding hasNoZeroWeights test to fse tables
* Forbiding nbBits != 0 when weight == 0
* Reverting the last commit
* Setting table log to 0 when weight == 0
* Small (invalid) zero weight dict test
* Small (valid) zero weight dict test
* Initializing repeatMode vars to check before zero check
* Removing FSE changes to seperate pr
* Reverting accidentally changed file
* Negating bool, using unsigned, optimization nit
2019-11-26 12:24:19 -08:00
Bimba Shrestha
d4e17d0776
Negating bool, updating bool on inner branches
2019-11-26 12:17:43 -08:00
Nick Terrell
718f00ff6f
Optimize decompression speed for gcc and clang ( #1892 )
...
* Optimize `ZSTD_decodeSequence()`
* Optimize Huffman decoding
* Optimize `ZSTD_decompressSequences()`
* Delete `ZSTD_decodeSequenceLong()`
2019-11-25 18:26:19 -08:00
Bimba Shrestha
826b555463
Merge branch 'dev' into oss
2019-11-22 17:29:33 -08:00
Bimba Shrestha
10bce1919e
Mixed declration fix
2019-11-21 13:08:27 -08:00
Bimba Shrestha
0451accab1
Checking noCompressBlock explicitly for rep code confirmation
2019-11-21 13:06:26 -08:00
Nick Terrell
659e9f05cf
Fix null pointer addition
2019-11-20 18:36:04 -08:00
Yann Collet
2d4dcce55f
Merge pull request #1894 from felixhandte/doc-clarify-dctx-reset
...
Easy: Update Comment on `ZSTD_initDStream()`
2019-11-19 16:18:56 -08:00
Nick Terrell
e0d6daabac
Fix Appveyor failure
2019-11-19 11:12:26 -08:00
Bimba Shrestha
8f0c2d04c8
Going back to original flow but removing else return
2019-11-19 10:03:07 -08:00
W. Felix Handte
722149cf2b
Easy: Update Comment on `ZSTD_initDStream()`
2019-11-19 01:57:15 -05:00
Nick Terrell
6a7f65117e
Merge pull request #1866 from legrosbuffle/dev
...
Optimized loop bounds to allow the compiler to unroll the loop.
2019-11-18 16:16:30 -08:00
Nick Terrell
a839d6852c
Merge pull request #1888 from senhuang42/superblocks_fixed
...
RLE test and re-enable RLE in main compression loop
2019-11-18 16:09:33 -08:00
Bimba Shrestha
80586f5e80
Reversing condition order and forwarding error
2019-11-18 13:53:55 -08:00
Bimba Shrestha
dade64428f
Output regular uncompressed block when compressSequences fails
2019-11-18 08:43:14 -08:00
Bimba Shrestha
2d5d961a60
Typo in comment
2019-11-15 19:00:53 -08:00
Bimba Shrestha
dba767c0bb
Leaving room for checksum
2019-11-15 18:44:51 -08:00
Vincent Torri
6b5c10b48c
shared library: rename import library with .dll.a extension
...
mort of open source project are using this extension for the import library.
The Win32 linker is supporting this extension, see
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/win32.html
section "direct linking to a dll"
2019-11-15 19:46:06 +01:00
Clement Courbet
b3c9fc27b4
Optimized loop bounds to allow the compiler to unroll the loop.
...
This has no measurable impact on large files but improves small file
decompression by ~1-2% for 10kB, benchmarked with:
head -c 10000 silesia.tar > /tmp/test
make CC=/usr/local/bin/clang-9 BUILD_STATIC=1 && ./lzbench -ezstd -t1,5 /tmp/test
2019-11-15 08:27:05 +01:00
Sen Huang
d9646dcbb5
Fixed main compression logic changes
2019-11-14 19:39:09 -05:00
Yann Collet
4b1ac69f19
Merge pull request #1868 from senhuang42/superblocks_fixed
...
Superblocks rebased for merge
2019-11-14 13:31:34 -08:00
Sen Huang
c26d32c91c
Change superblock #include to be last
2019-11-14 13:12:17 -05:00
Yann Collet
d67742bc5d
Merge pull request #1858 from senhuang42/dictionary_header_size
...
Method to get dictionary header size
2019-11-14 09:44:07 -08:00
Sen Huang
c85d10d0ea
Remove mixed declarations
2019-11-08 13:57:26 -05:00
Sen Huang
d9c475f3b3
Fix static analyze error, use proper bounds for dictEnd
2019-11-08 13:57:26 -05:00
Sen Huang
d06b90692b
Move asserts to loadZstdDictionary()
2019-11-08 13:57:26 -05:00
Sen Huang
b39149e156
Expose ZSTD_reset_compressedBlockState() to shared API
2019-11-08 13:57:26 -05:00
Sen Huang
6ce335371b
Add error forwarding to loadCEntropy(), make check for dictSize >= 8 from bad merge
2019-11-08 13:57:26 -05:00
Sen Huang
4a61aaf368
Remove redundant comment
2019-11-08 13:57:26 -05:00
Sen Huang
c787b351ea
Use ZSTD Error codes, improve explanation of ZSTD_loadCEntropy() and ZSTD_loadDEntropy()
2019-11-08 13:57:26 -05:00
Sen Huang
04fb42b4f3
Integrated refactor into getDictHeaderSize, now passes tests
2019-11-08 13:57:26 -05:00
Sen Huang
0bcaf6db08
First working pass at refactor of loadZstdDictionary()
2019-11-08 13:57:26 -05:00
Sen Huang
4b141b63e0
Revert "Move decompress symbols into zstd_internal.h, remove dependency"
...
This reverts commit a152b4c67a5266f611db4a2eac4a79003852a795.
2019-11-08 13:57:26 -05:00
Sen Huang
84404cff6e
Move decompress symbols into zstd_internal.h, remove dependency
2019-11-08 13:57:26 -05:00
Sen Huang
341e0641ed
Checks malloc() for failure, returns 0 if so
2019-11-08 13:57:26 -05:00
Sen Huang
97b7f712f3
Change to heap allocation, remove implicit type conversion
2019-11-08 13:57:25 -05:00
Sen Huang
3c36a7f13a
Add ZDICT_getHeaderSize()
2019-11-08 13:57:08 -05:00
Nick Terrell
8c474f9845
Fix parameter selection and adjustment with srcSize == 0
2019-11-07 08:58:43 -08:00
Felix Handte
5688447758
Merge pull request #1873 from felixhandte/make-overlap-log-multithread-only
...
Fix #1861 : Restrict overlapLog Parameter When Not Built With Multithreading
2019-11-06 16:56:37 -05:00
Felix Handte
ba4613602f
Merge pull request #1843 from moozzyk/issue-1637
...
Take ZSTD_parameters as a const pointer
2019-11-06 16:56:14 -05:00
W. Felix Handte
c13f81905a
Fix #1861 : Restrict overlapLog Parameter When Not Built With Multithreading
...
This parameter is unused in single-threaded compression. We should make it
behave like the other multithread-only parameters, for which we only accept
zero when we are not built with multithreading.
2019-11-06 16:05:02 -05:00
Sen Huang
13bb7500e8
Fix frame argument to compression
2019-11-05 16:15:55 -05:00
Sen Huang
f2932fb5eb
Fix more merge conflicts
2019-11-05 15:54:05 -05:00
Sen Huang
7ce891870c
Fix merge conflicts
2019-11-05 15:51:25 -05:00
Bimba Shrestha
3fb5b106da
Replacing some literals with constants
2019-11-05 10:26:57 -08:00
Nick Terrell
60205fec02
Fix 2 bugs in dictionary loading
...
* Silently skip dictionaries less than 8 bytes, unless using `ZSTD_dct_fullDict`.
This changes the compressor, which silently skips dictionaries <= 8 bytes.
* Allow repcodes that are equal to the dictionary content size, since it is in bounds.
2019-11-01 16:52:07 -07:00
Sen Huang
b9ede1c8c2
Make sure contentsize is known
2019-10-30 16:03:58 -04:00
Nick Terrell
9c1860861e
Fix assert in ZSTD_safecopy
...
In the case that `op >= oend_w` it is possible that `diff < 8` because
the two buffers could be adjacent.
Credit to OSS-Fuzz, which found the bug. It isn't reproducible because
it depends on the memory layout.
2019-10-28 17:51:17 -07:00
Felix Handte
01ec595b85
Merge pull request #1851 from felixhandte/pkg-config-prefix-fix
...
In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time
2019-10-28 14:24:56 -04:00
Yann Collet
74065da4c5
updated API inline doc and manual
...
regarding ZSTD_CDict created without a dictBuffer.
2019-10-28 11:15:41 -07:00
W. Felix Handte
74bd76c3ff
In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time
...
Addresses #1794 . Instead of deriving the lib dir and include dir at
build-time, let's do it like everyone else does at pkg-config run-time.
This has the disadvantage that we can no longer override LIBDIR and
INCLUDEDIR in the Makefile and have that reflected in the .pc file.
2019-10-25 15:07:31 -04:00
Yann Collet
c2140e9db0
Merge pull request #1845 from facebook/zbuff
...
improve deprecation warning macro
2019-10-25 09:59:00 -07:00
Yann Collet
a9a216a846
Merge pull request #1824 from senhuang42/new_path_for_cdict
...
Avoid using CDict params when input is large.
2019-10-23 12:04:40 -07:00
Yann Collet
63e435dda1
improve deprecation warning macro
...
fix #1488
although, curiously enough, I was never able to reproduce the issue
(according to the bug report, it should be present while using gcc 4.8).
2019-10-23 11:59:32 -07:00
moozzyk
eda7946a36
Take ZSTD_parameters as a const pointer
...
Fixes : #1637
2019-10-22 23:21:54 -07:00
Yann Collet
f966cd080a
added documentation on DYNAMIC_BMI2 build macro
2019-10-22 17:43:09 -07:00
Yann Collet
5d5c895b18
fix initCStream_advanced() for fast strategies
...
Compression ratio of fast strategies (levels 1 & 2)
was seriously reduced, due to accidental disabling of Literals compression.
Credit to @QrczakMK, which perfectly described the issue, and implementation details,
making the fix straightforward.
Example : initCStream with level 1 on synthetic sample P50 :
Before : 5,273,976 bytes
After : 3,154,678 bytes
ZSTD_compress (for comparison) : 3,154,550
Fix #1787 .
To follow : refactor the test which was supposed to catch this issue (and failed)
2019-10-22 15:01:38 -07:00
Yann Collet
111b0c53b0
update documentation on deprecated functions
...
mostly : note that these functions will soon generate deprecation warnings
2019-10-22 13:51:18 -07:00
Nick Terrell
b1ec94e63c
Fix ZSTD_f_zstd1_magicless for small data
...
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.
Fixes #1813 .
2019-10-21 21:16:17 -07:00
Sen Huang
c2e1e54f24
((x or y) or z) == (x or y or z), remove brackets
2019-10-21 19:16:50 -04:00
Sen Huang
59c81aa31b
Line up comments :)
2019-10-21 19:12:15 -04:00
Sen Huang
dbda8c318a
Trailing comma
2019-10-21 19:10:13 -04:00
Sen Huang
0c00455ea6
Merge branch 'dev' of github.com:senhuang42/zstd into new_path_for_cdict
2019-10-21 19:06:51 -04:00
Sen Huang
5b2f4ac1a8
merge
2019-10-21 19:02:52 -04:00
Sen Huang
2ab484a5f9
Fix bad merge
2019-10-21 18:55:17 -04:00
Nick Terrell
919d1d8e93
Merge pull request #1831 from terrelln/zstdmt-bad-memset
...
[zstdmt] Don't memset the jobDescription
2019-10-21 15:53:57 -07:00
Sen Huang
b6c3459d50
merge
2019-10-21 18:46:17 -04:00
Yann Collet
6cf04c0344
Merge pull request #1834 from facebook/winFix
...
Windows fixes
2019-10-21 13:45:17 -07:00
Sen Huang
676f89902a
Added multiplier, renamed new enum to something more useful
2019-10-21 15:36:12 -04:00
Sen Huang
1f3a51fb52
Updated forceAttachDict param bounds
2019-10-21 15:36:12 -04:00
Sen Huang
8f69c47643
Add enum to decision process
2019-10-21 15:36:12 -04:00
Sen Huang
e4de8b098a
Added support for forcing new CDict behavior and updated enum
2019-10-21 15:36:12 -04:00
Sen Huang
9294f4826b
Changed to int from BYTE
2019-10-21 15:36:12 -04:00
Sen Huang
f0fccc8847
Changed to int from BYTE
2019-10-21 15:36:12 -04:00
Sen Huang
bb2df8c499
Trailing whitespace
2019-10-21 15:36:12 -04:00
Sen Huang
cf51501d2f
Fix test
2019-10-21 15:36:12 -04:00
Sen Huang
ea3cb6988f
Cast to BYTE to appease appveyor
2019-10-21 15:36:12 -04:00
Sen Huang
a727a85a7e
merge conflicts round 2
2019-10-21 15:36:12 -04:00
Sen Huang
053a35fd64
formatting
2019-10-21 15:35:33 -04:00
Sen Huang
3fa4daaa55
Fix error
2019-10-21 15:35:33 -04:00
Sen Huang
3328348c63
Add compressionlevel to cdict
2019-10-21 15:32:39 -04:00
Felix Handte
cf725630a6
Merge pull request #1795 from felixhandte/workspace-asan
...
Add Poisoned Redzones to the Workspace When Compiling with ASAN
2019-10-21 12:15:17 -04:00
Sen Huang
e8aa3e486d
Updated forceAttachDict param bounds
2019-10-20 22:01:08 -04:00
Sen Huang
6d297265f9
Add enum to decision process
2019-10-20 19:02:47 -04:00
Sen Huang
1daa898c93
Added support for forcing new CDict behavior and updated enum
2019-10-20 14:03:09 -04:00
Nick Terrell
0bc39bc3a0
[zstdmt] Don't memset the jobDescription
2019-10-18 15:05:51 -07:00
Nick Terrell
243824551f
[threading] Add debug utilities
2019-10-18 15:05:34 -07:00
Yann Collet
1795133c45
refactored FIO_compressMultipleFilenames() prototype
...
for consistency
2019-10-17 15:32:03 -07:00
Yann Collet
6446ffb277
Merge pull request #1827 from facebook/dm_Dct
...
updated erroneous comments using ZSTD_dm_*
2019-10-17 10:30:58 -07:00
Yann Collet
19741c7d99
Merge pull request #1815 from facebook/zlibwrap
...
make zlibWrapper strict ISO-C90 compatible
2019-10-16 16:45:15 -07:00
Yann Collet
6323966e53
updated erroneous comments using ZSTD_dm_*
...
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822 )
2019-10-16 16:14:04 -07:00
Yann Collet
2d5201b0ab
removed wildcopy8()
...
which is no longer used,
noticed by @davidbolvansky
2019-10-16 14:51:33 -07:00
Sen Huang
4455f00cb8
Changed to int from BYTE
2019-10-16 15:06:02 -04:00
Sen Huang
4f7d26b0ee
Changed to int from BYTE
2019-10-16 15:05:29 -04:00
Sen Huang
cf00ea367a
Trailing whitespace
2019-10-16 10:31:27 -04:00
Sen Huang
8cb2174446
Fix test
2019-10-16 10:29:31 -04:00
Sen Huang
5e901b6f32
Cast to BYTE to appease appveyor
2019-10-15 13:58:44 -04:00
Sen Huang
5c010c9d2d
merge conflicts round 2
2019-10-15 13:10:05 -04:00
Sen Huang
a06b51879c
merge conflict
2019-10-15 12:58:50 -04:00
Sen Huang
23dac23a49
formatting
2019-10-15 12:44:48 -04:00
Sen Huang
0c8df5c928
Fix error
2019-10-15 12:28:23 -04:00
Sen Huang
a65eb39f9d
Add compressionlevel to cdict
2019-10-15 10:22:06 -04:00
Yann Collet
fb77afc626
Merge pull request #1760 from bimbashrestha/extract_sequences_api
...
Adding api for extracting sequences from seqstore
2019-10-10 13:11:18 -07:00
W. Felix Handte
ede31da2ea
Fix CCtx Size Estimation
2019-10-10 15:02:08 -04:00
W. Felix Handte
bd6a20b8a0
Expand Default Redzone Size
2019-10-10 13:45:55 -04:00
W. Felix Handte
2c80a9f8ac
Check if CCtx in Workspace after Null Check
2019-10-10 13:40:16 -04:00
W. Felix Handte
b6987acbbf
Declare the ASAN Functions We Need, Don't Include the Header
2019-10-10 13:40:16 -04:00
W. Felix Handte
0ffae7e440
Stop Allocating Extra Space for Table Redzones
2019-10-10 13:40:16 -04:00
W. Felix Handte
a07037b784
Don't Try to Redzone the Tables
2019-10-10 13:40:16 -04:00
W. Felix Handte
0cc481ef66
Fix Workspace Size Calculation
2019-10-10 13:40:16 -04:00
W. Felix Handte
b6c0a02a17
Fix ZSTD_sizeof_matchState() Calculation
2019-10-10 13:40:16 -04:00
W. Felix Handte
8cffd6ed08
Avoid ASAN Failure in ZSTD_cwksp_free()
2019-10-10 13:40:16 -04:00
W. Felix Handte
ef0b5707c5
Refactor Freeing CCtxes / CDicts Inside Workspaces
2019-10-10 13:40:16 -04:00
W. Felix Handte
143b296cf6
Surround Workspace Allocs with Dead Zone
2019-10-10 13:40:16 -04:00
W. Felix Handte
19a0955ec9
Add `ZSTD_cwksp_alloc_size()` to Help Calculate Needed Workspace Size
2019-10-10 13:40:16 -04:00
W. Felix Handte
da88c35d41
Stop Assuming Tables are Adjacent
2019-10-10 13:40:16 -04:00
W. Felix Handte
35c30d6ca7
Poison Unused Workspace Memory
2019-10-10 13:40:16 -04:00
W. Felix Handte
edb6d884a5
Detect Whether We're Being Compiled with ASAN
2019-10-10 13:40:16 -04:00
W. Felix Handte
dc1fb684bf
Remove Unused MEM_SKIP_MSAN Macro
2019-10-10 13:40:16 -04:00
Bimba Shrestha
36528b96c4
Manually moving instead of memcpy on decoder and using genBuffer()
2019-10-03 09:26:51 -07:00
Bimba Shrestha
61ec4c2e7f
Cleaning sequence parsing logic
2019-10-03 06:42:40 -07:00
Yann Collet
cb18fffe65
enforce C90 compatibility for zlibWrapper
2019-09-24 17:50:58 -07:00
Yann Collet
ad2a2785f7
bump version number to v1.4.4
...
so that future reports on `dev` branch use this number instead
2019-09-24 15:15:33 -07:00
Bimba Shrestha
c04245b257
Replacing assert with memory_allocation error code throw
2019-09-23 15:42:16 -07:00
Bimba Shrestha
be0bebd24e
Adding test and null check for malloc
2019-09-23 15:08:18 -07:00
Dávid Bolvanský
1ab1a40c9c
Fixed one more place
2019-09-23 21:32:56 +02:00
Dávid Bolvanský
1f7228c040
Use clz ^ 31 instead of 31 - clz; better codegen for GCC
2019-09-23 21:23:09 +02:00
Nick Terrell
7451c6578c
Merge pull request #1804 from terrelln/wild-and-fast
...
Optimize (de)compression and fix wildcopy overread
2019-09-21 17:04:36 -07:00
Nick Terrell
5cb7615f1f
Add UNUSED_ATTR to ZSTD_storeSeq()
2019-09-20 21:37:13 -07:00
Nick Terrell
5dc0a1d659
HINT_INLINE ZSTD_storeSeq()
...
Clang on Mac wasn't inlining `ZSTD_storeSeq()` in level 1, which was
causing a 5% performance regression. This fixes it.
2019-09-20 16:39:27 -07:00
Bimba Shrestha
f3c4fd17e3
Passing in dummy dst buffer of compressbound(srcSize)
2019-09-20 15:50:58 -07:00
Felix Handte
c047fcf7bf
Merge pull request #1806 from felixhandte/estimate-cctx-doc
...
Update Comment on `ZSTD_estimateCCtxSize()`
2019-09-20 15:36:00 -04:00
Nick Terrell
44c65da97e
Remove literals overread in ZSTD_storeSeq() for ~neutral perf
2019-09-20 12:23:25 -07:00
W. Felix Handte
f7d9b36835
Update Comment on `ZSTD_estimateCCtxSize()`
2019-09-20 14:11:29 -04:00
Nick Terrell
fde217df04
Fix bounds check in ZSTD_storeSeq()
2019-09-20 08:25:12 -07:00
Nick Terrell
67b1f5fc72
Fix too strict assert
2019-09-20 01:23:35 -07:00
Nick Terrell
ddab2a94e8
Pass iend into ZSTD_storeSeq() to allow ZSTD_wildcopy()
2019-09-20 00:56:20 -07:00
Nick Terrell
cdad7fa512
Widen ZSTD_wildcopy to 32 bytes
2019-09-20 00:52:15 -07:00
Nick Terrell
efd37a64ea
Optimize decompression and fix wildcopy overread
...
* Bump `WILDCOPY_OVERLENGTH` to 16 to fix the wildcopy overread.
* Optimize `ZSTD_wildcopy()` by removing unnecessary branches and
unrolling the loop.
* Extract `ZSTD_overlapCopy8()` into its own function.
* Add `ZSTD_safecopy()` for `ZSTD_execSequenceEnd()`. It is
optimized for single long sequences, since that is the important
case that can end up in `ZSTD_execSequenceEnd()`. Without this
optimization, decompressing a block with 1 long match goes
from 5.7 GB/s to 800 MB/s.
* Refactor `ZSTD_execSequenceEnd()`.
* Increase the literal copy shortcut to 16.
* Add a shortcut for offset >= 16.
* Simplify `ZSTD_execSequence()` by pushing more cases into
`ZSTD_execSequenceEnd()`.
* Delete `ZSTD_execSequenceLong()` since it is exactly the
same as `ZSTD_execSequence()`.
clang-8 seeds +17.5% on silesia and +21.8% on enwik8.
gcc-9 sees +12% on silesia and +15.5% on enwik8.
TODO: More detailed measurements, and on more datasets.
Crdit to OSS-Fuzz for finding the wildcopy overread.
2019-09-19 21:07:14 -07:00
Bimba Shrestha
ae6d0e64ae
Addressing comments
2019-09-19 15:25:20 -07:00
Yann Collet
3cac061db5
Merge pull request #1802 from bimbashrestha/rle_block_bound_fix_pt2
...
Adding 4 blocks to FSE_BLOCKBOUND() in lib/common (different from las…
2019-09-18 16:32:37 -07:00
Bimba Shrestha
6e9f6813bb
adding bit container size
2019-09-18 13:49:45 -07:00
Bimba Shrestha
f9b6abb896
Adding 4 blocks to FSE_BLOCKBOUND() in lib/common (different from last week)
2019-09-18 13:29:05 -07:00
Yann Collet
bfff5b30a4
Merge pull request #1756 from mgrice/dev
...
Improvements in zstd decode performance
2019-09-18 11:35:50 -07:00
Yann Collet
243200e5bf
minor refactor of ZSTD_fast
...
- reduced variables lifetime
- more accurate code comments
2019-09-17 14:02:57 -07:00
Bimba Shrestha
76fea3fb99
Resolving appveyor test failure implicit conversion
2019-09-16 14:02:23 -07:00
Bimba Shrestha
a874435478
Merge branch 'dev' into extract_sequences_api
2019-09-16 13:29:59 -07:00
Felix Handte
2164a130f3
Merge pull request #1780 from felixhandte/workspace-efficiency-3
...
Avoid Clearing Tables Even When Changing CParams
2019-09-16 14:37:05 -04:00
W. Felix Handte
72ea79cacd
Don't Include `sanitizer/msan_interface.h`, Since Not All Platforms Provide It
...
Instead, explicitly declare the functions we use.
2019-09-16 12:08:03 -04:00
Bimba Shrestha
bff6072e3a
Bailing early when collecting sequences and documentation
2019-09-16 08:26:21 -07:00
Nick Terrell
fbeaf6989e
[libzstd] Improve advanced API docs
2019-09-15 12:41:24 -07:00
Yann Collet
09b1844d9b
Merge pull request #1784 from bimbashrestha/fse_block_bound_err
...
Rearranging assert and allowing 4 extra for FSE_BLOCKBOUND()
2019-09-12 19:09:27 -07:00
Bimba Shrestha
fe9af338ed
Added assert to BIT_flushBits()
2019-09-12 15:35:27 -07:00
Bimba Shrestha
43da5bf27e
Rearranging assert and allowing 4 extra for FSE_BLOCKBOUND()
2019-09-12 14:43:50 -07:00
W. Felix Handte
20c69077d1
Shrink Table Valid End During Alloc Alignment / Phase Change
2019-09-11 17:14:59 -04:00
W. Felix Handte
51d90668ba
Add Assertions to Confirm that Workspace Pointers are Correctly Ordered
2019-09-11 17:14:59 -04:00
W. Felix Handte
a10c191613
`__msan_poison()` Workspace When Preparing for Re-Use
2019-09-11 17:14:45 -04:00
W. Felix Handte
7c57e2b9ca
Zero `h3size` When `h3log` is 0
...
This led to a nasty edgecase, where index reduction for modes that don't use
the h3 table would have a degenerate table (size 4) allocated and marked clean,
but which would not be re-indexed.
2019-09-11 13:14:26 -04:00
W. Felix Handte
bc020eec92
Also Shrink Clean Table Area When Reducing Indices
2019-09-11 11:40:57 -04:00
W. Felix Handte
1999b2ed9b
Update DEBUGLOG Statements
2019-09-11 11:21:00 -04:00
W. Felix Handte
13e29a56de
Shrink Clean Table Area When Copying Table Contents into Context
...
The source matchState is potentially at a lower current index, which means
that any extra table space not overwritten by the copy may now contain
invalid indices. The simple solution is to unconditionally shrink the valid
table area to just the area overwritten.
2019-09-11 11:18:45 -04:00
W. Felix Handte
edb3ad053e
Comments
2019-09-10 18:25:45 -04:00
W. Felix Handte
f31ef28ff8
Only Reset Indexing in `ZSTD_resetCCtx_internal()` When Necessary
2019-09-10 18:25:45 -04:00
W. Felix Handte
9968a53e91
Remove No-Longer-Used Continuation Functions
2019-09-10 18:25:45 -04:00
W. Felix Handte
1b28e80416
Remove Fast Continue Path in `ZSTD_resetCCtx_internal()`
2019-09-10 18:25:45 -04:00
W. Felix Handte
ad16eda5e4
`ZSTD_reset_matchState` Optionally Doesn't Restart Indexing
2019-09-10 18:25:45 -04:00
W. Felix Handte
5b10bb5ec3
Rename `ZSTD_compResetPolicy_e` Values and Add Comment
2019-09-10 18:25:45 -04:00
W. Felix Handte
0492b9a9ec
Accept `ZSTD_indexResetPolicy_e` Param in `ZSTD_reset_matchState()`
2019-09-10 18:25:45 -04:00
W. Felix Handte
14c5471d5e
Introduce `ZSTD_indexResetPolicy_e` Enum
2019-09-10 18:25:45 -04:00
W. Felix Handte
17b6da2e0f
Track Usable Table Space in Compression Workspace
2019-09-10 18:25:37 -04:00
Yann Collet
22bd158e0f
Merge pull request #1712 from felixhandte/workspace-efficiency-2
...
Allocate Internal Buffers via Workspace Abstraction
2019-09-10 15:20:29 -07:00
Bimba Shrestha
1407919d13
Addressing comments on parsing
2019-09-10 15:10:50 -07:00
Bimba Shrestha
47199480da
Cleaning up parsing per suggestion
2019-09-10 13:18:59 -07:00
W. Felix Handte
a9d373f093
Remove Empty lib/compress/zstd_cwksp.c
2019-09-10 16:03:13 -04:00
Yann Collet
5ba495b622
Merge pull request #1775 from facebook/edufix
...
fix educational decoder
2019-09-10 12:12:08 -07:00
Yann Collet
41416f0927
Merge pull request #1773 from bimbashrestha/rle_first_block_decompression_fix
...
Removing redundant condition in decompression, making first block rle…
2019-09-10 11:17:29 -07:00
Bimba Shrestha
e3c5825918
Fizing litLength == 0 case
2019-09-10 10:38:13 -07:00
Bimba Shrestha
9e7bb55e14
Addressing comments
2019-09-09 20:04:46 -07:00
W. Felix Handte
81208fd7c2
Forward Declare `ZSTD_cwksp_available_space` to Fix Build
2019-09-09 19:10:09 -04:00
W. Felix Handte
91bf1babd1
Inline Workspace Functions
2019-09-09 18:53:53 -04:00
W. Felix Handte
0db3ffe7ee
Forward resetCCtx Errors when Using CDict
2019-09-09 16:47:19 -04:00
W. Felix Handte
eb6f69d978
Fix sizeof_CCtx and sizeof_CDict Calculations for Statically Init'ed Objects
2019-09-09 16:45:17 -04:00
W. Felix Handte
e3703825a8
Fix workspaceTooSmall Calculation
2019-09-09 15:12:14 -04:00
W. Felix Handte
0a65a67901
Shorten `&zc->workspace` -> `ws` in `ZSTD_resetCCtx_internal()`
2019-09-09 14:59:09 -04:00
W. Felix Handte
1120e4d962
Clean Up TODOs and Comments pt. II
2019-09-09 14:04:39 -04:00
W. Felix Handte
c60e1c3be5
Nit
2019-09-09 13:34:08 -04:00
W. Felix Handte
7d7b665c90
Pull Phase Advance Logic Out into Internal Function
2019-09-09 13:34:08 -04:00
W. Felix Handte
8549ae9f1d
Hide Workspace Movement Behind Helper Function
2019-09-09 13:34:08 -04:00
W. Felix Handte
2405c03bcd
Fix DEBUGLOG Statement Levels
2019-09-09 13:34:08 -04:00
W. Felix Handte
7100d24221
Fix Rescale Continue Special Case
2019-09-09 13:34:08 -04:00
W. Felix Handte
7321e4c9f3
Remove Unused noRealloc CRP Value
2019-09-09 13:34:08 -04:00
W. Felix Handte
901bba4ca6
Re-Implement Workspace Shrinking when Oversized
2019-09-09 13:34:08 -04:00
W. Felix Handte
881bcd80ca
Cleanup from Move
2019-09-09 13:34:08 -04:00
W. Felix Handte
b511a84adc
Move Workspace Functions to Their Own File
2019-09-09 13:34:08 -04:00
W. Felix Handte
077a2d7dc9
Rename
2019-09-09 13:34:08 -04:00
W. Felix Handte
ebd162194f
Clean Up TODOs and Comments
2019-09-09 13:34:08 -04:00
W. Felix Handte
2abe0145b1
Improve Comments a Bit
2019-09-09 13:34:08 -04:00
W. Felix Handte
7a2416a863
Allocate CDict in Workspace (Rather than in Separate Allocation)
2019-09-09 13:34:08 -04:00
W. Felix Handte
65057cf009
Rewrite ZSTD_initStaticCCtx to Alloc CCtx in Workspace
2019-09-09 13:34:08 -04:00