80c26117a9
Line-wrapping
2020-02-03 09:38:16 -08:00
ee8a712af3
Using appliedParams instead of supplied params
2020-01-31 15:49:07 -08:00
e32e3e8662
Improve wildcopy performance across the board
2020-01-28 20:37:04 -08:00
7627759b4e
Merge pull request #1972 from terrelln/check-cont
...
Move ZSTD_checkContinuity() to zstd_decompress_block.c
2020-01-23 22:02:50 -08:00
fa6a772f38
Initialize dctx->bType to silence valgrind false positive
2020-01-23 17:54:48 -08:00
cb2abc3dbe
Fix performance regression on aarch64 with clang
2020-01-23 17:31:14 -08:00
6e3cd5b024
Move ZSTD_checkContinuity() to zstd_decompress_block.c
2020-01-23 12:27:39 -08:00
a11a9271d6
Fix lowLimit underflow in overflow correction
2020-01-17 12:10:18 -08:00
036b30b555
Fix super block compression and stream raw blocks in decompression ( #1947 )
...
Super blocks must never violate the zstd block bound of input_size + ZSTD_blockHeaderSize. The individual sub-blocks may, but not the super block. If the superblock violates the block bound we are liable to violate ZSTD_compressBound(), which we must not do. Whenever the super block violates the block bound we instead emit an uncompressed block.
This means we increase the latency because of the single uncompressed block. I fix this by enabling streaming an uncompressed block, so the latency of an uncompressed block is 1 byte. This doesn't reduce the latency of the buffer-less API, but I don't think we really care.
* I added a test case that verifies that the decompression has 1 byte latency.
* I rely on existing zstreamtest / fuzzer / libfuzzer regression tests for correctness. During development I had several correctness bugs, and they easily caught them.
* The added assert that the superblock doesn't violate the block bound will help us discover any missed conditions (though I think I got them all).
Credit to OSS-Fuzz.
2020-01-10 18:02:11 -08:00
d1cc9d2797
[fuzz] Allow zero sized buffers for streaming fuzzers ( #1945 )
...
* Allow zero sized buffers in `stream_decompress`. Ensure that we never have two
zero sized buffers in a row so we guarantee forwards progress.
* Make case 4 in `stream_round_trip` do a zero sized buffers call followed by
a full call to guarantee forwards progress.
* Fix `limitCopy()` in legacy decoders.
* Fix memcpy in `zstdmt_compress.c`.
Catches the bug fixed in PR #1939
2020-01-09 11:38:50 -08:00
03ffda7b88
fix UBSAN's invalid-null-argument error in zstd_decompress.c ( #1939 )
2020-01-08 16:17:42 -08:00
b1f53b1a10
[fuzz] Dividing by targetCBlockSize instead of blockSize for nbBlocks fit ( #1936 )
...
* Adding fail logging for superblock flow
* Dividing by targetCBlockSize instead of blockSize
* Adding new const and using more acurate formula for nbBlocks
* Only do dstCapacity check if using superblock
* Remvoing disabling logic
* Updating test to make it catch more extreme case of previou bug
* Also updating comment
* Only taking compressEnd shortcut on non-superblock
2020-01-03 16:53:51 -08:00
56415efc76
Constifying, malloc check and naming nit
2019-12-17 17:16:51 -08:00
5225dcfc0f
Adding bool to check if enough room left for noCompress superblocks
2019-12-13 15:47:28 -08:00
d73e2fb465
Merge pull request #1891 from bimbashrestha/oss
...
[fuzz] Superblock fuzz issues
2019-12-10 13:17:00 -08:00
e1913dc87f
Making const, removing unnecessary indent, changing parameter order
2019-12-04 15:51:17 -08:00
2ec556fec2
Moving init/end functions, moving compressSuperBlock inside body()
2019-12-04 15:23:13 -08:00
ffb0463041
Refactor
2019-12-04 14:52:27 -08:00
49c6d49247
[fuzz] msan uninitialized unsigned value ( #1908 )
...
Fixes new fuzz issue
Credit to OSS-Fuzz
* Initializing unsigned value
* Initialilzing to 1 instead of 0 because its more conservative
* Unconditionoally setting to check first and then checking zero
* Moving bool to before block for c90
* Move check set before block
2019-12-04 10:02:17 -08:00
5120883a9c
bumped version number
...
so that potential issue report do not confuse `dev` with latest release
2019-12-03 17:06:42 -08:00
1fc9352f81
Using bss var instead of creating new bool
2019-12-02 21:39:06 -08:00
1f681d8592
Merge branch 'oss' of https://github.com/bimbashrestha/zstd into oss
2019-11-27 10:56:54 -08:00
a3a3c62b81
[fuzz] Only set HUF_repeat_valid if loaded table has all non-zero weights ( #1898 )
...
Fixes a fuzz issue where dictionary_round_trip failed because the compressor was generating corrupt files thanks to zero weights in the table.
* Only setting loaded dict huf table to valid on non-zero
* Adding hasNoZeroWeights test to fse tables
* Forbiding nbBits != 0 when weight == 0
* Reverting the last commit
* Setting table log to 0 when weight == 0
* Small (invalid) zero weight dict test
* Small (valid) zero weight dict test
* Initializing repeatMode vars to check before zero check
* Removing FSE changes to seperate pr
* Reverting accidentally changed file
* Negating bool, using unsigned, optimization nit
2019-11-26 12:24:19 -08:00
d4e17d0776
Negating bool, updating bool on inner branches
2019-11-26 12:17:43 -08:00
718f00ff6f
Optimize decompression speed for gcc and clang ( #1892 )
...
* Optimize `ZSTD_decodeSequence()`
* Optimize Huffman decoding
* Optimize `ZSTD_decompressSequences()`
* Delete `ZSTD_decodeSequenceLong()`
2019-11-25 18:26:19 -08:00
826b555463
Merge branch 'dev' into oss
2019-11-22 17:29:33 -08:00
10bce1919e
Mixed declration fix
2019-11-21 13:08:27 -08:00
0451accab1
Checking noCompressBlock explicitly for rep code confirmation
2019-11-21 13:06:26 -08:00
659e9f05cf
Fix null pointer addition
2019-11-20 18:36:04 -08:00
2d4dcce55f
Merge pull request #1894 from felixhandte/doc-clarify-dctx-reset
...
Easy: Update Comment on `ZSTD_initDStream()`
2019-11-19 16:18:56 -08:00
e0d6daabac
Fix Appveyor failure
2019-11-19 11:12:26 -08:00
8f0c2d04c8
Going back to original flow but removing else return
2019-11-19 10:03:07 -08:00
722149cf2b
Easy: Update Comment on `ZSTD_initDStream()`
2019-11-19 01:57:15 -05:00
6a7f65117e
Merge pull request #1866 from legrosbuffle/dev
...
Optimized loop bounds to allow the compiler to unroll the loop.
2019-11-18 16:16:30 -08:00
a839d6852c
Merge pull request #1888 from senhuang42/superblocks_fixed
...
RLE test and re-enable RLE in main compression loop
2019-11-18 16:09:33 -08:00
80586f5e80
Reversing condition order and forwarding error
2019-11-18 13:53:55 -08:00
dade64428f
Output regular uncompressed block when compressSequences fails
2019-11-18 08:43:14 -08:00
2d5d961a60
Typo in comment
2019-11-15 19:00:53 -08:00
dba767c0bb
Leaving room for checksum
2019-11-15 18:44:51 -08:00
6b5c10b48c
shared library: rename import library with .dll.a extension
...
mort of open source project are using this extension for the import library.
The Win32 linker is supporting this extension, see
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/win32.html
section "direct linking to a dll"
2019-11-15 19:46:06 +01:00
b3c9fc27b4
Optimized loop bounds to allow the compiler to unroll the loop.
...
This has no measurable impact on large files but improves small file
decompression by ~1-2% for 10kB, benchmarked with:
head -c 10000 silesia.tar > /tmp/test
make CC=/usr/local/bin/clang-9 BUILD_STATIC=1 && ./lzbench -ezstd -t1,5 /tmp/test
2019-11-15 08:27:05 +01:00
d9646dcbb5
Fixed main compression logic changes
2019-11-14 19:39:09 -05:00
4b1ac69f19
Merge pull request #1868 from senhuang42/superblocks_fixed
...
Superblocks rebased for merge
2019-11-14 13:31:34 -08:00
c26d32c91c
Change superblock #include to be last
2019-11-14 13:12:17 -05:00
d67742bc5d
Merge pull request #1858 from senhuang42/dictionary_header_size
...
Method to get dictionary header size
2019-11-14 09:44:07 -08:00
c85d10d0ea
Remove mixed declarations
2019-11-08 13:57:26 -05:00
d9c475f3b3
Fix static analyze error, use proper bounds for dictEnd
2019-11-08 13:57:26 -05:00
d06b90692b
Move asserts to loadZstdDictionary()
2019-11-08 13:57:26 -05:00
b39149e156
Expose ZSTD_reset_compressedBlockState() to shared API
2019-11-08 13:57:26 -05:00
6ce335371b
Add error forwarding to loadCEntropy(), make check for dictSize >= 8 from bad merge
2019-11-08 13:57:26 -05:00
4a61aaf368
Remove redundant comment
2019-11-08 13:57:26 -05:00
c787b351ea
Use ZSTD Error codes, improve explanation of ZSTD_loadCEntropy() and ZSTD_loadDEntropy()
2019-11-08 13:57:26 -05:00
04fb42b4f3
Integrated refactor into getDictHeaderSize, now passes tests
2019-11-08 13:57:26 -05:00
0bcaf6db08
First working pass at refactor of loadZstdDictionary()
2019-11-08 13:57:26 -05:00
4b141b63e0
Revert "Move decompress symbols into zstd_internal.h, remove dependency"
...
This reverts commit a152b4c67a5266f611db4a2eac4a79003852a795.
2019-11-08 13:57:26 -05:00
84404cff6e
Move decompress symbols into zstd_internal.h, remove dependency
2019-11-08 13:57:26 -05:00
341e0641ed
Checks malloc() for failure, returns 0 if so
2019-11-08 13:57:26 -05:00
97b7f712f3
Change to heap allocation, remove implicit type conversion
2019-11-08 13:57:25 -05:00
3c36a7f13a
Add ZDICT_getHeaderSize()
2019-11-08 13:57:08 -05:00
8c474f9845
Fix parameter selection and adjustment with srcSize == 0
2019-11-07 08:58:43 -08:00
5688447758
Merge pull request #1873 from felixhandte/make-overlap-log-multithread-only
...
Fix #1861 : Restrict overlapLog Parameter When Not Built With Multithreading
2019-11-06 16:56:37 -05:00
ba4613602f
Merge pull request #1843 from moozzyk/issue-1637
...
Take ZSTD_parameters as a const pointer
2019-11-06 16:56:14 -05:00
c13f81905a
Fix #1861 : Restrict overlapLog Parameter When Not Built With Multithreading
...
This parameter is unused in single-threaded compression. We should make it
behave like the other multithread-only parameters, for which we only accept
zero when we are not built with multithreading.
2019-11-06 16:05:02 -05:00
13bb7500e8
Fix frame argument to compression
2019-11-05 16:15:55 -05:00
f2932fb5eb
Fix more merge conflicts
2019-11-05 15:54:05 -05:00
7ce891870c
Fix merge conflicts
2019-11-05 15:51:25 -05:00
3fb5b106da
Replacing some literals with constants
2019-11-05 10:26:57 -08:00
60205fec02
Fix 2 bugs in dictionary loading
...
* Silently skip dictionaries less than 8 bytes, unless using `ZSTD_dct_fullDict`.
This changes the compressor, which silently skips dictionaries <= 8 bytes.
* Allow repcodes that are equal to the dictionary content size, since it is in bounds.
2019-11-01 16:52:07 -07:00
b9ede1c8c2
Make sure contentsize is known
2019-10-30 16:03:58 -04:00
9c1860861e
Fix assert in ZSTD_safecopy
...
In the case that `op >= oend_w` it is possible that `diff < 8` because
the two buffers could be adjacent.
Credit to OSS-Fuzz, which found the bug. It isn't reproducible because
it depends on the memory layout.
2019-10-28 17:51:17 -07:00
01ec595b85
Merge pull request #1851 from felixhandte/pkg-config-prefix-fix
...
In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time
2019-10-28 14:24:56 -04:00
74065da4c5
updated API inline doc and manual
...
regarding ZSTD_CDict created without a dictBuffer.
2019-10-28 11:15:41 -07:00
74bd76c3ff
In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time
...
Addresses #1794 . Instead of deriving the lib dir and include dir at
build-time, let's do it like everyone else does at pkg-config run-time.
This has the disadvantage that we can no longer override LIBDIR and
INCLUDEDIR in the Makefile and have that reflected in the .pc file.
2019-10-25 15:07:31 -04:00
c2140e9db0
Merge pull request #1845 from facebook/zbuff
...
improve deprecation warning macro
2019-10-25 09:59:00 -07:00
a9a216a846
Merge pull request #1824 from senhuang42/new_path_for_cdict
...
Avoid using CDict params when input is large.
2019-10-23 12:04:40 -07:00
63e435dda1
improve deprecation warning macro
...
fix #1488
although, curiously enough, I was never able to reproduce the issue
(according to the bug report, it should be present while using gcc 4.8).
2019-10-23 11:59:32 -07:00
eda7946a36
Take ZSTD_parameters as a const pointer
...
Fixes : #1637
2019-10-22 23:21:54 -07:00
f966cd080a
added documentation on DYNAMIC_BMI2 build macro
2019-10-22 17:43:09 -07:00
5d5c895b18
fix initCStream_advanced() for fast strategies
...
Compression ratio of fast strategies (levels 1 & 2)
was seriously reduced, due to accidental disabling of Literals compression.
Credit to @QrczakMK, which perfectly described the issue, and implementation details,
making the fix straightforward.
Example : initCStream with level 1 on synthetic sample P50 :
Before : 5,273,976 bytes
After : 3,154,678 bytes
ZSTD_compress (for comparison) : 3,154,550
Fix #1787 .
To follow : refactor the test which was supposed to catch this issue (and failed)
2019-10-22 15:01:38 -07:00
111b0c53b0
update documentation on deprecated functions
...
mostly : note that these functions will soon generate deprecation warnings
2019-10-22 13:51:18 -07:00
b1ec94e63c
Fix ZSTD_f_zstd1_magicless for small data
...
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.
Fixes #1813 .
2019-10-21 21:16:17 -07:00
c2e1e54f24
((x or y) or z) == (x or y or z), remove brackets
2019-10-21 19:16:50 -04:00
59c81aa31b
Line up comments :)
2019-10-21 19:12:15 -04:00
dbda8c318a
Trailing comma
2019-10-21 19:10:13 -04:00
0c00455ea6
Merge branch 'dev' of github.com:senhuang42/zstd into new_path_for_cdict
2019-10-21 19:06:51 -04:00
5b2f4ac1a8
merge
2019-10-21 19:02:52 -04:00
2ab484a5f9
Fix bad merge
2019-10-21 18:55:17 -04:00
919d1d8e93
Merge pull request #1831 from terrelln/zstdmt-bad-memset
...
[zstdmt] Don't memset the jobDescription
2019-10-21 15:53:57 -07:00
b6c3459d50
merge
2019-10-21 18:46:17 -04:00
6cf04c0344
Merge pull request #1834 from facebook/winFix
...
Windows fixes
2019-10-21 13:45:17 -07:00
676f89902a
Added multiplier, renamed new enum to something more useful
2019-10-21 15:36:12 -04:00
1f3a51fb52
Updated forceAttachDict param bounds
2019-10-21 15:36:12 -04:00
8f69c47643
Add enum to decision process
2019-10-21 15:36:12 -04:00
e4de8b098a
Added support for forcing new CDict behavior and updated enum
2019-10-21 15:36:12 -04:00
9294f4826b
Changed to int from BYTE
2019-10-21 15:36:12 -04:00
f0fccc8847
Changed to int from BYTE
2019-10-21 15:36:12 -04:00
bb2df8c499
Trailing whitespace
2019-10-21 15:36:12 -04:00
cf51501d2f
Fix test
2019-10-21 15:36:12 -04:00
ea3cb6988f
Cast to BYTE to appease appveyor
2019-10-21 15:36:12 -04:00
a727a85a7e
merge conflicts round 2
2019-10-21 15:36:12 -04:00
053a35fd64
formatting
2019-10-21 15:35:33 -04:00
3fa4daaa55
Fix error
2019-10-21 15:35:33 -04:00
3328348c63
Add compressionlevel to cdict
2019-10-21 15:32:39 -04:00
cf725630a6
Merge pull request #1795 from felixhandte/workspace-asan
...
Add Poisoned Redzones to the Workspace When Compiling with ASAN
2019-10-21 12:15:17 -04:00
e8aa3e486d
Updated forceAttachDict param bounds
2019-10-20 22:01:08 -04:00
6d297265f9
Add enum to decision process
2019-10-20 19:02:47 -04:00
1daa898c93
Added support for forcing new CDict behavior and updated enum
2019-10-20 14:03:09 -04:00
0bc39bc3a0
[zstdmt] Don't memset the jobDescription
2019-10-18 15:05:51 -07:00
243824551f
[threading] Add debug utilities
2019-10-18 15:05:34 -07:00
1795133c45
refactored FIO_compressMultipleFilenames() prototype
...
for consistency
2019-10-17 15:32:03 -07:00
6446ffb277
Merge pull request #1827 from facebook/dm_Dct
...
updated erroneous comments using ZSTD_dm_*
2019-10-17 10:30:58 -07:00
19741c7d99
Merge pull request #1815 from facebook/zlibwrap
...
make zlibWrapper strict ISO-C90 compatible
2019-10-16 16:45:15 -07:00
6323966e53
updated erroneous comments using ZSTD_dm_*
...
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822 )
2019-10-16 16:14:04 -07:00
2d5201b0ab
removed wildcopy8()
...
which is no longer used,
noticed by @davidbolvansky
2019-10-16 14:51:33 -07:00
4455f00cb8
Changed to int from BYTE
2019-10-16 15:06:02 -04:00
4f7d26b0ee
Changed to int from BYTE
2019-10-16 15:05:29 -04:00
cf00ea367a
Trailing whitespace
2019-10-16 10:31:27 -04:00
8cb2174446
Fix test
2019-10-16 10:29:31 -04:00
5e901b6f32
Cast to BYTE to appease appveyor
2019-10-15 13:58:44 -04:00
5c010c9d2d
merge conflicts round 2
2019-10-15 13:10:05 -04:00
a06b51879c
merge conflict
2019-10-15 12:58:50 -04:00
23dac23a49
formatting
2019-10-15 12:44:48 -04:00
0c8df5c928
Fix error
2019-10-15 12:28:23 -04:00
a65eb39f9d
Add compressionlevel to cdict
2019-10-15 10:22:06 -04:00
fb77afc626
Merge pull request #1760 from bimbashrestha/extract_sequences_api
...
Adding api for extracting sequences from seqstore
2019-10-10 13:11:18 -07:00
ede31da2ea
Fix CCtx Size Estimation
2019-10-10 15:02:08 -04:00
bd6a20b8a0
Expand Default Redzone Size
2019-10-10 13:45:55 -04:00
2c80a9f8ac
Check if CCtx in Workspace after Null Check
2019-10-10 13:40:16 -04:00
b6987acbbf
Declare the ASAN Functions We Need, Don't Include the Header
2019-10-10 13:40:16 -04:00
0ffae7e440
Stop Allocating Extra Space for Table Redzones
2019-10-10 13:40:16 -04:00
a07037b784
Don't Try to Redzone the Tables
2019-10-10 13:40:16 -04:00
0cc481ef66
Fix Workspace Size Calculation
2019-10-10 13:40:16 -04:00
b6c0a02a17
Fix ZSTD_sizeof_matchState() Calculation
2019-10-10 13:40:16 -04:00
8cffd6ed08
Avoid ASAN Failure in ZSTD_cwksp_free()
2019-10-10 13:40:16 -04:00
ef0b5707c5
Refactor Freeing CCtxes / CDicts Inside Workspaces
2019-10-10 13:40:16 -04:00
143b296cf6
Surround Workspace Allocs with Dead Zone
2019-10-10 13:40:16 -04:00
19a0955ec9
Add `ZSTD_cwksp_alloc_size()` to Help Calculate Needed Workspace Size
2019-10-10 13:40:16 -04:00
da88c35d41
Stop Assuming Tables are Adjacent
2019-10-10 13:40:16 -04:00
35c30d6ca7
Poison Unused Workspace Memory
2019-10-10 13:40:16 -04:00
edb6d884a5
Detect Whether We're Being Compiled with ASAN
2019-10-10 13:40:16 -04:00
dc1fb684bf
Remove Unused MEM_SKIP_MSAN Macro
2019-10-10 13:40:16 -04:00
36528b96c4
Manually moving instead of memcpy on decoder and using genBuffer()
2019-10-03 09:26:51 -07:00
61ec4c2e7f
Cleaning sequence parsing logic
2019-10-03 06:42:40 -07:00
cb18fffe65
enforce C90 compatibility for zlibWrapper
2019-09-24 17:50:58 -07:00
ad2a2785f7
bump version number to v1.4.4
...
so that future reports on `dev` branch use this number instead
2019-09-24 15:15:33 -07:00
c04245b257
Replacing assert with memory_allocation error code throw
2019-09-23 15:42:16 -07:00
be0bebd24e
Adding test and null check for malloc
2019-09-23 15:08:18 -07:00
1ab1a40c9c
Fixed one more place
2019-09-23 21:32:56 +02:00
1f7228c040
Use clz ^ 31 instead of 31 - clz; better codegen for GCC
2019-09-23 21:23:09 +02:00
7451c6578c
Merge pull request #1804 from terrelln/wild-and-fast
...
Optimize (de)compression and fix wildcopy overread
2019-09-21 17:04:36 -07:00
Bimba Shrestha
Nick Terrell
Nick Terrell
Igor Sugak
Yann Collet
Yann Collet
W. Felix Handte
Vincent Torri
Clement Courbet
Sen Huang
moozzyk
Dávid Bolvanský