Commit Graph

3275 Commits (c6548eac8ea6c9c6faf464d7dda9b6ec81123e5d)

Author SHA1 Message Date
Bimba Shrestha 1f681d8592 Merge branch 'oss' of https://github.com/bimbashrestha/zstd into oss 2019-11-27 10:56:54 -08:00
Bimba Shrestha a3a3c62b81 [fuzz] Only set HUF_repeat_valid if loaded table has all non-zero weights (#1898)
Fixes a fuzz issue where dictionary_round_trip failed because the compressor was generating corrupt files thanks to zero weights in the table.

* Only setting loaded dict huf table to valid on non-zero

* Adding hasNoZeroWeights test to fse tables

* Forbiding nbBits != 0 when weight == 0

* Reverting the last commit

* Setting table log to 0 when weight == 0

* Small (invalid) zero weight dict test

* Small (valid) zero weight dict test

* Initializing repeatMode vars to check before zero check

* Removing FSE changes to seperate pr

* Reverting accidentally changed file

* Negating bool, using unsigned, optimization nit
2019-11-26 12:24:19 -08:00
Bimba Shrestha d4e17d0776 Negating bool, updating bool on inner branches 2019-11-26 12:17:43 -08:00
Nick Terrell 718f00ff6f
Optimize decompression speed for gcc and clang (#1892)
* Optimize `ZSTD_decodeSequence()`
* Optimize Huffman decoding
* Optimize `ZSTD_decompressSequences()`
* Delete `ZSTD_decodeSequenceLong()`
2019-11-25 18:26:19 -08:00
Bimba Shrestha 826b555463
Merge branch 'dev' into oss 2019-11-22 17:29:33 -08:00
Bimba Shrestha 10bce1919e Mixed declration fix 2019-11-21 13:08:27 -08:00
Bimba Shrestha 0451accab1 Checking noCompressBlock explicitly for rep code confirmation 2019-11-21 13:06:26 -08:00
Nick Terrell 659e9f05cf Fix null pointer addition 2019-11-20 18:36:04 -08:00
Yann Collet 2d4dcce55f
Merge pull request #1894 from felixhandte/doc-clarify-dctx-reset
Easy: Update Comment on `ZSTD_initDStream()`
2019-11-19 16:18:56 -08:00
Nick Terrell e0d6daabac Fix Appveyor failure 2019-11-19 11:12:26 -08:00
Bimba Shrestha 8f0c2d04c8 Going back to original flow but removing else return 2019-11-19 10:03:07 -08:00
W. Felix Handte 722149cf2b Easy: Update Comment on `ZSTD_initDStream()` 2019-11-19 01:57:15 -05:00
Nick Terrell 6a7f65117e
Merge pull request #1866 from legrosbuffle/dev
Optimized loop bounds to allow the compiler to unroll the loop.
2019-11-18 16:16:30 -08:00
Nick Terrell a839d6852c
Merge pull request #1888 from senhuang42/superblocks_fixed
RLE test and re-enable RLE in main compression loop
2019-11-18 16:09:33 -08:00
Bimba Shrestha 80586f5e80 Reversing condition order and forwarding error 2019-11-18 13:53:55 -08:00
Bimba Shrestha dade64428f Output regular uncompressed block when compressSequences fails 2019-11-18 08:43:14 -08:00
Bimba Shrestha 2d5d961a60 Typo in comment 2019-11-15 19:00:53 -08:00
Bimba Shrestha dba767c0bb Leaving room for checksum 2019-11-15 18:44:51 -08:00
Vincent Torri 6b5c10b48c shared library: rename import library with .dll.a extension
mort of open source project are using this extension for the import library.
The Win32 linker is supporting this extension, see
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/win32.html
section "direct linking to a dll"
2019-11-15 19:46:06 +01:00
Clement Courbet b3c9fc27b4 Optimized loop bounds to allow the compiler to unroll the loop.
This has no measurable impact on large files but improves small file
decompression by ~1-2% for 10kB, benchmarked with:

head -c 10000 silesia.tar > /tmp/test
make CC=/usr/local/bin/clang-9 BUILD_STATIC=1 && ./lzbench -ezstd -t1,5 /tmp/test
2019-11-15 08:27:05 +01:00
Sen Huang d9646dcbb5 Fixed main compression logic changes 2019-11-14 19:39:09 -05:00
Yann Collet 4b1ac69f19
Merge pull request #1868 from senhuang42/superblocks_fixed
Superblocks rebased for merge
2019-11-14 13:31:34 -08:00
Sen Huang c26d32c91c Change superblock #include to be last 2019-11-14 13:12:17 -05:00
Yann Collet d67742bc5d
Merge pull request #1858 from senhuang42/dictionary_header_size
Method to get dictionary header size
2019-11-14 09:44:07 -08:00
Sen Huang c85d10d0ea Remove mixed declarations 2019-11-08 13:57:26 -05:00
Sen Huang d9c475f3b3 Fix static analyze error, use proper bounds for dictEnd 2019-11-08 13:57:26 -05:00
Sen Huang d06b90692b Move asserts to loadZstdDictionary() 2019-11-08 13:57:26 -05:00
Sen Huang b39149e156 Expose ZSTD_reset_compressedBlockState() to shared API 2019-11-08 13:57:26 -05:00
Sen Huang 6ce335371b Add error forwarding to loadCEntropy(), make check for dictSize >= 8 from bad merge 2019-11-08 13:57:26 -05:00
Sen Huang 4a61aaf368 Remove redundant comment 2019-11-08 13:57:26 -05:00
Sen Huang c787b351ea Use ZSTD Error codes, improve explanation of ZSTD_loadCEntropy() and ZSTD_loadDEntropy() 2019-11-08 13:57:26 -05:00
Sen Huang 04fb42b4f3 Integrated refactor into getDictHeaderSize, now passes tests 2019-11-08 13:57:26 -05:00
Sen Huang 0bcaf6db08 First working pass at refactor of loadZstdDictionary() 2019-11-08 13:57:26 -05:00
Sen Huang 4b141b63e0 Revert "Move decompress symbols into zstd_internal.h, remove dependency"
This reverts commit a152b4c67a5266f611db4a2eac4a79003852a795.
2019-11-08 13:57:26 -05:00
Sen Huang 84404cff6e Move decompress symbols into zstd_internal.h, remove dependency 2019-11-08 13:57:26 -05:00
Sen Huang 341e0641ed Checks malloc() for failure, returns 0 if so 2019-11-08 13:57:26 -05:00
Sen Huang 97b7f712f3 Change to heap allocation, remove implicit type conversion 2019-11-08 13:57:25 -05:00
Sen Huang 3c36a7f13a Add ZDICT_getHeaderSize() 2019-11-08 13:57:08 -05:00
Nick Terrell 8c474f9845 Fix parameter selection and adjustment with srcSize == 0 2019-11-07 08:58:43 -08:00
Felix Handte 5688447758
Merge pull request #1873 from felixhandte/make-overlap-log-multithread-only
Fix #1861: Restrict overlapLog Parameter When Not Built With Multithreading
2019-11-06 16:56:37 -05:00
Felix Handte ba4613602f
Merge pull request #1843 from moozzyk/issue-1637
Take ZSTD_parameters as a const pointer
2019-11-06 16:56:14 -05:00
W. Felix Handte c13f81905a Fix #1861: Restrict overlapLog Parameter When Not Built With Multithreading
This parameter is unused in single-threaded compression. We should make it
behave like the other multithread-only parameters, for which we only accept
zero when we are not built with multithreading.
2019-11-06 16:05:02 -05:00
Sen Huang 13bb7500e8 Fix frame argument to compression 2019-11-05 16:15:55 -05:00
Sen Huang f2932fb5eb Fix more merge conflicts 2019-11-05 15:54:05 -05:00
Sen Huang 7ce891870c Fix merge conflicts 2019-11-05 15:51:25 -05:00
Bimba Shrestha 3fb5b106da Replacing some literals with constants 2019-11-05 10:26:57 -08:00
Nick Terrell 60205fec02 Fix 2 bugs in dictionary loading
* Silently skip dictionaries less than 8 bytes, unless using `ZSTD_dct_fullDict`.
  This changes the compressor, which silently skips dictionaries <= 8 bytes.
* Allow repcodes that are equal to the dictionary content size, since it is in bounds.
2019-11-01 16:52:07 -07:00
Sen Huang b9ede1c8c2 Make sure contentsize is known 2019-10-30 16:03:58 -04:00
Nick Terrell 9c1860861e Fix assert in ZSTD_safecopy
In the case that `op >= oend_w` it is possible that `diff < 8` because
the two buffers could be adjacent.

Credit to OSS-Fuzz, which found the bug. It isn't reproducible because
it depends on the memory layout.
2019-10-28 17:51:17 -07:00
Felix Handte 01ec595b85
Merge pull request #1851 from felixhandte/pkg-config-prefix-fix
In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time
2019-10-28 14:24:56 -04:00
Yann Collet 74065da4c5 updated API inline doc and manual
regarding ZSTD_CDict created without a dictBuffer.
2019-10-28 11:15:41 -07:00
W. Felix Handte 74bd76c3ff In pkg-config File, Derive Lib and Include Dir from Prefix at Use-Time
Addresses #1794. Instead of deriving the lib dir and include dir at
build-time, let's do it like everyone else does at pkg-config run-time.

This has the disadvantage that we can no longer override LIBDIR and
INCLUDEDIR in the Makefile and have that reflected in the .pc file.
2019-10-25 15:07:31 -04:00
Yann Collet c2140e9db0
Merge pull request #1845 from facebook/zbuff
improve deprecation warning macro
2019-10-25 09:59:00 -07:00
Yann Collet a9a216a846
Merge pull request #1824 from senhuang42/new_path_for_cdict
Avoid using CDict params when input is large.
2019-10-23 12:04:40 -07:00
Yann Collet 63e435dda1 improve deprecation warning macro
fix #1488

although, curiously enough, I was never able to reproduce the issue
(according to the bug report, it should be present while using gcc 4.8).
2019-10-23 11:59:32 -07:00
moozzyk eda7946a36 Take ZSTD_parameters as a const pointer
Fixes: #1637
2019-10-22 23:21:54 -07:00
Yann Collet f966cd080a added documentation on DYNAMIC_BMI2 build macro 2019-10-22 17:43:09 -07:00
Yann Collet 5d5c895b18 fix initCStream_advanced() for fast strategies
Compression ratio of fast strategies (levels 1 & 2)
was seriously reduced, due to accidental disabling of Literals compression.

Credit to @QrczakMK, which perfectly described the issue, and implementation details,
making the fix straightforward.

Example : initCStream with level 1 on synthetic sample P50 :
Before : 5,273,976 bytes
After  : 3,154,678 bytes
ZSTD_compress (for comparison) : 3,154,550

Fix #1787.

To follow : refactor the test which was supposed to catch this issue (and failed)
2019-10-22 15:01:38 -07:00
Yann Collet 111b0c53b0 update documentation on deprecated functions
mostly : note that these functions will soon generate deprecation warnings
2019-10-22 13:51:18 -07:00
Nick Terrell b1ec94e63c Fix ZSTD_f_zstd1_magicless for small data
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
  take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
  taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.

Fixes #1813.
2019-10-21 21:16:17 -07:00
Sen Huang c2e1e54f24 ((x or y) or z) == (x or y or z), remove brackets 2019-10-21 19:16:50 -04:00
Sen Huang 59c81aa31b Line up comments :) 2019-10-21 19:12:15 -04:00
Sen Huang dbda8c318a Trailing comma 2019-10-21 19:10:13 -04:00
Sen Huang 0c00455ea6 Merge branch 'dev' of github.com:senhuang42/zstd into new_path_for_cdict 2019-10-21 19:06:51 -04:00
Sen Huang 5b2f4ac1a8 merge 2019-10-21 19:02:52 -04:00
Sen Huang 2ab484a5f9 Fix bad merge 2019-10-21 18:55:17 -04:00
Nick Terrell 919d1d8e93
Merge pull request #1831 from terrelln/zstdmt-bad-memset
[zstdmt] Don't memset the jobDescription
2019-10-21 15:53:57 -07:00
Sen Huang b6c3459d50 merge 2019-10-21 18:46:17 -04:00
Yann Collet 6cf04c0344
Merge pull request #1834 from facebook/winFix
Windows fixes
2019-10-21 13:45:17 -07:00
Sen Huang 676f89902a Added multiplier, renamed new enum to something more useful 2019-10-21 15:36:12 -04:00
Sen Huang 1f3a51fb52 Updated forceAttachDict param bounds 2019-10-21 15:36:12 -04:00
Sen Huang 8f69c47643 Add enum to decision process 2019-10-21 15:36:12 -04:00
Sen Huang e4de8b098a Added support for forcing new CDict behavior and updated enum 2019-10-21 15:36:12 -04:00
Sen Huang 9294f4826b Changed to int from BYTE 2019-10-21 15:36:12 -04:00
Sen Huang f0fccc8847 Changed to int from BYTE 2019-10-21 15:36:12 -04:00
Sen Huang bb2df8c499 Trailing whitespace 2019-10-21 15:36:12 -04:00
Sen Huang cf51501d2f Fix test 2019-10-21 15:36:12 -04:00
Sen Huang ea3cb6988f Cast to BYTE to appease appveyor 2019-10-21 15:36:12 -04:00
Sen Huang a727a85a7e merge conflicts round 2 2019-10-21 15:36:12 -04:00
Sen Huang 053a35fd64 formatting 2019-10-21 15:35:33 -04:00
Sen Huang 3fa4daaa55 Fix error 2019-10-21 15:35:33 -04:00
Sen Huang 3328348c63 Add compressionlevel to cdict 2019-10-21 15:32:39 -04:00
Felix Handte cf725630a6
Merge pull request #1795 from felixhandte/workspace-asan
Add Poisoned Redzones to the Workspace When Compiling with ASAN
2019-10-21 12:15:17 -04:00
Sen Huang e8aa3e486d Updated forceAttachDict param bounds 2019-10-20 22:01:08 -04:00
Sen Huang 6d297265f9 Add enum to decision process 2019-10-20 19:02:47 -04:00
Sen Huang 1daa898c93 Added support for forcing new CDict behavior and updated enum 2019-10-20 14:03:09 -04:00
Nick Terrell 0bc39bc3a0 [zstdmt] Don't memset the jobDescription 2019-10-18 15:05:51 -07:00
Nick Terrell 243824551f [threading] Add debug utilities 2019-10-18 15:05:34 -07:00
Yann Collet 1795133c45 refactored FIO_compressMultipleFilenames() prototype
for consistency
2019-10-17 15:32:03 -07:00
Yann Collet 6446ffb277
Merge pull request #1827 from facebook/dm_Dct
updated erroneous comments using ZSTD_dm_*
2019-10-17 10:30:58 -07:00
Yann Collet 19741c7d99
Merge pull request #1815 from facebook/zlibwrap
make zlibWrapper strict ISO-C90 compatible
2019-10-16 16:45:15 -07:00
Yann Collet 6323966e53 updated erroneous comments using ZSTD_dm_*
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822)
2019-10-16 16:14:04 -07:00
Yann Collet 2d5201b0ab removed wildcopy8()
which is no longer used,
noticed by @davidbolvansky
2019-10-16 14:51:33 -07:00
Sen Huang 4455f00cb8 Changed to int from BYTE 2019-10-16 15:06:02 -04:00
Sen Huang 4f7d26b0ee Changed to int from BYTE 2019-10-16 15:05:29 -04:00
Sen Huang cf00ea367a Trailing whitespace 2019-10-16 10:31:27 -04:00
Sen Huang 8cb2174446 Fix test 2019-10-16 10:29:31 -04:00
Sen Huang 5e901b6f32 Cast to BYTE to appease appveyor 2019-10-15 13:58:44 -04:00
Sen Huang 5c010c9d2d merge conflicts round 2 2019-10-15 13:10:05 -04:00
Sen Huang a06b51879c merge conflict 2019-10-15 12:58:50 -04:00
Sen Huang 23dac23a49 formatting 2019-10-15 12:44:48 -04:00
Sen Huang 0c8df5c928 Fix error 2019-10-15 12:28:23 -04:00
Sen Huang a65eb39f9d Add compressionlevel to cdict 2019-10-15 10:22:06 -04:00
Yann Collet fb77afc626
Merge pull request #1760 from bimbashrestha/extract_sequences_api
Adding api for extracting sequences from seqstore
2019-10-10 13:11:18 -07:00
W. Felix Handte ede31da2ea Fix CCtx Size Estimation 2019-10-10 15:02:08 -04:00
W. Felix Handte bd6a20b8a0 Expand Default Redzone Size 2019-10-10 13:45:55 -04:00
W. Felix Handte 2c80a9f8ac Check if CCtx in Workspace after Null Check 2019-10-10 13:40:16 -04:00
W. Felix Handte b6987acbbf Declare the ASAN Functions We Need, Don't Include the Header 2019-10-10 13:40:16 -04:00
W. Felix Handte 0ffae7e440 Stop Allocating Extra Space for Table Redzones 2019-10-10 13:40:16 -04:00
W. Felix Handte a07037b784 Don't Try to Redzone the Tables 2019-10-10 13:40:16 -04:00
W. Felix Handte 0cc481ef66 Fix Workspace Size Calculation 2019-10-10 13:40:16 -04:00
W. Felix Handte b6c0a02a17 Fix ZSTD_sizeof_matchState() Calculation 2019-10-10 13:40:16 -04:00
W. Felix Handte 8cffd6ed08 Avoid ASAN Failure in ZSTD_cwksp_free() 2019-10-10 13:40:16 -04:00
W. Felix Handte ef0b5707c5 Refactor Freeing CCtxes / CDicts Inside Workspaces 2019-10-10 13:40:16 -04:00
W. Felix Handte 143b296cf6 Surround Workspace Allocs with Dead Zone 2019-10-10 13:40:16 -04:00
W. Felix Handte 19a0955ec9 Add `ZSTD_cwksp_alloc_size()` to Help Calculate Needed Workspace Size 2019-10-10 13:40:16 -04:00
W. Felix Handte da88c35d41 Stop Assuming Tables are Adjacent 2019-10-10 13:40:16 -04:00
W. Felix Handte 35c30d6ca7 Poison Unused Workspace Memory 2019-10-10 13:40:16 -04:00
W. Felix Handte edb6d884a5 Detect Whether We're Being Compiled with ASAN 2019-10-10 13:40:16 -04:00
W. Felix Handte dc1fb684bf Remove Unused MEM_SKIP_MSAN Macro 2019-10-10 13:40:16 -04:00
Bimba Shrestha 36528b96c4 Manually moving instead of memcpy on decoder and using genBuffer() 2019-10-03 09:26:51 -07:00
Bimba Shrestha 61ec4c2e7f Cleaning sequence parsing logic 2019-10-03 06:42:40 -07:00
Yann Collet cb18fffe65 enforce C90 compatibility for zlibWrapper 2019-09-24 17:50:58 -07:00
Yann Collet ad2a2785f7 bump version number to v1.4.4
so that future reports on `dev` branch use this number instead
2019-09-24 15:15:33 -07:00
Bimba Shrestha c04245b257 Replacing assert with memory_allocation error code throw 2019-09-23 15:42:16 -07:00
Bimba Shrestha be0bebd24e Adding test and null check for malloc 2019-09-23 15:08:18 -07:00
Dávid Bolvanský 1ab1a40c9c Fixed one more place 2019-09-23 21:32:56 +02:00
Dávid Bolvanský 1f7228c040 Use clz ^ 31 instead of 31 - clz; better codegen for GCC 2019-09-23 21:23:09 +02:00
Nick Terrell 7451c6578c
Merge pull request #1804 from terrelln/wild-and-fast
Optimize (de)compression and fix wildcopy overread
2019-09-21 17:04:36 -07:00
Nick Terrell 5cb7615f1f Add UNUSED_ATTR to ZSTD_storeSeq() 2019-09-20 21:37:13 -07:00
Nick Terrell 5dc0a1d659 HINT_INLINE ZSTD_storeSeq()
Clang on Mac wasn't inlining `ZSTD_storeSeq()` in level 1, which was
causing a 5% performance regression. This fixes it.
2019-09-20 16:39:27 -07:00
Bimba Shrestha f3c4fd17e3 Passing in dummy dst buffer of compressbound(srcSize) 2019-09-20 15:50:58 -07:00
Felix Handte c047fcf7bf
Merge pull request #1806 from felixhandte/estimate-cctx-doc
Update Comment on `ZSTD_estimateCCtxSize()`
2019-09-20 15:36:00 -04:00
Nick Terrell 44c65da97e Remove literals overread in ZSTD_storeSeq() for ~neutral perf 2019-09-20 12:23:25 -07:00
W. Felix Handte f7d9b36835 Update Comment on `ZSTD_estimateCCtxSize()` 2019-09-20 14:11:29 -04:00
Nick Terrell fde217df04 Fix bounds check in ZSTD_storeSeq() 2019-09-20 08:25:12 -07:00
Nick Terrell 67b1f5fc72 Fix too strict assert 2019-09-20 01:23:35 -07:00
Nick Terrell ddab2a94e8 Pass iend into ZSTD_storeSeq() to allow ZSTD_wildcopy() 2019-09-20 00:56:20 -07:00
Nick Terrell cdad7fa512 Widen ZSTD_wildcopy to 32 bytes 2019-09-20 00:52:15 -07:00
Nick Terrell efd37a64ea Optimize decompression and fix wildcopy overread
* Bump `WILDCOPY_OVERLENGTH` to 16 to fix the wildcopy overread.
* Optimize `ZSTD_wildcopy()` by removing unnecessary branches and
  unrolling the loop.
* Extract `ZSTD_overlapCopy8()` into its own function.
* Add `ZSTD_safecopy()` for `ZSTD_execSequenceEnd()`. It is
  optimized for single long sequences, since that is the important
  case that can end up in `ZSTD_execSequenceEnd()`. Without this
  optimization, decompressing a block with 1 long match goes
  from 5.7 GB/s to 800 MB/s.
* Refactor `ZSTD_execSequenceEnd()`.
* Increase the literal copy shortcut to 16.
* Add a shortcut for offset >= 16.
* Simplify `ZSTD_execSequence()` by pushing more cases into
  `ZSTD_execSequenceEnd()`.
* Delete `ZSTD_execSequenceLong()` since it is exactly the
  same as `ZSTD_execSequence()`.

clang-8 seeds +17.5% on silesia and +21.8% on enwik8.
gcc-9 sees +12% on silesia and +15.5% on enwik8.

TODO: More detailed measurements, and on more datasets.

Crdit to OSS-Fuzz for finding the wildcopy overread.
2019-09-19 21:07:14 -07:00
Bimba Shrestha ae6d0e64ae Addressing comments 2019-09-19 15:25:20 -07:00
Yann Collet 3cac061db5
Merge pull request #1802 from bimbashrestha/rle_block_bound_fix_pt2
Adding 4 blocks to FSE_BLOCKBOUND() in lib/common (different from las…
2019-09-18 16:32:37 -07:00
Bimba Shrestha 6e9f6813bb adding bit container size 2019-09-18 13:49:45 -07:00
Bimba Shrestha f9b6abb896 Adding 4 blocks to FSE_BLOCKBOUND() in lib/common (different from last week) 2019-09-18 13:29:05 -07:00
Yann Collet bfff5b30a4
Merge pull request #1756 from mgrice/dev
Improvements in zstd decode performance
2019-09-18 11:35:50 -07:00
Yann Collet 243200e5bf minor refactor of ZSTD_fast
- reduced variables lifetime
- more accurate code comments
2019-09-17 14:02:57 -07:00
Bimba Shrestha 76fea3fb99 Resolving appveyor test failure implicit conversion 2019-09-16 14:02:23 -07:00
Bimba Shrestha a874435478
Merge branch 'dev' into extract_sequences_api 2019-09-16 13:29:59 -07:00
Felix Handte 2164a130f3
Merge pull request #1780 from felixhandte/workspace-efficiency-3
Avoid Clearing Tables Even When Changing CParams
2019-09-16 14:37:05 -04:00
W. Felix Handte 72ea79cacd Don't Include `sanitizer/msan_interface.h`, Since Not All Platforms Provide It
Instead, explicitly declare the functions we use.
2019-09-16 12:08:03 -04:00
Bimba Shrestha bff6072e3a Bailing early when collecting sequences and documentation 2019-09-16 08:26:21 -07:00
Nick Terrell fbeaf6989e [libzstd] Improve advanced API docs 2019-09-15 12:41:24 -07:00
Yann Collet 09b1844d9b
Merge pull request #1784 from bimbashrestha/fse_block_bound_err
Rearranging assert and allowing 4 extra for FSE_BLOCKBOUND()
2019-09-12 19:09:27 -07:00
Bimba Shrestha fe9af338ed Added assert to BIT_flushBits() 2019-09-12 15:35:27 -07:00
Bimba Shrestha 43da5bf27e Rearranging assert and allowing 4 extra for FSE_BLOCKBOUND() 2019-09-12 14:43:50 -07:00
W. Felix Handte 20c69077d1 Shrink Table Valid End During Alloc Alignment / Phase Change 2019-09-11 17:14:59 -04:00
W. Felix Handte 51d90668ba Add Assertions to Confirm that Workspace Pointers are Correctly Ordered 2019-09-11 17:14:59 -04:00
W. Felix Handte a10c191613 `__msan_poison()` Workspace When Preparing for Re-Use 2019-09-11 17:14:45 -04:00
W. Felix Handte 7c57e2b9ca Zero `h3size` When `h3log` is 0
This led to a nasty edgecase, where index reduction for modes that don't use
the h3 table would have a degenerate table (size 4) allocated and marked clean,
but which would not be re-indexed.
2019-09-11 13:14:26 -04:00
W. Felix Handte bc020eec92 Also Shrink Clean Table Area When Reducing Indices 2019-09-11 11:40:57 -04:00
W. Felix Handte 1999b2ed9b Update DEBUGLOG Statements 2019-09-11 11:21:00 -04:00
W. Felix Handte 13e29a56de Shrink Clean Table Area When Copying Table Contents into Context
The source matchState is potentially at a lower current index, which means
that any extra table space not overwritten by the copy may now contain
invalid indices. The simple solution is to unconditionally shrink the valid
table area to just the area overwritten.
2019-09-11 11:18:45 -04:00
W. Felix Handte edb3ad053e Comments 2019-09-10 18:25:45 -04:00
W. Felix Handte f31ef28ff8 Only Reset Indexing in `ZSTD_resetCCtx_internal()` When Necessary 2019-09-10 18:25:45 -04:00
W. Felix Handte 9968a53e91 Remove No-Longer-Used Continuation Functions 2019-09-10 18:25:45 -04:00
W. Felix Handte 1b28e80416 Remove Fast Continue Path in `ZSTD_resetCCtx_internal()` 2019-09-10 18:25:45 -04:00
W. Felix Handte ad16eda5e4 `ZSTD_reset_matchState` Optionally Doesn't Restart Indexing 2019-09-10 18:25:45 -04:00
W. Felix Handte 5b10bb5ec3 Rename `ZSTD_compResetPolicy_e` Values and Add Comment 2019-09-10 18:25:45 -04:00
W. Felix Handte 0492b9a9ec Accept `ZSTD_indexResetPolicy_e` Param in `ZSTD_reset_matchState()` 2019-09-10 18:25:45 -04:00
W. Felix Handte 14c5471d5e Introduce `ZSTD_indexResetPolicy_e` Enum 2019-09-10 18:25:45 -04:00
W. Felix Handte 17b6da2e0f Track Usable Table Space in Compression Workspace 2019-09-10 18:25:37 -04:00
Yann Collet 22bd158e0f
Merge pull request #1712 from felixhandte/workspace-efficiency-2
Allocate Internal Buffers via Workspace Abstraction
2019-09-10 15:20:29 -07:00
Bimba Shrestha 1407919d13 Addressing comments on parsing 2019-09-10 15:10:50 -07:00
Bimba Shrestha 47199480da Cleaning up parsing per suggestion 2019-09-10 13:18:59 -07:00
W. Felix Handte a9d373f093 Remove Empty lib/compress/zstd_cwksp.c 2019-09-10 16:03:13 -04:00
Yann Collet 5ba495b622
Merge pull request #1775 from facebook/edufix
fix educational decoder
2019-09-10 12:12:08 -07:00
Yann Collet 41416f0927
Merge pull request #1773 from bimbashrestha/rle_first_block_decompression_fix
Removing redundant condition in decompression, making first block rle…
2019-09-10 11:17:29 -07:00
Bimba Shrestha e3c5825918 Fizing litLength == 0 case 2019-09-10 10:38:13 -07:00
Bimba Shrestha 9e7bb55e14 Addressing comments 2019-09-09 20:04:46 -07:00
W. Felix Handte 81208fd7c2 Forward Declare `ZSTD_cwksp_available_space` to Fix Build 2019-09-09 19:10:09 -04:00
W. Felix Handte 91bf1babd1 Inline Workspace Functions 2019-09-09 18:53:53 -04:00
W. Felix Handte 0db3ffe7ee Forward resetCCtx Errors when Using CDict 2019-09-09 16:47:19 -04:00
W. Felix Handte eb6f69d978 Fix sizeof_CCtx and sizeof_CDict Calculations for Statically Init'ed Objects 2019-09-09 16:45:17 -04:00
W. Felix Handte e3703825a8 Fix workspaceTooSmall Calculation 2019-09-09 15:12:14 -04:00
W. Felix Handte 0a65a67901 Shorten `&zc->workspace` -> `ws` in `ZSTD_resetCCtx_internal()` 2019-09-09 14:59:09 -04:00
W. Felix Handte 1120e4d962 Clean Up TODOs and Comments pt. II 2019-09-09 14:04:39 -04:00
W. Felix Handte c60e1c3be5 Nit 2019-09-09 13:34:08 -04:00
W. Felix Handte 7d7b665c90 Pull Phase Advance Logic Out into Internal Function 2019-09-09 13:34:08 -04:00
W. Felix Handte 8549ae9f1d Hide Workspace Movement Behind Helper Function 2019-09-09 13:34:08 -04:00
W. Felix Handte 2405c03bcd Fix DEBUGLOG Statement Levels 2019-09-09 13:34:08 -04:00
W. Felix Handte 7100d24221 Fix Rescale Continue Special Case 2019-09-09 13:34:08 -04:00
W. Felix Handte 7321e4c9f3 Remove Unused noRealloc CRP Value 2019-09-09 13:34:08 -04:00
W. Felix Handte 901bba4ca6 Re-Implement Workspace Shrinking when Oversized 2019-09-09 13:34:08 -04:00
W. Felix Handte 881bcd80ca Cleanup from Move 2019-09-09 13:34:08 -04:00
W. Felix Handte b511a84adc Move Workspace Functions to Their Own File 2019-09-09 13:34:08 -04:00
W. Felix Handte 077a2d7dc9 Rename 2019-09-09 13:34:08 -04:00
W. Felix Handte ebd162194f Clean Up TODOs and Comments 2019-09-09 13:34:08 -04:00
W. Felix Handte 2abe0145b1 Improve Comments a Bit 2019-09-09 13:34:08 -04:00
W. Felix Handte 7a2416a863 Allocate CDict in Workspace (Rather than in Separate Allocation) 2019-09-09 13:34:08 -04:00
W. Felix Handte 65057cf009 Rewrite ZSTD_initStaticCCtx to Alloc CCtx in Workspace 2019-09-09 13:34:08 -04:00