Commit Graph

5887 Commits (b7421f8e6994fc8317b50993a0b5ccf5a843395a)

Author SHA1 Message Date
W. Felix Handte 01ff945eae Split Attach and Copy Reset Strategies into Separate Implementation Functions 2018-09-28 17:12:54 -07:00
W. Felix Handte a6d6bbeae1 Pull Attachment Decision into Separate Function 2018-09-28 17:12:54 -07:00
W. Felix Handte b7fba599ae And Then Avoid the Unused Parameter Warning 2018-09-28 17:12:54 -07:00
W. Felix Handte 1f188ae655 Move Asserts into Function to Avoid Unused Function Warning 2018-09-28 17:12:54 -07:00
W. Felix Handte 7212b5e5c2 Move Match State CParams Setting into `resetCCtx` and `continueCCtx` 2018-09-28 17:12:54 -07:00
W. Felix Handte 01e34d365b Strengthen Assertion to Assert Equality 2018-09-28 17:12:53 -07:00
W. Felix Handte 50cc1cf4d5 Remove CParams Arg from ZSTD_ldm_blockCompress 2018-09-28 17:12:53 -07:00
W. Felix Handte 14764de49f Stop Separately Passing CParams in ZSTD_lazy Internal Functions 2018-09-28 17:12:53 -07:00
W. Felix Handte 97149f22c3 Stop Separately Passing CParams in ZSTD_opt Internal Functions 2018-09-28 17:10:42 -07:00
W. Felix Handte dcdf437fed Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte 3483f89101 Also Assert Equivalency When Filling MatchState with Prefix 2018-09-28 17:10:42 -07:00
W. Felix Handte 6cb2454646 Remove CParams from Block Compressor Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte 03103269de Assert `ctx` and `ms` cparams Equivalency 2018-09-28 17:10:42 -07:00
W. Felix Handte 4e3ecee9ed Remove cParams from CDict 2018-09-28 17:10:42 -07:00
W. Felix Handte 76ef87ed9d Add ZSTD_compressionParameters to ZSTD_matchState_t 2018-09-28 17:10:42 -07:00
Yann Collet ec1cb8e996 changed macro name
from EXIT_IF() to RETURN_IF()

EXIT could be misunderstood as exit(), which terminates program execution.
But the macro only leaves the function, not the program.
2018-09-28 16:04:00 -07:00
Yann Collet 05c0a072b7 minor improvement in the multi-format suffix selection 2018-09-28 15:57:35 -07:00
Nick Terrell 5aa9a1dd2d Fix minigzip in std=c99 mode 2018-09-28 15:52:25 -07:00
Nick Terrell 09231dad4d [util] Fix lstat feature test macro 2018-09-28 15:23:09 -07:00
Nick Terrell 43146d8a16 Add -Werror to *build rules 2018-09-28 14:59:40 -07:00
Nick Terrell eb4423e7ed Fix another warning 2018-09-28 14:24:38 -07:00
Nick Terrell 0e7a7f1def Fix warnings 2018-09-28 12:14:24 -07:00
Nick Terrell 6391cd1030 [zstd] Fix newly added test case 2018-09-28 12:09:28 -07:00
Nick Terrell 146049a1ea [zstreamtest] Add failing test case 2018-09-28 12:09:14 -07:00
Yann Collet d987ab5983 fixed unreachable section warning on Visual 2018-09-28 09:34:16 -07:00
Nick Terrell e06f91a169
Merge pull request #1343 from terrelln/faster-ci
[zstreamtest] Reduce memory of newapi tests
2018-09-27 19:29:02 -07:00
Yann Collet 913a0365b6 Merge branch 'dev' into donotdelete 2018-09-27 19:23:55 -07:00
Yann Collet 8b86004cff
Merge pull request #1345 from facebook/fixLegacy
fixed legacy compilation tests
2018-09-27 19:23:27 -07:00
Yann Collet ef1272737b fixed minor Visual conversion warnings 2018-09-27 18:29:15 -07:00
Yann Collet ff36513556 fixed longmatch test too 2018-09-27 18:24:41 -07:00
Yann Collet 73773c6b6a fixed legacy compilation tests
for some reason, these tests started failing recently on CircleCI
2018-09-27 18:15:14 -07:00
Yann Collet 7318753d47
Merge pull request #1344 from terrelln/grub-fixes
Fixes for GRUB
2018-09-27 17:36:54 -07:00
Yann Collet 9b45db7fa6 minor refactoring of --list
trying to reduce recurrent patterns.
2018-09-27 16:49:08 -07:00
Nick Terrell a180ea07c4 Restore ZSTD_noCompressBlock() for clarity 2018-09-27 16:06:02 -07:00
Nick Terrell d8c73cd607 Reset number of threads less often 2018-09-27 15:49:31 -07:00
Nick Terrell aec1a3ec58 Change byte to value to avoid a GRUB typedef 2018-09-27 15:24:48 -07:00
Nick Terrell 109bd37474 Include stddef.h for size_t 2018-09-27 15:24:48 -07:00
Nick Terrell f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
Nick Terrell 7ee910e86b More aggressive limitations 2018-09-27 13:55:24 -07:00
Nick Terrell ca0cfa3dbd [zstreamtest] Reduce memory of newapi tests
We could allocate up to 2^28 bytes of memory when using 2 threads with
window log = 24. Now, we limit it to 2^26 bytes of memory when not running
big tests.

I chose max window log = 22 since that is the maximum source size when
big tests are disabled. Hopefully this will be enough to reduce or
eliminate the test failures.
2018-09-27 12:48:29 -07:00
Yann Collet 2a5cd8535a
Merge pull request #1342 from facebook/fixcatyd
fix : huge (>4GB) chain of blocks
2018-09-27 10:20:14 -07:00
Yann Collet 404a7bfed0 moved again overflow correction
cannot work from within ZSTD_compressBlock()
2018-09-26 18:06:53 -07:00
Yann Collet 9bce916732
Merge pull request #1340 from terrelln/generate-artifacts
Publish artifacts with CircleCI
2018-09-26 15:52:23 -07:00
Yann Collet 0e2dbac18a changed overflow correction place
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
2018-09-26 15:35:38 -07:00
Yann Collet e74eade251
Merge pull request #1339 from facebook/grep_colors
fixed usage of grep in Makefile
2018-09-26 14:39:20 -07:00
Yann Collet 8883af6a1e
Merge pull request #1327 from facebook/adapt
Adaptive compression
2018-09-26 14:39:08 -07:00
Yann Collet f98c69d77c fix : huge (>4GB) stream of blocks
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
2018-09-26 14:24:28 -07:00
Yann Collet 3dae90c4e0
Merge pull request #1338 from facebook/rfc
updated format documentation
2018-09-26 13:40:35 -07:00
Yann Collet 65ed6eeefb
Merge pull request #1337 from facebook/test_failure
fixed `!` tests
2018-09-26 13:40:20 -07:00
Yann Collet 8ff17a6a09
Merge pull request #1329 from facebook/v04isout
Changed default legacy support to v0.5+
2018-09-26 13:39:05 -07:00