Commit Graph

637 Commits (dce386f658bac7ddb35d279bc7a929ee7da9b0cd)

Author SHA1 Message Date
Nick Terrell 8adebbd0f8 Add test case for PR #972 2018-01-03 15:52:18 -08:00
Yann Collet cafedcbbe4 ZSTD_resetCCtx_internal: fixed order of arguments
params1 was swapped with params2.
This used to be a non-issue when testing for strict equality,
but now that some tests look for "sufficient size" `<=`, order matters.
2017-12-19 21:49:04 +01:00
Yann Collet 574e75354b fuzzer: ensure existence of CHECK_Z macro beyond OS-X systems 2017-12-19 11:24:14 +01:00
Yann Collet d88c671663 added test case for "wrong blockSize in continue mode" 2017-12-19 10:16:09 +01:00
Yann Collet a0e0985d38 added test on small file
on top of test on small stream
2017-12-14 13:32:24 -08:00
Yann Collet 281f06e01f saves 3-bytes on small input with streaming API
zstd streaming API was adding a null-block at end of frame for small input.

Reason is : on small input, a single block is enough.
ZSTD_CStream would size its input buffer to expect a single block of this size,
automatically triggering a flush on reaching this size.

Unfortunately, that last byte was generally received before the "end" directive (at least in `fileio`).
The later "end" directive would force the creation of a 3-bytes last block to indicate end of frame.

The solution is to not flush automatically, which is btw the expected behavior.
It happens in this case because blocksize is defined with exactly the same size as input.
Just adding one-byte is enough to stop triggering the automatic flush.

I initially looked at another solution, solving the problem directly in the compression context.
But it felt awkward.
Now, the underlying compression API `ZSTD_compressContinue()` would take the decision the close a frame
on reaching its expected end (`pledgedSrcSize`).
This feels awkward, a responsability over-reach, beyond the definition of this API.
ZSTD_compressContinue() is clearly documented as a guaranteed flush,
with ZSTD_compressEnd() generating a guaranteed end.

I faced similar issue when trying to port a similar mechanism at the higher streaming layer.
Having ZSTD_CStream end a frame automatically on reaching `pledgedSrcSize` can surprise the caller,
since it did not explicitly requested an end of frame.
The only sensible action remaining after that is to end the frame with no additional input.
This adds additional logic in the ZSTD_CStream state to check this condition.
Plus some potential confusion on the meaning of ZSTD_endStream() with no additional input (ending confirmation ? new 0-size frame ?)

In the end, just enlarging input buffer by 1 byte feels the least intrusive change.
It's also a contract remaining inside the streaming layer, so the logic is contained in this part of the code.

The patch also introduces a new test checking that size of small frame is as expected, without additional 3-bytes null block.
2017-12-14 11:47:02 -08:00
Yann Collet 5b2ce2c043
Merge pull request #946 from terrelln/r-o
Allow -o with multiple files
2017-12-14 10:02:05 -08:00
Yann Collet c005df136f
Merge pull request #947 from facebook/fix944
Fix #944
2017-12-14 10:01:52 -08:00
Nick Terrell 4680e85bdf Allow -o with multiple files 2017-12-13 17:44:34 -08:00
Yann Collet 4d0dfafa7b
Merge pull request #949 from terrelln/rrm
[fileio] Refuse to remove non-regular file
2017-12-13 17:36:39 -08:00
Yann Collet d23eb9a098 zstreamtest : added missing CHECK_Z() 2017-12-13 15:35:49 -08:00
Nick Terrell 90d38f6a53
Merge pull request #945 from terrelln/dev
Fix cdict compressor repcodes
2017-12-13 14:24:21 -08:00
Nick Terrell 82bc8fe0cc [fileio] Refuse to remove non-regular file 2017-12-13 13:38:26 -08:00
Yann Collet 311878dec3 Improved tests
- building cli from /tests preserves potential flags in MOREFLAGS (such as asan/usan)
- MT dictionary tests check for MT capability (MT is not enabled by default for zstd32)
2017-12-13 11:48:30 -08:00
Nick Terrell 22727a7467 Fix cdict compressor repcodes 2017-12-13 11:31:20 -08:00
Yann Collet dba8016d2d Merge branch 'dev' into fix944 2017-12-13 11:20:09 -08:00
Yann Collet 04a1557e28 fuzzer tests : added -v command
so that unit tests get listed in CI console
and we can see which one failed.
2017-12-13 10:08:52 -08:00
Yann Collet 98ccc115ed added cli test veryfying bug #944
this test fails on v1.3.2
2017-12-12 19:47:38 -08:00
Yann Collet e9dc0ab884 zstd binary, when compiled from /tests, enables debug mode
adds ZSTD_DEBUG=1 and -g
2017-12-12 18:20:04 -08:00
Yann Collet e28305fcca fix #944 : ZSTDMT with large files and dictionary now works correctly
windowLog is now enforced from provided compression parameters,
instead of being copied blindly from `cdict`
where it could be smaller.

also :
- fix a minor bug in zstreamtest --mt : advanced parameters must be set before init
- changed advanced parameter name to ZSTDMT_jobSize
2017-12-12 18:04:58 -08:00
Yann Collet 03832b7aa5 re-added test case
messing with revert ... :(
2017-12-12 14:01:54 -08:00
Yann Collet 8a104fda05 Revert "Created a test case which reliably reproduces bug #944"
This reverts commit 5098d1fbe2.
2017-12-12 12:51:49 -08:00
Yann Collet 5098d1fbe2 Created a test case which reliably reproduces bug #944
in zstreamtest.
2017-12-12 12:48:31 -08:00
Yann Collet 21f7672d43 zstreamtest : add test which detects when ZSTD_initCStream() inflate data
as in #942, when ZSTD_initCStream() believes data is empty, instead of unknown.
2017-12-07 03:06:01 -05:00
Yann Collet 7e05ef851a Merge branch 'dev' into qemu32panic 2017-12-03 11:14:36 -08:00
Yann Collet a1b24e6262
Merge pull request #938 from terrelln/time
Use util.h for timing
2017-12-01 16:40:38 -08:00
Yann Collet 6fc63f521f Merge branch 'dev' into qemu32panic 2017-12-01 10:51:01 -08:00
Yann Collet 453fb5e830 removed -ftrapv from tests/ debug flags
-ftrapv is apparently buggy for gcc.
versions >= 5 are supposed to work better,
but even then, some complaints say it's still flaky when optimizations are enabled.
I even saw a post saying it only works if one creates its own signal handler,
which would make this flag no longer transparent.

on clang, it seems to work correctly.
But we would need to add a method to selectively add flags depending on compiler.
That's too much troubles for the intended benefit
(just catch integer overflows, which we can also do using ubsan).
2017-12-01 10:30:53 -08:00
Nick Terrell dab8cfa3c7 Combine definitions of SEC_TO_MICRO 2017-11-30 19:40:53 -08:00
Nick Terrell 9a2f6f477b Use util.h for timing 2017-11-30 14:57:25 -08:00
Yann Collet d3c59edac9 removed long-range-mode tests from `zstreamtest --no-big-tests` 2017-11-29 16:42:20 -08:00
Yann Collet b3d76e0a94 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-11-28 15:03:38 -08:00
Yann Collet f5747afe0f add integer overflow trapping to debug builds
doc: https://gcc.gnu.org/onlinedocs/gcc-4.0.2/gcc/Code-Gen-Options.html
2017-11-28 15:02:48 -08:00
W. Felix Handte 62c746dcf9 Add Test on LZ4 Format Input Buffer Overrun 2017-11-28 12:06:48 -05:00
Yann Collet bd88f633ac zstreamtest : in `-T#s`, s considered a suffix meaning "seconds"
avoid unintentionnally triggering `seedset`,
so that seed gets automatically determined when not set.
2017-11-27 12:15:23 -08:00
Yann Collet 42c1e64270 slightly improved ratio at -22
merging of repcode search into btsearch introduced a small compression ratio regressio at max level :
1.3.2 : 52728769
after repMerge patch : 52760789 (+32020)

A few minor changes have produced this difference.
They can be hard to spot.

This patch buys back about half of the difference,
by no longer inserting position at hc3 when a long match is found there.
It feels strangely counter-intuitive, but works :
after this patch : 52742555 (-18234)
2017-11-19 14:00:55 -08:00
Yann Collet 99435dbbab minor : search early-out on sufficient_len for hc3 and rep
very very small speed and ratio increases
2017-11-19 12:58:04 -08:00
Yann Collet 23767e950a fix one UB pointer arithmetic in encoder
Instead of calculating distance between 2 memory objects, which is UB,
we extract the offset from object 1, and transfer it into object 2.
2017-11-17 13:24:51 -08:00
Yann Collet 15768cabb5 fixed some complex scenarios
Fixed : multithreading to compress some small data with dictionary
Fixed : ZSTD_initCStream_usingCDict()
Improved streaming memory usage when pledgedSrcSize is known.
2017-11-16 15:18:18 -08:00
Yann Collet aec56a52fb
Merge pull request #908 from facebook/ubsan
Modified one pointer arithmetic expression to a more conformant way.
2017-11-07 11:45:34 -08:00
Yann Collet 3b59a01039 fixed zstreamtest declaration after statement 2017-11-01 17:36:31 -07:00
Yann Collet 150354c5fe minor refactor
added some traces and assert
related to hunting a potential ubsan error in 32-bits more
(it ends up being a compiler-side issue : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82802).

Modified one pointer arithmetic expression for a more conformant way.
2017-11-01 16:57:48 -07:00
Nick Terrell e19b0822bc Test large skippable frames 2017-11-01 13:10:03 -07:00
Nick Terrell 86b8134cad [libzstd] Fix parameter selection for empty input
ZSTD_compress() and friends would treat an empty input as an unknown size
when selecting parameters. Thus, they would drastically overallocate the
context. Tell ZSTD_getParams() that the source size is 1 when it is empty.
2017-10-25 17:24:15 -07:00
Yann Collet 97dccbbb2b fixed zbufftest
preserve "pledgedSrcSize=0" means "unknown" in init_advanced()
2017-10-19 14:06:02 -07:00
Yann Collet 9d377c6e88 Merge pull request #895 from facebook/fileSize_unknown
Distinguish 0-size from size-unavailable
2017-10-18 16:50:17 -07:00
Yann Collet 1ff8a8c109 Merge pull request #891 from facebook/contentSize
Content size
2017-10-17 17:24:51 -07:00
Yann Collet 5891f1d952 added a check to ensure 0-size is present in frame header 2017-10-17 16:23:20 -07:00
Yann Collet 18b795374a UTIL_getFileSize() returns UTIL_FILESIZE_UNKNOWN on failure
UTIL_getFileSize() used to return zero on failure.
This made it impossible to distinguish a failure from a genuine empty file.
Both cases where coalesced.

Adding UTIL_FILESIZE_UNKNOWN constant has many consequences on user code,
since in many places, the `0` was assumed to mean "error".
This is no longer the case, and the error code must be actively checked.
2017-10-17 16:14:25 -07:00
Nick Terrell e600b5d0da [test] Exercise all codes in dictionary tables 2017-10-16 18:05:36 -07:00