709 Commits

Author SHA1 Message Date
Yann Collet
ea1f50bf73 removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal()
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.

Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.

A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
2017-09-27 13:51:05 -07:00
Yann Collet
bfabd1d4dc fixed zstreamtest decoding error
same error (wrong output buffer size) was present
on --mt and --new_api tests.
2017-09-27 01:01:11 -07:00
Yann Collet
ca306c1c84 fixed a bug in zstreamtest
decoder output buffer would receive a wrong size.

In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.

This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
2017-09-27 00:39:41 -07:00
Yann Collet
4791561c4a silence minor gcc warning -Wempty-body
also silence fuzz test artefacts
2017-09-26 17:57:38 -07:00
Yann Collet
91ffe71ec8 Merge pull request #864 from terrelln/long
Increase maximum window size
2017-09-26 16:26:29 -07:00
Yann Collet
df4e9bba25 fixed constant errors for gcc in c99 mode
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
2017-09-26 14:31:06 -07:00
Yann Collet
9f0b8dfbe9 Merge branch 'dev' into newFormats 2017-09-26 14:22:39 -07:00
Nick Terrell
471aa385b3 [fuzz] Speed up round trip tests
* Enforce smaller maximum values for parameters
* Adjust parameters to the source size

The memory usage is reduced by about 5x, which makes the fuzzers run at
least twice as fast, even more so with ASAN/MSAN enabled.
2017-09-26 14:03:43 -07:00
Nick Terrell
c233bdbaee Increase maximum window size
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
2017-09-26 14:00:01 -07:00
Yann Collet
8a4d0abc1c Merge pull request #861 from terrelln/zip
[fuzz][CI] Set up regression tests
2017-09-25 17:11:35 -07:00
Yann Collet
52a1d1c6dc added ZSTD_DCtx_reset() 2017-09-25 16:56:48 -07:00
Yann Collet
5d8fdd1641 Merge pull request #855 from terrelln/maxoff
[libzstd] Increase MaxOff
2017-09-25 16:34:29 -07:00
Nick Terrell
917a213254 [fuzz] Determine flags based on compiler version 2017-09-25 15:32:36 -07:00
Nick Terrell
11e21f23cb [fuzz] Mention the corpora in the README 2017-09-25 15:31:38 -07:00
Nick Terrell
6bb781e0f1 [fuzz] Add regressiontest targets 2017-09-25 15:31:33 -07:00
Yann Collet
62568c9a42 added capability to generate magic-less frames
decoder not implemented yet
2017-09-25 14:26:26 -07:00
Nick Terrell
bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Nick Terrell
bfad5568b5 [fuzz] Make simple_round_trip compile cleanly 2017-09-25 13:28:45 -07:00
Nick Terrell
23199b6daf [fuzz] Fix fuzz.py env flags parsing 2017-09-25 13:28:18 -07:00
Nick Terrell
1c23b64049 [fuzz] fuzz.py can minimize and zip corpora
* "minimize" minimizes the corpora into an output directory.
* "zip" zips up the minimized corpora, which are ready to deploy.
2017-09-25 12:04:12 -07:00
Yann Collet
96f0cde31a minor function rename
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
Yann Collet
cd3115b284 added control from frame content size at end of decompression
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3b4fcc73df82126fd5ecfb751268fc6
2017-09-21 16:21:10 -07:00
Nick Terrell
1fe762e236 [zstdcli] Fix LDM advanced options parsing 2017-09-18 14:49:35 -07:00
Yann Collet
31829cb057 Merge pull request #847 from terrelln/fuzzer
[fuzzer] Fuzz long range matching & new API
2017-09-15 12:09:00 -07:00
Nick Terrell
39357c41cb [fuzzer] Fuzz long range matching & new API 2017-09-14 14:48:08 -07:00
Yann Collet
218c09e5b3 Merge pull request #844 from terrelln/fuzzer
Fuzzer
2017-09-14 11:40:25 -07:00
Nick Terrell
9712d5ebe6 [fuzzer] Fix bugs in fuzz.py 2017-09-13 19:08:35 -07:00
Nick Terrell
a6f08b4783 [fuzzer] Fix FUZZ_seed() 2017-09-13 18:41:32 -07:00
Nick Terrell
6c6412cef9 [fuzzer] Update README.md 2017-09-13 18:23:52 -07:00
Nick Terrell
6b8236cf7e [fuzz] Add fuzzing helper script 2017-09-13 17:45:21 -07:00
Nick Terrell
b7e1522330 Add block fuzzers 2017-09-13 17:44:41 -07:00
Nick Terrell
def3214d74 [fuzzer] Handle single empty directory 2017-09-13 17:44:30 -07:00
Yann Collet
739b620814 Merge pull request #842 from stellamplau/decodeCorpus-maxSize
Add flag to limit max decompressed size in decodeCorpus
2017-09-13 17:26:55 -07:00
Nick Terrell
8b6c80ada8 Update fuzzer Makefile 2017-09-13 16:16:57 -07:00
Nick Terrell
677c2cbf89 Update fuzzer sources 2017-09-13 16:16:57 -07:00
Stella Lau
963558a072 Fix implicit conversion error 2017-09-13 16:01:16 -07:00
Stella Lau
40bf0ced7d Add flag to limit max decompressed size in decodeCorpus 2017-09-13 15:16:56 -07:00
Yann Collet
f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev
Add long distance matcher
2017-09-13 13:24:08 -07:00
Yann Collet
be1f2dac5b Merge pull request #841 from facebook/utilTimeAPI
modified util::time API (T19505791)
2017-09-13 11:41:01 -07:00
Yann Collet
a1bc08834f Merge pull request #840 from stellamplau/decodeCorpus-blocks
Make decodecorpus generate raw compressed blocks
2017-09-13 09:34:04 -07:00
Yann Collet
c95c0c9725 modified util::time API
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
2017-09-12 18:12:46 -07:00
Stella Lau
e89065506e Make decodecorpus generate raw compressed blocks 2017-09-12 17:18:45 -07:00
Stella Lau
3d8e313f64 Reduce ldm hash table size in test 2017-09-11 17:21:28 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Yann Collet
b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
Yann Collet
058ed2ad33 ZSTD_decodingBufferSize_min()
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
Yann Collet
3128e03be6 updated license header
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Stella Lau
eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Stella Lau
9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
Stella Lau
af4068a697 Fix function name in tests/fuzz/regression_driver 2017-09-05 22:14:41 -07:00