1ff8a8c109
Merge pull request #891 from facebook/contentSize
...
Content size
2017-10-17 17:24:51 -07:00
5891f1d952
added a check to ensure 0-size is present in frame header
2017-10-17 16:23:20 -07:00
18b795374a
UTIL_getFileSize() returns UTIL_FILESIZE_UNKNOWN on failure
...
UTIL_getFileSize() used to return zero on failure.
This made it impossible to distinguish a failure from a genuine empty file.
Both cases where coalesced.
Adding UTIL_FILESIZE_UNKNOWN constant has many consequences on user code,
since in many places, the `0` was assumed to mean "error".
This is no longer the case, and the error code must be actively checked.
2017-10-17 16:14:25 -07:00
e600b5d0da
[test] Exercise all codes in dictionary tables
2017-10-16 18:05:36 -07:00
241c57adff
playtest : minor cosmetic changes
2017-10-16 14:01:42 -07:00
43914f6a20
add test for presence of decompressed size field in header
...
previous test was "display" only, it wouldn't trigger an error in case of missing header.
2017-10-13 23:47:01 -07:00
beb9b4b398
fixed ZSTDMT_initCStream() when contentSizeFlag==1 by default
...
and a wrong test in zstreamtest --mt
2017-10-13 19:09:30 -07:00
213ef3b510
fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag,
...
and a stream fuzzer test, which was incorrect
(relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)
2017-10-13 19:01:58 -07:00
3c1e3f8ec9
contentSizeFlag enabled by default would also fail for streaming and MT operations
...
fixed
2017-10-13 18:32:06 -07:00
fb44516641
ensure fParams.contentSizeFlag starts at 1
...
such default was failing for ZSTD_compressBegin/ZSTD_compressContinue
fixed too
2017-10-13 17:39:13 -07:00
7f6a783862
fixed a small error in decodeCorpus
...
a compressed block must be strictly smaller than its decompressed size.
2017-10-07 15:19:52 -07:00
4252621e26
playtests: do not use cat on large files
...
some target have limitation making cat incompatible with large files
(namely debian hurd-i386)
2017-10-05 20:21:59 -07:00
e580dc6a4a
Merge pull request #860 from felixhandte/zstd-lz4-support-tests
...
Add Default LZ4 Support When Available
2017-09-29 22:32:54 -07:00
e963800e27
zstdmt : fixed : buffer dst0 wasn't properly set to null after usage
...
now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources()
wether previous compression was completed correctly or not.
2017-09-28 23:01:31 -07:00
e0065cf660
make test : removed zstreamtest unit tests for variants
...
slightly reduced time to create dictionary
at beginning of unit tests
2017-09-28 18:34:38 -07:00
47c6a95d07
zstreamtest : run unit tests only during "normal" session
...
not during --mt, --newapi and --opaque
this avoids running them 4x during `make test`
2017-09-28 18:27:22 -07:00
d0519d4b0c
Add CLI Program Name Detection for LZ4
2017-09-28 19:18:15 -04:00
5705d9f25a
Add basic tests for the lz4 integration
2017-09-28 19:16:43 -04:00
e4ec427720
Merge branch 'dev' into shorterTests
...
fixed conflicts
2017-09-28 12:19:28 -07:00
bbef058ae6
zstreamtest --newapi : reduced maximum allocated memory
2017-09-28 11:48:45 -07:00
d6770f80af
minor : rewrite unit tests using CHECK_Z macro
2017-09-28 02:14:48 -07:00
9fe50ed623
fixed maximum windowLog for zstreamtest --newapi
...
for compatibility with low memory VM
2017-09-28 01:42:06 -07:00
9b5b47ac93
ensure adjustCParams adjust hLog and cLog even without srcSize
...
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.
Reduces maximum memory allocated during zstreamtest --newapi
2017-09-28 01:25:40 -07:00
824f75ea7c
Merge pull request #863 from facebook/newFormats
...
magicless frames (#591 )
2017-09-28 00:32:16 -07:00
d9c1e9125f
[fuzz] Small changes for oss-fuzz integration
2017-09-27 18:23:06 -07:00
aa800c4793
reduced memory usage of zstreamtest --newapi
...
to run on memory-constrained VM
2017-09-27 18:00:15 -07:00
bc32b40b98
reduced zstreamtest --mt memory load
...
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
2017-09-27 17:27:38 -07:00
54a827fff0
Merge branch 'dev' into newFormats
...
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
02502191e5
separated --long tests between short and long tests
...
A fast subset of these tests is now played in short test mode
2017-09-27 15:48:06 -07:00
f9de54acfb
reduced memory requirements for --long tests
...
in new --long test section
2017-09-27 15:38:27 -07:00
bdd0f6f046
improved make clean in tests/fuzz
2017-09-27 15:20:08 -07:00
60059df051
shorter make test to avoid time out on travis CI
...
Timed tests (fuzzer) are reduced
Long tests are shortened (less data generated)
2017-09-27 15:16:27 -07:00
ea1f50bf73
removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal()
...
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.
Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.
A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
2017-09-27 13:51:05 -07:00
bfabd1d4dc
fixed zstreamtest decoding error
...
same error (wrong output buffer size) was present
on --mt and --new_api tests.
2017-09-27 01:01:11 -07:00
ca306c1c84
fixed a bug in zstreamtest
...
decoder output buffer would receive a wrong size.
In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.
This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
2017-09-27 00:39:41 -07:00
4791561c4a
silence minor gcc warning -Wempty-body
...
also silence fuzz test artefacts
2017-09-26 17:57:38 -07:00
91ffe71ec8
Merge pull request #864 from terrelln/long
...
Increase maximum window size
2017-09-26 16:26:29 -07:00
df4e9bba25
fixed constant errors for gcc in c99 mode
...
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
2017-09-26 14:31:06 -07:00
9f0b8dfbe9
Merge branch 'dev' into newFormats
2017-09-26 14:22:39 -07:00
471aa385b3
[fuzz] Speed up round trip tests
...
* Enforce smaller maximum values for parameters
* Adjust parameters to the source size
The memory usage is reduced by about 5x, which makes the fuzzers run at
least twice as fast, even more so with ASAN/MSAN enabled.
2017-09-26 14:03:43 -07:00
c233bdbaee
Increase maximum window size
...
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
`--zstd=wlog=#`. These options also set the window size during
decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
and decompression.
2017-09-26 14:00:01 -07:00
8a4d0abc1c
Merge pull request #861 from terrelln/zip
...
[fuzz][CI] Set up regression tests
2017-09-25 17:11:35 -07:00
52a1d1c6dc
added ZSTD_DCtx_reset()
2017-09-25 16:56:48 -07:00
5d8fdd1641
Merge pull request #855 from terrelln/maxoff
...
[libzstd] Increase MaxOff
2017-09-25 16:34:29 -07:00
917a213254
[fuzz] Determine flags based on compiler version
2017-09-25 15:32:36 -07:00
11e21f23cb
[fuzz] Mention the corpora in the README
2017-09-25 15:31:38 -07:00
6bb781e0f1
[fuzz] Add regressiontest targets
2017-09-25 15:31:33 -07:00
62568c9a42
added capability to generate magic-less frames
...
decoder not implemented yet
2017-09-25 14:26:26 -07:00
bbe77212ef
[libzstd] Increase MaxOff
2017-09-25 13:36:18 -07:00
bfad5568b5
[fuzz] Make simple_round_trip compile cleanly
2017-09-25 13:28:45 -07:00
23199b6daf
[fuzz] Fix fuzz.py env flags parsing
2017-09-25 13:28:18 -07:00
1c23b64049
[fuzz] fuzz.py can minimize and zip corpora
...
* "minimize" minimizes the corpora into an output directory.
* "zip" zips up the minimized corpora, which are ready to deploy.
2017-09-25 12:04:12 -07:00
96f0cde31a
minor function rename
...
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
cd3115b284
added control from frame content size at end of decompression
...
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
2017-09-21 16:21:10 -07:00
1fe762e236
[zstdcli] Fix LDM advanced options parsing
2017-09-18 14:49:35 -07:00
31829cb057
Merge pull request #847 from terrelln/fuzzer
...
[fuzzer] Fuzz long range matching & new API
2017-09-15 12:09:00 -07:00
39357c41cb
[fuzzer] Fuzz long range matching & new API
2017-09-14 14:48:08 -07:00
218c09e5b3
Merge pull request #844 from terrelln/fuzzer
...
Fuzzer
2017-09-14 11:40:25 -07:00
9712d5ebe6
[fuzzer] Fix bugs in fuzz.py
2017-09-13 19:08:35 -07:00
a6f08b4783
[fuzzer] Fix FUZZ_seed()
2017-09-13 18:41:32 -07:00
6c6412cef9
[fuzzer] Update README.md
2017-09-13 18:23:52 -07:00
6b8236cf7e
[fuzz] Add fuzzing helper script
2017-09-13 17:45:21 -07:00
b7e1522330
Add block fuzzers
2017-09-13 17:44:41 -07:00
def3214d74
[fuzzer] Handle single empty directory
2017-09-13 17:44:30 -07:00
739b620814
Merge pull request #842 from stellamplau/decodeCorpus-maxSize
...
Add flag to limit max decompressed size in decodeCorpus
2017-09-13 17:26:55 -07:00
8b6c80ada8
Update fuzzer Makefile
2017-09-13 16:16:57 -07:00
677c2cbf89
Update fuzzer sources
2017-09-13 16:16:57 -07:00
963558a072
Fix implicit conversion error
2017-09-13 16:01:16 -07:00
40bf0ced7d
Add flag to limit max decompressed size in decodeCorpus
2017-09-13 15:16:56 -07:00
f1571dad8f
Merge pull request #838 from stellamplau/ldm-mergeDev
...
Add long distance matcher
2017-09-13 13:24:08 -07:00
be1f2dac5b
Merge pull request #841 from facebook/utilTimeAPI
...
modified util::time API (T19505791)
2017-09-13 11:41:01 -07:00
a1bc08834f
Merge pull request #840 from stellamplau/decodeCorpus-blocks
...
Make decodecorpus generate raw compressed blocks
2017-09-13 09:34:04 -07:00
c95c0c9725
modified util::time API
...
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
2017-09-12 18:12:46 -07:00
e89065506e
Make decodecorpus generate raw compressed blocks
2017-09-12 17:18:45 -07:00
3d8e313f64
Reduce ldm hash table size in test
2017-09-11 17:21:28 -07:00
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
b3f33ccfb3
use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
...
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
058ed2ad33
ZSTD_decodingBufferSize_min()
...
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.
also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
eeff55dfa8
Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev
2017-09-06 15:56:32 -07:00
9e4060200b
Add tests and fix pointer alignment
2017-09-06 09:14:05 -07:00
af4068a697
Fix function name in tests/fuzz/regression_driver
2017-09-05 22:14:41 -07:00
67d4a6161c
Add ldmBucketSizeLog param
2017-09-02 21:55:29 -07:00
a1f04d518d
Move hashEveryLog to cctxParams and update cli
2017-09-01 15:05:47 -07:00
767a0b3be1
Move ldm hashLog, bucketLog, and mml to cctxParams
2017-09-01 12:24:59 -07:00
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
7492e7f1c7
tests/fuzz: change ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
...
ZSTD_BLOCKSIZE_ABSOLUTEMAX is changed at the commit:
fa3671eac7
2017-09-01 16:37:39 +09:00
6639395979
tests/fuzz: fix make all target names
2017-09-01 16:32:40 +09:00
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
e0cecd8736
fixed poolTests
...
needs more dependencies from zstd for custom allocators and error codes
2017-08-31 15:13:31 -07:00
6a546efb8c
Add long distance matcher
...
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
b0cb081dc8
last batch of header files changed to reflect new license ( #825 )
...
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
e21384fffb
fixed more file headers after license change ( #825 )
2017-08-31 12:11:57 -07:00
e9dc204f42
fixed a bunch of headers after license change ( #825 )
2017-08-31 11:24:54 -07:00
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
ee65701720
Minor fixes; remove formatting only changes
2017-08-29 20:27:35 -07:00
a6e20e1bd7
Add test for raw content starting with dict header
2017-08-29 18:36:18 -07:00
82d636b76a
Rename applyCCtxParams()
2017-08-29 18:03:06 -07:00
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
Yann Collet
Yann Collet
Nick Terrell
W. Felix Handte
Stella Lau
Eiichi Tsukata