9a2f6f477b
Use util.h for timing
2017-11-30 14:57:25 -08:00
d3c59edac9
removed long-range-mode tests from `zstreamtest --no-big-tests`
2017-11-29 16:42:20 -08:00
b3d76e0a94
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-11-28 15:03:38 -08:00
f5747afe0f
add integer overflow trapping to debug builds
...
doc: https://gcc.gnu.org/onlinedocs/gcc-4.0.2/gcc/Code-Gen-Options.html
2017-11-28 15:02:48 -08:00
62c746dcf9
Add Test on LZ4 Format Input Buffer Overrun
2017-11-28 12:06:48 -05:00
bd88f633ac
zstreamtest : in `-T#s`, s considered a suffix meaning "seconds"
...
avoid unintentionnally triggering `seedset`,
so that seed gets automatically determined when not set.
2017-11-27 12:15:23 -08:00
42c1e64270
slightly improved ratio at -22
...
merging of repcode search into btsearch introduced a small compression ratio regressio at max level :
1.3.2 : 52728769
after repMerge patch : 52760789 (+32020)
A few minor changes have produced this difference.
They can be hard to spot.
This patch buys back about half of the difference,
by no longer inserting position at hc3 when a long match is found there.
It feels strangely counter-intuitive, but works :
after this patch : 52742555 (-18234)
2017-11-19 14:00:55 -08:00
99435dbbab
minor : search early-out on sufficient_len for hc3 and rep
...
very very small speed and ratio increases
2017-11-19 12:58:04 -08:00
23767e950a
fix one UB pointer arithmetic in encoder
...
Instead of calculating distance between 2 memory objects, which is UB,
we extract the offset from object 1, and transfer it into object 2.
2017-11-17 13:24:51 -08:00
15768cabb5
fixed some complex scenarios
...
Fixed : multithreading to compress some small data with dictionary
Fixed : ZSTD_initCStream_usingCDict()
Improved streaming memory usage when pledgedSrcSize is known.
2017-11-16 15:18:18 -08:00
aec56a52fb
Merge pull request #908 from facebook/ubsan
...
Modified one pointer arithmetic expression to a more conformant way.
2017-11-07 11:45:34 -08:00
3b59a01039
fixed zstreamtest declaration after statement
2017-11-01 17:36:31 -07:00
150354c5fe
minor refactor
...
added some traces and assert
related to hunting a potential ubsan error in 32-bits more
(it ends up being a compiler-side issue : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82802 ).
Modified one pointer arithmetic expression for a more conformant way.
2017-11-01 16:57:48 -07:00
e19b0822bc
Test large skippable frames
2017-11-01 13:10:03 -07:00
86b8134cad
[libzstd] Fix parameter selection for empty input
...
ZSTD_compress() and friends would treat an empty input as an unknown size
when selecting parameters. Thus, they would drastically overallocate the
context. Tell ZSTD_getParams() that the source size is 1 when it is empty.
2017-10-25 17:24:15 -07:00
97dccbbb2b
fixed zbufftest
...
preserve "pledgedSrcSize=0" means "unknown" in init_advanced()
2017-10-19 14:06:02 -07:00
9d377c6e88
Merge pull request #895 from facebook/fileSize_unknown
...
Distinguish 0-size from size-unavailable
2017-10-18 16:50:17 -07:00
1ff8a8c109
Merge pull request #891 from facebook/contentSize
...
Content size
2017-10-17 17:24:51 -07:00
5891f1d952
added a check to ensure 0-size is present in frame header
2017-10-17 16:23:20 -07:00
18b795374a
UTIL_getFileSize() returns UTIL_FILESIZE_UNKNOWN on failure
...
UTIL_getFileSize() used to return zero on failure.
This made it impossible to distinguish a failure from a genuine empty file.
Both cases where coalesced.
Adding UTIL_FILESIZE_UNKNOWN constant has many consequences on user code,
since in many places, the `0` was assumed to mean "error".
This is no longer the case, and the error code must be actively checked.
2017-10-17 16:14:25 -07:00
e600b5d0da
[test] Exercise all codes in dictionary tables
2017-10-16 18:05:36 -07:00
241c57adff
playtest : minor cosmetic changes
2017-10-16 14:01:42 -07:00
43914f6a20
add test for presence of decompressed size field in header
...
previous test was "display" only, it wouldn't trigger an error in case of missing header.
2017-10-13 23:47:01 -07:00
beb9b4b398
fixed ZSTDMT_initCStream() when contentSizeFlag==1 by default
...
and a wrong test in zstreamtest --mt
2017-10-13 19:09:30 -07:00
213ef3b510
fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag,
...
and a stream fuzzer test, which was incorrect
(relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)
2017-10-13 19:01:58 -07:00
3c1e3f8ec9
contentSizeFlag enabled by default would also fail for streaming and MT operations
...
fixed
2017-10-13 18:32:06 -07:00
fb44516641
ensure fParams.contentSizeFlag starts at 1
...
such default was failing for ZSTD_compressBegin/ZSTD_compressContinue
fixed too
2017-10-13 17:39:13 -07:00
7f6a783862
fixed a small error in decodeCorpus
...
a compressed block must be strictly smaller than its decompressed size.
2017-10-07 15:19:52 -07:00
4252621e26
playtests: do not use cat on large files
...
some target have limitation making cat incompatible with large files
(namely debian hurd-i386)
2017-10-05 20:21:59 -07:00
e580dc6a4a
Merge pull request #860 from felixhandte/zstd-lz4-support-tests
...
Add Default LZ4 Support When Available
2017-09-29 22:32:54 -07:00
e963800e27
zstdmt : fixed : buffer dst0 wasn't properly set to null after usage
...
now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources()
wether previous compression was completed correctly or not.
2017-09-28 23:01:31 -07:00
e0065cf660
make test : removed zstreamtest unit tests for variants
...
slightly reduced time to create dictionary
at beginning of unit tests
2017-09-28 18:34:38 -07:00
47c6a95d07
zstreamtest : run unit tests only during "normal" session
...
not during --mt, --newapi and --opaque
this avoids running them 4x during `make test`
2017-09-28 18:27:22 -07:00
d0519d4b0c
Add CLI Program Name Detection for LZ4
2017-09-28 19:18:15 -04:00
5705d9f25a
Add basic tests for the lz4 integration
2017-09-28 19:16:43 -04:00
e4ec427720
Merge branch 'dev' into shorterTests
...
fixed conflicts
2017-09-28 12:19:28 -07:00
bbef058ae6
zstreamtest --newapi : reduced maximum allocated memory
2017-09-28 11:48:45 -07:00
d6770f80af
minor : rewrite unit tests using CHECK_Z macro
2017-09-28 02:14:48 -07:00
9fe50ed623
fixed maximum windowLog for zstreamtest --newapi
...
for compatibility with low memory VM
2017-09-28 01:42:06 -07:00
9b5b47ac93
ensure adjustCParams adjust hLog and cLog even without srcSize
...
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.
Reduces maximum memory allocated during zstreamtest --newapi
2017-09-28 01:25:40 -07:00
824f75ea7c
Merge pull request #863 from facebook/newFormats
...
magicless frames (#591 )
2017-09-28 00:32:16 -07:00
d9c1e9125f
[fuzz] Small changes for oss-fuzz integration
2017-09-27 18:23:06 -07:00
aa800c4793
reduced memory usage of zstreamtest --newapi
...
to run on memory-constrained VM
2017-09-27 18:00:15 -07:00
bc32b40b98
reduced zstreamtest --mt memory load
...
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
2017-09-27 17:27:38 -07:00
54a827fff0
Merge branch 'dev' into newFormats
...
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
02502191e5
separated --long tests between short and long tests
...
A fast subset of these tests is now played in short test mode
2017-09-27 15:48:06 -07:00
f9de54acfb
reduced memory requirements for --long tests
...
in new --long test section
2017-09-27 15:38:27 -07:00
bdd0f6f046
improved make clean in tests/fuzz
2017-09-27 15:20:08 -07:00
60059df051
shorter make test to avoid time out on travis CI
...
Timed tests (fuzzer) are reduced
Long tests are shortened (less data generated)
2017-09-27 15:16:27 -07:00
ea1f50bf73
removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal()
...
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.
Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.
A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
2017-09-27 13:51:05 -07:00
bfabd1d4dc
fixed zstreamtest decoding error
...
same error (wrong output buffer size) was present
on --mt and --new_api tests.
2017-09-27 01:01:11 -07:00
ca306c1c84
fixed a bug in zstreamtest
...
decoder output buffer would receive a wrong size.
In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.
This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
2017-09-27 00:39:41 -07:00
4791561c4a
silence minor gcc warning -Wempty-body
...
also silence fuzz test artefacts
2017-09-26 17:57:38 -07:00
91ffe71ec8
Merge pull request #864 from terrelln/long
...
Increase maximum window size
2017-09-26 16:26:29 -07:00
df4e9bba25
fixed constant errors for gcc in c99 mode
...
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
2017-09-26 14:31:06 -07:00
9f0b8dfbe9
Merge branch 'dev' into newFormats
2017-09-26 14:22:39 -07:00
471aa385b3
[fuzz] Speed up round trip tests
...
* Enforce smaller maximum values for parameters
* Adjust parameters to the source size
The memory usage is reduced by about 5x, which makes the fuzzers run at
least twice as fast, even more so with ASAN/MSAN enabled.
2017-09-26 14:03:43 -07:00
c233bdbaee
Increase maximum window size
...
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
`--zstd=wlog=#`. These options also set the window size during
decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
and decompression.
2017-09-26 14:00:01 -07:00
8a4d0abc1c
Merge pull request #861 from terrelln/zip
...
[fuzz][CI] Set up regression tests
2017-09-25 17:11:35 -07:00
52a1d1c6dc
added ZSTD_DCtx_reset()
2017-09-25 16:56:48 -07:00
5d8fdd1641
Merge pull request #855 from terrelln/maxoff
...
[libzstd] Increase MaxOff
2017-09-25 16:34:29 -07:00
917a213254
[fuzz] Determine flags based on compiler version
2017-09-25 15:32:36 -07:00
11e21f23cb
[fuzz] Mention the corpora in the README
2017-09-25 15:31:38 -07:00
6bb781e0f1
[fuzz] Add regressiontest targets
2017-09-25 15:31:33 -07:00
62568c9a42
added capability to generate magic-less frames
...
decoder not implemented yet
2017-09-25 14:26:26 -07:00
bbe77212ef
[libzstd] Increase MaxOff
2017-09-25 13:36:18 -07:00
bfad5568b5
[fuzz] Make simple_round_trip compile cleanly
2017-09-25 13:28:45 -07:00
23199b6daf
[fuzz] Fix fuzz.py env flags parsing
2017-09-25 13:28:18 -07:00
1c23b64049
[fuzz] fuzz.py can minimize and zip corpora
...
* "minimize" minimizes the corpora into an output directory.
* "zip" zips up the minimized corpora, which are ready to deploy.
2017-09-25 12:04:12 -07:00
96f0cde31a
minor function rename
...
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
cd3115b284
added control from frame content size at end of decompression
...
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
2017-09-21 16:21:10 -07:00
1fe762e236
[zstdcli] Fix LDM advanced options parsing
2017-09-18 14:49:35 -07:00
31829cb057
Merge pull request #847 from terrelln/fuzzer
...
[fuzzer] Fuzz long range matching & new API
2017-09-15 12:09:00 -07:00
39357c41cb
[fuzzer] Fuzz long range matching & new API
2017-09-14 14:48:08 -07:00
218c09e5b3
Merge pull request #844 from terrelln/fuzzer
...
Fuzzer
2017-09-14 11:40:25 -07:00
9712d5ebe6
[fuzzer] Fix bugs in fuzz.py
2017-09-13 19:08:35 -07:00
a6f08b4783
[fuzzer] Fix FUZZ_seed()
2017-09-13 18:41:32 -07:00
6c6412cef9
[fuzzer] Update README.md
2017-09-13 18:23:52 -07:00
6b8236cf7e
[fuzz] Add fuzzing helper script
2017-09-13 17:45:21 -07:00
b7e1522330
Add block fuzzers
2017-09-13 17:44:41 -07:00
def3214d74
[fuzzer] Handle single empty directory
2017-09-13 17:44:30 -07:00
739b620814
Merge pull request #842 from stellamplau/decodeCorpus-maxSize
...
Add flag to limit max decompressed size in decodeCorpus
2017-09-13 17:26:55 -07:00
8b6c80ada8
Update fuzzer Makefile
2017-09-13 16:16:57 -07:00
677c2cbf89
Update fuzzer sources
2017-09-13 16:16:57 -07:00
963558a072
Fix implicit conversion error
2017-09-13 16:01:16 -07:00
40bf0ced7d
Add flag to limit max decompressed size in decodeCorpus
2017-09-13 15:16:56 -07:00
f1571dad8f
Merge pull request #838 from stellamplau/ldm-mergeDev
...
Add long distance matcher
2017-09-13 13:24:08 -07:00
be1f2dac5b
Merge pull request #841 from facebook/utilTimeAPI
...
modified util::time API (T19505791)
2017-09-13 11:41:01 -07:00
a1bc08834f
Merge pull request #840 from stellamplau/decodeCorpus-blocks
...
Make decodecorpus generate raw compressed blocks
2017-09-13 09:34:04 -07:00
c95c0c9725
modified util::time API
...
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
2017-09-12 18:12:46 -07:00
e89065506e
Make decodecorpus generate raw compressed blocks
2017-09-12 17:18:45 -07:00
3d8e313f64
Reduce ldm hash table size in test
2017-09-11 17:21:28 -07:00
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
b3f33ccfb3
use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
...
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
058ed2ad33
ZSTD_decodingBufferSize_min()
...
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.
also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
eeff55dfa8
Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev
2017-09-06 15:56:32 -07:00
9e4060200b
Add tests and fix pointer alignment
2017-09-06 09:14:05 -07:00
af4068a697
Fix function name in tests/fuzz/regression_driver
2017-09-05 22:14:41 -07:00
67d4a6161c
Add ldmBucketSizeLog param
2017-09-02 21:55:29 -07:00
a1f04d518d
Move hashEveryLog to cctxParams and update cli
2017-09-01 15:05:47 -07:00
767a0b3be1
Move ldm hashLog, bucketLog, and mml to cctxParams
2017-09-01 12:24:59 -07:00
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
7492e7f1c7
tests/fuzz: change ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
...
ZSTD_BLOCKSIZE_ABSOLUTEMAX is changed at the commit:
fa3671eac7
2017-09-01 16:37:39 +09:00
6639395979
tests/fuzz: fix make all target names
2017-09-01 16:32:40 +09:00
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
e0cecd8736
fixed poolTests
...
needs more dependencies from zstd for custom allocators and error codes
2017-08-31 15:13:31 -07:00
6a546efb8c
Add long distance matcher
...
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
b0cb081dc8
last batch of header files changed to reflect new license ( #825 )
...
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
e21384fffb
fixed more file headers after license change ( #825 )
2017-08-31 12:11:57 -07:00
e9dc204f42
fixed a bunch of headers after license change ( #825 )
2017-08-31 11:24:54 -07:00
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
ee65701720
Minor fixes; remove formatting only changes
2017-08-29 20:27:35 -07:00
a6e20e1bd7
Add test for raw content starting with dict header
2017-08-29 18:36:18 -07:00
82d636b76a
Rename applyCCtxParams()
2017-08-29 18:03:06 -07:00
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
b5b9275e67
Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced()
2017-08-29 10:49:29 -07:00
18224608ff
Remove ZSTD_setCCtxParameter()
2017-08-25 13:58:41 -07:00
9911153723
Move jobSize and overlapLog in zstdmt to cctxParams
2017-08-25 13:14:51 -07:00
eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
15fdeb9e41
Enforce nbThreads<=1 for estimateCCtxSize
2017-08-24 16:28:49 -07:00
1c81f725ff
Remove duplicated testing code
2017-08-23 15:47:15 -07:00
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
8fd1636776
Remove unused functions
2017-08-22 13:33:58 -07:00
73c73bf16a
Reduce code duplication in zstreamtest
2017-08-21 12:41:19 -07:00
3587556873
[cover] Test small maxdict
2017-08-21 11:16:47 -07:00
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
d6394cc4c3
fixed test-zstd-nolegacy
2017-08-20 10:15:44 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
8049556928
Merge pull request #778 from terrelln/bad-huff
...
[libzstd] Fix bug in Huffman decompresser
2017-08-07 14:05:58 -07:00
abe12b3399
[libzstd] Fix bug in Huffman decompresser
...
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.
* Make the decompresser accept Huffman compressed literals whose size
increases.
* Add a test case that exposes the bug. The compressed file has to be
statically generated, since the compressor won't normally produce files
that expose the bug.
2017-08-07 12:37:48 -07:00
e1abc2a367
Switch the sleep function to UTIL_sleepMilli
2017-08-07 11:49:13 -07:00
1e366f9dea
Add test for deadlock
2017-08-02 11:27:50 -07:00
5adceeed01
Allow queueSize=0 in pool.c and update poolTests
2017-07-31 10:10:16 -07:00
38ba7002f2
fixed minor warning on unused variable in shell function
2017-07-20 18:39:04 -07:00
5e6c5203f3
fixed fuzzer test for non OS-X platforms
2017-07-20 15:11:56 -07:00
1ca1288689
added --memtest=# command to fuzzer
...
to jump directly to relevant test section
2017-07-19 16:01:16 -07:00
44b0838253
Merge pull request #770 from terrelln/test-mode
...
[zstdcli] Fix -t in streaming mode
2017-07-18 15:40:59 -07:00
d0b27483ae
[zstdcli] Fix -t in streaming mode
2017-07-18 14:45:49 -07:00
cc1522351f
[libzstd] Fix bug in Huffman encoding
...
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.
The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.
Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s
Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
2017-07-18 13:20:40 -07:00
fa3aa04ccd
Merge pull request #761 from paulcruz74/file-rename
...
renamed pool.c to poolTests.c
2017-07-14 09:09:45 -07:00
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
ee3423d709
extended fuzzer MT memory tests
2017-07-10 14:09:16 -07:00
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
f9524cf366
added --memtest to fuzzer
2017-07-10 13:48:41 -07:00
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
89190ef07d
renamed pool.c to poolTests.c
2017-07-10 11:32:30 -07:00
ed0243a63c
removed zbufftest from list of `all` tests
2017-07-07 16:16:14 -07:00
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
7758ed8458
fixed fullbench, part 2
2017-07-06 02:48:00 -07:00
9b2c1acfc0
fixed fullbench
2017-07-06 02:22:57 -07:00
27e883371d
fixed wrong assert() condition
...
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
2017-07-04 19:33:16 -07:00
2cb9774f5e
more precise estimation of amount to flush at end of stream (single thread mode)
...
also : can use DEBUGLEVEL variable in /tests
2017-07-04 12:39:26 -07:00
5051dd39ca
Merge pull request #743 from facebook/fullbench
...
compress_generic() automatic optimization opportunities
2017-07-03 21:26:38 -07:00
95c4a6e2c8
Merge pull request #745 from terrelln/libfuzzer
...
[fuzz] Add libFuzzer targets
2017-07-03 15:15:20 -07:00
bea0f0cfa0
[fuzz] Move from fuzz/ to tests/fuzz/
2017-07-03 12:40:12 -07:00
4b26306cb8
blindfix : fullbench's one-time leak, detected by valgrind
2017-07-01 08:03:59 -07:00
c07e43c2b5
added --show-leak-kind=all to valgrind tests
2017-07-01 07:05:11 -07:00
b5bb7c6d95
fixed Visual compilation of fullbench-dll
2017-06-29 19:59:37 -07:00
e7e5a8cef7
made fullbench compatible with multi-threading
...
fullbench 61/62 measure speed of ZSTD_compress_generic with 2 threads
2017-06-29 18:56:24 -07:00
afb0aca739
zstreamtest : big tests are only enabled in 64-bits mode
...
to avoid requesting too much memory in 32-bits mode during MT tests
2017-06-29 18:19:09 -07:00
2e84bec9ac
updated fullbench to also measure ZSTD_compress_generic()
...
will make it possible to visualize
optimization opportunity for ZSTD_e_end
2017-06-29 13:03:10 -07:00
7303ed5d74
minor : specify compression level in tags
2017-06-29 11:30:31 -07:00
590937df20
Merge pull request #739 from facebook/refPrefix
...
ZSTD_refPrefix
2017-06-29 04:36:03 -07:00
811deaea6f
Merge pull request #736 from terrelln/cover-default-api
...
[zdict] Make COVER the default algorithm
2017-06-28 20:25:36 -07:00
ad5f29918e
Merge pull request #737 from facebook/advancedAPI2
...
simplified API for *initStatic*()
2017-06-28 20:25:22 -07:00
62f7efc04a
added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job
2017-06-28 16:25:13 -07:00
7ac4724bd2
removed fnum from DISPLAY statements
2017-06-28 13:00:49 -07:00
e667d33b0b
fixed generation of buggy test, corrected DISPLAY statements for errors
2017-06-28 12:19:37 -07:00
33a6639039
fixed ZSTD_refPrefix with Multithread-enabled CCtx
2017-06-28 11:09:43 -07:00
1fd2df9c2c
fixed zstreamtest
...
modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary()
but it is still allowed after ZSTD_refPrefix()
2017-06-28 09:57:59 -07:00
2e4274262d
controlled dictMode
2017-06-27 17:09:12 -07:00
b7372933b8
implemented ZSTD_refPrefix()
2017-06-27 15:49:12 -07:00
fecc721fd9
added parameter ZSTD_p_refDictContent
2017-06-27 11:46:39 -07:00
09a5bbe22e
Merge pull request #735 from terrelln/xz-default
...
[zstdcli] Support xz by default when liblzma is available
2017-06-27 09:42:02 -07:00
5b7fd7c422
[zdict] Make COVER the default algorithm
2017-06-26 21:09:22 -07:00
20eeb243d1
Merge pull request #729 from paulcruz74/corpus
...
Corpus
2017-06-26 17:47:28 -07:00
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
31af8290d1
ZSTD_estimateCCtx_advanced()
...
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.
ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
2017-06-26 15:52:39 -07:00
6aeb50ea0d
[zstdcli] Add unlzma and unxz symlinks
2017-06-26 11:24:36 -07:00
298fe9f23b
added cli tests to test-all
2017-06-23 17:10:41 -07:00
3a295a91f8
added additional condition so large offsets into the dictionary are not generated past windowSize
2017-06-23 15:54:51 -07:00
2085375816
fixed bug detected by the API test
2017-06-23 13:44:24 -07:00
8cd134559d
type warnings
2017-06-23 12:00:48 -07:00
4219acc60a
fixed bus error bug
2017-06-23 11:22:29 -07:00
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
cca1ed14db
Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2
2017-06-23 00:13:03 -07:00
32f546c763
zstreamtest : disabled multi-threading tests with --newapi --no-big-tests
...
--no-big-tests is typically used in combination with qemu-user-static
qemu-user-static allocated 4 GB of RAM upfront.
On 2 GB VM, this can degenerate into a crash.
It's not a problem as long as memory is not used.
But with multi-threading enabled, memory fragmentation kicks in,
so the amoung of RAM effectively touched increases,
and can pass beyond the 2 GB limit of the VM.
In single-threaded mode, there is no such issue :
memory requirement is smaller, and remains well-located,
so very little fragmentation is expected.
This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests`
work fine on Travis CI.
2017-06-23 00:09:02 -07:00
2e8cc6f12a
added sizeof for clarity
2017-06-22 15:52:33 -07:00
6d7f442120
changed makefile test
2017-06-22 15:44:01 -07:00
b325a2e4db
changed assignment
2017-06-22 15:36:28 -07:00
2935239544
added description for dictionary cli tests
2017-06-22 15:23:59 -07:00
97a8f89bec
made changes to cli test so it runs properly
2017-06-22 15:13:41 -07:00
829eb29033
added cli test for decodecorpus inside tests/Makefile. Also changed calculation of offset
2017-06-22 14:43:44 -07:00
028fb9d01b
added cli test case in Makefile
2017-06-22 13:53:39 -07:00
98751f69e7
should be updating seed whenever multiple files are generated
2017-06-22 10:23:36 -07:00
84cfa07d2d
changed format of command to --use-dict=#
2017-06-22 10:04:14 -07:00
f99c2c1a22
reduced --no-big-tests even more
...
to pass tests on qemu-aarch64
2017-06-21 23:35:58 -07:00
04094f37e9
fixed offset in this case os that it always goes past src start
2017-06-21 18:47:40 -07:00
49f8459d11
fixed minor cast warning
2017-06-21 18:43:39 -07:00
0950b3159a
more meaningful names for count variables
2017-06-21 18:30:27 -07:00
0b6eedeace
malloc samples instead of static allocation
2017-06-21 18:24:19 -07:00
4d3bdcf130
reduced CLevelMax for --no-big-tests
2017-06-21 18:20:02 -07:00
52a13abf3f
removed comment regarding windowSize requirement
2017-06-21 18:13:52 -07:00
32e36c2d9a
removed requirement that offset be less than windowSize
2017-06-21 18:00:47 -07:00
bfc2f00080
--no-big-tests for zstreamtest
...
Apply --no-big-tests for tsan tests
2017-06-21 17:57:14 -07:00
eeff2d0e3c
added const x2
2017-06-21 17:48:37 -07:00
04253e21bb
added const
2017-06-21 17:48:01 -07:00
2ceaec1aad
got rid of unnecessary casting to void*
2017-06-21 17:46:37 -07:00
ccae9ec807
added comment addressing use of bits in frame header descriptor
2017-06-21 17:43:21 -07:00
dce789281b
fixed : decompression of skippable frames in streaming mode
2017-06-21 15:53:42 -07:00
c16748b137
added test cases for new command --list
2017-06-21 12:09:53 -07:00
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
db3606e249
added test for files with no frame content size
2017-06-20 17:43:36 -07:00
acf0df678c
fixed strange spacing
2017-06-20 15:41:43 -07:00
69bc4fab25
free the context object once done with decompression
2017-06-20 15:37:14 -07:00
ceb4f65620
allocate statically so we don't have to deal with freeing
2017-06-20 15:31:54 -07:00
a73c2a444a
added tests with null files, changed condition to check that the file is more than 0 bytes
2017-06-20 14:33:08 -07:00
58c19b4202
spacing matters for test scripts
2017-06-20 14:14:53 -07:00
ff8f83bd47
fixed fuzzer test
2017-06-20 12:17:32 -07:00
b07d0af2ba
added additional tests for error detection
2017-06-20 11:54:44 -07:00
43ffcdde40
const dictOffset
2017-06-20 11:27:39 -07:00
1e0d125e76
const value
2017-06-20 11:15:26 -07:00
0034129270
made changes to accommodate single file use case
2017-06-20 11:07:38 -07:00
6a99de2dd8
change test to list info on .zst files
2017-06-20 10:45:06 -07:00
36e14b1963
added description to advanced usage, changed to use-dict
2017-06-20 10:34:34 -07:00
15102069a1
removed testing code from generation code
2017-06-20 10:25:16 -07:00
ff5589c091
added comments for clarity
2017-06-20 10:17:38 -07:00
551fb7ca05
changed error condition
2017-06-20 10:14:03 -07:00
b8237c6c66
refactoring for readability
2017-06-20 10:12:13 -07:00
6f17de2132
changed computation for readability
2017-06-20 09:50:28 -07:00
4a4f9ba262
const value that doesn't change
2017-06-19 17:46:21 -07:00
3a6856edca
changed scope to avoid errors
2017-06-19 17:43:16 -07:00
7d2b5613b4
adhere to min dict size
2017-06-19 17:37:14 -07:00
ea2af1ffc4
changed dictionary to stay reasonable size during test
2017-06-19 17:34:50 -07:00
fdbb07283a
added test to runTestMode
2017-06-19 17:29:15 -07:00
bdec7b8ce7
fixed for loop declaration, fixed test case
2017-06-19 17:18:43 -07:00
23098b53e8
wrote a test for dictionary corpus
2017-06-19 17:13:00 -07:00
7a26390989
changing for loop declaration
2017-06-19 16:39:57 -07:00
147ef05f12
for loop declaration not allowed
2017-06-19 16:38:51 -07:00
Nick Terrell
Yann Collet
W. Felix Handte
Yann Collet
Stella Lau
Eiichi Tsukata
Paul Cruz
cyan4973