5e6c5203f3
fixed fuzzer test for non OS-X platforms
2017-07-20 15:11:56 -07:00
1ca1288689
added --memtest=# command to fuzzer
...
to jump directly to relevant test section
2017-07-19 16:01:16 -07:00
cc1522351f
[libzstd] Fix bug in Huffman encoding
...
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.
The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.
Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s
Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
2017-07-18 13:20:40 -07:00
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
ee3423d709
extended fuzzer MT memory tests
2017-07-10 14:09:16 -07:00
f9524cf366
added --memtest to fuzzer
2017-07-10 13:48:41 -07:00
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
811deaea6f
Merge pull request #736 from terrelln/cover-default-api
...
[zdict] Make COVER the default algorithm
2017-06-28 20:25:36 -07:00
5b7fd7c422
[zdict] Make COVER the default algorithm
2017-06-26 21:09:22 -07:00
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
ff8f83bd47
fixed fuzzer test
2017-06-20 12:17:32 -07:00
cdf7e82222
Added ZSTD_initStaticCDict()
2017-05-25 18:05:49 -07:00
57827f906f
added ZSTD_initStaticDDict()
2017-05-25 15:44:06 -07:00
25989e361c
updated ZSTD_estimate?DictSize() to pass parameter byReference
...
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
0fdc71c3dc
added ZSTD_initStaticDCtx()
2017-05-24 17:41:41 -07:00
c7fe262dc9
added ZSTD_initStaticCCtx()
...
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
2017-05-23 13:20:41 -07:00
fa3671eac7
changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
...
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
2017-05-19 10:51:30 -07:00
a5ffe3d370
pushed enum values for strategy by one (ZSTD_fast==1)
...
this makes it possible to use `0` to mean:
"do not change strategy"
2017-05-12 16:29:19 -07:00
542c9dfcf8
changed name frameParams into frameHeader
...
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()
The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.
ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
2017-05-09 15:46:07 -07:00
a1d6704d7f
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
...
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
2017-05-08 17:51:49 -07:00
a51cab6e68
Merge pull request #678 from facebook/apiChange
...
Breaking API Change around CDict
2017-04-28 10:02:45 -07:00
c6915429f2
shortened Appveyor release tests
...
fuzzer supports time suffix `s` for "seconds"
2017-04-27 16:24:53 -07:00
2f73427d35
added test for ZSTD_compress_usingCDict_advanced()
2017-04-27 14:39:39 -07:00
7321345fd2
fixed another VS2008 init error
2017-04-27 14:19:34 -07:00
f4bd857d81
created ZSTD_compress_usingCDict_advanced()
2017-04-27 11:31:55 -07:00
31533bacce
Changed ZSTD_createCDict_advanced()
...
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.
Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
4f818182b8
clarified frame parameters for ZSTD_compress*_usingCDict()
...
created ZSTD_compressBegin_usingCDict_internal(),
which gives direct control to frame Parameters.
ZSTD_resetCStream_internal() now points into it.
2017-04-17 18:29:06 -07:00
f913cbed33
fixed : memory leak in fuzzer test
2017-04-13 22:46:51 -07:00
20d5e03893
content size is controlled at bufferless level
...
so it's active for all entry points
Also : added relevant test (wrong content size) in fuzzer
2017-04-11 18:34:02 -07:00
39a6cc5172
Make ZSTD_compress_usingCDict() respect contentSizeFlag
2017-04-03 21:09:55 -07:00
1e1e26f2ac
fixed #634 : ZSTDMT_compressCCtx() doesn't provide frame content size in header
2017-03-29 17:09:59 -07:00
0ef6803707
added ZSTD_getFrameContentSize() to ZSTDMT unit test in fuzzer
...
Now tests fail, because ZSTDMT_compress() doesn't fill frame content size correctly.
2017-03-29 16:58:57 -07:00
042ba122ae
Change g_displayLevel to int and fix DISPLAYUPDATE flush
2017-03-23 11:21:59 -07:00
7ebf2de02d
Add ability to strongly limit fuzzer test size with flag
2017-03-20 11:25:00 -07:00
f5e50519e0
Prevent fuzz testers from combining large dicts with high clevel
2017-03-15 15:04:54 -07:00
c0b1731bce
added test for decompression with NULL dict and NULL DDict
...
previous version of ZSTD_decompressMultiFrame() would fail that test
2017-02-28 01:02:46 -08:00
bd7fa21deb
added ZSTD_refDDict()
...
Now DDict does no longer depends on DCtx duplication
2017-02-26 14:43:07 -08:00
8dff956dbf
Added DDict unit test in fuzzer
...
also : slightly modified loadEntropy :
know src must points at start of dictionary
2017-02-25 10:11:15 -08:00
9050e1925e
Change name to to findFrameCompressedSize and add skippable support
2017-02-22 12:12:34 -08:00
5069b6c2c3
Merge branch 'dev' into multiframe
2017-02-10 10:08:55 -08:00
84b37cc1f1
Fix failing unit test
2017-02-09 12:27:39 -08:00
e0b3265e87
Fix ZSTD_getErrorString and add tests
2017-02-08 17:28:49 -08:00
ba2ad9f25c
ZSTD_decompress now handles multiple frames
2017-02-08 14:50:10 -08:00
4e709712e1
Decompressed size functions now handle multiframes and distinguish cases
...
- Add ZSTD_findDecompressedSize
- Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
- Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
2017-02-08 14:50:10 -08:00
06e7697f96
added test of new parameter ZSTD_p_forceWindow
2017-01-25 16:39:03 -08:00
c44c4d5223
Fix missing 'OK' logging on fuzzer testcase
2017-01-12 09:38:33 -08:00
834ab50fa3
Fixed decompress_usingDict not propagating corrupted dictionary error
2017-01-11 17:31:34 -08:00
a8b4fe0481
Add COVER dictionary builder to fuzzer unit tests
2017-01-02 18:45:19 -08:00
f586bdfe23
fixed fuzzer test
2016-12-06 06:11:46 +01:00
e7a41a5955
added : dictID retrieval functions.
...
added : unit tests for dictID retrieval functions
2016-12-05 16:21:06 -08:00
4c0b44f8ce
minor display improvement in fuzzer when nb of tests is very large
2016-11-01 11:13:22 -07:00
a17fd7312a
changed error_public.h into zstd_errors.h
2016-10-11 16:41:09 -07:00
97b378a6f8
Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize.
2016-09-21 17:20:19 +02:00
64deef3bee
Fixed srcSize=1
2016-09-14 00:16:07 +02:00
a7737f6a60
improved compression on small files when using same parameters
2016-09-06 09:44:59 +02:00
2b26ad1947
removed timeb.h ( #319 )
2016-09-02 15:34:41 -07:00
4ded9e591c
added boilerplate
2016-08-30 11:06:28 -07:00
cb3276329a
added sizeof CStream and DStream
2016-08-23 00:31:59 +02:00
da3fbcb302
Added ZDICT_getDictID()
2016-08-19 14:23:58 +02:00
2868d565e4
test-related files moved to tests/
2016-08-18 13:18:11 +02:00
Yann Collet
Nick Terrell
Yann Collet
Sean Purcell
Yann Collet
inikep