e70912c72b
Changed : input divided into roughly equal parts.
...
Debug : can measure time waiting for mutexes to unlock.
2016-12-29 01:24:01 +01:00
6c0ed9483a
compression threads use ZSTD_compressCCtx()
2016-12-28 17:08:28 +01:00
ce9e1452fd
protect buffer pool with a mutex
2016-12-28 15:31:19 +01:00
3d93f2fce7
first zstdmt sketch
2016-12-27 07:19:36 +01:00
aca113f4f5
fixed ZSTD_sizeof_?Dict()
2016-12-23 22:25:03 +01:00
4e5eea61a8
added ZSTD_createDDict_byReference()
2016-12-21 16:44:35 +01:00
1f57c2ed32
added : ZSTD_createCDict_byReference()
2016-12-21 16:20:11 +01:00
8157a4c3cc
Fix dictionary loading bug causing an MSAN failure
...
Offset rep codes must be in the range `[1, dictSize)`.
Fix dictionary loading to reject `0` as a offset rep code.
2016-12-20 10:47:52 -08:00
d564faa3c6
fix : ZSTD_initCStream_srcSize() correctly set srcSize in frame header
2016-12-18 21:39:15 +01:00
e795c8a5f6
Added ZSTD_initCStream_srcSize().
...
Added relevant test cases in zstreamtest
2016-12-13 17:00:14 +01:00
c3a5c4bef8
introduced cycleLog
2016-12-12 00:47:30 +01:00
c261f71f6a
minor variation of rescale fix
2016-12-12 00:25:07 +01:00
3826207a70
Simplify segfault fix
...
Take advantage of the fact that `chainLog <= windowLog`.
2016-12-10 18:46:55 -08:00
0012332ce0
Fix compression segfault
...
When the overflow protection kicks in, it makes sure that ip - ctx->base
isn't too large. However, it didn't ensure that saved offsets are
still valid. This change ensures that any valid offsets (<= windowLog)
are still representable after the update.
The bug would shop up on line 1056, when `offset_1 > current + 1`, which
causes an underflow. This in turn, would cause a segfault on line 1063.
The input must necessarily be longer than 1 GB for this issue to occur.
Even then, it only occurs if one of the last 3 matches is larger than
the chain size and block size.
2016-12-09 17:15:33 -08:00
825dffbc43
moved zbuff source files into lib/deprecated
2016-12-05 19:28:19 -08:00
a0d742b1e4
introduced HUF_buildCTable_wksp(), to reduce stack memory usage
2016-12-01 17:47:30 -08:00
643d9a234b
replaced usage of FSE_buildCTable by FSE_buildCTable_wksp, using less stack space in the process
2016-12-01 16:24:04 -08:00
e928f7e16d
introduced ext_wksp variants of count to reduce stack memory usage
2016-12-01 16:13:35 -08:00
979cab412b
fixed some minor visual silent cast warnings.
...
introduced FSE_count_parallel_wksp().
2016-11-30 18:10:38 -08:00
5e00b848a8
FSE_compress_wksp() uses less stack space
2016-11-30 16:46:13 -08:00
d79a9a00d9
Introduced FSE_compress_wksp() and FSE_buildCTable_wksp() to reduce stack memory usage
2016-11-30 15:52:20 -08:00
25f46dcc0f
minor const
2016-11-29 16:59:27 -08:00
fc4193bda5
fixed g++ warnings
2016-11-23 18:17:18 +01:00
9ca65af810
zstd_opt.h: improved price function
2016-11-23 17:22:54 +01:00
ad3e94512c
fixed warnings from static analyzer in zstd_opt.h
2016-11-21 20:22:12 +01:00
3d18088b38
updated windres
2016-11-17 18:04:41 +01:00
407a11f63e
fixed Visual compatibility
2016-11-03 15:52:01 -07:00
d82efd8a70
ZSTD_compress_usingDict() when dict gets loaded
...
Specify that when `dict == NULL || dictSize < 8` no dictionary
gets loaded.
Also add some periods.
2016-11-02 18:07:16 -07:00
ee5b725823
ZSTD_initCStream() optimization : do not allocate a CDict when no dictionary used
2016-10-27 14:20:55 -07:00
335ad5d4d4
added ZSTD_initDStream_usingDDict() .
...
slightly optimized ZSTD_initDStream() when no dictionary .
fixed ZSTD_sizeof_CStream() .
2016-10-25 17:47:02 -07:00
9516234e67
first sketch for ZSTD_initCStream_usingCDict()
2016-10-25 16:19:52 -07:00
62d9a7ddfd
Merge pull request #429 from inikep/btopt2
...
Btopt2
2016-10-25 14:48:43 -07:00
5c5f01f3da
added ZSTD_btopt2 strategy
2016-10-25 12:25:07 +02:00
b2c39a22b0
Fix compiler narrowing warning
2016-10-24 14:50:13 -07:00
f698ad6deb
Merge remote-tracking branch 'upstream/dev' into fixes
...
* upstream/dev:
added doc\zstd_manual.html
added contrib\gen_html
zstd_compression_format.md moved to doc/
Fix small bug in ZSTD_execSequence()
improved ZSTD_compressBlock_opt_extDict_generic
protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
zstd_opt.h: small improvement in compression ratio
improved dicitonary segment merge
use implicit rules to compile zstd_decompress.c
detect early impossible decompression scenario in legacy decoder v0.5
no repeat mode in legacy v0.5
fixed invalid invocation of dictionary in legacy decoder v0.5
fix edge case
fix command line interpretation
fixed minor corner case
zstd.h: added the Introduction section
fixed clang 3.5 warnings
zstd.h: updated comments
2016-10-24 13:10:13 -07:00
f9c9af3c2e
Reject dictionaries with incomplete entropy tables
...
If a dictionary specifies that a symbol has probability zero in its
`matchLength`, `literalLength`, or `offset` FSE table, but the symbol
appears when compressing input, the compressor fails.
Ensure that dictionaries support all `matchLength`, and `literalLength`
codes. They must also support all of the `offset` codes required to
represent every possible offset that can appear in the first block.
2016-10-24 10:42:44 -07:00
3ee94a7600
zstd_compression_format.md moved to doc/
2016-10-24 15:58:07 +02:00
4732074a71
improved ZSTD_compressBlock_opt_extDict_generic
2016-10-21 11:19:00 +02:00
d365ae3497
zstd_opt.h: small improvement in compression ratio
2016-10-20 11:49:02 +02:00
197a55ee7b
fix edge case
2016-10-18 11:27:52 -07:00
fd98087047
Fix stack buffer overflow in HUF_readCTable()
...
If `w ==0` on line 153, then `CTable[n].nbBits == tableLog + 1`.
Then `nbPerRank[CTable[n].nbBits]` and `valPerRank[CTable[n].nbBits]`
are stack buffer overflows.
2016-10-17 18:16:59 -07:00
bfd943ace5
Fix buffer overrun in ZSTD_loadDictEntropyStats()
...
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadDictEntropyStats()`. This caused `FSE_buildCTable()`
to stack/heap overflow in a few places.
The benchmarks look good, there is no obvious compression performance regression:
> ./zstds/zstd.opt.0 -i10 -b1 -e10 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 271.6 MB/s , 716.8 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 204.8 MB/s , 671.1 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 156.8 MB/s , 658.6 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 136.4 MB/s , 665.3 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 98.9 MB/s , 647.0 MB/s
6#silesia.tar : 211988480 -> 62979643 (3.366), 65.2 MB/s , 670.4 MB/s
7#silesia.tar : 211988480 -> 61974560 (3.421), 44.9 MB/s , 688.2 MB/s
8#silesia.tar : 211988480 -> 61028308 (3.474), 32.4 MB/s , 711.9 MB/s
9#silesia.tar : 211988480 -> 60416751 (3.509), 21.1 MB/s , 718.1 MB/s
10#silesia.tar : 211988480 -> 60174239 (3.523), 22.2 MB/s , 721.8 MB/s
> ./compress_zstds/zstd.opt.1 -i10 -b1 -e10 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 273.8 MB/s , 722.0 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 203.2 MB/s , 666.6 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 157.4 MB/s , 666.5 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 132.1 MB/s , 661.9 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 96.8 MB/s , 641.6 MB/s
6#silesia.tar : 211988480 -> 62979643 (3.366), 63.1 MB/s , 677.0 MB/s
7#silesia.tar : 211988480 -> 61974560 (3.421), 44.3 MB/s , 678.2 MB/s
8#silesia.tar : 211988480 -> 61028308 (3.474), 33.1 MB/s , 708.9 MB/s
9#silesia.tar : 211988480 -> 60416751 (3.509), 21.5 MB/s , 710.1 MB/s
10#silesia.tar : 211988480 -> 60174239 (3.523), 21.9 MB/s , 723.9 MB/s
2016-10-17 16:55:52 -07:00
2b361cf2f1
minor opt
2016-10-14 16:09:07 -07:00
3b9cdf9220
Fix ubsan failures (pass NULL to memcpy)
2016-10-12 20:54:42 -07:00
cf409a7e2a
fixed : init*_advanced() followed by reset() with different pledgedSrcSiz
2016-09-26 16:41:05 +02:00
97b378a6f8
Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize.
2016-09-21 17:20:19 +02:00
993060e0f2
cli : better adaptation to small files
2016-09-21 16:46:08 +02:00
a6bdf55759
fixed memory leak
2016-09-15 17:02:06 +02:00
644a8da88a
fixed minor conversion warning
2016-09-15 16:16:21 +02:00
4cb212938c
introduced ZSTD_resetCStream()
2016-09-15 14:54:07 +02:00
Yann Collet
Nick Terrell
Przemyslaw Skibinski
Yann Collet
Yann Collet