Nick Terrell
f9c9af3c2e
Reject dictionaries with incomplete entropy tables
...
If a dictionary specifies that a symbol has probability zero in its
`matchLength`, `literalLength`, or `offset` FSE table, but the symbol
appears when compressing input, the compressor fails.
Ensure that dictionaries support all `matchLength`, and `literalLength`
codes. They must also support all of the `offset` codes required to
represent every possible offset that can appear in the first block.
2016-10-24 10:42:44 -07:00
Przemyslaw Skibinski
3ee94a7600
zstd_compression_format.md moved to doc/
2016-10-24 15:58:07 +02:00
Nick Terrell
bfd943ace5
Fix buffer overrun in ZSTD_loadDictEntropyStats()
...
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadDictEntropyStats()`. This caused `FSE_buildCTable()`
to stack/heap overflow in a few places.
The benchmarks look good, there is no obvious compression performance regression:
> ./zstds/zstd.opt.0 -i10 -b1 -e10 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 271.6 MB/s , 716.8 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 204.8 MB/s , 671.1 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 156.8 MB/s , 658.6 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 136.4 MB/s , 665.3 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 98.9 MB/s , 647.0 MB/s
6#silesia.tar : 211988480 -> 62979643 (3.366), 65.2 MB/s , 670.4 MB/s
7#silesia.tar : 211988480 -> 61974560 (3.421), 44.9 MB/s , 688.2 MB/s
8#silesia.tar : 211988480 -> 61028308 (3.474), 32.4 MB/s , 711.9 MB/s
9#silesia.tar : 211988480 -> 60416751 (3.509), 21.1 MB/s , 718.1 MB/s
10#silesia.tar : 211988480 -> 60174239 (3.523), 22.2 MB/s , 721.8 MB/s
> ./compress_zstds/zstd.opt.1 -i10 -b1 -e10 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 273.8 MB/s , 722.0 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 203.2 MB/s , 666.6 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 157.4 MB/s , 666.5 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 132.1 MB/s , 661.9 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 96.8 MB/s , 641.6 MB/s
6#silesia.tar : 211988480 -> 62979643 (3.366), 63.1 MB/s , 677.0 MB/s
7#silesia.tar : 211988480 -> 61974560 (3.421), 44.3 MB/s , 678.2 MB/s
8#silesia.tar : 211988480 -> 61028308 (3.474), 33.1 MB/s , 708.9 MB/s
9#silesia.tar : 211988480 -> 60416751 (3.509), 21.5 MB/s , 710.1 MB/s
10#silesia.tar : 211988480 -> 60174239 (3.523), 21.9 MB/s , 723.9 MB/s
2016-10-17 16:55:52 -07:00
Yann Collet
2b361cf2f1
minor opt
2016-10-14 16:09:07 -07:00
Nick Terrell
3b9cdf9220
Fix ubsan failures (pass NULL to memcpy)
2016-10-12 20:54:42 -07:00
Yann Collet
cf409a7e2a
fixed : init*_advanced() followed by reset() with different pledgedSrcSiz
2016-09-26 16:41:05 +02:00
Yann Collet
97b378a6f8
Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize.
2016-09-21 17:20:19 +02:00
Yann Collet
993060e0f2
cli : better adaptation to small files
2016-09-21 16:46:08 +02:00
Yann Collet
a6bdf55759
fixed memory leak
2016-09-15 17:02:06 +02:00
Yann Collet
644a8da88a
fixed minor conversion warning
2016-09-15 16:16:21 +02:00
Yann Collet
4cb212938c
introduced ZSTD_resetCStream()
2016-09-15 14:54:07 +02:00
Yann Collet
fa0c09760c
variable renaming
2016-09-15 14:11:01 +02:00
Yann Collet
d7c6589df8
support ZSTD_sizeof_*() on NULL
...
added ZSTD_sizeof_CDict()
2016-09-15 02:57:27 +02:00
Yann Collet
64deef3bee
Fixed srcSize=1
2016-09-14 00:16:07 +02:00
Yann Collet
ac8bace6b1
support large skippable frames
2016-09-07 14:54:23 +02:00
Yann Collet
95d07d7447
introduced CHECK_E
2016-09-06 16:38:51 +02:00
Yann Collet
3e21ec5b01
introduced CHECK_F
2016-09-06 15:36:19 +02:00
Yann Collet
5c956d593c
FORCE_INLINE common definition
2016-09-06 15:05:19 +02:00
Yann Collet
edbcd9f5b2
fixed zbufftest
2016-09-06 14:30:57 +02:00
Yann Collet
b624922b14
fixed checksum
2016-09-06 11:16:57 +02:00
Yann Collet
a7737f6a60
improved compression on small files when using same parameters
2016-09-06 09:44:59 +02:00
Yann Collet
7ae67bb18a
small compression speed gains with using_CDict
2016-09-06 06:28:05 +02:00
Yann Collet
855766d73d
clarified dictionary in format description
2016-09-02 17:04:49 -07:00
Yann Collet
1563bfeabc
fixing FORCE_INLINE for older compilers ( #330 )
2016-09-02 11:44:21 -07:00
David Lam
e10f7f3dcb
merge
2016-08-30 12:03:36 -07:00
Yann Collet
4ded9e591c
added boilerplate
2016-08-30 11:06:28 -07:00
Yann Collet
14200a20f0
Fixed issue #304 , reported by @borzunov
2016-08-30 06:51:00 -07:00
David Lam
da9d3b7057
Cleanup some errors in typedef comments and remove duplicated HOWTO from zbuff_decompress.c
2016-08-29 17:31:51 -07:00
Yann Collet
23b6e05d8e
ZSTD_malloc() and ZSTD_free(), to simplify customMem
2016-08-28 21:05:43 -07:00
Yann Collet
e19a9ef05d
update compression level table
2016-08-26 20:02:49 +02:00
Yann Collet
87c18b2ebd
fixed multiple minor warnings for XCode
2016-08-26 01:43:47 +02:00
Yann Collet
0d59a6f73a
removed debug strings
2016-08-25 22:42:46 +02:00
Yann Collet
5a02b69215
reinforced fix for huge files
2016-08-25 22:24:59 +02:00
Yann Collet
96bdd87de4
fixed : compression bug on very large files
2016-08-25 14:34:42 +02:00
inikep
a3a47ec4d0
Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into Other
2016-08-24 21:25:49 +02:00
Yann Collet
a2cdffe556
fixed wrong parameter issue
2016-08-24 19:42:15 +02:00
inikep
e416e30019
remove unnecessary comments
2016-08-24 17:32:09 +02:00
inikep
4e90f6c1e0
removed ZSTD_LOG_ENCODE and ZSTD_LOG_BLOCK
2016-08-24 17:24:11 +02:00
inikep
8a36f8527c
removed stats in debug mode
2016-08-24 17:19:12 +02:00
Yann Collet
24b68a5b23
update cLevel table for 256KB
2016-08-24 14:22:26 +02:00
Yann Collet
c54692faeb
improved level 3
2016-08-24 01:45:46 +02:00
Yann Collet
70e3b31306
fixed playtests on os-x
2016-08-23 01:18:06 +02:00
Yann Collet
cb3276329a
added sizeof CStream and DStream
2016-08-23 00:31:59 +02:00
Yann Collet
c411902230
fixed g++ conversion warning
2016-08-17 01:50:54 +02:00
Yann Collet
53e17fbd5e
updated streaming API
2016-08-17 01:39:22 +02:00
Yann Collet
104e5b072d
added : streaming decompression API
2016-08-16 15:11:28 +02:00
Yann Collet
5a0c8e2439
new streaming API (compression)
2016-08-16 15:11:27 +02:00
inikep
98e08cbe34
fixed: tree not updated after finding very long rep matches
2016-08-10 15:00:30 +02:00
Yann Collet
280f9a8754
minor comment
2016-08-08 00:44:00 +02:00
Yann Collet
0763905f44
ZSTD_compress_usingCDict() correctly provides original size by default in frame header
...
Fixed dictionary examples
2016-08-03 01:57:57 +02:00
