b2c39a22b0
Fix compiler narrowing warning
2016-10-24 14:50:13 -07:00
f698ad6deb
Merge remote-tracking branch 'upstream/dev' into fixes
...
* upstream/dev:
added doc\zstd_manual.html
added contrib\gen_html
zstd_compression_format.md moved to doc/
Fix small bug in ZSTD_execSequence()
improved ZSTD_compressBlock_opt_extDict_generic
protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
zstd_opt.h: small improvement in compression ratio
improved dicitonary segment merge
use implicit rules to compile zstd_decompress.c
detect early impossible decompression scenario in legacy decoder v0.5
no repeat mode in legacy v0.5
fixed invalid invocation of dictionary in legacy decoder v0.5
fix edge case
fix command line interpretation
fixed minor corner case
zstd.h: added the Introduction section
fixed clang 3.5 warnings
zstd.h: updated comments
2016-10-24 13:10:13 -07:00
4239a207dd
Merge pull request #425 from inikep/dev11
...
Doc
2016-10-24 11:11:40 -07:00
f9c9af3c2e
Reject dictionaries with incomplete entropy tables
...
If a dictionary specifies that a symbol has probability zero in its
`matchLength`, `literalLength`, or `offset` FSE table, but the symbol
appears when compressing input, the compressor fails.
Ensure that dictionaries support all `matchLength`, and `literalLength`
codes. They must also support all of the `offset` codes required to
represent every possible offset that can appear in the first block.
2016-10-24 10:42:44 -07:00
984b66cd72
added contrib\gen_html
2016-10-24 15:59:51 +02:00
3ee94a7600
zstd_compression_format.md moved to doc/
2016-10-24 15:58:07 +02:00
97611611a3
Merge pull request #423 from terrelln/exec-seq-patch
...
Fix small bug in ZSTD_execSequence()
2016-10-21 17:02:06 -07:00
ae1cb3b3d0
Fix small bug in ZSTD_execSequence()
...
`memmove(op, match, sequence.matchLength)` is not the desired behavior.
Overlap is allowed, and handled as if we did `*op++ = *match++`, which
is not how `memmove()` handles overlap.
Only triggered if both of the following conditions are met:
* The match spans extDict & currentPrefixSegment
* `oLitEnd <= oend_w < oLitEnd + length1 < oMatchEnd <= oend`.
These two conditions imply that the block is less than 15 bytes long.
This bug isn't triggered by the streaming API, because it allocates
enough space for the window size + the block size, so there cannot be
a match that is within 8 bytes of the end and overlaps with itself.
It cannot be triggered by the block decompression API because all of
the decompressed data is in the currentPrefixSegment.
Introduced by commit 7158584399
2016-10-21 12:13:44 -07:00
4732074a71
improved ZSTD_compressBlock_opt_extDict_generic
2016-10-21 11:19:00 +02:00
da3bd8b6de
protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
2016-10-20 20:11:00 -07:00
d365ae3497
zstd_opt.h: small improvement in compression ratio
2016-10-20 11:49:02 +02:00
575ab00db7
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-10-20 11:01:52 +02:00
d760529a05
Fix stack buffer overrun when weightTotal == 0
...
If `weightTotal == 0`, then `BIT_highbit32(weightTotal)` is
undefined behavior in the case that it calls `__builtin_clz()`.
If `tableLog == HUF_TABLELOG_ABSOLUTEMAX` then we will access one
byte beyond the end of the buffer.
2016-10-19 11:39:11 -07:00
bb68062c59
Unitialized memory read in ZSTD_decodeSeqHeaders()
...
Caused by two things:
1. Not checking that `ip` is in range except for the first byte.
2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.
2016-10-18 16:41:33 -07:00
52c1bf93fe
improved dicitonary segment merge
2016-10-18 16:34:58 -07:00
7b06ad7a05
Backport fix from commit 125d817
...
This fixes a read of unitialized memory.
Full commit hash: 125d81774f
2016-10-18 14:52:34 -07:00
f45b157d95
Backport fix from commit 9e8b09a
...
Fixes uninitialized memory reads.
Full commit hash: 9e8b09a7bd
2016-10-18 14:22:49 -07:00
f7906d5955
detect early impossible decompression scenario in legacy decoder v0.5
2016-10-18 13:48:32 -07:00
9313c8d953
no repeat mode in legacy v0.5
2016-10-18 13:36:15 -07:00
83d7bdee4b
fixed invalid invocation of dictionary in legacy decoder v0.5
2016-10-18 12:25:43 -07:00
197a55ee7b
fix edge case
2016-10-18 11:27:52 -07:00
fd98087047
Fix stack buffer overflow in HUF_readCTable()
...
If `w ==0` on line 153, then `CTable[n].nbBits == tableLog + 1`.
Then `nbPerRank[CTable[n].nbBits]` and `valPerRank[CTable[n].nbBits]`
are stack buffer overflows.
2016-10-17 18:16:59 -07:00
06573e17be
fixed minor corner case
2016-10-17 17:28:28 -07:00
bfd943ace5
Fix buffer overrun in ZSTD_loadDictEntropyStats()
...
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadDictEntropyStats()`. This caused `FSE_buildCTable()`
to stack/heap overflow in a few places.
The benchmarks look good, there is no obvious compression performance regression:
> ./zstds/zstd.opt.0 -i10 -b1 -e10 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 271.6 MB/s , 716.8 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 204.8 MB/s , 671.1 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 156.8 MB/s , 658.6 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 136.4 MB/s , 665.3 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 98.9 MB/s , 647.0 MB/s
6#silesia.tar : 211988480 -> 62979643 (3.366), 65.2 MB/s , 670.4 MB/s
7#silesia.tar : 211988480 -> 61974560 (3.421), 44.9 MB/s , 688.2 MB/s
8#silesia.tar : 211988480 -> 61028308 (3.474), 32.4 MB/s , 711.9 MB/s
9#silesia.tar : 211988480 -> 60416751 (3.509), 21.1 MB/s , 718.1 MB/s
10#silesia.tar : 211988480 -> 60174239 (3.523), 22.2 MB/s , 721.8 MB/s
> ./compress_zstds/zstd.opt.1 -i10 -b1 -e10 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 273.8 MB/s , 722.0 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 203.2 MB/s , 666.6 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 157.4 MB/s , 666.5 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 132.1 MB/s , 661.9 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 96.8 MB/s , 641.6 MB/s
6#silesia.tar : 211988480 -> 62979643 (3.366), 63.1 MB/s , 677.0 MB/s
7#silesia.tar : 211988480 -> 61974560 (3.421), 44.3 MB/s , 678.2 MB/s
8#silesia.tar : 211988480 -> 61028308 (3.474), 33.1 MB/s , 708.9 MB/s
9#silesia.tar : 211988480 -> 60416751 (3.509), 21.5 MB/s , 710.1 MB/s
10#silesia.tar : 211988480 -> 60174239 (3.523), 21.9 MB/s , 723.9 MB/s
2016-10-17 16:55:52 -07:00
4db751668f
Fix buffer overrun in ZSTD_loadEntropy()
...
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadEntropy()`. This caused `FSE_buildDTable(dctx->MLTable, ...)`
to overwrite the beginning of `dctx->hufTable`.
The benchmarks look good, there is no obvious performance regression:
> ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 268.2 MB/s , 701.0 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 199.5 MB/s , 666.9 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 154.9 MB/s , 655.6 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 128.9 MB/s , 648.4 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 98.4 MB/s , 633.4 MB/s
> ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar
1#silesia.tar : 211988480 -> 73656930 (2.878), 266.1 MB/s , 703.7 MB/s
2#silesia.tar : 211988480 -> 70162842 (3.021), 199.0 MB/s , 666.6 MB/s
3#silesia.tar : 211988480 -> 66997986 (3.164), 156.2 MB/s , 656.2 MB/s
4#silesia.tar : 211988480 -> 66002591 (3.212), 133.2 MB/s , 647.4 MB/s
5#silesia.tar : 211988480 -> 65008480 (3.261), 96.3 MB/s , 633.3 MB/s
2016-10-17 15:51:15 -07:00
ccfcc643da
Check if dict is empty before reading first byte
2016-10-17 11:46:03 -07:00
2b361cf2f1
minor opt
2016-10-14 16:09:07 -07:00
7933434fdf
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-10-14 13:32:35 -07:00
d4cda27b63
new command -M#, to limit memory usage during decompression ( #403 )
2016-10-14 13:32:20 -07:00
3b9cdf9220
Fix ubsan failures (pass NULL to memcpy)
2016-10-12 20:54:42 -07:00
5d919e7ac3
added ZSTD_error_frameParameter_windowTooLarge ( #403 )
2016-10-12 17:29:24 -07:00
e19111c42f
make creates libzstd binaries ( #415 )
2016-10-12 11:09:36 -07:00
8b70d012f0
fix cmake
2016-10-12 10:23:53 -07:00
38fb0dc4cf
Merge pull request #416 from terrelln/exec-sequence
...
Fix ZSTD_execSequence() edge case
2016-10-12 10:17:53 -07:00
7158584399
Fix ZSTD_execSequence() edge case
2016-10-12 10:05:26 -07:00
f52cd03e73
bumped version number
2016-10-11 17:29:27 -07:00
ef2357d0d3
created error_private.c, so that a single list of error strings get included
2016-10-11 17:24:50 -07:00
14efab827b
added zstd_errors.h to include installation
2016-10-11 16:51:29 -07:00
a17fd7312a
changed error_public.h into zstd_errors.h
2016-10-11 16:41:09 -07:00
18b51b99c0
sync fse
2016-10-11 08:21:09 -07:00
2d2613399a
zstd.h: added the Introduction section
2016-10-06 16:28:21 +02:00
ba1db376ac
fixed clang 3.5 warnings
2016-10-06 14:22:48 +02:00
82057aa7ec
zstd.h: updated comments
2016-10-06 13:23:52 +02:00
df6797447f
update dictionary builder warning comments
2016-09-27 15:14:32 +02:00
47094ea66b
added comment on filePos
2016-09-26 18:03:33 +02:00
cf409a7e2a
fixed : init*_advanced() followed by reset() with different pledgedSrcSiz
2016-09-26 16:41:05 +02:00
2f2639438a
zstreamtest can fuzztest pledgedSrcSize
2016-09-26 14:06:08 +02:00
dc245e91cb
Changed to use ZSTDLIBv06_API and ZSTDLIBv07_API for DLL exports to fix warning
...
- changed name to prevent collision with ZSTDLIB_API used by non-legacy dll exports
2016-09-23 17:09:36 +02:00
21412bb3f6
Merge branch 'dev' of github.com:Cyan4973/zstd into dev
2016-09-22 15:57:56 +02:00
51f4d566c2
small decompression speed boost for very small data
2016-09-22 15:57:28 +02:00
97b378a6f8
Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize.
2016-09-21 17:20:19 +02:00
993060e0f2
cli : better adaptation to small files
2016-09-21 16:46:08 +02:00
1eb2fdc74f
bumped version number
2016-09-18 12:21:47 +02:00
a6bdf55759
fixed memory leak
2016-09-15 17:02:06 +02:00
644a8da88a
fixed minor conversion warning
2016-09-15 16:16:21 +02:00
4cb212938c
introduced ZSTD_resetCStream()
2016-09-15 14:54:07 +02:00
fa0c09760c
variable renaming
2016-09-15 14:11:01 +02:00
d7c6589df8
support ZSTD_sizeof_*() on NULL
...
added ZSTD_sizeof_CDict()
2016-09-15 02:57:27 +02:00
e91c4b4cef
introduced ZSTD_resetDStream() .
...
added : ZSTD_sizeof_DDict()
2016-09-14 16:55:44 +02:00
d092d77cfc
minor variable renaming
2016-09-14 16:14:57 +02:00
64deef3bee
Fixed srcSize=1
2016-09-14 00:16:07 +02:00
26ec254066
new strategy for faster DDict decompression
2016-09-13 16:52:16 +02:00
ac175d46d4
updated comments
2016-09-13 00:51:47 +02:00
a3481d6de0
make uninstall
2016-09-12 05:04:26 +02:00
b3060f7a9e
changed streaming decoder behavior : now, when all compressed frame is consumed, it means decompression is completed, with regenerated data fully flushed.
2016-09-09 16:44:16 +02:00
01c199226a
updated decompression streaming example
2016-09-08 19:29:04 +02:00
5c6d244973
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-09-07 14:54:54 +02:00
ac8bace6b1
support large skippable frames
2016-09-07 14:54:23 +02:00
0e07bf3f60
added comments on searchLength min / max ( #337 )
2016-09-07 06:33:02 +02:00
95d07d7447
introduced CHECK_E
2016-09-06 16:38:51 +02:00
3e21ec5b01
introduced CHECK_F
2016-09-06 15:36:19 +02:00
5c956d593c
FORCE_INLINE common definition
2016-09-06 15:05:19 +02:00
edbcd9f5b2
fixed zbufftest
2016-09-06 14:30:57 +02:00
b624922b14
fixed checksum
2016-09-06 11:16:57 +02:00
a7737f6a60
improved compression on small files when using same parameters
2016-09-06 09:44:59 +02:00
7ae67bb18a
small compression speed gains with using_CDict
2016-09-06 06:28:05 +02:00
1d4208c029
clarified streaming decompression inlined doc
2016-09-06 05:16:40 +02:00
7c83dfd5c2
ZSTD_frameHeaderSize_prefix ( #340 ), as result of ZSTD_initStream
2016-09-05 19:47:43 +02:00
fa72f6bdce
clarified inline doc for streaming
2016-09-05 17:39:56 +02:00
c73a8109bb
Merge pull request #344 from inikep/dev10
...
unified error codes for legacy decoders
2016-09-05 07:46:33 -07:00
45db83f98d
ZSTD_decodeLiteralsBlock renamed to ZSTDv01_decodeLiteralsBlock
2016-09-05 14:46:24 +02:00
476964f6a1
ZSTD_decodeSeqHeaders renamed to ZSTDv01_decodeSeqHeaders
2016-09-05 13:34:57 +02:00
c13faa1b0f
legacy decoders: restored #include <intrin.h> for VC++
2016-09-05 13:25:07 +02:00
8161e7321a
unified error codes for legacy decoders
2016-09-05 12:29:51 +02:00
b85cdabd50
Enable install targets for NetBSD.
2016-09-04 14:37:57 +02:00
33a0465a51
fixed a few links
2016-09-02 22:11:49 -07:00
d56dbc02d3
removed g_displayLevel
2016-09-02 17:28:41 -07:00
855766d73d
clarified dictionary in format description
2016-09-02 17:04:49 -07:00
d725427a3c
g_time => local displayTime
2016-09-02 15:32:39 -07:00
1563bfeabc
fixing FORCE_INLINE for older compilers ( #330 )
2016-09-02 11:44:21 -07:00
7304eb7c09
bumped version number
2016-09-01 15:49:26 -07:00
901e85fe26
version bump
2016-08-31 07:51:25 -07:00
1c59c20903
removed redundant files
2016-08-31 07:15:44 -07:00
599c69d917
minor Makefile updates
2016-08-30 13:33:20 -07:00
e10f7f3dcb
merge
2016-08-30 12:03:36 -07:00
4ded9e591c
added boilerplate
2016-08-30 11:06:28 -07:00
3b15f1f10f
minor refactor
2016-08-30 09:58:50 -07:00
240795bef7
Merge branch 'dev' of github.com:Cyan4973/zstd into dev
2016-08-30 06:51:55 -07:00
14200a20f0
Fixed issue #304 , reported by @borzunov
2016-08-30 06:51:00 -07:00
da9d3b7057
Cleanup some errors in typedef comments and remove duplicated HOWTO from zbuff_decompress.c
2016-08-29 17:31:51 -07:00
Nick Terrell
Yann Collet
Przemyslaw Skibinski
Yann Collet
Yann Collet
Christophe Chevalier
Thomas Klausner
David Lam