Commit Graph

84 Commits (29297c6751289087d3d52688012dc776eccf1d2b)

Author SHA1 Message Date
Nick Terrell 5152fb2cb2 Convert all tabs to spaces 2017-03-29 18:51:58 -07:00
Sean Purcell 9830aeeea6 Fix legacy support=0 case and accidental double include of version headers 2017-03-13 17:19:37 -07:00
Sean Purcell 334cb34edb ZSTD_LEGACY_SUPPORT defines lowest supported version 2017-03-13 14:32:30 -07:00
Yann Collet 4bcc69b761 solves warnings when compiling with global XXH_STATIC_LINKING_ONLY
XXH_STATIC_LINKING_ONLY protection macro is intended to be triggered just before the include.
The main idea is to keep this setting local :
user module shall explicitly understand and accept the static linking restriction
which becomes transparent when triggering the macro at project level.
Global definition also triggers redefinition warnings for user modules which do locally define the macro.

This new version compiles lib and cli without warning when the macro is set globally.
That's not a scenario to be recommended, since it trades a local effect for a global one,
but it was easy enough to provide from zstd side.
2017-03-01 11:33:25 -08:00
Sean Purcell 9050e1925e Change name to to findFrameCompressedSize and add skippable support 2017-02-22 12:12:34 -08:00
Sean Purcell 6b010dec80 execSequence copies up to 2*WILDCOPY_OVERLENGTH extra 2017-02-16 12:05:40 -08:00
Sean Purcell 887eaa9e21 Fix wildcopy overwriting data still in window 2017-02-15 16:43:45 -08:00
Sean Purcell d7bfcac18a Expose frameSrcSize to experimental API 2017-02-10 11:55:44 -08:00
Sean Purcell ba2ad9f25c ZSTD_decompress now handles multiple frames 2017-02-08 14:50:10 -08:00
Sean Purcell 4e709712e1 Decompressed size functions now handle multiframes and distinguish cases
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
2017-02-08 14:50:10 -08:00
Yann Collet b5fd15ccb2 fixed : legacy decoders v04 and v05 2017-01-30 10:45:58 -08:00
Yann Collet cafdd31a38 fixed MSAN warnings in legacy decoders
In some extraordinary circumstances,
*Length field can be generated from reading a partially uninitialized memory segment.
Data is correctly identified as corrupted later on,
but the read taints some later pointer arithmetic operation.
2017-01-27 10:44:03 -08:00
Nick Terrell 8157a4c3cc Fix dictionary loading bug causing an MSAN failure
Offset rep codes must be in the range `[1, dictSize)`.
Fix dictionary loading to reject `0` as a offset rep code.
2016-12-20 10:47:52 -08:00
Yann Collet 35168679bd Merge pull request #478 from terrelln/wildcopy-ub
Fix execSequence wildcopy undefined behavior
2016-12-13 11:33:00 +01:00
Nick Terrell 064a143520 Fix execSequence wildcopy undefined behavior
execSequence relied on pointer overflow to handle cases where
`sequence.matchLength < 8`.  Instead of passing an `size_t` to
wildcopy, pass a `ptrdiff_t`.
2016-12-12 19:01:23 -08:00
Nick Terrell e474aa55b4 Fix decompression buffer overrun
Allows an adversary to write up to 3 bytes beyond the end of the buffer.
Occurs if the match overlaps the `extDict` and `currentPrefix`, and the
match length in the `currentPrefix` is less than `MINMATCH`, and
`op-(16-MINMATCH) >= oMatchEnd > op-16`.
2016-12-12 18:05:30 -08:00
Nick Terrell 4359d21ad7 Merge two memset() calls into one 2016-11-14 17:52:51 -08:00
Nick Terrell 24701de877 Fix uninitialized memory read 2016-11-14 13:57:05 -08:00
Nick Terrell dc904ad17b Fix bug in zstd v0.{5, 6} dictionary decompression
Introduced by bb68062c59.
2016-11-04 16:18:59 -07:00
Nick Terrell f698ad6deb Merge remote-tracking branch 'upstream/dev' into fixes
* upstream/dev:
  added doc\zstd_manual.html
  added contrib\gen_html
  zstd_compression_format.md moved to doc/
  Fix small bug in ZSTD_execSequence()
  improved ZSTD_compressBlock_opt_extDict_generic
  protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
  zstd_opt.h: small improvement in compression ratio
  improved dicitonary segment merge
  use implicit rules to compile zstd_decompress.c
  detect early impossible decompression scenario in legacy decoder v0.5
  no repeat mode in legacy v0.5
  fixed invalid invocation of dictionary in legacy decoder v0.5
  fix edge case
  fix command line interpretation
  fixed minor corner case
  zstd.h: added the Introduction section
  fixed clang 3.5 warnings
  zstd.h: updated comments
2016-10-24 13:10:13 -07:00
Nick Terrell ae1cb3b3d0 Fix small bug in ZSTD_execSequence()
`memmove(op, match, sequence.matchLength)` is not the desired behavior.
Overlap is allowed, and handled as if we did `*op++ = *match++`, which
is not how `memmove()` handles overlap.

Only triggered if both of the following conditions are met:
* The match spans extDict & currentPrefixSegment
* `oLitEnd <= oend_w < oLitEnd + length1 < oMatchEnd <= oend`.

These two conditions imply that the block is less than 15 bytes long.
This bug isn't triggered by the streaming API, because it allocates
enough space for the window size + the block size, so there cannot be
a match that is within 8 bytes of the end and overlaps with itself.
It cannot be triggered by the block decompression API because all of
the decompressed data is in the currentPrefixSegment.

Introduced by commit 7158584399
2016-10-21 12:13:44 -07:00
Nick Terrell d760529a05 Fix stack buffer overrun when weightTotal == 0
If `weightTotal == 0`, then `BIT_highbit32(weightTotal)` is
undefined behavior in the case that it calls `__builtin_clz()`.
If `tableLog == HUF_TABLELOG_ABSOLUTEMAX` then we will access one
byte beyond the end of the buffer.
2016-10-19 11:39:11 -07:00
Nick Terrell bb68062c59 Unitialized memory read in ZSTD_decodeSeqHeaders()
Caused by two things:
1. Not checking that `ip` is in range except for the first byte.
2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.
2016-10-18 16:41:33 -07:00
Nick Terrell 7b06ad7a05 Backport fix from commit 125d817
This fixes a read of unitialized memory.
Full commit hash: 125d81774f.
2016-10-18 14:52:34 -07:00
Nick Terrell f45b157d95 Backport fix from commit 9e8b09a
Fixes uninitialized memory reads.
Full commit hash: 9e8b09a7bd
2016-10-18 14:22:49 -07:00
Yann Collet f7906d5955 detect early impossible decompression scenario in legacy decoder v0.5 2016-10-18 13:48:32 -07:00
Yann Collet 9313c8d953 no repeat mode in legacy v0.5 2016-10-18 13:36:15 -07:00
Yann Collet 83d7bdee4b fixed invalid invocation of dictionary in legacy decoder v0.5 2016-10-18 12:25:43 -07:00
Nick Terrell 4db751668f Fix buffer overrun in ZSTD_loadEntropy()
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadEntropy()`.  This caused `FSE_buildDTable(dctx->MLTable, ...)`
to overwrite the beginning of `dctx->hufTable`.

The benchmarks look good, there is no obvious performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 268.2 MB/s , 701.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.5 MB/s , 666.9 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 154.9 MB/s , 655.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 128.9 MB/s , 648.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.4 MB/s , 633.4 MB/s

  > ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 266.1 MB/s , 703.7 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.0 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.2 MB/s , 656.2 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 133.2 MB/s , 647.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.3 MB/s , 633.3 MB/s
2016-10-17 15:51:15 -07:00
Nick Terrell ccfcc643da Check if dict is empty before reading first byte 2016-10-17 11:46:03 -07:00
Nick Terrell 7158584399 Fix ZSTD_execSequence() edge case 2016-10-12 10:05:26 -07:00
Christophe Chevalier dc245e91cb Changed to use ZSTDLIBv06_API and ZSTDLIBv07_API for DLL exports to fix warning
- changed name to prevent collision with ZSTDLIB_API used by non-legacy dll exports
2016-09-23 17:09:36 +02:00
inikep 45db83f98d ZSTD_decodeLiteralsBlock renamed to ZSTDv01_decodeLiteralsBlock 2016-09-05 14:46:24 +02:00
inikep 476964f6a1 ZSTD_decodeSeqHeaders renamed to ZSTDv01_decodeSeqHeaders 2016-09-05 13:34:57 +02:00
inikep c13faa1b0f legacy decoders: restored #include <intrin.h> for VC++ 2016-09-05 13:25:07 +02:00
inikep 8161e7321a unified error codes for legacy decoders 2016-09-05 12:29:51 +02:00
Yann Collet 1563bfeabc fixing FORCE_INLINE for older compilers (#330) 2016-09-02 11:44:21 -07:00
Yann Collet 4ded9e591c added boilerplate 2016-08-30 11:06:28 -07:00
Yann Collet 5f53b0335e fixed continuation context 2016-08-28 10:00:49 -07:00
Yann Collet 767d8f66fa legacy contexts can be re-used 2016-08-28 08:19:47 -07:00
Yann Collet 4bf317dd00 first version supporting legacy streams (transparent decoding) 2016-08-28 07:43:34 -07:00
Yann Collet 9a021c1aae fixed some minor clang warnings 2016-08-26 09:05:06 +02:00
Yann Collet 87c18b2ebd fixed multiple minor warnings for XCode 2016-08-26 01:43:47 +02:00
inikep 57ef4b1a0d zstd_v07.c: removed unused macros 2016-08-24 17:16:56 +02:00
inikep a7bb322a93 removed never referenced functions 2016-08-18 10:30:21 +02:00
inikep 48849f86f0 fixed compilation with Intel Compiler with Windows 2016-08-10 14:26:35 +02:00
Yann Collet 7adc2328a3 fixed --test on zero-length files, reported by @amnilsson 2016-07-26 15:49:24 +02:00
inikep 4178f5c289 fixed gcc warning: always_inline function might not be inlinable 2016-07-25 21:17:45 +02:00
inikep fca90f8f60 legacy decoder for v0.7 format 2016-07-25 17:49:08 +02:00
Yann Collet d5c5a77990 minor comments clarifications 2016-07-20 13:35:14 +02:00