facebook
/
zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,093 Commits
2 Branches
0 Tags
28 MiB
Commit Graph

142 Commits (03982b5ddb178077ab0b867cc1e993a08d856d00)

Author SHA1 Message Date
Yann Collet 2115724c22 Merge pull request #430 from terrelln/exec-sequences 
ZSTD_execSequence() accepts match in last 7 bytes
 2016-10-28 10:45:05 -07:00
Nick Terrell 10bfd0c0d5 Fix ZSTD_execSequence() performance regression 
Commit ae1cb3b3d0 caused the regression.
It is an instruction alignment issue, because if it is `U64 i` instead
of `U32 i`, the regression returns.  This patch fixes the regression
in gcc, but only gets some of the clang performance back.

Benchmarks:
Run on `silesia.tar`.  I only show levels 1-5 because the performance
regression was uniform across all levels.  I did one run on levels
1-19 and it looked good.

| Build | Level | Before | While | After |
|-------|-------|-------:|------:|------:|
| gcc   |     1 |  931.4 | 904.4 | 932.8 |
| gcc   |     2 |  849.1 | 822.6 | 851.2 |
| gcc   |     3 |  815.6 | 790.6 | 818.9 |
| gcc   |     4 |  794.1 | 770.7 | 798.0 |
| gcc   |     5 |  785.7 | 760.7 | 788.8 |
| clang |     1 |  705.5 | 683.2 | 693.8 |
| clang |     2 |  670.0 | 649.2 | 660.7 |
| clang |     3 |  659.6 | 639.8 | 651.4 |
| clang |     4 |  652.5 | 634.7 | 645.9 |
| clang |     5 |  646.9 | 625.5 | 637.7 |
 2016-10-27 16:19:57 -07:00
Nick Terrell eb7873a048 ZSTD_execSequence() accepts match in last 7 bytes 
The zstd reference compressor will not emit a match in the last 7
bytes of a block.  The decompressor will also not accept a match
in the last 7 bytes.  This patch makes the decompressor accept a
match in the last 7 bytes.
 2016-10-25 21:24:15 -07:00
Yann Collet 335ad5d4d4 added ZSTD_initDStream_usingDDict() . 
slightly optimized ZSTD_initDStream() when no dictionary .
fixed ZSTD_sizeof_CStream() .
 2016-10-25 17:47:02 -07:00
Nick Terrell f698ad6deb Merge remote-tracking branch 'upstream/dev' into fixes 
* upstream/dev:
  added doc\zstd_manual.html
  added contrib\gen_html
  zstd_compression_format.md moved to doc/
  Fix small bug in ZSTD_execSequence()
  improved ZSTD_compressBlock_opt_extDict_generic
  protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
  zstd_opt.h: small improvement in compression ratio
  improved dicitonary segment merge
  use implicit rules to compile zstd_decompress.c
  detect early impossible decompression scenario in legacy decoder v0.5
  no repeat mode in legacy v0.5
  fixed invalid invocation of dictionary in legacy decoder v0.5
  fix edge case
  fix command line interpretation
  fixed minor corner case
  zstd.h: added the Introduction section
  fixed clang 3.5 warnings
  zstd.h: updated comments
 2016-10-24 13:10:13 -07:00
Yann Collet 4239a207dd Merge pull request #425 from inikep/dev11 
Doc
 2016-10-24 11:11:40 -07:00
Przemyslaw Skibinski 3ee94a7600 zstd_compression_format.md moved to doc/ 2016-10-24 15:58:07 +02:00
Yann Collet 97611611a3 Merge pull request #423 from terrelln/exec-seq-patch 
Fix small bug in ZSTD_execSequence()
 2016-10-21 17:02:06 -07:00
Nick Terrell ae1cb3b3d0 Fix small bug in ZSTD_execSequence() 
`memmove(op, match, sequence.matchLength)` is not the desired behavior.
Overlap is allowed, and handled as if we did `*op++ = *match++`, which
is not how `memmove()` handles overlap.

Only triggered if both of the following conditions are met:
* The match spans extDict & currentPrefixSegment
* `oLitEnd <= oend_w < oLitEnd + length1 < oMatchEnd <= oend`.

These two conditions imply that the block is less than 15 bytes long.
This bug isn't triggered by the streaming API, because it allocates
enough space for the window size + the block size, so there cannot be
a match that is within 8 bytes of the end and overlaps with itself.
It cannot be triggered by the block decompression API because all of
the decompressed data is in the currentPrefixSegment.

Introduced by commit 7158584399
 2016-10-21 12:13:44 -07:00
Yann Collet da3bd8b6de protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob 2016-10-20 20:11:00 -07:00
Nick Terrell bb68062c59 Unitialized memory read in ZSTD_decodeSeqHeaders() 
Caused by two things:
1. Not checking that `ip` is in range except for the first byte.
2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.
 2016-10-18 16:41:33 -07:00
Yann Collet 06573e17be fixed minor corner case 2016-10-17 17:28:28 -07:00
Nick Terrell 4db751668f Fix buffer overrun in ZSTD_loadEntropy() 
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadEntropy()`.  This caused `FSE_buildDTable(dctx->MLTable, ...)`
to overwrite the beginning of `dctx->hufTable`.

The benchmarks look good, there is no obvious performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 268.2 MB/s , 701.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.5 MB/s , 666.9 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 154.9 MB/s , 655.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 128.9 MB/s , 648.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.4 MB/s , 633.4 MB/s

  > ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 266.1 MB/s , 703.7 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.0 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.2 MB/s , 656.2 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 133.2 MB/s , 647.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.3 MB/s , 633.3 MB/s
 2016-10-17 15:51:15 -07:00
Yann Collet 7933434fdf Merge branch 'dev' of github.com:facebook/zstd into dev 2016-10-14 13:32:35 -07:00
Yann Collet d4cda27b63 new command -M#, to limit memory usage during decompression (#403) 2016-10-14 13:32:20 -07:00
Nick Terrell 3b9cdf9220 Fix ubsan failures (pass NULL to memcpy) 2016-10-12 20:54:42 -07:00
Yann Collet 5d919e7ac3 added ZSTD_error_frameParameter_windowTooLarge (#403) 2016-10-12 17:29:24 -07:00
Nick Terrell 7158584399 Fix ZSTD_execSequence() edge case 2016-10-12 10:05:26 -07:00
Yann Collet 2f2639438a zstreamtest can fuzztest pledgedSrcSize 2016-09-26 14:06:08 +02:00
Yann Collet 51f4d566c2 small decompression speed boost for very small data 2016-09-22 15:57:28 +02:00
Yann Collet d7c6589df8 support ZSTD_sizeof_*() on NULL 
added ZSTD_sizeof_CDict()
 2016-09-15 02:57:27 +02:00
Yann Collet e91c4b4cef introduced ZSTD_resetDStream() . 
added : ZSTD_sizeof_DDict()
 2016-09-14 16:55:44 +02:00
Yann Collet d092d77cfc minor variable renaming 2016-09-14 16:14:57 +02:00
Yann Collet 26ec254066 new strategy for faster DDict decompression 2016-09-13 16:52:16 +02:00
Yann Collet b3060f7a9e changed streaming decoder behavior : now, when all compressed frame is consumed, it means decompression is completed, with regenerated data fully flushed. 2016-09-09 16:44:16 +02:00
Yann Collet 95d07d7447 introduced CHECK_E 2016-09-06 16:38:51 +02:00
Yann Collet 3e21ec5b01 introduced CHECK_F 2016-09-06 15:36:19 +02:00
Yann Collet 5c956d593c FORCE_INLINE common definition 2016-09-06 15:05:19 +02:00
Yann Collet 7c83dfd5c2 ZSTD_frameHeaderSize_prefix (#340), as result of ZSTD_initStream 2016-09-05 19:47:43 +02:00
Yann Collet 1563bfeabc fixing FORCE_INLINE for older compilers (#330) 2016-09-02 11:44:21 -07:00
David Lam e10f7f3dcb merge 2016-08-30 12:03:36 -07:00
Yann Collet 4ded9e591c added boilerplate 2016-08-30 11:06:28 -07:00
David Lam da9d3b7057 Cleanup some errors in typedef comments and remove duplicated HOWTO from zbuff_decompress.c 2016-08-29 17:31:51 -07:00
Yann Collet 23b6e05d8e ZSTD_malloc() and ZSTD_free(), to simplify customMem 2016-08-28 21:05:43 -07:00
Yann Collet 5f53b0335e fixed continuation context 2016-08-28 10:00:49 -07:00
Yann Collet 767d8f66fa legacy contexts can be re-used 2016-08-28 08:19:47 -07:00
Yann Collet 4bf317dd00 first version supporting legacy streams (transparent decoding) 2016-08-28 07:43:34 -07:00
inikep a3a47ec4d0 Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into Other 2016-08-24 21:25:49 +02:00
inikep e416e30019 remove unnecessary comments 2016-08-24 17:32:09 +02:00
Yann Collet 17e482efdd added ZSTD_setDStreamParameter() 2016-08-23 16:58:10 +02:00
Yann Collet 3071c3e303 STREAM_WINDOW_MAX : protect streaming from unreasonable memory requirements 2016-08-23 01:34:34 +02:00
Yann Collet 70e3b31306 fixed playtests on os-x 2016-08-23 01:18:06 +02:00
Yann Collet cb3276329a added sizeof CStream and DStream 2016-08-23 00:31:59 +02:00
Yann Collet 8baf78a291 minor coding style 2016-08-20 13:04:20 +02:00
Yann Collet 1bee2d5e08 slight decompression speed improvement 2016-08-20 02:59:04 +02:00
Yann Collet 18442c1482 minor refactoring 2016-08-18 01:40:32 +02:00
Yann Collet 53e17fbd5e updated streaming API 2016-08-17 01:39:22 +02:00
Yann Collet 104e5b072d added : streaming decompression API 2016-08-16 15:11:28 +02:00
inikep 038d1497c9 fixed compilation with Visual Studio 2005 2016-08-10 14:30:10 +02:00
Yann Collet 917fe188f1 Implemented repOffset "minus 1" on ll==0 2016-07-31 04:01:57 +02:00