b555b7ef41
[libzstd][opt] Simplify repcode logic
2017-09-27 15:30:12 -07:00
ea1f50bf73
removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal()
...
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.
Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.
A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
2017-09-27 13:51:05 -07:00
c994932788
fixed ZSTD_format_e value validation
2017-09-27 12:22:22 -07:00
6c41adfb28
[libzstd] pthread function prefixed with ZSTD_
...
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
2017-09-27 11:48:48 -07:00
ecf1778e23
updated ZSTD_format_e value validation
...
also updated manual
2017-09-27 11:19:21 -07:00
9416195221
changed error code when pos<=size condition is not respected
...
Now pointing towards src_size or dst_size,
instead of error_GENERIC.
2017-09-27 10:35:56 -07:00
d56a350402
removed unsupported formats
2017-09-27 10:29:31 -07:00
ca306c1c84
fixed a bug in zstreamtest
...
decoder output buffer would receive a wrong size.
In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.
This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
2017-09-27 00:39:41 -07:00
cd53ac831b
fixed DCtx initialization error
...
now relying on initialization of dctx->format first
2017-09-26 18:26:09 -07:00
4791561c4a
silence minor gcc warning -Wempty-body
...
also silence fuzz test artefacts
2017-09-26 17:57:38 -07:00
c0dd960363
switch assert() position
2017-09-26 15:36:57 -07:00
319c699991
created ZSTD_startingInputLength()
...
as suggested by @terrelln
2017-09-26 15:36:14 -07:00
8d1e97ea9c
minor fixes following @terrelln comments
2017-09-26 15:06:30 -07:00
df4e9bba25
fixed constant errors for gcc in c99 mode
...
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
2017-09-26 14:31:06 -07:00
9f0b8dfbe9
Merge branch 'dev' into newFormats
2017-09-26 14:22:39 -07:00
c233bdbaee
Increase maximum window size
...
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
`--zstd=wlog=#`. These options also set the window size during
decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
and decompression.
2017-09-26 14:00:01 -07:00
586df82a78
Merge pull request #862 from terrelln/static
...
[zstd] Backport kernel patch from @ColinIanKing
2017-09-25 17:02:40 -07:00
52a1d1c6dc
added ZSTD_DCtx_reset()
2017-09-25 16:56:48 -07:00
5d8fdd1641
Merge pull request #855 from terrelln/maxoff
...
[libzstd] Increase MaxOff
2017-09-25 16:34:29 -07:00
76cb38d085
[zstd] Backport kernel patch from @ColinIanKing
...
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com .
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED ).
* Benchmarks showed no difference in speed.
2017-09-25 16:18:23 -07:00
f2a913862c
added ZSTD_decompress_generic_simpleArgs()
2017-09-25 15:46:34 -07:00
6ee05a02b8
added ZSTD_decompress_generic()
...
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
2017-09-25 15:41:48 -07:00
b8d4a3887f
introduced constant ZSTD_frameIdSize
...
within zstd_internal.h
This is the size of magic number.
Avoids using `4` directly in source code, which is a bit less meaningful.
2017-09-25 15:26:18 -07:00
044fb4c057
implemented magic-less frame decoder
2017-09-25 15:12:09 -07:00
62568c9a42
added capability to generate magic-less frames
...
decoder not implemented yet
2017-09-25 14:26:26 -07:00
bbe77212ef
[libzstd] Increase MaxOff
2017-09-25 13:36:18 -07:00
96f0cde31a
minor function rename
...
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
7c3dea42ce
added prototypes for advanced parameters for decompression API
...
required to decode custom formats
2017-09-24 15:57:29 -07:00
e60f48c549
Merge branch 'dev' into newFormats
2017-09-24 14:33:37 -07:00
8977224b9b
Merge pull request #859 from terrelln/31
...
Prepare for ZSTD_WINDOWLOG_MAX == 31
2017-09-22 09:01:39 -07:00
d6abb28951
Prepare for ZSTD_WINDOWLOG_MAX == 31
2017-09-21 17:18:41 -07:00
cd3115b284
added control from frame content size at end of decompression
...
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
2017-09-21 16:21:10 -07:00
645563583e
Merge branch 'dev' into newFormats
2017-09-21 16:08:06 -07:00
f97c2dbd39
created ZSTD_format declaration
2017-09-21 16:07:29 -07:00
da74aabc00
Merge pull request #850 from terrelln/fse-optimal
...
[fse] Fix FSE_optimalTableLog() for srcSize==1
2017-09-19 14:59:21 -07:00
c399ab4804
Merge pull request #849 from terrelln/30
...
[bitstream] Allow adding 31 bits at a time
2017-09-19 14:25:10 -07:00
74718d7e43
[bitstream] Allow adding 31 bits at a time
2017-09-19 13:57:33 -07:00
6c9ed76676
[ldm] Fix corner case where minMatch < 8
...
There is a potential read buffer overflow when minMatch < 8.
fix-fuzz-failure
2017-09-19 13:49:37 -07:00
18442a31ff
[libzstd] Fix bad window size assert
...
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.
fix-fuzz-failure
2017-09-19 13:47:59 -07:00
cb8b471e8b
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-09-18 14:48:23 -07:00
7d1ff3817b
fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream()
...
previous result was over-estimated
by counting streaming buffers twice
2017-09-18 14:47:34 -07:00
cae3e3c652
[fse] Fix FSE_optimalTableLog() for srcSize==1
2017-09-18 14:11:18 -07:00
72a80515ec
Merge pull request #848 from terrelln/fparams
...
[block] Don't use fParams in ZSTD_decompressBlock()
2017-09-18 13:48:31 -07:00
539b91ee9b
minor : added assert in bt
2017-09-16 23:41:58 -07:00
5f22479517
[block] Don't use fParams in ZSTD_decompressBlock()
2017-09-15 17:37:20 -07:00
77c137b3ae
minor comment refactor
2017-09-14 15:12:57 -07:00
335780c427
fixed too strong alignment assert in ZSTD_initStaticCCtx()
...
64-bits fields are only 32-bits aligned on 32-bits CPU
2017-09-13 16:35:29 -07:00
f1571dad8f
Merge pull request #838 from stellamplau/ldm-mergeDev
...
Add long distance matcher
2017-09-13 13:24:08 -07:00
4120a7fd5a
Merge pull request #837 from facebook/libzstd-nomt
...
makes it possible to compile libzstd in single-thread mode without zs…
2017-09-12 17:13:17 -07:00
3306bcb0e6
fix #820 : GCC v3.x 32-bits doesn't define 64-bits intrinsic
...
resulting in undefined symbol error.
Push the requirement to GCC 4 for now.
Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead.
__LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits)
Might be better, but seems also more complex, hence more prone to side effects.
Keeping the simple solution for now (just rely on __GNUC__)
2017-09-11 15:17:31 -07:00
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
f902bf9676
Merge branch 'ldm-integrate' into ldm-mergeDev
2017-09-11 14:55:29 -07:00
f325ee4e84
fixed pass-through warning
2017-09-11 14:37:03 -07:00
0d1b54db61
Explicitly cast raw numerals when left-shifting
2017-09-11 14:28:18 -07:00
0d6ecc72a3
makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c ( #819 )
2017-09-11 14:09:34 -07:00
ce31004f20
fix following suggestions by @terrelln
2017-09-11 13:12:52 -07:00
b3f33ccfb3
use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
...
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
058ed2ad33
ZSTD_decodingBufferSize_min()
...
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.
also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
360428c5d9
Move ldm functions to their own file
2017-09-06 18:09:26 -07:00
baa37c3362
programs/Makefile : better support for GNU conventions
...
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
2017-09-06 16:53:59 -07:00
3a12531a3d
lib/Makefile : better support for GNU conventions
...
see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
2017-09-06 16:35:49 -07:00
1c7b914cdf
update README on BUCK file
2017-09-06 16:23:39 -07:00
36374cc3b4
update and clarify lib/README
2017-09-06 16:15:18 -07:00
2b99d696de
Remove debug code
2017-09-06 15:57:26 -07:00
eeff55dfa8
Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev
2017-09-06 15:56:32 -07:00
ad0046244f
Merge pull request #831 from terrelln/split-compress
...
Split parsers out of zstd_compress.c
2017-09-06 10:01:27 -07:00
9e4060200b
Add tests and fix pointer alignment
2017-09-06 09:14:05 -07:00
c706de5395
Rename and add short ldm parameters in cli
2017-09-05 21:11:18 -07:00
98b85426f1
Fix setting of nextToUpdate at end of ldm matcher
2017-09-05 20:41:37 -07:00
721726d688
Split parsers out of zstd_compress.c
2017-09-05 17:10:25 -07:00
08d33fe1c9
Fix parameter handling in copyCCtx with cdict
2017-09-05 15:50:20 -07:00
fd0071da29
Fix parameter handling with ZSTD_copyCCtx
2017-09-05 15:34:17 -07:00
643d28c701
Add ldm options to 'man zstd'
2017-09-05 11:27:15 -07:00
423b133568
[POOL] Allow free on NULL when multithreading is disabled
2017-09-05 11:18:13 -07:00
67d4a6161c
Add ldmBucketSizeLog param
2017-09-02 21:55:29 -07:00
a1f04d518d
Move hashEveryLog to cctxParams and update cli
2017-09-01 15:05:47 -07:00
767a0b3be1
Move ldm hashLog, bucketLog, and mml to cctxParams
2017-09-01 12:24:59 -07:00
8a5c0c98ae
restored 32-bits decoder ability to decode long offsets (>32 MB, levels 21+)
2017-09-01 11:56:57 -07:00
36aa8b5999
improved decoding speed
2017-09-01 11:40:59 -07:00
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
d963daa6a9
fixed minor warning (empty translation unit)
2017-09-01 00:12:07 -07:00
3704507774
fixed decompression bug reported by @Etsukata ( #828 )
2017-09-01 00:05:37 -07:00
369c29dd1a
fixed impact of merge conflict for longRange
2017-08-31 18:25:56 -07:00
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
6a546efb8c
Add long distance matcher
...
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
b0cb081dc8
last batch of header files changed to reflect new license ( #825 )
...
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
e21384fffb
fixed more file headers after license change ( #825 )
2017-08-31 12:11:57 -07:00
e9dc204f42
fixed a bunch of headers after license change ( #825 )
2017-08-31 11:24:54 -07:00
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
ee65701720
Minor fixes; remove formatting only changes
2017-08-29 20:27:35 -07:00
a6e20e1bd7
Add test for raw content starting with dict header
2017-08-29 18:36:18 -07:00
623e3cd40b
Use ZSTD_dm_rawContent in zstdmt_compress
2017-08-29 18:04:32 -07:00
82d636b76a
Rename applyCCtxParams()
2017-08-29 18:03:06 -07:00
4e835720bf
Delay creation of ZSTDMT_CCtx
2017-08-29 17:58:32 -07:00
c7a18b7c21
Localize 'dictMode' from cctx to function param
2017-08-29 15:52:24 -07:00
d6ddb879da
Merge pull request #817 from terrelln/pool-custom-alloc
...
[pool] Accept custom allocators
2017-08-29 13:05:39 -07:00
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
9822f97721
[error] Don't guard undef X with ifdef X
2017-08-29 11:54:38 -07:00
b5b9275e67
Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced()
2017-08-29 10:49:29 -07:00
0e56a84a1e
Fix getting cParams from CCtxParams
2017-08-28 19:25:17 -07:00
02033be08c
[pool] Visual Studios disallows empty structs
2017-08-28 17:19:01 -07:00
7c365eb02c
[threading] Fix ERROR macro after including windows.h
2017-08-28 16:25:02 -07:00
cf689b84f9
Sort input file list
...
in order to make builds reproducible
in spite of indeterministic filesystem readdir order.
See https://reproducible-builds.org/ for why this is good.
2017-08-26 17:08:00 +02:00
024098a47d
Fix parameter retrieval from cdict
2017-08-25 17:58:28 -07:00
2adde898c8
Fix typo with ZSTDMT_parameter
2017-08-25 16:13:40 -07:00
18224608ff
Remove ZSTD_setCCtxParameter()
2017-08-25 13:58:41 -07:00
0744592d38
Add function initializing cctxParams from clevel
2017-08-25 13:36:47 -07:00
9911153723
Move jobSize and overlapLog in zstdmt to cctxParams
2017-08-25 13:14:51 -07:00
de5193422d
Distinguish between jobParams and cctxParams in zstdmt
2017-08-25 11:36:17 -07:00
eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
db3f5372df
[zstdmt] Use POOL_create_advanced()
2017-08-24 18:12:28 -07:00
de6c6bce85
Fix zstd_internal.h for C++ mode
2017-08-24 18:09:50 -07:00
26dc040a7b
[pool] Accept custom allocators
2017-08-24 17:01:41 -07:00
89dc856cae
[pool] Fix formatting
2017-08-24 16:48:32 -07:00
15fdeb9e41
Enforce nbThreads<=1 for estimateCCtxSize
2017-08-24 16:28:49 -07:00
376f435914
[dictBuilder] Set default compression level to 3
2017-08-24 16:21:05 -07:00
2fbf0285b2
Fix interaction with ZSTD_setCCtxParameter() and cleanup
2017-08-24 11:25:41 -07:00
fd9bf42516
Fix forceWindow and dictMode setting for zstdmt jobs
2017-08-23 19:16:57 -07:00
bf3108fb50
Ensure zstdmt uses 'job version' of cctx parameters
2017-08-23 17:03:31 -07:00
1c81f725ff
Remove duplicated testing code
2017-08-23 15:47:15 -07:00
64ce49426b
Fix cstream compression level
2017-08-23 12:30:47 -07:00
5bc2c1e982
Add prototype support for customMem with cctxParams
2017-08-23 12:03:30 -07:00
e9ce1208a1
Merge pull request #812 from facebook/longRangeFix
...
fixed extraordinary scenario where all fields use maximum nbBits
2017-08-23 11:35:28 -07:00
74cde5a4d8
Merge pull request #813 from stellamplau/highbit32fix
...
Fix undefined behavior when srcSize==1
2017-08-23 11:31:06 -07:00
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
20f715d709
Fix displayLevel overflow
2017-08-23 15:56:15 +05:00
11303778d0
Add function to make cctxParams from ZSTD_parameters
2017-08-22 14:53:13 -07:00
bd9c8ca146
Merge pull request #811 from terrelln/segmentSize
...
[cover] Fix end condition for small dictionary
2017-08-22 14:36:30 -07:00
23fc0e41fa
Remove 'opaque' naming from internal functions
2017-08-22 14:24:47 -07:00
8fd1636776
Remove unused functions
2017-08-22 13:33:58 -07:00
6b2b6a9bd5
fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously
...
can only happen if windowLog>=27 (level 22 --ultra)
2017-08-22 12:09:21 -07:00
e50ed1fa3a
Fix undefined behavior when srcSize==1
2017-08-22 11:55:42 -07:00
60e1bc617c
Explicitly create a job cctxParam for multithreading
2017-08-21 15:39:37 -07:00
5b956f4753
Comment out CCtx_param versions of CDict functions
2017-08-21 14:49:16 -07:00
29c2d9a4d0
[cover] Turn down notification for ZDICT subroutines
2017-08-21 14:28:31 -07:00
98de3f6847
[cover] Add dictionary size to compressed size
2017-08-21 14:23:17 -07:00
78c3d16bf4
Merge pull request #809 from terrelln/dev
...
[cover] Fix divide by zero
2017-08-21 13:33:19 -07:00
9a54a315aa
[cover] Convert score to U32 and check for zero
2017-08-21 13:30:07 -07:00
fd8a25786e
Check parameters are valid in initCCtxParams
2017-08-21 13:23:35 -07:00
1c0dbe81b1
Add documentation for CCtx_params
2017-08-21 13:18:00 -07:00
d49eb40c03
[cover] Stop when segmentSize is less than d
2017-08-21 13:10:03 -07:00
939f954285
Pass ZSTD_CCtx_params as const ptr when possible
2017-08-21 12:57:18 -07:00
73c73bf16a
Reduce code duplication in zstreamtest
2017-08-21 12:41:19 -07:00
560b34f6d2
Return error code when initializing NULL cctxParams
2017-08-21 11:52:26 -07:00
25be09c6b4
Set some parameters to zero before initializing cdict
2017-08-21 11:35:46 -07:00
232d62b637
fixed a few headers that were too hastily copy/pasted during last license change
2017-08-21 11:24:32 -07:00
f306d400c0
[cover] Fix divide by zero
2017-08-21 11:12:11 -07:00
502031ca10
Use cctxParam version of createCDict internally
2017-08-21 11:00:44 -07:00
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
7db552676e
reduced pool queue to 0 to save memory
...
fixed : pool performance when jobs are fires fast and queueSize==0
2017-08-19 15:07:54 -07:00
6cee6e07e5
Add internal createCDict function
2017-08-18 22:48:31 -07:00
d775519296
Add cctxParam versions of internal functions
2017-08-18 17:37:58 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
63b8c98531
Pass cctx parameters to MTCtx
2017-08-18 16:17:24 -07:00
399ae013d4
Add function to apply cctx params
2017-08-18 13:01:55 -07:00
81d89d82a6
Move nbThreads to cctx params
2017-08-18 12:08:57 -07:00
2300c58a6f
Move dictContentByRef to cctx params
2017-08-18 12:03:16 -07:00
b6cb2ed8cb
Move dictMode to cctxParams
2017-08-18 11:43:31 -07:00
97e27affcb
Move compression level to cctx params
2017-08-18 11:20:08 -07:00
c0221124d5
Add function to set opaque parameters
2017-08-17 19:30:22 -07:00
4169f49171
Add initialization/allocation functions for opaque params
2017-08-17 18:45:04 -07:00
ade95b8bed
Add opaque interfaces for static initialization
2017-08-17 18:13:08 -07:00
699f11b4f7
Create opaque parameter structure
2017-08-17 17:33:46 -07:00
f9e6590715
Merge pull request #796 from terrelln/is-error
...
[FSE][HUF] Inline error checks
2017-08-15 12:37:28 -07:00
2dbcfc6994
Merge pull request #794 from terrelln/force-inline
...
[libzstd] Fix FORCE_INLINE macro
2017-08-15 12:03:44 -07:00
07c6ff588e
[FSE][HUF] Inline error checks
...
Caught by Clang's optimization remarks.
2017-08-15 11:23:28 -07:00
565e925eb7
[libzstd] Fix FORCE_INLINE macro
2017-08-14 21:12:05 -07:00
b9d4f4fb74
Fix ZSTD_estimateDStreamSize function after ZSTD_DStream and ZSTD_DCtx were merged
2017-08-13 13:29:42 +03:00
9ba97182d1
[CI] Add gcc7build test
2017-08-08 13:28:56 -07:00
d9f2893eb9
Merge pull request #782 from terrelln/dstSizeTooSmall
...
Fix compression failure on incompressible data
2017-08-07 14:52:02 -07:00
8049556928
Merge pull request #778 from terrelln/bad-huff
...
[libzstd] Fix bug in Huffman decompresser
2017-08-07 14:05:58 -07:00
abe12b3399
[libzstd] Fix bug in Huffman decompresser
...
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.
* Make the decompresser accept Huffman compressed literals whose size
increases.
* Add a test case that exposes the bug. The compressed file has to be
statically generated, since the compressor won't normally produce files
that expose the bug.
2017-08-07 12:37:48 -07:00
308047eb5d
Fix compression failure on incompressible data
...
If the destination buffer is the minimum allowed size in
`ZSTD_compressSequences()` (2^17), then if the block isn't compressible
compression might fail with `dstSize_tooSmall`, when it should instead emit
a raw uncompressed block.
Additionally, `ZSTD_compressLiterals()` implicitly called
`ZSTD_noCompressLiterals()` if Huffman compression failed. Make that
explicit.
2017-08-07 11:45:24 -07:00
73ba58955f
Signal after finishing job when queueSize=0
2017-08-01 20:12:06 -07:00
1d76da1d87
Replace marker with queueEmpty variable and update pool.h comment
2017-08-01 12:30:16 -07:00
5adceeed01
Allow queueSize=0 in pool.c and update poolTests
2017-07-31 10:10:16 -07:00
e1222544be
Merge pull request #753 from paulcruz74/adapt-approach-3
...
adaptive compression v1
2017-07-27 10:00:10 -07:00
ae20d413da
[libzstd] Fix CHECK_V_F macros
2017-07-25 12:52:01 -07:00
a90b16e150
Visual blind fix 2
2017-07-20 15:57:55 -07:00
b4d460f32c
pool.c : blindfix for Visual warnings
2017-07-20 01:13:14 -07:00
3974d2b38a
blind fix for Windows Multithreading module
...
adds a fake 0 return value for mutex/cond init
2017-07-19 13:33:21 -07:00
6945b3c43d
removed previous version of completion for compression
2017-07-19 11:51:50 -07:00
b71363b967
check pthread_*_init() success condition
2017-07-19 01:05:40 -07:00
cc1522351f
[libzstd] Fix bug in Huffman encoding
...
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.
The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.
Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s
Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
2017-07-18 13:20:40 -07:00
77d67fb167
Merge pull request #766 from terrelln/real-block-split
...
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-18 08:26:24 -07:00
14c83b05c7
Merge pull request #765 from terrelln/real-block-split
...
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 19:25:55 -07:00
7a28b9e4a3
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-17 15:29:11 -07:00
3381bf4b84
Merge pull request #764 from terrelln/real-block-split
...
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 14:46:01 -07:00
e198230645
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 12:27:24 -07:00
634f012420
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 11:36:11 -07:00
50ce4eaeb6
added error detection for pthread initialization, added compression completion measurement, fixed const values
2017-07-17 10:12:44 -07:00
3b0cff3c33
fixed clang's -Wdocumentation
2017-07-13 18:58:30 -07:00
2bd6440be0
pinned down error code enum values
...
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.
Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
2017-07-13 17:12:16 -07:00
830ef4152a
[libzstd] Increase granularity of FSECTable repeat mode
2017-07-13 12:45:39 -07:00
d985319337
Merge pull request #759 from terrelln/real-block-split
...
[libzstd] Pull CTables into sub-structure
2017-07-13 10:24:19 -07:00
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
Nick Terrell
Yann Collet
Yann Collet
Stella Lau
Bernhard M. Wiedemann
Dmitriy Titarenko
Roman Gershman
Paul Cruz