Commit Graph

286 Commits (241c57adffe34a8267dd9526cece87b8751468d2)

Author SHA1 Message Date
Yann Collet 5eed8e7a55 changed API comments to invite using macro ZSTD_CONTENTSIZE_UNKNOWN
to mean "pledgedSrcSize is not known at init time" instead of `0`.
Note that, a few prototypes created and documented with `0` to mean "unknown" still interpret "0" as unknown,
to avoid breaking 3rd party applications which depend on this behavior.
But this value is no longer recommended to mean "unknown".

In some future version, it might be possible to switch "0" to mean "empty",
as is already the case for several prototypes.
The advantage is that pledgedSrcSize field would have same behavior accross entire API,
making it easier to reason about.

Note that all concerned prototypes belong to the "experimental" API section.

srcSize is controlled at end of compression,
so if someone uses "0" to mean "unknown" while it effectively means "empty",
this is immediately caught by the compression function, which generates an error code : ZSTD_ERROR_srcSize_wrong
2017-10-14 00:32:06 -07:00
Yann Collet 213ef3b510 fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag,
and a stream fuzzer test, which was incorrect
(relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)
2017-10-13 19:01:58 -07:00
Yann Collet fb44516641 ensure fParams.contentSizeFlag starts at 1
such default was failing for ZSTD_compressBegin/ZSTD_compressContinue
fixed too
2017-10-13 17:39:13 -07:00
Yann Collet 3b27ed41fd Merge branch 'srcSize' into dev 2017-10-02 16:34:14 -07:00
Yann Collet 7e00df4a49 bumped version number
and updated NEWS in anticipation for release
2017-10-02 16:27:25 -07:00
Yann Collet 5db19b8685 added comment on ZSTD_COMPRESSBOUND()
as requested by @terrelln
2017-10-01 11:32:38 -07:00
Yann Collet 76ac0b2d99 macro compatible with scenario where windowSize = 1024 (minimum) 2017-09-30 15:34:44 -07:00
Yann Collet 5b10345b26 added ZSTD_COMPRESSBOUND() as a macro
ZSTD_compressBound() works fine, but is only useful for dynamic allocation.
For static allocation, only a macro can provide the amount during compilation time.
2017-09-29 23:17:41 -07:00
Yann Collet e4ec427720 Merge branch 'dev' into shorterTests
fixed conflicts
2017-09-28 12:19:28 -07:00
Yann Collet 9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
2017-09-28 01:25:40 -07:00
Yann Collet 54a827fff0 Merge branch 'dev' into newFormats
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
Yann Collet d56a350402 removed unsupported formats 2017-09-27 10:29:31 -07:00
Yann Collet df4e9bba25 fixed constant errors for gcc in c99 mode
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
2017-09-26 14:31:06 -07:00
Nick Terrell c233bdbaee Increase maximum window size
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
2017-09-26 14:00:01 -07:00
Yann Collet 52a1d1c6dc added ZSTD_DCtx_reset() 2017-09-25 16:56:48 -07:00
Yann Collet f2a913862c added ZSTD_decompress_generic_simpleArgs() 2017-09-25 15:46:34 -07:00
Yann Collet 6ee05a02b8 added ZSTD_decompress_generic()
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
2017-09-25 15:41:48 -07:00
Yann Collet 62568c9a42 added capability to generate magic-less frames
decoder not implemented yet
2017-09-25 14:26:26 -07:00
Yann Collet 96f0cde31a minor function rename
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
Yann Collet 7c3dea42ce added prototypes for advanced parameters for decompression API
required to decode custom formats
2017-09-24 15:57:29 -07:00
Yann Collet e60f48c549 Merge branch 'dev' into newFormats 2017-09-24 14:33:37 -07:00
Nick Terrell d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Yann Collet f97c2dbd39 created ZSTD_format declaration 2017-09-21 16:07:29 -07:00
Yann Collet f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev
Add long distance matcher
2017-09-13 13:24:08 -07:00
Stella Lau eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Yann Collet ce31004f20 fix following suggestions by @terrelln 2017-09-11 13:12:52 -07:00
Yann Collet 058ed2ad33 ZSTD_decodingBufferSize_min()
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
Yann Collet 3128e03be6 updated license header
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Stella Lau 9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
Stella Lau c706de5395 Rename and add short ldm parameters in cli 2017-09-05 21:11:18 -07:00
Stella Lau 643d28c701 Add ldm options to 'man zstd' 2017-09-05 11:27:15 -07:00
Stella Lau 67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00
Stella Lau a1f04d518d Move hashEveryLog to cctxParams and update cli 2017-09-01 15:05:47 -07:00
Stella Lau 767a0b3be1 Move ldm hashLog, bucketLog, and mml to cctxParams 2017-09-01 12:24:59 -07:00
Stella Lau 17d8e0bdcc Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate 2017-09-01 10:19:38 -07:00
Stella Lau 8081becadc Add long distance matching as a CCtxParam 2017-09-01 09:18:58 -07:00
Yann Collet d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
Stella Lau 6a546efb8c Add long distance matcher
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
Stella Lau 90a31bfa16 Pass dictMode to ZSTDMT_initCStream; fix nits
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
Stella Lau 82d636b76a Rename applyCCtxParams() 2017-08-29 18:03:06 -07:00
Stella Lau c7a18b7c21 Localize 'dictMode' from cctx to function param 2017-08-29 15:52:24 -07:00
Stella Lau c88fb9267f Replace 'byReference' with enum 2017-08-29 11:55:02 -07:00
Stella Lau b5b9275e67 Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced() 2017-08-29 10:49:29 -07:00
Stella Lau 18224608ff Remove ZSTD_setCCtxParameter() 2017-08-25 13:58:41 -07:00
Stella Lau 0744592d38 Add function initializing cctxParams from clevel 2017-08-25 13:36:47 -07:00
Stella Lau eb7bbab36a Remove ZSTD_p_refDictContent and dictContentByRef 2017-08-25 11:11:45 -07:00
Stella Lau 15fdeb9e41 Enforce nbThreads<=1 for estimateCCtxSize 2017-08-24 16:28:49 -07:00
Stella Lau 5bc2c1e982 Add prototype support for customMem with cctxParams 2017-08-23 12:03:30 -07:00
Stella Lau 6f1a21c7e9 Remove formatting-only changes 2017-08-23 10:24:19 -07:00
Stella Lau 5b956f4753 Comment out CCtx_param versions of CDict functions 2017-08-21 14:49:16 -07:00