facebook/zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #863 from facebook/newFormats

Browse Source 
magicless frames (#591)
This commit is contained in:
Yann Collet 2017-09-28 00:32:16 -07:00 committed by GitHub
commit 824f75ea7c
18 changed files with 701 additions and 259 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
NEWS
View File

@ -3,7 +3,8 @@ new : long range mode, using --long command, by Stella Lau (@stellamplau)
license : changed /examples license to BSD + GPLv2 license : changed /examples license to BSD + GPLv2
license : fix a few header files to reflect new license (#825) license : fix a few header files to reflect new license (#825)
fix : multi-threading compression works with custom allocators fix : multi-threading compression works with custom allocators
fix : a rare compression bug when compression generates very large distances (only possible at --ultra -22) fix : ZSTD_sizeof_CStream() was over-evaluating memory usage
fix : a rare compression bug when compression generates very large distances and bunch of other conditions (only possible at --ultra -22)
fix : 32-bits build can now decode large offsets (levels 21+) fix : 32-bits build can now decode large offsets (levels 21+)
cli : new : can split input file for dictionary training, using command -B# cli : new : can split input file for dictionary training, using command -B#
cli : fix : do not change /dev/null permissions when using command -t with root access, reported by @mike155 (#851) cli : fix : do not change /dev/null permissions when using command -t with root access, reported by @mike155 (#851)

220
doc/zstd_manual.html
View File

@ -27,8 +27,8 @@
<li><a href="#Chapter17">Buffer-less and synchronous inner streaming functions</a></li> <li><a href="#Chapter17">Buffer-less and synchronous inner streaming functions</a></li>
<li><a href="#Chapter18">Buffer-less streaming compression (synchronous mode)</a></li> <li><a href="#Chapter18">Buffer-less streaming compression (synchronous mode)</a></li>
<li><a href="#Chapter19">Buffer-less streaming decompression (synchronous mode)</a></li> <li><a href="#Chapter19">Buffer-less streaming decompression (synchronous mode)</a></li>
<li><a href="#Chapter20">ZSTD_CCtx_params</a></li> <li><a href="#Chapter20">New advanced API (experimental)</a></li>
<li><a href="#Chapter21">Block functions</a></li> <li><a href="#Chapter21">Block level API</a></li>
</ol> </ol>
<hr> <hr>
<a name="Chapter1"></a><h2>Introduction</h2><pre> <a name="Chapter1"></a><h2>Introduction</h2><pre>
@ -399,33 +399,33 @@ size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds);
size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict); size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict);
size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
</b><p> These functions give the current memory usage of selected object. </b><p> These functions give the current memory usage of selected object.
Object memory usage can evolve if it's re-used multiple times. Object memory usage can evolve when re-used multiple times.
</p></pre><BR> </p></pre><BR>
<pre><b>size_t ZSTD_estimateCCtxSize(int compressionLevel); <pre><b>size_t ZSTD_estimateCCtxSize(int compressionLevel);
size_t ZSTD_estimateCCtxSize_advanced_usingCParams(ZSTD_compressionParameters cParams); size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
size_t ZSTD_estimateCCtxSize_advanced_usingCCtxParams(const ZSTD_CCtx_params* params); size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
size_t ZSTD_estimateDCtxSize(void); size_t ZSTD_estimateDCtxSize(void);
</b><p> These functions make it possible to estimate memory usage </b><p> These functions make it possible to estimate memory usage
of a future {D,C}Ctx, before its creation. of a future {D,C}Ctx, before its creation.
ZSTD_estimateCCtxSize() will provide a budget large enough for any compression level up to selected one. ZSTD_estimateCCtxSize() will provide a budget large enough for any compression level up to selected one.
It will also consider src size to be arbitrarily "large", which is worst case. It will also consider src size to be arbitrarily "large", which is worst case.
If srcSize is known to always be small, ZSTD_estimateCCtxSize_advanced_usingCParams() can provide a tighter estimation. If srcSize is known to always be small, ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation.
ZSTD_estimateCCtxSize_advanced_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
ZSTD_estimateCCtxSize_advanced_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is > 1. ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is > 1.
Note : CCtx estimation is only correct for single-threaded compression Note : CCtx estimation is only correct for single-threaded compression
</p></pre><BR> </p></pre><BR>
<pre><b>size_t ZSTD_estimateCStreamSize(int compressionLevel); <pre><b>size_t ZSTD_estimateCStreamSize(int compressionLevel);
size_t ZSTD_estimateCStreamSize_advanced_usingCParams(ZSTD_compressionParameters cParams); size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams);
size_t ZSTD_estimateCStreamSize_advanced_usingCCtxParams(const ZSTD_CCtx_params* params); size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params);
size_t ZSTD_estimateDStreamSize(size_t windowSize); size_t ZSTD_estimateDStreamSize(size_t windowSize);
size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize); size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
</b><p> ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one. </b><p> ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one.
It will also consider src size to be arbitrarily "large", which is worst case. It will also consider src size to be arbitrarily "large", which is worst case.
If srcSize is known to always be small, ZSTD_estimateCStreamSize_advanced_usingCParams() can provide a tighter estimation. If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation.
ZSTD_estimateCStreamSize_advanced_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
ZSTD_estimateCStreamSize_advanced_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is set to a value > 1. ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is set to a value > 1.
Note : CStream estimation is only correct for single-threaded compression. Note : CStream estimation is only correct for single-threaded compression.
ZSTD_DStream memory budget depends on window Size. ZSTD_DStream memory budget depends on window Size.
This information can be passed manually, using ZSTD_estimateDStreamSize, This information can be passed manually, using ZSTD_estimateDStreamSize,
@ -436,8 +436,8 @@ size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
</p></pre><BR> </p></pre><BR>
<pre><b>typedef enum { <pre><b>typedef enum {
ZSTD_dlm_byCopy = 0, </b>/* Copy dictionary content internally. */<b> ZSTD_dlm_byCopy = 0, </b>/**< Copy dictionary content internally */<b>
ZSTD_dlm_byRef, </b>/* Reference dictionary content -- the dictionary buffer must outlives its users. */<b> ZSTD_dlm_byRef, </b>/**< Reference dictionary content -- the dictionary buffer must outlive its users. */<b>
} ZSTD_dictLoadMethod_e; } ZSTD_dictLoadMethod_e;
</b></pre><BR> </b></pre><BR>
<pre><b>size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel); <pre><b>size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel);
@ -646,18 +646,18 @@ size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs, const ZSTD_CDict*
@return : 0, or an error code (which can be tested using ZSTD_isError()) @return : 0, or an error code (which can be tested using ZSTD_isError())
</p></pre><BR> </p></pre><BR>
<h3>Advanced Streaming decompression functions</h3><pre></pre><b><pre>typedef enum { DStream_p_maxWindowSize } ZSTD_DStreamParameter_e; <h3>Advanced Streaming decompression functions</h3><pre></pre><b><pre>ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); </b>/**< same as ZSTD_initStaticDCtx() */<b> ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); </b>/**< same as ZSTD_initStaticDCtx() */<b>
size_t ZSTD_setDStreamParameter(ZSTD_DStream* zds, ZSTD_DStreamParameter_e paramType, unsigned paramValue); typedef enum { DStream_p_maxWindowSize } ZSTD_DStreamParameter_e;
size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); </b>/**< note: a dict will not be used if dict == NULL or dictSize < 8 */<b> size_t ZSTD_setDStreamParameter(ZSTD_DStream* zds, ZSTD_DStreamParameter_e paramType, unsigned paramValue); </b>/* obsolete : this API will be removed in a future version */<b>
size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); </b>/**< note : ddict will just be referenced, and must outlive decompression session */<b> size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); </b>/**< note: no dictionary will be used if dict == NULL or dictSize < 8 */<b>
size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); </b>/**< note : ddict is referenced, it must outlive decompression session */<b>
size_t ZSTD_resetDStream(ZSTD_DStream* zds); </b>/**< re-use decompression parameters from previous init; saves dictionary loading */<b> size_t ZSTD_resetDStream(ZSTD_DStream* zds); </b>/**< re-use decompression parameters from previous init; saves dictionary loading */<b>
</pre></b><BR> </pre></b><BR>
<a name="Chapter17"></a><h2>Buffer-less and synchronous inner streaming functions</h2><pre> <a name="Chapter17"></a><h2>Buffer-less and synchronous inner streaming functions</h2><pre>
This is an advanced API, giving full control over buffer management, for users which need direct control over memory. This is an advanced API, giving full control over buffer management, for users which need direct control over memory.
But it's also a complex one, with many restrictions (documented below). But it's also a complex one, with several restrictions, documented below.
Prefer using normal streaming API for an easier experience Prefer normal streaming API for an easier experience.
<BR></pre> <BR></pre>
@ -673,8 +673,8 @@ size_t ZSTD_resetDStream(ZSTD_DStream* zds); </b>/**< re-use decompression para
Then, consume your input using ZSTD_compressContinue(). Then, consume your input using ZSTD_compressContinue().
There are some important considerations to keep in mind when using this advanced function : There are some important considerations to keep in mind when using this advanced function :
- ZSTD_compressContinue() has no internal buffer. It uses externally provided buffer only. - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
- Interface is synchronous : input is consumed entirely and produce 1+ (or more) compressed blocks. - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
- Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario. - Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario.
Worst case evaluation is provided by ZSTD_compressBound(). Worst case evaluation is provided by ZSTD_compressBound().
ZSTD_compressContinue() doesn't guarantee recover after a failed compression. ZSTD_compressContinue() doesn't guarantee recover after a failed compression.
@ -685,9 +685,9 @@ size_t ZSTD_resetDStream(ZSTD_DStream* zds); </b>/**< re-use decompression para
Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum. Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame. It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
Without last block mark, frames will be considered unfinished (corrupted) by decoders. Without last block mark, frames are considered unfinished (hence corrupted) by compliant decoders.
`ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress some new frame. `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
<BR></pre> <BR></pre>
<h3>Buffer-less streaming compression functions</h3><pre></pre><b><pre>size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel); <h3>Buffer-less streaming compression functions</h3><pre></pre><b><pre>size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
@ -783,8 +783,29 @@ size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long
</pre></b><BR> </pre></b><BR>
<pre><b>typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e; <pre><b>typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e;
</b></pre><BR> </b></pre><BR>
<h3>New advanced API (experimental, and compression only)</h3><pre></pre><b><pre></pre></b><BR> <a name="Chapter20"></a><h2>New advanced API (experimental)</h2><pre></pre>
<pre><b>typedef enum { <pre><b>typedef enum {
</b>/* Question : should we have a format ZSTD_f_auto ?<b>
* For the time being, it would mean exactly the same as ZSTD_f_zstd1.
* But, in the future, should several formats be supported,
* on the compression side, it would mean "default format".
* On the decompression side, it would mean "multi format",
* and ZSTD_f_zstd1 could be reserved to mean "accept *only* zstd frames".
* Since meaning is a little different, another option could be to define different enums for compression and decompression.
* This question could be kept for later, when there are actually multiple formats to support,
* but there is also the question of pinning enum values, and pinning value `0` is especially important */
ZSTD_f_zstd1 = 0, </b>/* zstd frame format, specified in zstd_compression_format.md (default) */<b>
ZSTD_f_zstd1_magicless, </b>/* Variant of zstd frame format, without initial 4-bytes magic number.<b>
* Useful to save 4 bytes per generated frame.
* Decoder cannot recognise automatically this format, requiring instructions. */
} ZSTD_format_e;
</b></pre><BR>
<pre><b>typedef enum {
</b>/* compression format */<b>
ZSTD_p_format = 10, </b>/* See ZSTD_format_e enum definition.<b>
* Cast selected format as unsigned for ZSTD_CCtx_setParameter() compatibility. */
</b>/* compression parameters */<b> </b>/* compression parameters */<b>
ZSTD_p_compressionLevel=100, </b>/* Update all compression parameters according to pre-defined cLevel table<b> ZSTD_p_compressionLevel=100, </b>/* Update all compression parameters according to pre-defined cLevel table<b>
* Default level is ZSTD_CLEVEL_DEFAULT==3. * Default level is ZSTD_CLEVEL_DEFAULT==3.
@ -949,7 +970,7 @@ size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t
<pre><b>typedef enum { <pre><b>typedef enum {
ZSTD_e_continue=0, </b>/* collect more data, encoder transparently decides when to output result, for optimal conditions */<b> ZSTD_e_continue=0, </b>/* collect more data, encoder transparently decides when to output result, for optimal conditions */<b>
ZSTD_e_flush, </b>/* flush any data provided so far - frame will continue, future data can still reference previous data for better compression */<b> ZSTD_e_flush, </b>/* flush any data provided so far - frame will continue, future data can still reference previous data for better compression */<b>
ZSTD_e_end </b>/* flush any remaining data and ends current frame. Any future compression starts a new frame. */<b> ZSTD_e_end </b>/* flush any remaining data and close current frame. Any additional data starts a new frame. */<b>
} ZSTD_EndDirective; } ZSTD_EndDirective;
</b></pre><BR> </b></pre><BR>
<pre><b>size_t ZSTD_compress_generic (ZSTD_CCtx* cctx, <pre><b>size_t ZSTD_compress_generic (ZSTD_CCtx* cctx,
@ -959,8 +980,8 @@ size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t
</b><p> Behave about the same as ZSTD_compressStream. To note : </b><p> Behave about the same as ZSTD_compressStream. To note :
- Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_setParameter() - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_setParameter()
- Compression parameters cannot be changed once compression is started. - Compression parameters cannot be changed once compression is started.
- *dstPos must be <= dstCapacity, *srcPos must be <= srcSize - outpot->pos must be <= dstCapacity, input->pos must be <= srcSize
- *dspPos and *srcPos will be updated. They are guaranteed to remain below their respective limit. - outpot->pos and input->pos will be updated. They are guaranteed to remain below their respective limit.
- @return provides the minimum amount of data still to flush from internal buffers - @return provides the minimum amount of data still to flush from internal buffers
or an error code, which can be tested using ZSTD_isError(). or an error code, which can be tested using ZSTD_isError().
if @return != 0, flush is not fully completed, there is some data left within internal buffers. if @return != 0, flush is not fully completed, there is some data left within internal buffers.
@ -976,6 +997,7 @@ size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t
Useful after an error, or to interrupt an ongoing compression job and start a new one. Useful after an error, or to interrupt an ongoing compression job and start a new one.
Any internal data not yet flushed is cancelled. Any internal data not yet flushed is cancelled.
Dictionary (if any) is dropped. Dictionary (if any) is dropped.
All parameters are back to default values.
It's possible to modify compression parameters after a reset. It's possible to modify compression parameters after a reset.
</p></pre><BR> </p></pre><BR>
@ -987,26 +1009,30 @@ size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t
ZSTD_EndDirective endOp); ZSTD_EndDirective endOp);
</b><p> Same as ZSTD_compress_generic(), </b><p> Same as ZSTD_compress_generic(),
but using only integral types as arguments. but using only integral types as arguments.
Argument list is larger and less expressive than ZSTD_{in,out}Buffer, Argument list is larger than ZSTD_{in,out}Buffer,
but can be helpful for binders from dynamic languages but can be helpful for binders from dynamic languages
which have troubles handling structures containing memory pointers. which have troubles handling structures containing memory pointers.
</p></pre><BR> </p></pre><BR>
<a name="Chapter20"></a><h2>ZSTD_CCtx_params</h2><pre> <pre><b>ZSTD_CCtx_params* ZSTD_createCCtxParams(void);
</b><p> Quick howto :
- ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
- ZSTD_CCtxParam_setParameter() : Push parameters one by one into an - ZSTD_CCtxParam_setParameter() : Push parameters one by one into
existing ZSTD_CCtx_params structure. This is similar to an existing ZSTD_CCtx_params structure.
ZSTD_CCtx_setParameter(). This is similar to
- ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to an existing CCtx. These ZSTD_CCtx_setParameter().
parameters will be applied to all subsequent compression jobs. - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
an existing CCtx.
These parameters will be applied to
all subsequent compression jobs.
- ZSTD_compress_generic() : Do compression using the CCtx. - ZSTD_compress_generic() : Do compression using the CCtx.
- ZSTD_freeCCtxParams() : Free the memory. - ZSTD_freeCCtxParams() : Free the memory.
This can be used with ZSTD_estimateCCtxSize_opaque() for static allocation This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams()
for single-threaded compression. for static allocation for single-threaded compression.
<BR></pre> </p></pre><BR>
<pre><b>size_t ZSTD_resetCCtxParams(ZSTD_CCtx_params* params); <pre><b>size_t ZSTD_resetCCtxParams(ZSTD_CCtx_params* params);
</b><p> Reset params to default, with the default compression level. </b><p> Reset params to default, with the default compression level.
@ -1030,22 +1056,116 @@ size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t
Set one compression parameter, selected by enum ZSTD_cParameter. Set one compression parameter, selected by enum ZSTD_cParameter.
Parameters must be applied to a ZSTD_CCtx using ZSTD_CCtx_setParametersUsingCCtxParams(). Parameters must be applied to a ZSTD_CCtx using ZSTD_CCtx_setParametersUsingCCtxParams().
Note : when `value` is an enum, cast it to unsigned for proper type checking. Note : when `value` is an enum, cast it to unsigned for proper type checking.
@result : 0, or an error code (which can be tested with ZSTD_isError()). @result : 0, or an error code (which can be tested with ZSTD_isError()).
</p></pre><BR> </p></pre><BR>
<pre><b>size_t ZSTD_CCtx_setParametersUsingCCtxParams( <pre><b>size_t ZSTD_CCtx_setParametersUsingCCtxParams(
ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params); ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params);
</b><p> Apply a set of ZSTD_CCtx_params to the compression context. </b><p> Apply a set of ZSTD_CCtx_params to the compression context.
This must be done before the dictionary is loaded. This must be done before the dictionary is loaded.
The pledgedSrcSize is treated as unknown. The pledgedSrcSize is treated as unknown.
Multithreading parameters are applied only if nbThreads > 1. Multithreading parameters are applied only if nbThreads > 1.
</p></pre><BR> </p></pre><BR>
<a name="Chapter21"></a><h2>Block functions</h2><pre> <h3>Advanced parameters for decompression API</h3><pre></pre><b><pre></pre></b><BR>
Block functions produce and decode raw zstd blocks, without frame metadata. <pre><b>size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); </b>/* not implemented */<b>
Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes). size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); </b>/* not implemented */<b>
size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictMode_e dictMode); </b>/* not implemented */<b>
</b><p> Create an internal DDict from dict buffer,
to be used to decompress next frames.
@result : 0, or an error code (which can be tested with ZSTD_isError()).
Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
meaning "return to no-dictionary mode".
Note 1 : `dict` content will be copied internally.
Use ZSTD_DCtx_loadDictionary_byReference()
to reference dictionary content instead.
In which case, the dictionary buffer must outlive its users.
Note 2 : Loading a dictionary involves building tables,
which has a non-negligible impact on CPU usage and latency.
Note 3 : Use ZSTD_DCtx_loadDictionary_advanced() to select
how dictionary content will be interpreted and loaded.
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict); </b>/* not implemented */<b>
</b><p> Reference a prepared dictionary, to be used to decompress next frames.
The dictionary remains active for decompression of future frames using same DCtx.
@result : 0, or an error code (which can be tested with ZSTD_isError()).
Note 1 : Currently, only one dictionary can be managed.
Referencing a new dictionary effectively "discards" any previous one.
Special : adding a NULL DDict means "return to no-dictionary mode".
Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx.
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize); </b>/* not implemented */<b>
size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictMode_e dictMode); </b>/* not implemented */<b>
</b><p> Reference a prefix (single-usage dictionary) for next compression job.
Prefix is **only used once**. It must be explicitly referenced before each frame.
If there is a need to use same prefix multiple times, consider embedding it into a ZSTD_DDict instead.
@result : 0, or an error code (which can be tested with ZSTD_isError()).
Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary
Note 2 : Prefix buffer is referenced. It must outlive compression job.
Note 3 : By default, the prefix is treated as raw content (ZSTD_dm_rawContent).
Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode.
Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost.
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
</b><p> Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
This is useful to prevent a decoder context from reserving too much memory for itself (potential attack scenario).
This parameter is only useful in streaming mode, since no internal buffer is allocated in direct mode.
By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_MAX)
@return : 0, or an error code (which can be tested using ZSTD_isError()).
</p></pre><BR>
<pre><b>size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
</b><p> Instruct the decoder context about what kind of data to decode next.
This instruction is mandatory to decode data without a fully-formed header,
such ZSTD_f_zstd1_magicless for example.
@return : 0, or an error code (which can be tested using ZSTD_isError()).
</p></pre><BR>
<pre><b>size_t ZSTD_decompress_generic(ZSTD_DCtx* dctx,
ZSTD_outBuffer* output,
ZSTD_inBuffer* input);
</b><p> Behave the same as ZSTD_decompressStream.
Decompression parameters cannot be changed once decompression is started.
@return : an error code, which can be tested using ZSTD_isError()
if >0, a hint, nb of expected input bytes for next invocation.
`0` means : a frame has just been fully decoded and flushed.
</p></pre><BR>
<pre><b>size_t ZSTD_decompress_generic_simpleArgs (
ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos);
</b><p> Same as ZSTD_decompress_generic(),
but using only integral types as arguments.
Argument list is larger than ZSTD_{in,out}Buffer,
but can be helpful for binders from dynamic languages
which have troubles handling structures containing memory pointers.
</p></pre><BR>
<pre><b>void ZSTD_DCtx_reset(ZSTD_DCtx* dctx);
</b><p> Return a DCtx to clean state.
If a decompression was ongoing, any internal data not yet flushed is cancelled.
All parameters are back to default values, including sticky ones.
Dictionary (if any) is dropped.
Parameters can be modified again after a reset.
</p></pre><BR>
<a name="Chapter21"></a><h2>Block level API</h2><pre></pre>
<pre><b></b><p> Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes).
User will have to take in charge required information to regenerate data, such as compressed and content sizes. User will have to take in charge required information to regenerate data, such as compressed and content sizes.
A few rules to respect : A few rules to respect :
@ -1055,7 +1175,7 @@ size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t
+ compression : any ZSTD_compressBegin*() variant, including with dictionary + compression : any ZSTD_compressBegin*() variant, including with dictionary
+ decompression : any ZSTD_decompressBegin*() variant, including with dictionary + decompression : any ZSTD_decompressBegin*() variant, including with dictionary
+ copyCCtx() and copyDCtx() can be used too + copyCCtx() and copyDCtx() can be used too
- Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
+ If input is larger than a block size, it's necessary to split input data into multiple blocks + If input is larger than a block size, it's necessary to split input data into multiple blocks
+ For inputs larger than a single block size, consider using the regular ZSTD_compress() instead. + For inputs larger than a single block size, consider using the regular ZSTD_compress() instead.
Frame metadata is not that costly, and quickly becomes negligible as source size grows larger. Frame metadata is not that costly, and quickly becomes negligible as source size grows larger.
@ -1066,7 +1186,7 @@ size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t
+ In case of multiple successive blocks, should some of them be uncompressed, + In case of multiple successive blocks, should some of them be uncompressed,
decoder must be informed of their existence in order to follow proper history. decoder must be informed of their existence in order to follow proper history.
Use ZSTD_insertBlock() for such a case. Use ZSTD_insertBlock() for such a case.
<BR></pre> </p></pre><BR>
<h3>Raw zstd block functions</h3><pre></pre><b><pre>size_t ZSTD_getBlockSize (const ZSTD_CCtx* cctx); <h3>Raw zstd block functions</h3><pre></pre><b><pre>size_t ZSTD_getBlockSize (const ZSTD_CCtx* cctx);
size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize); size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);

2
examples/streaming_memory_usage.c
View File

@ -133,7 +133,7 @@ int main(int argc, char const *argv[]) {
size_t const cstreamSize = ZSTD_sizeof_CStream(cstream); size_t const cstreamSize = ZSTD_sizeof_CStream(cstream);
size_t const cstreamEstimatedSize = wLog ? size_t const cstreamEstimatedSize = wLog ?
ZSTD_estimateCStreamSize_advanced_usingCParams(params.cParams) : ZSTD_estimateCStreamSize_usingCParams(params.cParams) :
ZSTD_estimateCStreamSize(compressionLevel); ZSTD_estimateCStreamSize(compressionLevel);
size_t const dstreamSize = ZSTD_sizeof_DStream(dstream); size_t const dstreamSize = ZSTD_sizeof_DStream(dstream);

5
lib/common/error_private.c
View File

@ -30,14 +30,15 @@ const char* ERR_getErrorString(ERR_enum code)
case PREFIX(init_missing): return "Context should be init first"; case PREFIX(init_missing): return "Context should be init first";
case PREFIX(memory_allocation): return "Allocation error : not enough memory"; case PREFIX(memory_allocation): return "Allocation error : not enough memory";
case PREFIX(stage_wrong): return "Operation not authorized at current processing stage"; case PREFIX(stage_wrong): return "Operation not authorized at current processing stage";
case PREFIX(dstSize_tooSmall): return "Destination buffer is too small";
case PREFIX(srcSize_wrong): return "Src size is incorrect";
case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported"; case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported";
case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large"; case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large";
case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small"; case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small";
case PREFIX(dictionary_corrupted): return "Dictionary is corrupted"; case PREFIX(dictionary_corrupted): return "Dictionary is corrupted";
case PREFIX(dictionary_wrong): return "Dictionary mismatch"; case PREFIX(dictionary_wrong): return "Dictionary mismatch";
case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples"; case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples";
case PREFIX(dstSize_tooSmall): return "Destination buffer is too small";
case PREFIX(srcSize_wrong): return "Src size is incorrect";
/* following error codes are not stable and may be removed or changed in a future version */
case PREFIX(frameIndex_tooLarge): return "Frame index is too large"; case PREFIX(frameIndex_tooLarge): return "Frame index is too large";
case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking"; case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking";
case PREFIX(maxCode): case PREFIX(maxCode):

3
lib/common/zstd_errors.h
View File

@ -63,9 +63,10 @@ typedef enum {
ZSTD_error_memory_allocation = 64, ZSTD_error_memory_allocation = 64,
ZSTD_error_dstSize_tooSmall = 70, ZSTD_error_dstSize_tooSmall = 70,
ZSTD_error_srcSize_wrong = 72, ZSTD_error_srcSize_wrong = 72,
/* following error codes are not stable and may be removed or changed in a future version */
ZSTD_error_frameIndex_tooLarge = 100, ZSTD_error_frameIndex_tooLarge = 100,
ZSTD_error_seekableIO = 102, ZSTD_error_seekableIO = 102,
ZSTD_error_maxCode = 120 /* never EVER use this value directly, it may change in future versions! Use ZSTD_isError() instead */ ZSTD_error_maxCode = 120 /* never EVER use this value directly, it can change in future versions! Use ZSTD_isError() instead */
} ZSTD_ErrorCode; } ZSTD_ErrorCode;
/*! ZSTD_getErrorCode() : /*! ZSTD_getErrorCode() :

4
lib/common/zstd_internal.h
View File

@ -106,6 +106,9 @@ static const U32 repStartValue[ZSTD_REP_NUM] = { 1, 4, 8 };
static const size_t ZSTD_fcs_fieldSize[4] = { 0, 2, 4, 8 }; static const size_t ZSTD_fcs_fieldSize[4] = { 0, 2, 4, 8 };
static const size_t ZSTD_did_fieldSize[4] = { 0, 1, 2, 4 }; static const size_t ZSTD_did_fieldSize[4] = { 0, 1, 2, 4 };
#define ZSTD_FRAMEIDSIZE 4
static const size_t ZSTD_frameIdSize = ZSTD_FRAMEIDSIZE; /* magic number size */
#define ZSTD_BLOCKHEADERSIZE 3 /* C standard doesn't allow `static const` variable to be init using another `static const` variable */ #define ZSTD_BLOCKHEADERSIZE 3 /* C standard doesn't allow `static const` variable to be init using another `static const` variable */
static const size_t ZSTD_blockHeaderSize = ZSTD_BLOCKHEADERSIZE; static const size_t ZSTD_blockHeaderSize = ZSTD_BLOCKHEADERSIZE;
typedef enum { bt_raw, bt_rle, bt_compressed, bt_reserved } blockType_e; typedef enum { bt_raw, bt_rle, bt_compressed, bt_reserved } blockType_e;
@ -284,6 +287,7 @@ typedef struct {
} ZSTD_entropyCTables_t; } ZSTD_entropyCTables_t;
struct ZSTD_CCtx_params_s { struct ZSTD_CCtx_params_s {
ZSTD_format_e format;
ZSTD_compressionParameters cParams; ZSTD_compressionParameters cParams;
ZSTD_frameParameters fParams; ZSTD_frameParameters fParams;

46
lib/compress/zstd_compress.c
View File

@ -256,6 +256,9 @@ size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, unsigned v
switch(param) switch(param)
{ {
case ZSTD_p_format :
return ZSTD_CCtxParam_setParameter(&cctx->requestedParams, param, value);
case ZSTD_p_compressionLevel: case ZSTD_p_compressionLevel:
if (value == 0) return 0; /* special value : 0 means "don't change anything" */ if (value == 0) return 0; /* special value : 0 means "don't change anything" */
if (cctx->cdict) return ERROR(stage_wrong); if (cctx->cdict) return ERROR(stage_wrong);
@ -326,6 +329,12 @@ size_t ZSTD_CCtxParam_setParameter(
{ {
switch(param) switch(param)
{ {
case ZSTD_p_format :
if (value > (unsigned)ZSTD_f_zstd1_magicless)
return ERROR(parameter_unsupported);
params->format = (ZSTD_format_e)value;
return 0;
case ZSTD_p_compressionLevel : case ZSTD_p_compressionLevel :
if ((int)value > ZSTD_maxCLevel()) value = ZSTD_maxCLevel(); if ((int)value > ZSTD_maxCLevel()) value = ZSTD_maxCLevel();
if (value == 0) return 0; if (value == 0) return 0;
@ -664,7 +673,7 @@ ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, u
return ZSTD_adjustCParams_internal(cPar, srcSize, dictSize); return ZSTD_adjustCParams_internal(cPar, srcSize, dictSize);
} }
size_t ZSTD_estimateCCtxSize_advanced_usingCCtxParams(const ZSTD_CCtx_params* params) size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params)
{ {
/* Estimate CCtx size is supported for single-threaded compression only. */ /* Estimate CCtx size is supported for single-threaded compression only. */
if (params->nbThreads > 1) { return ERROR(GENERIC); } if (params->nbThreads > 1) { return ERROR(GENERIC); }
@ -701,22 +710,22 @@ size_t ZSTD_estimateCCtxSize_advanced_usingCCtxParams(const ZSTD_CCtx_params* pa
} }
} }
size_t ZSTD_estimateCCtxSize_advanced_usingCParams(ZSTD_compressionParameters cParams) size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams)
{ {
ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams); ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
return ZSTD_estimateCCtxSize_advanced_usingCCtxParams(&params); return ZSTD_estimateCCtxSize_usingCCtxParams(&params);
} }
size_t ZSTD_estimateCCtxSize(int compressionLevel) size_t ZSTD_estimateCCtxSize(int compressionLevel)
{ {
ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, 0, 0); ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, 0, 0);
return ZSTD_estimateCCtxSize_advanced_usingCParams(cParams); return ZSTD_estimateCCtxSize_usingCParams(cParams);
} }
size_t ZSTD_estimateCStreamSize_advanced_usingCCtxParams(const ZSTD_CCtx_params* params) size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params)
{ {
if (params->nbThreads > 1) { return ERROR(GENERIC); } if (params->nbThreads > 1) { return ERROR(GENERIC); }
{ size_t const CCtxSize = ZSTD_estimateCCtxSize_advanced_usingCCtxParams(params); { size_t const CCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(params);
size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << params->cParams.windowLog); size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << params->cParams.windowLog);
size_t const inBuffSize = ((size_t)1 << params->cParams.windowLog) + blockSize; size_t const inBuffSize = ((size_t)1 << params->cParams.windowLog) + blockSize;
size_t const outBuffSize = ZSTD_compressBound(blockSize) + 1; size_t const outBuffSize = ZSTD_compressBound(blockSize) + 1;
@ -726,15 +735,15 @@ size_t ZSTD_estimateCStreamSize_advanced_usingCCtxParams(const ZSTD_CCtx_params*
} }
} }
size_t ZSTD_estimateCStreamSize_advanced_usingCParams(ZSTD_compressionParameters cParams) size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams)
{ {
ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams); ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
return ZSTD_estimateCStreamSize_advanced_usingCCtxParams(&params); return ZSTD_estimateCStreamSize_usingCCtxParams(&params);
} }
size_t ZSTD_estimateCStreamSize(int compressionLevel) { size_t ZSTD_estimateCStreamSize(int compressionLevel) {
ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, 0, 0); ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, 0, 0);
return ZSTD_estimateCStreamSize_advanced_usingCParams(cParams); return ZSTD_estimateCStreamSize_usingCParams(cParams);
} }
static U32 ZSTD_equivalentCParams(ZSTD_compressionParameters cParams1, static U32 ZSTD_equivalentCParams(ZSTD_compressionParameters cParams1,
@ -1694,14 +1703,18 @@ static size_t ZSTD_writeFrameHeader(void* dst, size_t dstCapacity,
U32 const fcsCode = params.fParams.contentSizeFlag ? U32 const fcsCode = params.fParams.contentSizeFlag ?
(pledgedSrcSize>=256) + (pledgedSrcSize>=65536+256) + (pledgedSrcSize>=0xFFFFFFFFU) : 0; /* 0-3 */ (pledgedSrcSize>=256) + (pledgedSrcSize>=65536+256) + (pledgedSrcSize>=0xFFFFFFFFU) : 0; /* 0-3 */
BYTE const frameHeaderDecriptionByte = (BYTE)(dictIDSizeCode + (checksumFlag<<2) + (singleSegment<<5) + (fcsCode<<6) ); BYTE const frameHeaderDecriptionByte = (BYTE)(dictIDSizeCode + (checksumFlag<<2) + (singleSegment<<5) + (fcsCode<<6) );
size_t pos; size_t pos=0;
if (dstCapacity < ZSTD_frameHeaderSize_max) return ERROR(dstSize_tooSmall); if (dstCapacity < ZSTD_frameHeaderSize_max) return ERROR(dstSize_tooSmall);
DEBUGLOG(5, "ZSTD_writeFrameHeader : dictIDFlag : %u ; dictID : %u ; dictIDSizeCode : %u", DEBUGLOG(4, "ZSTD_writeFrameHeader : dictIDFlag : %u ; dictID : %u ; dictIDSizeCode : %u",
!params.fParams.noDictIDFlag, dictID, dictIDSizeCode); !params.fParams.noDictIDFlag, dictID, dictIDSizeCode);
MEM_writeLE32(dst, ZSTD_MAGICNUMBER); if (params.format == ZSTD_f_zstd1) {
op[4] = frameHeaderDecriptionByte; pos=5; DEBUGLOG(4, "writing zstd magic number");
MEM_writeLE32(dst, ZSTD_MAGICNUMBER);
pos = 4;
}
op[pos++] = frameHeaderDecriptionByte;
if (!singleSegment) op[pos++] = windowLogByte; if (!singleSegment) op[pos++] = windowLogByte;
switch(dictIDSizeCode) switch(dictIDSizeCode)
{ {
@ -2186,8 +2199,8 @@ size_t ZSTD_estimateCDictSize_advanced(
{ {
DEBUGLOG(5, "sizeof(ZSTD_CDict) : %u", (U32)sizeof(ZSTD_CDict)); DEBUGLOG(5, "sizeof(ZSTD_CDict) : %u", (U32)sizeof(ZSTD_CDict));
DEBUGLOG(5, "CCtx estimate : %u", DEBUGLOG(5, "CCtx estimate : %u",
(U32)ZSTD_estimateCCtxSize_advanced_usingCParams(cParams)); (U32)ZSTD_estimateCCtxSize_usingCParams(cParams));
return sizeof(ZSTD_CDict) + ZSTD_estimateCCtxSize_advanced_usingCParams(cParams) return sizeof(ZSTD_CDict) + ZSTD_estimateCCtxSize_usingCParams(cParams)
+ (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize); + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize);
} }
@ -2312,7 +2325,7 @@ ZSTD_CDict* ZSTD_initStaticCDict(void* workspace, size_t workspaceSize,
ZSTD_dictMode_e dictMode, ZSTD_dictMode_e dictMode,
ZSTD_compressionParameters cParams) ZSTD_compressionParameters cParams)
{ {
size_t const cctxSize = ZSTD_estimateCCtxSize_advanced_usingCParams(cParams); size_t const cctxSize = ZSTD_estimateCCtxSize_usingCParams(cParams);
size_t const neededSize = sizeof(ZSTD_CDict) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize) size_t const neededSize = sizeof(ZSTD_CDict) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize)
+ cctxSize; + cctxSize;
ZSTD_CDict* const cdict = (ZSTD_CDict*) workspace; ZSTD_CDict* const cdict = (ZSTD_CDict*) workspace;
@ -2574,6 +2587,7 @@ MEM_STATIC size_t ZSTD_limitCopy(void* dst, size_t dstCapacity,
/** ZSTD_compressStream_generic(): /** ZSTD_compressStream_generic():
* internal function for all *compressStream*() variants and *compress_generic() * internal function for all *compressStream*() variants and *compress_generic()
* non-static, because can be called from zstdmt.c
* @return : hint size for next input */ * @return : hint size for next input */
size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs, size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs,
ZSTD_outBuffer* output, ZSTD_outBuffer* output,

10
lib/compress/zstdmt_compress.c
View File

@ -53,17 +53,19 @@ static unsigned long long GetCurrentClockTimeMicroseconds(void)
} }
#define MUTEX_WAIT_TIME_DLEVEL 6 #define MUTEX_WAIT_TIME_DLEVEL 6
#define ZSTD_PTHREAD_MUTEX_LOCK(mutex) { \ #define ZSTD_PTHREAD_MUTEX_LOCK(mutex) { \
if (ZSTD_DEBUG>=MUTEX_WAIT_TIME_DLEVEL) { \ if (ZSTD_DEBUG >= MUTEX_WAIT_TIME_DLEVEL) { \
unsigned long long const beforeTime = GetCurrentClockTimeMicroseconds(); \ unsigned long long const beforeTime = GetCurrentClockTimeMicroseconds(); \
ZSTD_pthread_mutex_lock(mutex); \ ZSTD_pthread_mutex_lock(mutex); \
{ unsigned long long const afterTime = GetCurrentClockTimeMicroseconds(); \ { unsigned long long const afterTime = GetCurrentClockTimeMicroseconds(); \
unsigned long long const elapsedTime = (afterTime-beforeTime); \ unsigned long long const elapsedTime = (afterTime-beforeTime); \
if (elapsedTime > 1000) { /* or whatever threshold you like; I'm using 1 millisecond here */ \ if (elapsedTime > 1000) { /* or whatever threshold you like; I'm using 1 millisecond here */ \
DEBUGLOG(MUTEX_WAIT_TIME_DLEVEL, "Thread took %llu microseconds to acquire mutex %s \n", \ DEBUGLOG(MUTEX_WAIT_TIME_DLEVEL, "Thread took %llu microseconds to acquire mutex %s \n", \
elapsedTime, #mutex); \ elapsedTime, #mutex); \
} } \ } } \
} else ZSTD_pthread_mutex_lock(mutex); \ } else { \
ZSTD_pthread_mutex_lock(mutex); \
} \
} }
#else #else

338
lib/decompress/zstd_decompress.c
View File

@ -110,6 +110,7 @@ struct ZSTD_DCtx_s
XXH64_state_t xxhState; XXH64_state_t xxhState;
size_t headerSize; size_t headerSize;
U32 dictID; U32 dictID;
ZSTD_format_e format;
const BYTE* litPtr; const BYTE* litPtr;
ZSTD_customMem customMem; ZSTD_customMem customMem;
size_t litSize; size_t litSize;
@ -149,30 +150,21 @@ size_t ZSTD_sizeof_DCtx (const ZSTD_DCtx* dctx)
size_t ZSTD_estimateDCtxSize(void) { return sizeof(ZSTD_DCtx); } size_t ZSTD_estimateDCtxSize(void) { return sizeof(ZSTD_DCtx); }
size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx)
static size_t ZSTD_startingInputLength(ZSTD_format_e format)
{ {
dctx->expected = ZSTD_frameHeaderSize_prefix; size_t const startingInputLength = (format==ZSTD_f_zstd1_magicless) ?
dctx->stage = ZSTDds_getFrameHeaderSize; ZSTD_frameHeaderSize_prefix - ZSTD_frameIdSize :
dctx->decodedSize = 0; ZSTD_frameHeaderSize_prefix;
dctx->previousDstEnd = NULL; ZSTD_STATIC_ASSERT(ZSTD_FRAMEHEADERSIZE_PREFIX >= ZSTD_FRAMEIDSIZE);
dctx->base = NULL; /* only supports formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless */
dctx->vBase = NULL; assert( (format == ZSTD_f_zstd1) || (format == ZSTD_f_zstd1_magicless) );
dctx->dictEnd = NULL; return startingInputLength;
dctx->entropy.hufTable[0] = (HUF_DTable)((HufLog)*0x1000001); /* cover both little and big endian */
dctx->litEntropy = dctx->fseEntropy = 0;
dctx->dictID = 0;
MEM_STATIC_ASSERT(sizeof(dctx->entropy.rep) == sizeof(repStartValue));
memcpy(dctx->entropy.rep, repStartValue, sizeof(repStartValue)); /* initial repcodes */
dctx->LLTptr = dctx->entropy.LLTable;
dctx->MLTptr = dctx->entropy.MLTable;
dctx->OFTptr = dctx->entropy.OFTable;
dctx->HUFptr = dctx->entropy.hufTable;
return 0;
} }
static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx) static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx)
{ {
ZSTD_decompressBegin(dctx); /* cannot fail */ dctx->format = ZSTD_f_zstd1; /* ZSTD_decompressBegin() invokes ZSTD_startingInputLength() with argument dctx->format */
dctx->staticSize = 0; dctx->staticSize = 0;
dctx->maxWindowSize = ZSTD_MAXWINDOWSIZE_DEFAULT; dctx->maxWindowSize = ZSTD_MAXWINDOWSIZE_DEFAULT;
dctx->ddict = NULL; dctx->ddict = NULL;
@ -183,6 +175,19 @@ static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx)
dctx->streamStage = zdss_init; dctx->streamStage = zdss_init;
} }
ZSTD_DCtx* ZSTD_initStaticDCtx(void *workspace, size_t workspaceSize)
{
ZSTD_DCtx* const dctx = (ZSTD_DCtx*) workspace;
if ((size_t)workspace & 7) return NULL; /* 8-aligned */
if (workspaceSize < sizeof(ZSTD_DCtx)) return NULL; /* minimum size */
ZSTD_initDCtx_internal(dctx);
dctx->staticSize = workspaceSize;
dctx->inBuff = (char*)(dctx+1);
return dctx;
}
ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem) ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem)
{ {
if (!customMem.customAlloc ^ !customMem.customFree) return NULL; if (!customMem.customAlloc ^ !customMem.customFree) return NULL;
@ -197,19 +202,6 @@ ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem)
} }
} }
ZSTD_DCtx* ZSTD_initStaticDCtx(void *workspace, size_t workspaceSize)
{
ZSTD_DCtx* dctx = (ZSTD_DCtx*) workspace;
if ((size_t)workspace & 7) return NULL; /* 8-aligned */
if (workspaceSize < sizeof(ZSTD_DCtx)) return NULL; /* minimum size */
ZSTD_initDCtx_internal(dctx);
dctx->staticSize = workspaceSize;
dctx->inBuff = (char*)(dctx+1);
return dctx;
}
ZSTD_DCtx* ZSTD_createDCtx(void) ZSTD_DCtx* ZSTD_createDCtx(void)
{ {
return ZSTD_createDCtx_advanced(ZSTD_defaultCMem); return ZSTD_createDCtx_advanced(ZSTD_defaultCMem);
@ -252,7 +244,7 @@ void ZSTD_copyDCtx(ZSTD_DCtx* dstDCtx, const ZSTD_DCtx* srcDCtx)
* Note 3 : Skippable Frame Identifiers are considered valid. */ * Note 3 : Skippable Frame Identifiers are considered valid. */
unsigned ZSTD_isFrame(const void* buffer, size_t size) unsigned ZSTD_isFrame(const void* buffer, size_t size)
{ {
if (size < 4) return 0; if (size < ZSTD_frameIdSize) return 0;
{ U32 const magic = MEM_readLE32(buffer); { U32 const magic = MEM_readLE32(buffer);
if (magic == ZSTD_MAGICNUMBER) return 1; if (magic == ZSTD_MAGICNUMBER) return 1;
if ((magic & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) return 1; if ((magic & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) return 1;
@ -263,40 +255,56 @@ unsigned ZSTD_isFrame(const void* buffer, size_t size)
return 0; return 0;
} }
/** ZSTD_frameHeaderSize_internal() :
/** ZSTD_frameHeaderSize() : * srcSize must be large enough to reach header size fields.
* srcSize must be >= ZSTD_frameHeaderSize_prefix. * note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless
* @return : size of the Frame Header */ * @return : size of the Frame Header
size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize) * or an error code, which can be tested with ZSTD_isError() */
static size_t ZSTD_frameHeaderSize_internal(const void* src, size_t srcSize, ZSTD_format_e format)
{ {
if (srcSize < ZSTD_frameHeaderSize_prefix) return ERROR(srcSize_wrong); size_t const minInputSize = ZSTD_startingInputLength(format);
{ BYTE const fhd = ((const BYTE*)src)[4]; if (srcSize < minInputSize) return ERROR(srcSize_wrong);
{ BYTE const fhd = ((const BYTE*)src)[minInputSize-1];
U32 const dictID= fhd & 3; U32 const dictID= fhd & 3;
U32 const singleSegment = (fhd >> 5) & 1; U32 const singleSegment = (fhd >> 5) & 1;
U32 const fcsId = fhd >> 6; U32 const fcsId = fhd >> 6;
return ZSTD_frameHeaderSize_prefix + !singleSegment + ZSTD_did_fieldSize[dictID] + ZSTD_fcs_fieldSize[fcsId] return minInputSize + !singleSegment
+ (singleSegment && !fcsId); + ZSTD_did_fieldSize[dictID] + ZSTD_fcs_fieldSize[fcsId]
+ (singleSegment && !fcsId);
} }
} }
/** ZSTD_frameHeaderSize() :
* srcSize must be >= ZSTD_frameHeaderSize_prefix.
* @return : size of the Frame Header */
size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize)
{
return ZSTD_frameHeaderSize_internal(src, srcSize, ZSTD_f_zstd1);
}
/** ZSTD_getFrameHeader() :
* decode Frame Header, or require larger `srcSize`. /** ZSTD_getFrameHeader_internal() :
* @return : 0, `zfhPtr` is correctly filled, * decode Frame Header, or require larger `srcSize`.
* >0, `srcSize` is too small, result is expected `srcSize`, * note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless
* or an error code, which can be tested using ZSTD_isError() */ * @return : 0, `zfhPtr` is correctly filled,
size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize) * >0, `srcSize` is too small, value is wanted `srcSize` amount,
* or an error code, which can be tested using ZSTD_isError() */
static size_t ZSTD_getFrameHeader_internal(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format)
{ {
const BYTE* ip = (const BYTE*)src; const BYTE* ip = (const BYTE*)src;
if (srcSize < ZSTD_frameHeaderSize_prefix) return ZSTD_frameHeaderSize_prefix; size_t const minInputSize = ZSTD_startingInputLength(format);
if (MEM_readLE32(src) != ZSTD_MAGICNUMBER) { if (srcSize < minInputSize) return minInputSize;
if ( (format != ZSTD_f_zstd1_magicless)
&& (MEM_readLE32(src) != ZSTD_MAGICNUMBER) ) {
if ((MEM_readLE32(src) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) { if ((MEM_readLE32(src) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) {
/* skippable frame */ /* skippable frame */
if (srcSize < ZSTD_skippableHeaderSize) if (srcSize < ZSTD_skippableHeaderSize)
return ZSTD_skippableHeaderSize; /* magic number + frame length */ return ZSTD_skippableHeaderSize; /* magic number + frame length */
memset(zfhPtr, 0, sizeof(*zfhPtr)); memset(zfhPtr, 0, sizeof(*zfhPtr));
zfhPtr->frameContentSize = MEM_readLE32((const char *)src + 4); zfhPtr->frameContentSize = MEM_readLE32((const char *)src + ZSTD_frameIdSize);
zfhPtr->frameType = ZSTD_skippableFrame; zfhPtr->frameType = ZSTD_skippableFrame;
return 0; return 0;
} }
@ -304,13 +312,13 @@ size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t src
} }
/* ensure there is enough `srcSize` to fully read/decode frame header */ /* ensure there is enough `srcSize` to fully read/decode frame header */
{ size_t const fhsize = ZSTD_frameHeaderSize(src, srcSize); { size_t const fhsize = ZSTD_frameHeaderSize_internal(src, srcSize, format);
if (srcSize < fhsize) return fhsize; if (srcSize < fhsize) return fhsize;
zfhPtr->headerSize = (U32)fhsize; zfhPtr->headerSize = (U32)fhsize;
} }
{ BYTE const fhdByte = ip[4]; { BYTE const fhdByte = ip[minInputSize-1];
size_t pos = 5; size_t pos = minInputSize;
U32 const dictIDSizeCode = fhdByte&3; U32 const dictIDSizeCode = fhdByte&3;
U32 const checksumFlag = (fhdByte>>2)&1; U32 const checksumFlag = (fhdByte>>2)&1;
U32 const singleSegment = (fhdByte>>5)&1; U32 const singleSegment = (fhdByte>>5)&1;
@ -357,6 +365,18 @@ size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t src
return 0; return 0;
} }
/** ZSTD_getFrameHeader() :
* decode Frame Header, or require larger `srcSize`.
* note : this function does not consume input, it only reads it.
* @return : 0, `zfhPtr` is correctly filled,
* >0, `srcSize` is too small, value is wanted `srcSize` amount,
* or an error code, which can be tested using ZSTD_isError() */
size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize)
{
return ZSTD_getFrameHeader_internal(zfhPtr, src, srcSize, ZSTD_f_zstd1);
}
/** ZSTD_getFrameContentSize() : /** ZSTD_getFrameContentSize() :
* compatible with legacy mode * compatible with legacy mode
* @return : decompressed size of the single frame pointed to be `src` if known, otherwise * @return : decompressed size of the single frame pointed to be `src` if known, otherwise
@ -390,14 +410,14 @@ unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize)
unsigned long long totalDstSize = 0; unsigned long long totalDstSize = 0;
while (srcSize >= ZSTD_frameHeaderSize_prefix) { while (srcSize >= ZSTD_frameHeaderSize_prefix) {
const U32 magicNumber = MEM_readLE32(src); U32 const magicNumber = MEM_readLE32(src);
if ((magicNumber & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) { if ((magicNumber & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) {
size_t skippableSize; size_t skippableSize;
if (srcSize < ZSTD_skippableHeaderSize) if (srcSize < ZSTD_skippableHeaderSize)
return ERROR(srcSize_wrong); return ERROR(srcSize_wrong);
skippableSize = MEM_readLE32((const BYTE *)src + 4) + skippableSize = MEM_readLE32((const BYTE *)src + ZSTD_frameIdSize)
ZSTD_skippableHeaderSize; + ZSTD_skippableHeaderSize;
if (srcSize < skippableSize) { if (srcSize < skippableSize) {
return ZSTD_CONTENTSIZE_ERROR; return ZSTD_CONTENTSIZE_ERROR;
} }
@ -422,11 +442,9 @@ unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize)
src = (const BYTE *)src + frameSrcSize; src = (const BYTE *)src + frameSrcSize;
srcSize -= frameSrcSize; srcSize -= frameSrcSize;
} }
} } /* while (srcSize >= ZSTD_frameHeaderSize_prefix) */
if (srcSize) { if (srcSize) return ZSTD_CONTENTSIZE_ERROR;
return ZSTD_CONTENTSIZE_ERROR;
}
return totalDstSize; return totalDstSize;
} }
@ -442,7 +460,8 @@ unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize)
unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize) unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize)
{ {
unsigned long long const ret = ZSTD_getFrameContentSize(src, srcSize); unsigned long long const ret = ZSTD_getFrameContentSize(src, srcSize);
return ret >= ZSTD_CONTENTSIZE_ERROR ? 0 : ret; ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_ERROR < ZSTD_CONTENTSIZE_UNKNOWN);
return (ret >= ZSTD_CONTENTSIZE_ERROR) ? 0 : ret;
} }
@ -451,9 +470,9 @@ unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize)
* @return : 0 if success, or an error code, which can be tested using ZSTD_isError() */ * @return : 0 if success, or an error code, which can be tested using ZSTD_isError() */
static size_t ZSTD_decodeFrameHeader(ZSTD_DCtx* dctx, const void* src, size_t headerSize) static size_t ZSTD_decodeFrameHeader(ZSTD_DCtx* dctx, const void* src, size_t headerSize)
{ {
size_t const result = ZSTD_getFrameHeader(&(dctx->fParams), src, headerSize); size_t const result = ZSTD_getFrameHeader_internal(&(dctx->fParams), src, headerSize, dctx->format);
if (ZSTD_isError(result)) return result; /* invalid header */ if (ZSTD_isError(result)) return result; /* invalid header */
if (result>0) return ERROR(srcSize_wrong); /* headerSize too small */ if (result>0) return ERROR(srcSize_wrong); /* headerSize too small */
if (dctx->fParams.dictID && (dctx->dictID != dctx->fParams.dictID)) if (dctx->fParams.dictID && (dctx->dictID != dctx->fParams.dictID))
return ERROR(dictionary_wrong); return ERROR(dictionary_wrong);
if (dctx->fParams.checksumFlag) XXH64_reset(&dctx->xxhState, 0); if (dctx->fParams.checksumFlag) XXH64_reset(&dctx->xxhState, 0);
@ -499,7 +518,8 @@ static size_t ZSTD_setRleBlock(void* dst, size_t dstCapacity,
} }
/*! ZSTD_decodeLiteralsBlock() : /*! ZSTD_decodeLiteralsBlock() :
@return : nb of bytes read from src (< srcSize ) */ * @return : nb of bytes read from src (< srcSize )
* note : symbol not declared but exposed for fullbench */
size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx, size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
const void* src, size_t srcSize) /* note : srcSize < BLOCKSIZE */ const void* src, size_t srcSize) /* note : srcSize < BLOCKSIZE */
{ {
@ -700,9 +720,9 @@ static const FSE_decode_t4 OF_defaultDTable[(1<<OF_DEFAULTNORMLOG)+1] = {
}; /* OF_defaultDTable */ }; /* OF_defaultDTable */
/*! ZSTD_buildSeqTable() : /*! ZSTD_buildSeqTable() :
@return : nb bytes read from src, * @return : nb bytes read from src,
or an error code if it fails, testable with ZSTD_isError() * or an error code if it fails, testable with ZSTD_isError()
*/ */
static size_t ZSTD_buildSeqTable(FSE_DTable* DTableSpace, const FSE_DTable** DTablePtr, static size_t ZSTD_buildSeqTable(FSE_DTable* DTableSpace, const FSE_DTable** DTablePtr,
symbolEncodingType_e type, U32 max, U32 maxLog, symbolEncodingType_e type, U32 max, U32 maxLog,
const void* src, size_t srcSize, const void* src, size_t srcSize,
@ -1445,7 +1465,7 @@ ZSTDLIB_API size_t ZSTD_insertBlock(ZSTD_DCtx* dctx, const void* blockStart, siz
} }
size_t ZSTD_generateNxBytes(void* dst, size_t dstCapacity, BYTE byte, size_t length) static size_t ZSTD_generateNxBytes(void* dst, size_t dstCapacity, BYTE byte, size_t length)
{ {
if (length > dstCapacity) return ERROR(dstSize_tooSmall); if (length > dstCapacity) return ERROR(dstSize_tooSmall);
memset(dst, byte, length); memset(dst, byte, length);
@ -1465,7 +1485,7 @@ size_t ZSTD_findFrameCompressedSize(const void *src, size_t srcSize)
#endif #endif
if ( (srcSize >= ZSTD_skippableHeaderSize) if ( (srcSize >= ZSTD_skippableHeaderSize)
&& (MEM_readLE32(src) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START ) { && (MEM_readLE32(src) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START ) {
return ZSTD_skippableHeaderSize + MEM_readLE32((const BYTE*)src + 4); return ZSTD_skippableHeaderSize + MEM_readLE32((const BYTE*)src + ZSTD_frameIdSize);
} else { } else {
const BYTE* ip = (const BYTE*)src; const BYTE* ip = (const BYTE*)src;
const BYTE* const ipstart = ip; const BYTE* const ipstart = ip;
@ -1628,13 +1648,15 @@ static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx* dctx,
#endif #endif
magicNumber = MEM_readLE32(src); magicNumber = MEM_readLE32(src);
DEBUGLOG(4, "reading magic number %08X (expecting %08X)",
(U32)magicNumber, (U32)ZSTD_MAGICNUMBER);
if (magicNumber != ZSTD_MAGICNUMBER) { if (magicNumber != ZSTD_MAGICNUMBER) {
if ((magicNumber & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) { if ((magicNumber & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) {
size_t skippableSize; size_t skippableSize;
if (srcSize < ZSTD_skippableHeaderSize) if (srcSize < ZSTD_skippableHeaderSize)
return ERROR(srcSize_wrong); return ERROR(srcSize_wrong);
skippableSize = MEM_readLE32((const BYTE *)src + 4) + skippableSize = MEM_readLE32((const BYTE*)src + ZSTD_frameIdSize)
ZSTD_skippableHeaderSize; + ZSTD_skippableHeaderSize;
if (srcSize < skippableSize) return ERROR(srcSize_wrong); if (srcSize < skippableSize) return ERROR(srcSize_wrong);
src = (const BYTE *)src + skippableSize; src = (const BYTE *)src + skippableSize;
@ -1737,33 +1759,31 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
{ {
DEBUGLOG(5, "ZSTD_decompressContinue"); DEBUGLOG(5, "ZSTD_decompressContinue");
/* Sanity check */ /* Sanity check */
if (srcSize != dctx->expected) return ERROR(srcSize_wrong); /* unauthorized */ if (srcSize != dctx->expected) return ERROR(srcSize_wrong); /* not allowed */
if (dstCapacity) ZSTD_checkContinuity(dctx, dst); if (dstCapacity) ZSTD_checkContinuity(dctx, dst);
switch (dctx->stage) switch (dctx->stage)
{ {
case ZSTDds_getFrameHeaderSize : case ZSTDds_getFrameHeaderSize :
if (srcSize != ZSTD_frameHeaderSize_prefix) return ERROR(srcSize_wrong); /* unauthorized */
assert(src != NULL); assert(src != NULL);
if ((MEM_readLE32(src) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */ if (dctx->format == ZSTD_f_zstd1) { /* allows header */
memcpy(dctx->headerBuffer, src, ZSTD_frameHeaderSize_prefix); assert(srcSize >= ZSTD_frameIdSize); /* to read skippable magic number */
dctx->expected = ZSTD_skippableHeaderSize - ZSTD_frameHeaderSize_prefix; /* magic number + skippable frame length */ if ((MEM_readLE32(src) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */
dctx->stage = ZSTDds_decodeSkippableHeader; memcpy(dctx->headerBuffer, src, srcSize);
return 0; dctx->expected = ZSTD_skippableHeaderSize - srcSize; /* remaining to load to get full skippable frame header */
} dctx->stage = ZSTDds_decodeSkippableHeader;
dctx->headerSize = ZSTD_frameHeaderSize(src, ZSTD_frameHeaderSize_prefix); return 0;
} }
dctx->headerSize = ZSTD_frameHeaderSize_internal(src, srcSize, dctx->format);
if (ZSTD_isError(dctx->headerSize)) return dctx->headerSize; if (ZSTD_isError(dctx->headerSize)) return dctx->headerSize;
memcpy(dctx->headerBuffer, src, ZSTD_frameHeaderSize_prefix); memcpy(dctx->headerBuffer, src, srcSize);
if (dctx->headerSize > ZSTD_frameHeaderSize_prefix) { dctx->expected = dctx->headerSize - srcSize;
dctx->expected = dctx->headerSize - ZSTD_frameHeaderSize_prefix; dctx->stage = ZSTDds_decodeFrameHeader;
dctx->stage = ZSTDds_decodeFrameHeader; return 0;
return 0;
}
dctx->expected = 0; /* not necessary to copy more */
/* fall-through */
case ZSTDds_decodeFrameHeader: case ZSTDds_decodeFrameHeader:
assert(src != NULL); assert(src != NULL);
memcpy(dctx->headerBuffer + ZSTD_frameHeaderSize_prefix, src, dctx->expected); memcpy(dctx->headerBuffer + (dctx->headerSize - srcSize), src, srcSize);
CHECK_F(ZSTD_decodeFrameHeader(dctx, dctx->headerBuffer, dctx->headerSize)); CHECK_F(ZSTD_decodeFrameHeader(dctx, dctx->headerBuffer, dctx->headerSize));
dctx->expected = ZSTD_blockHeaderSize; dctx->expected = ZSTD_blockHeaderSize;
dctx->stage = ZSTDds_decodeBlockHeader; dctx->stage = ZSTDds_decodeBlockHeader;
@ -1795,6 +1815,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
} }
return 0; return 0;
} }
case ZSTDds_decompressLastBlock: case ZSTDds_decompressLastBlock:
case ZSTDds_decompressBlock: case ZSTDds_decompressBlock:
DEBUGLOG(5, "case ZSTDds_decompressBlock"); DEBUGLOG(5, "case ZSTDds_decompressBlock");
@ -1840,29 +1861,31 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
} }
return rSize; return rSize;
} }
case ZSTDds_checkChecksum: case ZSTDds_checkChecksum:
DEBUGLOG(4, "case ZSTDds_checkChecksum");
assert(srcSize == 4); /* guaranteed by dctx->expected */ assert(srcSize == 4); /* guaranteed by dctx->expected */
{ U32 const h32 = (U32)XXH64_digest(&dctx->xxhState); { U32 const h32 = (U32)XXH64_digest(&dctx->xxhState);
U32 const check32 = MEM_readLE32(src); U32 const check32 = MEM_readLE32(src);
DEBUGLOG(4, "calculated %08X :: %08X read", h32, check32); DEBUGLOG(4, "checksum : calculated %08X :: %08X read", h32, check32);
if (check32 != h32) return ERROR(checksum_wrong); if (check32 != h32) return ERROR(checksum_wrong);
dctx->expected = 0; dctx->expected = 0;
dctx->stage = ZSTDds_getFrameHeaderSize; dctx->stage = ZSTDds_getFrameHeaderSize;
return 0; return 0;
} }
case ZSTDds_decodeSkippableHeader: case ZSTDds_decodeSkippableHeader:
{ assert(src != NULL); assert(src != NULL);
memcpy(dctx->headerBuffer + ZSTD_frameHeaderSize_prefix, src, dctx->expected); assert(srcSize <= ZSTD_skippableHeaderSize);
dctx->expected = MEM_readLE32(dctx->headerBuffer + 4); memcpy(dctx->headerBuffer + (ZSTD_skippableHeaderSize - srcSize), src, srcSize); /* complete skippable header */
dctx->stage = ZSTDds_skipFrame; dctx->expected = MEM_readLE32(dctx->headerBuffer + ZSTD_frameIdSize); /* note : dctx->expected can grow seriously large, beyond local buffer size */
return 0; dctx->stage = ZSTDds_skipFrame;
} return 0;
case ZSTDds_skipFrame: case ZSTDds_skipFrame:
{ dctx->expected = 0; dctx->expected = 0;
dctx->stage = ZSTDds_getFrameHeaderSize; dctx->stage = ZSTDds_getFrameHeaderSize;
return 0; return 0;
}
default: default:
return ERROR(GENERIC); /* impossible */ return ERROR(GENERIC); /* impossible */
} }
@ -1943,7 +1966,7 @@ static size_t ZSTD_decompress_insertDictionary(ZSTD_DCtx* dctx, const void* dict
if (magic != ZSTD_MAGIC_DICTIONARY) { if (magic != ZSTD_MAGIC_DICTIONARY) {
return ZSTD_refDictContent(dctx, dict, dictSize); /* pure content mode */ return ZSTD_refDictContent(dctx, dict, dictSize); /* pure content mode */
} } } }
dctx->dictID = MEM_readLE32((const char*)dict + 4); dctx->dictID = MEM_readLE32((const char*)dict + ZSTD_frameIdSize);
/* load entropy tables */ /* load entropy tables */
{ size_t const eSize = ZSTD_loadEntropy(&dctx->entropy, dict, dictSize); { size_t const eSize = ZSTD_loadEntropy(&dctx->entropy, dict, dictSize);
@ -1957,6 +1980,29 @@ static size_t ZSTD_decompress_insertDictionary(ZSTD_DCtx* dctx, const void* dict
return ZSTD_refDictContent(dctx, dict, dictSize); return ZSTD_refDictContent(dctx, dict, dictSize);
} }
/* Note : this function cannot fail */
size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx)
{
assert(dctx != NULL);
dctx->expected = ZSTD_startingInputLength(dctx->format); /* dctx->format must be properly set */
dctx->stage = ZSTDds_getFrameHeaderSize;
dctx->decodedSize = 0;
dctx->previousDstEnd = NULL;
dctx->base = NULL;
dctx->vBase = NULL;
dctx->dictEnd = NULL;
dctx->entropy.hufTable[0] = (HUF_DTable)((HufLog)*0x1000001); /* cover both little and big endian */
dctx->litEntropy = dctx->fseEntropy = 0;
dctx->dictID = 0;
ZSTD_STATIC_ASSERT(sizeof(dctx->entropy.rep) == sizeof(repStartValue));
memcpy(dctx->entropy.rep, repStartValue, sizeof(repStartValue)); /* initial repcodes */
dctx->LLTptr = dctx->entropy.LLTable;
dctx->MLTptr = dctx->entropy.MLTable;
dctx->OFTptr = dctx->entropy.OFTable;
dctx->HUFptr = dctx->entropy.hufTable;
return 0;
}
size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize) size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
{ {
CHECK_F( ZSTD_decompressBegin(dctx) ); CHECK_F( ZSTD_decompressBegin(dctx) );
@ -2023,7 +2069,7 @@ static size_t ZSTD_loadEntropy_inDDict(ZSTD_DDict* ddict)
{ U32 const magic = MEM_readLE32(ddict->dictContent); { U32 const magic = MEM_readLE32(ddict->dictContent);
if (magic != ZSTD_MAGIC_DICTIONARY) return 0; /* pure content mode */ if (magic != ZSTD_MAGIC_DICTIONARY) return 0; /* pure content mode */
} }
ddict->dictID = MEM_readLE32((const char*)ddict->dictContent + 4); ddict->dictID = MEM_readLE32((const char*)ddict->dictContent + ZSTD_frameIdSize);
/* load entropy tables */ /* load entropy tables */
CHECK_E( ZSTD_loadEntropy(&ddict->entropy, ddict->dictContent, ddict->dictSize), dictionary_corrupted ); CHECK_E( ZSTD_loadEntropy(&ddict->entropy, ddict->dictContent, ddict->dictSize), dictionary_corrupted );
@ -2144,7 +2190,7 @@ unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize)
{ {
if (dictSize < 8) return 0; if (dictSize < 8) return 0;
if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) return 0; if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) return 0;
return MEM_readLE32((const char*)dict + 4); return MEM_readLE32((const char*)dict + ZSTD_frameIdSize);
} }
/*! ZSTD_getDictID_fromDDict() : /*! ZSTD_getDictID_fromDDict() :
@ -2239,13 +2285,15 @@ size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t di
return ZSTD_frameHeaderSize_prefix; return ZSTD_frameHeaderSize_prefix;
} }
/* note : this variant can't fail */
size_t ZSTD_initDStream(ZSTD_DStream* zds) size_t ZSTD_initDStream(ZSTD_DStream* zds)
{ {
return ZSTD_initDStream_usingDict(zds, NULL, 0); return ZSTD_initDStream_usingDict(zds, NULL, 0);
} }
/* ZSTD_initDStream_usingDDict() : /* ZSTD_initDStream_usingDDict() :
* ddict will just be referenced, and must outlive decompression session */ * ddict will just be referenced, and must outlive decompression session
* this function cannot fail */
size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict) size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict)
{ {
size_t const initResult = ZSTD_initDStream(zds); size_t const initResult = ZSTD_initDStream(zds);
@ -2265,14 +2313,39 @@ size_t ZSTD_resetDStream(ZSTD_DStream* zds)
size_t ZSTD_setDStreamParameter(ZSTD_DStream* zds, size_t ZSTD_setDStreamParameter(ZSTD_DStream* zds,
ZSTD_DStreamParameter_e paramType, unsigned paramValue) ZSTD_DStreamParameter_e paramType, unsigned paramValue)
{ {
ZSTD_STATIC_ASSERT((unsigned)zdss_loadHeader >= (unsigned)zdss_init);
if ((unsigned)zds->streamStage > (unsigned)zdss_loadHeader)
return ERROR(stage_wrong);
switch(paramType) switch(paramType)
{ {
default : return ERROR(parameter_unsupported); default : return ERROR(parameter_unsupported);
case DStream_p_maxWindowSize : zds->maxWindowSize = paramValue ? paramValue : (U32)(-1); break; case DStream_p_maxWindowSize :
DEBUGLOG(4, "setting maxWindowSize = %u KB", paramValue >> 10);
zds->maxWindowSize = paramValue ? paramValue : (U32)(-1);
break;
} }
return 0; return 0;
} }
size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize)
{
ZSTD_STATIC_ASSERT((unsigned)zdss_loadHeader >= (unsigned)zdss_init);
if ((unsigned)dctx->streamStage > (unsigned)zdss_loadHeader)
return ERROR(stage_wrong);
dctx->maxWindowSize = maxWindowSize;
return 0;
}
size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format)
{
DEBUGLOG(4, "ZSTD_DCtx_setFormat : %u", (unsigned)format);
ZSTD_STATIC_ASSERT((unsigned)zdss_loadHeader >= (unsigned)zdss_init);
if ((unsigned)dctx->streamStage > (unsigned)zdss_loadHeader)
return ERROR(stage_wrong);
dctx->format = format;
return 0;
}
size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds) size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds)
{ {
@ -2297,7 +2370,7 @@ size_t ZSTD_estimateDStreamSize(size_t windowSize)
return ZSTD_estimateDCtxSize() + inBuffSize + outBuffSize; return ZSTD_estimateDCtxSize() + inBuffSize + outBuffSize;
} }
ZSTDLIB_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize) size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize)
{ {
U32 const windowSizeMax = 1U << ZSTD_WINDOWLOG_MAX; /* note : should be user-selectable */ U32 const windowSizeMax = 1U << ZSTD_WINDOWLOG_MAX; /* note : should be user-selectable */
ZSTD_frameHeader zfh; ZSTD_frameHeader zfh;
@ -2331,7 +2404,18 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
U32 someMoreWork = 1; U32 someMoreWork = 1;
DEBUGLOG(5, "ZSTD_decompressStream"); DEBUGLOG(5, "ZSTD_decompressStream");
if (input->pos > input->size) { /* forbidden */
DEBUGLOG(5, "in: pos: %u vs size: %u",
(U32)input->pos, (U32)input->size);
return ERROR(srcSize_wrong);
}
if (output->pos > output->size) { /* forbidden */
DEBUGLOG(5, "out: pos: %u vs size: %u",
(U32)output->pos, (U32)output->size);
return ERROR(dstSize_tooSmall);
}
DEBUGLOG(5, "input size : %u", (U32)(input->size - input->pos)); DEBUGLOG(5, "input size : %u", (U32)(input->size - input->pos));
#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1)
if (zds->legacyVersion) { if (zds->legacyVersion) {
/* legacy support is incompatible with static dctx */ /* legacy support is incompatible with static dctx */
@ -2348,7 +2432,9 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
/* fall-through */ /* fall-through */
case zdss_loadHeader : case zdss_loadHeader :
{ size_t const hSize = ZSTD_getFrameHeader(&zds->fParams, zds->headerBuffer, zds->lhSize); DEBUGLOG(5, "stage zdss_loadHeader (srcSize : %u)", (U32)(iend - ip));
{ size_t const hSize = ZSTD_getFrameHeader_internal(&zds->fParams, zds->headerBuffer, zds->lhSize, zds->format);
DEBUGLOG(5, "header size : %u", (U32)hSize);
if (ZSTD_isError(hSize)) { if (ZSTD_isError(hSize)) {
#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) #if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1)
U32 const legacyVersion = ZSTD_isLegacy(istart, iend-istart); U32 const legacyVersion = ZSTD_isLegacy(istart, iend-istart);
@ -2401,7 +2487,7 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
CHECK_F(ZSTD_decompressBegin_usingDDict(zds, zds->ddict)); CHECK_F(ZSTD_decompressBegin_usingDDict(zds, zds->ddict));
if ((MEM_readLE32(zds->headerBuffer) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */ if ((MEM_readLE32(zds->headerBuffer) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */
zds->expected = MEM_readLE32(zds->headerBuffer + 4); zds->expected = MEM_readLE32(zds->headerBuffer + ZSTD_frameIdSize);
zds->stage = ZSTDds_skipFrame; zds->stage = ZSTDds_skipFrame;
} else { } else {
CHECK_F(ZSTD_decodeFrameHeader(zds, zds->headerBuffer, zds->lhSize)); CHECK_F(ZSTD_decodeFrameHeader(zds, zds->headerBuffer, zds->lhSize));
@ -2410,7 +2496,8 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
} }
/* control buffer memory usage */ /* control buffer memory usage */
DEBUGLOG(4, "Control max buffer memory usage"); DEBUGLOG(4, "Control max buffer memory usage (max %u KB)",
(U32)(zds->maxWindowSize >> 10));
zds->fParams.windowSize = MAX(zds->fParams.windowSize, 1U << ZSTD_WINDOWLOG_ABSOLUTEMIN); zds->fParams.windowSize = MAX(zds->fParams.windowSize, 1U << ZSTD_WINDOWLOG_ABSOLUTEMIN);
if (zds->fParams.windowSize > zds->maxWindowSize) return ERROR(frameParameter_windowTooLarge); if (zds->fParams.windowSize > zds->maxWindowSize) return ERROR(frameParameter_windowTooLarge);
@ -2539,3 +2626,30 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
return nextSrcSizeHint; return nextSrcSizeHint;
} }
} }
size_t ZSTD_decompress_generic(ZSTD_DCtx* dctx, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
{
return ZSTD_decompressStream(dctx, output, input);
}
size_t ZSTD_decompress_generic_simpleArgs (
ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos)
{
ZSTD_outBuffer output = { dst, dstCapacity, *dstPos };
ZSTD_inBuffer input = { src, srcSize, *srcPos };
/* ZSTD_compress_generic() will check validity of dstPos and srcPos */
size_t const cErr = ZSTD_decompress_generic(dctx, &output, &input);
*dstPos = output.pos;
*srcPos = input.pos;
return cErr;
}
void ZSTD_DCtx_reset(ZSTD_DCtx* dctx)
{
(void)ZSTD_initDStream(dctx);
dctx->format = ZSTD_f_zstd1;
dctx->maxWindowSize = ZSTD_MAXWINDOWSIZE_DEFAULT;
}

252
lib/zstd.h
View File

@ -395,11 +395,12 @@ ZSTDLIB_API size_t ZSTD_DStreamOutSize(void); /*!< recommended size for output
#define ZSTD_LDM_MINMATCH_MAX 4096 #define ZSTD_LDM_MINMATCH_MAX 4096
#define ZSTD_LDM_BUCKETSIZELOG_MAX 8 #define ZSTD_LDM_BUCKETSIZELOG_MAX 8
#define ZSTD_FRAMEHEADERSIZE_MAX 18 /* for static allocation */ #define ZSTD_FRAMEHEADERSIZE_PREFIX 5 /* minimum input size to know frame header size */
#define ZSTD_FRAMEHEADERSIZE_MIN 6 #define ZSTD_FRAMEHEADERSIZE_MIN 6
static const size_t ZSTD_frameHeaderSize_prefix = 5; /* minimum input size to know frame header size */ #define ZSTD_FRAMEHEADERSIZE_MAX 18 /* for static allocation */
static const size_t ZSTD_frameHeaderSize_max = ZSTD_FRAMEHEADERSIZE_MAX; static const size_t ZSTD_frameHeaderSize_prefix = ZSTD_FRAMEHEADERSIZE_PREFIX;
static const size_t ZSTD_frameHeaderSize_min = ZSTD_FRAMEHEADERSIZE_MIN; static const size_t ZSTD_frameHeaderSize_min = ZSTD_FRAMEHEADERSIZE_MIN;
static const size_t ZSTD_frameHeaderSize_max = ZSTD_FRAMEHEADERSIZE_MAX;
static const size_t ZSTD_skippableHeaderSize = 8; /* magic number + skippable frame length */ static const size_t ZSTD_skippableHeaderSize = 8; /* magic number + skippable frame length */
@ -486,7 +487,7 @@ ZSTDLIB_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);
/*! ZSTD_sizeof_*() : /*! ZSTD_sizeof_*() :
* These functions give the current memory usage of selected object. * These functions give the current memory usage of selected object.
* Object memory usage can evolve if it's re-used multiple times. */ * Object memory usage can evolve when re-used multiple times. */
ZSTDLIB_API size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx); ZSTDLIB_API size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx);
ZSTDLIB_API size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx); ZSTDLIB_API size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx);
ZSTDLIB_API size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs); ZSTDLIB_API size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs);
@ -499,21 +500,21 @@ ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
* of a future {D,C}Ctx, before its creation. * of a future {D,C}Ctx, before its creation.
* ZSTD_estimateCCtxSize() will provide a budget large enough for any compression level up to selected one. * ZSTD_estimateCCtxSize() will provide a budget large enough for any compression level up to selected one.
* It will also consider src size to be arbitrarily "large", which is worst case. * It will also consider src size to be arbitrarily "large", which is worst case.
* If srcSize is known to always be small, ZSTD_estimateCCtxSize_advanced_usingCParams() can provide a tighter estimation. * If srcSize is known to always be small, ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation.
* ZSTD_estimateCCtxSize_advanced_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. * ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
* ZSTD_estimateCCtxSize_advanced_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is > 1. * ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is > 1.
* Note : CCtx estimation is only correct for single-threaded compression */ * Note : CCtx estimation is only correct for single-threaded compression */
ZSTDLIB_API size_t ZSTD_estimateCCtxSize(int compressionLevel); ZSTDLIB_API size_t ZSTD_estimateCCtxSize(int compressionLevel);
ZSTDLIB_API size_t ZSTD_estimateCCtxSize_advanced_usingCParams(ZSTD_compressionParameters cParams); ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
ZSTDLIB_API size_t ZSTD_estimateCCtxSize_advanced_usingCCtxParams(const ZSTD_CCtx_params* params); ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
ZSTDLIB_API size_t ZSTD_estimateDCtxSize(void); ZSTDLIB_API size_t ZSTD_estimateDCtxSize(void);
/*! ZSTD_estimateCStreamSize() : /*! ZSTD_estimateCStreamSize() :
* ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one. * ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one.
* It will also consider src size to be arbitrarily "large", which is worst case. * It will also consider src size to be arbitrarily "large", which is worst case.
* If srcSize is known to always be small, ZSTD_estimateCStreamSize_advanced_usingCParams() can provide a tighter estimation. * If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation.
* ZSTD_estimateCStreamSize_advanced_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. * ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
* ZSTD_estimateCStreamSize_advanced_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is set to a value > 1. * ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbThreads is set to a value > 1.
* Note : CStream estimation is only correct for single-threaded compression. * Note : CStream estimation is only correct for single-threaded compression.
* ZSTD_DStream memory budget depends on window Size. * ZSTD_DStream memory budget depends on window Size.
* This information can be passed manually, using ZSTD_estimateDStreamSize, * This information can be passed manually, using ZSTD_estimateDStreamSize,
@ -522,14 +523,14 @@ ZSTDLIB_API size_t ZSTD_estimateDCtxSize(void);
* an internal ?Dict will be created, which additional size is not estimated here. * an internal ?Dict will be created, which additional size is not estimated here.
* In this case, get total size by adding ZSTD_estimate?DictSize */ * In this case, get total size by adding ZSTD_estimate?DictSize */
ZSTDLIB_API size_t ZSTD_estimateCStreamSize(int compressionLevel); ZSTDLIB_API size_t ZSTD_estimateCStreamSize(int compressionLevel);
ZSTDLIB_API size_t ZSTD_estimateCStreamSize_advanced_usingCParams(ZSTD_compressionParameters cParams); ZSTDLIB_API size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams);
ZSTDLIB_API size_t ZSTD_estimateCStreamSize_advanced_usingCCtxParams(const ZSTD_CCtx_params* params); ZSTDLIB_API size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params);
ZSTDLIB_API size_t ZSTD_estimateDStreamSize(size_t windowSize); ZSTDLIB_API size_t ZSTD_estimateDStreamSize(size_t windowSize);
ZSTDLIB_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize); ZSTDLIB_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
typedef enum { typedef enum {
ZSTD_dlm_byCopy = 0, /* Copy dictionary content internally. */ ZSTD_dlm_byCopy = 0, /**< Copy dictionary content internally */
ZSTD_dlm_byRef, /* Reference dictionary content -- the dictionary buffer must outlives its users. */ ZSTD_dlm_byRef, /**< Reference dictionary content -- the dictionary buffer must outlive its users. */
} ZSTD_dictLoadMethod_e; } ZSTD_dictLoadMethod_e;
/*! ZSTD_estimate?DictSize() : /*! ZSTD_estimate?DictSize() :
@ -747,12 +748,12 @@ ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledg
/*===== Advanced Streaming decompression functions =====*/ /*===== Advanced Streaming decompression functions =====*/
typedef enum { DStream_p_maxWindowSize } ZSTD_DStreamParameter_e;
ZSTDLIB_API ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem); ZSTDLIB_API ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
ZSTDLIB_API ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); /**< same as ZSTD_initStaticDCtx() */ ZSTDLIB_API ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); /**< same as ZSTD_initStaticDCtx() */
ZSTDLIB_API size_t ZSTD_setDStreamParameter(ZSTD_DStream* zds, ZSTD_DStreamParameter_e paramType, unsigned paramValue); typedef enum { DStream_p_maxWindowSize } ZSTD_DStreamParameter_e;
ZSTDLIB_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); /**< note: a dict will not be used if dict == NULL or dictSize < 8 */ ZSTDLIB_API size_t ZSTD_setDStreamParameter(ZSTD_DStream* zds, ZSTD_DStreamParameter_e paramType, unsigned paramValue); /* obsolete : this API will be removed in a future version */
ZSTDLIB_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); /**< note : ddict will just be referenced, and must outlive decompression session */ ZSTDLIB_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize); /**< note: no dictionary will be used if dict == NULL or dictSize < 8 */
ZSTDLIB_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict); /**< note : ddict is referenced, it must outlive decompression session */
ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds); /**< re-use decompression parameters from previous init; saves dictionary loading */ ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds); /**< re-use decompression parameters from previous init; saves dictionary loading */
@ -760,8 +761,8 @@ ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds); /**< re-use decompress
* Buffer-less and synchronous inner streaming functions * Buffer-less and synchronous inner streaming functions
* *
* This is an advanced API, giving full control over buffer management, for users which need direct control over memory. * This is an advanced API, giving full control over buffer management, for users which need direct control over memory.
* But it's also a complex one, with many restrictions (documented below). * But it's also a complex one, with several restrictions, documented below.
* Prefer using normal streaming API for an easier experience * Prefer normal streaming API for an easier experience.
********************************************************************* */ ********************************************************************* */
/** /**
@ -778,8 +779,8 @@ ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds); /**< re-use decompress
Then, consume your input using ZSTD_compressContinue(). Then, consume your input using ZSTD_compressContinue().
There are some important considerations to keep in mind when using this advanced function : There are some important considerations to keep in mind when using this advanced function :
- ZSTD_compressContinue() has no internal buffer. It uses externally provided buffer only. - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
- Interface is synchronous : input is consumed entirely and produce 1+ (or more) compressed blocks. - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
- Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario. - Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario.
Worst case evaluation is provided by ZSTD_compressBound(). Worst case evaluation is provided by ZSTD_compressBound().
ZSTD_compressContinue() doesn't guarantee recover after a failed compression. ZSTD_compressContinue() doesn't guarantee recover after a failed compression.
@ -790,9 +791,9 @@ ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds); /**< re-use decompress
Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum. Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame. It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
Without last block mark, frames will be considered unfinished (corrupted) by decoders. Without last block mark, frames are considered unfinished (hence corrupted) by compliant decoders.
`ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress some new frame. `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
*/ */
/*===== Buffer-less streaming compression functions =====*/ /*===== Buffer-less streaming compression functions =====*/
@ -908,33 +909,55 @@ ZSTDLIB_API ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx);
/*=== New advanced API (experimental, and compression only) ===*/ /* ============================================ */
/** New advanced API (experimental) */
/* ============================================ */
/* notes on API design : /* notes on API design :
* In this proposal, parameters are pushed one by one into an existing CCtx, * In this proposal, parameters are pushed one by one into an existing context,
* and then applied on all subsequent compression jobs. * and then applied on all subsequent compression jobs.
* When no parameter is ever provided, CCtx is created with compression level ZSTD_CLEVEL_DEFAULT. * When no parameter is ever provided, CCtx is created with compression level ZSTD_CLEVEL_DEFAULT.
* *
* This API is intended to replace all others experimental API. * This API is intended to replace all others experimental API.
* It can basically do all other use cases, and even new ones. * It can basically do all other use cases, and even new ones.
* It stands a good chance to become "stable", * In constrast with _advanced() variants, it stands a reasonable chance to become "stable",
* after a reasonable testing period. * after a good testing period.
*/ */
/* note on naming convention : /* note on naming convention :
* Initially, the API favored names like ZSTD_setCCtxParameter() . * Initially, the API favored names like ZSTD_setCCtxParameter() .
* In this proposal, convention is changed towards ZSTD_CCtx_setParameter() . * In this proposal, convention is changed towards ZSTD_CCtx_setParameter() .
* The main driver is that it identifies more clearly the target object type. * The main driver is that it identifies more clearly the target object type.
* It feels clearer in light of potential variants : * It feels clearer when considering multiple targets :
* ZSTD_CDict_setParameter() (rather than ZSTD_setCDictParameter()) * ZSTD_CDict_setParameter() (rather than ZSTD_setCDictParameter())
* ZSTD_DCtx_setParameter() (rather than ZSTD_setDCtxParameter() ) * ZSTD_CCtxParams_setParameter() (rather than ZSTD_setCCtxParamsParameter() )
* Left variant feels easier to distinguish. * etc...
*/ */
/* note on enum design : /* note on enum design :
* All enum will be manually set to explicit values before reaching "stable API" status */ * All enum will be pinned to explicit values before reaching "stable API" status */
typedef enum { typedef enum {
/* Question : should we have a format ZSTD_f_auto ?
* For the time being, it would mean exactly the same as ZSTD_f_zstd1.
* But, in the future, should several formats be supported,
* on the compression side, it would mean "default format".
* On the decompression side, it would mean "multi format",
* and ZSTD_f_zstd1 could be reserved to mean "accept *only* zstd frames".
* Since meaning is a little different, another option could be to define different enums for compression and decompression.
* This question could be kept for later, when there are actually multiple formats to support,
* but there is also the question of pinning enum values, and pinning value `0` is especially important */
ZSTD_f_zstd1 = 0, /* zstd frame format, specified in zstd_compression_format.md (default) */
ZSTD_f_zstd1_magicless, /* Variant of zstd frame format, without initial 4-bytes magic number.
* Useful to save 4 bytes per generated frame.
* Decoder cannot recognise automatically this format, requiring instructions. */
} ZSTD_format_e;
typedef enum {
/* compression format */
ZSTD_p_format = 10, /* See ZSTD_format_e enum definition.
* Cast selected format as unsigned for ZSTD_CCtx_setParameter() compatibility. */
/* compression parameters */ /* compression parameters */
ZSTD_p_compressionLevel=100, /* Update all compression parameters according to pre-defined cLevel table ZSTD_p_compressionLevel=100, /* Update all compression parameters according to pre-defined cLevel table
* Default level is ZSTD_CLEVEL_DEFAULT==3. * Default level is ZSTD_CLEVEL_DEFAULT==3.
@ -1105,15 +1128,15 @@ ZSTDLIB_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* pre
typedef enum { typedef enum {
ZSTD_e_continue=0, /* collect more data, encoder transparently decides when to output result, for optimal conditions */ ZSTD_e_continue=0, /* collect more data, encoder transparently decides when to output result, for optimal conditions */
ZSTD_e_flush, /* flush any data provided so far - frame will continue, future data can still reference previous data for better compression */ ZSTD_e_flush, /* flush any data provided so far - frame will continue, future data can still reference previous data for better compression */
ZSTD_e_end /* flush any remaining data and ends current frame. Any future compression starts a new frame. */ ZSTD_e_end /* flush any remaining data and close current frame. Any additional data starts a new frame. */
} ZSTD_EndDirective; } ZSTD_EndDirective;
/*! ZSTD_compress_generic() : /*! ZSTD_compress_generic() :
* Behave about the same as ZSTD_compressStream. To note : * Behave about the same as ZSTD_compressStream. To note :
* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_setParameter() * - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_setParameter()
* - Compression parameters cannot be changed once compression is started. * - Compression parameters cannot be changed once compression is started.
* - *dstPos must be <= dstCapacity, *srcPos must be <= srcSize * - outpot->pos must be <= dstCapacity, input->pos must be <= srcSize
* - *dspPos and *srcPos will be updated. They are guaranteed to remain below their respective limit. * - outpot->pos and input->pos will be updated. They are guaranteed to remain below their respective limit.
* - @return provides the minimum amount of data still to flush from internal buffers * - @return provides the minimum amount of data still to flush from internal buffers
* or an error code, which can be tested using ZSTD_isError(). * or an error code, which can be tested using ZSTD_isError().
* if @return != 0, flush is not fully completed, there is some data left within internal buffers. * if @return != 0, flush is not fully completed, there is some data left within internal buffers.
@ -1132,6 +1155,7 @@ ZSTDLIB_API size_t ZSTD_compress_generic (ZSTD_CCtx* cctx,
* Useful after an error, or to interrupt an ongoing compression job and start a new one. * Useful after an error, or to interrupt an ongoing compression job and start a new one.
* Any internal data not yet flushed is cancelled. * Any internal data not yet flushed is cancelled.
* Dictionary (if any) is dropped. * Dictionary (if any) is dropped.
* All parameters are back to default values.
* It's possible to modify compression parameters after a reset. * It's possible to modify compression parameters after a reset.
*/ */
ZSTDLIB_API void ZSTD_CCtx_reset(ZSTD_CCtx* cctx); /* Not ready yet ! */ ZSTDLIB_API void ZSTD_CCtx_reset(ZSTD_CCtx* cctx); /* Not ready yet ! */
@ -1140,30 +1164,33 @@ ZSTDLIB_API void ZSTD_CCtx_reset(ZSTD_CCtx* cctx); /* Not ready yet ! */
/*! ZSTD_compress_generic_simpleArgs() : /*! ZSTD_compress_generic_simpleArgs() :
* Same as ZSTD_compress_generic(), * Same as ZSTD_compress_generic(),
* but using only integral types as arguments. * but using only integral types as arguments.
* Argument list is larger and less expressive than ZSTD_{in,out}Buffer, * Argument list is larger than ZSTD_{in,out}Buffer,
* but can be helpful for binders from dynamic languages * but can be helpful for binders from dynamic languages
* which have troubles handling structures containing memory pointers. * which have troubles handling structures containing memory pointers.
*/ */
size_t ZSTD_compress_generic_simpleArgs ( ZSTDLIB_API size_t ZSTD_compress_generic_simpleArgs (
ZSTD_CCtx* cctx, ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, size_t* dstPos, void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos, const void* src, size_t srcSize, size_t* srcPos,
ZSTD_EndDirective endOp); ZSTD_EndDirective endOp);
/** ZSTD_CCtx_params /*! ZSTD_CCtx_params :
* * Quick howto :
* - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
* - ZSTD_CCtxParam_setParameter() : Push parameters one by one into an * - ZSTD_CCtxParam_setParameter() : Push parameters one by one into
* existing ZSTD_CCtx_params structure. This is similar to * an existing ZSTD_CCtx_params structure.
* ZSTD_CCtx_setParameter(). * This is similar to
* - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to an existing CCtx. These * ZSTD_CCtx_setParameter().
* parameters will be applied to all subsequent compression jobs. * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
* an existing CCtx.
* These parameters will be applied to
* all subsequent compression jobs.
* - ZSTD_compress_generic() : Do compression using the CCtx. * - ZSTD_compress_generic() : Do compression using the CCtx.
* - ZSTD_freeCCtxParams() : Free the memory. * - ZSTD_freeCCtxParams() : Free the memory.
* *
* This can be used with ZSTD_estimateCCtxSize_opaque() for static allocation * This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams()
* for single-threaded compression. * for static allocation for single-threaded compression.
*/ */
ZSTDLIB_API ZSTD_CCtx_params* ZSTD_createCCtxParams(void); ZSTDLIB_API ZSTD_CCtx_params* ZSTD_createCCtxParams(void);
@ -1191,22 +1218,133 @@ ZSTDLIB_API size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params);
* Set one compression parameter, selected by enum ZSTD_cParameter. * Set one compression parameter, selected by enum ZSTD_cParameter.
* Parameters must be applied to a ZSTD_CCtx using ZSTD_CCtx_setParametersUsingCCtxParams(). * Parameters must be applied to a ZSTD_CCtx using ZSTD_CCtx_setParametersUsingCCtxParams().
* Note : when `value` is an enum, cast it to unsigned for proper type checking. * Note : when `value` is an enum, cast it to unsigned for proper type checking.
* @result : 0, or an error code (which can be tested with ZSTD_isError()). * @result : 0, or an error code (which can be tested with ZSTD_isError()).
*/ */
ZSTDLIB_API size_t ZSTD_CCtxParam_setParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, unsigned value); ZSTDLIB_API size_t ZSTD_CCtxParam_setParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, unsigned value);
/*! ZSTD_CCtx_setParametersUsingCCtxParams() : /*! ZSTD_CCtx_setParametersUsingCCtxParams() :
* Apply a set of ZSTD_CCtx_params to the compression context. * Apply a set of ZSTD_CCtx_params to the compression context.
* This must be done before the dictionary is loaded. * This must be done before the dictionary is loaded.
* The pledgedSrcSize is treated as unknown. * The pledgedSrcSize is treated as unknown.
* Multithreading parameters are applied only if nbThreads > 1. * Multithreading parameters are applied only if nbThreads > 1.
*/ */
ZSTDLIB_API size_t ZSTD_CCtx_setParametersUsingCCtxParams( ZSTDLIB_API size_t ZSTD_CCtx_setParametersUsingCCtxParams(
ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params); ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params);
/**
Block functions
/*=== Advanced parameters for decompression API ===*/
/* The following parameters must be set after creating a ZSTD_DCtx* (or ZSTD_DStream*) object,
* but before starting decompression of a frame.
*/
/*! ZSTD_DCtx_loadDictionary() :
* Create an internal DDict from dict buffer,
* to be used to decompress next frames.
* @result : 0, or an error code (which can be tested with ZSTD_isError()).
* Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
* meaning "return to no-dictionary mode".
* Note 1 : `dict` content will be copied internally.
* Use ZSTD_DCtx_loadDictionary_byReference()
* to reference dictionary content instead.
* In which case, the dictionary buffer must outlive its users.
* Note 2 : Loading a dictionary involves building tables,
* which has a non-negligible impact on CPU usage and latency.
* Note 3 : Use ZSTD_DCtx_loadDictionary_advanced() to select
* how dictionary content will be interpreted and loaded.
*/
ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); /* not implemented */
ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize); /* not implemented */
ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictMode_e dictMode); /* not implemented */
/*! ZSTD_DCtx_refDDict() :
* Reference a prepared dictionary, to be used to decompress next frames.
* The dictionary remains active for decompression of future frames using same DCtx.
* @result : 0, or an error code (which can be tested with ZSTD_isError()).
* Note 1 : Currently, only one dictionary can be managed.
* Referencing a new dictionary effectively "discards" any previous one.
* Special : adding a NULL DDict means "return to no-dictionary mode".
* Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx.
*/
ZSTDLIB_API size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict); /* not implemented */
/*! ZSTD_DCtx_refPrefix() :
* Reference a prefix (single-usage dictionary) for next compression job.
* Prefix is **only used once**. It must be explicitly referenced before each frame.
* If there is a need to use same prefix multiple times, consider embedding it into a ZSTD_DDict instead.
* @result : 0, or an error code (which can be tested with ZSTD_isError()).
* Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary
* Note 2 : Prefix buffer is referenced. It must outlive compression job.
* Note 3 : By default, the prefix is treated as raw content (ZSTD_dm_rawContent).
* Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode.
* Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost.
*/
ZSTDLIB_API size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize); /* not implemented */
ZSTDLIB_API size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictMode_e dictMode); /* not implemented */
/*! ZSTD_DCtx_setMaxWindowSize() :
* Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
* This is useful to prevent a decoder context from reserving too much memory for itself (potential attack scenario).
* This parameter is only useful in streaming mode, since no internal buffer is allocated in direct mode.
* By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_MAX)
* @return : 0, or an error code (which can be tested using ZSTD_isError()).
*/
ZSTDLIB_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
/*! ZSTD_DCtx_setFormat() :
* Instruct the decoder context about what kind of data to decode next.
* This instruction is mandatory to decode data without a fully-formed header,
* such ZSTD_f_zstd1_magicless for example.
* @return : 0, or an error code (which can be tested using ZSTD_isError()).
*/
ZSTDLIB_API size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
/*! ZSTD_decompress_generic() :
* Behave the same as ZSTD_decompressStream.
* Decompression parameters cannot be changed once decompression is started.
* @return : an error code, which can be tested using ZSTD_isError()
* if >0, a hint, nb of expected input bytes for next invocation.
* `0` means : a frame has just been fully decoded and flushed.
*/
ZSTDLIB_API size_t ZSTD_decompress_generic(ZSTD_DCtx* dctx,
ZSTD_outBuffer* output,
ZSTD_inBuffer* input);
/*! ZSTD_decompress_generic_simpleArgs() :
* Same as ZSTD_decompress_generic(),
* but using only integral types as arguments.
* Argument list is larger than ZSTD_{in,out}Buffer,
* but can be helpful for binders from dynamic languages
* which have troubles handling structures containing memory pointers.
*/
ZSTDLIB_API size_t ZSTD_decompress_generic_simpleArgs (
ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos);
/*! ZSTD_DCtx_reset() :
* Return a DCtx to clean state.
* If a decompression was ongoing, any internal data not yet flushed is cancelled.
* All parameters are back to default values, including sticky ones.
* Dictionary (if any) is dropped.
* Parameters can be modified again after a reset.
*/
ZSTDLIB_API void ZSTD_DCtx_reset(ZSTD_DCtx* dctx);
/* ============================ */
/** Block level API */
/* ============================ */
/*!
Block functions produce and decode raw zstd blocks, without frame metadata. Block functions produce and decode raw zstd blocks, without frame metadata.
Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes). Frame metadata cost is typically ~18 bytes, which can be non-negligible for very small blocks (< 100 bytes).
User will have to take in charge required information to regenerate data, such as compressed and content sizes. User will have to take in charge required information to regenerate data, such as compressed and content sizes.
@ -1218,7 +1356,7 @@ ZSTDLIB_API size_t ZSTD_CCtx_setParametersUsingCCtxParams(
+ compression : any ZSTD_compressBegin*() variant, including with dictionary + compression : any ZSTD_compressBegin*() variant, including with dictionary
+ decompression : any ZSTD_decompressBegin*() variant, including with dictionary + decompression : any ZSTD_decompressBegin*() variant, including with dictionary
+ copyCCtx() and copyDCtx() can be used too + copyCCtx() and copyDCtx() can be used too
- Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
+ If input is larger than a block size, it's necessary to split input data into multiple blocks + If input is larger than a block size, it's necessary to split input data into multiple blocks
+ For inputs larger than a single block size, consider using the regular ZSTD_compress() instead. + For inputs larger than a single block size, consider using the regular ZSTD_compress() instead.
Frame metadata is not that costly, and quickly becomes negligible as source size grows larger. Frame metadata is not that costly, and quickly becomes negligible as source size grows larger.

2
programs/Makefile
View File

@ -40,7 +40,7 @@ CPPFLAGS+= -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \
-DZSTD_NEWAPI \ -DZSTD_NEWAPI \
-DXXH_NAMESPACE=ZSTD_ # because xxhash.o already compiled with this macro from library -DXXH_NAMESPACE=ZSTD_ # because xxhash.o already compiled with this macro from library
CFLAGS ?= -O3 CFLAGS ?= -O3
DEBUGFLAGS= -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \ DEBUGFLAGS+=-Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
-Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \ -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \
-Wstrict-prototypes -Wundef -Wpointer-arith -Wformat-security \ -Wstrict-prototypes -Wundef -Wpointer-arith -Wformat-security \
-Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \ -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \

2
programs/fileio.c
View File

@ -1040,7 +1040,7 @@ static dRess_t FIO_createDResources(const char* dictFileName)
/* Allocation */ /* Allocation */
ress.dctx = ZSTD_createDStream(); ress.dctx = ZSTD_createDStream();
if (ress.dctx==NULL) EXM_THROW(60, "Can't create ZSTD_DStream"); if (ress.dctx==NULL) EXM_THROW(60, "Can't create ZSTD_DStream");
ZSTD_setDStreamParameter(ress.dctx, DStream_p_maxWindowSize, g_memLimit); CHECK( ZSTD_setDStreamParameter(ress.dctx, DStream_p_maxWindowSize, g_memLimit) );
ress.srcBufferSize = ZSTD_DStreamInSize(); ress.srcBufferSize = ZSTD_DStreamInSize();
ress.srcBuffer = malloc(ress.srcBufferSize); ress.srcBuffer = malloc(ress.srcBufferSize);
ress.dstBufferSize = ZSTD_DStreamOutSize(); ress.dstBufferSize = ZSTD_DStreamOutSize();

1
tests/fullbench.c
View File

@ -376,6 +376,7 @@ static size_t benchMem(const void* src, size_t srcSize, U32 benchNb)
skippedSize = frameHeaderSize + ZSTD_blockHeaderSize; skippedSize = frameHeaderSize + ZSTD_blockHeaderSize;
memcpy(buff2, dstBuff+skippedSize, g_cSize-skippedSize); memcpy(buff2, dstBuff+skippedSize, g_cSize-skippedSize);
srcSize = srcSize > 128 KB ? 128 KB : srcSize; /* speed relative to block */ srcSize = srcSize > 128 KB ? 128 KB : srcSize; /* speed relative to block */
ZSTD_decompressBegin(g_zdc);
break; break;
} }
case 32: /* ZSTD_decodeSeqHeaders */ case 32: /* ZSTD_decodeSeqHeaders */

5
tests/fuzz/.gitignore vendored Normal file
View File

@ -0,0 +1,5 @@
# test artefacts
corpora
block_decompress
block_round_trip
simple_round_trip

39
tests/fuzzer.c
View File

@ -918,6 +918,43 @@ static int basicUnitTests(U32 seed, double compressibility)
ZSTD_freeCCtx(cctx); ZSTD_freeCCtx(cctx);
} }
/* custom formats tests */
{ ZSTD_CCtx* const cctx = ZSTD_createCCtx();
size_t const inputSize = CNBuffSize / 2; /* won't cause pb with small dict size */
/* basic block compression */
DISPLAYLEVEL(4, "test%3i : magic-less format test : ", testNb++);
CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_p_format, ZSTD_f_zstd1_magicless) );
{ ZSTD_inBuffer in = { CNBuffer, inputSize, 0 };
ZSTD_outBuffer out = { compressedBuffer, ZSTD_compressBound(inputSize), 0 };
size_t const result = ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end);
if (result != 0) goto _output_error;
if (in.pos != in.size) goto _output_error;
cSize = out.pos;
}
DISPLAYLEVEL(4, "OK (compress : %u -> %u bytes)\n", (U32)inputSize, (U32)cSize);
DISPLAYLEVEL(4, "test%3i : decompress normally (should fail) : ", testNb++);
{ size_t const decodeResult = ZSTD_decompressDCtx(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize);
if (ZSTD_getErrorCode(decodeResult) != ZSTD_error_prefix_unknown) goto _output_error;
DISPLAYLEVEL(4, "OK : %s \n", ZSTD_getErrorName(decodeResult));
}
DISPLAYLEVEL(4, "test%3i : decompress with magic-less instruction : ", testNb++);
ZSTD_DCtx_reset(dctx);
CHECK( ZSTD_DCtx_setFormat(dctx, ZSTD_f_zstd1_magicless) );
{ ZSTD_inBuffer in = { compressedBuffer, cSize, 0 };
ZSTD_outBuffer out = { decodedBuffer, CNBuffSize, 0 };
size_t const result = ZSTD_decompress_generic(dctx, &out, &in);
if (result != 0) goto _output_error;
if (in.pos != in.size) goto _output_error;
if (out.pos != inputSize) goto _output_error;
DISPLAYLEVEL(4, "OK : regenerated %u bytes \n", (U32)out.pos);
}
ZSTD_freeCCtx(cctx);
}
/* block API tests */ /* block API tests */
{ ZSTD_CCtx* const cctx = ZSTD_createCCtx(); { ZSTD_CCtx* const cctx = ZSTD_createCCtx();
static const size_t dictSize = 65 KB; static const size_t dictSize = 65 KB;
@ -961,8 +998,8 @@ static int basicUnitTests(U32 seed, double compressibility)
DISPLAYLEVEL(4, "OK \n"); DISPLAYLEVEL(4, "OK \n");
ZSTD_freeCCtx(cctx); ZSTD_freeCCtx(cctx);
ZSTD_freeDCtx(dctx);
} }
ZSTD_freeDCtx(dctx);
/* long rle test */ /* long rle test */
{ size_t sampleSize = 0; { size_t sampleSize = 0;

4
tests/paramgrill.c
View File

@ -391,8 +391,8 @@ static int BMK_seed(winnerInfo_t* winners, const ZSTD_compressionParameters para
double W_DMemUsed_note = W_ratioNote * ( 40 + 9*cLevel) - log((double)W_DMemUsed); double W_DMemUsed_note = W_ratioNote * ( 40 + 9*cLevel) - log((double)W_DMemUsed);
double O_DMemUsed_note = O_ratioNote * ( 40 + 9*cLevel) - log((double)O_DMemUsed); double O_DMemUsed_note = O_ratioNote * ( 40 + 9*cLevel) - log((double)O_DMemUsed);
size_t W_CMemUsed = (1 << params.windowLog) + ZSTD_estimateCCtxSize_advanced_usingCParams(params); size_t W_CMemUsed = (1 << params.windowLog) + ZSTD_estimateCCtxSize_usingCParams(params);
size_t O_CMemUsed = (1 << winners[cLevel].params.windowLog) + ZSTD_estimateCCtxSize_advanced_usingCParams(winners[cLevel].params); size_t O_CMemUsed = (1 << winners[cLevel].params.windowLog) + ZSTD_estimateCCtxSize_usingCParams(winners[cLevel].params);
double W_CMemUsed_note = W_ratioNote * ( 50 + 13*cLevel) - log((double)W_CMemUsed); double W_CMemUsed_note = W_ratioNote * ( 50 + 13*cLevel) - log((double)W_CMemUsed);
double O_CMemUsed_note = O_ratioNote * ( 50 + 13*cLevel) - log((double)O_CMemUsed); double O_CMemUsed_note = O_ratioNote * ( 50 + 13*cLevel) - log((double)O_CMemUsed);

6
tests/playTests.sh
View File

@ -55,7 +55,6 @@ then
fi fi
isWindows=false isWindows=false
ECHO="echo -e"
INTOVOID="/dev/null" INTOVOID="/dev/null"
case "$OS" in case "$OS" in
Windows*) Windows*)
@ -76,6 +75,11 @@ case "$UNAME" in
SunOS) DIFF="gdiff" ;; SunOS) DIFF="gdiff" ;;
esac esac
ECHO="echo -e"
case "$UNAME" in
Darwin) ECHO="echo" ;;
esac
$ECHO "\nStarting playTests.sh isWindows=$isWindows ZSTD='$ZSTD'" $ECHO "\nStarting playTests.sh isWindows=$isWindows ZSTD='$ZSTD'"
[ -n "$ZSTD" ] || die "ZSTD variable must be defined!" [ -n "$ZSTD" ] || die "ZSTD variable must be defined!"

16
tests/zstreamtest.c
View File

@ -201,11 +201,11 @@ static int basicUnitTests(U32 seed, double compressibility, ZSTD_customMem custo
/* context size functions */ /* context size functions */
DISPLAYLEVEL(3, "test%3i : estimate CStream size : ", testNb++); DISPLAYLEVEL(3, "test%3i : estimate CStream size : ", testNb++);
{ ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBufferSize, dictSize); { ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBufferSize, dictSize);
size_t const s = ZSTD_estimateCStreamSize_advanced_usingCParams(cParams) size_t const cstreamSize = ZSTD_estimateCStreamSize_usingCParams(cParams);
/* uses ZSTD_initCStream_usingDict() */ size_t const cdictSize = ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy); /* uses ZSTD_initCStream_usingDict() */
+ ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy); if (ZSTD_isError(cstreamSize)) goto _output_error;
if (ZSTD_isError(s)) goto _output_error; if (ZSTD_isError(cdictSize)) goto _output_error;
DISPLAYLEVEL(3, "OK (%u bytes) \n", (U32)s); DISPLAYLEVEL(3, "OK (%u bytes) \n", (U32)(cstreamSize + cdictSize));
} }
DISPLAYLEVEL(3, "test%3i : check actual CStream size : ", testNb++); DISPLAYLEVEL(3, "test%3i : check actual CStream size : ", testNb++);
@ -915,7 +915,7 @@ static int fuzzerTests(U32 seed, U32 nbTests, unsigned startTest, double compres
size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog); size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
size_t const dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize); size_t const dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize);
inBuff.size = inBuff.pos + readCSrcSize; inBuff.size = inBuff.pos + readCSrcSize;
outBuff.size = inBuff.pos + dstBuffSize; outBuff.size = outBuff.pos + dstBuffSize;
decompressionResult = ZSTD_decompressStream(zd, &outBuff, &inBuff); decompressionResult = ZSTD_decompressStream(zd, &outBuff, &inBuff);
if (ZSTD_getErrorCode(decompressionResult) == ZSTD_error_checksum_wrong) { if (ZSTD_getErrorCode(decompressionResult) == ZSTD_error_checksum_wrong) {
DISPLAY("checksum error : \n"); DISPLAY("checksum error : \n");
@ -1179,7 +1179,7 @@ static int fuzzerTests_MT(U32 seed, U32 nbTests, unsigned startTest, double comp
size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog); size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
size_t const dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize); size_t const dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize);
inBuff.size = inBuff.pos + readCSrcSize; inBuff.size = inBuff.pos + readCSrcSize;
outBuff.size = inBuff.pos + dstBuffSize; outBuff.size = outBuff.pos + dstBuffSize;
DISPLAYLEVEL(5, "ZSTD_decompressStream input %u bytes \n", (U32)readCSrcSize); DISPLAYLEVEL(5, "ZSTD_decompressStream input %u bytes \n", (U32)readCSrcSize);
decompressionResult = ZSTD_decompressStream(zd, &outBuff, &inBuff); decompressionResult = ZSTD_decompressStream(zd, &outBuff, &inBuff);
CHECK (ZSTD_isError(decompressionResult), "decompression error : %s", ZSTD_getErrorName(decompressionResult)); CHECK (ZSTD_isError(decompressionResult), "decompression error : %s", ZSTD_getErrorName(decompressionResult));
@ -1507,7 +1507,7 @@ static int fuzzerTests_newAPI(U32 seed, U32 nbTests, unsigned startTest, double
size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog); size_t const randomDstSize = FUZ_randomLength(&lseed, maxSampleLog);
size_t const dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize); size_t const dstBuffSize = MIN(dstBufferSize - totalGenSize, randomDstSize);
inBuff.size = inBuff.pos + readCSrcSize; inBuff.size = inBuff.pos + readCSrcSize;
outBuff.size = inBuff.pos + dstBuffSize; outBuff.size = outBuff.pos + dstBuffSize;
DISPLAYLEVEL(5, "ZSTD_decompressStream input %u bytes (pos:%u/%u)\n", DISPLAYLEVEL(5, "ZSTD_decompressStream input %u bytes (pos:%u/%u)\n",
(U32)readCSrcSize, (U32)inBuff.pos, (U32)cSize); (U32)readCSrcSize, (U32)inBuff.pos, (U32)cSize);
decompressionResult = ZSTD_decompressStream(zd, &outBuff, &inBuff); decompressionResult = ZSTD_decompressStream(zd, &outBuff, &inBuff);