facebook/zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

created ZSTD_compress2() and ZSTD_compressStream2()

Browse Source 
ZSTD_compress_generic() is renamed ZSTD_compressStream2().

Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
This commit is contained in:
Yann Collet 2018-11-30 11:16:26 -08:00
parent d3a0c71259
commit d8e215cbee
12 changed files with 133 additions and 141 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
doc/zstd_manual.html
View File

@ -484,10 +484,11 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
</b>/* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).<b> </b>/* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).<b>
* They return an error otherwise. */ * They return an error otherwise. */
ZSTD_p_nbWorkers=400, </b>/* Select how many threads will be spawned to compress in parallel.<b> ZSTD_p_nbWorkers=400, </b>/* Select how many threads will be spawned to compress in parallel.<b>
* When nbWorkers >= 1, triggers asynchronous mode : * When nbWorkers >= 1, triggers asynchronous mode when used with ZSTD_compressStream2() :
* ZSTD_compress_generic() consumes input and flush output if possible, but immediately gives back control to caller, * ZSTD_compressStream2() consumes input and flush output if possible, but immediately gives back control to caller,
* while compression work is performed in parallel, within worker threads. * while compression work is performed in parallel, within worker threads.
* (note : a strong exception to this rule is when first invocation sets ZSTD_e_end : it becomes a blocking call). * (note : a strong exception to this rule is when first invocation sets ZSTD_e_end :
* in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call).
* More workers improve speed, but also increase memory usage. * More workers improve speed, but also increase memory usage.
* Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */ * Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */
ZSTD_p_jobSize=401, </b>/* Size of a compression job. This value is enforced only when nbWorkers >= 1.<b> ZSTD_p_jobSize=401, </b>/* Size of a compression job. This value is enforced only when nbWorkers >= 1.<b>
@ -630,6 +631,18 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
</p></pre><BR> </p></pre><BR>
<pre><b>size_t ZSTD_compress2( ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize);
</b><p> Behave the same as ZSTD_compressCCtx(), but compression parameters are set using the advanced API.
- Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
- The function is always blocking.
Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`.
@return : compressed size written into `dst` (<= `dstCapacity),
or an error code if it fails (which can be tested using ZSTD_isError()).
</p></pre><BR>
<pre><b>typedef enum { <pre><b>typedef enum {
ZSTD_e_continue=0, </b>/* collect more data, encoder decides when to output compressed result, for optimal compression ratio */<b> ZSTD_e_continue=0, </b>/* collect more data, encoder decides when to output compressed result, for optimal compression ratio */<b>
ZSTD_e_flush=1, </b>/* flush any data provided so far,<b> ZSTD_e_flush=1, </b>/* flush any data provided so far,<b>
@ -640,11 +653,11 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
* each frame is independent (does not reference any content from previous frame). */ * each frame is independent (does not reference any content from previous frame). */
} ZSTD_EndDirective; } ZSTD_EndDirective;
</b></pre><BR> </b></pre><BR>
<pre><b>size_t ZSTD_compress_generic (ZSTD_CCtx* cctx, <pre><b>size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
ZSTD_outBuffer* output, ZSTD_outBuffer* output,
ZSTD_inBuffer* input, ZSTD_inBuffer* input,
ZSTD_EndDirective endOp); ZSTD_EndDirective endOp);
</b><p> Behave about the same as ZSTD_compressStream. To note : </b><p> Behave about the same as ZSTD_compressStream, with additional control on end directive.
- Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*() - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
- Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode) - Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode)
- outpot->pos must be <= dstCapacity, input->pos must be <= srcSize - outpot->pos must be <= dstCapacity, input->pos must be <= srcSize
@ -653,7 +666,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
- In multi-thread mode, function is non-blocking : it just acquires a copy of input, and distribute job to internal worker threads, - In multi-thread mode, function is non-blocking : it just acquires a copy of input, and distribute job to internal worker threads,
and then immediately returns, just indicating that there is some data remaining to be flushed. and then immediately returns, just indicating that there is some data remaining to be flushed.
The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte. The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte.
- Exception : in multi-threading mode, if the first call requests a ZSTD_e_end directive, it is blocking : it will complete compression before giving back control to caller. - Exception : if the first call requests a ZSTD_e_end directive, the function delegates to ZSTD_compress2() which is always blocking.
- @return provides a minimum amount of data remaining to be flushed from internal buffers - @return provides a minimum amount of data remaining to be flushed from internal buffers
or an error code, which can be tested using ZSTD_isError(). or an error code, which can be tested using ZSTD_isError().
if @return != 0, flush is not fully completed, there is still some data left within internal buffers. if @return != 0, flush is not fully completed, there is still some data left within internal buffers.
@ -666,7 +679,7 @@ size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
</p></pre><BR> </p></pre><BR>
<pre><b> ZSTDLIB_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize); <pre><b>size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
</b><p> Refuses allocating internal buffers for frames requiring a window size larger than provided limit. </b><p> Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
This protects a decoder context from reserving too much memory for itself (potential attack scenario). This protects a decoder context from reserving too much memory for itself (potential attack scenario).
This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode. This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
@ -1018,7 +1031,7 @@ size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params);
an existing CCtx. an existing CCtx.
These parameters will be applied to These parameters will be applied to
all subsequent compression jobs. all subsequent compression jobs.
- ZSTD_compress_generic() : Do compression using the CCtx. - ZSTD_compressStream2() : Do compression using the CCtx.
- ZSTD_freeCCtxParams() : Free the memory. - ZSTD_freeCCtxParams() : Free the memory.
This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams() This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams()
@ -1068,12 +1081,12 @@ size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params);
</p></pre><BR> </p></pre><BR>
<pre><b>size_t ZSTD_compress_generic_simpleArgs ( <pre><b>size_t ZSTD_compressStream2_simpleArgs (
ZSTD_CCtx* cctx, ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, size_t* dstPos, void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos, const void* src, size_t srcSize, size_t* srcPos,
ZSTD_EndDirective endOp); ZSTD_EndDirective endOp);
</b><p> Same as ZSTD_compress_generic(), </b><p> Same as ZSTD_compressStream2(),
but using only integral types as arguments. but using only integral types as arguments.
This variant might be helpful for binders from dynamic languages This variant might be helpful for binders from dynamic languages
which have troubles handling structures containing memory pointers. which have troubles handling structures containing memory pointers.

32
lib/compress/zstd_compress.c
View File

@ -3953,12 +3953,12 @@ size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuf
} }
size_t ZSTD_compress_generic (ZSTD_CCtx* cctx, size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
ZSTD_outBuffer* output, ZSTD_outBuffer* output,
ZSTD_inBuffer* input, ZSTD_inBuffer* input,
ZSTD_EndDirective endOp) ZSTD_EndDirective endOp)
{ {
DEBUGLOG(5, "ZSTD_compress_generic, endOp=%u ", (U32)endOp); DEBUGLOG(5, "ZSTD_compressStream2, endOp=%u ", (U32)endOp);
/* check conditions */ /* check conditions */
if (output->pos > output->size) return ERROR(GENERIC); if (output->pos > output->size) return ERROR(GENERIC);
if (input->pos > input->size) return ERROR(GENERIC); if (input->pos > input->size) return ERROR(GENERIC);
@ -3970,7 +3970,7 @@ size_t ZSTD_compress_generic (ZSTD_CCtx* cctx,
ZSTD_prefixDict const prefixDict = cctx->prefixDict; ZSTD_prefixDict const prefixDict = cctx->prefixDict;
memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict)); /* single usage */ memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict)); /* single usage */
assert(prefixDict.dict==NULL || cctx->cdict==NULL); /* only one can be set */ assert(prefixDict.dict==NULL || cctx->cdict==NULL); /* only one can be set */
DEBUGLOG(4, "ZSTD_compress_generic : transparent init stage"); DEBUGLOG(4, "ZSTD_compressStream2 : transparent init stage");
if (endOp == ZSTD_e_end) cctx->pledgedSrcSizePlusOne = input->size + 1; /* auto-fix pledgedSrcSize */ if (endOp == ZSTD_e_end) cctx->pledgedSrcSizePlusOne = input->size + 1; /* auto-fix pledgedSrcSize */
params.cParams = ZSTD_getCParamsFromCCtxParams( params.cParams = ZSTD_getCParamsFromCCtxParams(
&cctx->requestedParams, cctx->pledgedSrcSizePlusOne-1, 0 /*dictSize*/); &cctx->requestedParams, cctx->pledgedSrcSizePlusOne-1, 0 /*dictSize*/);
@ -3983,7 +3983,7 @@ size_t ZSTD_compress_generic (ZSTD_CCtx* cctx,
if (params.nbWorkers > 0) { if (params.nbWorkers > 0) {
/* mt context creation */ /* mt context creation */
if (cctx->mtctx == NULL) { if (cctx->mtctx == NULL) {
DEBUGLOG(4, "ZSTD_compress_generic: creating new mtctx for nbWorkers=%u", DEBUGLOG(4, "ZSTD_compressStream2: creating new mtctx for nbWorkers=%u",
params.nbWorkers); params.nbWorkers);
cctx->mtctx = ZSTDMT_createCCtx_advanced(params.nbWorkers, cctx->customMem); cctx->mtctx = ZSTDMT_createCCtx_advanced(params.nbWorkers, cctx->customMem);
if (cctx->mtctx == NULL) return ERROR(memory_allocation); if (cctx->mtctx == NULL) return ERROR(memory_allocation);
@ -4018,16 +4018,16 @@ size_t ZSTD_compress_generic (ZSTD_CCtx* cctx,
|| (endOp == ZSTD_e_end && flushMin == 0) ) { /* compression completed */ || (endOp == ZSTD_e_end && flushMin == 0) ) { /* compression completed */
ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only); ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
} }
DEBUGLOG(5, "completed ZSTD_compress_generic delegating to ZSTDMT_compressStream_generic"); DEBUGLOG(5, "completed ZSTD_compressStream2 delegating to ZSTDMT_compressStream_generic");
return flushMin; return flushMin;
} } } }
#endif #endif
CHECK_F( ZSTD_compressStream_generic(cctx, output, input, endOp) ); CHECK_F( ZSTD_compressStream_generic(cctx, output, input, endOp) );
DEBUGLOG(5, "completed ZSTD_compress_generic"); DEBUGLOG(5, "completed ZSTD_compressStream2");
return cctx->outBuffContentSize - cctx->outBuffFlushedSize; /* remaining to flush */ return cctx->outBuffContentSize - cctx->outBuffFlushedSize; /* remaining to flush */
} }
size_t ZSTD_compress_generic_simpleArgs ( size_t ZSTD_compressStream2_simpleArgs (
ZSTD_CCtx* cctx, ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, size_t* dstPos, void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos, const void* src, size_t srcSize, size_t* srcPos,
@ -4035,13 +4035,27 @@ size_t ZSTD_compress_generic_simpleArgs (
{ {
ZSTD_outBuffer output = { dst, dstCapacity, *dstPos }; ZSTD_outBuffer output = { dst, dstCapacity, *dstPos };
ZSTD_inBuffer input = { src, srcSize, *srcPos }; ZSTD_inBuffer input = { src, srcSize, *srcPos };
/* ZSTD_compress_generic() will check validity of dstPos and srcPos */ /* ZSTD_compressStream2() will check validity of dstPos and srcPos */
size_t const cErr = ZSTD_compress_generic(cctx, &output, &input, endOp); size_t const cErr = ZSTD_compressStream2(cctx, &output, &input, endOp);
*dstPos = output.pos; *dstPos = output.pos;
*srcPos = input.pos; *srcPos = input.pos;
return cErr; return cErr;
} }
size_t ZSTD_compress2(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize)
{
size_t oPos = 0;
size_t iPos = 0;
size_t const result = ZSTD_compressStream2_simpleArgs(cctx,
dst, dstCapacity, &oPos,
src, srcSize, &iPos,
ZSTD_e_end);
assert(iPos == srcSize);
if (ZSTD_isError(result)) return result;
return oPos;
}
/*====== Finalize ======*/ /*====== Finalize ======*/

42
lib/zstd.h
View File

@ -466,7 +466,7 @@ ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
* In this API, parameters are pushed one by one into an existing context, * In this API, parameters are pushed one by one into an existing context,
* using ZSTD_CCtx_set*() functions. * using ZSTD_CCtx_set*() functions.
* Pushed parameters are sticky : they are applied to next job, and any subsequent job. * Pushed parameters are sticky : they are applied to next job, and any subsequent job.
* Note that "sticky" parameters are only applicable with `ZSTD_compress_generic()` ! * Note that "sticky" parameters are only applicable with `ZSTD_compress2()` and `ZSTD_compressStream2()` !
* They do not apply should the context be used with a "simple" variant such as ZSTD_compressCCtx() * They do not apply should the context be used with a "simple" variant such as ZSTD_compressCCtx()
* *
* It's possible to reset all parameters to "default" using ZSTD_CCtx_reset(). * It's possible to reset all parameters to "default" using ZSTD_CCtx_reset().
@ -581,10 +581,11 @@ typedef enum {
/* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD). /* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).
* They return an error otherwise. */ * They return an error otherwise. */
ZSTD_p_nbWorkers=400, /* Select how many threads will be spawned to compress in parallel. ZSTD_p_nbWorkers=400, /* Select how many threads will be spawned to compress in parallel.
* When nbWorkers >= 1, triggers asynchronous mode : * When nbWorkers >= 1, triggers asynchronous mode when used with ZSTD_compressStream2() :
* ZSTD_compress_generic() consumes input and flush output if possible, but immediately gives back control to caller, * ZSTD_compressStream2() consumes input and flush output if possible, but immediately gives back control to caller,
* while compression work is performed in parallel, within worker threads. * while compression work is performed in parallel, within worker threads.
* (note : a strong exception to this rule is when first invocation sets ZSTD_e_end : it becomes a blocking call). * (note : a strong exception to this rule is when first invocation sets ZSTD_e_end :
* in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call).
* More workers improve speed, but also increase memory usage. * More workers improve speed, but also increase memory usage.
* Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */ * Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */
ZSTD_p_jobSize=401, /* Size of a compression job. This value is enforced only when nbWorkers >= 1. ZSTD_p_jobSize=401, /* Size of a compression job. This value is enforced only when nbWorkers >= 1.
@ -731,6 +732,18 @@ ZSTDLIB_API size_t ZSTD_CCtx_reset(ZSTD_CCtx* cctx, ZSTD_ResetDirective reset);
/*! ZSTD_compress2() :
* Behave the same as ZSTD_compressCCtx(), but compression parameters are set using the advanced API.
* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
* - The function is always blocking.
* Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`.
* @return : compressed size written into `dst` (<= `dstCapacity),
* or an error code if it fails (which can be tested using ZSTD_isError()).
*/
ZSTDLIB_API size_t ZSTD_compress2( ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize);
typedef enum { typedef enum {
ZSTD_e_continue=0, /* collect more data, encoder decides when to output compressed result, for optimal compression ratio */ ZSTD_e_continue=0, /* collect more data, encoder decides when to output compressed result, for optimal compression ratio */
ZSTD_e_flush=1, /* flush any data provided so far, ZSTD_e_flush=1, /* flush any data provided so far,
@ -741,8 +754,8 @@ typedef enum {
* each frame is independent (does not reference any content from previous frame). */ * each frame is independent (does not reference any content from previous frame). */
} ZSTD_EndDirective; } ZSTD_EndDirective;
/*! ZSTD_compress_generic() : /*! ZSTD_compressStream2() :
* Behave about the same as ZSTD_compressStream. To note : * Behave about the same as ZSTD_compressStream, with additional control on end directive.
* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*() * - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
* - Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode) * - Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode)
* - outpot->pos must be <= dstCapacity, input->pos must be <= srcSize * - outpot->pos must be <= dstCapacity, input->pos must be <= srcSize
@ -751,7 +764,7 @@ typedef enum {
* - In multi-thread mode, function is non-blocking : it just acquires a copy of input, and distribute job to internal worker threads, * - In multi-thread mode, function is non-blocking : it just acquires a copy of input, and distribute job to internal worker threads,
* and then immediately returns, just indicating that there is some data remaining to be flushed. * and then immediately returns, just indicating that there is some data remaining to be flushed.
* The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte. * The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte.
* - Exception : in multi-threading mode, if the first call requests a ZSTD_e_end directive, it is blocking : it will complete compression before giving back control to caller. * - Exception : if the first call requests a ZSTD_e_end directive, the function delegates to ZSTD_compress2() which is always blocking.
* - @return provides a minimum amount of data remaining to be flushed from internal buffers * - @return provides a minimum amount of data remaining to be flushed from internal buffers
* or an error code, which can be tested using ZSTD_isError(). * or an error code, which can be tested using ZSTD_isError().
* if @return != 0, flush is not fully completed, there is still some data left within internal buffers. * if @return != 0, flush is not fully completed, there is still some data left within internal buffers.
@ -762,12 +775,13 @@ typedef enum {
* Before starting a new compression job, or changing compression parameters, * Before starting a new compression job, or changing compression parameters,
* it is required to fully flush internal buffers. * it is required to fully flush internal buffers.
*/ */
ZSTDLIB_API size_t ZSTD_compress_generic (ZSTD_CCtx* cctx, ZSTDLIB_API size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
ZSTD_outBuffer* output, ZSTD_outBuffer* output,
ZSTD_inBuffer* input, ZSTD_inBuffer* input,
ZSTD_EndDirective endOp); ZSTD_EndDirective endOp);
/* ============================== */ /* ============================== */
/* Advanced decompression API */ /* Advanced decompression API */
/* ============================== */ /* ============================== */
@ -780,14 +794,14 @@ ZSTDLIB_API size_t ZSTD_compress_generic (ZSTD_CCtx* cctx,
* They are not valid if a "simple" function is used on the context (like `ZSTD_decompressDCtx()`). * They are not valid if a "simple" function is used on the context (like `ZSTD_decompressDCtx()`).
*/ */
/*! ZSTD_DCtx_setMaxWindowSize() : /*! ZSTD_DCtx_setMaxWindowSize() :
* Refuses allocating internal buffers for frames requiring a window size larger than provided limit. * Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
* This protects a decoder context from reserving too much memory for itself (potential attack scenario). * This protects a decoder context from reserving too much memory for itself (potential attack scenario).
* This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode. * This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
* By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT) * By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT)
* @return : 0, or an error code (which can be tested using ZSTD_isError()). * @return : 0, or an error code (which can be tested using ZSTD_isError()).
*/ */
ZSTDLIB_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize); ZSTDLIB_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
/*! ZSTD_DCtx_loadDictionary() : /*! ZSTD_DCtx_loadDictionary() :
* Create an internal DDict from dict buffer, * Create an internal DDict from dict buffer,
@ -1265,7 +1279,7 @@ ZSTDLIB_API size_t ZSTD_CCtx_getParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param
* an existing CCtx. * an existing CCtx.
* These parameters will be applied to * These parameters will be applied to
* all subsequent compression jobs. * all subsequent compression jobs.
* - ZSTD_compress_generic() : Do compression using the CCtx. * - ZSTD_compressStream2() : Do compression using the CCtx.
* - ZSTD_freeCCtxParams() : Free the memory. * - ZSTD_freeCCtxParams() : Free the memory.
* *
* This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams() * This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams()
@ -1316,13 +1330,13 @@ ZSTDLIB_API size_t ZSTD_CCtxParam_getParameter(ZSTD_CCtx_params* params, ZSTD_cP
ZSTDLIB_API size_t ZSTD_CCtx_setParametersUsingCCtxParams( ZSTDLIB_API size_t ZSTD_CCtx_setParametersUsingCCtxParams(
ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params); ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params);
/*! ZSTD_compress_generic_simpleArgs() : /*! ZSTD_compressStream2_simpleArgs() :
* Same as ZSTD_compress_generic(), * Same as ZSTD_compressStream2(),
* but using only integral types as arguments. * but using only integral types as arguments.
* This variant might be helpful for binders from dynamic languages * This variant might be helpful for binders from dynamic languages
* which have troubles handling structures containing memory pointers. * which have troubles handling structures containing memory pointers.
*/ */
ZSTDLIB_API size_t ZSTD_compress_generic_simpleArgs ( ZSTDLIB_API size_t ZSTD_compressStream2_simpleArgs (
ZSTD_CCtx* cctx, ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, size_t* dstPos, void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos, const void* src, size_t srcSize, size_t* srcPos,

16
programs/benchzstd.c
View File

@ -223,22 +223,8 @@ static size_t local_defaultCompress(
void* dstBuffer, size_t dstSize, void* dstBuffer, size_t dstSize,
void* addArgs) void* addArgs)
{ {
size_t moreToFlush = 1;
ZSTD_CCtx* const cctx = (ZSTD_CCtx*)addArgs; ZSTD_CCtx* const cctx = (ZSTD_CCtx*)addArgs;
ZSTD_inBuffer in; return ZSTD_compress2(cctx, dstBuffer, dstSize, srcBuffer, srcSize);
ZSTD_outBuffer out;
in.src = srcBuffer; in.size = srcSize; in.pos = 0;
out.dst = dstBuffer; out.size = dstSize; out.pos = 0;
while (moreToFlush) {
if(out.pos == out.size) {
return (size_t)-ZSTD_error_dstSize_tooSmall;
}
moreToFlush = ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end);
if (ZSTD_isError(moreToFlush)) {
return moreToFlush;
}
}
return out.pos;
} }
/* `addArgs` is the context */ /* `addArgs` is the context */

2
programs/fileio.c
View File

@ -884,7 +884,7 @@ FIO_compressZstdFrame(const cRess_t* ressPtr,
size_t const oldIPos = inBuff.pos; size_t const oldIPos = inBuff.pos;
ZSTD_outBuffer outBuff = { ress.dstBuffer, ress.dstBufferSize, 0 }; ZSTD_outBuffer outBuff = { ress.dstBuffer, ress.dstBufferSize, 0 };
size_t const toFlushNow = ZSTD_toFlushNow(ress.cctx); size_t const toFlushNow = ZSTD_toFlushNow(ress.cctx);
CHECK_V(stillToFlush, ZSTD_compress_generic(ress.cctx, &outBuff, &inBuff, directive)); CHECK_V(stillToFlush, ZSTD_compressStream2(ress.cctx, &outBuff, &inBuff, directive));
/* count stats */ /* count stats */
inputPresented++; inputPresented++;

30
tests/fullbench.c
View File

@ -171,17 +171,8 @@ local_ZSTD_compress_generic_end(const void* src, size_t srcSize,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
void* buff2) void* buff2)
{ {
ZSTD_outBuffer buffOut;
ZSTD_inBuffer buffIn;
(void)buff2; (void)buff2;
buffOut.dst = dst; return ZSTD_compress2(g_cstream, dst, dstCapacity, src, srcSize);
buffOut.size = dstCapacity;
buffOut.pos = 0;
buffIn.src = src;
buffIn.size = srcSize;
buffIn.pos = 0;
ZSTD_compress_generic(g_cstream, &buffOut, &buffIn, ZSTD_e_end);
return buffOut.pos;
} }
static size_t static size_t
@ -198,8 +189,8 @@ local_ZSTD_compress_generic_continue(const void* src, size_t srcSize,
buffIn.src = src; buffIn.src = src;
buffIn.size = srcSize; buffIn.size = srcSize;
buffIn.pos = 0; buffIn.pos = 0;
ZSTD_compress_generic(g_cstream, &buffOut, &buffIn, ZSTD_e_continue); ZSTD_compressStream2(g_cstream, &buffOut, &buffIn, ZSTD_e_continue);
ZSTD_compress_generic(g_cstream, &buffOut, &buffIn, ZSTD_e_end); ZSTD_compressStream2(g_cstream, &buffOut, &buffIn, ZSTD_e_end);
return buffOut.pos; return buffOut.pos;
} }
@ -208,18 +199,9 @@ local_ZSTD_compress_generic_T2_end(const void* src, size_t srcSize,
void* dst, size_t dstCapacity, void* dst, size_t dstCapacity,
void* buff2) void* buff2)
{ {
ZSTD_outBuffer buffOut;
ZSTD_inBuffer buffIn;
(void)buff2; (void)buff2;
ZSTD_CCtx_setParameter(g_cstream, ZSTD_p_nbWorkers, 2); ZSTD_CCtx_setParameter(g_cstream, ZSTD_p_nbWorkers, 2);
buffOut.dst = dst; return ZSTD_compress2(g_cstream, dst, dstCapacity, src, srcSize);
buffOut.size = dstCapacity;
buffOut.pos = 0;
buffIn.src = src;
buffIn.size = srcSize;
buffIn.pos = 0;
while (ZSTD_compress_generic(g_cstream, &buffOut, &buffIn, ZSTD_e_end)) {}
return buffOut.pos;
} }
static size_t static size_t
@ -237,8 +219,8 @@ local_ZSTD_compress_generic_T2_continue(const void* src, size_t srcSize,
buffIn.src = src; buffIn.src = src;
buffIn.size = srcSize; buffIn.size = srcSize;
buffIn.pos = 0; buffIn.pos = 0;
ZSTD_compress_generic(g_cstream, &buffOut, &buffIn, ZSTD_e_continue); ZSTD_compressStream2(g_cstream, &buffOut, &buffIn, ZSTD_e_continue);
while(ZSTD_compress_generic(g_cstream, &buffOut, &buffIn, ZSTD_e_end)) {} while(ZSTD_compressStream2(g_cstream, &buffOut, &buffIn, ZSTD_e_end)) {}
return buffOut.pos; return buffOut.pos;
} }

2
tests/fuzz/simple_round_trip.c
View File

@ -42,7 +42,7 @@ static size_t roundTripTest(void *result, size_t resultCapacity,
ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only); ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
FUZZ_setRandomParameters(cctx, srcSize, &seed); FUZZ_setRandomParameters(cctx, srcSize, &seed);
err = ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end); err = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
FUZZ_ZASSERT(err); FUZZ_ZASSERT(err);
FUZZ_ASSERT(err == 0); FUZZ_ASSERT(err == 0);
cSize = out.pos; cSize = out.pos;

8
tests/fuzz/stream_round_trip.c
View File

@ -72,7 +72,7 @@ static size_t compress(uint8_t *dst, size_t capacity,
case 1: /* fall-though */ case 1: /* fall-though */
case 2: { case 2: {
size_t const ret = size_t const ret =
ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_flush); ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_flush);
FUZZ_ZASSERT(ret); FUZZ_ZASSERT(ret);
if (ret == 0) if (ret == 0)
mode = -1; mode = -1;
@ -80,7 +80,7 @@ static size_t compress(uint8_t *dst, size_t capacity,
} }
case 3: { case 3: {
size_t ret = size_t ret =
ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end); ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
FUZZ_ZASSERT(ret); FUZZ_ZASSERT(ret);
/* Reset the compressor when the frame is finished */ /* Reset the compressor when the frame is finished */
if (ret == 0) { if (ret == 0) {
@ -95,7 +95,7 @@ static size_t compress(uint8_t *dst, size_t capacity,
} }
default: { default: {
size_t const ret = size_t const ret =
ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_continue); ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_continue);
FUZZ_ZASSERT(ret); FUZZ_ZASSERT(ret);
mode = -1; mode = -1;
} }
@ -108,7 +108,7 @@ static size_t compress(uint8_t *dst, size_t capacity,
for (;;) { for (;;) {
ZSTD_inBuffer in = {NULL, 0, 0}; ZSTD_inBuffer in = {NULL, 0, 0};
ZSTD_outBuffer out = makeOutBuffer(dst, capacity); ZSTD_outBuffer out = makeOutBuffer(dst, capacity);
size_t const ret = ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end); size_t const ret = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
FUZZ_ZASSERT(ret); FUZZ_ZASSERT(ret);
dst += out.pos; dst += out.pos;

16
tests/fuzzer.c
View File

@ -237,7 +237,7 @@ static int FUZ_mallocTests_internal(unsigned seed, double compressibility, unsig
ZSTD_inBuffer in = { inBuffer, inSize, 0 }; ZSTD_inBuffer in = { inBuffer, inSize, 0 };
CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_p_compressionLevel, compressionLevel) ); CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_p_compressionLevel, compressionLevel) );
CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_p_nbWorkers, nbThreads) ); CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_p_nbWorkers, nbThreads) );
while ( ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end) ) {} while ( ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end) ) {}
ZSTD_freeCCtx(cctx); ZSTD_freeCCtx(cctx);
DISPLAYLEVEL(3, "compress_generic,-T%u,end level %i : ", DISPLAYLEVEL(3, "compress_generic,-T%u,end level %i : ",
nbThreads, compressionLevel); nbThreads, compressionLevel);
@ -257,8 +257,8 @@ static int FUZ_mallocTests_internal(unsigned seed, double compressibility, unsig
ZSTD_inBuffer in = { inBuffer, inSize, 0 }; ZSTD_inBuffer in = { inBuffer, inSize, 0 };
CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_p_compressionLevel, compressionLevel) ); CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_p_compressionLevel, compressionLevel) );
CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_p_nbWorkers, nbThreads) ); CHECK_Z( ZSTD_CCtx_setParameter(cctx, ZSTD_p_nbWorkers, nbThreads) );
CHECK_Z( ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_continue) ); CHECK_Z( ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_continue) );
while ( ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end) ) {} while ( ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end) ) {}
ZSTD_freeCCtx(cctx); ZSTD_freeCCtx(cctx);
DISPLAYLEVEL(3, "compress_generic,-T%u,continue level %i : ", DISPLAYLEVEL(3, "compress_generic,-T%u,continue level %i : ",
nbThreads, compressionLevel); nbThreads, compressionLevel);
@ -521,7 +521,7 @@ static int basicUnitTests(U32 seed, double compressibility)
CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_p_hashLog, &value)); CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_p_hashLog, &value));
CHECK_EQ(value, ZSTD_HASHLOG_MIN); CHECK_EQ(value, ZSTD_HASHLOG_MIN);
/* Start a compression job */ /* Start a compression job */
ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_continue); ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_continue);
CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_p_compressionLevel, &value)); CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_p_compressionLevel, &value));
CHECK_EQ(value, 7); CHECK_EQ(value, 7);
CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_p_hashLog, &value)); CHECK_Z(ZSTD_CCtx_getParameter(cctx, ZSTD_p_hashLog, &value));
@ -1255,7 +1255,7 @@ static int basicUnitTests(U32 seed, double compressibility)
CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_p_compressionLevel, compressionLevel) ); CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_p_compressionLevel, compressionLevel) );
{ ZSTD_inBuffer in = { CNBuffer, srcSize, 0 }; { ZSTD_inBuffer in = { CNBuffer, srcSize, 0 };
ZSTD_outBuffer out = { compressedBuffer, compressedBufferSize, 0 }; ZSTD_outBuffer out = { compressedBuffer, compressedBufferSize, 0 };
size_t const compressionResult = ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end); size_t const compressionResult = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
DISPLAYLEVEL(5, "simple=%zu vs %zu=advanced : ", cSize_1pass, out.pos); DISPLAYLEVEL(5, "simple=%zu vs %zu=advanced : ", cSize_1pass, out.pos);
if (ZSTD_isError(compressionResult)) goto _output_error; if (ZSTD_isError(compressionResult)) goto _output_error;
if (out.pos != cSize_1pass) goto _output_error; if (out.pos != cSize_1pass) goto _output_error;
@ -1276,7 +1276,7 @@ static int basicUnitTests(U32 seed, double compressibility)
CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_p_windowLog, 18) ); CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_p_windowLog, 18) );
{ ZSTD_inBuffer in = { CNBuffer, inputSize, 0 }; { ZSTD_inBuffer in = { CNBuffer, inputSize, 0 };
ZSTD_outBuffer out = { compressedBuffer, ZSTD_compressBound(inputSize), 0 }; ZSTD_outBuffer out = { compressedBuffer, ZSTD_compressBound(inputSize), 0 };
size_t const result = ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end); size_t const result = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
if (result != 0) goto _output_error; if (result != 0) goto _output_error;
if (in.pos != in.size) goto _output_error; if (in.pos != in.size) goto _output_error;
cSize = out.pos; cSize = out.pos;
@ -1293,7 +1293,7 @@ static int basicUnitTests(U32 seed, double compressibility)
CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_p_compressionLevel, 2) ); CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_p_compressionLevel, 2) );
{ ZSTD_inBuffer in = { CNBuffer, inputSize, 0 }; { ZSTD_inBuffer in = { CNBuffer, inputSize, 0 };
ZSTD_outBuffer out = { compressedBuffer, ZSTD_compressBound(inputSize), 0 }; ZSTD_outBuffer out = { compressedBuffer, ZSTD_compressBound(inputSize), 0 };
size_t const result = ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end); size_t const result = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
if (result != 0) goto _output_error; if (result != 0) goto _output_error;
if (in.pos != in.size) goto _output_error; if (in.pos != in.size) goto _output_error;
if (out.pos != cSize) goto _output_error; /* must result in same compressed result, hence same size */ if (out.pos != cSize) goto _output_error; /* must result in same compressed result, hence same size */
@ -1313,7 +1313,7 @@ static int basicUnitTests(U32 seed, double compressibility)
CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_p_format, ZSTD_f_zstd1_magicless) ); CHECK( ZSTD_CCtx_setParameter(cctx, ZSTD_p_format, ZSTD_f_zstd1_magicless) );
{ ZSTD_inBuffer in = { CNBuffer, inputSize, 0 }; { ZSTD_inBuffer in = { CNBuffer, inputSize, 0 };
ZSTD_outBuffer out = { compressedBuffer, ZSTD_compressBound(inputSize), 0 }; ZSTD_outBuffer out = { compressedBuffer, ZSTD_compressBound(inputSize), 0 };
size_t const result = ZSTD_compress_generic(cctx, &out, &in, ZSTD_e_end); size_t const result = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_end);
if (result != 0) goto _output_error; if (result != 0) goto _output_error;
if (in.pos != in.size) goto _output_error; if (in.pos != in.size) goto _output_error;
cSize = out.pos; cSize = out.pos;

25
tests/paramgrill.c
View File

@ -922,28 +922,11 @@ static size_t local_initDCtx(void* payload) {
static size_t local_defaultCompress( static size_t local_defaultCompress(
const void* srcBuffer, size_t srcSize, const void* srcBuffer, size_t srcSize,
void* dstBuffer, size_t dstSize, void* dstBuffer, size_t dstSize,
void* addArgs) { void* addArgs)
size_t moreToFlush = 1; {
ZSTD_CCtx* ctx = (ZSTD_CCtx*)addArgs; ZSTD_CCtx* cctx = (ZSTD_CCtx*)addArgs;
ZSTD_inBuffer in;
ZSTD_outBuffer out;
in.src = srcBuffer;
in.size = srcSize;
in.pos = 0;
out.dst = dstBuffer;
out.size = dstSize;
out.pos = 0;
assert(dstSize == ZSTD_compressBound(srcSize)); /* specific to this version, which is only used in paramgrill */ assert(dstSize == ZSTD_compressBound(srcSize)); /* specific to this version, which is only used in paramgrill */
while (moreToFlush) { return ZSTD_compress2(cctx, dstBuffer, dstSize, srcBuffer, srcSize);
if(out.pos == out.size) {
return (size_t)-ZSTD_error_dstSize_tooSmall;
}
moreToFlush = ZSTD_compress_generic(ctx, &out, &in, ZSTD_e_end);
if (ZSTD_isError(moreToFlush)) {
return moreToFlush;
}
}
return out.pos;
} }
/* additional argument is just the context */ /* additional argument is just the context */

4
tests/roundTripCrash.c
View File

@ -85,7 +85,7 @@ static size_t cctxParamRoundTripTest(void* resultBuff, size_t resultBuffCapacity
ZSTD_CCtx* const cctx = ZSTD_createCCtx(); ZSTD_CCtx* const cctx = ZSTD_createCCtx();
ZSTD_CCtx_params* const cctxParams = ZSTD_createCCtxParams(); ZSTD_CCtx_params* const cctxParams = ZSTD_createCCtxParams();
ZSTD_inBuffer inBuffer = { srcBuff, srcBuffSize, 0 }; ZSTD_inBuffer inBuffer = { srcBuff, srcBuffSize, 0 };
ZSTD_outBuffer outBuffer = {compressedBuff, compressedBuffCapacity, 0 }; ZSTD_outBuffer outBuffer = { compressedBuff, compressedBuffCapacity, 0 };
static const int maxClevel = 19; static const int maxClevel = 19;
size_t const hashLength = MIN(128, srcBuffSize); size_t const hashLength = MIN(128, srcBuffSize);
@ -101,7 +101,7 @@ static size_t cctxParamRoundTripTest(void* resultBuff, size_t resultBuffCapacity
/* Apply parameters */ /* Apply parameters */
CHECK_Z( ZSTD_CCtx_setParametersUsingCCtxParams(cctx, cctxParams) ); CHECK_Z( ZSTD_CCtx_setParametersUsingCCtxParams(cctx, cctxParams) );
CHECK_Z (ZSTD_compress_generic(cctx, &outBuffer, &inBuffer, ZSTD_e_end) ); CHECK_Z (ZSTD_compressStream2(cctx, &outBuffer, &inBuffer, ZSTD_e_end) );
ZSTD_freeCCtxParams(cctxParams); ZSTD_freeCCtxParams(cctxParams);
ZSTD_freeCCtx(cctx); ZSTD_freeCCtx(cctx);

28
tests/zstreamtest.c
View File

@ -175,11 +175,11 @@ static size_t SEQ_roundTrip(ZSTD_CCtx* cctx, ZSTD_DCtx* dctx,
size_t cret; size_t cret;
do { do {
ZSTD_outBuffer cout = {compressed, sizeof(compressed), 0}; ZSTD_outBuffer cout = { compressed, sizeof(compressed), 0 };
ZSTD_inBuffer din = {compressed, 0, 0}; ZSTD_inBuffer din = { compressed, 0, 0 };
ZSTD_outBuffer dout = {uncompressed, 0, 0}; ZSTD_outBuffer dout = { uncompressed, 0, 0 };
cret = ZSTD_compress_generic(cctx, &cout, &cin, endOp); cret = ZSTD_compressStream2(cctx, &cout, &cin, endOp);
if (ZSTD_isError(cret)) if (ZSTD_isError(cret))
return cret; return cret;
@ -686,7 +686,7 @@ static int basicUnitTests(U32 seed, double compressibility)
inBuff.size = size; inBuff.size = size;
inBuff.pos = 0; inBuff.pos = 0;
CHECK_Z(ZSTD_CCtx_refCDict(cctx, cdict)); CHECK_Z(ZSTD_CCtx_refCDict(cctx, cdict));
CHECK_Z(ZSTD_compress_generic(cctx, &outBuff, &inBuff, ZSTD_e_end)); CHECK_Z(ZSTD_compressStream2(cctx, &outBuff, &inBuff, ZSTD_e_end));
CHECK(badParameters(cctx, savedParams), "Bad CCtx params"); CHECK(badParameters(cctx, savedParams), "Bad CCtx params");
if (inBuff.pos != inBuff.size) goto _output_error; if (inBuff.pos != inBuff.size) goto _output_error;
{ ZSTD_outBuffer decOut = {decodedBuffer, size, 0}; { ZSTD_outBuffer decOut = {decodedBuffer, size, 0};
@ -746,7 +746,7 @@ static int basicUnitTests(U32 seed, double compressibility)
inBuff.src = CNBuffer; inBuff.src = CNBuffer;
inBuff.size = CNBufferSize; inBuff.size = CNBufferSize;
inBuff.pos = 0; inBuff.pos = 0;
CHECK_Z( ZSTD_compress_generic(zc, &outBuff, &inBuff, ZSTD_e_end) ); CHECK_Z( ZSTD_compressStream2(zc, &outBuff, &inBuff, ZSTD_e_end) );
if (inBuff.pos != inBuff.size) goto _output_error; /* entire input should be consumed */ if (inBuff.pos != inBuff.size) goto _output_error; /* entire input should be consumed */
cSize = outBuff.pos; cSize = outBuff.pos;
DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (U32)cSize, (double)cSize/CNBufferSize*100); DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (U32)cSize, (double)cSize/CNBufferSize*100);
@ -770,14 +770,14 @@ static int basicUnitTests(U32 seed, double compressibility)
DISPLAYLEVEL(3, "OK (%s)\n", ZSTD_getErrorName(r)); DISPLAYLEVEL(3, "OK (%s)\n", ZSTD_getErrorName(r));
} }
DISPLAYLEVEL(3, "test%3i : compress again with ZSTD_compress_generic : ", testNb++); DISPLAYLEVEL(3, "test%3i : compress again with ZSTD_compressStream2 : ", testNb++);
outBuff.dst = compressedBuffer; outBuff.dst = compressedBuffer;
outBuff.size = compressedBufferSize; outBuff.size = compressedBufferSize;
outBuff.pos = 0; outBuff.pos = 0;
inBuff.src = CNBuffer; inBuff.src = CNBuffer;
inBuff.size = CNBufferSize; inBuff.size = CNBufferSize;
inBuff.pos = 0; inBuff.pos = 0;
CHECK_Z( ZSTD_compress_generic(zc, &outBuff, &inBuff, ZSTD_e_end) ); CHECK_Z( ZSTD_compressStream2(zc, &outBuff, &inBuff, ZSTD_e_end) );
if (inBuff.pos != inBuff.size) goto _output_error; /* entire input should be consumed */ if (inBuff.pos != inBuff.size) goto _output_error; /* entire input should be consumed */
cSize = outBuff.pos; cSize = outBuff.pos;
DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (U32)cSize, (double)cSize/CNBufferSize*100); DISPLAYLEVEL(3, "OK (%u bytes : %.2f%%)\n", (U32)cSize, (double)cSize/CNBufferSize*100);
@ -885,7 +885,7 @@ static int basicUnitTests(U32 seed, double compressibility)
ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dictionary.start, dictionary.filled, ZSTD_dlm_byRef, ZSTD_dct_fullDict, cParams, ZSTD_defaultCMem); ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dictionary.start, dictionary.filled, ZSTD_dlm_byRef, ZSTD_dct_fullDict, cParams, ZSTD_defaultCMem);
DISPLAYLEVEL(5, "cParams.windowLog = %u : ", cParams.windowLog); DISPLAYLEVEL(5, "cParams.windowLog = %u : ", cParams.windowLog);
CHECK_Z( ZSTD_CCtx_refCDict(zc, cdict) ); CHECK_Z( ZSTD_CCtx_refCDict(zc, cdict) );
CHECK_Z( ZSTD_compress_generic(zc, &outBuff, &inBuff, ZSTD_e_end) ); CHECK_Z( ZSTD_compressStream2(zc, &outBuff, &inBuff, ZSTD_e_end) );
CHECK_Z( ZSTD_CCtx_refCDict(zc, NULL) ); /* do not keep a reference to cdict, as its lifetime ends */ CHECK_Z( ZSTD_CCtx_refCDict(zc, NULL) ); /* do not keep a reference to cdict, as its lifetime ends */
ZSTD_freeCDict(cdict); ZSTD_freeCDict(cdict);
} }
@ -1086,7 +1086,7 @@ static int basicUnitTests(U32 seed, double compressibility)
const size_t kSmallBlockSize = sizeof(testBuffer); const size_t kSmallBlockSize = sizeof(testBuffer);
ZSTD_inBuffer in = {testBuffer, kSmallBlockSize, 0}; ZSTD_inBuffer in = {testBuffer, kSmallBlockSize, 0};
CHECK_Z(ZSTD_compress_generic(zc, &out, &in, ZSTD_e_flush)); CHECK_Z(ZSTD_compressStream2(zc, &out, &in, ZSTD_e_flush));
CHECK(in.pos != in.size, "input not fully consumed"); CHECK(in.pos != in.size, "input not fully consumed");
remainingInput -= kSmallBlockSize; remainingInput -= kSmallBlockSize;
} }
@ -1094,7 +1094,7 @@ static int basicUnitTests(U32 seed, double compressibility)
for (offset = 1024; offset >= 0; offset -= 128) { for (offset = 1024; offset >= 0; offset -= 128) {
ZSTD_inBuffer in = {dictionary.start + offset, 128, 0}; ZSTD_inBuffer in = {dictionary.start + offset, 128, 0};
ZSTD_EndDirective flush = offset > 0 ? ZSTD_e_continue : ZSTD_e_end; ZSTD_EndDirective flush = offset > 0 ? ZSTD_e_continue : ZSTD_e_end;
CHECK_Z(ZSTD_compress_generic(zc, &out, &in, flush)); CHECK_Z(ZSTD_compressStream2(zc, &out, &in, flush));
CHECK(in.pos != in.size, "input not fully consumed"); CHECK(in.pos != in.size, "input not fully consumed");
} }
/* Ensure decompression works */ /* Ensure decompression works */
@ -1956,7 +1956,7 @@ static int fuzzerTests_newAPI(U32 seed, U32 nbTests, unsigned startTest,
ZSTD_inBuffer inBuff = { srcBuffer+srcStart, srcSize, 0 }; ZSTD_inBuffer inBuff = { srcBuffer+srcStart, srcSize, 0 };
outBuff.size = outBuff.pos + dstBuffSize; outBuff.size = outBuff.pos + dstBuffSize;
CHECK_Z( ZSTD_compress_generic(zc, &outBuff, &inBuff, flush) ); CHECK_Z( ZSTD_compressStream2(zc, &outBuff, &inBuff, flush) );
DISPLAYLEVEL(6, "t%u: compress consumed %u bytes (total : %u) ; flush: %u (total : %u) \n", DISPLAYLEVEL(6, "t%u: compress consumed %u bytes (total : %u) ; flush: %u (total : %u) \n",
testNb, (U32)inBuff.pos, (U32)(totalTestSize + inBuff.pos), (U32)flush, (U32)outBuff.pos); testNb, (U32)inBuff.pos, (U32)(totalTestSize + inBuff.pos), (U32)flush, (U32)outBuff.pos);
@ -1973,10 +1973,10 @@ static int fuzzerTests_newAPI(U32 seed, U32 nbTests, unsigned startTest,
size_t const adjustedDstSize = MIN(cBufferSize - cSize, randomDstSize); size_t const adjustedDstSize = MIN(cBufferSize - cSize, randomDstSize);
outBuff.size = outBuff.pos + adjustedDstSize; outBuff.size = outBuff.pos + adjustedDstSize;
DISPLAYLEVEL(6, "t%u: End-flush into dst buffer of size %u \n", testNb, (U32)adjustedDstSize); DISPLAYLEVEL(6, "t%u: End-flush into dst buffer of size %u \n", testNb, (U32)adjustedDstSize);
remainingToFlush = ZSTD_compress_generic(zc, &outBuff, &inBuff, ZSTD_e_end); remainingToFlush = ZSTD_compressStream2(zc, &outBuff, &inBuff, ZSTD_e_end);
DISPLAYLEVEL(6, "t%u: Total flushed so far : %u bytes \n", testNb, (U32)outBuff.pos); DISPLAYLEVEL(6, "t%u: Total flushed so far : %u bytes \n", testNb, (U32)outBuff.pos);
CHECK( ZSTD_isError(remainingToFlush), CHECK( ZSTD_isError(remainingToFlush),
"ZSTD_compress_generic w/ ZSTD_e_end error : %s", "ZSTD_compressStream2 w/ ZSTD_e_end error : %s",
ZSTD_getErrorName(remainingToFlush) ); ZSTD_getErrorName(remainingToFlush) );
} } } }
crcOrig = XXH64_digest(&xxhState); crcOrig = XXH64_digest(&xxhState);