Added comments on I/O buffer sizes for streaming

It seems this is still a confusing topic,
as in https://github.com/klauspost/compress/issues/109 .
dev
Yann Collet 2019-06-04 10:26:16 -07:00
parent b13a9207f9
commit b5c98fbfd0
2 changed files with 25 additions and 8 deletions

View File

@ -18,7 +18,7 @@
<li><a href="#Chapter8">Advanced decompression API</a></li> <li><a href="#Chapter8">Advanced decompression API</a></li>
<li><a href="#Chapter9">Streaming</a></li> <li><a href="#Chapter9">Streaming</a></li>
<li><a href="#Chapter10">Streaming compression - HowTo</a></li> <li><a href="#Chapter10">Streaming compression - HowTo</a></li>
<li><a href="#Chapter11">This is a legacy streaming API, and can be replaced by ZSTD_CCtx_reset() and</a></li> <li><a href="#Chapter11">This following is a legacy streaming API.</a></li>
<li><a href="#Chapter12">Equivalent to:</a></li> <li><a href="#Chapter12">Equivalent to:</a></li>
<li><a href="#Chapter13">Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue).</a></li> <li><a href="#Chapter13">Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue).</a></li>
<li><a href="#Chapter14">Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_flush).</a></li> <li><a href="#Chapter14">Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_flush).</a></li>
@ -592,9 +592,10 @@ size_t ZSTD_freeCStream(ZSTD_CStream* zcs);
<pre><b>size_t ZSTD_CStreamInSize(void); </b>/**< recommended size for input buffer */<b> <pre><b>size_t ZSTD_CStreamInSize(void); </b>/**< recommended size for input buffer */<b>
</b></pre><BR> </b></pre><BR>
<pre><b>size_t ZSTD_CStreamOutSize(void); </b>/**< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block in all circumstances. */<b> <pre><b>size_t ZSTD_CStreamOutSize(void); </b>/**< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block. */<b>
</b></pre><BR> </b></pre><BR>
<a name="Chapter11"></a><h2>This is a legacy streaming API, and can be replaced by ZSTD_CCtx_reset() and</h2><pre> ZSTD_compressStream2(). It is redundant, but is still fully supported. <a name="Chapter11"></a><h2>This following is a legacy streaming API.</h2><pre> It can be replaced by ZSTD_CCtx_reset() and ZSTD_compressStream2().
It is redundant, but remains fully supported.
Advanced parameters and dictionary compression can only be used through the Advanced parameters and dictionary compression can only be used through the
new API. new API.
<BR></pre> <BR></pre>
@ -608,7 +609,7 @@ size_t ZSTD_freeCStream(ZSTD_CStream* zcs);
<a name="Chapter13"></a><h2>Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue).</h2><pre> NOTE: The return value is different. ZSTD_compressStream() returns a hint for <a name="Chapter13"></a><h2>Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue).</h2><pre> NOTE: The return value is different. ZSTD_compressStream() returns a hint for
the next read size (if non-zero and not an error). ZSTD_compressStream2() the next read size (if non-zero and not an error). ZSTD_compressStream2()
returns the number of bytes left to flush (if non-zero and not an error). returns the minimum nb of bytes left to flush (if non-zero and not an error).
<BR></pre> <BR></pre>

View File

@ -657,12 +657,28 @@ ZSTDLIB_API size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
ZSTD_inBuffer* input, ZSTD_inBuffer* input,
ZSTD_EndDirective endOp); ZSTD_EndDirective endOp);
/* These buffer sizes are softly recommended.
* They are not required : ZSTD_compressStream*() happily accepts any buffer size, for both input and output.
* Respecting the recommended size just makes it a bit easier for ZSTD_compressStream*(),
* reducing the amount of memory shuffling and buffering, resulting in minor performance savings.
*
* However, note that these recommendations are from the perspective of a C caller program.
* If the streaming interface is invoked from some other language,
* especially managed ones such as Java or Go, through a foreign function interface such as jni or cgo,
* a major performance rule is to reduce crossing such interface to an absolute minimum.
* It's not rare that performance ends being spent more into the interface, rather than compression itself.
* In which cases, prefer using large buffers, as large as practical,
* for both input and output, to reduce the nb of roundtrips.
*/
ZSTDLIB_API size_t ZSTD_CStreamInSize(void); /**< recommended size for input buffer */ ZSTDLIB_API size_t ZSTD_CStreamInSize(void); /**< recommended size for input buffer */
ZSTDLIB_API size_t ZSTD_CStreamOutSize(void); /**< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block in all circumstances. */ ZSTDLIB_API size_t ZSTD_CStreamOutSize(void); /**< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block. */
/******************************************************************************* /*******************************************************************************
* This is a legacy streaming API, and can be replaced by ZSTD_CCtx_reset() and * This following is a legacy streaming API.
* ZSTD_compressStream2(). It is redundant, but is still fully supported. * It can be replaced by ZSTD_CCtx_reset() and ZSTD_compressStream2().
* It is redundant, but remains fully supported.
* Advanced parameters and dictionary compression can only be used through the * Advanced parameters and dictionary compression can only be used through the
* new API. * new API.
******************************************************************************/ ******************************************************************************/
@ -679,7 +695,7 @@ ZSTDLIB_API size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel);
* Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue). * Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue).
* NOTE: The return value is different. ZSTD_compressStream() returns a hint for * NOTE: The return value is different. ZSTD_compressStream() returns a hint for
* the next read size (if non-zero and not an error). ZSTD_compressStream2() * the next read size (if non-zero and not an error). ZSTD_compressStream2()
* returns the number of bytes left to flush (if non-zero and not an error). * returns the minimum nb of bytes left to flush (if non-zero and not an error).
*/ */
ZSTDLIB_API size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input); ZSTDLIB_API size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
/** Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_flush). */ /** Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_flush). */