Commit Graph

85 Commits (6466fd340049d7785d2bcbe60b405a179920f40c)

Author SHA1 Message Date
Yann Collet 052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
Yann Collet 4616fad18b improved ZSTDMT_compress() memory usage
does not need the input buffer for streaming operations

also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
Yann Collet afb0aca739 zstreamtest : big tests are only enabled in 64-bits mode
to avoid requesting too much memory in 32-bits mode during MT tests
2017-06-29 18:19:09 -07:00
Yann Collet 62f7efc04a added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job 2017-06-28 16:25:13 -07:00
Yann Collet 33a6639039 fixed ZSTD_refPrefix with Multithread-enabled CCtx 2017-06-28 11:09:43 -07:00
Yann Collet 1fd2df9c2c fixed zstreamtest
modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary()
but it is still allowed after ZSTD_refPrefix()
2017-06-28 09:57:59 -07:00
Yann Collet 2e4274262d controlled dictMode 2017-06-27 17:09:12 -07:00
Yann Collet b7372933b8 implemented ZSTD_refPrefix() 2017-06-27 15:49:12 -07:00
Yann Collet fecc721fd9 added parameter ZSTD_p_refDictContent 2017-06-27 11:46:39 -07:00
Yann Collet dde10b23fe refactored ZSTD_estimateDStreamSize()
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
Yann Collet 09ae03a570 ZSTD_estimateCDictSize_advanced()
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
Yann Collet 0c9a915a28 ZSTD_estimateCStreamSize_advanced() 2017-06-26 16:02:25 -07:00
Yann Collet cca1ed14db Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-06-23 00:13:03 -07:00
Yann Collet 32f546c763 zstreamtest : disabled multi-threading tests with --newapi --no-big-tests
--no-big-tests is typically used in combination with qemu-user-static
qemu-user-static allocated 4 GB of RAM upfront.
On 2 GB VM, this can degenerate into a crash.
It's not a problem as long as memory is not used.
But with multi-threading enabled, memory fragmentation kicks in,
so the amoung of RAM effectively touched increases,
and can pass beyond the 2 GB limit of the VM.

In single-threaded mode, there is no such issue :
memory requirement is smaller, and remains well-located,
so very little fragmentation is expected.

This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests`
work fine on Travis CI.
2017-06-23 00:09:02 -07:00
Yann Collet f99c2c1a22 reduced --no-big-tests even more
to pass tests on qemu-aarch64
2017-06-21 23:35:58 -07:00
Yann Collet 49f8459d11 fixed minor cast warning 2017-06-21 18:43:39 -07:00
Yann Collet 4d3bdcf130 reduced CLevelMax for --no-big-tests 2017-06-21 18:20:02 -07:00
Yann Collet bfc2f00080 --no-big-tests for zstreamtest
Apply --no-big-tests for tsan tests
2017-06-21 17:57:14 -07:00
Yann Collet dce789281b fixed : decompression of skippable frames in streaming mode 2017-06-21 15:53:42 -07:00
Yann Collet 7bd1a2900e added ZSTD_dictMode_e to control dictionary loading mode 2017-06-21 11:50:33 -07:00
Yann Collet 688952062b minor declaration statement warning fix 2017-06-19 11:53:42 -07:00
Yann Collet d7a3bffba9 new api : setting compression parameters is refused if a dictionary is already loaded 2017-06-19 11:53:01 -07:00
Yann Collet ed1d039127 newapi fuzzer tests : random flush orders in main loop 2017-06-19 11:07:33 -07:00
Yann Collet f5deae8a67 new api fuzzer tests : frame parameters are randonly set 2017-06-18 23:41:38 -07:00
Yann Collet ea3630a889 new api fuzzer set : dictionary is randomly set 2017-06-18 23:31:55 -07:00
Yann Collet 8dee0ec99e new api fuzzer tests : compression parameters are randomly set 2017-06-18 23:25:15 -07:00
Yann Collet 01743a36e7 fuzzer tests for new API 2017-06-16 17:56:41 -07:00
Yann Collet 8c910d2097 updated ZSTDMT streaming API
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
Yann Collet 25989e361c updated ZSTD_estimate?DictSize() to pass parameter byReference
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
Yann Collet 009d604e00 ZSTD_compress_generic() supports multiple successive frames
also : clarified streaming API implementation
2017-05-19 10:17:59 -07:00
Yann Collet a5ffe3d370 pushed enum values for strategy by one (ZSTD_fast==1)
this makes it possible to use `0` to mean:
"do not change strategy"
2017-05-12 16:29:19 -07:00
Yann Collet 30ab64e21d added test for ZSTD_estimateCStreamSize() 2017-05-10 11:30:19 -07:00
Yann Collet f16f4497ca added ZSTD_estimateDStreamSize() 2017-05-09 16:18:17 -07:00
Yann Collet 0be6fd3429 merged CCtx and CStream as a single same object
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
2017-05-08 16:08:01 -07:00
Yann Collet a92cbb7004 Added a secondary test, checking dictID presence after setting noDictIdFLag=1 2017-04-27 15:08:56 -07:00
Yann Collet 7d283cdfa2 added test for ZSTD_initCStream_usingCDict_advanced() 2017-04-27 14:48:34 -07:00
Yann Collet 31533bacce Changed ZSTD_createCDict_advanced()
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
Yann Collet 2c5514c759 fixed ZSTDMT_initCStream_advanced()
Must use the new ZSTD_compressBegin_usingCDict_advanced()
to enforce correct frame parameters
2017-04-18 22:52:41 -07:00
Yann Collet 0bb381dad8 added test for ZSTD_initCStream_advanced()
when params.fParams.contentSizeFlags = 1
and pledgedSrcSize = 0
then srcSize should be considered 0 (empty), and not "unknown"
2017-04-18 15:08:52 -07:00
Yann Collet 4ee6b15dac force contentSizeFlag=0 when using ZSTD_initCStream_usingCDict()
because by definition srcSize is not known when using this prototype.
added relevant test

Note : this use was already working, because at a later stage
(both ZSTD_compressBegin_usingCDict() and ZSTD_copyCCtx())
pledgedSrcSize=0 is translated into "unknown", no matter the frame parameter.
This is not correct, but of little importance,
as the medium term plan is to no longer set fParams within CDict
2017-04-11 11:59:44 -07:00
Yann Collet 4b987ad8ce Introduce ZSTD_initCStream_internal()
This is now the regroup point for ZSTD_initCStream*() functions

ZSTD_initCStream_advanced() now properly checks for parameters validity.

Also : added <assert.h> usage inside zstd_compress.c
Needs ZSTD_DEBUG=1 macro to be triggered.
Will be triggered by default from `tests` directory
2017-04-10 17:50:44 -07:00
Yann Collet ce80098f14 improved zstreamtest --mt to trap bug #644 2017-04-05 16:34:09 -07:00
Nick Terrell 62ecad3819 Fix ZSTD_initCStream_usingCDict() to use dictionary 2017-04-03 21:05:59 -07:00
Yann Collet 30c7698970 optimize ZSTDMT_compress() memory usage
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).

Also : fix ZSTD_compressBound() for small srcSize
2017-03-31 18:27:03 -07:00
Sean Purcell 042ba122ae Change g_displayLevel to int and fix DISPLAYUPDATE flush 2017-03-23 11:21:59 -07:00
Sean Purcell 7ebf2de02d Add ability to strongly limit fuzzer test size with flag 2017-03-20 11:25:00 -07:00
Sean Purcell f5e50519e0 Prevent fuzz testers from combining large dicts with high clevel 2017-03-15 15:04:54 -07:00
Yann Collet a33ae64204 fixed decoding skippable frames 2017-02-28 01:15:28 -08:00
Przemyslaw Skibinski 684858e7b7 fix memory leaks 2017-02-21 18:17:24 +01:00
Sean Purcell 0ed3901b05 Update overlength match test case 2017-02-16 13:36:57 -08:00