Commit Graph

588 Commits (18b795374a4d1e0b308fdf6e86b1ea08b4489489)

Author SHA1 Message Date
Yann Collet 7303ed5d74 minor : specify compression level in tags 2017-06-29 11:30:31 -07:00
Yann Collet 590937df20 Merge pull request #739 from facebook/refPrefix
ZSTD_refPrefix
2017-06-29 04:36:03 -07:00
Yann Collet 811deaea6f Merge pull request #736 from terrelln/cover-default-api
[zdict] Make COVER the default algorithm
2017-06-28 20:25:36 -07:00
Yann Collet ad5f29918e Merge pull request #737 from facebook/advancedAPI2
simplified API for *initStatic*()
2017-06-28 20:25:22 -07:00
Yann Collet 62f7efc04a added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job 2017-06-28 16:25:13 -07:00
Paul Cruz 7ac4724bd2 removed fnum from DISPLAY statements 2017-06-28 13:00:49 -07:00
Paul Cruz e667d33b0b fixed generation of buggy test, corrected DISPLAY statements for errors 2017-06-28 12:19:37 -07:00
Yann Collet 33a6639039 fixed ZSTD_refPrefix with Multithread-enabled CCtx 2017-06-28 11:09:43 -07:00
Yann Collet 1fd2df9c2c fixed zstreamtest
modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary()
but it is still allowed after ZSTD_refPrefix()
2017-06-28 09:57:59 -07:00
Yann Collet 2e4274262d controlled dictMode 2017-06-27 17:09:12 -07:00
Yann Collet b7372933b8 implemented ZSTD_refPrefix() 2017-06-27 15:49:12 -07:00
Yann Collet fecc721fd9 added parameter ZSTD_p_refDictContent 2017-06-27 11:46:39 -07:00
Yann Collet 09a5bbe22e Merge pull request #735 from terrelln/xz-default
[zstdcli] Support xz by default when liblzma is available
2017-06-27 09:42:02 -07:00
Nick Terrell 5b7fd7c422 [zdict] Make COVER the default algorithm 2017-06-26 21:09:22 -07:00
Yann Collet 20eeb243d1 Merge pull request #729 from paulcruz74/corpus
Corpus
2017-06-26 17:47:28 -07:00
Yann Collet dde10b23fe refactored ZSTD_estimateDStreamSize()
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
Yann Collet 09ae03a570 ZSTD_estimateCDictSize_advanced()
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
Yann Collet 0c9a915a28 ZSTD_estimateCStreamSize_advanced() 2017-06-26 16:02:25 -07:00
Yann Collet 31af8290d1 ZSTD_estimateCCtx_advanced()
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.

ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
2017-06-26 15:52:39 -07:00
Nick Terrell 6aeb50ea0d [zstdcli] Add unlzma and unxz symlinks 2017-06-26 11:24:36 -07:00
Paul Cruz 298fe9f23b added cli tests to test-all 2017-06-23 17:10:41 -07:00
Paul Cruz 3a295a91f8 added additional condition so large offsets into the dictionary are not generated past windowSize 2017-06-23 15:54:51 -07:00
Paul Cruz 2085375816 fixed bug detected by the API test 2017-06-23 13:44:24 -07:00
Paul Cruz 8cd134559d type warnings 2017-06-23 12:00:48 -07:00
Paul Cruz 4219acc60a fixed bus error bug 2017-06-23 11:22:29 -07:00
Yann Collet ef269c1b68 Merge pull request #725 from facebook/advancedAPI2
New Advanced API
2017-06-23 09:50:47 -07:00
Yann Collet cca1ed14db Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-06-23 00:13:03 -07:00
Yann Collet 32f546c763 zstreamtest : disabled multi-threading tests with --newapi --no-big-tests
--no-big-tests is typically used in combination with qemu-user-static
qemu-user-static allocated 4 GB of RAM upfront.
On 2 GB VM, this can degenerate into a crash.
It's not a problem as long as memory is not used.
But with multi-threading enabled, memory fragmentation kicks in,
so the amoung of RAM effectively touched increases,
and can pass beyond the 2 GB limit of the VM.

In single-threaded mode, there is no such issue :
memory requirement is smaller, and remains well-located,
so very little fragmentation is expected.

This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests`
work fine on Travis CI.
2017-06-23 00:09:02 -07:00
Paul Cruz 2e8cc6f12a added sizeof for clarity 2017-06-22 15:52:33 -07:00
Paul Cruz 6d7f442120 changed makefile test 2017-06-22 15:44:01 -07:00
Paul Cruz b325a2e4db changed assignment 2017-06-22 15:36:28 -07:00
Paul Cruz 2935239544 added description for dictionary cli tests 2017-06-22 15:23:59 -07:00
Paul Cruz 97a8f89bec made changes to cli test so it runs properly 2017-06-22 15:13:41 -07:00
Paul Cruz 829eb29033 added cli test for decodecorpus inside tests/Makefile. Also changed calculation of offset 2017-06-22 14:43:44 -07:00
Paul Cruz 028fb9d01b added cli test case in Makefile 2017-06-22 13:53:39 -07:00
Paul Cruz 98751f69e7 should be updating seed whenever multiple files are generated 2017-06-22 10:23:36 -07:00
Paul Cruz 84cfa07d2d changed format of command to --use-dict=# 2017-06-22 10:04:14 -07:00
Yann Collet f99c2c1a22 reduced --no-big-tests even more
to pass tests on qemu-aarch64
2017-06-21 23:35:58 -07:00
Paul Cruz 04094f37e9 fixed offset in this case os that it always goes past src start 2017-06-21 18:47:40 -07:00
Yann Collet 49f8459d11 fixed minor cast warning 2017-06-21 18:43:39 -07:00
Paul Cruz 0950b3159a more meaningful names for count variables 2017-06-21 18:30:27 -07:00
Paul Cruz 0b6eedeace malloc samples instead of static allocation 2017-06-21 18:24:19 -07:00
Yann Collet 4d3bdcf130 reduced CLevelMax for --no-big-tests 2017-06-21 18:20:02 -07:00
Paul Cruz 52a13abf3f removed comment regarding windowSize requirement 2017-06-21 18:13:52 -07:00
Paul Cruz 32e36c2d9a removed requirement that offset be less than windowSize 2017-06-21 18:00:47 -07:00
Yann Collet bfc2f00080 --no-big-tests for zstreamtest
Apply --no-big-tests for tsan tests
2017-06-21 17:57:14 -07:00
Paul Cruz eeff2d0e3c added const x2 2017-06-21 17:48:37 -07:00
Paul Cruz 04253e21bb added const 2017-06-21 17:48:01 -07:00
Paul Cruz 2ceaec1aad got rid of unnecessary casting to void* 2017-06-21 17:46:37 -07:00
Paul Cruz ccae9ec807 added comment addressing use of bits in frame header descriptor 2017-06-21 17:43:21 -07:00
Yann Collet dce789281b fixed : decompression of skippable frames in streaming mode 2017-06-21 15:53:42 -07:00
Yann Collet c16748b137 added test cases for new command --list 2017-06-21 12:09:53 -07:00
Yann Collet 7bd1a2900e added ZSTD_dictMode_e to control dictionary loading mode 2017-06-21 11:50:33 -07:00
Paul Cruz db3606e249 added test for files with no frame content size 2017-06-20 17:43:36 -07:00
Paul Cruz acf0df678c fixed strange spacing 2017-06-20 15:41:43 -07:00
Paul Cruz 69bc4fab25 free the context object once done with decompression 2017-06-20 15:37:14 -07:00
Paul Cruz ceb4f65620 allocate statically so we don't have to deal with freeing 2017-06-20 15:31:54 -07:00
Paul Cruz a73c2a444a added tests with null files, changed condition to check that the file is more than 0 bytes 2017-06-20 14:33:08 -07:00
Paul Cruz 58c19b4202 spacing matters for test scripts 2017-06-20 14:14:53 -07:00
Yann Collet ff8f83bd47 fixed fuzzer test 2017-06-20 12:17:32 -07:00
Paul Cruz b07d0af2ba added additional tests for error detection 2017-06-20 11:54:44 -07:00
Paul Cruz 43ffcdde40 const dictOffset 2017-06-20 11:27:39 -07:00
Paul Cruz 1e0d125e76 const value 2017-06-20 11:15:26 -07:00
Paul Cruz 0034129270 made changes to accommodate single file use case 2017-06-20 11:07:38 -07:00
Paul Cruz 6a99de2dd8 change test to list info on .zst files 2017-06-20 10:45:06 -07:00
Paul Cruz 36e14b1963 added description to advanced usage, changed to use-dict 2017-06-20 10:34:34 -07:00
Paul Cruz 15102069a1 removed testing code from generation code 2017-06-20 10:25:16 -07:00
Paul Cruz ff5589c091 added comments for clarity 2017-06-20 10:17:38 -07:00
Paul Cruz 551fb7ca05 changed error condition 2017-06-20 10:14:03 -07:00
Paul Cruz b8237c6c66 refactoring for readability 2017-06-20 10:12:13 -07:00
Paul Cruz 6f17de2132 changed computation for readability 2017-06-20 09:50:28 -07:00
Paul Cruz 4a4f9ba262 const value that doesn't change 2017-06-19 17:46:21 -07:00
Paul Cruz 3a6856edca changed scope to avoid errors 2017-06-19 17:43:16 -07:00
Paul Cruz 7d2b5613b4 adhere to min dict size 2017-06-19 17:37:14 -07:00
Paul Cruz ea2af1ffc4 changed dictionary to stay reasonable size during test 2017-06-19 17:34:50 -07:00
Paul Cruz fdbb07283a added test to runTestMode 2017-06-19 17:29:15 -07:00
Paul Cruz bdec7b8ce7 fixed for loop declaration, fixed test case 2017-06-19 17:18:43 -07:00
Paul Cruz 23098b53e8 wrote a test for dictionary corpus 2017-06-19 17:13:00 -07:00
Paul Cruz 7a26390989 changing for loop declaration 2017-06-19 16:39:57 -07:00
Paul Cruz 147ef05f12 for loop declaration not allowed 2017-06-19 16:38:51 -07:00
Paul Cruz 791352ad0e scope change 2017-06-19 16:32:38 -07:00
Paul Cruz 365e265a8e changed scope so that warning disappears 2017-06-19 16:28:40 -07:00
Paul Cruz f4f928595b changed scope to get rid of mixed code/variables warning 2017-06-19 16:27:05 -07:00
Paul Cruz 7ea025cc7d added in struct to keep functions more orderly 2017-06-19 16:23:00 -07:00
Paul Cruz b91b810c64 made some small changes to dictInfo struct and initDictInfo 2017-06-19 16:05:05 -07:00
Paul Cruz 1ee4f6b36c cleaning up code 2017-06-19 15:55:01 -07:00
Paul Cruz 2a52d72712 added comment to anotate decompressed error code 2017-06-19 14:21:35 -07:00
Paul Cruz 73ebb646d8 small stylistic changes 2017-06-19 14:20:35 -07:00
Paul Cruz 34f4e5cab2 changed if statement styling 2017-06-19 14:07:56 -07:00
Paul Cruz 1fc1a35dc4 added calls to free to match calls to malloc, made some stylistic changes, added init function for new struct 2017-06-19 14:02:27 -07:00
Paul Cruz 8f4fa43fe8 got files to decompress 2017-06-19 13:21:50 -07:00
Yann Collet 688952062b minor declaration statement warning fix 2017-06-19 11:53:42 -07:00
Yann Collet d7a3bffba9 new api : setting compression parameters is refused if a dictionary is already loaded 2017-06-19 11:53:01 -07:00
Paul Cruz da4ddc1f76 fixed decompression so that it uses compressed version 2017-06-19 11:40:45 -07:00
Paul Cruz 33467add60 made changes so that HUF_writeCTable no longer fails 2017-06-19 11:27:50 -07:00
Yann Collet ed1d039127 newapi fuzzer tests : random flush orders in main loop 2017-06-19 11:07:33 -07:00
Paul Cruz 09ac51b56b small style changes 2017-06-19 09:24:38 -07:00
Yann Collet f5deae8a67 new api fuzzer tests : frame parameters are randonly set 2017-06-18 23:41:38 -07:00
Yann Collet ea3630a889 new api fuzzer set : dictionary is randomly set 2017-06-18 23:31:55 -07:00
Yann Collet 8dee0ec99e new api fuzzer tests : compression parameters are randomly set 2017-06-18 23:25:15 -07:00
Yann Collet bd05b19713 removed macro CLAMP from decodecorpus
never used,
and would duplicate with CLAMP from zstd_compress.c
2017-06-16 18:16:27 -07:00
Yann Collet 01743a36e7 fuzzer tests for new API 2017-06-16 17:56:41 -07:00
Paul Cruz 2f30433cd7 fixing general style 2017-06-15 18:06:32 -07:00
Paul Cruz ddd1ab710c added tests in playTest.sh 2017-06-15 16:53:03 -07:00
Paul Cruz 623b8fce85 add print statements for debugging and change malloc with updated size 2017-06-15 11:37:24 -07:00
Paul Cruz 52934fbfc8 added print statements for debugging, fixed first segfault 2017-06-15 09:47:29 -07:00
Paul Cruz 03a208e27c changed samples to geometric distribution so that it is huffman compressible 2017-06-14 18:06:16 -07:00
Paul Cruz d93207a79f changed dictionary from using fixed amount of bytes for the header / entropy tables 2017-06-14 17:23:56 -07:00
Paul Cruz 664ed05ff6 changed randomly generated samples to randomly selected from the dictionary content 2017-06-14 16:42:51 -07:00
Paul Cruz 3b0e24c06b added error message to catch case where size not given 2017-06-14 12:04:22 -07:00
Paul Cruz aa95bc0a5f code refactoring and adding some print statements for debugging 2017-06-14 10:54:47 -07:00
Paul Cruz 07cfc975bb debugging error with finalizeDictionary() not writing dictionary properly 2017-06-13 17:27:02 -07:00
Paul Cruz 8dd621f788 changed makefile to include zdict functions, wrote out dictionary to path 2017-06-13 14:19:35 -07:00
Paul Cruz f35f252e36 added code to generate dictionary using finalizeDictionary 2017-06-13 11:54:43 -07:00
Paul Cruz 37e1b1488b removed previous double free crash, used new crash function 2017-06-09 13:57:18 -07:00
Paul Cruz be9b0ae627 added error function for different compilation modes 2017-06-09 13:41:34 -07:00
Paul Cruz 11c3987baf added code to extend the offset when a dictionary is detected 2017-06-09 09:48:56 -07:00
Paul Cruz 125ed59968 made sure dictionary ID was being written in the frame header of each file 2017-06-08 17:27:15 -07:00
Paul Cruz 233ee5334e set the lower bits of frame header descriptor if dictionary is needed 2017-06-08 17:11:33 -07:00
Paul Cruz c2d909e396 added code for generating dictionary/test files randomly. Still need to make sure dictionary ID matches 2017-06-08 17:06:30 -07:00
Yann Collet 8c910d2097 updated ZSTDMT streaming API
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
Yann Collet cdf7e82222 Added ZSTD_initStaticCDict() 2017-05-25 18:05:49 -07:00
Yann Collet 57827f906f added ZSTD_initStaticDDict() 2017-05-25 15:44:06 -07:00
Yann Collet 25989e361c updated ZSTD_estimate?DictSize() to pass parameter byReference
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
Yann Collet 0fdc71c3dc added ZSTD_initStaticDCtx() 2017-05-24 17:41:41 -07:00
Yann Collet c7fe262dc9 added ZSTD_initStaticCCtx()
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
2017-05-23 13:20:41 -07:00
Yann Collet 60a557e7fd fixing symbol.c test
I believe it would be better to rely on fuzzer tests compiled with dll
2017-05-19 11:27:43 -07:00
Yann Collet fa3671eac7 changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
2017-05-19 10:51:30 -07:00
Yann Collet 009d604e00 ZSTD_compress_generic() supports multiple successive frames
also : clarified streaming API implementation
2017-05-19 10:17:59 -07:00
Yann Collet 0bdb575c31 Merge branch 'dev' into advancedAPI2 2017-05-16 16:32:29 -07:00
Yann Collet 2d4d31c18a removed gcc compilation flag -Wbad-function-cast
It makes it more difficult to directly cast the result of a function,
requiring to store the result in an intermediate variable.
It does not necessarily help readability,
and this restriction can be difficult to overcome in some constructions,
like some macros.

also : fixed minor Visual conversion warnings in datagencli.c
2017-05-16 11:34:38 -07:00
Yann Collet 83d0c764dc added several compilation flags 2017-05-15 17:15:46 -07:00
Yann Collet a5ffe3d370 pushed enum values for strategy by one (ZSTD_fast==1)
this makes it possible to use `0` to mean:
"do not change strategy"
2017-05-12 16:29:19 -07:00
Nick Terrell 5f2c7213c7 Merge remote-tracking branch 'upstream/dev' into btopt
* upstream/dev: (305 commits)
  added test for ZSTD_estimateCStreamSize()
  changed variable name, for clarity
  fixed ZSTD_estimateCStreamSize()
  shortened ZSTD_createCStream_Advanced()
  fixed symbols test
  added ZSTD_estimateDStreamSize()
  changed name frameParams into frameHeader
  regroup memory usage function declarations
  separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
  bumped version number
  added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
  Updated ZSTD_freeCCtx()
  updated ZSTD_estimateCCtxSize()
  Updated ZSTD_sizeof_CCtx()
  merged CCtx and CStream as a single same object
  cli : -d and -t do not stop after a failed decompression
  added dev branch CircleCI badge
  added dev branch Appveyor badge
  keep dev branch status only
  creates a binary archive without the `programs` directory
  ...
2017-05-10 16:49:58 -07:00
Yann Collet 30ab64e21d added test for ZSTD_estimateCStreamSize() 2017-05-10 11:30:19 -07:00
Yann Collet 461a9cc7c6 fixed symbols test 2017-05-09 16:53:09 -07:00
Yann Collet f16f4497ca added ZSTD_estimateDStreamSize() 2017-05-09 16:18:17 -07:00
Yann Collet 542c9dfcf8 changed name frameParams into frameHeader
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()

The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.

ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
2017-05-09 15:46:07 -07:00
Yann Collet fa8dadb294 separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
for clarity
2017-05-08 18:24:16 -07:00
Yann Collet a1d6704d7f added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
2017-05-08 17:51:49 -07:00
Yann Collet fc5145955a updated ZSTD_estimateCCtxSize()
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).

Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
2017-05-08 17:07:59 -07:00
Yann Collet 0be6fd3429 merged CCtx and CStream as a single same object
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
2017-05-08 16:08:01 -07:00
Yann Collet 01a1abfdb5 cli : -d and -t do not stop after a failed decompression
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
2017-05-05 19:15:24 -07:00
Nick Terrell f376d47c11 [CLI] Switch dictionary builder on CLI to cover 2017-05-02 11:18:27 -07:00
Yann Collet a51cab6e68 Merge pull request #678 from facebook/apiChange
Breaking API Change around CDict
2017-04-28 10:02:45 -07:00
Yann Collet 36e79a1a1f tests : -g is part of DEBUGFLAGS 2017-04-27 17:29:40 -07:00
Yann Collet c6915429f2 shortened Appveyor release tests
fuzzer supports time suffix `s` for "seconds"
2017-04-27 16:24:53 -07:00
Yann Collet a92cbb7004 Added a secondary test, checking dictID presence after setting noDictIdFLag=1 2017-04-27 15:08:56 -07:00
Yann Collet 7d283cdfa2 added test for ZSTD_initCStream_usingCDict_advanced() 2017-04-27 14:48:34 -07:00
Yann Collet 2f73427d35 added test for ZSTD_compress_usingCDict_advanced() 2017-04-27 14:39:39 -07:00
Yann Collet 7321345fd2 fixed another VS2008 init error 2017-04-27 14:19:34 -07:00
Yann Collet f4bd857d81 created ZSTD_compress_usingCDict_advanced() 2017-04-27 11:31:55 -07:00
Yann Collet 31533bacce Changed ZSTD_createCDict_advanced()
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
Sean Purcell 4de8632957 Add LZ4 compress/decompress support to CLI 2017-04-24 16:48:25 -07:00
Yann Collet 230d7acc7d cli : add support for --threads=# command
updated documentation
add relevant test case
2017-04-21 11:38:13 -07:00
Yann Collet c17e020c9a disable assert when compiling paramgrill
paramgrill is a benchmark calibration function.
Speed accuracy is critical, it cannot be altered by assert.
2017-04-20 12:50:02 -07:00
Yann Collet 2c5514c759 fixed ZSTDMT_initCStream_advanced()
Must use the new ZSTD_compressBegin_usingCDict_advanced()
to enforce correct frame parameters
2017-04-18 22:52:41 -07:00
Yann Collet 0bb381dad8 added test for ZSTD_initCStream_advanced()
when params.fParams.contentSizeFlags = 1
and pledgedSrcSize = 0
then srcSize should be considered 0 (empty), and not "unknown"
2017-04-18 15:08:52 -07:00
Yann Collet 4f818182b8 clarified frame parameters for ZSTD_compress*_usingCDict()
created ZSTD_compressBegin_usingCDict_internal(),
which gives direct control to frame Parameters.
ZSTD_resetCStream_internal() now points into it.
2017-04-17 18:29:06 -07:00
Yann Collet 9c4c970399 Merge branch 'dev' into compressionFlow 2017-04-17 16:33:09 -07:00
Sean Purcell d845dab69c Fix input size too small to trigger zstdmt 2017-04-17 12:10:58 -07:00
Sean Purcell c424ec2eae Add multithreading tests to playTests.sh 2017-04-17 11:41:46 -07:00
Yann Collet 2afb075382 Merge branch 'dev' into compressionFlow 2017-04-14 15:08:31 -07:00
Yann Collet f913cbed33 fixed : memory leak in fuzzer test 2017-04-13 22:46:51 -07:00
Yann Collet 20d5e03893 content size is controlled at bufferless level
so it's active for all entry points

Also : added relevant test (wrong content size) in fuzzer
2017-04-11 18:34:02 -07:00
Sean Purcell d37e1df2ab Fix message 2017-04-11 17:33:26 -07:00
Sean Purcell eb70d219fd Add test of file > 4GB to playTests 2017-04-11 17:15:13 -07:00
Yann Collet 4ee6b15dac force contentSizeFlag=0 when using ZSTD_initCStream_usingCDict()
because by definition srcSize is not known when using this prototype.
added relevant test

Note : this use was already working, because at a later stage
(both ZSTD_compressBegin_usingCDict() and ZSTD_copyCCtx())
pledgedSrcSize=0 is translated into "unknown", no matter the frame parameter.
This is not correct, but of little importance,
as the medium term plan is to no longer set fParams within CDict
2017-04-11 11:59:44 -07:00
Yann Collet 4b987ad8ce Introduce ZSTD_initCStream_internal()
This is now the regroup point for ZSTD_initCStream*() functions

ZSTD_initCStream_advanced() now properly checks for parameters validity.

Also : added <assert.h> usage inside zstd_compress.c
Needs ZSTD_DEBUG=1 macro to be triggered.
Will be triggered by default from `tests` directory
2017-04-10 17:50:44 -07:00
Yann Collet 1fa3b75369 Merge pull request #648 from facebook/zstdmt
improved zstreamtest --mt to trap bug #644
2017-04-07 06:09:52 -07:00
Sean Purcell 1652172b2d Add refuse to overwrite test and fix format 2017-04-06 17:06:30 -07:00
Sean Purcell 9da11c6aae Add test for overwriting read-only 2017-04-06 12:58:49 -07:00
Yann Collet ce80098f14 improved zstreamtest --mt to trap bug #644 2017-04-05 16:34:09 -07:00
Yann Collet 9631e603ae paramgrill : results from optimizer use cli format
Best found configuration is displayed
using documented cli advanced parameter syntax.
2017-04-04 16:54:33 -07:00
Yann Collet 89b32f3db0 fix paramgrill -O#
ensure proposed config does not require more memory than necessary
2017-04-04 16:41:11 -07:00
Yann Collet 2724f25567 fixed paramgrill -O#
find optimal settings for a minimum speed
2017-04-04 16:31:17 -07:00
Yann Collet c2007388a5 fixed bench.c : optional advanced parameters applied
before creating cdict
2017-04-04 15:35:06 -07:00
Nick Terrell 39a6cc5172 Make ZSTD_compress_usingCDict() respect contentSizeFlag 2017-04-03 21:09:55 -07:00
Nick Terrell 62ecad3819 Fix ZSTD_initCStream_usingCDict() to use dictionary 2017-04-03 21:05:59 -07:00
Yann Collet 30c7698970 optimize ZSTDMT_compress() memory usage
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).

Also : fix ZSTD_compressBound() for small srcSize
2017-03-31 18:27:03 -07:00
Yann Collet 6476c51b86 Merge pull request #637 from facebook/zstdmt
Zstdmt
2017-03-30 14:18:37 -07:00
Nick Terrell 5152fb2cb2 Convert all tabs to spaces 2017-03-29 18:51:58 -07:00
Yann Collet 1e1e26f2ac fixed #634 : ZSTDMT_compressCCtx() doesn't provide frame content size in header 2017-03-29 17:09:59 -07:00
Yann Collet 0ef6803707 added ZSTD_getFrameContentSize() to ZSTDMT unit test in fuzzer
Now tests fail, because ZSTDMT_compress() doesn't fill frame content size correctly.
2017-03-29 16:58:57 -07:00
Yann Collet 41fefd573a Improved speed tests
Better compatibility with Mac OS-X
Force attribution to a selected core # (Linux)
2017-03-26 23:52:19 -07:00
Yann Collet 4c41d37fcc changed test for new syntax
--dictID= and --maxdict=
2017-03-24 18:36:56 -07:00
Yann Collet f332ece468 dictBuilder fails to create dictionary on certain input
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
2017-03-23 16:24:02 -07:00
Sean Purcell 680e4e0953 Ignore symbolic links unless --force specified 2017-03-23 11:52:09 -07:00
Sean Purcell 042ba122ae Change g_displayLevel to int and fix DISPLAYUPDATE flush 2017-03-23 11:21:59 -07:00
Sean Purcell 1b418bd3da Merge remote-tracking branch 'origin/dev' into appveyor 2017-03-21 10:59:54 -07:00
Sean Purcell 5a2b857029 Update appveyor.yml to have short and long tests 2017-03-20 17:04:44 -07:00
Sean Purcell c771977489 Fix decodecorpus clang compile errors 2017-03-20 16:02:24 -07:00
Sean Purcell 2f6c7e6a53 Fix windows compiler warnings for decodecorpus 2017-03-20 13:45:12 -07:00
Yann Collet 170861da70 Merge pull request #610 from iburinoc/slowfuzzer
Strongly limit compression level when source is large for tests running on QEMU
2017-03-20 13:36:01 -07:00
Sean Purcell 0b0b10ec83 Add --no-big-tests flag for qemu fuzz-tests 2017-03-20 11:32:24 -07:00
Sean Purcell 7ebf2de02d Add ability to strongly limit fuzzer test size with flag 2017-03-20 11:25:00 -07:00
Yann Collet 35c147ab6a Merge pull request #605 from iburinoc/term
Fix zstd not erroring on compressing to terminal without forceStdout
2017-03-17 14:02:48 -07:00
Sean Purcell 9a38dfa794 Only run IS_CONSOLE tests with a TTY 2017-03-17 12:32:18 -07:00
Sean Purcell d973071e90 Add tests for compressed data on console 2017-03-16 16:25:19 -07:00
Sean Purcell f5e50519e0 Prevent fuzz testers from combining large dicts with high clevel 2017-03-15 15:04:54 -07:00