Commit Graph

316 Commits (0d0fea96bced5a5fc1c2805815ce6e41744c9694)

Author SHA1 Message Date
Yann Collet ce80098f14 improved zstreamtest --mt to trap bug #644 2017-04-05 16:34:09 -07:00
Yann Collet 9631e603ae paramgrill : results from optimizer use cli format
Best found configuration is displayed
using documented cli advanced parameter syntax.
2017-04-04 16:54:33 -07:00
Yann Collet 89b32f3db0 fix paramgrill -O#
ensure proposed config does not require more memory than necessary
2017-04-04 16:41:11 -07:00
Yann Collet 2724f25567 fixed paramgrill -O#
find optimal settings for a minimum speed
2017-04-04 16:31:17 -07:00
Yann Collet c2007388a5 fixed bench.c : optional advanced parameters applied
before creating cdict
2017-04-04 15:35:06 -07:00
Nick Terrell 39a6cc5172 Make ZSTD_compress_usingCDict() respect contentSizeFlag 2017-04-03 21:09:55 -07:00
Nick Terrell 62ecad3819 Fix ZSTD_initCStream_usingCDict() to use dictionary 2017-04-03 21:05:59 -07:00
Yann Collet 30c7698970 optimize ZSTDMT_compress() memory usage
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).

Also : fix ZSTD_compressBound() for small srcSize
2017-03-31 18:27:03 -07:00
Yann Collet 6476c51b86 Merge pull request #637 from facebook/zstdmt
Zstdmt
2017-03-30 14:18:37 -07:00
Nick Terrell 5152fb2cb2 Convert all tabs to spaces 2017-03-29 18:51:58 -07:00
Yann Collet 1e1e26f2ac fixed #634 : ZSTDMT_compressCCtx() doesn't provide frame content size in header 2017-03-29 17:09:59 -07:00
Yann Collet 0ef6803707 added ZSTD_getFrameContentSize() to ZSTDMT unit test in fuzzer
Now tests fail, because ZSTDMT_compress() doesn't fill frame content size correctly.
2017-03-29 16:58:57 -07:00
Yann Collet 41fefd573a Improved speed tests
Better compatibility with Mac OS-X
Force attribution to a selected core # (Linux)
2017-03-26 23:52:19 -07:00
Yann Collet 4c41d37fcc changed test for new syntax
--dictID= and --maxdict=
2017-03-24 18:36:56 -07:00
Yann Collet f332ece468 dictBuilder fails to create dictionary on certain input
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
2017-03-23 16:24:02 -07:00
Sean Purcell 680e4e0953 Ignore symbolic links unless --force specified 2017-03-23 11:52:09 -07:00
Sean Purcell 042ba122ae Change g_displayLevel to int and fix DISPLAYUPDATE flush 2017-03-23 11:21:59 -07:00
Sean Purcell 1b418bd3da Merge remote-tracking branch 'origin/dev' into appveyor 2017-03-21 10:59:54 -07:00
Sean Purcell 5a2b857029 Update appveyor.yml to have short and long tests 2017-03-20 17:04:44 -07:00
Sean Purcell c771977489 Fix decodecorpus clang compile errors 2017-03-20 16:02:24 -07:00
Sean Purcell 2f6c7e6a53 Fix windows compiler warnings for decodecorpus 2017-03-20 13:45:12 -07:00
Yann Collet 170861da70 Merge pull request #610 from iburinoc/slowfuzzer
Strongly limit compression level when source is large for tests running on QEMU
2017-03-20 13:36:01 -07:00
Sean Purcell 0b0b10ec83 Add --no-big-tests flag for qemu fuzz-tests 2017-03-20 11:32:24 -07:00
Sean Purcell 7ebf2de02d Add ability to strongly limit fuzzer test size with flag 2017-03-20 11:25:00 -07:00
Yann Collet 35c147ab6a Merge pull request #605 from iburinoc/term
Fix zstd not erroring on compressing to terminal without forceStdout
2017-03-17 14:02:48 -07:00
Sean Purcell 9a38dfa794 Only run IS_CONSOLE tests with a TTY 2017-03-17 12:32:18 -07:00
Sean Purcell d973071e90 Add tests for compressed data on console 2017-03-16 16:25:19 -07:00
Sean Purcell f5e50519e0 Prevent fuzz testers from combining large dicts with high clevel 2017-03-15 15:04:54 -07:00
Yann Collet 3773bd318f Merge pull request #601 from terrelln/nxz
Add xz and lzma support.
2017-03-13 22:00:03 -07:00
Nick Terrell aa8bcf360f Add xz and lzma support.
Finish feature started by @inikep.

* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
2017-03-13 18:45:41 -07:00
Sean Purcell 120df494e9 Update builds to not support legacy v01-v03 2017-03-13 14:44:08 -07:00
Sean Purcell 7c8f5d5bc7 Make test times overwritable 2017-03-09 16:05:23 -08:00
Sean Purcell daec40db24 Update .travis.yml and Makefile for medium tests 2017-03-09 16:05:22 -08:00
Nick Terrell eeb31eed39 s/ZSTD_btopt2/ZSTD_btultra/g 2017-03-09 11:44:25 -08:00
Yann Collet 8b1d004031 added -Wformat-security flag, as recommended by @pixelb 2017-03-05 21:17:32 -08:00
Yann Collet 27526c7201 make : added target shortest
shortest only run fast part of playTests.sh .
cc @iburinoc
2017-03-01 17:02:49 -08:00
Yann Collet c1c040eae1 added gzip tests
also : made sure zstd --format=gzip -V
would fail if gzip compatibility is not supported
2017-03-01 16:49:20 -08:00
Yann Collet 51598510c0 Merge pull request #580 from facebook/speedStream
Improve streaming decompression speed
2017-03-01 10:59:51 -08:00
Yann Collet a33ae64204 fixed decoding skippable frames 2017-02-28 01:15:28 -08:00
Yann Collet c0b1731bce added test for decompression with NULL dict and NULL DDict
previous version of ZSTD_decompressMultiFrame() would fail that test
2017-02-28 01:02:46 -08:00
Przemyslaw Skibinski 8b3560e196 update gzip tests 2017-02-28 09:41:23 +01:00
Yann Collet 1d7f30f9d4 Merge branch 'decompressStream' into dev 2017-02-27 20:55:22 -08:00
Yann Collet a81c2e7e44 Merge pull request #573 from facebook/ddict
Improved DDict memory usage
2017-02-27 20:54:42 -08:00
Yann Collet 952d06fa9c fullbench : -i0 displays list of functions to bench 2017-02-27 17:58:02 -08:00
Yann Collet 67d86a74a5 added test case : --rm on stdin
must remain silent (instead of failing)
2017-02-27 16:09:20 -08:00
Yann Collet ea7589ce07 Merge pull request #571 from inikep/dev11
gzip tests
2017-02-27 13:54:33 -08:00
Przemyslaw Skibinski 5d848527e6 use "./gzip" for gzip tests 2017-02-27 22:02:03 +01:00
Przemyslaw Skibinski b43d75154d update gzip tests 2017-02-27 09:07:35 +01:00
Przemyslaw Skibinski 1479c98661 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-27 08:56:43 +01:00
Sean Purcell 2302bfa4bf Merge branch 'dev' into testcorpus 2017-02-26 22:27:31 -08:00
Yann Collet bd7fa21deb added ZSTD_refDDict()
Now DDict does no longer depends on DCtx duplication
2017-02-26 14:43:07 -08:00
Yann Collet 8dff956dbf Added DDict unit test in fuzzer
also : slightly modified loadEntropy :
know src must points at start of dictionary
2017-02-25 10:11:15 -08:00
Yann Collet 682ae8a10e Merge pull request #567 from inikep/dev11
faster Travis tests
2017-02-25 06:57:31 -08:00
Sean Purcell 22468b0cc3 Fix some more ARM compile errors
https://travis-ci.org/facebook/zstd/jobs/204807461

Can't get them to reproduce the compile errors locally,
tested fix by forcing that test to run on Travis.
2017-02-24 10:55:42 -08:00
Przemyslaw Skibinski a66b764d79 added tests for gzip 2017-02-24 16:09:17 +01:00
Przemyslaw Skibinski 8740d6bcf5 fix uninitialized value warning 2017-02-24 09:24:55 +01:00
Yann Collet df9f9296e3 attempt to fix pthreat linking error
replacing -lpthread by -pthread
2017-02-24 00:16:05 -08:00
Yann Collet 831b4890ce minor tests/Makefile refactoring
and update of zstd_manual,html
2017-02-23 23:09:10 -08:00
Sean Purcell d590291d86 Fix -Wsign-compare issues in decodecorpus.c
https://travis-ci.org/facebook/zstd/jobs/204423280
2017-02-23 16:11:08 -08:00
Yann Collet 7e224ea03e Merge pull request #562 from iburinoc/testcorpus
Create a tool that generates random, valid, Zstd frames for decoder testing
2017-02-23 15:19:36 -08:00
Sean Purcell 3cd8d50c34 Update CLI and link from educational decoder 2017-02-23 13:06:50 -08:00
Sean Purcell 485ca8c352 Update tests/README.md 2017-02-23 10:27:00 -08:00
Sean Purcell f119b62055 Create a tool that generates random, valid, Zstd frames for decoder testing
Note: Does not handle dictionaries currently
2017-02-22 16:08:34 -08:00
Sean Purcell 9050e1925e Change name to to findFrameCompressedSize and add skippable support 2017-02-22 12:12:34 -08:00
Przemyslaw Skibinski 337ec875b6 minor tweaks 2017-02-22 10:31:30 +01:00
Przemyslaw Skibinski 3d836bfd18 travis.yml: fix versionsTest target 2017-02-22 09:36:42 +01:00
Przemyslaw Skibinski 346ce32ade legacy.c: fix memory leaks 2017-02-21 20:10:21 +01:00
Przemyslaw Skibinski 684858e7b7 fix memory leaks 2017-02-21 18:17:24 +01:00
Yann Collet f2da1b1f22 Merge pull request #550 from iburinoc/matchoverlenwrite
execSequence copies up to 2*WILDCOPY_OVERLENGTH extra
2017-02-16 18:14:47 -08:00
Yann Collet ff176d3208 Merge pull request #548 from inikep/gz_compress
Circle CI tests
2017-02-16 14:57:59 -08:00
Sean Purcell 0ed3901b05 Update overlength match test case 2017-02-16 13:36:57 -08:00
Sean Purcell 887eaa9e21 Fix wildcopy overwriting data still in window 2017-02-15 16:43:45 -08:00
Przemyslaw Skibinski 9b5a1e9d97 added circle.yml 2017-02-14 20:06:41 +01:00
Nick Terrell 74b81ada25 Don't run test-pool with QEMU
> make test -n
    ...
    ./pool
    > make test -n QEMU_SYS=valgrind
    ...
    ./legacy
    # ./pool not run
2017-02-14 10:08:18 -08:00
Sean Purcell 5069b6c2c3 Merge branch 'dev' into multiframe 2017-02-10 10:08:55 -08:00
Yann Collet a28c34cb7a Merge pull request #538 from iburinoc/errorstring
Fix ZSTD_getErrorString and add tests
2017-02-10 03:59:56 -08:00
Sean Purcell 84b37cc1f1 Fix failing unit test 2017-02-09 12:27:39 -08:00
Sean Purcell 478d7174e5 Merge branch 'dev' into multiframe 2017-02-09 11:54:22 -08:00
Sean Purcell 2db7249265 Make pledgedSrcSize meaning clear for other functions
- Added tests
- Moved new size functions to static link only
2017-02-09 11:49:58 -08:00
Przemyslaw Skibinski 80c1e04c33 Merge branch 'dev' into dev11 2017-02-09 09:42:16 +01:00
Sean Purcell e0b3265e87 Fix ZSTD_getErrorString and add tests 2017-02-08 17:28:49 -08:00
Sean Purcell f07ddf88e8 Test multiframe legacy decoding with simple and streaming APIs 2017-02-08 15:13:20 -08:00
Sean Purcell ba2ad9f25c ZSTD_decompress now handles multiple frames 2017-02-08 14:50:10 -08:00
Sean Purcell 4e709712e1 Decompressed size functions now handle multiframes and distinguish cases
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
2017-02-08 14:50:10 -08:00
Nick Terrell b5e46b1255 Remove test-longmatch from test target and only run it once 2017-02-08 12:00:21 -08:00
Przemyslaw Skibinski 4b4f8c2d71 turn off test-pool for qemu-ppc64-static 2017-02-08 13:58:04 +01:00
Yann Collet 48bed91606 Merge pull request #527 from facebook/zstdmt
zstdmt refinements
2017-01-31 16:36:46 -08:00
Yann Collet 92c98a5b21 zstreamtest uses random section sizes for fuzzing 2017-01-30 12:50:31 -08:00
Yann Collet cd23dd24af zstreamtest uses random overlapLog for fuzzing 2017-01-30 12:46:35 -08:00
Yann Collet 64bf8ffce6 report @terrelln patch to ST fuzzer tests 2017-01-27 17:25:07 -08:00
Nick Terrell d98bf49224 Fix segfault in zstreamtest MT
It was reading beyond the end of the input buffer because no errors were
detected. Once that was fixed, it wasn't making forward progress because
no errors were detected and it was waiting for input.
2017-01-27 15:42:36 -08:00
Yann Collet 717c65d690 Merge pull request #519 from inikep/dev11
Dev11
2017-01-26 14:23:44 -08:00
Yann Collet 8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet 06e7697f96 added test of new parameter ZSTD_p_forceWindow 2017-01-25 16:39:03 -08:00
Yann Collet bb0027405a fixed zstdmt corruption issue when enabling overlapped sections
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
Przemyslaw Skibinski 3bb010a667 .travis.yml: optimized order of short tests 2017-01-25 11:19:35 +01:00
Yann Collet bd6bc22612 playtest.sh : changed sdiff into $DIFF 2017-01-22 15:54:14 -08:00
Yann Collet 317604e0ad fixed : compilation of zstreamtest in dll mode 2017-01-20 17:18:41 -08:00
Yann Collet d7e3cb58c5 Resolved merge conflict dev+zstdmt 2017-01-20 16:44:50 -08:00
Yann Collet 19d670ba9d Added ZSTDMT_initCStream_advanced() variant
Correctly compress with custom params and dictionary
Added relevant fuzzer test in zstreamtest

Also :
new macro ZSTDMT_SECTION_LOGSIZE_MIN, which sets a minimum size for a full job
(note : a flush() command can still generate a partial job anytime)
2017-01-19 15:32:07 -08:00