Commit Graph

1082 Commits (b93598d6a4b50c85f43e59803afe31d6ac84a6b0)

Author SHA1 Message Date
Yann Collet f47284fec0 reorganized Makefile for multiple targets 2017-05-02 16:55:57 -07:00
Yann Collet 3791d2105d added xzstd4 target
support for all formats, xz/lzma/lz4 included
2017-05-02 16:38:37 -07:00
Yann Collet 2e63a877f3 fixed xzstd
--format=xz was missing a break, making the execution continue into lz4 error message
2017-05-02 15:40:42 -07:00
Nick Terrell f376d47c11 [CLI] Switch dictionary builder on CLI to cover 2017-05-02 11:18:27 -07:00
Nick Terrell 020b960e13 [cover] Make optimization faster 2017-05-02 11:02:48 -07:00
Nick Terrell f2d9ef1dc0 [cover] Optimize case where d <= 8 2017-05-02 11:02:43 -07:00
Nick Terrell feba969a69 Fix LZ4_MSG in xzstd 2017-05-02 11:02:16 -07:00
Yann Collet 31533bacce Changed ZSTD_createCDict_advanced()
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
Sean Purcell eab41c1872 Fix LZ4 wrapper deprecation warnings 2017-04-26 10:17:38 -07:00
Sean Purcell 2c4b6fe6b3 Make lz4 compression/decompression compatible with library r123 2017-04-25 11:00:54 -07:00
Sean Purcell 4de8632957 Add LZ4 compress/decompress support to CLI 2017-04-24 16:48:25 -07:00
Yann Collet 230d7acc7d cli : add support for --threads=# command
updated documentation
add relevant test case
2017-04-21 11:38:13 -07:00
Sean Purcell 9606256a8d Fix no thread message 2017-04-18 13:52:00 -07:00
Yann Collet 1c286950af Merge pull request #663 from iburinoc/zstdmt-default
Compile CLI using files instead of objs
2017-04-18 02:59:54 -07:00
Sean Purcell c8b2df7d62 Compile CLI using files instead of objs
This avoids conflicts between how the library was
configured and how the CLI was configured.
2017-04-17 17:13:47 -07:00
Sean Purcell e6c504dbe6 Update -T0 comment in man page 2017-04-17 17:12:02 -07:00
Sean Purcell 5935c990a0 Add zstdmt and -T0 to man page 2017-04-17 16:05:20 -07:00
Sean Purcell f6ef4db20e Install zstdmt even without threading support 2017-04-17 12:21:11 -07:00
Sean Purcell 5a61f36474 Make zstd compile with mt by default 2017-04-17 12:06:02 -07:00
Baptiste Daroussin 7dd14d03b0 Enable multithreading on BSD 2017-04-15 16:25:08 +02:00
Sean Purcell e4f3235c85 Add 0 initializers to static variables 2017-04-13 16:34:28 -07:00
Sean Purcell 42bac7fa84 Change ifndef's to undef's 2017-04-13 15:35:05 -07:00
Sean Purcell 9227aae001 Fix clang linux compilation 2017-04-13 14:06:40 -07:00
Sean Purcell 3b6207d4bd Fix compilation on windows 2017-04-13 14:04:10 -07:00
Sean Purcell f876f1200c Fix compilation on macOS 2017-04-13 12:33:45 -07:00
Sean Purcell afa48518e2 -T0 detects number of physical cores 2017-04-13 12:28:28 -07:00
Yann Collet 5c42d0edc8 cli : better status display for zstdmt in 1-thread mode 2017-04-11 16:57:32 -07:00
Yann Collet 0e30059ba1 cli : FIO_createDictBuffer() replaces FIO_loadFile()
makes it more explicit that it allocates a buffer
and that it's meant to be used for dictionary.

Also : simplified function a bit,
now only works for dictionaries up to DICTSIZE_MAX
2017-04-11 14:41:02 -07:00
Sean Purcell 279be2015b Let zstd overwrite read-only files 2017-04-06 12:56:40 -07:00
Yann Collet c2007388a5 fixed bench.c : optional advanced parameters applied
before creating cdict
2017-04-04 15:35:06 -07:00
Yann Collet 81d6380139 minor bench.c adjustments
shorter debug messages
no need to check decompressedLength==0 twice
2017-04-04 15:21:09 -07:00
Yann Collet a36330a27d minor man update, for `overlapSize` 2017-04-03 14:57:24 -07:00
Yann Collet 805c5a3efb updated documentation on multithreading modes 2017-04-01 00:36:31 -07:00
Nick Terrell eb38617175 Clean up default sparse logic 2017-03-31 15:20:50 -07:00
Nick Terrell 96fe545a18 Turn off sparse mode for OS X by default 2017-03-31 15:16:43 -07:00
Nick Terrell 5152fb2cb2 Convert all tabs to spaces 2017-03-29 18:51:58 -07:00
Yann Collet 5bde4be5a1 fix : bench automatically adapts parameters to srcSize 2017-03-29 12:10:38 -07:00
Sean Purcell efdaf8bb7c Fix inline compile errors 2017-03-27 12:26:40 -07:00
Sean Purcell 894bf49713 Fix IS_CONSOLE returning 1 for NUL on windows 2017-03-27 12:19:30 -07:00
Yann Collet 0246d5c531 Merge pull request #630 from facebook/advancedCliCommands
changed advanced commands --maxdict= and --dictID=
2017-03-26 00:13:35 -07:00
Sean Purcell ef30af843f Ignore extension in command name matching 2017-03-24 17:06:09 -07:00
Yann Collet 96aa3019b2 changed advanced commands --maxdict= and --dictID=
now works with the `=` variant, which is the recommended one.
Old variant `--dictID #` still works, for compatibility with existing scripts.
Long term objective is to remove the old variant..
2017-03-24 16:04:29 -07:00
Yann Collet f332ece468 dictBuilder fails to create dictionary on certain input
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
2017-03-23 16:24:02 -07:00
Sean Purcell dee08caabb Fix displayLevel redefinition issues 2017-03-23 12:09:35 -07:00
Sean Purcell 680e4e0953 Ignore symbolic links unless --force specified 2017-03-23 11:52:09 -07:00
Sean Purcell 042ba122ae Change g_displayLevel to int and fix DISPLAYUPDATE flush 2017-03-23 11:21:59 -07:00
Nick Terrell dfc75f74da Add $(EXT) to binary to fix zlib detection for some MinGW versions 2017-03-22 13:24:52 -07:00
Nick Terrell 741e3e8314 Add $(FLAGS) when testing for zlib/lzma support.
* If zlib/lzma isn't in the usual spot, it won't be used,
  even if `$CFLAGS` and `$LDFLAGS` add the location it is in.
* Update the test code snippets to not trigger any warnings.
2017-03-22 12:33:08 -07:00
Yann Collet 35c147ab6a Merge pull request #605 from iburinoc/term
Fix zstd not erroring on compressing to terminal without forceStdout
2017-03-17 14:02:48 -07:00
Yann Collet 0a189b63fe fix minor details in man page 2017-03-16 16:33:53 -07:00
Sean Purcell b8e52d3c83 Fix zstd not erroring on compressing to terminal without forceStdout 2017-03-16 16:06:19 -07:00
Sean Purcell 72a243e3bd Convert manpage to Markdown, converted with ronn 2017-03-16 15:37:32 -07:00
Yann Collet 3773bd318f Merge pull request #601 from terrelln/nxz
Add xz and lzma support.
2017-03-13 22:00:03 -07:00
Nick Terrell aa8bcf360f Add xz and lzma support.
Finish feature started by @inikep.

* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
2017-03-13 18:45:41 -07:00
Sean Purcell 120df494e9 Update builds to not support legacy v01-v03 2017-03-13 14:44:08 -07:00
Sean Purcell 334cb34edb ZSTD_LEGACY_SUPPORT defines lowest supported version 2017-03-13 14:32:30 -07:00
Dmitry V. Levin 201e8c8157 programs/Makefile: remove zstd-internal target
zstd-internal was intended to be a helper target, but it doesn't help
at all, what it does in practice is a useless rebuild of zstd every time
"make zstd" is invoked.

Fixes: 030ac243a0 ("Changed Makefile to generate zstd with .gz support by default")
2017-03-09 22:47:14 +00:00
Nick Terrell eeb31eed39 s/ZSTD_btopt2/ZSTD_btultra/g 2017-03-09 11:44:25 -08:00
Yann Collet 8b1d004031 added -Wformat-security flag, as recommended by @pixelb 2017-03-05 21:17:32 -08:00
Yann Collet 78208bd8be fixed : build zstd cli after libzstd 2017-03-01 21:02:06 -08:00
Yann Collet c1c040eae1 added gzip tests
also : made sure zstd --format=gzip -V
would fail if gzip compatibility is not supported
2017-03-01 16:49:20 -08:00
Przemyslaw Skibinski 8e5032a965 cli : fix : --rm is silent when input is stdin (decompression) 2017-02-28 09:42:37 +01:00
Yann Collet a81c2e7e44 Merge pull request #573 from facebook/ddict
Improved DDict memory usage
2017-02-27 20:54:42 -08:00
Yann Collet ef569bf75f Merge branch 'dev' of github.com:facebook/zstd into dev 2017-02-27 15:58:38 -08:00
Yann Collet dccd6b6f65 cli : fix : --rm is silent when input is stdin
previously, app would produce an error message, and stop.
2017-02-27 15:57:50 -08:00
Przemyslaw Skibinski 862698f479 minor tweaks in FIO_decompressGzFrame 2017-02-27 13:21:05 +01:00
Yann Collet 0b9b894b2d reduced ZSTD_DDict memory usage
saved 128 KB
2017-02-27 00:27:30 -08:00
Przemyslaw Skibinski 337ec875b6 minor tweaks 2017-02-22 10:31:30 +01:00
Przemyslaw Skibinski 684858e7b7 fix memory leaks 2017-02-21 18:17:24 +01:00
Przemyslaw Skibinski 74dcd8d15f bench.c: use a single ticksPerSecond 2017-02-21 12:22:05 +01:00
Przemyslaw Skibinski 1b59333c82 util.h: restore times() 2017-02-21 07:33:45 +01:00
Przemyslaw Skibinski da4a0f30af util.h: use mach_absolute_time for macOS 2017-02-20 12:18:15 +01:00
Przemyslaw Skibinski e052c60540 introduce UTIL_freq_t 2017-02-20 11:27:11 +01:00
Przemyslaw Skibinski 83775d9e05 replace times() with clock_gettime(CLOCK_MONOTONIC, x) 2017-02-20 11:11:50 +01:00
Przemyslaw Skibinski acb6e57ad2 use FindFirstFileA instead of FindFirstFile 2017-02-15 17:13:35 +01:00
Przemyslaw Skibinski 6e59b3ce01 added UTIL_fseek 2017-02-15 17:03:16 +01:00
Przemyslaw Skibinski 3aaa1dae4e simplified zlib detection 2017-02-15 09:17:39 +01:00
Yann Collet f0b9a8dddb Merge pull request #547 from inikep/dev11
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
2017-02-14 12:29:00 -08:00
Yann Collet c09d16ba8c preset behavior for gzip, gunzip and gzcat
when zstd is called through a link named gzip, gunzip or gzcat,
provides the same behavior as the related program.
gzip compresses using --format=gz
both gzip and gunzip enable --rm by default
2017-02-14 10:45:19 -08:00
Przemyslaw Skibinski ce13d087d9 fix LONG_SEEK 2017-02-14 09:52:52 +01:00
Przemyslaw Skibinski 970419535f fixed function name (2) 2017-02-14 09:47:29 +01:00
Przemyslaw Skibinski bf336572bf Avoid fseek()'s 2GiB barrier with MacOS and *BSD 2017-02-14 09:45:33 +01:00
Przemyslaw Skibinski abd6302423 Windows resources updated to v1.1.4 2017-02-14 09:39:09 +01:00
Przemyslaw Skibinski 442c75f132 removed UTIL_doesFileExists (replaced with UTIL_isRegFile) 2017-02-14 09:38:51 +01:00
Przemyslaw Skibinski b876b96ce1 Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-14 09:26:03 +01:00
Przemyslaw Skibinski 98509a70ac fixed function name 2017-02-14 09:23:32 +01:00
Przemyslaw Skibinski 1a195b3b7a fixed unitialized variable warning 2017-02-13 22:56:31 +01:00
Przemyslaw Skibinski 48466b36e8 Resolve conflict with branch 'refs/remotes/facebook/dev'
# Conflicts:
#	programs/fileio.c
2017-02-13 21:35:39 +01:00
Yann Collet db2666c10c Merge pull request #536 from iburinoc/multiframe
Simple API multiframe decoding
2017-02-13 12:18:16 -08:00
Przemyslaw Skibinski 64f7221958 limit zlib compression level to Z_BEST_COMPRESSION 2017-02-13 21:00:41 +01:00
Przemyslaw Skibinski 5a3bb05bb2 Merge remote-tracking branch 'refs/remotes/facebook/dev' into gz_compress
# Conflicts:
#	programs/Makefile
2017-02-13 20:47:01 +01:00
Przemyslaw Skibinski 35bf23c086 MinGW-w64 requires _FILE_OFFSET_BITS 64 2017-02-13 13:57:29 +01:00
ds77 6220bfc924 fix indentation in previous commit 2017-02-13 12:00:59 +01:00
ds77 168d9b8006 fix seeking 2GB+ files under Windows
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes  fallback workaround using Win32 API).
2017-02-12 16:05:19 +01:00
Yann Collet acfa151622 Merge pull request #540 from ds77/dev-stat64-fix
zstdcli: Fix reporting incorrect sizes of large flies on MinGW
2017-02-11 21:08:08 -08:00
Przemyslaw Skibinski eb132530cd revert last commit 2017-02-10 21:15:49 +01:00
Przemyslaw Skibinski 645f5b9856 fix for original MinGW 2017-02-10 20:09:28 +01:00
Sean Purcell 5069b6c2c3 Merge branch 'dev' into multiframe 2017-02-10 10:08:55 -08:00
ds77 45f0c207ab use _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces previously preprocessor checks. 2017-02-10 18:37:57 +01:00
Przemyslaw Skibinski 456330948a Merge remote-tracking branch 'refs/remotes/facebook/dev' into mingw
# Conflicts:
#	programs/platform.h
2017-02-10 13:38:54 +01:00
Yann Collet bbba42acd1 Merge pull request #537 from terrelln/small-bugs
Fix small bugs
2017-02-10 04:35:43 -08:00
- 7ec315df0d fix previous commit
* struct _stat64 is not defined by (non-w64) MinGW releases, __stat64 should be everywhere
* proper detection of _stat64() availability (as in MinGW sys/stat.h)
2017-02-10 13:27:43 +01:00
- 19f61b534e use _stat64 only when targetting Win2k or later 2017-02-10 10:56:45 +01:00
Przemyslaw Skibinski 429e13099a fix 64-bit file support for MinGW 2017-02-10 10:36:44 +01:00
Nick Terrell d08019813b Improvement from @inikep 2017-02-09 14:20:52 -08:00
ds77 9cde3f8b2e use _stat64() on MinGW
On MinGW, use _stat64() and struct _stat64 instead of stat() and struct stat_t. This fixes reporting incorrect sizes for large files.
2017-02-09 22:49:31 +01:00
Przemyslaw Skibinski 896638a8a2 echo replaced with printf 2017-02-09 17:01:17 +01:00
Przemyslaw Skibinski 13127fd05b don't use "echo -e" 2017-02-09 11:32:21 +01:00
Nick Terrell a0f9006e5a #undef _POSIX_C_SOURCE if already defined 2017-02-08 17:25:01 -08:00
Sean Purcell 0f5c95af44 Disambiguate pledgedSrcSize == 0
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
2017-02-08 15:12:46 -08:00
Sean Purcell 4e709712e1 Decompressed size functions now handle multiframes and distinguish cases
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
2017-02-08 14:50:10 -08:00
Przemyslaw Skibinski 93901fe85c remove redundant "ifeq ($(HAVE_ZLIB), 1)" 2017-02-08 21:11:18 +01:00
Przemyslaw Skibinski 4f9eaa7bb3 fixed gcc warnings 2017-02-08 18:08:09 +01:00
Przemyslaw Skibinski cb56306a50 .gz suffix for gzip compressed files 2017-02-08 17:37:14 +01:00
Przemyslaw Skibinski 02018c83cf added FIO_compressGzFrame 2017-02-08 16:54:23 +01:00
Przemyslaw Skibinski cfd4dc299a add "--format=gzip" option 2017-02-08 15:17:55 +01:00
Przemyslaw Skibinski ca20edd960 fixed zlib detection with MinGW 2017-02-08 14:32:49 +01:00
Przemyslaw Skibinski cdf5a7bd9f Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-08 13:49:35 +01:00
Przemyslaw Skibinski 40580ff669 added description of "--priority=rt" 2017-02-08 13:49:06 +01:00
Przemyslaw Skibinski 0665a359aa "--rt-prio" renamed to "--priority=rt" 2017-02-07 20:12:59 +01:00
Przemyslaw Skibinski d05014c739 added the "--rt-prio" option 2017-02-07 16:48:01 +01:00
Przemyslaw Skibinski 94abd6a26c SET_REALTIME_PRIORITY 2017-02-07 16:36:19 +01:00
Nick Terrell 7e3fc73795 Ensure <zlib.h> can be included in HAVE_ZLIB test 2017-02-06 11:54:31 -08:00
Nick Terrell 2cb8ee8784 Change zlib include to be a system include 2017-02-06 11:32:13 -08:00
Yann Collet f6e9c4f1e1 Merge pull request #529 from facebook/gzstd_default
Gzstd default
2017-02-05 18:03:36 -08:00
cyan4973 613087c02b Silence zlib detection routine
When it fails, $(CC) sends error message into stderr
redirected to /dev/null
2017-02-04 23:36:12 -08:00
Nick Terrell b4016ff02f Add cover dictionary training to zstd.1
Tested with `make install && man zstd` and visual inspection.
2017-02-03 16:42:07 -08:00
Yann Collet 762ddeeb9e fixed zstdmt compilation under Windows minGW/MSYS2, by @inikep 2017-02-03 14:35:42 -08:00
Yann Collet 21eb80d485 remove zlib detection artefact
result of compilation test is sent to /dev/null
2017-02-03 14:34:41 -08:00
Yann Collet b02ac8d613 fixed pointer conversion warnings (C++)
in gz module
2017-02-03 08:43:06 -08:00
Yann Collet c2a4632789 release builds use less debug symbols and warnings
release build are triggered through either `make`,
or their specific target `make zstd-release` and `make lib-release`.
2017-02-02 20:54:41 -08:00
Yann Collet c3cba9d858 fixed silent conversion warnings in GZDECOMPRESS path 2017-02-02 17:12:50 -08:00
Yann Collet 030ac243a0 Changed Makefile to generate zstd with .gz support by default
.gz support is detected by a runtime test.
2017-02-02 16:49:34 -08:00
Yann Collet 3dc85bae66 minor : fixed zstd-frugal
fixed a minor unused variable warning when compiling zstd-frugal target
2017-02-02 16:19:09 -08:00
Yann Collet 48bed91606 Merge pull request #527 from facebook/zstdmt
zstdmt refinements
2017-01-31 16:36:46 -08:00
Yann Collet 8d8513fb64 fixed C constant restrictions 2017-01-30 14:37:08 -08:00
Yann Collet 3672d06d06 zstdmt : section size is set to be a minimum of overlapSize
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
2017-01-30 13:35:45 -08:00
Yann Collet 6ccd37c8d4 cli : added advanced parameter overlapLog
as a hidden (undocumented) parameter for now
2017-01-30 13:07:24 -08:00
Yann Collet 6be2337c26 added command --block-size=
for Multi-threading only.
alias : -B#
2017-01-30 11:17:26 -08:00
Yann Collet cc3d1bc262 Merge pull request #525 from terrelln/covermt
Multithreaded COVER dictionary training
2017-01-30 10:15:33 -08:00
Yann Collet 5d9b894e46 Fixed status display for zstdmt
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.

This patch disables the displaying of % compression
when multi-threading is enabled.

It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.

There is no modification to decompression side,
since decompression is only single-threaded for now.
2017-01-27 13:30:18 -08:00
Nick Terrell 2fe9126591 Add multithread support to COVER 2017-01-27 11:56:02 -08:00
Nick Terrell 9c018cc140 Add BUCK files for Nuclide support 2017-01-27 10:43:12 -08:00
Przemyslaw Skibinski eb2d23a90c improved #232 fix 2017-01-27 10:43:03 -08:00
Przemyslaw Skibinski 92a4dbf2e4 Fixed https://github.com/facebook/zstd/issues/232 2017-01-27 10:43:03 -08:00
Nick Terrell 83c387eb8e Fix zstdmt_compress.h include 2017-01-26 15:25:32 -08:00
Yann Collet 717c65d690 Merge pull request #519 from inikep/dev11
Dev11
2017-01-26 14:23:44 -08:00
Yann Collet 8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet bb0027405a fixed zstdmt corruption issue when enabling overlapped sections
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
Yann Collet 943cff9c37 fixed zstdmt cli freeze issue with large nb of threads
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski 5022a18d51 improved #232 fix 2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski 64fa2dbc5e Fixed https://github.com/facebook/zstd/issues/232 2017-01-25 13:02:33 +01:00
Yann Collet 512cbe8c10 zstdmt cli and API allow selection of section sizes
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.

It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()

The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
2017-01-24 17:08:53 -08:00
Yann Collet f8804d1014 convert tabs to space
joys of using multiple editors from multiple environments ...
2017-01-20 17:23:19 -08:00
Yann Collet d7e3cb58c5 Resolved merge conflict dev+zstdmt 2017-01-20 16:44:50 -08:00
cyan4973 5fba09fa41 updated util's time for Windows compatibility
Correctly measures time on Posix systems when running with
Multi-threading

Todo : check Windows measurement under multi-threading
2017-01-20 12:57:31 -08:00
Yann Collet 458c8a94b4 minor refactoring : cleaner MT integration within bench 2017-01-19 17:44:15 -08:00
Yann Collet 500014af49 zstd cli can now compress using multi-threading
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads()  (fileio)
2017-01-19 17:04:28 -08:00
Yann Collet 0f984d94c4 changed MT enabling macro to ZSTD_MULTITHREAD 2017-01-19 14:05:07 -08:00
Yann Collet f22adae984 fixed minor warning (unused variable) in fuzzer 2017-01-19 13:46:30 -08:00
Yann Collet 736788f8e8 added streaming fuzzer tests for MT API
Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize
which is possible when many flushes are issued
while there is not enough dst buffer to flush completed ones.
2017-01-19 12:15:29 -08:00
Przemyslaw Skibinski 8a0bc30a2d Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-01-17 13:02:29 +01:00
Przemyslaw Skibinski d72f4b6b7a added "Makefile is validated" 2017-01-17 12:40:06 +01:00
Yann Collet 5eb749e734 ZSTDMT_compress() creates a single frame
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()

As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().

This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
2017-01-11 18:21:25 +01:00
Nick Terrell c220d4c74d Use COVER_MEMMULT when training with COVER. 2017-01-09 16:49:04 -08:00
Yann Collet 56958500fc minor man page update 2017-01-09 19:47:09 +01:00
Nick Terrell 3a1fefcf00 Simplify COVER parameters 2017-01-02 17:51:38 -08:00
Nick Terrell df8415c502 Add COVER to the zstd cli 2017-01-02 14:43:08 -08:00
Yann Collet 6334b04d61 compile object files, for faster recompilation 2017-01-02 03:22:18 +01:00
Yann Collet 747452677d fixed cmake tests 2017-01-02 02:05:45 +01:00
Yann Collet cdb2763f4a new Makefile target zstdmt 2017-01-02 01:43:56 +01:00
Yann Collet f1cb55192c fixed linux warnings 2017-01-02 01:11:55 +01:00
Yann Collet 0ec6a95ba1 minor fixes 2017-01-02 00:49:42 +01:00
Yann Collet c6a6417458 bench correctly measures time for multi-threaded compression (posix only) 2016-12-31 03:31:26 +01:00
Yann Collet f765a375a5 Merge pull request #504 from terrelln/thread-pool
[zstdmt] Add thread pool
2016-12-30 15:31:49 +01:00
Yann Collet e70912c72b Changed : input divided into roughly equal parts.
Debug : can measure time waiting for mutexes to unlock.
2016-12-29 01:24:01 +01:00
Yann Collet 8d7432914f Merge pull request #503 from inikep/dev11
Dev11
2016-12-28 16:50:39 +01:00
Yann Collet ab7a579180 added -T command , to set nb of threads 2016-12-28 16:11:09 +01:00
Przemyslaw Skibinski 75f3a3a335 changed default PREFIX and MANDIR 2016-12-28 12:32:41 +01:00
Yann Collet 3d93f2fce7 first zstdmt sketch 2016-12-27 07:19:36 +01:00
Yann Collet c07d2e3a31 Merge pull request #499 from inikep/dev11
improved *BSD and Solaris compatibility
2016-12-23 21:32:03 +01:00
Przemyslaw Skibinski 63b0014b96 BSD: improved "make install" 2016-12-23 10:05:49 +01:00
Andrew Janke fce374a100 zstdless: add shebang and quote $@ 2016-12-22 17:40:10 -05:00
Przemyslaw Skibinski b999170311 Solaris: working "make -C lib install" 2016-12-22 20:14:37 +01:00
Przemyslaw Skibinski aab442133d Solaris: working "make -C programs install" 2016-12-22 19:26:01 +01:00
Yann Collet 8333106b8a Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-21 16:44:24 +01:00
Yann Collet 0d7e84899f Merge pull request #489 from inikep/v112
improved detection of POSIX
2016-12-21 16:42:46 +01:00
Yann Collet 9518744025 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-21 16:20:41 +01:00
Yann Collet 1f57c2ed32 added : ZSTD_createCDict_byReference() 2016-12-21 16:20:11 +01:00
Przemyslaw Skibinski 7a8a03c20d util.h: restore BSD license for Facebook Open-Source 2016-12-21 15:08:44 +01:00
Przemyslaw Skibinski 97a258d71d updated comments 2016-12-21 14:00:41 +01:00
Przemyslaw Skibinski e679741b18 _CRT_SECURE_NO_WARNINGS moved to util.h 2016-12-21 13:47:11 +01:00
Przemyslaw Skibinski 2f6ccee6af platform.h: removed Compiler Options 2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski 101df4f636 fixed Visual Studio compilation 2016-12-21 11:43:11 +01:00
Przemyslaw Skibinski a35b9448ac improved MinGW support 2016-12-21 11:18:45 +01:00
Przemyslaw Skibinski 5736db219e fix basic types redefinition 2016-12-21 09:26:00 +01:00
Przemyslaw Skibinski 20b089e53d simplified zstdcli.c 2016-12-21 09:19:15 +01:00
Przemyslaw Skibinski 16ae6563a2 executables use new util.h and platform.h 2016-12-21 09:06:14 +01:00
Przemyslaw Skibinski ead350bdc0 improved util.h and platform.h 2016-12-21 09:04:59 +01:00
Przemyslaw Skibinski 1c1db6b845 windres updated to v1.1.3 2016-12-20 11:21:26 +01:00