c9f21c8a71
gzip : only display inflateEnd error when there is no error before
2017-07-03 13:45:09 -07:00
6c35112d08
gzip decoding does no longer exit() on invalid input
2017-07-03 13:24:50 -07:00
e97ff3b54a
decoding malformed lz4 frames does no longer exit()
2017-07-03 11:27:29 -07:00
c34185df13
fileio : decoding malformed lzma frame does no longer exit()
...
makes it possible to continue decoding file list
2017-07-03 10:27:16 -07:00
1d39550471
Merge pull request #744 from terrelln/dev
...
[pzstd] Remove appveyor tests
2017-07-01 05:58:33 -07:00
56e3964d85
[man] Specify that strategies start at 1
2017-06-30 16:29:37 -07:00
811deaea6f
Merge pull request #736 from terrelln/cover-default-api
...
[zdict] Make COVER the default algorithm
2017-06-28 20:25:36 -07:00
5b7fd7c422
[zdict] Make COVER the default algorithm
2017-06-26 21:09:22 -07:00
d88a441406
[zstdcli] Document HAVE_LZMA and zstd-noxz
2017-06-26 11:25:00 -07:00
6aeb50ea0d
[zstdcli] Add unlzma and unxz symlinks
2017-06-26 11:24:36 -07:00
849ecf3510
[zstdcli] Support xz by default when liblzma is available
2017-06-23 17:11:38 -07:00
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
991f9dfcde
switched fileio.c to ZSTD_DEBUG trigger macro
2017-06-21 15:16:13 -07:00
5be0f5544e
added newline to align output
2017-06-21 12:41:10 -07:00
6f5fe71041
added error check for when file could not be opened
2017-06-21 12:37:23 -07:00
74a725da69
reversed calculation of ratio
2017-06-21 12:27:20 -07:00
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
a73c2a444a
added tests with null files, changed condition to check that the file is more than 0 bytes
2017-06-20 14:33:08 -07:00
b44ab82f7a
ensure new ZSTD_strategy starts at value 1
2017-06-20 14:11:49 -07:00
0765602903
fixed error where extremely small files were not being detected as not compressed with zstd
2017-06-20 14:04:46 -07:00
3701dde3ba
changed to use LONG_SEEK
2017-06-20 13:44:05 -07:00
93a5510962
added progress metric to display
2017-06-20 13:26:25 -07:00
bda57754a2
moved signifcant operations to fileio.c
2017-06-20 12:43:10 -07:00
c594507871
removed double if statements for same condition
2017-06-20 11:59:45 -07:00
72e0ac9929
minor update to logic
2017-06-20 11:58:27 -07:00
cea55a9217
size_t const value
2017-06-20 11:56:11 -07:00
bed0dc95bc
changed displayInfo alignment and const
2017-06-20 11:47:24 -07:00
cb2dbe6434
changed fileInfo initialization, updated error code
2017-06-20 09:35:21 -07:00
c08e649e95
first implementation of bench.c with new API ZSTD_compress_generic()
...
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.
Also : fixed a long standing bug in ZSTDMT streaming API.
2017-06-19 18:25:35 -07:00
fe234bf48b
fix attempts : fullbench for VS2008
2017-06-19 15:23:19 -07:00
55b9b28dad
made style changes, added description in man file
2017-06-19 09:55:55 -07:00
4a0e4c856c
fixing more spacing on loops
2017-06-15 18:08:22 -07:00
b3e62446c6
added in correct error return from main in zstdcli.c
2017-06-15 17:46:49 -07:00
0d7c4d766a
initialized info manually to avoid error
2017-06-15 17:30:18 -07:00
d3b34e4a25
changed spacing on if statements
2017-06-15 17:12:41 -07:00
b316691f91
added const
2017-06-15 17:03:59 -07:00
60a2e55e2e
added case for when file is not compressed with zstd (incorrect magic number)
2017-06-15 17:00:59 -07:00
acaefb531b
refactoring brackets to allow for const values
2017-06-15 16:27:38 -07:00
0757eae6ff
rearranging code to allow for consts
2017-06-15 16:12:04 -07:00
e49afae2ab
changed implicit precision loss again
2017-06-15 16:03:40 -07:00
54e0b6c66b
got rid of imlpicit precision loss
2017-06-15 16:01:52 -07:00
b766211e10
made another function static
2017-06-15 15:55:49 -07:00
e7f02fc58a
closed file at end of function, created another variable to keep exit points simpler
2017-06-15 15:53:02 -07:00
8b3ff7c9bc
changed info to static allocation
2017-06-15 15:35:37 -07:00
96cabf7233
cleaned up if statements
2017-06-15 15:25:20 -07:00
9437cc7498
added --list command
2017-06-15 15:07:54 -07:00
4495e9a826
allow analysis even if file does not have zst extension
2017-06-15 15:02:54 -07:00
a9b77c83e5
cleaning up code for analyzing frames
2017-06-15 14:13:28 -07:00
e208992529
cleaning up code
2017-06-15 12:27:32 -07:00
fc428ab350
changing formatting again
2017-06-14 13:26:19 -07:00
607ee9f4bc
print message when no files are given, rearrange code for readability
2017-06-14 13:23:36 -07:00
ebd60349f5
changed U64 to unsigned long to get rid of build test errors
2017-06-14 13:11:18 -07:00
81fa33b55a
cleaning up code
2017-06-13 10:06:49 -07:00
a56dcbfcf8
changed print format for non-verbose version
2017-06-12 17:47:33 -07:00
c828b52111
yet more refactoring
2017-06-12 17:19:53 -07:00
28dbf8ee6c
more refactoring for C90
2017-06-12 17:09:08 -07:00
618a7b66e2
refactoring for C90 syntax
2017-06-12 16:53:50 -07:00
6e33c74133
formatting issues resolved
2017-06-12 16:29:00 -07:00
173a739717
code refactorings with explicit conversion
2017-06-12 16:14:04 -07:00
f3d6ab28a2
formated information for verbose output
2017-06-12 15:59:28 -07:00
6e0204470a
displayed decompressed size
2017-06-12 15:51:59 -07:00
dcd6ba6dc6
incremented decompressedSize instead of setting value
2017-06-12 15:40:47 -07:00
6996bd2598
removed useless lines
2017-06-12 15:24:50 -07:00
9cb602ee25
added in logic for parsing through blocks/frames
2017-06-12 15:22:48 -07:00
786b7cac27
added code to analyze the first frame header
2017-06-12 13:46:39 -07:00
a3d54cf73d
added line spacing for clarity
2017-06-12 10:58:34 -07:00
bb0aaf9579
minor man update on -B# option in benchmark mode
2017-06-12 05:19:15 -07:00
9e6a2eaab6
added MT support to NEWAPI
2017-06-11 18:39:46 -07:00
0f06f4f266
added display for compressed size
2017-06-06 09:21:42 -07:00
4128f67a4a
added in check suffix check to ensure file was compressed with zstd
2017-06-05 15:00:06 -07:00
901435e9ef
setup basic functions for adding --list functionality
2017-06-05 14:45:31 -07:00
8bcbf42617
fixed g++ prototype mismatch
2017-06-04 23:52:00 -07:00
6d4fef36de
Added ZSTD_compress_generic()
...
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
2017-05-17 18:36:15 -07:00
23c256e44b
removed useless variable from CCtx
...
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
2017-05-16 18:10:11 -07:00
e12ae02357
minor coding style update
...
reduced some long lines
2017-05-16 17:32:33 -07:00
2d4d31c18a
removed gcc compilation flag -Wbad-function-cast
...
It makes it more difficult to directly cast the result of a function,
requiring to store the result in an intermediate variable.
It does not necessarily help readability,
and this restriction can be difficult to overcome in some constructions,
like some macros.
also : fixed minor Visual conversion warnings in datagencli.c
2017-05-16 11:34:38 -07:00
46ac9ade9b
fixed vla warning on linux
2017-05-15 18:15:08 -07:00
83d0c764dc
added several compilation flags
2017-05-15 17:15:46 -07:00
a99ece9e79
Update zstd.1.md to use btultra
2017-05-10 16:51:18 -07:00
5f2c7213c7
Merge remote-tracking branch 'upstream/dev' into btopt
...
* upstream/dev: (305 commits)
added test for ZSTD_estimateCStreamSize()
changed variable name, for clarity
fixed ZSTD_estimateCStreamSize()
shortened ZSTD_createCStream_Advanced()
fixed symbols test
added ZSTD_estimateDStreamSize()
changed name frameParams into frameHeader
regroup memory usage function declarations
separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
bumped version number
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
Updated ZSTD_freeCCtx()
updated ZSTD_estimateCCtxSize()
Updated ZSTD_sizeof_CCtx()
merged CCtx and CStream as a single same object
cli : -d and -t do not stop after a failed decompression
added dev branch CircleCI badge
added dev branch Appveyor badge
keep dev branch status only
creates a binary archive without the `programs` directory
...
2017-05-10 16:49:58 -07:00
01a1abfdb5
cli : -d and -t do not stop after a failed decompression
...
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
2017-05-05 19:15:24 -07:00
a00e9599f1
removed -g from DEBUGFLAGS
...
It inflates binary sizes, which is negative for the Windows build.
It also makes it impossible to check if 2 different source codes
get nonetheless compiled to the same binary,
since checksum will be different, due to integrated source code.
2017-05-04 17:24:29 -07:00
c08e56861e
updated dict graphs to 2D mode
2017-05-03 14:33:28 -07:00
710497d8ea
updated programs/README.md, to introduce compilation variables
...
make it possible to enable/disable features individually
2017-05-02 17:18:24 -07:00
f47284fec0
reorganized Makefile for multiple targets
2017-05-02 16:55:57 -07:00
3791d2105d
added xzstd4 target
...
support for all formats, xz/lzma/lz4 included
2017-05-02 16:38:37 -07:00
2e63a877f3
fixed xzstd
...
--format=xz was missing a break, making the execution continue into lz4 error message
2017-05-02 15:40:42 -07:00
f376d47c11
[CLI] Switch dictionary builder on CLI to cover
2017-05-02 11:18:27 -07:00
020b960e13
[cover] Make optimization faster
2017-05-02 11:02:48 -07:00
f2d9ef1dc0
[cover] Optimize case where d <= 8
2017-05-02 11:02:43 -07:00
feba969a69
Fix LZ4_MSG in xzstd
2017-05-02 11:02:16 -07:00
31533bacce
Changed ZSTD_createCDict_advanced()
...
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.
Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
eab41c1872
Fix LZ4 wrapper deprecation warnings
2017-04-26 10:17:38 -07:00
2c4b6fe6b3
Make lz4 compression/decompression compatible with library r123
2017-04-25 11:00:54 -07:00
4de8632957
Add LZ4 compress/decompress support to CLI
2017-04-24 16:48:25 -07:00
230d7acc7d
cli : add support for --threads=# command
...
updated documentation
add relevant test case
2017-04-21 11:38:13 -07:00
9606256a8d
Fix no thread message
2017-04-18 13:52:00 -07:00
1c286950af
Merge pull request #663 from iburinoc/zstdmt-default
...
Compile CLI using files instead of objs
2017-04-18 02:59:54 -07:00
c8b2df7d62
Compile CLI using files instead of objs
...
This avoids conflicts between how the library was
configured and how the CLI was configured.
2017-04-17 17:13:47 -07:00
e6c504dbe6
Update -T0 comment in man page
2017-04-17 17:12:02 -07:00
5935c990a0
Add zstdmt and -T0 to man page
2017-04-17 16:05:20 -07:00
f6ef4db20e
Install zstdmt even without threading support
2017-04-17 12:21:11 -07:00
5a61f36474
Make zstd compile with mt by default
2017-04-17 12:06:02 -07:00
7dd14d03b0
Enable multithreading on BSD
2017-04-15 16:25:08 +02:00
e4f3235c85
Add 0 initializers to static variables
2017-04-13 16:34:28 -07:00
42bac7fa84
Change ifndef's to undef's
2017-04-13 15:35:05 -07:00
9227aae001
Fix clang linux compilation
2017-04-13 14:06:40 -07:00
3b6207d4bd
Fix compilation on windows
2017-04-13 14:04:10 -07:00
f876f1200c
Fix compilation on macOS
2017-04-13 12:33:45 -07:00
afa48518e2
-T0 detects number of physical cores
2017-04-13 12:28:28 -07:00
5c42d0edc8
cli : better status display for zstdmt in 1-thread mode
2017-04-11 16:57:32 -07:00
0e30059ba1
cli : FIO_createDictBuffer() replaces FIO_loadFile()
...
makes it more explicit that it allocates a buffer
and that it's meant to be used for dictionary.
Also : simplified function a bit,
now only works for dictionaries up to DICTSIZE_MAX
2017-04-11 14:41:02 -07:00
279be2015b
Let zstd overwrite read-only files
2017-04-06 12:56:40 -07:00
c2007388a5
fixed bench.c : optional advanced parameters applied
...
before creating cdict
2017-04-04 15:35:06 -07:00
81d6380139
minor bench.c adjustments
...
shorter debug messages
no need to check decompressedLength==0 twice
2017-04-04 15:21:09 -07:00
a36330a27d
minor man update, for `overlapSize`
2017-04-03 14:57:24 -07:00
805c5a3efb
updated documentation on multithreading modes
2017-04-01 00:36:31 -07:00
eb38617175
Clean up default sparse logic
2017-03-31 15:20:50 -07:00
96fe545a18
Turn off sparse mode for OS X by default
2017-03-31 15:16:43 -07:00
5152fb2cb2
Convert all tabs to spaces
2017-03-29 18:51:58 -07:00
5bde4be5a1
fix : bench automatically adapts parameters to srcSize
2017-03-29 12:10:38 -07:00
efdaf8bb7c
Fix inline compile errors
2017-03-27 12:26:40 -07:00
894bf49713
Fix IS_CONSOLE returning 1 for NUL on windows
2017-03-27 12:19:30 -07:00
0246d5c531
Merge pull request #630 from facebook/advancedCliCommands
...
changed advanced commands --maxdict= and --dictID=
2017-03-26 00:13:35 -07:00
ef30af843f
Ignore extension in command name matching
2017-03-24 17:06:09 -07:00
96aa3019b2
changed advanced commands --maxdict= and --dictID=
...
now works with the `=` variant, which is the recommended one.
Old variant `--dictID #` still works, for compatibility with existing scripts.
Long term objective is to remove the old variant..
2017-03-24 16:04:29 -07:00
f332ece468
dictBuilder fails to create dictionary on certain input
...
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
2017-03-23 16:24:02 -07:00
dee08caabb
Fix displayLevel redefinition issues
2017-03-23 12:09:35 -07:00
680e4e0953
Ignore symbolic links unless --force specified
2017-03-23 11:52:09 -07:00
042ba122ae
Change g_displayLevel to int and fix DISPLAYUPDATE flush
2017-03-23 11:21:59 -07:00
dfc75f74da
Add $(EXT) to binary to fix zlib detection for some MinGW versions
2017-03-22 13:24:52 -07:00
741e3e8314
Add $(FLAGS) when testing for zlib/lzma support.
...
* If zlib/lzma isn't in the usual spot, it won't be used,
even if `$CFLAGS` and `$LDFLAGS` add the location it is in.
* Update the test code snippets to not trigger any warnings.
2017-03-22 12:33:08 -07:00
35c147ab6a
Merge pull request #605 from iburinoc/term
...
Fix zstd not erroring on compressing to terminal without forceStdout
2017-03-17 14:02:48 -07:00
0a189b63fe
fix minor details in man page
2017-03-16 16:33:53 -07:00
b8e52d3c83
Fix zstd not erroring on compressing to terminal without forceStdout
2017-03-16 16:06:19 -07:00
72a243e3bd
Convert manpage to Markdown, converted with ronn
2017-03-16 15:37:32 -07:00
3773bd318f
Merge pull request #601 from terrelln/nxz
...
Add xz and lzma support.
2017-03-13 22:00:03 -07:00
aa8bcf360f
Add xz and lzma support.
...
Finish feature started by @inikep.
* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
2017-03-13 18:45:41 -07:00
120df494e9
Update builds to not support legacy v01-v03
2017-03-13 14:44:08 -07:00
334cb34edb
ZSTD_LEGACY_SUPPORT defines lowest supported version
2017-03-13 14:32:30 -07:00
201e8c8157
programs/Makefile: remove zstd-internal target
...
zstd-internal was intended to be a helper target, but it doesn't help
at all, what it does in practice is a useless rebuild of zstd every time
"make zstd" is invoked.
Fixes: 030ac243a0
2017-03-09 22:47:14 +00:00
eeb31eed39
s/ZSTD_btopt2/ZSTD_btultra/g
2017-03-09 11:44:25 -08:00
8b1d004031
added -Wformat-security flag, as recommended by @pixelb
2017-03-05 21:17:32 -08:00
78208bd8be
fixed : build zstd cli after libzstd
2017-03-01 21:02:06 -08:00
c1c040eae1
added gzip tests
...
also : made sure zstd --format=gzip -V
would fail if gzip compatibility is not supported
2017-03-01 16:49:20 -08:00
8e5032a965
cli : fix : --rm is silent when input is stdin (decompression)
2017-02-28 09:42:37 +01:00
a81c2e7e44
Merge pull request #573 from facebook/ddict
...
Improved DDict memory usage
2017-02-27 20:54:42 -08:00
ef569bf75f
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-02-27 15:58:38 -08:00
dccd6b6f65
cli : fix : --rm is silent when input is stdin
...
previously, app would produce an error message, and stop.
2017-02-27 15:57:50 -08:00
862698f479
minor tweaks in FIO_decompressGzFrame
2017-02-27 13:21:05 +01:00
0b9b894b2d
reduced ZSTD_DDict memory usage
...
saved 128 KB
2017-02-27 00:27:30 -08:00
337ec875b6
minor tweaks
2017-02-22 10:31:30 +01:00
684858e7b7
fix memory leaks
2017-02-21 18:17:24 +01:00
74dcd8d15f
bench.c: use a single ticksPerSecond
2017-02-21 12:22:05 +01:00
1b59333c82
util.h: restore times()
2017-02-21 07:33:45 +01:00
da4a0f30af
util.h: use mach_absolute_time for macOS
2017-02-20 12:18:15 +01:00
e052c60540
introduce UTIL_freq_t
2017-02-20 11:27:11 +01:00
83775d9e05
replace times() with clock_gettime(CLOCK_MONOTONIC, x)
2017-02-20 11:11:50 +01:00
acb6e57ad2
use FindFirstFileA instead of FindFirstFile
2017-02-15 17:13:35 +01:00
6e59b3ce01
added UTIL_fseek
2017-02-15 17:03:16 +01:00
3aaa1dae4e
simplified zlib detection
2017-02-15 09:17:39 +01:00
f0b9a8dddb
Merge pull request #547 from inikep/dev11
...
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
2017-02-14 12:29:00 -08:00
c09d16ba8c
preset behavior for gzip, gunzip and gzcat
...
when zstd is called through a link named gzip, gunzip or gzcat,
provides the same behavior as the related program.
gzip compresses using --format=gz
both gzip and gunzip enable --rm by default
2017-02-14 10:45:19 -08:00
ce13d087d9
fix LONG_SEEK
2017-02-14 09:52:52 +01:00
970419535f
fixed function name (2)
2017-02-14 09:47:29 +01:00
bf336572bf
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
2017-02-14 09:45:33 +01:00
abd6302423
Windows resources updated to v1.1.4
2017-02-14 09:39:09 +01:00
442c75f132
removed UTIL_doesFileExists (replaced with UTIL_isRegFile)
2017-02-14 09:38:51 +01:00
b876b96ce1
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2017-02-14 09:26:03 +01:00
98509a70ac
fixed function name
2017-02-14 09:23:32 +01:00
1a195b3b7a
fixed unitialized variable warning
2017-02-13 22:56:31 +01:00
48466b36e8
Resolve conflict with branch 'refs/remotes/facebook/dev'
...
# Conflicts:
# programs/fileio.c
2017-02-13 21:35:39 +01:00
db2666c10c
Merge pull request #536 from iburinoc/multiframe
...
Simple API multiframe decoding
2017-02-13 12:18:16 -08:00
64f7221958
limit zlib compression level to Z_BEST_COMPRESSION
2017-02-13 21:00:41 +01:00
5a3bb05bb2
Merge remote-tracking branch 'refs/remotes/facebook/dev' into gz_compress
...
# Conflicts:
# programs/Makefile
2017-02-13 20:47:01 +01:00
35bf23c086
MinGW-w64 requires _FILE_OFFSET_BITS 64
2017-02-13 13:57:29 +01:00
6220bfc924
fix indentation in previous commit
2017-02-13 12:00:59 +01:00
168d9b8006
fix seeking 2GB+ files under Windows
...
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes fallback workaround using Win32 API).
2017-02-12 16:05:19 +01:00
acfa151622
Merge pull request #540 from ds77/dev-stat64-fix
...
zstdcli: Fix reporting incorrect sizes of large flies on MinGW
2017-02-11 21:08:08 -08:00
eb132530cd
revert last commit
2017-02-10 21:15:49 +01:00
645f5b9856
fix for original MinGW
2017-02-10 20:09:28 +01:00
5069b6c2c3
Merge branch 'dev' into multiframe
2017-02-10 10:08:55 -08:00
45f0c207ab
use _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces previously preprocessor checks.
2017-02-10 18:37:57 +01:00
456330948a
Merge remote-tracking branch 'refs/remotes/facebook/dev' into mingw
...
# Conflicts:
# programs/platform.h
2017-02-10 13:38:54 +01:00
bbba42acd1
Merge pull request #537 from terrelln/small-bugs
...
Fix small bugs
2017-02-10 04:35:43 -08:00
7ec315df0d
fix previous commit
...
* struct _stat64 is not defined by (non-w64) MinGW releases, __stat64 should be everywhere
* proper detection of _stat64() availability (as in MinGW sys/stat.h)
2017-02-10 13:27:43 +01:00
19f61b534e
use _stat64 only when targetting Win2k or later
2017-02-10 10:56:45 +01:00
429e13099a
fix 64-bit file support for MinGW
2017-02-10 10:36:44 +01:00
d08019813b
Improvement from @inikep
2017-02-09 14:20:52 -08:00
9cde3f8b2e
use _stat64() on MinGW
...
On MinGW, use _stat64() and struct _stat64 instead of stat() and struct stat_t. This fixes reporting incorrect sizes for large files.
2017-02-09 22:49:31 +01:00
896638a8a2
echo replaced with printf
2017-02-09 17:01:17 +01:00
13127fd05b
don't use "echo -e"
2017-02-09 11:32:21 +01:00
a0f9006e5a
#undef _POSIX_C_SOURCE if already defined
2017-02-08 17:25:01 -08:00
0f5c95af44
Disambiguate pledgedSrcSize == 0
...
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
2017-02-08 15:12:46 -08:00
4e709712e1
Decompressed size functions now handle multiframes and distinguish cases
...
- Add ZSTD_findDecompressedSize
- Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
- Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
2017-02-08 14:50:10 -08:00
93901fe85c
remove redundant "ifeq ($(HAVE_ZLIB), 1)"
2017-02-08 21:11:18 +01:00
4f9eaa7bb3
fixed gcc warnings
2017-02-08 18:08:09 +01:00
cb56306a50
.gz suffix for gzip compressed files
2017-02-08 17:37:14 +01:00
02018c83cf
added FIO_compressGzFrame
2017-02-08 16:54:23 +01:00
cfd4dc299a
add "--format=gzip" option
2017-02-08 15:17:55 +01:00
ca20edd960
fixed zlib detection with MinGW
2017-02-08 14:32:49 +01:00
cdf5a7bd9f
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2017-02-08 13:49:35 +01:00
40580ff669
added description of "--priority=rt"
2017-02-08 13:49:06 +01:00
0665a359aa
"--rt-prio" renamed to "--priority=rt"
2017-02-07 20:12:59 +01:00
d05014c739
added the "--rt-prio" option
2017-02-07 16:48:01 +01:00
94abd6a26c
SET_REALTIME_PRIORITY
2017-02-07 16:36:19 +01:00
7e3fc73795
Ensure <zlib.h> can be included in HAVE_ZLIB test
2017-02-06 11:54:31 -08:00
2cb8ee8784
Change zlib include to be a system include
2017-02-06 11:32:13 -08:00
f6e9c4f1e1
Merge pull request #529 from facebook/gzstd_default
...
Gzstd default
2017-02-05 18:03:36 -08:00
613087c02b
Silence zlib detection routine
...
When it fails, $(CC) sends error message into stderr
redirected to /dev/null
2017-02-04 23:36:12 -08:00
b4016ff02f
Add cover dictionary training to zstd.1
...
Tested with `make install && man zstd` and visual inspection.
2017-02-03 16:42:07 -08:00
762ddeeb9e
fixed zstdmt compilation under Windows minGW/MSYS2, by @inikep
2017-02-03 14:35:42 -08:00
21eb80d485
remove zlib detection artefact
...
result of compilation test is sent to /dev/null
2017-02-03 14:34:41 -08:00
b02ac8d613
fixed pointer conversion warnings (C++)
...
in gz module
2017-02-03 08:43:06 -08:00
c2a4632789
release builds use less debug symbols and warnings
...
release build are triggered through either `make`,
or their specific target `make zstd-release` and `make lib-release`.
2017-02-02 20:54:41 -08:00
c3cba9d858
fixed silent conversion warnings in GZDECOMPRESS path
2017-02-02 17:12:50 -08:00
030ac243a0
Changed Makefile to generate zstd with .gz support by default
...
.gz support is detected by a runtime test.
2017-02-02 16:49:34 -08:00
3dc85bae66
minor : fixed zstd-frugal
...
fixed a minor unused variable warning when compiling zstd-frugal target
2017-02-02 16:19:09 -08:00
48bed91606
Merge pull request #527 from facebook/zstdmt
...
zstdmt refinements
2017-01-31 16:36:46 -08:00
8d8513fb64
fixed C constant restrictions
2017-01-30 14:37:08 -08:00
3672d06d06
zstdmt : section size is set to be a minimum of overlapSize
...
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
2017-01-30 13:35:45 -08:00
6ccd37c8d4
cli : added advanced parameter overlapLog
...
as a hidden (undocumented) parameter for now
2017-01-30 13:07:24 -08:00
6be2337c26
added command --block-size=
...
for Multi-threading only.
alias : -B#
2017-01-30 11:17:26 -08:00
cc3d1bc262
Merge pull request #525 from terrelln/covermt
...
Multithreaded COVER dictionary training
2017-01-30 10:15:33 -08:00
5d9b894e46
Fixed status display for zstdmt
...
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.
This patch disables the displaying of % compression
when multi-threading is enabled.
It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.
There is no modification to decompression side,
since decompression is only single-threaded for now.
2017-01-27 13:30:18 -08:00
2fe9126591
Add multithread support to COVER
2017-01-27 11:56:02 -08:00
9c018cc140
Add BUCK files for Nuclide support
2017-01-27 10:43:12 -08:00
eb2d23a90c
improved #232 fix
2017-01-27 10:43:03 -08:00
92a4dbf2e4
Fixed https://github.com/facebook/zstd/issues/232
2017-01-27 10:43:03 -08:00
83c387eb8e
Fix zstdmt_compress.h include
2017-01-26 15:25:32 -08:00
717c65d690
Merge pull request #519 from inikep/dev11
...
Dev11
2017-01-26 14:23:44 -08:00
8dafb1acf5
CLI : automatically set overlap size to max (windowSize) for max compression level
2017-01-25 17:01:13 -08:00
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
943cff9c37
fixed zstdmt cli freeze issue with large nb of threads
...
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
2017-01-25 12:35:19 -08:00
5022a18d51
improved #232 fix
2017-01-25 13:11:26 +01:00
64fa2dbc5e
Fixed https://github.com/facebook/zstd/issues/232
2017-01-25 13:02:33 +01:00
512cbe8c10
zstdmt cli and API allow selection of section sizes
...
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.
It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()
The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
2017-01-24 17:08:53 -08:00
f8804d1014
convert tabs to space
...
joys of using multiple editors from multiple environments ...
2017-01-20 17:23:19 -08:00
d7e3cb58c5
Resolved merge conflict dev+zstdmt
2017-01-20 16:44:50 -08:00
5fba09fa41
updated util's time for Windows compatibility
...
Correctly measures time on Posix systems when running with
Multi-threading
Todo : check Windows measurement under multi-threading
2017-01-20 12:57:31 -08:00
458c8a94b4
minor refactoring : cleaner MT integration within bench
2017-01-19 17:44:15 -08:00
500014af49
zstd cli can now compress using multi-threading
...
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads() (fileio)
2017-01-19 17:04:28 -08:00
0f984d94c4
changed MT enabling macro to ZSTD_MULTITHREAD
2017-01-19 14:05:07 -08:00
f22adae984
fixed minor warning (unused variable) in fuzzer
2017-01-19 13:46:30 -08:00
736788f8e8
added streaming fuzzer tests for MT API
...
Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize
which is possible when many flushes are issued
while there is not enough dst buffer to flush completed ones.
2017-01-19 12:15:29 -08:00
8a0bc30a2d
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2017-01-17 13:02:29 +01:00
d72f4b6b7a
added "Makefile is validated"
2017-01-17 12:40:06 +01:00
5eb749e734
ZSTDMT_compress() creates a single frame
...
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()
As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().
This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
2017-01-11 18:21:25 +01:00
c220d4c74d
Use COVER_MEMMULT when training with COVER.
2017-01-09 16:49:04 -08:00
56958500fc
minor man page update
2017-01-09 19:47:09 +01:00
Yann Collet
Yann Collet
Nick Terrell
Paul Cruz
cyan4973
Sean Purcell
Baptiste Daroussin
Dmitry V. Levin
Przemyslaw Skibinski
ds77
-
ds77