dd18d73e7e
fileio: content size is enabled by default
2017-10-13 16:32:18 -07:00
6dd958eea2
[zstdcli] Add window size to verbose list
...
```
> zstd --list -v file1 file2 file3
*** zstd command line interface 64-bits v1.3.2, by Yann Collet ***
Window Size: 512.00 KB (524288 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64
Window Size: 8192.00 KB (8388608 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64
Window Size: 512.00 KB (524288 B)
Compressed Size: 0.01 KB (15 B)
Check: None
```
2017-10-04 12:26:28 -07:00
3b27ed41fd
Merge branch 'srcSize' into dev
2017-10-02 16:34:14 -07:00
4946993f87
removed isRegularFile parameter
...
no longer useful : size of src is determined for each file.
2017-10-02 12:29:25 -07:00
7f580f9ee8
interruption handler and variable are static
2017-10-02 11:39:05 -07:00
fe5444bc66
removed the statement for all versions of Visual Studio
2017-10-02 02:02:16 -07:00
51d82d5516
same error in Visual Studio 2012 ...
2017-10-02 01:12:40 -07:00
ed7ae4c9bd
The issue also impacts Visual Studio 2010
2017-10-02 00:45:28 -07:00
6e7ba3df2f
added (void)sig to avoid compilers complaining that sig is not used.
2017-10-02 00:19:47 -07:00
82bc200f82
conditionnally removed invocation that generates a buggy warning with Visual Studio 2008
2017-10-02 00:02:24 -07:00
bd18095edc
blindfix for Visual : minor casting issue
...
should not happen since SIGIGN is provided by <signal.h>,
so it should work "ouf of the box"
2017-10-01 15:32:48 -07:00
00fc1ba8dd
cli: add Ctrl-C support, requested by @mike155 in #854
...
Now, pressing Ctrl-C during compression or decompression
will erase operation artefact (unfinished destination file)
before leaving execution.
2017-10-01 12:10:26 -07:00
e580dc6a4a
Merge pull request #860 from felixhandte/zstd-lz4-support-tests
...
Add Default LZ4 Support When Available
2017-09-29 22:32:54 -07:00
8afb151c9b
cli: fixed wrong initialization in MT mode
...
It's not good to mix old and new API
ZSTD_resetCStream() doesn't just set pledgedSrcSize :
it also sets the CCtx for a single thread compression.
Problem is, when 2+ threads are defined in cctx->requestedParams,
ZSTD_compress_generic() will want to start MT compression,
since initialization is supposed to have already happened (thanks to ZSTD_resetCStream())
except that the underlying ZSTDMT_CCtx* object is not created,
resulting in a segfault.
This is an invalid construction
(correct one is to use ZSTD_CCtx_setPledgedSrcSize()).
I haven't found a nice way to mitigate this impact if someone makes the same mistake.
At some point, removing the old API to keep only the new API within fileio.c will limit these risks.
2017-09-29 22:14:37 -07:00
fbd5ab7027
minor fix : no longer use fake srcSize during resource creation
...
srcSize is read and provided at each file, not at resource creation.
This used to be useful with older API, because it could not re-adapt parameters between sessions.
At some point, it will be better to remove the old code, and only keep the new_api.
It works fine by now.
2017-09-29 19:40:27 -07:00
db1668a43b
fix : srcSize written in frame header when multiple files compressed
...
This information used to be disabled when nbFiles>1.
It was badly initialized later in the code, resulting in an error.
2017-09-29 18:05:18 -07:00
8afcc80e07
decode more data before triggering error
...
fixes #874 :
when a frame is not properly terminated by a "last block" signal,
zstd -d used to detect it immediately and error out.
This version will decode and flush the last block, and only then issue an error.
2017-09-29 15:54:09 -07:00
54a827fff0
Merge branch 'dev' into newFormats
...
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
3182ea2e64
Merge pull request #866 from facebook/list
...
improved --list display
2017-09-27 16:34:29 -07:00
c233bdbaee
Increase maximum window size
...
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
`--zstd=wlog=#`. These options also set the window size during
decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
and decompression.
2017-09-26 14:00:01 -07:00
3095ca8c56
fixed minor conversion warnings for g++ on Linux
...
U64 is not considered equivalent to unsigned long long
2017-09-26 13:53:50 -07:00
56f1f0e3dd
write summary for --list on multiple files
2017-09-26 11:21:36 -07:00
62568c9a42
added capability to generate magic-less frames
...
decoder not implemented yet
2017-09-25 14:26:26 -07:00
360238733a
Adds LZ4 support by default if LZ4 is available
...
Simple makefile change + quick typename change
Test:
make clean
make
# successfully produces binary without lz4 support
make clean
# with flags to pick up my lz4 build
make MOREFLAGS="-L/home/felixh/prog/lz4/lib -I/home/felixh/prog/lz4/lib"
# successfully produces binary with lz4 support
echo "TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU" | \
./lz4/lz4 | \
LD_LIBRARY_PATH=/home/felixh/prog/lz4/lib ./zstd/zstd -d
# successfully prints TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU
2017-09-22 13:28:56 -07:00
92889709f9
fix #851 : sudo zstd -t file.zst changes /dev/null permissions
...
reported by @mike155
2017-09-18 13:41:54 -07:00
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
67d4a6161c
Add ldmBucketSizeLog param
2017-09-02 21:55:29 -07:00
a1f04d518d
Move hashEveryLog to cctxParams and update cli
2017-09-01 15:05:47 -07:00
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
6a546efb8c
Add long distance matcher
...
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
32e943b3ef
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-08-19 00:36:37 -07:00
582e19b01c
Merge pull request #801 from facebook/newLicense
...
New license
2017-08-19 00:34:34 -07:00
166645e7b3
fixed zstd-compress
...
file-information is dependent on decompression functions.
it should only be enabled when ZSTD_NODECOMPRESS is not set.
also : added zstd-compress compilation test into `make shortest`
2017-08-18 18:30:41 -07:00
f207b39f55
blindfix for Windows conversion warning
...
long type is 32-bits on Windows 64,
while it's 64-bits on Unix.
64-to-32 shortening conversion for long is a specific Windows issue.
2017-08-18 17:06:12 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
88d2f72df9
fixed --list command in presence of special blocks
...
block type RLE is special, compressed size is always 1.
block type 3 is "reserved", aka not supported.
2017-08-18 16:18:20 -07:00
c523c93b26
improved and fixed --list command, original patches by @ib ( #772 )
...
accepts all skippable frame identifiers.
display in MB or KB, depending on frame size.
fixed combination of skippable and zstd frames.
2017-08-18 15:57:53 -07:00
0fb7b23fcc
fix typo in lz4 support code
2017-08-14 14:04:24 +02:00
b8280fe929
minor code refactoring
2017-07-03 15:14:55 -07:00
368b974387
fixed gz error reporting
2017-07-03 13:47:46 -07:00
c9f21c8a71
gzip : only display inflateEnd error when there is no error before
2017-07-03 13:45:09 -07:00
6c35112d08
gzip decoding does no longer exit() on invalid input
2017-07-03 13:24:50 -07:00
e97ff3b54a
decoding malformed lz4 frames does no longer exit()
2017-07-03 11:27:29 -07:00
c34185df13
fileio : decoding malformed lzma frame does no longer exit()
...
makes it possible to continue decoding file list
2017-07-03 10:27:16 -07:00
849ecf3510
[zstdcli] Support xz by default when liblzma is available
2017-06-23 17:11:38 -07:00
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
991f9dfcde
switched fileio.c to ZSTD_DEBUG trigger macro
2017-06-21 15:16:13 -07:00
5be0f5544e
added newline to align output
2017-06-21 12:41:10 -07:00
6f5fe71041
added error check for when file could not be opened
2017-06-21 12:37:23 -07:00
74a725da69
reversed calculation of ratio
2017-06-21 12:27:20 -07:00
a73c2a444a
added tests with null files, changed condition to check that the file is more than 0 bytes
2017-06-20 14:33:08 -07:00
0765602903
fixed error where extremely small files were not being detected as not compressed with zstd
2017-06-20 14:04:46 -07:00
3701dde3ba
changed to use LONG_SEEK
2017-06-20 13:44:05 -07:00
93a5510962
added progress metric to display
2017-06-20 13:26:25 -07:00
bda57754a2
moved signifcant operations to fileio.c
2017-06-20 12:43:10 -07:00
c594507871
removed double if statements for same condition
2017-06-20 11:59:45 -07:00
72e0ac9929
minor update to logic
2017-06-20 11:58:27 -07:00
cea55a9217
size_t const value
2017-06-20 11:56:11 -07:00
bed0dc95bc
changed displayInfo alignment and const
2017-06-20 11:47:24 -07:00
cb2dbe6434
changed fileInfo initialization, updated error code
2017-06-20 09:35:21 -07:00
55b9b28dad
made style changes, added description in man file
2017-06-19 09:55:55 -07:00
4a0e4c856c
fixing more spacing on loops
2017-06-15 18:08:22 -07:00
b3e62446c6
added in correct error return from main in zstdcli.c
2017-06-15 17:46:49 -07:00
0d7c4d766a
initialized info manually to avoid error
2017-06-15 17:30:18 -07:00
d3b34e4a25
changed spacing on if statements
2017-06-15 17:12:41 -07:00
b316691f91
added const
2017-06-15 17:03:59 -07:00
60a2e55e2e
added case for when file is not compressed with zstd (incorrect magic number)
2017-06-15 17:00:59 -07:00
acaefb531b
refactoring brackets to allow for const values
2017-06-15 16:27:38 -07:00
0757eae6ff
rearranging code to allow for consts
2017-06-15 16:12:04 -07:00
e49afae2ab
changed implicit precision loss again
2017-06-15 16:03:40 -07:00
54e0b6c66b
got rid of imlpicit precision loss
2017-06-15 16:01:52 -07:00
b766211e10
made another function static
2017-06-15 15:55:49 -07:00
e7f02fc58a
closed file at end of function, created another variable to keep exit points simpler
2017-06-15 15:53:02 -07:00
8b3ff7c9bc
changed info to static allocation
2017-06-15 15:35:37 -07:00
96cabf7233
cleaned up if statements
2017-06-15 15:25:20 -07:00
4495e9a826
allow analysis even if file does not have zst extension
2017-06-15 15:02:54 -07:00
a9b77c83e5
cleaning up code for analyzing frames
2017-06-15 14:13:28 -07:00
e208992529
cleaning up code
2017-06-15 12:27:32 -07:00
fc428ab350
changing formatting again
2017-06-14 13:26:19 -07:00
ebd60349f5
changed U64 to unsigned long to get rid of build test errors
2017-06-14 13:11:18 -07:00
81fa33b55a
cleaning up code
2017-06-13 10:06:49 -07:00
a56dcbfcf8
changed print format for non-verbose version
2017-06-12 17:47:33 -07:00
c828b52111
yet more refactoring
2017-06-12 17:19:53 -07:00
28dbf8ee6c
more refactoring for C90
2017-06-12 17:09:08 -07:00
618a7b66e2
refactoring for C90 syntax
2017-06-12 16:53:50 -07:00
6e33c74133
formatting issues resolved
2017-06-12 16:29:00 -07:00
173a739717
code refactorings with explicit conversion
2017-06-12 16:14:04 -07:00
f3d6ab28a2
formated information for verbose output
2017-06-12 15:59:28 -07:00
6e0204470a
displayed decompressed size
2017-06-12 15:51:59 -07:00
dcd6ba6dc6
incremented decompressedSize instead of setting value
2017-06-12 15:40:47 -07:00
6996bd2598
removed useless lines
2017-06-12 15:24:50 -07:00
9cb602ee25
added in logic for parsing through blocks/frames
2017-06-12 15:22:48 -07:00
786b7cac27
added code to analyze the first frame header
2017-06-12 13:46:39 -07:00
a3d54cf73d
added line spacing for clarity
2017-06-12 10:58:34 -07:00
9e6a2eaab6
added MT support to NEWAPI
2017-06-11 18:39:46 -07:00
0f06f4f266
added display for compressed size
2017-06-06 09:21:42 -07:00
4128f67a4a
added in check suffix check to ensure file was compressed with zstd
2017-06-05 15:00:06 -07:00
901435e9ef
setup basic functions for adding --list functionality
2017-06-05 14:45:31 -07:00
6d4fef36de
Added ZSTD_compress_generic()
...
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
2017-05-17 18:36:15 -07:00
23c256e44b
removed useless variable from CCtx
...
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
2017-05-16 18:10:11 -07:00
e12ae02357
minor coding style update
...
reduced some long lines
2017-05-16 17:32:33 -07:00
01a1abfdb5
cli : -d and -t do not stop after a failed decompression
...
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
2017-05-05 19:15:24 -07:00
2e63a877f3
fixed xzstd
...
--format=xz was missing a break, making the execution continue into lz4 error message
2017-05-02 15:40:42 -07:00
31533bacce
Changed ZSTD_createCDict_advanced()
...
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.
Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
eab41c1872
Fix LZ4 wrapper deprecation warnings
2017-04-26 10:17:38 -07:00
2c4b6fe6b3
Make lz4 compression/decompression compatible with library r123
2017-04-25 11:00:54 -07:00
4de8632957
Add LZ4 compress/decompress support to CLI
2017-04-24 16:48:25 -07:00
42bac7fa84
Change ifndef's to undef's
2017-04-13 15:35:05 -07:00
f876f1200c
Fix compilation on macOS
2017-04-13 12:33:45 -07:00
5c42d0edc8
cli : better status display for zstdmt in 1-thread mode
2017-04-11 16:57:32 -07:00
0e30059ba1
cli : FIO_createDictBuffer() replaces FIO_loadFile()
...
makes it more explicit that it allocates a buffer
and that it's meant to be used for dictionary.
Also : simplified function a bit,
now only works for dictionaries up to DICTSIZE_MAX
2017-04-11 14:41:02 -07:00
279be2015b
Let zstd overwrite read-only files
2017-04-06 12:56:40 -07:00
eb38617175
Clean up default sparse logic
2017-03-31 15:20:50 -07:00
96fe545a18
Turn off sparse mode for OS X by default
2017-03-31 15:16:43 -07:00
042ba122ae
Change g_displayLevel to int and fix DISPLAYUPDATE flush
2017-03-23 11:21:59 -07:00
aa8bcf360f
Add xz and lzma support.
...
Finish feature started by @inikep.
* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
2017-03-13 18:45:41 -07:00
8e5032a965
cli : fix : --rm is silent when input is stdin (decompression)
2017-02-28 09:42:37 +01:00
a81c2e7e44
Merge pull request #573 from facebook/ddict
...
Improved DDict memory usage
2017-02-27 20:54:42 -08:00
ef569bf75f
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-02-27 15:58:38 -08:00
dccd6b6f65
cli : fix : --rm is silent when input is stdin
...
previously, app would produce an error message, and stop.
2017-02-27 15:57:50 -08:00
862698f479
minor tweaks in FIO_decompressGzFrame
2017-02-27 13:21:05 +01:00
0b9b894b2d
reduced ZSTD_DDict memory usage
...
saved 128 KB
2017-02-27 00:27:30 -08:00
f0b9a8dddb
Merge pull request #547 from inikep/dev11
...
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
2017-02-14 12:29:00 -08:00
ce13d087d9
fix LONG_SEEK
2017-02-14 09:52:52 +01:00
bf336572bf
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
2017-02-14 09:45:33 +01:00
442c75f132
removed UTIL_doesFileExists (replaced with UTIL_isRegFile)
2017-02-14 09:38:51 +01:00
98509a70ac
fixed function name
2017-02-14 09:23:32 +01:00
1a195b3b7a
fixed unitialized variable warning
2017-02-13 22:56:31 +01:00
48466b36e8
Resolve conflict with branch 'refs/remotes/facebook/dev'
...
# Conflicts:
# programs/fileio.c
2017-02-13 21:35:39 +01:00
db2666c10c
Merge pull request #536 from iburinoc/multiframe
...
Simple API multiframe decoding
2017-02-13 12:18:16 -08:00
64f7221958
limit zlib compression level to Z_BEST_COMPRESSION
2017-02-13 21:00:41 +01:00
6220bfc924
fix indentation in previous commit
2017-02-13 12:00:59 +01:00
168d9b8006
fix seeking 2GB+ files under Windows
...
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes fallback workaround using Win32 API).
2017-02-12 16:05:19 +01:00
0f5c95af44
Disambiguate pledgedSrcSize == 0
...
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
2017-02-08 15:12:46 -08:00
4f9eaa7bb3
fixed gcc warnings
2017-02-08 18:08:09 +01:00
cb56306a50
.gz suffix for gzip compressed files
2017-02-08 17:37:14 +01:00
02018c83cf
added FIO_compressGzFrame
2017-02-08 16:54:23 +01:00
2cb8ee8784
Change zlib include to be a system include
2017-02-06 11:32:13 -08:00
b02ac8d613
fixed pointer conversion warnings (C++)
...
in gz module
2017-02-03 08:43:06 -08:00
c3cba9d858
fixed silent conversion warnings in GZDECOMPRESS path
2017-02-02 17:12:50 -08:00
8d8513fb64
fixed C constant restrictions
2017-01-30 14:37:08 -08:00
3672d06d06
zstdmt : section size is set to be a minimum of overlapSize
...
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
2017-01-30 13:35:45 -08:00
6be2337c26
added command --block-size=
...
for Multi-threading only.
alias : -B#
2017-01-30 11:17:26 -08:00
5d9b894e46
Fixed status display for zstdmt
...
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.
This patch disables the displaying of % compression
when multi-threading is enabled.
It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.
There is no modification to decompression side,
since decompression is only single-threaded for now.
2017-01-27 13:30:18 -08:00
717c65d690
Merge pull request #519 from inikep/dev11
...
Dev11
2017-01-26 14:23:44 -08:00
8dafb1acf5
CLI : automatically set overlap size to max (windowSize) for max compression level
2017-01-25 17:01:13 -08:00
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
943cff9c37
fixed zstdmt cli freeze issue with large nb of threads
...
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
2017-01-25 12:35:19 -08:00
Yann Collet
Nick Terrell
Yann Collet
W. Felix Handte
Stella Lau
codicodi
Paul Cruz
Sean Purcell
Przemyslaw Skibinski
ds77