6f1dfa8adf
removed line with `//` comment
...
this is for a different topic
(better parameter adaptation for small files + dictionary and/or custome parameters)
2017-11-01 17:01:45 -07:00
428e8b3bf4
fix : ZSTD_compress_generic(,,,ZSTD_e_end) automatically sets pledgedSrcSize
...
as per documentation, on ZSTD_setPledgedSrcSize() :
> If all data is provided and consumed in a single round,
> this value (pledgedSrcSize) is overriden by srcSize instead.
This wasn't applied before compression level is transformed into compression parameters.
As a consequence, small input missed compression parameters adaptation.
It seems to work fine now : compression was compared with ZSTD_compress_advanced(),
results were the same.
2017-11-01 13:15:23 -07:00
b495140f67
Update BUCK files
...
* Correct XXH namespace (Fixes #901 )
* Multithreading always enabled
* GZIP/LZ4/LZMA always enabled
* Legacy support always fully enabled
2017-10-25 12:47:57 -07:00
91535d71ec
fixed missing zstdmt_compress.h dependency
...
we lose a warning message :
when a job size is chosen < minimum job size for multithreading,
it is automatically resized to minimum size.
If this information is really useful, it should be present in zstd.h now.
2017-10-19 12:09:34 -07:00
eac42534fe
bench: fixed Visual warning regarding struct initialization
...
also :
removed dependency on zstdmt_compress.h
removed several unused macros
fileio : small code refactoring to reduce some variable scope
2017-10-19 11:56:14 -07:00
d3b9547aa4
IO and bench : ZSTD_NEWAPI is the only remaining code path
...
removed the other 2 code paths (single thread, and ZSTDMT ones)
keeping only the new advanced API, for easier code coverage.
It shall also fix identified issue with Visual Studio
which doesn't have ZSTD_NEWAPI defined.
2017-10-18 17:01:53 -07:00
300e1df0a3
fixed wrong test to display compression status
2017-10-18 11:41:52 -07:00
18b795374a
UTIL_getFileSize() returns UTIL_FILESIZE_UNKNOWN on failure
...
UTIL_getFileSize() used to return zero on failure.
This made it impossible to distinguish a failure from a genuine empty file.
Both cases where coalesced.
Adding UTIL_FILESIZE_UNKNOWN constant has many consequences on user code,
since in many places, the `0` was assumed to mean "error".
This is no longer the case, and the error code must be actively checked.
2017-10-17 16:14:25 -07:00
32c9f715ae
fixed : Visual build compressing stdin with multi-threading enabled fails
...
It was multiple reasons stacked :
- Visual use a different code path, because ZSTD_NEWAPI is not defined
- fileio.c sends `0` as `pledgedSrcSize` to mean `ZSTD_CONTENTSIZE_UNKNOWN` (fixed)
- ZSTDMT_resetCCtx() interpreted `0` as "empty" instead of "unknown" (fixed)
2017-10-17 14:07:43 -07:00
fc8d293460
dictionary compression use correct file size estimation
...
when determining compression parameters
to compress one file only.
For multiple files, it still "bets" that files are going to be small.
There was also a bug recently added in ZSTD_CCtx_loadDictionary_advanced()
making it incapable to use pledgedSrcSize to determine compression parameters.
2017-10-14 01:21:43 -07:00
9ef32b3cf1
minor : zstd -l -v display each file name
2017-10-14 00:02:32 -07:00
dd18d73e7e
fileio: content size is enabled by default
2017-10-13 16:32:18 -07:00
6dd958eea2
[zstdcli] Add window size to verbose list
...
```
> zstd --list -v file1 file2 file3
*** zstd command line interface 64-bits v1.3.2, by Yann Collet ***
Window Size: 512.00 KB (524288 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64
Window Size: 8192.00 KB (8388608 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64
Window Size: 512.00 KB (524288 B)
Compressed Size: 0.01 KB (15 B)
Check: None
```
2017-10-04 12:26:28 -07:00
3b27ed41fd
Merge branch 'srcSize' into dev
2017-10-02 16:34:14 -07:00
4946993f87
removed isRegularFile parameter
...
no longer useful : size of src is determined for each file.
2017-10-02 12:29:25 -07:00
7f580f9ee8
interruption handler and variable are static
2017-10-02 11:39:05 -07:00
fe5444bc66
removed the statement for all versions of Visual Studio
2017-10-02 02:02:16 -07:00
51d82d5516
same error in Visual Studio 2012 ...
2017-10-02 01:12:40 -07:00
ed7ae4c9bd
The issue also impacts Visual Studio 2010
2017-10-02 00:45:28 -07:00
6e7ba3df2f
added (void)sig to avoid compilers complaining that sig is not used.
2017-10-02 00:19:47 -07:00
82bc200f82
conditionnally removed invocation that generates a buggy warning with Visual Studio 2008
2017-10-02 00:02:24 -07:00
bd18095edc
blindfix for Visual : minor casting issue
...
should not happen since SIGIGN is provided by <signal.h>,
so it should work "ouf of the box"
2017-10-01 15:32:48 -07:00
00fc1ba8dd
cli: add Ctrl-C support, requested by @mike155 in #854
...
Now, pressing Ctrl-C during compression or decompression
will erase operation artefact (unfinished destination file)
before leaving execution.
2017-10-01 12:10:26 -07:00
e580dc6a4a
Merge pull request #860 from felixhandte/zstd-lz4-support-tests
...
Add Default LZ4 Support When Available
2017-09-29 22:32:54 -07:00
8afb151c9b
cli: fixed wrong initialization in MT mode
...
It's not good to mix old and new API
ZSTD_resetCStream() doesn't just set pledgedSrcSize :
it also sets the CCtx for a single thread compression.
Problem is, when 2+ threads are defined in cctx->requestedParams,
ZSTD_compress_generic() will want to start MT compression,
since initialization is supposed to have already happened (thanks to ZSTD_resetCStream())
except that the underlying ZSTDMT_CCtx* object is not created,
resulting in a segfault.
This is an invalid construction
(correct one is to use ZSTD_CCtx_setPledgedSrcSize()).
I haven't found a nice way to mitigate this impact if someone makes the same mistake.
At some point, removing the old API to keep only the new API within fileio.c will limit these risks.
2017-09-29 22:14:37 -07:00
fbd5ab7027
minor fix : no longer use fake srcSize during resource creation
...
srcSize is read and provided at each file, not at resource creation.
This used to be useful with older API, because it could not re-adapt parameters between sessions.
At some point, it will be better to remove the old code, and only keep the new_api.
It works fine by now.
2017-09-29 19:40:27 -07:00
db1668a43b
fix : srcSize written in frame header when multiple files compressed
...
This information used to be disabled when nbFiles>1.
It was badly initialized later in the code, resulting in an error.
2017-09-29 18:05:18 -07:00
8afcc80e07
decode more data before triggering error
...
fixes #874 :
when a frame is not properly terminated by a "last block" signal,
zstd -d used to detect it immediately and error out.
This version will decode and flush the last block, and only then issue an error.
2017-09-29 15:54:09 -07:00
dc27c36495
Update documentation to reflect other format support
2017-09-28 19:43:12 -04:00
d0519d4b0c
Add CLI Program Name Detection for LZ4
2017-09-28 19:18:15 -04:00
54a827fff0
Merge branch 'dev' into newFormats
...
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
3182ea2e64
Merge pull request #866 from facebook/list
...
improved --list display
2017-09-27 16:34:29 -07:00
c233bdbaee
Increase maximum window size
...
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
`--zstd=wlog=#`. These options also set the window size during
decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
and decompression.
2017-09-26 14:00:01 -07:00
3095ca8c56
fixed minor conversion warnings for g++ on Linux
...
U64 is not considered equivalent to unsigned long long
2017-09-26 13:53:50 -07:00
56f1f0e3dd
write summary for --list on multiple files
2017-09-26 11:21:36 -07:00
62568c9a42
added capability to generate magic-less frames
...
decoder not implemented yet
2017-09-25 14:26:26 -07:00
360238733a
Adds LZ4 support by default if LZ4 is available
...
Simple makefile change + quick typename change
Test:
make clean
make
# successfully produces binary without lz4 support
make clean
# with flags to pick up my lz4 build
make MOREFLAGS="-L/home/felixh/prog/lz4/lib -I/home/felixh/prog/lz4/lib"
# successfully produces binary with lz4 support
echo "TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU" | \
./lz4/lz4 | \
LD_LIBRARY_PATH=/home/felixh/prog/lz4/lib ./zstd/zstd -d
# successfully prints TEST TEST TEST THIS IS A TEST STRING PLEASE TEST THIS PLEASE OK THANK YOU
2017-09-22 13:28:56 -07:00
b0c0e3a3fb
Merge pull request #853 from terrelln/blog
...
[zstdcli] Fix LDM advanced options parsing
2017-09-18 15:21:23 -07:00
1fe762e236
[zstdcli] Fix LDM advanced options parsing
2017-09-18 14:49:35 -07:00
92889709f9
fix #851 : sudo zstd -t file.zst changes /dev/null permissions
...
reported by @mike155
2017-09-18 13:41:54 -07:00
1722055799
add comment on using -B# to split input file for dictionary training
2017-09-15 16:23:50 -07:00
c68d17f2da
ensures that sampleSizes table is large enough
...
as recommended by @terrelln
2017-09-15 15:31:31 -07:00
25a60488dd
fixed 64-to-32 conversion warnings
2017-09-15 11:55:13 -07:00
a9694231ca
fixed minor conversion warning
2017-09-15 10:16:26 -07:00
086b9597d9
added ability to split input files for dictionary training
...
using command -B#
This is the same behavior as benchmark module,
which can also split input into arbitrary size blocks, using -B#.
2017-09-14 16:45:10 -07:00
77c137b3ae
minor comment refactor
2017-09-14 15:12:57 -07:00
f1571dad8f
Merge pull request #838 from stellamplau/ldm-mergeDev
...
Add long distance matcher
2017-09-13 13:24:08 -07:00
8f26dc3f9c
blindfix for Visual
...
LARGE_INTEGER is not an integer :
https://msdn.microsoft.com/en-us/library/windows/desktop/aa383713(v=vs.85).aspx
Do not take any risk with the structure definition :
use int init = 0; like Mac code
2017-09-12 21:21:17 -07:00
bc41c7f0eb
fixed minor prototype warning
2017-09-12 19:32:26 -07:00
c95c0c9725
modified util::time API
...
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
2017-09-12 18:12:46 -07:00
6ab4d5e904
[bench] Use higher resolution timer on POSIX
...
The timer used was only accurate up to 0.01 seconds. This timer is accurate up to 1 ns.
It is a monotonic timer that measures the real time difference, not on CPU time.
2017-09-12 16:46:44 -07:00
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
baa37c3362
programs/Makefile : better support for GNU conventions
...
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
2017-09-06 16:53:59 -07:00
eeff55dfa8
Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev
2017-09-06 15:56:32 -07:00
8c33cfe0bc
Add ldm documentation in README
2017-09-06 15:21:01 -07:00
c706de5395
Rename and add short ldm parameters in cli
2017-09-05 21:11:18 -07:00
643d28c701
Add ldm options to 'man zstd'
2017-09-05 11:27:15 -07:00
67d4a6161c
Add ldmBucketSizeLog param
2017-09-02 21:55:29 -07:00
a1f04d518d
Move hashEveryLog to cctxParams and update cli
2017-09-01 15:05:47 -07:00
0558850735
bench stops immediately on decoding error
2017-09-01 11:46:15 -07:00
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
c7818fc676
Merge branch 'modTests' into dev
...
fixed conflict
2017-08-31 17:00:16 -07:00
4299c27132
improved console log of utils.h
...
removed a warning when compiling on Windows
2017-08-31 16:58:47 -07:00
8e298382a8
changed target allarch into allzstd
...
allzstd contains only zstd-related tests.
allmost = allzstd + zwrapper tests (which require zlib)
2017-08-31 14:30:52 -07:00
6a546efb8c
Add long distance matcher
...
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
b0cb081dc8
last batch of header files changed to reflect new license ( #825 )
...
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
cf689b84f9
Sort input file list
...
in order to make builds reproducible
in spite of indeterministic filesystem readdir order.
See https://reproducible-builds.org/ for why this is good.
2017-08-26 17:08:00 +02:00
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
232d62b637
fixed a few headers that were too hastily copy/pasted during last license change
2017-08-21 11:24:32 -07:00
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
1c108c811e
cli : Display supported formats on -vV command
...
Requested and inspired by patch from @ib (#771 )
2017-08-19 13:33:50 -07:00
2ecd34ee5e
fixed unused variables warnings
2017-08-19 01:23:49 -07:00
23706fb743
updated doc on compilation variables
2017-08-19 01:14:36 -07:00
9203003d5f
fixed zstd-nolegacy
...
and added it to allVariants for CI testings
2017-08-19 01:01:53 -07:00
4b387729b6
fixed zstd-small
...
and added it to shortest for CI tests
2017-08-19 00:48:29 -07:00
7729ab83bb
Merge branch 'dev' into variants
2017-08-19 00:37:06 -07:00
32e943b3ef
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-08-19 00:36:37 -07:00
582e19b01c
Merge pull request #801 from facebook/newLicense
...
New license
2017-08-19 00:34:34 -07:00
288e263ec4
added zstd-decompress to the list of variants tested
2017-08-18 18:39:39 -07:00
166645e7b3
fixed zstd-compress
...
file-information is dependent on decompression functions.
it should only be enabled when ZSTD_NODECOMPRESS is not set.
also : added zstd-compress compilation test into `make shortest`
2017-08-18 18:30:41 -07:00
f207b39f55
blindfix for Windows conversion warning
...
long type is 32-bits on Windows 64,
while it's 64-bits on Unix.
64-to-32 shortening conversion for long is a specific Windows issue.
2017-08-18 17:06:12 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
88d2f72df9
fixed --list command in presence of special blocks
...
block type RLE is special, compressed size is always 1.
block type 3 is "reserved", aka not supported.
2017-08-18 16:18:20 -07:00
c523c93b26
improved and fixed --list command, original patches by @ib ( #772 )
...
accepts all skippable frame identifiers.
display in MB or KB, depending on frame size.
fixed combination of skippable and zstd frames.
2017-08-18 15:57:53 -07:00
3f54d788e9
removed --list from cli help (-h), reported by Agostino Sarubbo (@asarubbo) ( #800 )
...
redundant with shorter -l.
both -l and --list do the same thing, and are documented in man page.
2017-08-18 15:15:31 -07:00
0fb7b23fcc
fix typo in lz4 support code
2017-08-14 14:04:24 +02:00
d0b27483ae
[zstdcli] Fix -t in streaming mode
2017-07-18 14:45:49 -07:00
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
3345a91964
cli : use new advanced API by default
2017-07-05 17:34:15 -07:00
b8280fe929
minor code refactoring
2017-07-03 15:14:55 -07:00
368b974387
fixed gz error reporting
2017-07-03 13:47:46 -07:00
c9f21c8a71
gzip : only display inflateEnd error when there is no error before
2017-07-03 13:45:09 -07:00
6c35112d08
gzip decoding does no longer exit() on invalid input
2017-07-03 13:24:50 -07:00
e97ff3b54a
decoding malformed lz4 frames does no longer exit()
2017-07-03 11:27:29 -07:00
c34185df13
fileio : decoding malformed lzma frame does no longer exit()
...
makes it possible to continue decoding file list
2017-07-03 10:27:16 -07:00
1d39550471
Merge pull request #744 from terrelln/dev
...
[pzstd] Remove appveyor tests
2017-07-01 05:58:33 -07:00
56e3964d85
[man] Specify that strategies start at 1
2017-06-30 16:29:37 -07:00
811deaea6f
Merge pull request #736 from terrelln/cover-default-api
...
[zdict] Make COVER the default algorithm
2017-06-28 20:25:36 -07:00
5b7fd7c422
[zdict] Make COVER the default algorithm
2017-06-26 21:09:22 -07:00
d88a441406
[zstdcli] Document HAVE_LZMA and zstd-noxz
2017-06-26 11:25:00 -07:00
6aeb50ea0d
[zstdcli] Add unlzma and unxz symlinks
2017-06-26 11:24:36 -07:00
849ecf3510
[zstdcli] Support xz by default when liblzma is available
2017-06-23 17:11:38 -07:00
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
991f9dfcde
switched fileio.c to ZSTD_DEBUG trigger macro
2017-06-21 15:16:13 -07:00
5be0f5544e
added newline to align output
2017-06-21 12:41:10 -07:00
6f5fe71041
added error check for when file could not be opened
2017-06-21 12:37:23 -07:00
74a725da69
reversed calculation of ratio
2017-06-21 12:27:20 -07:00
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
a73c2a444a
added tests with null files, changed condition to check that the file is more than 0 bytes
2017-06-20 14:33:08 -07:00
b44ab82f7a
ensure new ZSTD_strategy starts at value 1
2017-06-20 14:11:49 -07:00
0765602903
fixed error where extremely small files were not being detected as not compressed with zstd
2017-06-20 14:04:46 -07:00
3701dde3ba
changed to use LONG_SEEK
2017-06-20 13:44:05 -07:00
93a5510962
added progress metric to display
2017-06-20 13:26:25 -07:00
bda57754a2
moved signifcant operations to fileio.c
2017-06-20 12:43:10 -07:00
c594507871
removed double if statements for same condition
2017-06-20 11:59:45 -07:00
72e0ac9929
minor update to logic
2017-06-20 11:58:27 -07:00
cea55a9217
size_t const value
2017-06-20 11:56:11 -07:00
bed0dc95bc
changed displayInfo alignment and const
2017-06-20 11:47:24 -07:00
cb2dbe6434
changed fileInfo initialization, updated error code
2017-06-20 09:35:21 -07:00
c08e649e95
first implementation of bench.c with new API ZSTD_compress_generic()
...
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.
Also : fixed a long standing bug in ZSTDMT streaming API.
2017-06-19 18:25:35 -07:00
fe234bf48b
fix attempts : fullbench for VS2008
2017-06-19 15:23:19 -07:00
55b9b28dad
made style changes, added description in man file
2017-06-19 09:55:55 -07:00
4a0e4c856c
fixing more spacing on loops
2017-06-15 18:08:22 -07:00
b3e62446c6
added in correct error return from main in zstdcli.c
2017-06-15 17:46:49 -07:00
0d7c4d766a
initialized info manually to avoid error
2017-06-15 17:30:18 -07:00
d3b34e4a25
changed spacing on if statements
2017-06-15 17:12:41 -07:00
b316691f91
added const
2017-06-15 17:03:59 -07:00
60a2e55e2e
added case for when file is not compressed with zstd (incorrect magic number)
2017-06-15 17:00:59 -07:00
acaefb531b
refactoring brackets to allow for const values
2017-06-15 16:27:38 -07:00
0757eae6ff
rearranging code to allow for consts
2017-06-15 16:12:04 -07:00
e49afae2ab
changed implicit precision loss again
2017-06-15 16:03:40 -07:00
54e0b6c66b
got rid of imlpicit precision loss
2017-06-15 16:01:52 -07:00
b766211e10
made another function static
2017-06-15 15:55:49 -07:00
e7f02fc58a
closed file at end of function, created another variable to keep exit points simpler
2017-06-15 15:53:02 -07:00
8b3ff7c9bc
changed info to static allocation
2017-06-15 15:35:37 -07:00
96cabf7233
cleaned up if statements
2017-06-15 15:25:20 -07:00
9437cc7498
added --list command
2017-06-15 15:07:54 -07:00
4495e9a826
allow analysis even if file does not have zst extension
2017-06-15 15:02:54 -07:00
a9b77c83e5
cleaning up code for analyzing frames
2017-06-15 14:13:28 -07:00
e208992529
cleaning up code
2017-06-15 12:27:32 -07:00
fc428ab350
changing formatting again
2017-06-14 13:26:19 -07:00
607ee9f4bc
print message when no files are given, rearrange code for readability
2017-06-14 13:23:36 -07:00
ebd60349f5
changed U64 to unsigned long to get rid of build test errors
2017-06-14 13:11:18 -07:00
81fa33b55a
cleaning up code
2017-06-13 10:06:49 -07:00
Yann Collet
Nick Terrell
Yann Collet
W. Felix Handte
Stella Lau
Bernhard M. Wiedemann
codicodi
Paul Cruz