ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
991f9dfcde
switched fileio.c to ZSTD_DEBUG trigger macro
2017-06-21 15:16:13 -07:00
5be0f5544e
added newline to align output
2017-06-21 12:41:10 -07:00
6f5fe71041
added error check for when file could not be opened
2017-06-21 12:37:23 -07:00
74a725da69
reversed calculation of ratio
2017-06-21 12:27:20 -07:00
a73c2a444a
added tests with null files, changed condition to check that the file is more than 0 bytes
2017-06-20 14:33:08 -07:00
0765602903
fixed error where extremely small files were not being detected as not compressed with zstd
2017-06-20 14:04:46 -07:00
3701dde3ba
changed to use LONG_SEEK
2017-06-20 13:44:05 -07:00
93a5510962
added progress metric to display
2017-06-20 13:26:25 -07:00
bda57754a2
moved signifcant operations to fileio.c
2017-06-20 12:43:10 -07:00
c594507871
removed double if statements for same condition
2017-06-20 11:59:45 -07:00
72e0ac9929
minor update to logic
2017-06-20 11:58:27 -07:00
cea55a9217
size_t const value
2017-06-20 11:56:11 -07:00
bed0dc95bc
changed displayInfo alignment and const
2017-06-20 11:47:24 -07:00
cb2dbe6434
changed fileInfo initialization, updated error code
2017-06-20 09:35:21 -07:00
55b9b28dad
made style changes, added description in man file
2017-06-19 09:55:55 -07:00
4a0e4c856c
fixing more spacing on loops
2017-06-15 18:08:22 -07:00
b3e62446c6
added in correct error return from main in zstdcli.c
2017-06-15 17:46:49 -07:00
0d7c4d766a
initialized info manually to avoid error
2017-06-15 17:30:18 -07:00
d3b34e4a25
changed spacing on if statements
2017-06-15 17:12:41 -07:00
b316691f91
added const
2017-06-15 17:03:59 -07:00
60a2e55e2e
added case for when file is not compressed with zstd (incorrect magic number)
2017-06-15 17:00:59 -07:00
acaefb531b
refactoring brackets to allow for const values
2017-06-15 16:27:38 -07:00
0757eae6ff
rearranging code to allow for consts
2017-06-15 16:12:04 -07:00
e49afae2ab
changed implicit precision loss again
2017-06-15 16:03:40 -07:00
54e0b6c66b
got rid of imlpicit precision loss
2017-06-15 16:01:52 -07:00
b766211e10
made another function static
2017-06-15 15:55:49 -07:00
e7f02fc58a
closed file at end of function, created another variable to keep exit points simpler
2017-06-15 15:53:02 -07:00
8b3ff7c9bc
changed info to static allocation
2017-06-15 15:35:37 -07:00
96cabf7233
cleaned up if statements
2017-06-15 15:25:20 -07:00
4495e9a826
allow analysis even if file does not have zst extension
2017-06-15 15:02:54 -07:00
a9b77c83e5
cleaning up code for analyzing frames
2017-06-15 14:13:28 -07:00
e208992529
cleaning up code
2017-06-15 12:27:32 -07:00
fc428ab350
changing formatting again
2017-06-14 13:26:19 -07:00
ebd60349f5
changed U64 to unsigned long to get rid of build test errors
2017-06-14 13:11:18 -07:00
81fa33b55a
cleaning up code
2017-06-13 10:06:49 -07:00
a56dcbfcf8
changed print format for non-verbose version
2017-06-12 17:47:33 -07:00
c828b52111
yet more refactoring
2017-06-12 17:19:53 -07:00
28dbf8ee6c
more refactoring for C90
2017-06-12 17:09:08 -07:00
618a7b66e2
refactoring for C90 syntax
2017-06-12 16:53:50 -07:00
6e33c74133
formatting issues resolved
2017-06-12 16:29:00 -07:00
173a739717
code refactorings with explicit conversion
2017-06-12 16:14:04 -07:00
f3d6ab28a2
formated information for verbose output
2017-06-12 15:59:28 -07:00
6e0204470a
displayed decompressed size
2017-06-12 15:51:59 -07:00
dcd6ba6dc6
incremented decompressedSize instead of setting value
2017-06-12 15:40:47 -07:00
6996bd2598
removed useless lines
2017-06-12 15:24:50 -07:00
9cb602ee25
added in logic for parsing through blocks/frames
2017-06-12 15:22:48 -07:00
786b7cac27
added code to analyze the first frame header
2017-06-12 13:46:39 -07:00
a3d54cf73d
added line spacing for clarity
2017-06-12 10:58:34 -07:00
9e6a2eaab6
added MT support to NEWAPI
2017-06-11 18:39:46 -07:00
0f06f4f266
added display for compressed size
2017-06-06 09:21:42 -07:00
4128f67a4a
added in check suffix check to ensure file was compressed with zstd
2017-06-05 15:00:06 -07:00
901435e9ef
setup basic functions for adding --list functionality
2017-06-05 14:45:31 -07:00
6d4fef36de
Added ZSTD_compress_generic()
...
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
2017-05-17 18:36:15 -07:00
23c256e44b
removed useless variable from CCtx
...
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
2017-05-16 18:10:11 -07:00
e12ae02357
minor coding style update
...
reduced some long lines
2017-05-16 17:32:33 -07:00
01a1abfdb5
cli : -d and -t do not stop after a failed decompression
...
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
2017-05-05 19:15:24 -07:00
2e63a877f3
fixed xzstd
...
--format=xz was missing a break, making the execution continue into lz4 error message
2017-05-02 15:40:42 -07:00
31533bacce
Changed ZSTD_createCDict_advanced()
...
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.
Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
eab41c1872
Fix LZ4 wrapper deprecation warnings
2017-04-26 10:17:38 -07:00
2c4b6fe6b3
Make lz4 compression/decompression compatible with library r123
2017-04-25 11:00:54 -07:00
4de8632957
Add LZ4 compress/decompress support to CLI
2017-04-24 16:48:25 -07:00
42bac7fa84
Change ifndef's to undef's
2017-04-13 15:35:05 -07:00
f876f1200c
Fix compilation on macOS
2017-04-13 12:33:45 -07:00
5c42d0edc8
cli : better status display for zstdmt in 1-thread mode
2017-04-11 16:57:32 -07:00
0e30059ba1
cli : FIO_createDictBuffer() replaces FIO_loadFile()
...
makes it more explicit that it allocates a buffer
and that it's meant to be used for dictionary.
Also : simplified function a bit,
now only works for dictionaries up to DICTSIZE_MAX
2017-04-11 14:41:02 -07:00
279be2015b
Let zstd overwrite read-only files
2017-04-06 12:56:40 -07:00
eb38617175
Clean up default sparse logic
2017-03-31 15:20:50 -07:00
96fe545a18
Turn off sparse mode for OS X by default
2017-03-31 15:16:43 -07:00
042ba122ae
Change g_displayLevel to int and fix DISPLAYUPDATE flush
2017-03-23 11:21:59 -07:00
aa8bcf360f
Add xz and lzma support.
...
Finish feature started by @inikep.
* Add xz and lzma compression and decompression support to target `xzstd`.
* Fix bug in gzip decompression that silently accepted truncated files.
* Add gzip frame composition tests.
* Add xz/lzma compatibility tests.
* Add xz/lzma frame composition tests.
2017-03-13 18:45:41 -07:00
8e5032a965
cli : fix : --rm is silent when input is stdin (decompression)
2017-02-28 09:42:37 +01:00
a81c2e7e44
Merge pull request #573 from facebook/ddict
...
Improved DDict memory usage
2017-02-27 20:54:42 -08:00
ef569bf75f
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-02-27 15:58:38 -08:00
dccd6b6f65
cli : fix : --rm is silent when input is stdin
...
previously, app would produce an error message, and stop.
2017-02-27 15:57:50 -08:00
862698f479
minor tweaks in FIO_decompressGzFrame
2017-02-27 13:21:05 +01:00
0b9b894b2d
reduced ZSTD_DDict memory usage
...
saved 128 KB
2017-02-27 00:27:30 -08:00
f0b9a8dddb
Merge pull request #547 from inikep/dev11
...
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
2017-02-14 12:29:00 -08:00
ce13d087d9
fix LONG_SEEK
2017-02-14 09:52:52 +01:00
bf336572bf
Avoid fseek()'s 2GiB barrier with MacOS and *BSD
2017-02-14 09:45:33 +01:00
442c75f132
removed UTIL_doesFileExists (replaced with UTIL_isRegFile)
2017-02-14 09:38:51 +01:00
98509a70ac
fixed function name
2017-02-14 09:23:32 +01:00
1a195b3b7a
fixed unitialized variable warning
2017-02-13 22:56:31 +01:00
48466b36e8
Resolve conflict with branch 'refs/remotes/facebook/dev'
...
# Conflicts:
# programs/fileio.c
2017-02-13 21:35:39 +01:00
db2666c10c
Merge pull request #536 from iburinoc/multiframe
...
Simple API multiframe decoding
2017-02-13 12:18:16 -08:00
64f7221958
limit zlib compression level to Z_BEST_COMPRESSION
2017-02-13 21:00:41 +01:00
6220bfc924
fix indentation in previous commit
2017-02-13 12:00:59 +01:00
168d9b8006
fix seeking 2GB+ files under Windows
...
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes fallback workaround using Win32 API).
2017-02-12 16:05:19 +01:00
0f5c95af44
Disambiguate pledgedSrcSize == 0
...
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
2017-02-08 15:12:46 -08:00
4f9eaa7bb3
fixed gcc warnings
2017-02-08 18:08:09 +01:00
cb56306a50
.gz suffix for gzip compressed files
2017-02-08 17:37:14 +01:00
02018c83cf
added FIO_compressGzFrame
2017-02-08 16:54:23 +01:00
2cb8ee8784
Change zlib include to be a system include
2017-02-06 11:32:13 -08:00
b02ac8d613
fixed pointer conversion warnings (C++)
...
in gz module
2017-02-03 08:43:06 -08:00
c3cba9d858
fixed silent conversion warnings in GZDECOMPRESS path
2017-02-02 17:12:50 -08:00
8d8513fb64
fixed C constant restrictions
2017-01-30 14:37:08 -08:00
3672d06d06
zstdmt : section size is set to be a minimum of overlapSize
...
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
2017-01-30 13:35:45 -08:00
6be2337c26
added command --block-size=
...
for Multi-threading only.
alias : -B#
2017-01-30 11:17:26 -08:00
5d9b894e46
Fixed status display for zstdmt
...
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.
This patch disables the displaying of % compression
when multi-threading is enabled.
It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.
There is no modification to decompression side,
since decompression is only single-threaded for now.
2017-01-27 13:30:18 -08:00
717c65d690
Merge pull request #519 from inikep/dev11
...
Dev11
2017-01-26 14:23:44 -08:00
8dafb1acf5
CLI : automatically set overlap size to max (windowSize) for max compression level
2017-01-25 17:01:13 -08:00
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
943cff9c37
fixed zstdmt cli freeze issue with large nb of threads
...
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
2017-01-25 12:35:19 -08:00
5022a18d51
improved #232 fix
2017-01-25 13:11:26 +01:00
64fa2dbc5e
Fixed https://github.com/facebook/zstd/issues/232
2017-01-25 13:02:33 +01:00
512cbe8c10
zstdmt cli and API allow selection of section sizes
...
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.
It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()
The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
2017-01-24 17:08:53 -08:00
500014af49
zstd cli can now compress using multi-threading
...
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads() (fileio)
2017-01-19 17:04:28 -08:00
7a8a03c20d
util.h: restore BSD license for Facebook Open-Source
2016-12-21 15:08:44 +01:00
97a258d71d
updated comments
2016-12-21 14:00:41 +01:00
e679741b18
_CRT_SECURE_NO_WARNINGS moved to util.h
2016-12-21 13:47:11 +01:00
2f6ccee6af
platform.h: removed Compiler Options
2016-12-21 13:23:34 +01:00
a35b9448ac
improved MinGW support
2016-12-21 11:18:45 +01:00
20b089e53d
simplified zstdcli.c
2016-12-21 09:19:15 +01:00
16ae6563a2
executables use new util.h and platform.h
2016-12-21 09:06:14 +01:00
f8046b8e72
Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112
...
# Conflicts:
# appveyor.yml
2016-12-19 08:20:26 +01:00
b866e72826
tools use platform.h
2016-12-16 14:24:01 +01:00
c71e552b2e
fixed "strategy" in advanced compression parameters
2016-12-13 20:04:32 +01:00
8349d675e0
fileio.c: support advanced compression parameters
2016-12-13 13:24:59 +01:00
7c6bbc3298
updated FIO_passThrough
2016-12-05 18:31:14 +01:00
6b508b1770
updated test-gzstd
2016-12-05 18:02:40 +01:00
b493e3b3d3
introduced srcBufferLoaded
2016-12-05 17:39:38 +01:00
3c69760275
improved FIO_decompressGzFrame
2016-12-05 15:58:23 +01:00
8489f184f6
improved detection of -lz
2016-12-05 13:47:00 +01:00
743b33f57e
fix zstdcat
2016-12-02 15:18:57 -08:00
5bd4237beb
minor refactor
2016-12-02 12:40:57 -08:00
690753ea1a
improved formatting in FIO_decompressSrcFile
2016-12-02 16:20:16 +01:00
4e49580407
removed testing artifacts
2016-12-02 15:19:00 +01:00
c5eebca128
rewritten FIO_decompressGzFile
2016-12-02 15:01:31 +01:00
b0f2ef2119
improved gzip_* functions
2016-12-02 13:50:29 +01:00
4b504f131a
added gzip_open, gzip_close, gzip_read
2016-12-02 13:11:39 +01:00
daaf754539
detect stream with ungetc
2016-12-01 13:29:19 +01:00
19aad42ee1
added FIO_decompressGzFile
2016-12-01 11:56:31 +01:00
abfb51f5f2
gzstd: decompresses .gz files
2016-11-30 15:05:54 +01:00
0e14675df2
fileio.c: detect .gz files
2016-11-30 13:34:21 +01:00
cdff19c4b3
minor comment change
2016-11-11 17:26:54 -08:00
0018ca28dc
zstd cli : displays total decoded size, even when a stream consists of multiple frames
2016-11-07 14:42:37 -08:00
1f60eb6b17
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-11-03 11:38:04 +01:00
26306fcacf
BMK_SetNbIterations renamed to BMK_SetNbSeconds
2016-11-03 11:38:01 +01:00
179b19776f
fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h
...
Added : ZSTD_isFrame() in experimental section
2016-11-02 17:30:49 -07:00
fcf22e3473
set permissions, access and modification times
2016-11-02 14:08:07 +01:00
a42794df61
preserve file modification time
2016-11-02 13:08:39 +01:00
d4cda27b63
new command -M#, to limit memory usage during decompression ( #403 )
2016-10-14 13:32:20 -07:00
3e82f2b61a
Remove references to GPLv2 license
...
License headers added to Makefiles were taken from `zstd/Makefile`.
2016-10-11 15:28:40 -07:00
993060e0f2
cli : better adaptation to small files
2016-09-21 16:46:08 +02:00
43eeea4725
fileio uses ZSTD_resetCStream()
2016-09-15 15:38:44 +02:00
3ecbe6a37c
fileio uses ZSTD_resetDStream()
2016-09-14 17:26:59 +02:00
d77f2ceee6
Merge branch 'dev' of github.com:Cyan4973/zstd into dev
2016-09-12 03:14:03 +02:00
5c6d244973
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-09-07 14:54:54 +02:00
ac8bace6b1
support large skippable frames
2016-09-07 14:54:23 +02:00
03d3f238de
minor comment
2016-09-07 07:01:33 +02:00
aad9fe5470
don't remove() /dev/null ( #316 )
2016-09-07 07:00:08 +02:00
4ded9e591c
added boilerplate
2016-08-30 11:06:28 -07:00
1c69baa1a9
CLI supports Legacy formats v0.4+
2016-08-28 12:47:17 -07:00
5f53b0335e
fixed continuation context
2016-08-28 10:00:49 -07:00
ab267e772c
removed fileio specific legacy support;
...
now depends on ZSTD_decompressStream() transparent legacy support
2016-08-28 08:46:25 -07:00
e80d15304a
Merge pull request #296 from inikep/Other
...
Other
2016-08-18 11:48:48 +02:00
5a54870047
fixed Intel Compiler warnings with Visual Studio
...
http://encode.ru/threads/2119-Zstandard?p=49504&viewfull=1#post49504
2016-08-18 09:00:25 +02:00
bfdc34e5a3
Merge pull request #294 from inikep/Other
...
improved Makefiles
2016-08-17 19:41:56 +02:00
78f3e0630b
disabled %llu warnings with MinGW on Windows
2016-08-17 14:52:11 +02:00
3d9211e961
Merge pull request #290 from david-y-lam/fix_file_name
...
Remove file name truncation in zstd CLI output
2016-08-17 14:05:29 +02:00
53e17fbd5e
updated streaming API
2016-08-17 01:39:22 +02:00
94ca85d01b
fixed Visual warning
2016-08-16 15:11:28 +02:00
6263ba5451
fileio uses new streaming API
2016-08-16 15:11:28 +02:00
fb5e385164
minor changes
2016-08-16 15:11:28 +02:00
0f2704581e
Remove truncation of filename in zstd CLI log lines
2016-08-13 11:26:21 -07:00
60ba31c570
zbuff uses ZSTD_compressEnd()
2016-07-28 19:55:09 +02:00
fbd557d5c2
multi-files -t doesn't stop after detecting magic number read failure
2016-07-26 17:13:58 +02:00
a1dd6b97d2
fixed null-length round trip
2016-07-26 16:44:09 +02:00
7adc2328a3
fixed --test on zero-length files, reported by @amnilsson
2016-07-26 15:49:24 +02:00
24a3d90bf0
strengthened integrity tests
2016-07-26 01:33:20 +02:00
b402490546
fixed #260 , reported by @amnilsson
2016-07-26 00:49:47 +02:00
17508f1a16
fixed a few minor coverity warnings
2016-07-14 17:18:20 +02:00
44f684ded3
fixed minor coverity warning
2016-07-13 20:12:33 +02:00
5e80dd3261
fixed minor coverity warnings
2016-07-13 19:21:57 +02:00
3c242e79d3
updated compression levels table
2016-07-13 14:56:24 +02:00
2cac5b30b9
changed default compression level to 3 (can be modified with macro ZSTDCLI_DEFAULT_CLEVEL)
2016-07-13 14:15:08 +02:00
19c27d27f1
simplified legacy functions, no longer need magic number
2016-07-07 14:40:13 +02:00
c62cda9def
fileio.c no longer depends on zstd_internal.h
2016-07-03 01:36:57 +02:00
92d75667e4
fix for fast mode
2016-07-03 01:10:53 +02:00
b2b5309c6f
removes Visual warning on strerror()
2016-07-03 00:20:03 +02:00
ed7fb8413c
extended use of strerror(errno)
2016-07-02 21:01:54 +02:00
b71adf45c1
extended use of strerror(errno)
2016-07-02 01:51:12 +02:00
b7f197633a
use strerror(errno) to return system error message
2016-07-02 00:59:25 +02:00
cebab1d322
Explicit error message when dst file cannot be created / opened
2016-07-02 00:12:14 +02:00
6c6e1751f6
use ZSTD_getParams() to simplify code
2016-06-27 15:28:45 +02:00
391a128794
fix : segfault in command line during automatic overwrite protection mode
2016-06-21 17:06:25 +02:00
c98f8e7b57
Modified : zstd CLI enables checksum by default
2016-06-20 16:31:24 +02:00
b09b12ce10
Added command `--rm` : remove source file after successful de/compression
2016-06-09 23:01:19 +02:00
2cc72f1fd3
fixed initialization issue in bench
2016-06-06 17:50:07 +02:00
673f0d7cdc
new frame format, allowing custom window size
2016-06-06 00:26:38 +02:00
d3b7f8d21f
Merged `zstd_static.h` into `zstd.h` . Now requires `ZSTD_STATIC_LINKING_ONLY` macro
2016-06-04 19:47:02 +02:00
5347aee8f7
merged `zbuff_static.h` into `zbuff.h` . Now requires `ZBUFF_STATIC_LINKING_ONLY` macro
2016-06-04 19:12:48 +02:00
0d31160b4e
better dynamic memory adaptation when using CLI
2016-06-04 00:09:02 +02:00
87cfbe3e5f
checksum generation can be selected from command line (`--checksum`)
2016-06-01 19:22:15 +02:00
a6b942018d
Merge remote-tracking branch 'refs/remotes/origin/dev' into dev070
...
# Conflicts:
# .travis.yml
# Makefile
# lib/common/zstd_static.h
# programs/Makefile
# projects/VS2008/zstd/zstd.vcproj
# projects/VS2008/zstdlib/zstdlib.vcproj
# projects/cmake/lib/CMakeLists.txt
# projects/cmake/programs/CMakeLists.txt
2016-06-01 00:07:09 +02:00
f772bf54a5
support for skippable frames
2016-05-31 12:43:46 +02:00
6fca9e7545
Changed : max dictionary file size increased to 8 MB
2016-05-31 02:40:42 +02:00
6381e99fb2
Added : ability to create compressed frames without dictID
2016-05-31 02:29:45 +02:00
c46fb924df
added dictionary ID (incomplete)
2016-05-29 05:01:04 +02:00
e75909ef93
ZBUFF_createCCtx_advanced moved to zbuff_static.h
2016-05-25 11:31:16 +02:00
Yann Collet
Yann Collet
Paul Cruz
Sean Purcell
Nick Terrell
Przemyslaw Skibinski
ds77
Yann Collet
David Lam