6be2337c26
added command --block-size=
...
for Multi-threading only.
alias : -B#
2017-01-30 11:17:26 -08:00
5d9b894e46
Fixed status display for zstdmt
...
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.
This patch disables the displaying of % compression
when multi-threading is enabled.
It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.
There is no modification to decompression side,
since decompression is only single-threaded for now.
2017-01-27 13:30:18 -08:00
717c65d690
Merge pull request #519 from inikep/dev11
...
Dev11
2017-01-26 14:23:44 -08:00
8dafb1acf5
CLI : automatically set overlap size to max (windowSize) for max compression level
2017-01-25 17:01:13 -08:00
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
943cff9c37
fixed zstdmt cli freeze issue with large nb of threads
...
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
2017-01-25 12:35:19 -08:00
5022a18d51
improved #232 fix
2017-01-25 13:11:26 +01:00
64fa2dbc5e
Fixed https://github.com/facebook/zstd/issues/232
2017-01-25 13:02:33 +01:00
512cbe8c10
zstdmt cli and API allow selection of section sizes
...
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.
It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()
The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
2017-01-24 17:08:53 -08:00
500014af49
zstd cli can now compress using multi-threading
...
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads() (fileio)
2017-01-19 17:04:28 -08:00
7a8a03c20d
util.h: restore BSD license for Facebook Open-Source
2016-12-21 15:08:44 +01:00
97a258d71d
updated comments
2016-12-21 14:00:41 +01:00
e679741b18
_CRT_SECURE_NO_WARNINGS moved to util.h
2016-12-21 13:47:11 +01:00
2f6ccee6af
platform.h: removed Compiler Options
2016-12-21 13:23:34 +01:00
a35b9448ac
improved MinGW support
2016-12-21 11:18:45 +01:00
20b089e53d
simplified zstdcli.c
2016-12-21 09:19:15 +01:00
16ae6563a2
executables use new util.h and platform.h
2016-12-21 09:06:14 +01:00
f8046b8e72
Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112
...
# Conflicts:
# appveyor.yml
2016-12-19 08:20:26 +01:00
b866e72826
tools use platform.h
2016-12-16 14:24:01 +01:00
c71e552b2e
fixed "strategy" in advanced compression parameters
2016-12-13 20:04:32 +01:00
8349d675e0
fileio.c: support advanced compression parameters
2016-12-13 13:24:59 +01:00
7c6bbc3298
updated FIO_passThrough
2016-12-05 18:31:14 +01:00
6b508b1770
updated test-gzstd
2016-12-05 18:02:40 +01:00
b493e3b3d3
introduced srcBufferLoaded
2016-12-05 17:39:38 +01:00
3c69760275
improved FIO_decompressGzFrame
2016-12-05 15:58:23 +01:00
8489f184f6
improved detection of -lz
2016-12-05 13:47:00 +01:00
743b33f57e
fix zstdcat
2016-12-02 15:18:57 -08:00
5bd4237beb
minor refactor
2016-12-02 12:40:57 -08:00
690753ea1a
improved formatting in FIO_decompressSrcFile
2016-12-02 16:20:16 +01:00
4e49580407
removed testing artifacts
2016-12-02 15:19:00 +01:00
c5eebca128
rewritten FIO_decompressGzFile
2016-12-02 15:01:31 +01:00
b0f2ef2119
improved gzip_* functions
2016-12-02 13:50:29 +01:00
4b504f131a
added gzip_open, gzip_close, gzip_read
2016-12-02 13:11:39 +01:00
daaf754539
detect stream with ungetc
2016-12-01 13:29:19 +01:00
19aad42ee1
added FIO_decompressGzFile
2016-12-01 11:56:31 +01:00
abfb51f5f2
gzstd: decompresses .gz files
2016-11-30 15:05:54 +01:00
0e14675df2
fileio.c: detect .gz files
2016-11-30 13:34:21 +01:00
cdff19c4b3
minor comment change
2016-11-11 17:26:54 -08:00
0018ca28dc
zstd cli : displays total decoded size, even when a stream consists of multiple frames
2016-11-07 14:42:37 -08:00
1f60eb6b17
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-11-03 11:38:04 +01:00
26306fcacf
BMK_SetNbIterations renamed to BMK_SetNbSeconds
2016-11-03 11:38:01 +01:00
179b19776f
fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h
...
Added : ZSTD_isFrame() in experimental section
2016-11-02 17:30:49 -07:00
fcf22e3473
set permissions, access and modification times
2016-11-02 14:08:07 +01:00
a42794df61
preserve file modification time
2016-11-02 13:08:39 +01:00
d4cda27b63
new command -M#, to limit memory usage during decompression ( #403 )
2016-10-14 13:32:20 -07:00
3e82f2b61a
Remove references to GPLv2 license
...
License headers added to Makefiles were taken from `zstd/Makefile`.
2016-10-11 15:28:40 -07:00
993060e0f2
cli : better adaptation to small files
2016-09-21 16:46:08 +02:00
43eeea4725
fileio uses ZSTD_resetCStream()
2016-09-15 15:38:44 +02:00
3ecbe6a37c
fileio uses ZSTD_resetDStream()
2016-09-14 17:26:59 +02:00
d77f2ceee6
Merge branch 'dev' of github.com:Cyan4973/zstd into dev
2016-09-12 03:14:03 +02:00
Yann Collet
Yann Collet
Przemyslaw Skibinski
Nick Terrell
Yann Collet