5d9b894e46
Fixed status display for zstdmt
...
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.
This patch disables the displaying of % compression
when multi-threading is enabled.
It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.
There is no modification to decompression side,
since decompression is only single-threaded for now.
2017-01-27 13:30:18 -08:00
2fe9126591
Add multithread support to COVER
2017-01-27 11:56:02 -08:00
9c018cc140
Add BUCK files for Nuclide support
2017-01-27 10:43:12 -08:00
eb2d23a90c
improved #232 fix
2017-01-27 10:43:03 -08:00
92a4dbf2e4
Fixed https://github.com/facebook/zstd/issues/232
2017-01-27 10:43:03 -08:00
83c387eb8e
Fix zstdmt_compress.h include
2017-01-26 15:25:32 -08:00
717c65d690
Merge pull request #519 from inikep/dev11
...
Dev11
2017-01-26 14:23:44 -08:00
8dafb1acf5
CLI : automatically set overlap size to max (windowSize) for max compression level
2017-01-25 17:01:13 -08:00
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
943cff9c37
fixed zstdmt cli freeze issue with large nb of threads
...
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
2017-01-25 12:35:19 -08:00
5022a18d51
improved #232 fix
2017-01-25 13:11:26 +01:00
64fa2dbc5e
Fixed https://github.com/facebook/zstd/issues/232
2017-01-25 13:02:33 +01:00
512cbe8c10
zstdmt cli and API allow selection of section sizes
...
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.
It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()
The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
2017-01-24 17:08:53 -08:00
f8804d1014
convert tabs to space
...
joys of using multiple editors from multiple environments ...
2017-01-20 17:23:19 -08:00
d7e3cb58c5
Resolved merge conflict dev+zstdmt
2017-01-20 16:44:50 -08:00
5fba09fa41
updated util's time for Windows compatibility
...
Correctly measures time on Posix systems when running with
Multi-threading
Todo : check Windows measurement under multi-threading
2017-01-20 12:57:31 -08:00
458c8a94b4
minor refactoring : cleaner MT integration within bench
2017-01-19 17:44:15 -08:00
500014af49
zstd cli can now compress using multi-threading
...
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads() (fileio)
2017-01-19 17:04:28 -08:00
0f984d94c4
changed MT enabling macro to ZSTD_MULTITHREAD
2017-01-19 14:05:07 -08:00
f22adae984
fixed minor warning (unused variable) in fuzzer
2017-01-19 13:46:30 -08:00
736788f8e8
added streaming fuzzer tests for MT API
...
Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize
which is possible when many flushes are issued
while there is not enough dst buffer to flush completed ones.
2017-01-19 12:15:29 -08:00
8a0bc30a2d
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2017-01-17 13:02:29 +01:00
d72f4b6b7a
added "Makefile is validated"
2017-01-17 12:40:06 +01:00
5eb749e734
ZSTDMT_compress() creates a single frame
...
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()
As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().
This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
2017-01-11 18:21:25 +01:00
c220d4c74d
Use COVER_MEMMULT when training with COVER.
2017-01-09 16:49:04 -08:00
56958500fc
minor man page update
2017-01-09 19:47:09 +01:00
3a1fefcf00
Simplify COVER parameters
2017-01-02 17:51:38 -08:00
df8415c502
Add COVER to the zstd cli
2017-01-02 14:43:08 -08:00
6334b04d61
compile object files, for faster recompilation
2017-01-02 03:22:18 +01:00
747452677d
fixed cmake tests
2017-01-02 02:05:45 +01:00
cdb2763f4a
new Makefile target zstdmt
2017-01-02 01:43:56 +01:00
f1cb55192c
fixed linux warnings
2017-01-02 01:11:55 +01:00
0ec6a95ba1
minor fixes
2017-01-02 00:49:42 +01:00
c6a6417458
bench correctly measures time for multi-threaded compression (posix only)
2016-12-31 03:31:26 +01:00
f765a375a5
Merge pull request #504 from terrelln/thread-pool
...
[zstdmt] Add thread pool
2016-12-30 15:31:49 +01:00
e70912c72b
Changed : input divided into roughly equal parts.
...
Debug : can measure time waiting for mutexes to unlock.
2016-12-29 01:24:01 +01:00
8d7432914f
Merge pull request #503 from inikep/dev11
...
Dev11
2016-12-28 16:50:39 +01:00
ab7a579180
added -T command , to set nb of threads
2016-12-28 16:11:09 +01:00
75f3a3a335
changed default PREFIX and MANDIR
2016-12-28 12:32:41 +01:00
3d93f2fce7
first zstdmt sketch
2016-12-27 07:19:36 +01:00
c07d2e3a31
Merge pull request #499 from inikep/dev11
...
improved *BSD and Solaris compatibility
2016-12-23 21:32:03 +01:00
63b0014b96
BSD: improved "make install"
2016-12-23 10:05:49 +01:00
fce374a100
zstdless: add shebang and quote $@
2016-12-22 17:40:10 -05:00
b999170311
Solaris: working "make -C lib install"
2016-12-22 20:14:37 +01:00
aab442133d
Solaris: working "make -C programs install"
2016-12-22 19:26:01 +01:00
8333106b8a
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-12-21 16:44:24 +01:00
0d7e84899f
Merge pull request #489 from inikep/v112
...
improved detection of POSIX
2016-12-21 16:42:46 +01:00
9518744025
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-12-21 16:20:41 +01:00
1f57c2ed32
added : ZSTD_createCDict_byReference()
2016-12-21 16:20:11 +01:00
7a8a03c20d
util.h: restore BSD license for Facebook Open-Source
2016-12-21 15:08:44 +01:00
97a258d71d
updated comments
2016-12-21 14:00:41 +01:00
e679741b18
_CRT_SECURE_NO_WARNINGS moved to util.h
2016-12-21 13:47:11 +01:00
2f6ccee6af
platform.h: removed Compiler Options
2016-12-21 13:23:34 +01:00
101df4f636
fixed Visual Studio compilation
2016-12-21 11:43:11 +01:00
a35b9448ac
improved MinGW support
2016-12-21 11:18:45 +01:00
5736db219e
fix basic types redefinition
2016-12-21 09:26:00 +01:00
20b089e53d
simplified zstdcli.c
2016-12-21 09:19:15 +01:00
16ae6563a2
executables use new util.h and platform.h
2016-12-21 09:06:14 +01:00
ead350bdc0
improved util.h and platform.h
2016-12-21 09:04:59 +01:00
1c1db6b845
windres updated to v1.1.3
2016-12-20 11:21:26 +01:00
f8046b8e72
Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112
...
# Conflicts:
# appveyor.yml
2016-12-19 08:20:26 +01:00
1496c3dc47
Fix : size estimation when some samples are very large
2016-12-18 11:58:23 +01:00
d46ecb58a5
added dll compilation tests
2016-12-17 16:28:12 +01:00
0b37205098
util.h: minor improvement
2016-12-16 17:12:23 +01:00
b0f3663edc
imporved support for POSIX-type OSes
2016-12-16 15:41:18 +01:00
3cdfe266cf
use PLATFORM_POSIX_VERSION
2016-12-16 15:00:50 +01:00
b0e670a054
util.h uses platform.h
2016-12-16 14:25:12 +01:00
b866e72826
tools use platform.h
2016-12-16 14:24:01 +01:00
b3843afcf5
introduced platform.h
2016-12-16 14:13:15 +01:00
25314428c9
zstd.1: added advanced compression options
2016-12-14 16:10:13 +01:00
c71e552b2e
fixed "strategy" in advanced compression parameters
2016-12-13 20:04:32 +01:00
98ef0f98df
fixed conversion warning
2016-12-13 14:52:21 +01:00
8349d675e0
fileio.c: support advanced compression parameters
2016-12-13 13:24:59 +01:00
897b8bb5eb
bench.c: support advanced compression parameters
2016-12-13 13:03:41 +01:00
334b8962b0
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-12-13 12:19:19 +01:00
671f28d1e5
added parseCompressionParameters
2016-12-13 12:18:07 +01:00
83cc2fb083
Enable using isatty() and nanosleep() on *BSD
2016-12-12 19:24:51 +01:00
4c6ce5a57e
Merge pull request #476 from inikep/dev11
...
Dev11
2016-12-12 16:44:55 +01:00
c855da7533
allow all operationg systems with _POSIX_C_SOURCE >= 200112L
2016-12-12 15:37:43 +01:00
242c036875
turn on the '-r' option for HP-UX and AIX
2016-12-12 11:59:17 +01:00
c01ac6c30a
turn on the '-r' option for *BSD and Solaris
2016-12-12 11:23:21 +01:00
fda539f50b
minor coding style changes
2016-12-12 01:03:23 +01:00
128acb35db
improved MSYS support
2016-12-08 10:49:59 +01:00
13de72e0cc
programs\Makefile: use Linux paths
2016-12-08 10:43:55 +01:00
2a826e592b
compatibility with zstd-frugal (noBench mode)
2016-12-06 17:56:20 -08:00
e63c631aaf
decode benchmark, multi-files
2016-12-06 17:46:49 -08:00
d946501d2c
decode benchmark - single file (hidden option)
2016-12-06 16:49:23 -08:00
6b9a983261
changed gzstd build messages
2016-12-06 11:23:25 -08:00
7c6bbc3298
updated FIO_passThrough
2016-12-05 18:31:14 +01:00
6b508b1770
updated test-gzstd
2016-12-05 18:02:40 +01:00
b493e3b3d3
introduced srcBufferLoaded
2016-12-05 17:39:38 +01:00
3c69760275
improved FIO_decompressGzFrame
2016-12-05 15:58:23 +01:00
8489f184f6
improved detection of -lz
2016-12-05 13:47:00 +01:00
db85a6e09a
added zstdgrep
2016-12-02 15:57:07 -08:00
efaf104b2d
added zstdless
2016-12-02 15:24:40 -08:00
743b33f57e
fix zstdcat
2016-12-02 15:18:57 -08:00
5bd4237beb
minor refactor
2016-12-02 12:40:57 -08:00
690753ea1a
improved formatting in FIO_decompressSrcFile
2016-12-02 16:20:16 +01:00
4e49580407
removed testing artifacts
2016-12-02 15:19:00 +01:00
c5eebca128
rewritten FIO_decompressGzFile
2016-12-02 15:01:31 +01:00
b0f2ef2119
improved gzip_* functions
2016-12-02 13:50:29 +01:00
4b504f131a
added gzip_open, gzip_close, gzip_read
2016-12-02 13:11:39 +01:00
daaf754539
detect stream with ungetc
2016-12-01 13:29:19 +01:00
19aad42ee1
added FIO_decompressGzFile
2016-12-01 11:56:31 +01:00
166830ed0a
autodetect -lz
2016-11-30 16:43:07 +01:00
abfb51f5f2
gzstd: decompresses .gz files
2016-11-30 15:05:54 +01:00
0e14675df2
fileio.c: detect .gz files
2016-11-30 13:34:21 +01:00
167c494748
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-11-29 14:05:15 -08:00
4f5350f610
long matches support overflow
2016-11-29 13:12:24 -08:00
fd0ac93024
bench.c: use ZSTD_maxCLevel()
2016-11-23 21:45:29 +01:00
5ddcd9d9ae
bench.c: fixed MAX_CLEVEL
2016-11-21 16:37:56 +01:00
5a17223691
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-11-18 11:47:01 +01:00
2558b4cdbc
bench.c without dict uses ZSTD_compressCCtx
2016-11-18 11:46:30 +01:00
3d18088b38
updated windres
2016-11-17 18:04:41 +01:00
52afb3993e
zbuff API now generates deprecation warnings
2016-11-16 08:50:54 -08:00
cdff19c4b3
minor comment change
2016-11-11 17:26:54 -08:00
0018ca28dc
zstd cli : displays total decoded size, even when a stream consists of multiple frames
2016-11-07 14:42:37 -08:00
d0815583d9
Changed stdinmark and stdoutmark
2016-11-04 11:37:27 +01:00
38b590ad69
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
...
# Conflicts:
# lib/Makefile
2016-11-04 10:10:54 +01:00
407a11f63e
fixed Visual compatibility
2016-11-03 15:52:01 -07:00
3a415594b1
fixed MinGW compilation
2016-11-03 12:59:20 +01:00
1f60eb6b17
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-11-03 11:38:04 +01:00
26306fcacf
BMK_SetNbIterations renamed to BMK_SetNbSeconds
2016-11-03 11:38:01 +01:00
b40884f43d
preserve file modification time for Visual C++
2016-11-03 09:54:53 +01:00
7347869fb6
fixed make install
2016-11-02 22:28:37 -07:00
179b19776f
fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h
...
Added : ZSTD_isFrame() in experimental section
2016-11-02 17:30:49 -07:00
f3f13211ae
Fix #419 : no warning when setting custom LDFLAGS
2016-11-02 17:02:45 -07:00
fcf22e3473
set permissions, access and modification times
2016-11-02 14:08:07 +01:00
a42794df61
preserve file modification time
2016-11-02 13:08:39 +01:00
d872b64f52
added UTIL_setModificationTime, UTIL_getModificationTime
2016-11-02 12:52:20 +01:00
3d197d4c8c
updated man page
2016-10-31 18:10:32 -07:00
b9550d6d43
fixed benchmark mode typo .
...
added benchmark mode tests
2016-10-28 14:43:24 -07:00
22de81e87e
updated man page
2016-10-28 13:58:31 -07:00
bd0bd0fb65
generate RES files during each compilation
2016-10-28 18:15:34 +02:00
589a8f27b7
generate_res.bat moved to programs/windres
2016-10-28 13:17:35 +02:00
76d66dffd0
generate_res moved to programs/windres
2016-10-28 13:16:06 +02:00
347903cfb5
Windows resources moved to programs/windres/
2016-10-25 10:34:41 +02:00
a7a4690b0a
use implicit rules to compile zstd_decompress.c
2016-10-18 16:01:03 -07:00
33fdd099bb
fix command line interpretation
2016-10-17 17:48:48 -07:00
70077bc9bb
refactor for long commands
2016-10-14 14:41:17 -07:00
d7b120ab5c
added long commands --memory= and --memlimit-decompress=
2016-10-14 14:22:32 -07:00
1122349ac2
added long comment --memlimit=
2016-10-14 14:07:11 -07:00
d4cda27b63
new command -M#, to limit memory usage during decompression ( #403 )
2016-10-14 13:32:20 -07:00
e19111c42f
make creates libzstd binaries ( #415 )
2016-10-12 11:09:36 -07:00
3e82f2b61a
Remove references to GPLv2 license
...
License headers added to Makefiles were taken from `zstd/Makefile`.
2016-10-11 15:28:40 -07:00
23cec875cd
fixed Makefile targets: zstd-small, zstd-decompress, zstd-compress
2016-10-05 11:56:22 +02:00
2b4de225e1
Don't redefine macro in util.h
2016-09-22 18:02:39 -07:00
5b2c0dbed0
Add include guards to datagen.h
2016-09-22 17:12:50 -07:00
993060e0f2
cli : better adaptation to small files
2016-09-21 16:46:08 +02:00
714464f05d
fixed : cli : forgotten mandatory argument
2016-09-21 16:05:03 +02:00
27b5ac666e
Implemented "command must be followed by argument" protection
...
suggested by @terrelln (#375 )
2016-09-21 14:31:22 +02:00
0977f7ece6
minor refactor for clarity
2016-09-21 12:24:43 +02:00
dbe70bad48
completed change from projects to build
2016-09-19 15:08:43 +02:00
60038948e6
added -- command in help
2016-09-16 18:52:52 +02:00
3e70cf8036
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev
2016-09-15 20:41:40 +02:00
6173931868
fixed memory leak reported by bryongloden
2016-09-15 18:58:18 +02:00
a6bdf55759
fixed memory leak
2016-09-15 17:02:06 +02:00
43eeea4725
fileio uses ZSTD_resetCStream()
2016-09-15 15:38:44 +02:00
0b01a8ef09
Merge pull request #366 from inikep/dev
...
zstd.exe has FileVersion and ProductVersion with 32-bit gcc (MinGW)
2016-09-14 19:42:29 +02:00
3ecbe6a37c
fileio uses ZSTD_resetDStream()
2016-09-14 17:26:59 +02:00
c03f15e89d
zstd.exe has FileVersion and ProductVersion with 32-bit gcc (MinGW)
2016-09-14 16:16:24 +02:00
64a84edef5
added -r support for Mac OS-X
2016-09-13 17:54:37 +02:00
c4cc9bf973
-r generates an error on systems which do not support it
2016-09-13 17:50:08 +02:00
9f25fcf804
fixed precision warnigns
2016-09-13 16:38:54 +02:00
362708d4d2
zstd.exe has FileVersion and ProductVersion
2016-09-13 13:53:43 +02:00
1c5ba8a5e7
util.h: removed dependency from PATH_MAX
2016-09-13 13:13:10 +02:00
a3481d6de0
make uninstall
2016-09-12 05:04:26 +02:00
d77f2ceee6
Merge branch 'dev' of github.com:Cyan4973/zstd into dev
2016-09-12 03:14:03 +02:00
5c6d244973
Merge branch 'dev' of github.com:facebook/zstd into dev
2016-09-07 14:54:54 +02:00
ac8bace6b1
support large skippable frames
2016-09-07 14:54:23 +02:00
03d3f238de
minor comment
2016-09-07 07:01:33 +02:00
aad9fe5470
don't remove() /dev/null ( #316 )
2016-09-07 07:00:08 +02:00
7ae67bb18a
small compression speed gains with using_CDict
2016-09-06 06:28:05 +02:00
b85cdabd50
Enable install targets for NetBSD.
2016-09-04 14:37:57 +02:00
33a0465a51
fixed a few links
2016-09-02 22:11:49 -07:00
5b8c024716
fixed zstd-pgo ( #329 ) reported by @octoploid
2016-09-02 09:44:00 -07:00
60181e3aaf
zstd cli correctly detects console on Mac OS-X
2016-09-01 17:14:02 -07:00
c843142ad0
zstd -d writes to stdout when input is stdin
2016-09-01 15:05:57 -07:00
1c59c20903
removed redundant files
2016-08-31 07:15:44 -07:00
599c69d917
minor Makefile updates
2016-08-30 13:33:20 -07:00
4ded9e591c
added boilerplate
2016-08-30 11:06:28 -07:00
09c3c8e885
Merge pull request #307 from inikep/dev08
...
updated README.md
2016-08-29 16:32:33 -07:00
6416b0d705
updated README.md
2016-08-29 13:04:26 +02:00
54caf6893e
Removed fileio_legacy; now legacy supports relies entirely on streaming API.
...
CLI legacy support now limited to v0.4+
2016-08-28 19:20:13 -07:00
9097f7b156
fixed versionsTest
2016-08-28 16:25:56 -07:00
1c69baa1a9
CLI supports Legacy formats v0.4+
2016-08-28 12:47:17 -07:00
5f53b0335e
fixed continuation context
2016-08-28 10:00:49 -07:00
ab267e772c
removed fileio specific legacy support;
...
now depends on ZSTD_decompressStream() transparent legacy support
2016-08-28 08:46:25 -07:00
87c18b2ebd
fixed multiple minor warnings for XCode
2016-08-26 01:43:47 +02:00
0baa64a763
increased maximum memory size for 64-bits bench to 16 GB
2016-08-25 22:54:13 +02:00
de9d13068f
updated zstd_compression_format.md
2016-08-25 14:59:08 +02:00
0132375ba6
improved zstd_compression_format.md
2016-08-25 12:20:38 +02:00
637d33566d
minor README.md corrections
2016-08-25 10:42:49 +02:00
ab2f7708c1
created programs\README.md
2016-08-25 10:07:20 +02:00
e416e30019
remove unnecessary comments
2016-08-24 17:32:09 +02:00
7bcdbcf51d
Merge remote-tracking branch 'refs/remotes/Cyan4973/dev' into Other
2016-08-24 17:09:19 +02:00
d1733f7417
fixed crc bug in rare timing conditions within bench.c
2016-08-21 01:04:46 +02:00
c629776b31
make clean will remove decompress/zstd_decompress.o
2016-08-19 10:45:59 +02:00
b571a0eab4
Merge pull request #298 from inikep/Other
...
Other
2016-08-18 17:14:03 +02:00
b6514d2496
improved .gitignore
2016-08-18 16:53:29 +02:00
Yann Collet
Nick Terrell
Przemyslaw Skibinski
Yann Collet
cyan4973
Andrew Janke
Dimitry Andric
Thomas Klausner