1dba98d563
introduced parameter ZSTD_p_nonBlockingMode
...
This new parameter makes it possible to call
streaming ZSTDMT with a single thread set
which is non blocking.
It makes it possible for the main thread to do other tasks in parallel
while the worker thread does compression.
Typically, for zstd cli, it means it can do I/O stuff.
Applied within fileio.c, this patch provides non-negligible gains during compression.
Tested on my laptop, with enwik9 (1000000000 bytes) : time zstd -f enwik9
With traditional single-thread blocking mode :
real 0m9.557s
user 0m8.861s
sys 0m0.538s
With new single-worker non blocking mode :
real 0m7.938s
user 0m8.049s
sys 0m0.514s
=> 20% faster
2018-01-16 16:15:47 -08:00
6025465e42
ZSTDMT : minor CCtx memory optimization
...
can be useful when a compression job only has small amount of data to compress.
2018-01-16 15:34:41 -08:00
2e23333094
ZSTDMT can now work in non-blocking mode with 1 thread
...
it still fallbacks to single-thread blocking invocation
when input is small (<1job)
or when invoking ZSTDMT_compress(), which is blocking.
Also : fixed a bug in new block-granular compression routine.
2018-01-16 15:28:43 -08:00
8e83c5c910
Merge branch 'dev' into progressiveMT
2018-01-16 12:54:33 -08:00
f0d703243f
Merge pull request #956 from terrelln/mf-struct
...
Split ZSTD_CCtx into smaller sub-structures
2018-01-16 12:37:44 -08:00
0bc7a75468
Merge pull request #987 from facebook/checkTag
...
Check tag
2018-01-16 11:52:32 -08:00
f4e58455f6
ensure MOREFLAGS are not lost in root->tests Makefile invocation
2018-01-16 11:50:16 -08:00
19cb37f837
travis ci : added gcc-7 test
...
also added `-Werror` to sanitizer tests
2018-01-16 11:40:42 -08:00
aae267a2e1
Reorganize block state
2018-01-16 11:17:50 -08:00
887cd4e35e
Split ZSTD_CCtx into smaller sub-structures
2018-01-16 11:17:50 -08:00
a187486e3c
Merge pull request #990 from krab/dev-cmake-libdir
...
CMake: use GNUInstallDirs for library install dir
2018-01-15 18:55:21 -08:00
403e2db139
CMake: use GNUInstallDirs for library install dir
...
Libraries now will be installed in the correct directory on x86_64 linux systems,
and can be changed with `-DCMAKE_INSTALL_LIBDIR=<dirname>` option.
2018-01-15 22:48:46 +03:00
815eddeda4
added tag-triggered test
...
ensure tag and libzstd version are compatible
2018-01-14 17:06:21 -08:00
f6e17b8c7a
added tests/checkTag
...
compared provided tag with current libzstd version
2018-01-14 17:03:45 -08:00
4792ac6689
make -C tests legacy : minor flag alteration
...
ZSTD_LEGACY_SUPPORT is a macro constant,
so it should be part of CPPFLAGS, instead of CFLAGS
2018-01-14 14:09:17 -08:00
99a0a8bbd2
tests/Makefile : fixed target allnothread
...
ensures MT is disabled
2018-01-13 22:00:05 -08:00
f8a05932b9
added `make list` capability to `tests/Makefile`
2018-01-13 21:54:21 -08:00
9477f6529d
Merge pull request #984 from terrelln/dict-load
...
Load more dictionary positions into table if empty
2018-01-13 13:20:42 -08:00
58ecf13e02
zstdmt : can compress at block granularity
...
offering perspective of more accurate progression report.
2018-01-13 13:18:57 -08:00
9a211d1f05
Load more dictionary positions into table if empty
...
If the hash table is empty load positions into the hash table
that we would otherwise skip.
| Level | Data Set | Improvement |
|-------|--------------|-------------|
| 1 | github | 0.44% |
| 1 | hg-changelog | 0.13% |
| 1 | hg-commands | 1.28% |
| 1 | hg-manifest | 0.70% |
| 3 | github | 0.74% |
| 3 | hg-changelog | 0.87% |
| 3 | hg-commands | 1.74% |
| 3 | hg-manifest | 0.23% |
2018-01-12 16:17:22 -08:00
863b2f8db4
Merge pull request #983 from terrelln/dict-wlog
...
Increase windowLog from CDict based on the srcSize when known
2018-01-12 07:47:43 -08:00
9d65a5c316
Merge pull request #970 from facebook/dubtlazy
...
Delayed update Binary Tree for Lazy strategy
2018-01-12 07:28:59 -08:00
4b7c4e5f41
Add test for cdict window log adjustment
2018-01-11 16:45:16 -08:00
b610b777d3
Increase windowLog from CDict based on the srcSize when known
2018-01-11 16:23:21 -08:00
cacf47cbee
Merge branch 'dev' into dubtlazy
...
and fixed conflicts
2018-01-11 13:25:08 -08:00
04c00f9388
Merge pull request #982 from facebook/fix304
...
Fix for #304 and #977 : error during dictionary creation
2018-01-11 13:20:59 -08:00
b9a14900ff
changed function name to ZSTD_DUBT_findBestMatch()
2018-01-11 12:38:31 -08:00
1d623e60a1
Merge pull request #981 from facebook/fix976
...
fixed bug #976 , reported by @indygreg
2018-01-11 11:40:07 -08:00
752bae4a48
added warning message
...
when pathological dataset is detected
(note : cover_optimize needs -v to display the warning)
2018-01-11 11:29:28 -08:00
e8093dde09
fixed #304
...
Pathological samples may result in literal section being incompressible.
This case is now detected,
and literal distribution is replaced by one that can be written into the dictionary.
2018-01-11 11:16:32 -08:00
218e9fe0fc
added a test case for dictBuilder failure
...
cyclic data set makes the entropy stage fails
now, onto a fix for #304 ...
2018-01-11 09:42:38 -08:00
55cdbdb465
Merge pull request #980 from conor42/dev
...
Fix break condition in decompression noise test
2018-01-11 08:33:35 -08:00
2103a62b3d
fixed minor warning on prototype definition
2018-01-11 04:49:19 -08:00
ff795580f2
fixed bug #976 , reported by @indygreg
...
constants in zstd.h should not depend on MIN() macro which existence is not guaranteed.
Added a test to check the specific constants.
The test is a bit too specific.
But I have found no way to control a more generic "are all macro already defined" condition,
especially as this is a valid construction (the missing macro might be defined later, intentionnally).
2018-01-10 20:33:45 -08:00
0e88f6e97b
Fix break condition in decompression noise test
...
The bug prevents noise being added
2018-01-11 11:42:58 +10:00
06995775b0
Merge pull request #979 from facebook/memdoc
...
Update API documentation related to memory management capabilities
2018-01-10 17:39:37 -08:00
1edf33764e
Merge pull request #974 from terrelln/dstfile
...
[fileio] Improve safety of output file modifications
2018-01-10 19:02:48 +01:00
292eeb672f
api doc : grouped all ZSTD_create*_advanced() functions together
...
in a new "custom memory allocator" paragraph
which is itself part of "memory management" category.
This makes it simpler to see the relation between the type and its usages.
2018-01-10 09:07:47 -08:00
3ea156368c
API doc : grouped ZSTD_initStatic*() together
...
within "memory management" category.
2018-01-10 08:49:50 -08:00
b17fb488b0
fixed msan test
...
a pointer calculation was wrong in a corner case
2018-01-06 20:50:36 +01:00
76321d6bbc
Merge pull request #965 from facebook/tests_obj
...
%.o objects files in /tests
2018-01-06 13:15:53 +01:00
658d6b8588
Merge branch 'dev' into dubtlazy
2018-01-06 12:40:58 +01:00
a927fae2a1
fixed ZSTD_reduceIndex()
...
following suggestions from @terrelln.
Also added some comments to present logic behind ZSTD_preserveUnsortedMark().
2018-01-06 12:31:26 +01:00
3d578eef0c
minor tests/Makefile change, as suggested by @terrelln
...
https://github.com/facebook/zstd/pull/965#discussion_r160008006
2018-01-06 07:13:09 +01:00
752880ffed
Merge pull request #963 from facebook/benchfix
...
fix: bench can accept hlog custom parameter
2018-01-06 06:57:02 +01:00
ed9611dc62
[fileio] Don't call FIO_remove() on stdout or /dev/null
2018-01-05 11:50:24 -08:00
282ad05e0a
[fileio] Use FIO_remove() everywhere for safety
2018-01-05 11:44:45 -08:00
fd63140e1c
[util] Refuse to set file stat on non-regular file
2018-01-05 11:44:25 -08:00
dce386f658
Merge pull request #972 from pixelb/bz1530049
...
zstd: fix crash when not overwriting existing files
2018-01-04 12:14:50 +01:00
f51e861665
Merge pull request #973 from terrelln/test-case
...
Add test case for PR #972
2018-01-04 12:14:36 +01:00
Yann Collet
Yann Collet
Nick Terrell
Alexey Ivanov
conor42