38ba7002f2
fixed minor warning on unused variable in shell function
2017-07-20 18:39:04 -07:00
a90b16e150
Visual blind fix 2
2017-07-20 15:57:55 -07:00
4542d7118c
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-20 15:12:21 -07:00
5e6c5203f3
fixed fuzzer test for non OS-X platforms
2017-07-20 15:11:56 -07:00
57251f52cf
Merge pull request #773 from terrelln/lk-v3
...
[linux] Update patches for v3
2017-07-20 14:47:26 -07:00
7d3ac0710d
[linux] Update patches for v3
2017-07-20 13:33:55 -07:00
b4d460f32c
pool.c : blindfix for Visual warnings
2017-07-20 01:13:14 -07:00
1ca1288689
added --memtest=# command to fuzzer
...
to jump directly to relevant test section
2017-07-19 16:01:16 -07:00
3974d2b38a
blind fix for Windows Multithreading module
...
adds a fake 0 return value for mutex/cond init
2017-07-19 13:33:21 -07:00
b71363b967
check pthread_*_init() success condition
2017-07-19 01:05:40 -07:00
44b0838253
Merge pull request #770 from terrelln/test-mode
...
[zstdcli] Fix -t in streaming mode
2017-07-18 15:40:59 -07:00
2aba13b770
Merge pull request #769 from terrelln/real-block-split
...
[libzstd] Fix bug in Huffman encoding
2017-07-18 14:58:26 -07:00
d0b27483ae
[zstdcli] Fix -t in streaming mode
2017-07-18 14:45:49 -07:00
cc1522351f
[libzstd] Fix bug in Huffman encoding
...
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.
The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.
Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s
Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
2017-07-18 13:20:40 -07:00
77d67fb167
Merge pull request #766 from terrelln/real-block-split
...
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-18 08:26:24 -07:00
14c83b05c7
Merge pull request #765 from terrelln/real-block-split
...
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 19:25:55 -07:00
7a28b9e4a3
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-17 15:29:11 -07:00
3381bf4b84
Merge pull request #764 from terrelln/real-block-split
...
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 14:46:01 -07:00
e198230645
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 12:27:24 -07:00
634f012420
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 11:36:11 -07:00
fa3aa04ccd
Merge pull request #761 from paulcruz74/file-rename
...
renamed pool.c to poolTests.c
2017-07-14 09:09:45 -07:00
3841ee6fb3
Merge pull request #762 from facebook/errorCodes
...
pinned down error code enum values
2017-07-14 09:09:22 -07:00
3b0cff3c33
fixed clang's -Wdocumentation
2017-07-13 18:58:30 -07:00
2bd6440be0
pinned down error code enum values
...
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.
Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
2017-07-13 17:12:16 -07:00
3502426fd4
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-13 15:49:19 -07:00
6733c0777c
updated NEWS regarding #760
2017-07-13 15:34:44 -07:00
830ef4152a
[libzstd] Increase granularity of FSECTable repeat mode
2017-07-13 12:45:39 -07:00
d985319337
Merge pull request #759 from terrelln/real-block-split
...
[libzstd] Pull CTables into sub-structure
2017-07-13 10:24:19 -07:00
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
132e6efd76
switched ZSTDMT_compress_advanced() last argument to overlapLog
...
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
2017-07-13 02:22:58 -07:00
4e77f7761d
clarified comment on ZSTD_p_contentSizeFlag
2017-07-13 02:09:07 -07:00
de0414b736
[libzstd] Pull CTables into sub-structure
2017-07-12 19:49:19 -07:00
d6799635a5
Merge pull request #756 from facebook/memOpt
...
Memory optimisations for ZSTDMT
2017-07-12 18:32:37 -07:00
33cecaaaca
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-12 14:38:36 -07:00
8ef666c325
slightly increased buffer pool, to cover normal "full load" scenarios
...
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
2017-07-12 14:23:34 -07:00
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
2a62f48bf4
release input buffers from inside worker thread
...
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
2017-07-11 15:56:40 -07:00
57236184af
buffer pool : all buffers have same size
...
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
2017-07-11 15:17:25 -07:00
34b2b95631
zstdmt : intermediate outBuffer allocated from within worker
...
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
2017-07-11 14:59:10 -07:00
16261e6951
buffer pool can be invoked from multiple threads
2017-07-11 14:14:07 -07:00
fe80d45807
Merge pull request #755 from jacquesg/solaris
...
Solaris fix
2017-07-11 12:11:44 -07:00
c325c8db82
fix missing symbol 'nanosleep' for Solaris
2017-07-11 19:26:10 +02:00
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
ee3423d709
extended fuzzer MT memory tests
2017-07-10 14:09:16 -07:00
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
f9524cf366
added --memtest to fuzzer
2017-07-10 13:48:41 -07:00
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
Yann Collet
Yann Collet
Nick Terrell
Jacques Germishuys