3502426fd4
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-13 15:49:19 -07:00
6733c0777c
updated NEWS regarding #760
2017-07-13 15:34:44 -07:00
d985319337
Merge pull request #759 from terrelln/real-block-split
...
[libzstd] Pull CTables into sub-structure
2017-07-13 10:24:19 -07:00
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
132e6efd76
switched ZSTDMT_compress_advanced() last argument to overlapLog
...
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
2017-07-13 02:22:58 -07:00
4e77f7761d
clarified comment on ZSTD_p_contentSizeFlag
2017-07-13 02:09:07 -07:00
de0414b736
[libzstd] Pull CTables into sub-structure
2017-07-12 19:49:19 -07:00
d6799635a5
Merge pull request #756 from facebook/memOpt
...
Memory optimisations for ZSTDMT
2017-07-12 18:32:37 -07:00
33cecaaaca
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-12 14:38:36 -07:00
8ef666c325
slightly increased buffer pool, to cover normal "full load" scenarios
...
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
2017-07-12 14:23:34 -07:00
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
2a62f48bf4
release input buffers from inside worker thread
...
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
2017-07-11 15:56:40 -07:00
57236184af
buffer pool : all buffers have same size
...
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
2017-07-11 15:17:25 -07:00
34b2b95631
zstdmt : intermediate outBuffer allocated from within worker
...
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
2017-07-11 14:59:10 -07:00
16261e6951
buffer pool can be invoked from multiple threads
2017-07-11 14:14:07 -07:00
fe80d45807
Merge pull request #755 from jacquesg/solaris
...
Solaris fix
2017-07-11 12:11:44 -07:00
c325c8db82
fix missing symbol 'nanosleep' for Solaris
2017-07-11 19:26:10 +02:00
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
ee3423d709
extended fuzzer MT memory tests
2017-07-10 14:09:16 -07:00
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
f9524cf366
added --memtest to fuzzer
2017-07-10 13:48:41 -07:00
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
0f4fc6c20a
fixed several conversion warnings
2017-07-07 17:13:12 -07:00
ed0243a63c
removed zbufftest from list of `all` tests
2017-07-07 16:16:14 -07:00
9bde061a0b
fixed minor Visual compilation limitation
2017-07-07 16:14:17 -07:00
593d517ebf
fixed minor cast warning
2017-07-07 16:09:47 -07:00
842644e42e
target gpptest uses CXX environment variable
2017-07-07 15:55:41 -07:00
d4dcace34b
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-07 15:53:20 -07:00
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
46396523c0
ZSTD_getFrameHeader : control of windowSize limits is delegated to caller
...
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
2017-07-07 15:32:12 -07:00
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
e622330a3b
extended frameHeader.windowSize to unsigned long long
2017-07-07 14:19:01 -07:00
49d3d4f484
Merge pull request #751 from ivuk/fix-typo
...
Fix typos in README.md
2017-07-06 12:06:43 -07:00
e6e25c9507
Fix typos in README.md
2017-07-06 20:43:14 +02:00
7758ed8458
fixed fullbench, part 2
2017-07-06 02:48:00 -07:00
9b2c1acfc0
fixed fullbench
2017-07-06 02:22:57 -07:00
f04deff4fc
fixed #718 , reported by @GregSlazinski, solution suggested by @mcmilk
2017-07-06 01:42:46 -07:00
d75c0e71c4
minor code refactoring
2017-07-05 18:10:07 -07:00
3345a91964
cli : use new advanced API by default
2017-07-05 17:34:15 -07:00
49af41820d
clarified status of zstdmt_compress.h API
2017-07-05 17:21:13 -07:00
27e883371d
fixed wrong assert() condition
...
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
2017-07-04 19:33:16 -07:00
2cb9774f5e
more precise estimation of amount to flush at end of stream (single thread mode)
...
also : can use DEBUGLEVEL variable in /tests
2017-07-04 12:39:26 -07:00
6383372dec
fixed : 0-copy in NULL is UB
2017-07-04 10:36:41 -07:00
5051dd39ca
Merge pull request #743 from facebook/fullbench
...
compress_generic() automatic optimization opportunities
2017-07-03 21:26:38 -07:00
9ceea19dc1
Merge pull request #747 from facebook/noExit
...
No exit() on decoding errors
2017-07-03 18:15:13 -07:00
2de2396a36
refactor ZSTDMT_compress()
2017-07-03 16:23:36 -07:00
Yann Collet
Yann Collet
Nick Terrell
Jacques Germishuys
Igor Vuk