eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
eeff55dfa8
Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev
2017-09-06 15:56:32 -07:00
767a0b3be1
Move ldm hashLog, bucketLog, and mml to cctxParams
2017-09-01 12:24:59 -07:00
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
369c29dd1a
fixed impact of merge conflict for longRange
2017-08-31 18:25:56 -07:00
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
623e3cd40b
Use ZSTD_dm_rawContent in zstdmt_compress
2017-08-29 18:04:32 -07:00
4e835720bf
Delay creation of ZSTDMT_CCtx
2017-08-29 17:58:32 -07:00
c7a18b7c21
Localize 'dictMode' from cctx to function param
2017-08-29 15:52:24 -07:00
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
024098a47d
Fix parameter retrieval from cdict
2017-08-25 17:58:28 -07:00
2adde898c8
Fix typo with ZSTDMT_parameter
2017-08-25 16:13:40 -07:00
18224608ff
Remove ZSTD_setCCtxParameter()
2017-08-25 13:58:41 -07:00
0744592d38
Add function initializing cctxParams from clevel
2017-08-25 13:36:47 -07:00
9911153723
Move jobSize and overlapLog in zstdmt to cctxParams
2017-08-25 13:14:51 -07:00
de5193422d
Distinguish between jobParams and cctxParams in zstdmt
2017-08-25 11:36:17 -07:00
eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
db3f5372df
[zstdmt] Use POOL_create_advanced()
2017-08-24 18:12:28 -07:00
2fbf0285b2
Fix interaction with ZSTD_setCCtxParameter() and cleanup
2017-08-24 11:25:41 -07:00
fd9bf42516
Fix forceWindow and dictMode setting for zstdmt jobs
2017-08-23 19:16:57 -07:00
bf3108fb50
Ensure zstdmt uses 'job version' of cctx parameters
2017-08-23 17:03:31 -07:00
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
23fc0e41fa
Remove 'opaque' naming from internal functions
2017-08-22 14:24:47 -07:00
8fd1636776
Remove unused functions
2017-08-22 13:33:58 -07:00
60e1bc617c
Explicitly create a job cctxParam for multithreading
2017-08-21 15:39:37 -07:00
5b956f4753
Comment out CCtx_param versions of CDict functions
2017-08-21 14:49:16 -07:00
502031ca10
Use cctxParam version of createCDict internally
2017-08-21 11:00:44 -07:00
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
7db552676e
reduced pool queue to 0 to save memory
...
fixed : pool performance when jobs are fires fast and queueSize==0
2017-08-19 15:07:54 -07:00
d775519296
Add cctxParam versions of internal functions
2017-08-18 17:37:58 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
63b8c98531
Pass cctx parameters to MTCtx
2017-08-18 16:17:24 -07:00
b71363b967
check pthread_*_init() success condition
2017-07-19 01:05:40 -07:00
2bd6440be0
pinned down error code enum values
...
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.
Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
2017-07-13 17:12:16 -07:00
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
132e6efd76
switched ZSTDMT_compress_advanced() last argument to overlapLog
...
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
2017-07-13 02:22:58 -07:00
8ef666c325
slightly increased buffer pool, to cover normal "full load" scenarios
...
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
2017-07-12 14:23:34 -07:00
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
2a62f48bf4
release input buffers from inside worker thread
...
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
2017-07-11 15:56:40 -07:00
57236184af
buffer pool : all buffers have same size
...
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
2017-07-11 15:17:25 -07:00
34b2b95631
zstdmt : intermediate outBuffer allocated from within worker
...
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
2017-07-11 14:59:10 -07:00
16261e6951
buffer pool can be invoked from multiple threads
2017-07-11 14:14:07 -07:00
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
Stella Lau
Yann Collet
Yann Collet
Nick Terrell