b426bcc097
[zstdmt] Fix jobsize bugs ( #1205 )
...
[zstdmt] Fix jobsize bugs
* `ZSTDMT_serialState_reset()` should use `targetSectionSize`, not `jobSize` when sizing the seqstore.
Add an assert that checks that we sized the seqstore using the right job size.
* `ZSTDMT_compressionJob()` should check if `rawSeqStore.seq == NULL`.
* `ZSTDMT_initCStream_internal()` should not adjust `mtctx->params.jobSize` (clamping to MIN/MAX is okay).
2018-06-25 15:21:08 -07:00
3b53bfe4f3
Merge pull request #1200 from felixhandte/zstd-attach-dict-pref
...
Add CCtx Param Controlling Dict Attachment Behavior
2018-06-25 12:42:31 -07:00
59bb5f7d58
Merge pull request #1204 from facebook/noForwardProgress
...
Error code on no forward progress
2018-06-22 20:25:29 -07:00
31769ce702
error on no forward progress
...
streaming decoders, such as ZSTD_decompressStream() or ZSTD_decompress_generic(),
may end up making no forward progress,
(aka no byte read from input __and__ no byte written to output),
due to unusual parameters conditions,
such as providing an output buffer already full.
In such case, the caller may be caught in an infinite loop,
calling the streaming decompression function again and again,
without making any progress.
This version detects such situation, and generates an error instead :
ZSTD_error_dstSize_tooSmall when output buffer is full,
ZSTD_error_srcSize_wrong when input buffer is empty.
The detection tolerates a number of attempts before triggering an error,
controlled by ZSTD_NO_FORWARD_PROGRESS_MAX macro constant,
which is set to 16 by default, and can be re-defined at compilation time.
This behavior tolerates potentially existing implementations
where such cases happen sporadically, like once or twice,
which is not dangerous (only infinite loops are),
without generating an error, hence without breaking these implementations.
2018-06-22 17:58:21 -07:00
3934e010a2
Merge pull request #1197 from facebook/poolResize
...
Thread Pool resize
2018-06-22 14:20:07 -07:00
fbd5dfc1b1
changed POOL_resize() return type to int
...
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
2018-06-22 12:14:59 -07:00
d70c4a5074
Merge pull request #1201 from facebook/rfcUpdate
...
updated Zstandard frame format
2018-06-22 11:53:50 -07:00
1d5648ca10
Merge pull request #1196 from felixhandte/zstd-btopt-in-place-dict
...
ZSTD_btopt: Support Searching the Dictionary Context In-Place
2018-06-22 11:53:23 -07:00
f6242d30b7
Merge pull request #1202 from facebook/barelyCompressible
...
Increase threshold detection of poorly compressible data
2018-06-22 11:52:52 -07:00
698fd00afb
huf: increase threshold detection of poorly compressible data
2018-06-21 18:32:38 -07:00
c1e6347717
fixed minor typos, detected by @terrelln
2018-06-21 18:08:11 -07:00
243cd9d8bb
add a cond_broadcast after resize
...
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
2018-06-21 18:04:58 -07:00
7639db939f
updated Zstandard frame format
...
adding clarifications from IETF RFC DISCUSS.
2018-06-21 17:55:55 -07:00
818e72b4d5
added extended POOL test
...
abrupt end + downsizing with running jobs remaining in queue.
also : POOL_resize() requires numThreads >= 1
2018-06-21 14:58:59 -07:00
01bb1c1016
Add CCtx Param Controlling Dict Attachment Behavior
2018-06-21 17:29:25 -04:00
3e91dc4d6a
Add Repcode Bounds Check
2018-06-21 15:54:41 -04:00
5bd3d4b7d2
Add Debug Log Statement
2018-06-21 15:54:07 -04:00
3caba150c6
Fix `dmsBtLow` Test
2018-06-21 15:53:40 -04:00
7d80ada5ca
added a test for POOL (multithreading)
...
ensuring all jobs in queue are nonetheless completed
when POOL is instructed to end abruptly (POOL_free())
2018-06-21 12:24:36 -07:00
5da9bbc38e
Conceivably Dedup ZSTD_noDict and ZSTD_dictMatchState _insertBt1 Impls
...
By reverting to the bool extDict flag, we call ZSTD_insertBt1 with the same
const args in both non-extDict dictModes.
2018-06-21 11:20:01 -04:00
6de249c1c6
fixed: bug when counting nb of active threads
...
when queueSize > 1
also : added a test in testpool.c
verifying resizing is effective.
2018-06-20 18:28:49 -07:00
6b48eb12c0
change control of threadLimit
...
now limits maximum nb of active threads
even when queueSize > 1.
2018-06-20 14:35:39 -07:00
5d81f71e83
Consistency in Guarding DMS-Only Variable Initializations
2018-06-20 16:54:53 -04:00
9c14eafe3d
Also Use `matchLow` for HC3 Match
2018-06-20 15:51:14 -04:00
0a6cf7cd1d
Minor Changes
2018-06-20 15:27:23 -04:00
ae1f3898a2
Remove Dead(!) HC3 DMS Lookup
2018-06-20 15:27:12 -04:00
93702a7a62
Merge pull request #1198 from facebook/msdebug
...
made Visual Studio compatible with DEBUGLEVEL >= 2
2018-06-20 12:26:31 -07:00
ae0b7ffa0a
made Visual Studio compatible with DEBUGLEVEL >= 2
2018-06-20 09:45:02 -07:00
62469c9f41
fixed wrong size in pthread struct transfer
2018-06-19 20:14:03 -07:00
166901dc72
reduced POOL_resize() restriction
...
It's not necessary to ensure that no job is ongoing.
The pool is only expanded, existing threads are preserved.
In case of error, the only option is to return NULL and terminate the thread pool anyway.
2018-06-19 18:07:18 -07:00
066fbbfe1c
make zstdmt resize its context
...
when nbThreads change.
Technically, it only expands.
But when instructed to use less threads,
the thread pool will limit nb of concurrent threads.
2018-06-19 17:28:56 -07:00
4567c57199
finalized POOL_resize()
...
POOL_ctx* POOL_resize(POOL_ctx* ctx, size_t numThreads)
The function may fail, and returns a NULL pointer in this case.
2018-06-19 16:03:12 -07:00
6768cf53fd
Merge pull request #1190 from terrelln/ldm-adjust
...
Adjust advanced parameters to source size
2018-06-19 14:40:56 -07:00
c0b6ce95b1
Merge pull request #1179 from supertopher/dev
...
Improves UX for --list command's lack of support for pipes
2018-06-19 14:36:30 -07:00
03c39c540b
Fix Incorrect Param
2018-06-19 15:36:33 -04:00
de639502aa
Update Dict Attachment Cut-Offs
2018-06-19 15:36:13 -04:00
f0a13bcd68
Make Sure Position 0 Gets Into the Tree
2018-06-19 15:10:06 -04:00
cb45b41f16
Merge pull request #1194 from facebook/no-opaqueapi
...
removed specific --opaqueapi test
2018-06-19 10:24:26 -07:00
87fe4788a3
Fix Compression Ratio Regression #1
2018-06-19 13:01:21 -04:00
4bb79f9c55
Misc Changes
2018-06-19 13:01:21 -04:00
2091f34e9e
Find Proper Matches
2018-06-19 13:01:21 -04:00
64348a15f1
Misc Fixes
2018-06-19 13:01:21 -04:00
ade8586ce6
Find `mls == 3` Matches
2018-06-19 13:01:21 -04:00
ce743312e2
Fix Typo
2018-06-19 13:01:21 -04:00
a075864756
Switch `!= ZSTD_extDict` to `== ZSTD_noDict`
2018-06-19 13:01:21 -04:00
1e03377bde
Implement RepCode Check
2018-06-19 13:01:21 -04:00
ccbf067973
Add _dictMatchState Functions
2018-06-19 13:01:21 -04:00
d5d8240967
Convert `extDict` Flag to `dictMode` Enum
2018-06-19 13:01:21 -04:00
93c3184d44
Attach Dicts when Using ZSTD_btopt and ZSTD_btultra
2018-06-19 13:01:21 -04:00
5bac1db28f
Tests to verify piped input to `--list` exits 1
...
I'm following the pattern that i saw in the rest of the test file
please tell me if i am using the wrong conventions
2018-06-19 09:56:37 -07:00
Nick Terrell
Yann Collet
Yann Collet
W. Felix Handte
cyan4973
Topher Lubaway