121aa2c388
Merge pull request #1211 from facebook/staticAssert
...
updated DEBUG_STATIC_ASSERT()
2018-06-27 12:19:17 -07:00
4489daec09
slightly adjusted default-distribution threshold
...
depending on strategy.
fast favors faster compression and decompression speeds.
2018-06-26 20:10:45 -07:00
ff773bfcde
zeroise freq table with memset()
...
improves decoding speed by ~5% in github_users sample set
2018-06-26 17:24:41 -07:00
7b9bbf77c9
switched to a sizeof() version
...
avoid -Werror=unused-variable issue
2018-06-26 14:08:35 -07:00
f98ec46979
updated DEBUG_STATIC_ASSERT()
...
following suggestion from #1209
2018-06-26 12:04:59 -07:00
b426bcc097
[zstdmt] Fix jobsize bugs ( #1205 )
...
[zstdmt] Fix jobsize bugs
* `ZSTDMT_serialState_reset()` should use `targetSectionSize`, not `jobSize` when sizing the seqstore.
Add an assert that checks that we sized the seqstore using the right job size.
* `ZSTDMT_compressionJob()` should check if `rawSeqStore.seq == NULL`.
* `ZSTDMT_initCStream_internal()` should not adjust `mtctx->params.jobSize` (clamping to MIN/MAX is okay).
2018-06-25 15:21:08 -07:00
3b53bfe4f3
Merge pull request #1200 from felixhandte/zstd-attach-dict-pref
...
Add CCtx Param Controlling Dict Attachment Behavior
2018-06-25 12:42:31 -07:00
31769ce702
error on no forward progress
...
streaming decoders, such as ZSTD_decompressStream() or ZSTD_decompress_generic(),
may end up making no forward progress,
(aka no byte read from input __and__ no byte written to output),
due to unusual parameters conditions,
such as providing an output buffer already full.
In such case, the caller may be caught in an infinite loop,
calling the streaming decompression function again and again,
without making any progress.
This version detects such situation, and generates an error instead :
ZSTD_error_dstSize_tooSmall when output buffer is full,
ZSTD_error_srcSize_wrong when input buffer is empty.
The detection tolerates a number of attempts before triggering an error,
controlled by ZSTD_NO_FORWARD_PROGRESS_MAX macro constant,
which is set to 16 by default, and can be re-defined at compilation time.
This behavior tolerates potentially existing implementations
where such cases happen sporadically, like once or twice,
which is not dangerous (only infinite loops are),
without generating an error, hence without breaking these implementations.
2018-06-22 17:58:21 -07:00
3934e010a2
Merge pull request #1197 from facebook/poolResize
...
Thread Pool resize
2018-06-22 14:20:07 -07:00
fbd5dfc1b1
changed POOL_resize() return type to int
...
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
2018-06-22 12:14:59 -07:00
1d5648ca10
Merge pull request #1196 from felixhandte/zstd-btopt-in-place-dict
...
ZSTD_btopt: Support Searching the Dictionary Context In-Place
2018-06-22 11:53:23 -07:00
f6242d30b7
Merge pull request #1202 from facebook/barelyCompressible
...
Increase threshold detection of poorly compressible data
2018-06-22 11:52:52 -07:00
698fd00afb
huf: increase threshold detection of poorly compressible data
2018-06-21 18:32:38 -07:00
243cd9d8bb
add a cond_broadcast after resize
...
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
2018-06-21 18:04:58 -07:00
818e72b4d5
added extended POOL test
...
abrupt end + downsizing with running jobs remaining in queue.
also : POOL_resize() requires numThreads >= 1
2018-06-21 14:58:59 -07:00
01bb1c1016
Add CCtx Param Controlling Dict Attachment Behavior
2018-06-21 17:29:25 -04:00
3e91dc4d6a
Add Repcode Bounds Check
2018-06-21 15:54:41 -04:00
5bd3d4b7d2
Add Debug Log Statement
2018-06-21 15:54:07 -04:00
3caba150c6
Fix `dmsBtLow` Test
2018-06-21 15:53:40 -04:00
5da9bbc38e
Conceivably Dedup ZSTD_noDict and ZSTD_dictMatchState _insertBt1 Impls
...
By reverting to the bool extDict flag, we call ZSTD_insertBt1 with the same
const args in both non-extDict dictModes.
2018-06-21 11:20:01 -04:00
6de249c1c6
fixed: bug when counting nb of active threads
...
when queueSize > 1
also : added a test in testpool.c
verifying resizing is effective.
2018-06-20 18:28:49 -07:00
6b48eb12c0
change control of threadLimit
...
now limits maximum nb of active threads
even when queueSize > 1.
2018-06-20 14:35:39 -07:00
5d81f71e83
Consistency in Guarding DMS-Only Variable Initializations
2018-06-20 16:54:53 -04:00
9c14eafe3d
Also Use `matchLow` for HC3 Match
2018-06-20 15:51:14 -04:00
0a6cf7cd1d
Minor Changes
2018-06-20 15:27:23 -04:00
ae1f3898a2
Remove Dead(!) HC3 DMS Lookup
2018-06-20 15:27:12 -04:00
93702a7a62
Merge pull request #1198 from facebook/msdebug
...
made Visual Studio compatible with DEBUGLEVEL >= 2
2018-06-20 12:26:31 -07:00
ae0b7ffa0a
made Visual Studio compatible with DEBUGLEVEL >= 2
2018-06-20 09:45:02 -07:00
62469c9f41
fixed wrong size in pthread struct transfer
2018-06-19 20:14:03 -07:00
166901dc72
reduced POOL_resize() restriction
...
It's not necessary to ensure that no job is ongoing.
The pool is only expanded, existing threads are preserved.
In case of error, the only option is to return NULL and terminate the thread pool anyway.
2018-06-19 18:07:18 -07:00
066fbbfe1c
make zstdmt resize its context
...
when nbThreads change.
Technically, it only expands.
But when instructed to use less threads,
the thread pool will limit nb of concurrent threads.
2018-06-19 17:28:56 -07:00
4567c57199
finalized POOL_resize()
...
POOL_ctx* POOL_resize(POOL_ctx* ctx, size_t numThreads)
The function may fail, and returns a NULL pointer in this case.
2018-06-19 16:03:12 -07:00
6768cf53fd
Merge pull request #1190 from terrelln/ldm-adjust
...
Adjust advanced parameters to source size
2018-06-19 14:40:56 -07:00
03c39c540b
Fix Incorrect Param
2018-06-19 15:36:33 -04:00
de639502aa
Update Dict Attachment Cut-Offs
2018-06-19 15:36:13 -04:00
f0a13bcd68
Make Sure Position 0 Gets Into the Tree
2018-06-19 15:10:06 -04:00
87fe4788a3
Fix Compression Ratio Regression #1
2018-06-19 13:01:21 -04:00
4bb79f9c55
Misc Changes
2018-06-19 13:01:21 -04:00
2091f34e9e
Find Proper Matches
2018-06-19 13:01:21 -04:00
64348a15f1
Misc Fixes
2018-06-19 13:01:21 -04:00
ade8586ce6
Find `mls == 3` Matches
2018-06-19 13:01:21 -04:00
ce743312e2
Fix Typo
2018-06-19 13:01:21 -04:00
a075864756
Switch `!= ZSTD_extDict` to `== ZSTD_noDict`
2018-06-19 13:01:21 -04:00
1e03377bde
Implement RepCode Check
2018-06-19 13:01:21 -04:00
ccbf067973
Add _dictMatchState Functions
2018-06-19 13:01:21 -04:00
d5d8240967
Convert `extDict` Flag to `dictMode` Enum
2018-06-19 13:01:21 -04:00
93c3184d44
Attach Dicts when Using ZSTD_btopt and ZSTD_btultra
2018-06-19 13:01:21 -04:00
1c714fda3f
introduced POOL_resize()
...
not complete yet :
finalize behavior in case of unfinished expansion
2018-06-18 20:46:39 -07:00
3841dbac84
Adjust advanced parameters to source size
...
In the new advanced API, adjust the parameters even if they are explicitly
set. This mainly applies to the `windowLog`, and accordingly the `hashLog`
and `chainLog`, when the source size is known.
2018-06-18 15:49:31 -07:00
e30f13bde0
Merge pull request #1185 from felixhandte/zstd-btlazy-in-place-dict
...
ZSTD_btlazy2: Support Searching the Dictionary Context In-Place
2018-06-18 13:29:44 -07:00
Yann Collet
Yann Collet
Nick Terrell
W. Felix Handte
cyan4973
Nick Terrell