Commit Graph

5499 Commits (0f91b039ff934c4fd42d292a90ea62b52cb2d696)

Author SHA1 Message Date
Yann Collet 4746c30001
Merge pull request #1212 from facebook/decSpeed
Fix decoding speed
2018-06-27 12:11:45 -07:00
Yann Collet 4489daec09 slightly adjusted default-distribution threshold
depending on strategy.
fast favors faster compression and decompression speeds.
2018-06-26 20:10:45 -07:00
Yann Collet ff773bfcde zeroise freq table with memset()
improves decoding speed by ~5% in github_users sample set
2018-06-26 17:24:41 -07:00
Yann Collet 5ec2d38870
Merge pull request #1208 from facebook/msys2
minor fixes for MSYS2 compilation
2018-06-26 14:47:29 -07:00
Yann Collet 7b9bbf77c9 switched to a sizeof() version
avoid -Werror=unused-variable issue
2018-06-26 14:08:35 -07:00
Yann Collet 25f3d54c40
Merge pull request #1207 from oleid/cmake_multithread_logic
Correct multithread logic, fixing 'unsupported parameter' error
2018-06-26 12:13:16 -07:00
Yann Collet f98ec46979 updated DEBUG_STATIC_ASSERT()
following suggestion from #1209
2018-06-26 12:04:59 -07:00
cyan4973 f741fb8fcd minor fixes for MSYS2 compilation 2018-06-26 01:22:45 -07:00
oleid 4e196b2ac3
Correct multithread logic, fixing 'unsupported parameter' error
The original conditions only worked, when both, static and shared variants where built, resulting in an inconsistency between programs and library. The program was built with MT support enabled, the library not. That lead to error 11 'unsupported parameter' when compressing anything with the command line tool.

When changing the AND condition to `ZSTD_MULTITHREAD_SUPPORT AND (ZSTD_BUILD_SHARED OR ZSTD_BUILD_SHARED)`, cmake stopps complaining one of the targets wasn't built. This commit works for any case.
2018-06-26 08:36:41 +02:00
George Lu ceb4b9e670 New fullbench args
-l# gives some level
--zstd= style parameters also supported
2018-06-25 18:43:55 -07:00
Nick Terrell b426bcc097
[zstdmt] Fix jobsize bugs (#1205)
[zstdmt] Fix jobsize bugs

* `ZSTDMT_serialState_reset()` should use `targetSectionSize`, not `jobSize` when sizing the seqstore.
  Add an assert that checks that we sized the seqstore using the right job size.
* `ZSTDMT_compressionJob()` should check if `rawSeqStore.seq == NULL`.
* `ZSTDMT_initCStream_internal()` should not adjust `mtctx->params.jobSize` (clamping to MIN/MAX is okay).
2018-06-25 15:21:08 -07:00
George Lu 50d612f4f0 Interleave compression/decompression
Fix Bugs
2018-06-25 15:01:03 -07:00
Yann Collet 3b53bfe4f3
Merge pull request #1200 from felixhandte/zstd-attach-dict-pref
Add CCtx Param Controlling Dict Attachment Behavior
2018-06-25 12:42:31 -07:00
George Lu d6121ad0e1 Opaque State
And minor fixups (comments/alignment/checks/fix memory leak)
2018-06-25 08:07:43 -07:00
Yann Collet 59bb5f7d58
Merge pull request #1204 from facebook/noForwardProgress
Error code on no forward progress
2018-06-22 20:25:29 -07:00
Yann Collet 31769ce702 error on no forward progress
streaming decoders, such as ZSTD_decompressStream() or ZSTD_decompress_generic(),
may end up making no forward progress,
(aka no byte read from input __and__ no byte written to output),
due to unusual parameters conditions,
such as providing an output buffer already full.

In such case, the caller may be caught in an infinite loop,
calling the streaming decompression function again and again,
without making any progress.

This version detects such situation, and generates an error instead :
ZSTD_error_dstSize_tooSmall when output buffer is full,
ZSTD_error_srcSize_wrong when input buffer is empty.

The detection tolerates a number of attempts before triggering an error,
controlled by ZSTD_NO_FORWARD_PROGRESS_MAX macro constant,
which is set to 16 by default, and can be re-defined at compilation time.
This behavior tolerates potentially existing implementations
where such cases happen sporadically, like once or twice,
which is not dangerous (only infinite loops are),
without generating an error, hence without breaking these implementations.
2018-06-22 17:58:21 -07:00
Yann Collet 3934e010a2
Merge pull request #1197 from facebook/poolResize
Thread Pool resize
2018-06-22 14:20:07 -07:00
Yann Collet fbd5dfc1b1 changed POOL_resize() return type to int
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
2018-06-22 12:14:59 -07:00
Yann Collet d70c4a5074
Merge pull request #1201 from facebook/rfcUpdate
updated Zstandard frame format
2018-06-22 11:53:50 -07:00
Yann Collet 1d5648ca10
Merge pull request #1196 from felixhandte/zstd-btopt-in-place-dict
ZSTD_btopt: Support Searching the Dictionary Context In-Place
2018-06-22 11:53:23 -07:00
Yann Collet f6242d30b7
Merge pull request #1202 from facebook/barelyCompressible
Increase threshold detection of poorly compressible data
2018-06-22 11:52:52 -07:00
Yann Collet 698fd00afb huf: increase threshold detection of poorly compressible data 2018-06-21 18:32:38 -07:00
Yann Collet c1e6347717 fixed minor typos, detected by @terrelln 2018-06-21 18:08:11 -07:00
Yann Collet 243cd9d8bb add a cond_broadcast after resize
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
2018-06-21 18:04:58 -07:00
Yann Collet 7639db939f updated Zstandard frame format
adding clarifications from IETF RFC DISCUSS.
2018-06-21 17:55:55 -07:00
George Lu ab26f24c9c benchFunction Timed Wrappers
Add BMK_benchFunctionTimed
Add BMK_init_customResultCont..
Change benchMem to use benchFunctionTimed
Minor Fixes/Adjustments
2018-06-21 16:23:55 -07:00
George Lu a8eea99ebe Incremental Display + Fn Separations
Seperate syntheticTest and fileTableTest (now renamed as benchFiles)
Add incremental display to benchMem
Change to only iterMode for benchFunction
Make Synthetic test's compressibility configurable from cli (using -P#)
2018-06-21 16:23:18 -07:00
Yann Collet 818e72b4d5 added extended POOL test
abrupt end + downsizing with running jobs remaining in queue.

also : POOL_resize() requires numThreads >= 1
2018-06-21 14:58:59 -07:00
W. Felix Handte 01bb1c1016 Add CCtx Param Controlling Dict Attachment Behavior 2018-06-21 17:29:25 -04:00
W. Felix Handte 3e91dc4d6a Add Repcode Bounds Check 2018-06-21 15:54:41 -04:00
W. Felix Handte 5bd3d4b7d2 Add Debug Log Statement 2018-06-21 15:54:07 -04:00
W. Felix Handte 3caba150c6 Fix `dmsBtLow` Test 2018-06-21 15:53:40 -04:00
Yann Collet 7d80ada5ca added a test for POOL (multithreading)
ensuring all jobs in queue are nonetheless completed
when POOL is instructed to end abruptly (POOL_free())
2018-06-21 12:24:36 -07:00
W. Felix Handte 5da9bbc38e Conceivably Dedup ZSTD_noDict and ZSTD_dictMatchState _insertBt1 Impls
By reverting to the bool extDict flag, we call ZSTD_insertBt1 with the same
const args in both non-extDict dictModes.
2018-06-21 11:20:01 -04:00
Yann Collet 6de249c1c6 fixed: bug when counting nb of active threads
when queueSize > 1

also : added a test in testpool.c
       verifying resizing is effective.
2018-06-20 18:28:49 -07:00
Yann Collet 6b48eb12c0 change control of threadLimit
now limits maximum nb of active threads
even when queueSize > 1.
2018-06-20 14:35:39 -07:00
Jennifer Liu a3cb97b038 set up sample config.yml 2018-06-20 14:23:14 -07:00
W. Felix Handte 5d81f71e83 Consistency in Guarding DMS-Only Variable Initializations 2018-06-20 16:54:53 -04:00
W. Felix Handte 9c14eafe3d Also Use `matchLow` for HC3 Match 2018-06-20 15:51:14 -04:00
W. Felix Handte 0a6cf7cd1d Minor Changes 2018-06-20 15:27:23 -04:00
W. Felix Handte ae1f3898a2 Remove Dead(!) HC3 DMS Lookup 2018-06-20 15:27:12 -04:00
Yann Collet 93702a7a62
Merge pull request #1198 from facebook/msdebug
made Visual Studio compatible with DEBUGLEVEL >= 2
2018-06-20 12:26:31 -07:00
cyan4973 ae0b7ffa0a made Visual Studio compatible with DEBUGLEVEL >= 2 2018-06-20 09:45:02 -07:00
Yann Collet 62469c9f41 fixed wrong size in pthread struct transfer 2018-06-19 20:14:03 -07:00
Yann Collet 166901dc72 reduced POOL_resize() restriction
It's not necessary to ensure that no job is ongoing.
The pool is only expanded, existing threads are preserved.
In case of error, the only option is to return NULL and terminate the thread pool anyway.
2018-06-19 18:07:18 -07:00
Yann Collet 066fbbfe1c make zstdmt resize its context
when nbThreads change.

Technically, it only expands.
But when instructed to use less threads,
the thread pool will limit nb of concurrent threads.
2018-06-19 17:28:56 -07:00
Yann Collet 4567c57199 finalized POOL_resize()
POOL_ctx* POOL_resize(POOL_ctx* ctx, size_t numThreads)

The function may fail, and returns a NULL pointer in this case.
2018-06-19 16:03:12 -07:00
Yann Collet 6768cf53fd
Merge pull request #1190 from terrelln/ldm-adjust
Adjust advanced parameters to source size
2018-06-19 14:40:56 -07:00
Yann Collet c0b6ce95b1
Merge pull request #1179 from supertopher/dev
Improves UX for --list command's lack of support for pipes
2018-06-19 14:36:30 -07:00
W. Felix Handte 03c39c540b Fix Incorrect Param 2018-06-19 15:36:33 -04:00