1490 Commits

Author SHA1 Message Date
W. Felix Handte
2091f34e9e Find Proper Matches 2018-06-19 13:01:21 -04:00
W. Felix Handte
64348a15f1 Misc Fixes 2018-06-19 13:01:21 -04:00
W. Felix Handte
ade8586ce6 Find mls == 3 Matches 2018-06-19 13:01:21 -04:00
W. Felix Handte
ce743312e2 Fix Typo 2018-06-19 13:01:21 -04:00
W. Felix Handte
a075864756 Switch != ZSTD_extDict to == ZSTD_noDict 2018-06-19 13:01:21 -04:00
W. Felix Handte
1e03377bde Implement RepCode Check 2018-06-19 13:01:21 -04:00
W. Felix Handte
ccbf067973 Add _dictMatchState Functions 2018-06-19 13:01:21 -04:00
W. Felix Handte
d5d8240967 Convert extDict Flag to dictMode Enum 2018-06-19 13:01:21 -04:00
W. Felix Handte
93c3184d44 Attach Dicts when Using ZSTD_btopt and ZSTD_btultra 2018-06-19 13:01:21 -04:00
Nick Terrell
3841dbac84 Adjust advanced parameters to source size
In the new advanced API, adjust the parameters even if they are explicitly
set. This mainly applies to the `windowLog`, and accordingly the `hashLog`
and `chainLog`, when the source size is known.
2018-06-18 15:49:31 -07:00
Yann Collet
e30f13bde0
Merge pull request #1185 from felixhandte/zstd-btlazy-in-place-dict
ZSTD_btlazy2: Support Searching the Dictionary Context In-Place
2018-06-18 13:29:44 -07:00
Yann Collet
9698d2fb72
Merge pull request #1189 from facebook/hist
histogram module
2018-06-14 20:39:52 -04:00
Yann Collet
6901c94cd6 avoid duplicate code comments
when a function is decribed in hist.h,
do not describe it again in hist.c
to avoid future doc synchronization issues.
2018-06-14 19:47:05 -04:00
Yann Collet
a71513bec6
Merge pull request #1184 from facebook/debug
Grouped debug functions into debug.h
2018-06-14 16:21:53 -04:00
W. Felix Handte
0c654d22c8 Force Inline BtFindBestMatch 2018-06-14 14:54:39 -04:00
Yann Collet
2d76defbfe grouped all histogram functions into hist.c
renamed functions with HIST_* prefix
2018-06-13 19:49:31 -04:00
W. Felix Handte
0551de4b5a Search Dict for Matches 2018-06-13 16:06:28 -04:00
W. Felix Handte
ace9cfa950 Attach Dicts when Using ZSTD_btlazy2 2018-06-13 16:06:28 -04:00
Yann Collet
fa41bcc2c2 grouped debug functions into debug.h
There were 2 competing set of debug functions
within zstd_internal.h and bitstream.h.
They were mostly duplicate, and required care to avoid messing with each other.

There is now a single implementation, shared by both.

Significant change :
The macro variable ZSTD_DEBUG does no longer exist,
it has been replaced by DEBUGLEVEL,
which required modifying several source files.
2018-06-13 15:43:09 -04:00
W. Felix Handte
d53200a846 Fix Cast Warning 2018-06-13 14:58:36 -04:00
W. Felix Handte
b82063b266 Extend Dictionary Matches Backwards 2018-06-13 14:58:36 -04:00
W. Felix Handte
d53a04211c Update Dictionary Attachment Cutoff Values Again 2018-06-13 14:58:36 -04:00
W. Felix Handte
2162aa9f18 Do Not Inline DMS Search Function 2018-06-13 14:58:36 -04:00
W. Felix Handte
338bede9b5 Also Implement Depth Repcode Checks 2018-06-13 14:58:36 -04:00
W. Felix Handte
555ab9f8cf Apply Match Continuation Bug Fix 2018-06-13 14:58:36 -04:00
W. Felix Handte
c87dd2121d Update Dictionary Attachment Cutoff Values 2018-06-13 14:58:36 -04:00
W. Felix Handte
6204b6d592 Check Dict Match State in ZSTD_HcFindBestMatch_generic 2018-06-13 14:58:36 -04:00
W. Felix Handte
211a61b69b Focus on Non-BT Impls for the Moment 2018-06-13 14:58:36 -04:00
W. Felix Handte
2e93736a77 Remove Pre-Existing Repcode Check 2018-06-13 14:58:36 -04:00
W. Felix Handte
3b82a23a35 Second Repcode Check 2018-06-13 14:58:36 -04:00
W. Felix Handte
a2a24bebec First Repcode Check 2018-06-13 14:58:36 -04:00
W. Felix Handte
f74c2cd673 Disallow Too-Long Repcodes When Using an Attached Dict 2018-06-13 14:58:36 -04:00
W. Felix Handte
c14db94450 Rename base -> prefixLowest 2018-06-13 14:58:36 -04:00
W. Felix Handte
5d90708a0a Go Back to Separate Intermediate Functions for Different Dict Modes 2018-06-13 14:58:36 -04:00
W. Felix Handte
f84fc63a43 Further Templatize Intermediate Functions on dictMode 2018-06-13 14:58:36 -04:00
W. Felix Handte
529d3a5acd Convert Existing U32 extDict Vars to ZSTD_dictMode Enums 2018-06-13 14:58:36 -04:00
W. Felix Handte
33e2240fac Attach Dict When Using ZSTD_lazy Strategies 2018-06-13 14:58:36 -04:00
W. Felix Handte
90cfc799e5 Add _dictMatchState Stubs for ZSTD_lazy Functions 2018-06-13 14:58:36 -04:00
W. Felix Handte
a85ecb32bd Add dictMode Param to ZSTD_compressBlock_lazy_generic 2018-06-13 14:58:36 -04:00
Yann Collet
b2632bcf6c
Merge pull request #1174 from duc0/document_default_level
Expose ZSTD_CLEVEL_DEFAULT and update documentation
2018-06-12 12:09:01 -07:00
Duc Ngo
e34c000e44 Expose ZSTD_CLEVEL_DEFAULT and update documentation 2018-06-08 11:33:44 -07:00
Yann Collet
3050733042 Merge branch 'dev' into negLevels 2018-06-07 15:51:35 -07:00
Yann Collet
c2c47e24e0 support targetlen==0 with strategy==ZSTD_fast
to mean "normal compression",
targetlen >= 1 now means "disable huffman compression of literals"
2018-06-07 15:49:01 -07:00
Yann Collet
a57b4df85f removed literalCompression directive
in this version, literal compression is always disabled for ZSTD_fast strategy.

Performance parity between ZSTD_compress_advanced() and ZSTD_compress_generic()
2018-06-07 15:24:12 -07:00
Yann Collet
8537bfd85c fuzzer: make negative compression level fail
result of ZSTD_compress_advanced()
is different from ZSTD_compress_generic()
when using negative compression levels
because the disabling of huffman compression is not passed in parameters.
2018-06-07 15:12:13 -07:00
Yann Collet
e3c42c739b clean ZSTD_compress() initialization
The (pretty old) code inside ZSTD_compress()
was making some pretty bold assumptions
on what's inside a CCtx and how to init it.

This is pretty fragile by design.
CCtx content evolve.
Knowledge of how to handle that should be concentrate in one place.

A side effect of this strategy
is that ZSTD_compress() wouldn't check for BMI2 capability,
and is therefore missing out some potential speed opportunity.

This patch makes ZSTD_compress() use
the same initialization and release functions
as the normal creator / destructor ones.

Measured on my laptop, with a custom version of bench
manually modified to use ZSTD_compress() (instead of the advanced API) :
This patch :
 1#silesia.tar       : 211984896 ->  73651053 (2.878), 312.2 MB/s , 723.8 MB/s
 2#silesia.tar       : 211984896 ->  70163650 (3.021), 226.2 MB/s , 649.8 MB/s
 3#silesia.tar       : 211984896 ->  66996749 (3.164), 169.4 MB/s , 636.7 MB/s
 4#silesia.tar       : 211984896 ->  65998319 (3.212), 136.7 MB/s , 619.2 MB/s
dev branch :
 1#silesia.tar       : 211984896 ->  73651053 (2.878), 291.7 MB/s , 727.5 MB/s
 2#silesia.tar       : 211984896 ->  70163650 (3.021), 216.2 MB/s , 655.7 MB/s
 3#silesia.tar       : 211984896 ->  66996749 (3.164), 162.2 MB/s , 633.1 MB/s
 4#silesia.tar       : 211984896 ->  65998319 (3.212), 130.6 MB/s , 618.6 MB/s
2018-06-07 14:05:25 -07:00
Yann Collet
f1ea383f45 context can be sized down even with constant parameters
when parameters are "equivalent",
the context is re-used in continue mode,
hence needed workspace size is not recalculated.
This incidentally also evades the size-down check and action.

This patch intercepts the "continue mode"
so that the size-down check and action is actually triggered.
2018-06-06 15:04:12 -07:00
Yann Collet
e5e17d009f changed member name to workSpaceOversizedDuration 2018-06-06 15:00:27 -07:00
Yann Collet
f7392f3dc9 added test case 2018-06-05 14:53:28 -07:00
Yann Collet
3d523c741b added workSpaceTooLarge and workSpaceWasteful
also :
slightly increased speed of test fuzzer.16
2018-06-05 11:42:48 -07:00