facebook/zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,985 Commits 2 Branches 0 Tags
Commit Graph

100 Commits

Author SHA1 Message Date
Yann Collet
e874dacc08 changed searchLength into minMatch 
refactored all relevant API and calls
for consistency.
 2018-11-20 14:56:07 -08:00
Yann Collet
2caa995558 just add an assert() in ZSTD_insertBtAndGetAllMatches() 
to express a condition on ll0 .
May help static analyzer as in #1397
 2018-11-05 17:13:32 -08:00
W. Felix Handte
bad74c4781 Use Working Ctx Logs when not in DMS Mode 
We pre-hash the ptr for the dict match state sometimes. When that actually
happens, a hashlog of 0 can produce undefined behavior (right shift a long
long by 64). Only applies to unoptimized compilations, since when
optimizations are applied, those hash operations are dropped when we're not
actually in dms mode.
 2018-09-28 17:12:54 -07:00
W. Felix Handte
00c088b32d Support Split Logs in ZSTD_btopt..ZSTD_btultra 2018-09-28 17:12:54 -07:00
W. Felix Handte
97149f22c3 Stop Separately Passing CParams in ZSTD_opt Internal Functions 2018-09-28 17:10:42 -07:00
W. Felix Handte
dcdf437fed Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
W. Felix Handte
6cb2454646 Remove CParams from Block Compressor Functions' Args 2018-09-28 17:10:42 -07:00
Nick Terrell
f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
Yann Collet
6e66bbf5dd fixed several minor issues detected by scan-build 
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
 2018-08-14 16:55:35 -07:00
W. Felix Handte
3e91dc4d6a Add Repcode Bounds Check 2018-06-21 15:54:41 -04:00
W. Felix Handte
5bd3d4b7d2 Add Debug Log Statement 2018-06-21 15:54:07 -04:00
W. Felix Handte
3caba150c6 Fix dmsBtLow Test 2018-06-21 15:53:40 -04:00
W. Felix Handte
5da9bbc38e Conceivably Dedup ZSTD_noDict and ZSTD_dictMatchState _insertBt1 Impls 
By reverting to the bool extDict flag, we call ZSTD_insertBt1 with the same
const args in both non-extDict dictModes.
 2018-06-21 11:20:01 -04:00
W. Felix Handte
5d81f71e83 Consistency in Guarding DMS-Only Variable Initializations 2018-06-20 16:54:53 -04:00
W. Felix Handte
9c14eafe3d Also Use matchLow for HC3 Match 2018-06-20 15:51:14 -04:00
W. Felix Handte
0a6cf7cd1d Minor Changes 2018-06-20 15:27:23 -04:00
W. Felix Handte
ae1f3898a2 Remove Dead(!) HC3 DMS Lookup 2018-06-20 15:27:12 -04:00
W. Felix Handte
03c39c540b Fix Incorrect Param 2018-06-19 15:36:33 -04:00
W. Felix Handte
f0a13bcd68 Make Sure Position 0 Gets Into the Tree 2018-06-19 15:10:06 -04:00
W. Felix Handte
87fe4788a3 Fix Compression Ratio Regression #1 2018-06-19 13:01:21 -04:00
W. Felix Handte
4bb79f9c55 Misc Changes 2018-06-19 13:01:21 -04:00
W. Felix Handte
2091f34e9e Find Proper Matches 2018-06-19 13:01:21 -04:00
W. Felix Handte
64348a15f1 Misc Fixes 2018-06-19 13:01:21 -04:00
W. Felix Handte
ade8586ce6 Find mls == 3 Matches 2018-06-19 13:01:21 -04:00
W. Felix Handte
ce743312e2 Fix Typo 2018-06-19 13:01:21 -04:00
W. Felix Handte
a075864756 Switch != ZSTD_extDict to == ZSTD_noDict 2018-06-19 13:01:21 -04:00
W. Felix Handte
1e03377bde Implement RepCode Check 2018-06-19 13:01:21 -04:00
W. Felix Handte
ccbf067973 Add _dictMatchState Functions 2018-06-19 13:01:21 -04:00
W. Felix Handte
d5d8240967 Convert extDict Flag to dictMode Enum 2018-06-19 13:01:21 -04:00
Yann Collet
2d76defbfe grouped all histogram functions into hist.c 
renamed functions with HIST_* prefix
 2018-06-13 19:49:31 -04:00
Yann Collet
809f2f9322 minor update of literal cost function 
just assert() there is no negative cost evaluation for literals
 2018-05-29 15:34:50 -07:00
Yann Collet
463a0fe38b simplified optimal parser 
removed "cached" structure.
prices are now saved in the optimal table.

Primarily done for simplification.
Might improve speed by a little.
But actually, and surprisingly, also improves ratio in some circumstances.
 2018-05-29 14:07:25 -07:00
Yann Collet
e2c0e3d437 slightly nudge choices towards less sequences 
also slightly improve some strange detrimental corner cases.
 2018-05-25 14:52:21 -07:00
Yann Collet
f6ad59ab5c Merge branch 'dev' into staticDictCost 2018-05-24 16:21:02 -07:00
Nick Terrell
e3959d5eba Fixes 2018-05-22 16:06:33 -07:00
Yann Collet
a8ddf1d370 disable 2-passes strategy 2018-05-22 15:06:36 -07:00
Yann Collet
5cbef6e094 Merge branch 'dev' into staticDictCost 2018-05-18 16:03:06 -07:00
Yann Collet
a95e9e80d1 adding some debug functions to observe statistics 2018-05-18 14:09:42 -07:00
Yann Collet
af3da079d1 fixed minor conversion warning 2018-05-17 17:27:27 -07:00
Yann Collet
8572b4d09f fixed a pretty complex bug when combining ldm + btultra 2018-05-17 16:13:53 -07:00
Yann Collet
134388ba6b collect statistics for first block in ultra mode 
this patch makes btultra do 2 passes on the first block,
the first one being dedicated to collecting statistics
so that the 2nd pass is more accurate.

It translates into a very small compression ratio gain :

enwik7, level 20:
blocks  4K : 2.142 -> 2.153
blocks 16K : 2.447 -> 2.457
blocks 64K : 2.716 -> 2.726

On the other hand, the cpu cost is doubled.

The trade off looks bad.
Though, that's ultimately a price to pay to reach better compression ratio.
So it's only enabled when setting btultra.
 2018-05-17 12:24:30 -07:00
Yann Collet
a243020d37 slightly improved weight calculation 
translating into a tiny compression ratio improvement
 2018-05-17 11:19:44 -07:00
Yann Collet
18fc3d3cd5 introduced bit-fractional cost evaluation 
this improves compression ratio by a *tiny* amount.
It also reduces speed by a small amount.

Consequently, bit-fractional evaluation is only turned on for btultra.
 2018-05-16 14:53:35 -07:00
Nick Terrell
30d9c84b1a Fix failing Travis tests 2018-05-15 09:46:20 -07:00
Yann Collet
2c26df0e13 opt: removed static prices 
after testing, it's actually always better to use dynamic prices
albeit initialised from dictionary.
 2018-05-14 18:04:08 -07:00
Yann Collet
761758982e replaced FSE_count by FSE_count_simple 
to reduce usage of stack memory.

Also : tweaked a few comments, as suggested by @terrelln
 2018-05-11 16:03:37 -07:00
Yann Collet
09d0fa29ee minor adjusting of weights 2018-05-10 18:13:48 -07:00
Yann Collet
1a26ec6e8d opt: init statistics from dictionary 
instead of starting from fake "default" statistics.
 2018-05-10 17:59:12 -07:00
Yann Collet
74b1c75d64 btopt : minor adjustment of update frequencies 2018-05-10 16:32:36 -07:00
Yann Collet
ac6105463a opt: minor improvements to log traces 
slight improvement when using fractional-bit evaluation (opt:dictionay)
 2018-05-09 15:46:11 -07:00