Commit Graph

20 Commits (a65eb39f9d79a895ae246ac4b053828711b0eeed)

Author SHA1 Message Date
Josh Soref a880ca239b Spelling (#1582)
* spelling: accidentally

* spelling: across

* spelling: additionally

* spelling: addresses

* spelling: appropriate

* spelling: assumed

* spelling: available

* spelling: builder

* spelling: capacity

* spelling: compiler

* spelling: compressibility

* spelling: compressor

* spelling: compression

* spelling: contract

* spelling: convenience

* spelling: decompress

* spelling: description

* spelling: deflate

* spelling: deterministically

* spelling: dictionary

* spelling: display

* spelling: eliminate

* spelling: preemptively

* spelling: exclude

* spelling: failure

* spelling: independence

* spelling: independent

* spelling: intentionally

* spelling: matching

* spelling: maximum

* spelling: meaning

* spelling: mishandled

* spelling: memory

* spelling: occasionally

* spelling: occurrence

* spelling: official

* spelling: offsets

* spelling: original

* spelling: output

* spelling: overflow

* spelling: overridden

* spelling: parameter

* spelling: performance

* spelling: probability

* spelling: receives

* spelling: redundant

* spelling: recompression

* spelling: resources

* spelling: sanity

* spelling: segment

* spelling: series

* spelling: specified

* spelling: specify

* spelling: subtracted

* spelling: successful

* spelling: return

* spelling: translation

* spelling: update

* spelling: unrelated

* spelling: useless

* spelling: variables

* spelling: variety

* spelling: verbatim

* spelling: verification

* spelling: visited

* spelling: warming

* spelling: workers

* spelling: with
2019-04-12 11:18:11 -07:00
Jennifer Liu 9d6ed9def3 Merge fastCover into DictBuilder (#1274)
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
2018-08-23 12:06:20 -07:00
Jennifer Liu 0acb0abd1e Add non-optimize FASTCOVER (#1260)
* Add non-optimize FASTCOVER

* Minor fix

* Pass param as value instead of pointer
2018-08-01 11:06:16 -07:00
Jennifer Liu 4e29bc2469 Use CDict instead of CCtx in analyzeEntropy 2018-07-31 10:36:45 -07:00
Jennifer Liu 31229e527b Increment frequency for every dmer occurence within same sample instead of at most once per sample 2018-07-30 12:54:22 -07:00
Jennifer Liu 51b109c1b5 Delete old benchmarking result 2018-07-27 17:31:33 -07:00
Jennifer Liu 53ef22a4bc Undo deleting clean in make 2018-07-27 16:56:50 -07:00
Jennifer Liu 96d84ee235 Revert test.sh 2018-07-27 16:54:05 -07:00
Jennifer Liu 61262f6c0d Save segmentFreqs in ctx instead of malloc and memset in SelectSegment 2018-07-27 16:51:38 -07:00
Jennifer Liu 49b398e93f Use same param after optimizing cover and fastCover and record k and d for benchmarking 2018-07-27 13:39:19 -07:00
Jennifer Liu 759c543312 Rerun cover and fastCover with optimized values 2018-07-26 19:03:01 -07:00
Jennifer Liu 3d7941ce41 Benchmark different f values 2018-07-26 16:24:13 -07:00
Jennifer Liu 3b163e0b5b Add array to keep track of frequency within active segment, fix malloc bug, update benchmarking result 2018-07-26 13:53:13 -07:00
Jennifer Liu 2333ecb173 Allow d=6 2018-07-25 18:10:09 -07:00
Jennifer Liu 1e85f314d8 Benchmark fast cover optimize vs k=200 2018-07-25 17:53:38 -07:00
Jennifer Liu d1fc507ef9 Initial benchmarking result for fastCover 2018-07-25 17:05:54 -07:00
Jennifer Liu f5407e398a Make hash value const 2018-07-25 16:54:08 -07:00
Jennifer Liu 7f3f70f766 Add Fast Cover Dictionary Builder 2018-07-25 16:34:07 -07:00
Jennifer Liu b6c5d4982c Minor fix 2018-07-20 17:41:22 -07:00
Jennifer Liu 71e767ac09 Refactoring and benchmark without dictionary 2018-07-20 17:03:47 -07:00