Commit Graph

299 Commits (30ffc24ad7b763166e4583f234fc3688944a7225)

Author SHA1 Message Date
Yi Jin 30ffc24ad7 fix the code ahead of declaration issue 2018-12-19 17:49:04 -08:00
Yi Jin 0700335f57 change int init_cLevel() to int init_cLevel(void) 2018-12-19 17:38:28 -08:00
Yi Jin 26a9ae3f5f refactor readU32FromChar(...), improve init_cLevel(...), and add env var ZSTD_CLEVEL tests 2018-12-19 16:45:42 -08:00
Yi Jin 5324b1e386 add support for setting compression level through environment variable ZSTD_CLEVEL 2018-12-19 13:26:27 -08:00
Nick Terrell bdfcaecc0a [zstdcli] Add --no-progress flag
The `--no-progress` flag disables zstd's progress bars, but leaves
the summary.

I've added simple tests to `playTests.sh` to make sure the parsing
works.
2018-12-14 11:50:25 -08:00
Yann Collet 9c3265a53f
Merge pull request #1417 from facebook/advancedAPI
Advanced API
2018-12-10 18:48:15 -08:00
Denis Ahrens 71b8ee1bf1 fixed a typo 2018-11-28 04:51:16 +01:00
Yann Collet 41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet e874dacc08 changed searchLength into minMatch
refactored all relevant API and calls
for consistency.
2018-11-20 14:56:07 -08:00
Nick Terrell f9a671ac37 [cli] Add rsyncable 2018-11-14 17:00:16 -08:00
Yann Collet d38063f8ae separated bench module into benchfn and benchzstd
it shall be possible to use benchfn
without any dependency on zstd.
2018-11-13 11:01:59 -08:00
Yann Collet 6c51bf420c bounds for --adapt mode
can supply min and max compression level through advanced command :
--adapt=min=#,max=#
2018-09-25 16:03:28 -07:00
Yann Collet 04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet 9bb6c15f79
Merge pull request #1332 from facebook/minclevel
defined a minimum negative level
2018-09-24 16:01:13 -07:00
Yann Collet 0250ac74ce fixed minor scan-build warnings 2018-09-24 00:52:19 -07:00
Yann Collet 0fc07eb1fd fixed zstd-decompress
which cannot support ZSTD_minCLevel()
2018-09-22 17:21:39 -07:00
Yann Collet 364041c6dd enforce minimum compression level limit
using ZSTD_minCLevel()
2018-09-22 16:10:10 -07:00
Yann Collet 15519479ba fixed minor gcc warning on a unused variable 2018-09-20 13:00:11 -07:00
Yann Collet 2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
Casey McGinty 2224ecd718
Remove dead code and method name typo 2018-09-11 11:56:50 -07:00
Casey McGinty d4337b6f1d
Move ABRThandler func out of internal lib 2018-09-11 11:39:49 -07:00
Casey McGinty a06574fc97
Print a stack trace on unexpected term signal (e.g. SIGABRT)
For OSX and Linux, add a signal handler to SIGABRT, SGIFPE, SIGILL,
SIGSEGV, and SIGBUS. When the program terminates unexpectedly the
handler will print the current stack to the terminal to help determine
the location of the failure.

On OSX the output will look like:

```
Stack trace:
4   zstd                                0x000000010927ed96 main + 16886
5   libdyld.dylib                       0x00007fff767d1015 start + 1
6   ???                                 0x0000000000000001 0x0 + 1
```

On Linux the output will look like:

```
Stack trace:
./zstd() [0x4b8e1b]
./zstd() [0x4b928a]
./zstd() [0x403dc2]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5e0fbb0445]
./zstd() [0x405754]
```

As is, the code does not function on WIN32.

See also: https://oroboro.com/stack-trace-on-crash/
2018-09-06 18:46:52 -07:00
Jennifer Liu 21721b75a3 Change default f to 20 2018-09-04 17:15:14 -07:00
Yann Collet b37a0a6bde
Merge pull request #1298 from facebook/bench
Refactored bench.c
2018-08-28 12:25:02 -07:00
Yann Collet 55affc09de timedFn : measurement delay is programmable
instead of hard-coded 1 second per measurement
2018-08-28 11:26:27 -07:00
Jennifer Liu 9d6ed9def3 Merge fastCover into DictBuilder (#1274)
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
2018-08-23 12:06:20 -07:00
Yann Collet 33f7709c71 fileio: changed parameter type from ptr to plain structure
safer : this parameter is read-only,
we don't want original structure to be modified
2018-08-13 13:02:03 -07:00
Yann Collet e7a49c6683 introduced command --adapt 2018-08-11 20:48:06 -07:00
Yann Collet 2fdab1629b fix unused variable warning 2018-08-03 08:30:01 -07:00
Yann Collet 5203f01774 fix : zstd cli can be built with build macro ZSTD_NOBENCH
which disables bench.c module
2018-08-03 07:54:29 -07:00
Nick Terrell 58b8219475 zstdcli: Allow -o before --train
Only set the default value if `outFileName` is unset.

Fixes #1227.
2018-07-16 12:45:34 -07:00
Nick Terrell 45821fac0c
Merge pull request #1225 from jennifermliu/dev
Split samples when building dictionary for COVER
2018-07-13 13:26:15 -07:00
Jennifer Liu 5021441d86 Change default splitPoint to 100 2018-07-10 11:19:33 -07:00
Jennifer Liu 0881184c89 Some edits based on pull request comments 2018-07-03 17:53:27 -07:00
Yann Collet 689bfecd48
Merge pull request #1188 from GeorgeLu97/BenchModule
Bench module
2018-07-02 13:33:27 -07:00
Jennifer Liu 8afcb8eea7 Update documentation 2018-07-01 19:59:37 -07:00
Jennifer Liu 84e8b2a305 Fix another declaration issue 2018-06-29 18:02:02 -07:00
Jennifer Liu 348e5f77a9 Add split=# to cli 2018-06-29 17:54:41 -07:00
Yann Collet b5207aadfa make build tests more unforgiving
`-Werror` will ensure they fail if there is the slightest warning.

fix a minor warning specific to `zstd_decompress` variant.
2018-06-29 17:10:56 -07:00
Yann Collet a2c3a4cd0e
Merge pull request #1214 from jennifermliu/dev
Make --fast=0 fail
2018-06-27 18:53:39 -07:00
Jennifer Liu 1ab57a7ce1 Redirect failed test result to INTOVOID and update comment about parsing fast command 2018-06-27 16:27:45 -07:00
Jennifer Liu aef8486fee Make fast=0 fail 2018-06-27 14:27:27 -07:00
cyan4973 f741fb8fcd minor fixes for MSYS2 compilation 2018-06-26 01:22:45 -07:00
George Lu d6121ad0e1 Opaque State
And minor fixups (comments/alignment/checks/fix memory leak)
2018-06-25 08:07:43 -07:00
George Lu a8eea99ebe Incremental Display + Fn Separations
Seperate syntheticTest and fileTableTest (now renamed as benchFiles)
Add incremental display to benchMem
Change to only iterMode for benchFunction
Make Synthetic test's compressibility configurable from cli (using -P#)
2018-06-21 16:23:18 -07:00
George Lu a3c8b59990 Fix cli no print
Change looping behavior to match old
2018-06-18 15:38:14 -07:00
George Lu 0d1ee22990 Requested Changes
Add Comment
Simplify Interface (Remove resultSet)
Reorder Arguments
Remove customBench displayLevel
Reorder bench.h
Change benchFiles return type to match advanced
Rename stuff
2018-06-18 12:01:12 -07:00
George Lu 8522346322 Make Fullbench use new function
Rearrange Args
Add nothing function
Use new function, change locals to match
New Display
Comment cleanup
Change builds
2018-06-15 11:37:49 -04:00
George Lu 20f4f32379 Add to bench
-Remove global variables
-Remove gv setting functions
-Add advancedParams struct
-Add defaultAdvancedParams();
-Change return type of bench Files
-Change cli to use new interface
-Changed error returns to own struct value
-Change default compression benchmark to use decompress_generic
-Add CustomBench function
-Add Documentation for new functions
2018-06-14 14:23:24 -04:00
George Lu 01d940b670 Requested changes
-Remove g_displaylevel/setNotificationLevel function
-Add extern "C"
-Remove averaging
-Reorder arguments

More fixes

-Added BMK_return_t (result + possible error)
-Correct comment'
-Nullcheck ctx, dctx when allocated
-Remove extra assert
2018-06-12 17:02:44 -04:00