note : for some reason,
scan-build version on my laptop found problems within fastcover.c
that scan-build on travisCI does not flag.
They are, as usual, false positive :
the analyzer does not understand that a table (`offset`) is correctly filled before usage.
for Linux and Mac OS-X.
Note : the backtraces fires up through a trap
before the sanitizer get a chance to report.
There are situations where the sanitizer report is actually preferable.
It might be good to consider a kind of build macro
which can disable backtrace
when sanitizer is enabled.
from EXIT_IF() to RETURN_IF()
EXIT could be misunderstood as exit(), which terminates program execution.
But the macro only leaves the function, not the program.
- Print signal name to term
- Add -rdynamic option to generate Linux symbol names in backtrace
- Raise default signal after handler to ensure program termination
For OSX and Linux, add a signal handler to SIGABRT, SGIFPE, SIGILL,
SIGSEGV, and SIGBUS. When the program terminates unexpectedly the
handler will print the current stack to the terminal to help determine
the location of the failure.
On OSX the output will look like:
```
Stack trace:
4 zstd 0x000000010927ed96 main + 16886
5 libdyld.dylib 0x00007fff767d1015 start + 1
6 ??? 0x0000000000000001 0x0 + 1
```
On Linux the output will look like:
```
Stack trace:
./zstd() [0x4b8e1b]
./zstd() [0x4b928a]
./zstd() [0x403dc2]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5e0fbb0445]
./zstd() [0x405754]
```
As is, the code does not function on WIN32.
See also: https://oroboro.com/stack-trace-on-crash/
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block
current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size
dictionary can be selected on command line, with -D
* Minor fix
* Run non-optimize FASTCOVER 5 times in benchmark
* Merge fastCover into dictBuilder
* Fix mixed declaration issue
* Add fastcover to symbol.c
* Add fastCover.c and cover.h to build
* Change fastCover.c to fastcover.c
* Update benchmark to run FASTCOVER in dictBuilder
* Undo spliting fastcover_param into cover_param and f
* Remove convert param functions
* Assign f to parameter
* Add zdict.h to Makefile in lib
* Add cover.h to BUCK
* Cast 1 to U64 before shifting
* Remove trimming of zero freq head and tail in selectSegment and rebenchmark
* Remove f as a separate parameter of tryParam
* Read 8 bytes when d is 6
* Add trimming off zero frequency head and tail
* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)
* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary
* Change nbDmer to always read 8 bytes even when d=6
* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER
* Update comments and benchmarking result
* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover
* Add dictType enum and fix bug about passing zParam when converting to coverParam
* Combine finalize and skip into a single parameter
* Update acceleration parameters and benchmark on 3 sample sets
* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets
* Initialize variables outside of for loop in benchmark.c
* Update benchmark result for hg-manifest
* Remove cover.h from install-includes
* Add explanation of f
* Set default compression level for trainFromBuffer to 3
* Add assertion of fastCoverParams in DiB_trainFromFiles
* Add checkTotalCompressedSize function + some minor fixes
* Add test for multithreading fastCovr
* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish
* Free segmentFreqs
* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment
* Add FASTCOVER_MEMMULT
* Minor fix
* Update benchmarking result
Per warnings from flawfinder: "Does not check for buffer overflows when
copying to destination [MS-banned] (CWE-120). Consider using snprintf,
strcpy_s, or strlcpy (warning: strncpy easily misused).".
Replaced called to strcpy and strcat in `fileio.c` to calls with a
specified size (`strncpy` and `strncat`).
Tested the changes on OSX, Linux, Windows.
On OSX + Linux, changes were tested with ASAN. The following flags were
used: 'check_initialization_order=1:strict_init_order=1:detect_odr_violation=1:detect_stack_use_after_return=1'
To reproduce warning:
./flawfinder.py ./programs/fileio.c
tells in a non-blocking way if there is something ready to flush right now.
only works with multi-threading for the time being.
Useful to know if flush speed will be limited by lack of production.
Add different constraint types (decompression speed, compression memory, parameter constraints)
Separate search space by strategy + strategy selection
Memoize results
Real random restarts
Support multiple files
Support Dictionary inputs
Debug Macro for extra printing
Additional constraint checking
Minor fixes
more param parsing
Add Memory
Change paramVariation
work on feasibility
reformat bench
Changed Paramgrill to use bench.c benchmarking
customlevel macro
Printing Flag
Minor changes
Explicit casting
Makefile fix
casting, type fix
Printing Flag
Minor Changes
comments, helper fn's
Seperate syntheticTest and fileTableTest (now renamed as benchFiles)
Add incremental display to benchMem
Change to only iterMode for benchFunction
Make Synthetic test's compressibility configurable from cli (using -P#)
In the new advanced API, adjust the parameters even if they are explicitly
set. This mainly applies to the `windowLog`, and accordingly the `hashLog`
and `chainLog`, when the source size is known.
-Remove global variables
-Remove gv setting functions
-Add advancedParams struct
-Add defaultAdvancedParams();
-Change return type of bench Files
-Change cli to use new interface
-Changed error returns to own struct value
-Change default compression benchmark to use decompress_generic
-Add CustomBench function
-Add Documentation for new functions
IS_CONSOLE stolen wholesale from Options.cpp
not sure if i should have extracted that code for DRY-ness
tested in OSX and functionality seems appropriate
unstested in a windows environment
unistd.h is for unix standard tools.
There does not appear to be a simple isatty for windows
this we only run the logic and header include in
non-windows environments
Yann Collet
Nick Terrell
Yann Collet
Casey McGinty
Jennifer Liu
Jennifer Liu
Eden Zik
George Lu
cyan4973
Nick Terrell
W. Felix Handte
Topher Lubaway
Ryan Schmidt