Commit Graph

430 Commits (70d8c2a0313deb2a64b549a80fad4eb63596dec2)

Author SHA1 Message Date
Julian Fessard 70d8c2a031 fileio.c: Disable backtrace when built with address sanitizer
Covers clang and gcc's sanitizer flags.
Can still be overridden through CFLAGS on commandline.
2018-10-09 17:14:57 -07:00
Yann Collet 1e0c5466c5 fixed BACKTRACE_ENABLE macro test 2018-10-09 16:36:57 -07:00
Yann Collet f17c1df1ac backtrace support compiled with more conditions
following #1356,
only enable backtrace compilation on linux+glibc.

Also, disable backtrace by default from "release" compilation,
so that less platforms get impacted by the new requirements.
Can be manually enabled/disabled using BACKTRACE=1/0.
2018-10-08 17:03:06 -07:00
maxice8 1d75697d9b
don't assume __linux__ means __GLIBC__ on BACKTRACES_ENABLE
check for __GLIBC__ instead of __linux__ as musl libc doesn't provide
execinfo.h

fixes compilation on Alpine Linux and Void Linux musl arches.
2018-10-05 21:23:45 -03:00
Yann Collet b1407f9acd fixed wrong assert() position
could fire on invalid input.
blocking for afl tests.
2018-10-03 12:43:59 -07:00
Yann Collet 3ca6261223 fixed static analyzer warnings
note : for some reason,
scan-build version on my laptop found problems within fastcover.c
that scan-build on travisCI does not flag.

They are, as usual, false positive :
the analyzer does not understand that a table (`offset`) is correctly filled before usage.
2018-10-02 15:59:11 -07:00
Yann Collet 9012b6cba0 ./zstd -f do no longer overwrite destination file
if source file does not exist (#1082)
2018-10-01 17:16:34 -07:00
Yann Collet c7bd6a41ab zstd -d -f do no longer erase destination file
when source file does not exist (#1082)
2018-10-01 14:04:00 -07:00
Yann Collet 1ab71a8e72 regroup name creation logic into its own function
for a cleaner main file decompression loop
2018-09-28 18:19:23 -07:00
Yann Collet ec1cb8e996 changed macro name
from EXIT_IF() to RETURN_IF()

EXIT could be misunderstood as exit(), which terminates program execution.
But the macro only leaves the function, not the program.
2018-09-28 16:04:00 -07:00
Yann Collet 05c0a072b7 minor improvement in the multi-format suffix selection 2018-09-28 15:57:35 -07:00
Yann Collet d987ab5983 fixed unreachable section warning on Visual 2018-09-28 09:34:16 -07:00
Yann Collet 913a0365b6 Merge branch 'dev' into donotdelete 2018-09-27 19:23:55 -07:00
Yann Collet ef1272737b fixed minor Visual conversion warnings 2018-09-27 18:29:15 -07:00
Yann Collet 9b45db7fa6 minor refactoring of --list
trying to reduce recurrent patterns.
2018-09-27 16:49:08 -07:00
Nick Terrell f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
Yann Collet 6c51bf420c bounds for --adapt mode
can supply min and max compression level through advanced command :
--adapt=min=#,max=#
2018-09-25 16:03:28 -07:00
Yann Collet 04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet 0250ac74ce fixed minor scan-build warnings 2018-09-24 00:52:19 -07:00
Yann Collet 00c18c0c88 simplified "slows down when compression blocked" 2018-09-21 16:35:43 -07:00
Yann Collet c484345a82 Merge branch 'mingw' into adapt 2018-09-21 16:00:46 -07:00
Yann Collet bfff4f4809 ensure all writes to job->cSize are mutex protected
even when reporting errors,
using a macro for code brevity, as suggested by @terrelln,
2018-09-21 16:00:39 -07:00
Yann Collet 54001f3dbc fix mingw compatibility
only enable backtraces for platforms we know support it
aka mac OS-X and Linux.
can be extended later.
2018-09-21 14:46:09 -07:00
Yann Collet ca02ebee07 removed static variables
so that --adapt can work on multiple input files too
2018-09-19 15:25:50 -07:00
Yann Collet 89bc309d90 error out when --adapt is associated with --single-thread
since they are not compatible
2018-09-19 14:49:13 -07:00
Yann Collet 2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
Casey McGinty b9118ecdab
Update comments, and LD flag usage in Make 2018-09-11 14:49:47 -07:00
Casey McGinty d4337b6f1d
Move ABRThandler func out of internal lib 2018-09-11 11:39:49 -07:00
Casey McGinty a06574fc97
Print a stack trace on unexpected term signal (e.g. SIGABRT)
For OSX and Linux, add a signal handler to SIGABRT, SGIFPE, SIGILL,
SIGSEGV, and SIGBUS. When the program terminates unexpectedly the
handler will print the current stack to the terminal to help determine
the location of the failure.

On OSX the output will look like:

```
Stack trace:
4   zstd                                0x000000010927ed96 main + 16886
5   libdyld.dylib                       0x00007fff767d1015 start + 1
6   ???                                 0x0000000000000001 0x0 + 1
```

On Linux the output will look like:

```
Stack trace:
./zstd() [0x4b8e1b]
./zstd() [0x4b928a]
./zstd() [0x403dc2]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5e0fbb0445]
./zstd() [0x405754]
```

As is, the code does not function on WIN32.

See also: https://oroboro.com/stack-trace-on-crash/
2018-09-06 18:46:52 -07:00
Eden Zik 78af534f82 Fixed unsafe string copy and concat in `fileio.c`.
Per warnings from flawfinder: "Does not check for buffer overflows when
copying to destination [MS-banned] (CWE-120). Consider using snprintf,
strcpy_s, or strlcpy (warning: strncpy easily misused).".

Replaced called to strcpy and strcat in `fileio.c` to calls with a
specified size (`strncpy` and `strncat`).

Tested the changes on OSX, Linux, Windows.
On OSX + Linux, changes were tested with ASAN. The following flags were
used: 'check_initialization_order=1:strict_init_order=1:detect_odr_violation=1:detect_stack_use_after_return=1'

To reproduce warning:
./flawfinder.py ./programs/fileio.c
2018-08-20 22:15:24 -04:00
Yann Collet 105677c6db created ZSTDMT_toFlushNow()
tells in a non-blocking way if there is something ready to flush right now.
only works with multi-threading for the time being.

Useful to know if flush speed will be limited by lack of production.
2018-08-17 18:11:54 -07:00
Yann Collet 09e63c58ac fix : no longer slow down on input saturation
only slows down when all buffers are full
2018-08-17 16:27:43 -07:00
Yann Collet 8b674d7dc7 ensured compression level is maxed at ZSTD_maxCLevel() 2018-08-17 16:01:56 -07:00
Yann Collet 3e4617ef54 frameProgression reports nbActiveWorkers and output flushed 2018-08-14 11:49:25 -07:00
Yann Collet 0853f86044 adaptive mode uses default window size of 8 MB 2018-08-13 13:13:22 -07:00
Yann Collet 33f7709c71 fileio: changed parameter type from ptr to plain structure
safer : this parameter is read-only,
we don't want original structure to be modified
2018-08-13 13:02:03 -07:00
Yann Collet f3aa510738 rateLimiter does not "catch up" when input speed is slow 2018-08-13 11:38:55 -07:00
Yann Collet e7a49c6683 introduced command --adapt 2018-08-11 20:48:06 -07:00
Yann Collet 9d26cb6a75 slow down faster when output speed is limited 2018-08-09 17:44:30 -07:00
Yann Collet 3d7b533f68 Merge branch 'dev' into adapt 2018-08-09 15:57:36 -07:00
Yann Collet 754942cb79 fixed assert() condition 2018-08-09 15:57:19 -07:00
Yann Collet 2dd76037be zstd cli can increase level when input is too slow 2018-08-09 15:51:30 -07:00
Yann Collet 79a35ac20d minor code comments improvements 2018-08-09 15:16:31 -07:00
Nick Terrell 4e706d7f2c fileio: Error in compression on read errors
We can write a corrupted file if the input file errors during a read.
We should return a non-zero error code in this case.
2018-07-17 15:26:30 -07:00
W. Felix Handte 712a9fd972 Allow Invoking `zstd --list` When `stdin` is not a `tty`
Also now returns an error when no inputs are given.

New proposed behavior:

```
felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l; echo $?
No files given
1
felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l Makefile.zst; echo $?
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0     3.08 KB      10.92 KB  3.544  XXH64  Makefile.zst
0
felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l <Makefile.zst; echo $?
zstd: --list does not support reading from standard input
No files given
1
felix@odin:~/prog/zstd (list-stdin-check)$ ./zstd -l Makefile.zst <Makefile.zst; echo $?
Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename
     1      0     3.08 KB      10.92 KB  3.544  XXH64  Makefile.zst
0
felix@odin:~/prog/zstd (list-stdin-check)$
```
2018-06-29 15:33:44 -04:00
Yann Collet 6768cf53fd
Merge pull request #1190 from terrelln/ldm-adjust
Adjust advanced parameters to source size
2018-06-19 14:40:56 -07:00
Yann Collet c0b6ce95b1
Merge pull request #1179 from supertopher/dev
Improves UX for --list command's lack of support for pipes
2018-06-19 14:36:30 -07:00
Nick Terrell 1d0fcde45d Use debug.h in fileio.c 2018-06-18 15:51:21 -07:00
Nick Terrell 3841dbac84 Adjust advanced parameters to source size
In the new advanced API, adjust the parameters even if they are explicitly
set. This mainly applies to the `windowLog`, and accordingly the `hashLog`
and `chainLog`, when the source size is known.
2018-06-18 15:49:31 -07:00
Topher Lubaway 6bca3fb4bf Reduce noise in diff
putting the code block back on the exact line it came from
2018-06-13 14:32:59 -07:00