to produce same content on both 32 and 64-bit platforms
by removing floating from literal table determination.
also : added checksum trace in compression control test,
so that it's easier to determine if test fails
as a consequence of compressing a different sample.
utime is deprecated by POSIX 2008 and optionally not available with
uClibc-ng.
Got rid of a few useless headers in timefn.h.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Compiling with clang-8 fails with the following errors:
largeNbDicts.c:562:37: error: implicit conversion turns floating-point
number into integer: 'const double' to 'U64' (aka 'unsigned long')
[-Werror,-Wfloat-conversion]
U64 const dTime_ns = result.nanoSecPerRun;
~~~~~~~~ ~~~~~~~^~~~~~~~~~~~~
zstdcli.c:300:5: error: '@return' command used in a comment that is
not attached to a function or method declaration
[-Werror,-Wdocumentation]
* @return 1 means that cover parameters were correct
~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
zstdcli.c:301:5: error: '@return' command used in a comment that is
not attached to a function or method declaration
[-Werror,-Wdocumentation]
* @return 0 in case of malformed parameters
~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This resolves a race condition where zstd or unzstd may expose read
permissions beyond the original file allowed. Mode 600 is used
temporarily during the compression and decompression write stage
and the new file inherits the original file’s mode at the end.
Fixes#1630
not sure why, but msan fires an "unitialized variable" error
when time gets properly initialized by timespec_get().
Maybe in some cases, not all bytes of the structure are initialized ?
Or maybe msan fails to detect the initialization ?
Anyway, pre-initializing the variable before passing it to timespec_get() works.
C11 mandates the definition of timespec_get() and TIME_UTC.
However, FreeBSD11 announce C11 compliance, but does not provifr timespec_get(),
breaking link stage for benchfn.
Since it does not provide TIME_UTC either, which is also required by C11,
test this macro: this will automatically rule out FreeBSD 11 for this code path
(it will use the backup C90 path instead, based on clock_t).
The issue seeems fixed in FreeBSD 12.
benchfn used to rely on mem.h, and util,
which in turn relied on platform.h.
Using benchfn outside of zstd required to bring all these dependencies.
Now, dependency is reduced to timefn only.
This required to create a separate timefn from util,
and rewrite benchfn and timefn to no longer need mem.h.
Separating timefn from util has a wide effect accross the code base,
as usage of time functions is widespread.
A lot of build scripts had to be updated to also include timefn.
While fixing the detection of symbolic links on OpenBSD I noticed
inconsistent behaviour:
$ echo hello > hello
$ ln -s hello world
$ zstd hello world
Warning : world is a symbolic link, ignoring
hello :316.67% ( 6 => 19 bytes, hello.zst
$ ls *.zst
hello.zst
$ zstd world
world :316.67% ( 6 => 19 bytes, world.zst)
$ ls *.zst
hello.zst world.zst
In #1520 it is described that FreeBSD doesn't detect symbolic links. The
same is true for OpenBSD. This diff fixes this issue for OpenBSD. I'm
guessing that something similar works for FreeBSD as well. However, I'm
unable to test this.
was broken by #1505.
I'm surprised it passed CI tests.
LZ4 tests are part of the "Extended" tests on Travis CI,
which are run on "master" and in "cron" jobs.
Since latest cron job did not failed,
especially this one : https://travis-ci.org/facebook/zstd/jobs/484365040
it suggests cron jobs are no longer using `dev` branch.
To be investigated
fseek() doesn't indicate when it moves past the end of a file.
Consequently, if a file is truncated within its last block, the error would't be detected.
This PR adds a test scenario that induces this situation using a small compressed file of only one block in size.
This test is added to tests/playTests.sh
Check is implemented by ensuring that the filehandle position is equal to the filesize upon exit.
Similar to Apple, use the native physical core count sysctl, when available.
This is a little repetitive (it's basically the __APPLE__ method plus the
otherBSD method concatenated together) but seemed clearer than any way that
would totally eliminate repetition.
The __FreeBSD_version check only tests the version of the FreeBSD kernel
that zstd is compiled on; importantly, it may be run on a different version.
So the compile-time check is a little naive and needs to be able to fallback
to work on older versions of FreeBSD. For a similar reason, it may make
sense to simply eliminate the __FreeBSD_version check entirely. The
tradeoff is that a spurious sysctlbyname would be issued when -T0 is used on
older kernels.
due to bad support of inode identifiers.
On Visual, option is limited to same file name,
which is imperfect, but way better than disabling the feature entirely.
It's enough to pass associated tests.
On Windows, the equivalent of `/dev/null` is `NUL`.
When tests are run under msys2/minGW,
the environment identifies itself as Windows,
hence the script uses `NUL` instead of `/dev/null`
but the environment will consider `NUL` to be a regular file name.
Consequently, `NUL` will be overwritten during tests,
triggering an error.
This patch uses flag `-f` to force such overwrite
passing the test.
as suggested in #1441.
generally U32 and unsigned are the same thing,
except when they are not ...
case : 32-bit compilation for MIPS (uint32_t == unsigned long)
A vast majority of transformation consists in transforming U32 into unsigned.
In rare cases, it's the other way around (typically for internal code, such as seeds).
Among a few issues this patches solves :
- some parameters were declared with type `unsigned` in *.h,
but with type `U32` in their implementation *.c .
- some parameters have type unsigned*,
but the caller user a pointer to U32 instead.
These fixes are useful.
However, the bulk of changes is about %u formating,
which requires unsigned type,
but generally receives U32 values instead,
often just for brevity (U32 is shorter than unsigned).
These changes are generally minor, or even annoying.
As a consequence, the amount of code changed is larger than I would expect for such a patch.
Testing is also a pain :
it requires manually modifying `mem.h`,
in order to lie about `U32`
and force it to be an `unsigned long` typically.
On a 64-bit system, this will break the equivalence unsigned == U32.
Unfortunately, it will also break a few static_assert(), controlling structure sizes.
So it also requires modifying `debug.h` to make `static_assert()` a noop.
And then reverting these changes.
So it's inconvenient, and as a consequence,
this property is currently not checked during CI tests.
Therefore, these problems can emerge again in the future.
I wonder if it is worth ensuring proper distinction of U32 != unsigned in CI tests.
It's another restriction for coding, adding more frustration during merge tests,
since most platforms don't need this distinction (hence contributor will not see it),
and while this can matter in theory, the number of platforms impacted seems minimal.
Thoughts ?
Compare the input and output files by their inode number and
refuse to open the output file if the input file is the same.
This doesn't work when (de)compressing multiple files to a single
file, but that is a very uncommon use case, mostly used for
benchmarking by me.
Fixes#1422.
The `--no-progress` flag disables zstd's progress bars, but leaves
the summary.
I've added simple tests to `playTests.sh` to make sure the parsing
works.
from overlapSizeLog.
Reasoning :
`overlapLog` is already used everwhere, in the code, command line and documentation.
`ZSTD_c_overlapSizeLog` feels unnecessarily different.
ZSTD_compress_generic() is renamed ZSTD_compressStream2().
Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
When multiple -arch flags are used, the compiler invokes itself once for
each architecture. Apparently, input on stdin is consumed by the
compilation of the first arch and is no longer available to the
compilation of the second arch, which results in a build failure and the
potentially incorrect determination that a feature is not available. So
write the feature detection source to a file instead of using stdin.
answering #1407.
Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
fix#1379
decodecorpus was generating one extraneous byte when `nbSeq==0`.
This is disallowed by the specification.
The reference decoder was just skipping the extraneous byte.
It is now stricter, and flag such situation as an error.
gcc-8 on Linux doesn't like usage of strncat :
`warning: ‘strncat’ output truncated before terminating nul copying as many bytes from a string as its length`.
Not sure what was wrong, it might be a false positive,
but the logic is simple enough to replaced by a simple `memcpy()`,
thus avoiding the shenanigans of null-terminated strings.
following #1356,
only enable backtrace compilation on linux+glibc.
Also, disable backtrace by default from "release" compilation,
so that less platforms get impacted by the new requirements.
Can be manually enabled/disabled using BACKTRACE=1/0.
and slightly refactored affected function.
Honestly, the formula calculating variance should get a second reviewing round,
it's not clear if it's correct.
some non-trivial changes to platform.h and util.h,
initially related to compilation for Haiku,
but I used this opportunity to make them cleaner
and add some documentation.
Noticed several tests that could be improved
(too harsh conditions, useless exception, etc.)
but I did not dare modifying too many tests just before release.
note : for some reason,
scan-build version on my laptop found problems within fastcover.c
that scan-build on travisCI does not flag.
They are, as usual, false positive :
the analyzer does not understand that a table (`offset`) is correctly filled before usage.
for Linux and Mac OS-X.
Note : the backtraces fires up through a trap
before the sanitizer get a chance to report.
There are situations where the sanitizer report is actually preferable.
It might be good to consider a kind of build macro
which can disable backtrace
when sanitizer is enabled.
Bimba Shrestha
Sen Huang
Mike Swanson
Nick Magerko
Nick Terrell
Ed Maste
Yann Collet
Yann Collet
Rosen Penev
LeeYoung624
Tyler Tran
Qin Li
Ephraim Park
Tyler-Tran
W. Felix Handte
Yosuke Tanigawa
cev
myzhang1029
Josh Soref
Ori Peleg
shakeelrao
Björn Ketelaars
Peter (Stig) Edwards
Karl Ostmo
Lzu Tao
Conrad Meyer
Yi Jin
Yi Jin
Denis Ahrens
Ryan Schmidt
Bernhard M. Wiedemann
Sam Russell
Rohit Jain
Julian Fessard
maxice8
Jerome Duval