Commit Graph

8075 Commits (1ebe360d0f3213cc4e4664b461b37a5eec04949b)

Author SHA1 Message Date
Nick Terrell 49069a3f0a
Merge pull request #2265 from terrelln/prefetch-fix
Prefer __builtin_prefetch over inline asm
2020-08-11 10:07:53 -07:00
Nick Terrell e3bda594ae Prefer __builtin_prefetch over inline asm
Reorder the ifdefs for the PREFETCH macros so that the compiler builtin is
favored over the inline assembly for aarch64.
2020-08-10 22:17:18 -07:00
Yann Collet 63eebc86cd
Merge pull request #2262 from felixhandte/fewer-stat-syscalls
Share stat() Calls Where Convenient
2020-08-10 18:20:13 -07:00
W. Felix Handte 953f0a072a Fix MS Build 2020-08-10 17:28:34 -04:00
W. Felix Handte b02cdf63b0 Clean Up Redundant Checks, Rename FIO_remove() -> FIO_removeFile() 2020-08-10 15:43:17 -04:00
W. Felix Handte 51ac0207af Remove UTIL_statFile() and UTIL_statDir(); Decompose Former Call-Sites 2020-08-10 15:28:02 -04:00
W. Felix Handte 93dda988c8 Remove Unused Function UTIL_fileExist() 2020-08-10 15:22:53 -04:00
W. Felix Handte 76878697a4 Re-Organize and Document Prototypes in util.h 2020-08-10 15:16:14 -04:00
W. Felix Handte c1449143c5 Share stat() Calls in Uses of UTIL_chmod() 2020-08-05 12:10:42 -04:00
W. Felix Handte 0a8aacb4db Use stat() to Check that File Should be chmod()-ed
Rather than special-casing a check for `/dev/null`, this uses `stat()` to
avoid `chmod()`-ing any non-regular file. I believe this is the desirable
behavior. `UTIL_chmod()` is never called on directories at the moment, only
output files.
2020-08-05 12:00:12 -04:00
W. Felix Handte 7238cca1a1 Deduplicate Some Low-Hanging Fruit of Redundant Stat Calls 2020-08-05 01:08:34 -04:00
W. Felix Handte 44fa052599 Introduce Variants of Various UTIL Functions that Take Pre-Populated stat_t Structs
Instead of calling `stat()`, these functions accept the result of a previous
`stat()` call on the file in question, which will allow us to make multiple
decisions around a file without redundant `stat()` calls.
2020-08-05 01:00:06 -04:00
W. Felix Handte b6e24bc4dc Rename UTIL_getFileStat() -> UTIL_statFile() and UTIL_getDirectoryStat() -> UTIL_statDir()
I want to introduce versions of many of these functions that take pre-
populated `stat_t` objects and use those rather than doing their own redundant
`stat()` internally. These functions will have `...Stat()` suffixes. So this
commit renames these existing functions into the active voice, to avoid
confusion.
2020-08-05 00:40:16 -04:00
W. Felix Handte 1a1003f996 Mark stat_t Arg to UTIL_setFileStat() const 2020-08-05 00:35:21 -04:00
W. Felix Handte 5fbc6addb6 Additionally Convert UTIL_getFileStat() Calls to UTIL_stat() Where Appropriate 2020-08-05 00:31:48 -04:00
W. Felix Handte 69cb9e7798 Use New Stat Helper 2020-08-05 00:24:32 -04:00
W. Felix Handte b11bea56a5 Introduce Dedicated Helper to Call stat() 2020-08-05 00:21:21 -04:00
Yann Collet 38e38546a4
Merge pull request #2258 from Niadb/dev
Added STATIC_BMI2 for compile time detection of BMI2 on MSVC, when enabled various intrinsics are used
2020-08-04 09:43:59 -07:00
Yann Collet 60b56e3b5f
Merge pull request #2253 from facebook/histvec
optimized histogram
2020-08-02 14:08:42 -07:00
Nick Terrell f85a0f8bcf
Merge pull request #2256 from helloguo/dev
Optimize ZSTD_wildcopy
2020-07-29 11:57:49 -07:00
Yann Collet bff41a4190
Merge pull request #2259 from cwoffenden/vs-warning
Fixed VS variable shadowing warning (single file lib)
2020-07-29 09:21:03 -07:00
Carl Woffenden 5d81d44e40 Fixed VS variable shadowing warning (and added test) 2020-07-29 12:33:39 +02:00
helloguo acb3dd9a68 Use ZSTD_copy16 instead of memcpy 2020-07-28 11:58:46 -07:00
Niadb a8ebc14035
Update bitstream.h
Profiler showed some of these not being inlined on MSVC
2020-07-28 11:17:04 -06:00
Niadb 216a63dcf7
Add files via upload 2020-07-28 02:52:52 -06:00
Niadb 493fd40dca
Add files via upload 2020-07-28 02:52:15 -06:00
helloguo 82b0cd844f Optimize ZSTD_wildcopy 2020-07-27 22:08:52 -07:00
Nick Terrell 01261bc8b6
Merge pull request #2252 from helloguo/dev
Fix preprocessor in ZSTD_wildcopy
2020-07-27 12:59:10 -07:00
Yann Collet 8b9cdd2597 fixed overlapping count & workspace special case 2020-07-26 22:40:21 -07:00
Yann Collet 051232223f optimized histogram
new version easier to vectorize
leads to smaller code and faster execution
notably at the last recombination stage
(basically, fixed cost per block).

Assembly inspected with godbolt

On my laptop, with `clang` and `-mavx2` :
2K block : 1280 MB/s -> 1550 MB/s
8K block : 1750 MB/s -> 1860 MB/s
2020-07-26 22:24:22 -07:00
helloguo 6de87b3a74 fix preprocessor in ZSTD_wildcopy 2020-07-24 10:53:58 -07:00
Yann Collet 15c5e20023 updated documentation 2020-07-23 12:10:57 -07:00
Yann Collet e33e5eece3
Merge pull request #2250 from facebook/twoFieldsArgs
CLI: simplified handling of separated-fields arguments
2020-07-22 09:49:34 -07:00
Yann Collet cf6c4fcbd5
Merge pull request #2249 from facebook/fsefix
import some minor fixes from FSE project
2020-07-21 08:23:37 -07:00
Yann Collet 610c41f90f improved 2-args macro to support both syntaxes
both `--long-commmand=arg` and `--long-command arg`
are supported by macro `NEXT_FIELD()`
so that the long-command only needs to be listed once.

This extends the syntax to support new syntaxes like
`-o=FILE` or `-D=dict`,
though there is no need to advertise this capability for the time being.

Also : added `NEXT_UINT32()`,
which is wrapper around `NEXT_FIELD()`
to read integer parameters.
Use the wrapper to new field, such as `--memlimit`
which can now support both syntaxes too.
2020-07-20 17:41:32 -07:00
Yann Collet 23941eec04 added tests for newly enabled syntax
for --patch-from origin
and --filelist list

Also : removed some constrained syntax tests,
as the new argument parsing syntax is more permissive.

For example :
    zstd file -of dest
used to be disallowed.

It's now allowed, and understood as:
    zstd file -o dest -f
2020-07-17 13:31:15 -07:00
Yann Collet 7280c1dafa extended 2-fields macro usage to number arguments
leading to further simplications
2020-07-17 13:09:23 -07:00
Yann Collet a7e5652672 CLI: simplified handling of separated-fields arguments
this patch reduces complexity associated with
commands requiring a separated arguments
such as :
-o filename
-D dictionary
--output-dir-flat dir
--output-dir-mirror dir

It used to be a multi-stage process with explicit context,
it's now simplified as a single macro.

Thanks to this simplification,
separated arguments logic has also been extended to
--patch-from XXX
--filelist XXX
which extends existing capability using =XXX
--patch-from=XXX
--filelist=XXX

Separated argument is useful for filenames and directories,
as it benefits from shell expansion
such as ~/dir/file
where the ~ is automatically translated by the shell.
In contrast --long-command=FILE does not interpret FILE,
so ~/ is transmitted as is to the main() function,
generally resulting in incorrect file name.
2020-07-17 12:46:36 -07:00
Yann Collet 40f20b6453
Merge pull request #2197 from neheb/hj
meson: remove build requirement for distutils
2020-07-17 09:16:11 -07:00
Yann Collet 62df5bca4c
Merge pull request #2243 from facebook/uclibc
fix uclibc's st_mtim
2020-07-17 09:14:20 -07:00
Yann Collet c224367ede ensure workspace is large enough
even when MAX_TABLELOG is reduced
2020-07-16 20:33:50 -07:00
Yann Collet 21c273da84 import some minor fixes from FSE project 2020-07-16 20:25:15 -07:00
Felix Handte a35f135617
Merge pull request #2246 from ffontaine/dev
programs/util.c: fix build without st_mtime
2020-07-16 12:40:43 -04:00
Yann Collet 0d0fc4544c
Merge pull request #2247 from facebook/revert-2244-dev
Revert "Fix -Wunused-variable under FUZZING_BUILD_MODE..."
2020-07-15 12:42:31 -07:00
Yann Collet a44671b281
Revert "Fix -Wunused-variable under FUZZING_BUILD_MODE..." 2020-07-15 12:42:18 -07:00
Yann Collet 7d3d5f2a20
Merge pull request #2244 from hctim/dev
Fix -Wunused-variable under FUZZING_BUILD_MODE...
2020-07-15 12:40:17 -07:00
Fabrice Fontaine 26d01bdb26 programs/util.c: fix build without st_mtime
Since version 1.4.5 and commit
5af8cb7aea, if st_mtime is not defined,
programs/util.c uses utime without including utime.h which will raise
the following build failure on some of the buildroot autobuilders:

util.c: In function 'UTIL_setFileStat':
util.c:161:24: error: storage size of 'timebuf' isn't known
         struct utimbuf timebuf;
                        ^~~~~~~

Fixes:
 - http://autobuild.buildroot.org/results/be902c5d110f37bce622a2215191f155b7d3e7e0

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
2020-07-15 21:24:13 +02:00
Mitch Phillips 23b55d6b3e Fix -Wunused-variable under FUZZING_BUILD_MODE...
Fuzzing build modes (FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) doesn't
necessarily imply that assert() is enabled, according to the manual.

When the current do-nothing is expanded under -Wunused-variable (-Wall),
it results in unused variables in some of the FUZZING_BUILD_MODE...
blocks.

This patch extends the do-nothing to avoid the unused variable.
2020-07-14 09:03:02 -07:00
Yann Collet 1dcc478796 fix uclibc's st_mtim
reported and suggested by @ewildgoose (#1872)
2020-07-13 14:16:33 -07:00
Nick Terrell bd21e4b264
Merge pull request #2242 from evverx/cifuzz-ubsan-msan
cifuzz: turn on UBsan and MSan
2020-07-13 14:26:13 -04:00