Commit Graph

7745 Commits (c6548eac8ea6c9c6faf464d7dda9b6ec81123e5d)

Author SHA1 Message Date
Bimba Shrestha 9b13077cfb Revert to old zstdgrep 2020-05-13 15:12:14 -05:00
Yann Collet ad34d58218
Merge pull request #2134 from terrelln/align-lazy
[greedy] Fix performance instability
2020-05-12 20:49:10 -07:00
Nick Terrell 70c80e19e6 [greedy] Fix performance instability 2020-05-12 17:51:16 -07:00
Nick Terrell c3e921c639
Merge pull request #2131 from terrelln/raw-dict-fuzzer
Fix rare scenario with lazy parser, dictionary, and repcodes
2020-05-12 17:44:31 -07:00
W. Felix Handte d9a1e37aec Nit: Fix Size Type for 32-bit 2020-05-12 18:03:31 -04:00
Nick Terrell f800e72a3c [lib] Fix assertion when dictionary is prefix 2020-05-12 14:33:59 -07:00
W. Felix Handte 1aa6c7ccce Assert We Allocated Approximately What We Expected To 2020-05-12 16:55:03 -04:00
W. Felix Handte 27e2482217 Minor Refactor 2020-05-12 16:55:03 -04:00
W. Felix Handte afc2488973 Handle Non-Static CCtxes in Estimation 2020-05-12 16:54:33 -04:00
Yann Collet 5dca17db1f
Merge pull request #2132 from bimbashrestha/grep-rm-test
[build] removing -f test (grep usage not supported on mac)
2020-05-12 13:49:38 -07:00
Yann Collet f5f0bb2591
Merge pull request #2125 from Artoria2e5/patch-2
programs/Makefile: make it actually work with clang
2020-05-12 13:48:23 -07:00
W. Felix Handte 7ed996f5a0 Consolidate CCtx Size Estimation Code
This commit pulls out the internals of `ZSTD_estimateCCtxSize_usingCCtxParams`
into a helper. It then migrates two other callsites to use that helper,
a small optimization for `ZSTD_estimateCStreamSize_usingCCtxParams`, which
folds the buffer sizing into the helper, and then `ZSTD_resetCCtx_internal`,
which is more invasive.

This attempts to guarantee that the estimates returned to users are always
correct.
2020-05-12 16:26:53 -04:00
Bimba Shrestha 0453cfa8f5 removing -f test (grep usage not supported on mac) 2020-05-12 15:18:43 -05:00
Nick Terrell 3c1eba4d99 [lib] Fix lazy repcode validity checks 2020-05-12 12:25:06 -07:00
Yann Collet d6225775be
Merge pull request #2130 from facebook/initStatic_fix
Fix for initStatic
2020-05-12 12:03:22 -07:00
Nick Terrell 4e0515916d [lib] Fix repcode validation in no dict mode 2020-05-12 11:57:15 -07:00
Nick Terrell 6d687a8816 [lib] Fix dictionary + repcodes + optimal parser 2020-05-12 10:36:53 -07:00
Nick Terrell 4b88bd3ee0 [lib][fuzz] Assert sequences are valid in round trip tests 2020-05-11 20:38:49 -07:00
Yann Collet e001715b3d fixed asan test 2020-05-11 20:35:47 -07:00
Yann Collet 20bd246045 blindfix for VS macro redefinition 2020-05-11 19:29:36 -07:00
Yann Collet 76e726e3be updated documentation for ZSTD_estimate*()
make it clearer that tighter memory estimation
can be provided using advanced functions
on the condition of a defined input size bound.
2020-05-11 19:21:50 -07:00
Nick Terrell 80d3585e31 [lib] Fix lazy parser with dictionary + repcodes 2020-05-11 19:04:30 -07:00
Nick Terrell 1185dfb8d1 [fuzz] Add raw dictionary content fuzzer 2020-05-11 19:03:33 -07:00
Nick Terrell 301a62fe08 [fuzz] Fix compress bound for dictionary_round_trip 2020-05-11 19:00:52 -07:00
Bimba Shrestha e7d2391e9a
[doc] measuring performance docs (#2117)
* performance measuring docs

* spelling

* combining advanced and simple section

* zstd benchmark title change
2020-05-11 18:55:01 -07:00
Yann Collet ed9eb17471
Merge pull request #2124 from facebook/initStatic_tests
re-inforced tests for initStaticCCtx
2020-05-11 18:54:22 -07:00
Yann Collet f13eb2ec01
Merge pull request #2123 from facebook/zstd_help
updated `zstd -h`
2020-05-11 18:54:07 -07:00
Yann Collet 91ad01218e updated initStatic tests
differentiate small CCtx for small inputs
from full CCtx
from CStream contexts.

Ensure allocation & resize tests are more precise.
2020-05-11 18:50:10 -07:00
Yann Collet 608f1bfc4c fixed context downsize with initStatic
When context is created using initStatic,
no resize is possible.

fix : only bump oversizeDuration when !initStatic
2020-05-11 18:16:38 -07:00
Yann Collet 58227db405 Merge branch 'dev' into initStatic_tests 2020-05-11 16:51:13 -07:00
Yann Collet d8b40fe0be
Merge pull request #2129 from felixhandte/fix-cctx-size-estimate
Fix ZSTD_estimateCCtxSize() Under ASAN
2020-05-11 16:36:46 -07:00
W. Felix Handte c6636afbbb Fix ZSTD_estimateCCtxSize() Under ASAN
`ZSTD_estimateCCtxSize()` provides estimates for one-shot compression, which
is guaranteed not to buffer inputs or outputs. So it ignores the sizes of the
buffers, assuming they'll be zero. However, the actual workspace allocation
logic always allocates those buffers, and when running under ASAN, the
workspace surrounds every allocation with 256 bytes of redzone. So the 0-sized
buffers end up consuming 512 bytes of space, which is accounted for in the
actual allocation path through the use of `ZSTD_cwksp_alloc_size()` but isn't
in the estimation path, since it ignores the buffers entirely.

This commit fixes this.
2020-05-11 18:58:19 -04:00
Nick Terrell 738f49178b
Merge pull request #2128 from terrelln/benches
Update benchmarks and add fast mode
2020-05-11 13:53:18 -07:00
Nick Terrell f2f86b50ae Update benchmarks and add fast mode 2020-05-11 12:36:36 -07:00
Bimba Shrestha 0c04e641e9
Merge pull request #2127 from bimbashrestha/docs1
[doc] documenting non-supported grep options for zstdgrep to README
2020-05-11 12:51:35 -05:00
Bimba Shrestha ef467fd428 documenting non-supported grep options for zstdgrep 2020-05-11 12:22:45 -05:00
Yann Collet 93ff2fb329
Merge pull request #2120 from facebook/clevel_doc
updated zstd CLI documentation
2020-05-11 09:42:33 -07:00
Mingye Wang c3398d2dda
programs/Makefile: make it actually work with clang
Recent versions of clang seem to require a step of conversion from the profraw file to a "compiler-readable" profdata file to let PGO work.
2020-05-10 16:36:08 +08:00
Yann Collet dd026ca505 re-inforced tests for initStaticCCtx
ensure that `estimateCCtxSize()` works as intended.
2020-05-09 11:30:45 -07:00
Felix Handte aa9474ee9a
Merge pull request #2122 from felixhandte/unused-variables
Compile Under `-Wunused-variable`
2020-05-08 18:17:38 -04:00
Yann Collet 37b2dafbed minor clarification on `-o file` 2020-05-08 14:20:47 -07:00
Yann Collet 5c67a4e737 Merge branch 'dev' of github.com:facebook/zstd into dev 2020-05-08 14:15:56 -07:00
Yann Collet e5366bee16 updated `zstd -h`
to help readability.
Group arguments into categories,
try to give better visibility to more common arguments.
2020-05-08 14:14:46 -07:00
Felix Handte 3b39ce6b52
Merge pull request #2116 from felixhandte/pkg-config-again-again
Fix pkg-config File Generation Again Again
2020-05-08 17:14:43 -04:00
W. Felix Handte 376c26bf56 Use Unused Variable 2020-05-08 16:43:39 -04:00
Yann Collet d564d56440 document additional compilation macros
ZSTD_NOBENCH, ZSTD_NODICT, ZSTD_NOCOMPRESS, ZSTD_NODECOMPRESS
2020-05-08 13:35:59 -07:00
W. Felix Handte 87c541c5f9 Only Trigger libzstd.pc Build on Unix-Like Platforms
We don't even define the rule on unsupported platforms.
2020-05-08 16:11:32 -04:00
W. Felix Handte 85801b99d7 Test libzstd.pc Build on Travis OS X 2020-05-08 16:11:32 -04:00
W. Felix Handte 78aa9373cb Add libzstd.pc Build to More Aggregate Targets in Makefiles 2020-05-08 16:11:32 -04:00
W. Felix Handte 15561bcf74 Fix pkg-config File Generation Again Again
Resubmission of #2001. This switches the `sed` invocations to use `-E`,
extended regex syntax, which is better standardized across platforms.
I guess.

Same test plan:

```
make -C lib clean libzstd.pc
cat lib/libzstd.pc

echo # should fail
make -C lib clean libzstd.pc     LIBDIR=/foo
make -C lib clean libzstd.pc INCLUDEDIR=/foo
make -C lib clean libzstd.pc     LIBDIR=/usr/localfoo
make -C lib clean libzstd.pc INCLUDEDIR=/usr/localfoo
make -C lib clean libzstd.pc     LIBDIR=/usr/local/lib     prefix=/foo
make -C lib clean libzstd.pc INCLUDEDIR=/usr/local/include prefix=/foo

echo # should succeed
make -C lib clean libzstd.pc     LIBDIR=/usr/local/foo
make -C lib clean libzstd.pc INCLUDEDIR=/usr/local/foo
make -C lib clean libzstd.pc     LIBDIR=/usr/local/
make -C lib clean libzstd.pc INCLUDEDIR=/usr/local/
make -C lib clean libzstd.pc     LIBDIR=/usr/local
make -C lib clean libzstd.pc INCLUDEDIR=/usr/local
make -C lib clean libzstd.pc     LIBDIR=/tmp/foo prefix=/tmp
make -C lib clean libzstd.pc INCLUDEDIR=/tmp/foo prefix=/tmp
make -C lib clean libzstd.pc     LIBDIR=/tmp/foo prefix=/tmp/foo
make -C lib clean libzstd.pc INCLUDEDIR=/tmp/foo prefix=/tmp/foo

echo # should also succeed
make -C lib clean libzstd.pc prefix=/foo LIBDIR=/foo/bar INCLUDEDIR=/foo/
cat lib/libzstd.pc

mkdir out
cd out
cmake ../build/cmake
make
cat lib/libzstd.pc
```
2020-05-08 16:11:32 -04:00