Commit Graph

6149 Commits (0ed8ee4a371a11b91a68677cdff7a7efee79fe57)

Author SHA1 Message Date
Mitchell Grenier a424899637 Fix buck for lib 2018-11-30 13:45:16 +00:00
Yann Collet 090bc808a8
Merge pull request #1432 from terrelln/regression
[regression] Add initial regression test framework
2018-11-29 16:06:40 -08:00
Nick Terrell 4aaa36f74b [regression] Add initial regression test framework
The regression tests run nightly or on the `regression`
branch for convenience. The results get uploaded as the
artifacts of the job. If they change, check the diff
printed in the job. If all is well, download the new
results and commit them to the repo.

This code will only run on a UNIX like platform. It
could be made to run on Windows, but I don't think that
it is necessary. It also uses C99.

* data: This module defines the data to run tests on.
  It downloads data from a URL into a cache directory,
  checks it against a checksum, and unpacks it. It also
  provides helpers for accessing the data.
* config: This module defines the configs to run tests
  with. A config is a set of API parameters and a set of
  CLI flags.
* result: This module is a helper for method that defines
  the result type.
* method: This module defines the compression methods
  to test. It is what runs the regression test using the
  data and the config. It reports the total compressed
  size, or an error/skip.
* test: This is the test binary that runs the tests for
  every (data, config, method) tuple, and prints the
  results to the output file and stderr.
* results.csv: The results that the current commit is
  expected to produce.
2018-11-29 14:33:04 -08:00
Yann Collet f6591e4a04
Merge pull request #1430 from lzutao/fix-zstdgrep
Fix  #1428 - zstdgrep now returns 1 on unmatch
2018-11-28 14:07:40 -08:00
Lzu Tao d095adf9fb Add simple test for zstdgrep 2018-11-29 03:39:47 +07:00
Lzu Tao 3d18b4764d Prevent globbing on non-quoting variable 2018-11-29 03:04:40 +07:00
Lzu Tao c046e0b626 Fix #1428 - zstdgrep now returns 1 on unmatch 2018-11-29 02:45:01 +07:00
Yann Collet 0da2f1a582
Merge pull request #1429 from denis2342/dev
fixed a typo
2018-11-28 08:45:46 -08:00
Denis Ahrens 71b8ee1bf1 fixed a typo 2018-11-28 04:51:16 +01:00
Yann Collet 74f8a5c51d
Merge pull request #1426 from lzutao/upgrade-mesonbuild
Update meson build system
2018-11-27 10:55:41 -08:00
Lzu Tao 9a721e5216 Update meson build system
NOTE: This commit only tested on Linux (Ubuntu 18.04). Windows
build may not work as expected.

* Use meson >= 0.47.0 cause we use install_man function
* Add three helper Python script:
  * CopyFile.py: To copy file
  * CreateSymlink.py: To make symlink (both Windows and Unix)
  * GetZstdLibraryVersion.py: Parse lib/zstd.h to get zstd version
  These help emulating equivalent functions in CMake and Makefile.
* Use subdir from meson to split meson.build
  * Add contrib build
  * Fix other build
* Add new build options
  * build_programs: Enable programs build
  * build_contrib: Enable contrib build
  * build_tests: Enable tests build
  * use_static_runtime: Link to static run-time libraries on MSVC
  * zlib_support: Enable zlib support
  * lzma_support: Enable lzma support
2018-11-28 01:08:34 +07:00
Lzu Tao 9bd8f6a00c Rename and update build instruction in README file to README.md 2018-11-28 01:08:34 +07:00
Lzu Tao 2abd5139a5 Add meson build guide 2018-11-28 01:08:34 +07:00
Yann Collet 64569a184e
Merge pull request #1419 from ryandesign/universal
Fix feature detection with multiple -arch flags
2018-11-26 17:37:19 -08:00
Yann Collet d3a0c71259 pushed experimental parameters
into ZSTD_STATIC_LINKING_ONLY section
2018-11-21 16:18:55 -08:00
Yann Collet d4d4e109e9 getParameter fills an int*
rather than an unsigned*
for consistency
since type of setParameter() changed to int.
2018-11-21 15:37:26 -08:00
Yann Collet fea920615c promote ZSTD_findFrameCompressedSize() into staging area 2018-11-21 15:25:50 -08:00
Yann Collet 41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet 5d3592398d fixed fall-through 2018-11-20 16:09:33 -08:00
Yann Collet 5c6d4b18ac completed implementation of ZSTD_cParam_getBounds()
for all parameters
2018-11-20 16:06:00 -08:00
Yann Collet 2e7fd6a2cb fixed remaining searchLength invocations 2018-11-20 15:13:27 -08:00
Yann Collet e874dacc08 changed searchLength into minMatch
refactored all relevant API and calls
for consistency.
2018-11-20 14:56:07 -08:00
Yann Collet 114bd4346e changed enum type name to ZSTD_ResetDirective
for naming consistency :
types should start with a capital letter (after prefix)
2018-11-20 12:00:20 -08:00
Yann Collet 3b838abf97 ZSTD_CCtx_setParameter : `value` argument is now `int`
for compatibility with compression level
2018-11-20 11:53:01 -08:00
Yann Collet 19e5f2a35b removed some constants and _simpleArgs() from staging
constants that *may* change in the future
will be accessed through functions instead
(to be created).

_simpleArgs() variant do not have (yet) a clear enough added value
to deserve "stable" status.
2018-11-19 17:38:15 -08:00
Nick Terrell 787532b91a
Merge pull request #1418 from ryandesign/cpuid-i386
Fix i386 build failure "Junk character 13"
2018-11-16 11:01:46 -08:00
Ryan Schmidt 46d470c20c Fix feature detection with multiple -arch flags
When multiple -arch flags are used, the compiler invokes itself once for
each architecture. Apparently, input on stdin is consumed by the
compilation of the first arch and is no longer available to the
compilation of the second arch, which results in a build failure and the
potentially incorrect determination that a feature is not available. So
write the feature detection source to a file instead of using stdin.
2018-11-16 03:49:15 -06:00
Ryan Schmidt ef4df0df4a Fix i386 build failure "Junk character 13" 2018-11-16 02:16:21 -06:00
Yann Collet f07e13b729 fixed fuzz src 2018-11-15 16:53:45 -08:00
Yann Collet 5c68639186 updated ZSTD_DCtx_reset()
signature and behavior is now the same as ZSTD_CCtx_reset()
2018-11-15 16:12:39 -08:00
Yann Collet 4542b9f1cf updated manual 2018-11-15 10:52:24 -08:00
Yann Collet 06c8d5a4f4 Merge branch 'dev' into advancedAPI
fixed rsyncable
2018-11-15 10:51:24 -08:00
Nick Terrell f15f1bfefb
Merge pull request #1416 from terrelln/rsync
- Factor out LDM's hash function for reuse
- Add rsyncable mode to zstdmt and expose it via the advanced API
- Fix `-B`/`--block-size` to actually set the job size
- Add rsyncable tests to `zstreamtest` and `playTests.sh`

Tested by:

```
> cat A.100MB B.100MB C.100MB D.100MB E.100MB | zstd --rsyncable -fo src/file.zst
/*stdin*\            : 48.22%   (524288000 => 252837782 bytes, src/file.zst)
> rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
Total bytes sent: 252868779
total size is 252837782  speedup is 1.00
> echo test > test
> cat test A.100MB test B.100MB test C.100MB test D.100MB test E.100MB | zstd --rsyncable -fo src/file.zst
/*stdin*\            : 48.23%   (524288025 => 252838025 bytes, src/unicorn.tar.zst)
> rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
Total bytes sent: 4605696
total size is 252838025  speedup is 53.60
```

Close #1155.
2018-11-14 18:43:12 -08:00
Nick Terrell 71fe78cbf9 [tests] Add rsyncable tests 2018-11-14 17:00:29 -08:00
Nick Terrell 4316409427 [cli] Set the job size 2018-11-14 17:00:29 -08:00
Nick Terrell f9a671ac37 [cli] Add rsyncable 2018-11-14 17:00:16 -08:00
Nick Terrell b9693d3a49 [lib] Add rsyncable mode
- Add rsyncable mode to multithreaded mode
- Factor out LDM's hash function for reuse
2018-11-14 16:59:57 -08:00
Yann Collet 21a42bf5f9 added advanced decompression api 2018-11-14 16:54:54 -08:00
Yann Collet cf9f4b63b8 fixed fuzz test src code 2018-11-14 14:46:49 -08:00
Yann Collet 7b0391e37e finalized retrofit of ZSTD_CCtx_reset()
updated all depending sources
2018-11-14 13:05:35 -08:00
Yann Collet 5adbad4059 Merge branch 'dev' into advancedAPI 2018-11-14 13:00:37 -08:00
Yann Collet c584e84e68
Merge pull request #1415 from facebook/benchfn
Separating benchfn and benchzstd
2018-11-14 13:00:08 -08:00
Yann Collet ff8d371708 modified ZSTD_CCtx_reset()
which now accepts an enum,
to distinguish between resetting the session, or the parameters (or both).

removed ZSTD_CCtx_resetParameters(), which is redundant.

start replacing invocation of ZSTD_CCtx_reset*() functions

Updated advanced API documentation

trimmed down amount of API staged in RC,
in particular, all functions related to ZSTD_CCtxParams()
seem too advanced.
2018-11-14 12:33:57 -08:00
Yann Collet d7e10a774a added constant ZSTD_WINDOWLOG_LIMIT_DEFAULT
answering #1407.

Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
2018-11-13 18:12:34 -08:00
Yann Collet 2c8fde538f added constant ZSTD_MAGIC_SKIPPABLE_MASK
and updated several API comments
2018-11-13 17:36:35 -08:00
Yann Collet b83d1e7714 removed some `static const` variables
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
2018-11-13 16:56:32 -08:00
Yann Collet 975abf8f5d circumvent gcc-8 strict aliasing warning
because gcc considers that
`void* const *`  and `void**` are 2 different types ...
2018-11-13 16:04:35 -08:00
Yann Collet 768a264200 Merge branch 'dev' of github.com:facebook/zstd into dev 2018-11-13 15:56:36 -08:00
Yann Collet 092c4abd4c bumped version number to v1.3.8 2018-11-13 15:53:38 -08:00
Yann Collet 3ba0d6dd27 fixed decode-only test condition 2018-11-13 14:15:12 -08:00