1747 Commits

Author SHA1 Message Date
Yann Collet
72bd2a83a0 reduce length of scanbuild static analyzer test
This was ~30mn, by far the longest run on travisCI.
That's because it re-analyzes multiple times the same files (library files notably).
It also performs actions that make no sense for the static analyzer purpose,
such as building the single-file library.

Reduced time spent in this test by reducing its scope :
just build the CLI, and obviously the library along it.
These are the only ones that really deserve to be analyzed.

Unfortunately, it still results in a number of false positives when using newer versions of scanbuild
(each version of scanbuild generates a different list of false positives).
These will have to be fixed before transfering to Github Actions.
2021-08-29 15:26:31 -07:00
Yann Collet
7f37b8a547 accelerate versionsCompatibilityTest
by allowing parallel build of units,
and reducing optimization levels.

Parallel build is only effective on "recent" versions of `zstd`,
as previously, the list of units was passed as a list of source files,
which is something neither `make` nor `gcc` can parallelize.
So its impact is mildly effective (-20%).

Reducing optimization level to `-O1` makes compilation much faster.
It also makes runtime slower,
but in this test, compilation time dominates run time.
The savings are very significant (-50%).

On my test system, it reduces the length of this test from 13mn to 5mn.
2021-08-29 14:48:11 -07:00
Clément Chigot
6ef6cd7999 test: avoid /dev/full on AIX 2021-08-13 10:25:50 +02:00
Clément Chigot
399849e236 Makefile: add AIX support
For lib, AIX linker doesn't allow --soname.
2021-08-13 10:25:14 +02:00
sen
6a258043f5
Merge pull request #2692 from senhuang42/rebalance_clevel
[RFC] Rebalance compression levels
2021-08-06 12:51:31 -04:00
Sen Huang
539b3aab9b Optimize 32-bit VecMask_next() 2021-08-04 17:14:58 -04:00
Felix Handte
ae131282f5
Merge pull request #2742 from felixhandte/set-mtime
Set mtime on Output Files
2021-08-04 16:44:46 -04:00
senhuang42
e411040ea1 Add 64 row entry support for lazy 2021-08-04 16:19:12 -04:00
W. Felix Handte
a8f4612eab Remove sleep()s in Test; Replace with Artificial mtime
This behavior of `touch` is standardized in POSIX, so it should be available.
2021-08-04 15:56:46 -04:00
senhuang42
aa1957477b Improve Huffman sorting algorithm 2021-08-04 12:43:34 -04:00
W. Felix Handte
ead41bcb4e Add Test to Verify mtime is Copied to Destination 2021-08-03 17:22:58 -04:00
Nick Terrell
d8a0797268 [fuzz] Add Huffman round trip fuzzer
* Add a Huffman round trip fuzzer
* Fix two minor bugs in Huffman that aren't exposed in zstd
  - Incorrect weight comparison (weights are allowed to be equal to
    table log).
  - HUF_compress1X_usingCTable_internal() can return compressed
    size >= source size, so the assert that `cSize <= 65535` isn't
    correct, and it needs to be checked instead.
2021-08-03 08:10:06 -07:00
Nick Terrell
46f2710562 [HUF] Improve Huffman encoding speed
Improve Huffman encoding speed by 20% for gcc and 10% for clang.

| Compiler |     Benchmark     | Config  |   Dataset   | Ratio | Speed MB/s (dev) | Speed MB/s (huf-cspeed) | Speed MB/s (huf-cspeed - dev) |
|----------|-------------------|---------|-------------|-------|------------------|-------------------------|-------------------------------|
| gcc      | compress          | level_1 | enwik7      | 2.43  | 253.70           | 258.72                  | 2.0%                          |
| gcc      | compress          | level_1 | silesia     | 2.88  | 341.90           | 348.15                  | 1.8%                          |
| gcc      | compress_literals | level_1 | enwik7      | 1.49  | 761.83           | 912.76                  | 19.8%                         |
| gcc      | compress_literals | level_1 | silesia     | 1.28  | 754.83           | 902.37                  | 19.5%                         |
| gcc      | compress_literals | level_7 | enwik7      | 1.29  | 502.81           | 552.79                  | 9.9%                          |
| gcc      | compress_literals | level_7 | silesia     | 1.11  | 675.97           | 776.44                  | 14.9%                         |
| clang    | compress          | level_1 | enwik7      | 2.43  | 277.54           | 280.98                  | 1.2%                          |
| clang    | compress          | level_1 | silesia     | 2.88  | 369.98           | 375.46                  | 1.5%                          |
| clang    | compress_literals | level_1 | enwik7      | 1.49  | 828.83           | 918.41                  | 10.8%                         |
| clang    | compress_literals | level_1 | silesia     | 1.28  | 815.81           | 905.41                  | 11.0%                         |
| clang    | compress_literals | level_7 | enwik7      | 1.29  | 533.13           | 553.30                  | 3.8%                          |
| clang    | compress_literals | level_7 | silesia     | 1.11  | 714.52           | 775.38                  | 8.5%                          |
2021-07-27 15:10:35 -07:00
sen
d90bc0e0b6
Merge pull request #2720 from senhuang42/remove_folder
Remove folder when done with test
2021-06-30 14:37:21 -04:00
senhuang42
f5f6cc2e48 Remove folder when done with test 2021-06-30 13:09:00 -04:00
Binh Vo
6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
Binh Vo
9d9f7680f8 Add API for fetching skippable frame content 2021-06-14 16:01:28 -04:00
binhdvo
89127e5ee2
Merge pull request #2705 from binhdvo/bootcamp
Add support for negative values in advanced flags
2021-06-11 14:08:23 -04:00
Binh Vo
6fad35c6a1 Add support for negative levels in --adapt=min and --adapt=max" 2021-06-11 12:13:09 -04:00
W. Felix Handte
94cf57bb13 Update Tests to Reflect New Formatting 2021-06-10 13:14:18 -04:00
binhdvo
325952f878
Revert "Add support for --long-param flag, fix #2104" 2021-06-09 15:35:43 -04:00
binhdvo
78e16b15f1
Merge pull request #2703 from binhdvo/bootcamp
Add support for --long-param flag, fix #2104
2021-06-09 15:03:38 -04:00
Binh Vo
6583fa3f0a Add support for --long-param flag 2021-06-09 14:07:52 -04:00
Yann Collet
05d70903a6
Merge pull request #2698 from binhdvo/bootcamp
Fix --progress flag to properly control progress display and default …
2021-06-09 09:56:01 -07:00
Binh Vo
d2f31b6627 Fix --progress flag to properly control progress display and default progress display on when using -v 2021-06-08 17:24:38 -04:00
Sen Huang
2ff5c7b59f Add no intrinsics fuzztest, rowhash compression size test, and S390X to travis 2021-06-07 00:54:53 -07:00
senhuang42
939276cd0c Add ldm and block splitter auto-enable to old api 2021-05-24 13:09:32 -04:00
Nick Terrell
c468e1b9cb [test][regression] Update results.csv
Changing the repcode search slightly, and changing when zstd is in
ext-dict vs. prefix mode in edge cases, slightly changes the compressed
results.
2021-05-18 19:46:37 -07:00
Yann Collet
61afa154cd improve tar compatibility
This patch is supposed to improve compatibility with less featured tar variants
"when the tar program used does not support historical options (without hyphen) nor the '-z' option."

Patch proposed by Antonio Diaz Diaz
2021-05-15 23:09:42 -07:00
Nick Terrell
725c5e4e38 [fuzz] Add determinism fuzzing to simple & dictionary round trip
Compress the input twice in the `simple_round_trip` and
`dictionary_round_trip` fuzzers with exactly the same parameters, but
reusing the context. Then ensure that the compressed output is
identical.
2021-05-13 17:05:59 -07:00
sen
06718087f8
Remove deprecate flag for vcx (#2647) 2021-05-13 19:33:48 -04:00
sen
40def70387
Add source level deprecation warning disabling to certain tests/utils (#2645) 2021-05-13 14:41:21 -04:00
sen
b35c250bf3
Remove const data members in threadpooltest payload (#2639) 2021-05-12 12:56:57 -04:00
sen
d8d6e48a0a
Add threadPool unit tests to fuzzer.c (#2604) 2021-05-07 11:13:44 -04:00
sen
698f261b35
[1.5.0] Deprecate some functions (#2582)
* Add deprecated macro to zstd.h, mark certain functions as deprecated

* Remove ZSTD_compress.c dependencies on deprecated functions
2021-05-06 17:59:32 -04:00
Nick Terrell
207e33bb61
Merge pull request #2616 from terrelln/deterministic-dict
[lib] Add ZSTD_c_deterministicRefPrefix
2021-05-06 11:09:22 -07:00
Nick Terrell
fc8330b885
Merge pull request #2621 from terrelln/regression-test
[test][regression] Update results.csv
2021-05-06 10:32:07 -07:00
sen
d6be7659b0
Add seekable roundtrip fuzzer (#2617) 2021-05-06 10:08:21 -04:00
Nick Terrell
d2925de98a
Merge pull request #2615 from terrelln/stack-space
[lib] Move some ZSTD_CCtx_params off the stack
2021-05-05 19:43:39 -07:00
Nick Terrell
ce615d7fba [test][regression] Update results.csv
The LDM change in PR #2602 changed the algorithm slightly.
The compressed size is generally positive, and when it is worse,
it is only a few bytes.
2021-05-05 19:00:36 -07:00
Nick Terrell
172b4b6ac4 [lib] Add ZSTD_c_deterministicRefPrefix
This flag forces zstd to always load the prefix in ext-dict mode, even
if it happens to be contiguous, to force determinism. It also applies to
dictionaries that are re-processed.

A determinism test case is also added, which fails without
`ZSTD_c_deterministicRefPrefix` and passes with it set.

Question: Should this be the default behavior? It isn't in this PR.
2021-05-05 18:49:56 -07:00
Nick Terrell
eb7e74ccb7 [tests] Set DEBUGLEVEL=2 by default
This allows us to quickly check for compile errors in debug log
messages, which are compiled out when `DEBUGLEVEL < 2`.
2021-05-05 13:29:06 -07:00
W. Felix Handte
bea1b2ba70 rm -f in playTests.sh 2021-05-05 13:10:34 -04:00
W. Felix Handte
018ed6552a Attempt to Fix stat Format for BSDs 2021-05-05 13:10:34 -04:00
W. Felix Handte
1fb10ba831 Don't Block Removing File on Being Able to Read It
`open()`'s mode bits are only applied to files that are created by the call.
If the output file already exists, but is not readable, the `fopen()` would
fail, preventing us from removing it, which would mean that the file would
not end up with the correct permission bits.

It's not clear to me why the `fopen()` is there at all. `UTIL_isRegularFile()`
should be sufficient, AFAICT.
2021-05-05 13:10:34 -04:00
W. Felix Handte
4e10ff15f5 Add Tests Checking File Permissions of Created Files 2021-05-05 13:10:34 -04:00
Felix Handte
2d10544b84
Merge pull request #2613 from felixhandte/allow-block-device
Allow Reading from Block Devices with `--force`
2021-05-05 13:06:32 -04:00
Yann Collet
c077f257b4
Merge pull request #2611 from facebook/smallerJobs
allow jobSize to be as low as 512 KB
2021-05-05 00:03:29 -07:00
Nick Terrell
0b88c2582c [test] Add large dict/data --patch-from test
Dictionary size must be > `ZSTD_CHUNKSIZE_MAX`.
2021-05-04 17:31:32 -07:00
W. Felix Handte
e58e9c7928 Add Test Case (Behind Flag); Run in GitHub Action 2021-05-04 18:43:39 -04:00