49b2bf7106
'void* size issue' fix
2019-12-13 16:06:57 -08:00
e3cd2785e2
Add test to catch too many noCompress superblocks on streaming
2019-12-13 15:31:29 -08:00
d73e2fb465
Merge pull request #1891 from bimbashrestha/oss
...
[fuzz] Superblock fuzz issues
2019-12-10 13:17:00 -08:00
4b4de7493f
refactoring : remove global variable g_displayOut
...
displaying into stderr or stdout is now an explicit operation,
no longer depending on some global state set elsewhere in the code.
2019-12-02 16:19:03 -08:00
a49417b5af
fix recent issue combining -r with empty list of input files
...
This would resize the table of input filenames to zero,
delivering an empty table,
to which it was no longer possible to add stdin.
2019-12-02 14:28:18 -08:00
46bdbb9c06
Merge branch 'dev' into ahmed_file
2019-11-26 14:33:04 -08:00
49cacd858b
fixed fifo test
2019-11-26 14:18:09 -08:00
a3a3c62b81
[fuzz] Only set HUF_repeat_valid if loaded table has all non-zero weights ( #1898 )
...
Fixes a fuzz issue where dictionary_round_trip failed because the compressor was generating corrupt files thanks to zero weights in the table.
* Only setting loaded dict huf table to valid on non-zero
* Adding hasNoZeroWeights test to fse tables
* Forbiding nbBits != 0 when weight == 0
* Reverting the last commit
* Setting table log to 0 when weight == 0
* Small (invalid) zero weight dict test
* Small (valid) zero weight dict test
* Initializing repeatMode vars to check before zero check
* Removing FSE changes to seperate pr
* Reverting accidentally changed file
* Negating bool, using unsigned, optimization nit
2019-11-26 12:24:19 -08:00
c71bd45a3b
Merge branch 'dev' into ahmed_file
2019-11-26 11:20:26 -08:00
60db21a677
added decompression into /dev/null tests
2019-11-25 10:53:03 -08:00
7aaac3f69c
fix #1904
...
/dev/null permissions were modified when using sudo rights.
This fixes this bug during decompression.
More importantly, this patch adds a test, triggered in TravisCI,
ensuring unaltered /dev/null permissions.
2019-11-25 10:35:36 -08:00
826b555463
Merge branch 'dev' into oss
2019-11-22 17:29:33 -08:00
707a12c419
Test enough room for checksum in superblock
2019-11-22 17:25:36 -08:00
385a447e52
Merge pull request #1903 from terrelln/null
...
Fix null pointer addition
2019-11-21 17:35:06 -08:00
659e9f05cf
Fix null pointer addition
2019-11-20 18:36:04 -08:00
e68db76b4b
Update .gitignore
2019-11-20 16:36:40 -08:00
a839d6852c
Merge pull request #1888 from senhuang42/superblocks_fixed
...
RLE test and re-enable RLE in main compression loop
2019-11-18 16:09:33 -08:00
bc3e21578d
No margin on RLE test size check
2019-11-18 16:39:16 -05:00
db8efbfe7d
Updated comment to reflect actual compression behavior
2019-11-15 16:11:14 -05:00
75c34684c0
Modified existing RLE test to take compressed size into account
2019-11-15 12:26:48 -05:00
4b1ac69f19
Merge pull request #1868 from senhuang42/superblocks_fixed
...
Superblocks rebased for merge
2019-11-14 13:31:34 -08:00
d67742bc5d
Merge pull request #1858 from senhuang42/dictionary_header_size
...
Method to get dictionary header size
2019-11-14 09:44:07 -08:00
97b7f712f3
Change to heap allocation, remove implicit type conversion
2019-11-08 13:57:25 -05:00
e1edc554a3
Added 2 unit tests: one for sanity, one for correctnesson fixed dict
2019-11-08 13:57:25 -05:00
8c474f9845
Fix parameter selection and adjustment with srcSize == 0
2019-11-07 08:58:43 -08:00
aea2ff5d8d
fixed wrong assert() in regression driver
2019-11-06 14:56:21 -08:00
a7e33e3e10
updated fuzz tests to use FileNamesTable* abstraction
2019-11-06 14:42:13 -08:00
32f989c39a
Merge pull request #1869 from bket/openbsd
...
'head -c BYTES' is non-portable.
2019-11-06 16:54:03 -05:00
e488cabf25
'date -r FILE' is non-portable.
...
date(1) is used to display the last modification time of a file, which
is not supported on OpenBSD, FreeBSD and Darwin. Instead use stat(1).
Tested on OpenBSD.
2019-11-06 13:55:59 +01:00
66b6688274
'head -c BYTES' is non-portable.
...
Similar to #1321 : The '-c' option for 'head' is non-portable (not in
POSIX).
2019-11-06 11:39:59 +01:00
81c51483e9
Merge branch 'dev' into ahmed_file
2019-11-05 14:44:12 -08:00
e21a8bbecd
Fix FUZZ_rand32() bug
2019-11-05 16:43:24 -05:00
f2932fb5eb
Fix more merge conflicts
2019-11-05 15:54:05 -05:00
60205fec02
Fix 2 bugs in dictionary loading
...
* Silently skip dictionaries less than 8 bytes, unless using `ZSTD_dct_fullDict`.
This changes the compressor, which silently skips dictionaries <= 8 bytes.
* Allow repcodes that are equal to the dictionary content size, since it is in bounds.
2019-11-01 16:52:07 -07:00
75e7c0d107
[fuzz] Add dictionary_loader fuzzer
...
* Adds the fuzzer
* Adds an additional `InputType` for the fuzzer
I ran the fuzzer for about 10 minutes and it found 2 bugs:
* Catches the original bug without any help
* Catches an additional bug with 8-byte dictionaries
2019-11-01 15:54:24 -07:00
47034cd6c3
Merge pull request #1850 from bimbashrestha/named_pipes
...
Gating named pipe support on windows
2019-10-30 12:18:58 -07:00
6d0b7bd6ce
Changed extension comparision logic, added new test cases
2019-10-29 13:36:09 -07:00
9ab6a747d4
Created utility function to extract extension from filename, fixed tests
2019-10-29 12:27:54 -07:00
5e6dbad6c1
Deleting test directory before creation to pass travis-ci test
2019-10-29 09:54:54 -07:00
850ba66139
Minor fixes in test
2019-10-28 22:24:01 -07:00
da73b11b14
Merge branch 'dev' into named_pipes
2019-10-29 01:17:13 -04:00
0f2bff2faf
Addressing comments, removing cyclic dependency with header file, updating tests
2019-10-28 18:21:47 -07:00
85df7a4da8
added test which can overflow internal list of filenames
2019-10-28 16:05:42 -07:00
0e9a37daeb
Fixing tests and moving flag to advanced usage
2019-10-28 15:22:26 -07:00
170982fbd9
added symlink test for --filelist=FILE
2019-10-28 15:15:26 -07:00
f422914619
Resolving local merge conflicts
2019-10-28 15:15:13 -07:00
d9c634e13b
return final `\0` directly from readLine()
2019-10-28 15:03:32 -07:00
02433e0b15
Addressing comments: -Created a list of extensions defined in fileio.h, -Updated test
2019-10-28 14:54:54 -07:00
48f856640e
Added --exclude-compressed flag feature that skips compression of precompressed files
2019-10-28 13:57:40 -07:00
d7f258d845
change command name to `--filelist=`
...
updated documentation and man page
2019-10-28 13:54:36 -07:00
a78a8759a2
Merge branch 'dev' into ahmed_file
2019-10-28 13:43:28 -07:00
b4037b18ef
Merge pull request #1848 from felixhandte/tzst-ext-tmp
...
Support Decompressing Short Tar Extensions
2019-10-28 10:08:20 -07:00
8e414b586d
test resilience on garbage file
...
and clean traces
2019-10-25 17:01:26 -07:00
1ead0c5d5a
improved --file=FILE implementation
...
pass basic tests
2019-10-25 16:36:59 -07:00
55ee7d56e4
Added --exclude-compressed flag feature that skips compression of precompressed files
2019-10-25 15:49:11 -07:00
5e59d51991
Merge branch 'master' of https://github.com/bimbashrestha/zstd into named_pipes
2019-10-25 15:43:41 -07:00
5b45ff4f1c
Gating named file support on windows
2019-10-25 11:32:38 -07:00
4eccc82875
Minor Fixes
2019-10-25 13:59:19 -04:00
14e9010bb5
Merge pull request #1844 from AhmedAbdellah19/adding_read_files_from_file_feature
...
Adding --file=FILE feature
2019-10-25 10:11:47 -07:00
c2140e9db0
Merge pull request #1845 from facebook/zbuff
...
improve deprecation warning macro
2019-10-25 09:59:00 -07:00
1faeb222b2
adding some functional tests
2019-10-25 15:54:52 +01:00
24499036ba
Add Tests
2019-10-24 20:19:10 -04:00
37f90930da
Merge pull request #1839 from bimbashrestha/named_pipes
...
Allowing named pipes to go through in zstdcli
2019-10-24 10:53:12 -07:00
63e435dda1
improve deprecation warning macro
...
fix #1488
although, curiously enough, I was never able to reproduce the issue
(according to the bug report, it should be present while using gcc 4.8).
2019-10-23 11:59:32 -07:00
3c8ff35013
Merge pull request #1840 from terrelln/regression-fix
...
[regression] Fix the old streaming regression test
2019-10-22 17:58:31 -07:00
10f7147013
Adding shell test and removing bash test file
2019-10-22 17:09:44 -07:00
09d0b52998
Merge pull request #1838 from facebook/fixAdvanced
...
fix initCStream_advanced() for fast strategies
2019-10-22 16:22:55 -07:00
edae3c8f3c
Adding bash test for named pipes
2019-10-22 16:15:59 -07:00
cf21003995
[regression] Fix the old streaming regression test
...
* A copy-paste error made it so we weren't running the advanced/cdict
streaming tests with the old API.
* Clean up the old streaming tests to skip incompatible configs.
* Update `results.csv`.
The tests now catch the bug in #1787 .
2019-10-22 15:43:17 -07:00
514f941a70
Merge pull request #1836 from facebook/dirFlat2
...
Added --output-dir-flat=
2019-10-22 15:19:04 -07:00
5d5c895b18
fix initCStream_advanced() for fast strategies
...
Compression ratio of fast strategies (levels 1 & 2)
was seriously reduced, due to accidental disabling of Literals compression.
Credit to @QrczakMK, which perfectly described the issue, and implementation details,
making the fix straightforward.
Example : initCStream with level 1 on synthetic sample P50 :
Before : 5,273,976 bytes
After : 3,154,678 bytes
ZSTD_compress (for comparison) : 3,154,550
Fix #1787 .
To follow : refactor the test which was supposed to catch this issue (and failed)
2019-10-22 15:01:38 -07:00
b1ec94e63c
Fix ZSTD_f_zstd1_magicless for small data
...
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.
Fixes #1813 .
2019-10-21 21:16:17 -07:00
919d1d8e93
Merge pull request #1831 from terrelln/zstdmt-bad-memset
...
[zstdmt] Don't memset the jobDescription
2019-10-21 15:53:57 -07:00
989e0f0cee
Added --output-dir-flat=
...
which is a preferred construction when applicable.
Also, added --output-dir-flat into man page.
2019-10-21 14:39:18 -07:00
6cf04c0344
Merge pull request #1834 from facebook/winFix
...
Windows fixes
2019-10-21 13:45:17 -07:00
2ad75bb574
validated 'make test' for mingw environment
2019-10-18 15:37:07 -07:00
85a016ed92
made `make check` faster
...
for shorter feedback loop
2019-10-18 15:21:50 -07:00
243824551f
[threading] Add debug utilities
2019-10-18 15:05:34 -07:00
29e46ed0bd
fix test on windows
...
isDirectory() doesn't work on Windows
if directory name is followed by '/'
2019-10-18 14:28:34 -07:00
8c11f089a1
[fuzz] Increase output buffer size of stream_round_trip
...
Fixes OSS-Fuzz crash.
Credit to OSS-Fuzz
2019-10-18 13:39:08 -07:00
ad86a5d0bc
rewrite FIO_createFilename_fromOutDir()
2019-10-18 11:15:10 -07:00
ba7e2b6da7
tests: can override isTerminal with environment variable
2019-10-17 15:07:47 -07:00
6323966e53
updated erroneous comments using ZSTD_dm_*
...
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822 )
2019-10-16 16:14:04 -07:00
b3e92ad346
Merge pull request #1812 from senhuang42/output-dir-flat
...
Support for --output-dir-flat: output into a directory
2019-10-14 09:25:05 -07:00
fb77afc626
Merge pull request #1760 from bimbashrestha/extract_sequences_api
...
Adding api for extracting sequences from seqstore
2019-10-10 13:11:18 -07:00
36528b96c4
Manually moving instead of memcpy on decoder and using genBuffer()
2019-10-03 09:26:51 -07:00
f80437c586
Add support for --output-dir-flat
...
New flag to specify output directory destination for multiple files.
2019-10-02 11:08:20 -04:00
b63a1e7ae5
Typo fix
2019-09-27 07:20:20 -07:00
91daee5c06
Fixing appveyor test
2019-09-26 16:21:57 -07:00
75b1286354
Fixing shortest failure
2019-09-26 16:07:34 -07:00
bb27472afc
Adding more realistic test for get sequences
2019-09-26 15:38:31 -07:00
be0bebd24e
Adding test and null check for malloc
2019-09-23 15:08:18 -07:00
7451c6578c
Merge pull request #1804 from terrelln/wild-and-fast
...
Optimize (de)compression and fix wildcopy overread
2019-09-21 17:04:36 -07:00
c047fcf7bf
Merge pull request #1806 from felixhandte/estimate-cctx-doc
...
Update Comment on `ZSTD_estimateCCtxSize()`
2019-09-20 15:36:00 -04:00
f7d9b36835
Update Comment on `ZSTD_estimateCCtxSize()`
2019-09-20 14:11:29 -04:00
e068bd01df
[tests] Fix decodecorpus
2019-09-20 01:09:47 -07:00
51990246c3
[test] Test the bounds of ZSTD_c_srcSizeHint
2019-09-18 11:05:08 -07:00
3cacc0a30b
Casting void pointer to ZSTD_Sequence pointer
2019-09-17 17:44:08 -07:00
5b038f128f
Merge branch 'extract_sequences_api' of https://github.com/bimbashrestha/zstd into extract_sequences_api
2019-09-16 13:35:49 -07:00
1f93be0f6d
Handling memory leak and potential side effect
2019-09-16 13:35:45 -07:00
a874435478
Merge branch 'dev' into extract_sequences_api
2019-09-16 13:29:59 -07:00
2164a130f3
Merge pull request #1780 from felixhandte/workspace-efficiency-3
...
Avoid Clearing Tables Even When Changing CParams
2019-09-16 14:37:05 -04:00
d721fcf3ee
[fuzz] Fix leak in block_round_trip
2019-09-13 10:32:38 -07:00
7c4578160e
[fuzz] Generate seed data up to 256KB
2019-09-12 15:02:01 -07:00
51e9d29a51
Merge branch 'improvDataGen' of github.com:darxsys/zstd into improvDataGen
2019-09-12 13:11:02 -07:00
cd8588077e
It's time for all of rng seed code to go. Goodbye
2019-09-12 13:10:34 -07:00
47bb4c6a23
Update tests/fuzz/fuzz_data_producer.h
2019-09-12 12:45:28 -07:00
92c58c4d5d
Use range instead of the generic uint32 method to use less bytes when generating necessary numbers.
2019-09-12 12:40:12 -07:00
194c542598
Fix Memory Leak in Test
2019-09-11 14:25:30 -04:00
ff67c62458
Fix Compilation Error (`uint32_t` -> `size_t`)
2019-09-11 13:59:09 -04:00
5707c8a9d5
Speed Up Test a Little
2019-09-11 13:23:59 -04:00
ed4c2c60c3
Add Fuzzer Test Case for Index Reduction
2019-09-11 13:17:19 -04:00
b5b24c2a0d
Combining fuzz_data_producer restrict calls into a single function
2019-09-11 10:09:29 -07:00
23cc2d8510
All tests should give some portion of data to the producer and use the rest.
2019-09-10 16:52:38 -07:00
4dfc1bbf48
Merge branch 'dev' into improvDataGen
2019-09-10 16:20:27 -07:00
0630d084cb
[Fuzz] Improve data generation #1723
...
Converting the rest of the tests to use the new data producer.
2019-09-10 16:14:43 -07:00
458b602bb9
Merge pull request #1777 from darxsys/improvDataGen
...
[Fuzz] Improve data generation #1723
2019-09-10 11:51:23 -07:00
41416f0927
Merge pull request #1773 from bimbashrestha/rle_first_block_decompression_fix
...
Removing redundant condition in decompression, making first block rle…
2019-09-10 11:17:29 -07:00
caaf43b258
Using a number instead of M prefix in head call
2019-09-10 09:30:37 -07:00
9e7bb55e14
Addressing comments
2019-09-09 20:04:46 -07:00
ea1ad123da
Addressing nits
2019-09-09 16:13:24 -07:00
3932fcfebc
Fixing issues with double usage of data.
2019-09-09 15:39:04 -07:00
e6be4cf4eb
Changing test file directory names to be more descriptive
2019-09-09 12:08:33 -07:00
0b25ab2202
Testing first block decompression cli
2019-09-09 11:54:43 -07:00
a71bbba7be
[Fuzz] Improve data generation #1723
2019-09-09 08:43:22 -07:00
5f8b0f6890
Changing api to get sequences across all blocks
2019-08-30 09:18:44 -07:00
8ce9c1917e
[regression] small improvement from RLE blocks
2019-08-29 18:05:58 -07:00
5198347382
Merge pull request #1744 from bimbashrestha/dev
...
Generate RLE blocks in the encoder
2019-08-29 15:19:10 -07:00
e5704bbfdf
Added test for multiple blocks of zeros and fixed nit about comments
2019-08-28 08:32:34 -07:00
793c71d6ae
[test][dictBuilder] Add multithreaded tests
2019-08-26 18:32:08 -07:00
d0750a1c9c
Merge pull request #1733 from nmagerko/size-hint
...
Add --size-hint=# option
2019-08-23 10:16:10 -07:00
e2030a2c40
[fuzz] Add a DEBUGLOG(3) statement to print file
...
Enable it by building with this command:
```
./fuzz.py build all --debug 3
```
2019-08-22 17:27:15 -07:00
493f95c7df
Fix merge conflicts
2019-08-22 11:51:41 -07:00
a505463710
Merge pull request #1726 from nmagerko/stream-size
...
Add --stream-size=# option
2019-08-22 11:31:15 -07:00
b3540507f5
Remove bc from play tests
2019-08-21 10:27:54 -07:00
188311dd4a
Merge pull request #1736 from terrelln/fuzz-fix
...
[fuzz] Improve fuzzer build script and docs
2019-08-21 10:09:38 -07:00
3982935aef
[fuzz] Improve fuzzer build script and docs
...
* Remove the `make libFuzzer` target since it is broken and obsoleted
by `CC=clang CXX=clang++ ./fuzz.py build all --enable-fuzzer`. The
new `-fsanitize=fuzzer` is much better because it works with MSAN
by default.
* Improve the `./fuzz.py gen` command by making the input type explicit
when creating a new target.
* Update the `README` for `--enable-fuzzer`.
Fixes #1727 .
2019-08-20 16:44:50 -07:00
c7a24d7a14
Define ZSTD_SRCSIZEHINT_MIN as 0
2019-08-20 13:06:15 -07:00
f23402f1f5
Remove unnecessary test case
2019-08-19 17:20:46 -07:00
294f1e5cfe
Fix typo in test
2019-08-19 16:53:02 -07:00
f9af70ca8a
Fix playTests and add additional cases
2019-08-19 16:48:35 -07:00
ea9d35922c
Add size-hint to fuzz tests
2019-08-19 15:12:29 -07:00
dffbac5f89
Add --size-hint=# option
2019-08-19 11:38:49 -07:00
85d07c6c47
Tweak stdout, stderr redirection in new playTests
2019-08-16 12:49:21 -07:00
af0c9501d1
Add --stream-size=# command
2019-08-16 11:34:39 -07:00
b81d7cc6a0
remove extraneous doubled ;s
2019-08-15 21:17:06 -04:00
0b0b83e8f3
fix test 122
...
it's an unsupported scenario.
2019-08-03 16:51:26 +02:00
efe8496755
minor test refactoring
...
just for clarity, for the currently failing unit test
2019-08-02 19:31:19 +02:00
387e20d4f0
fixed minor conversion warning in datagen
2019-08-02 18:02:54 +02:00
37f47e51a8
fixed datagen
...
to produce same content on both 32 and 64-bit platforms
by removing floating from literal table determination.
also : added checksum trace in compression control test,
so that it's easier to determine if test fails
as a consequence of compressing a different sample.
2019-08-02 17:34:53 +02:00
d1927f0b39
regenerate sample to compress
...
to reduce chances of differences between 32 and 64-bit fuzzer tests
2019-08-02 15:31:00 +02:00
5cf1b24aca
fixed strategies greedy, lazy & lazy2
...
restore dictionary compression ratio
2019-08-02 14:21:39 +02:00
2115292616
minor : fixed ptr arithmetic
...
invalid on void ptr
2019-08-01 17:12:26 +02:00
810a9cac08
added efficiency test
...
to detect gross CR variations after a patch.
Tests normal and dictionary compression.
2019-08-01 16:59:22 +02:00
98692c2838
fixed compression ratio regression when dictionary-compressing medium-size inputs at levels 1-3
2019-08-01 15:58:17 +02:00
f2620697c7
Merge pull request #1685 from vivekmig/dev
...
Add Check if Block Size Exceeds Maximum
2019-07-19 15:22:29 -07:00
a3ce0c9d04
Fixing decodecorpus test issue
2019-07-18 14:32:09 -07:00
4c2943df23
[regression] Update results for ZSTD_double_fast update
2019-07-15 11:25:22 -07:00
75cfe1dc69
[ldm] Fix bug in overflow correction with large job size ( #1678 )
...
* [ldm] Fix bug in overflow correction with large job size
* [zstdmt] Respect ZSTDMT_JOBSIZE_MAX (1G in 64-bit mode)
* [test] Add test that exposes the bug
Sadly the test fails on our CI because it uses too much memory, so
I had to comment it out.
2019-07-12 18:45:18 -04:00
654cb9d439
fix gitignore errors
2019-07-09 21:08:13 +08:00
e962f07d19
[fuzz] Add a compression fuzzer with randomly sized output buffer ( #1670 )
2019-07-02 22:05:07 -07:00
4d611ca405
Merge pull request #1664 from ephiepark/dev
...
decodecorpus
2019-07-01 14:13:49 -07:00
28309520c0
reflect code review comments
2019-07-01 10:17:30 -07:00
01e8384000
Add test case for short bistream
2019-06-27 17:37:39 -07:00
c55d2e7ba3
Adding shrinking flag for cover and fastcover ( #1656 )
...
* Changed ERROR(GENERIC) excluding inits
* editing git ignore
* Edited init functions to size_t returns
* moved declarations earlier
* resolved issues with changes to init functions
* fixed style and an error check
* attempting to add tests that might trigger changes
* added && die to cases expecting to fail
* resolved no die on expected failed command
* fixed accel to be incorrect value
* Adding an automated shrinking option
* Fixing build
* finalizing fixes
* fix?
* Removing added comment in cover.h
* Styling fixes
* Merging with fb dev
* removing megic number for default regression
* Requested revisions
* fixing support for fast cover
* fixing casting errors
* parenthesis fix
* fixing some build nits
* resolving travis ci syntax
* might resolve all compilation issues
* removed unused variable
* remodeling the selectDict function
* fixing bad memory access
* fixing error checks
* fixed erroring check in selectDict
* fixing mixed declarations
* modify mixed declaration
* fixing nits and adding test cases
* Adding requested changes + fixed bug for error checking
* switched double comparison from != to <
* fixed declaration typing
* refactoring COVER_best_finish() and changing shrinkDict
* removing the const's
* modifying ZDICT_optimizeTrainFromBuffer_cover functions
* fixing potential bad memcpy
* fixing the error function for dict size
2019-06-27 16:26:57 -07:00
734eff70b8
enable repeat mode on rle
2019-06-26 16:39:00 -07:00
8e16b872e5
[tests] Add tests for big dictionaries
2019-06-21 17:58:24 -07:00
06d5e5ff5a
[regression] Update results for small wlog patch PR#1624
2019-06-17 10:40:33 -07:00
1166782eb7
Merge pull request #1640 from felixhandte/lstat-macro-guard
...
Protect lstat() With Better Macro Guard
2019-06-14 17:30:03 -07:00
344f942573
Merge pull request #1620 from michaelforney/test-no-threads
...
Skip --adapt and --rsyncable tests when built without thread support
2019-06-11 15:05:25 -04:00
81276c4ee4
Merge pull request #1646 from terrelln/oss-fuzz
...
[fuzz] Remove max_len from the options
2019-06-10 22:13:16 -07:00
6810dd6191
[fuzz] Remove max_len from the options
2019-06-10 11:05:45 -07:00
b878bd08dd
Clean Up Temp Files Produced By playTests.sh
2019-06-07 15:32:28 -04:00
f3800ba073
[regression] Update results.csv
2019-06-05 20:40:52 -07:00
78056a3e9c
consistency : changed constant name to kSampleSizeDefault
...
as suggested by @terrelln.
added <assert.h> include
2019-05-30 10:13:56 -07:00
ab3625cffd
define CONTROL, to ensure check cannot be disabled
...
assert() can be disabled with NDEBUG
2019-05-29 16:18:22 -07:00
904d4da239
fullbench : minor refactoring, for readability
2019-05-29 16:08:49 -07:00
ed38b645db
fullbench: pass proper parameters in scenario 43
2019-05-29 15:26:06 -07:00
c63081623f
fullbench 43: ensure context is freed after each usage
2019-05-28 17:27:52 -07:00
eb6b199038
fullbench: added streaming with fresh CCtx scenario
...
worst case situation, where context must be recreated every time
but without knowledge of the input size
thus sizing the context for some large input.
2019-05-28 16:50:49 -07:00
4c4149452c
fullbench can now select sample size with command `-B#`
2019-05-28 16:37:03 -07:00
4baecdf72a
added comments to better understand enforceMaxDist()
2019-05-28 13:15:48 -07:00
1bc77ee174
Skip --adapt and --rsyncable tests when built without thread support
...
These options require ZSTD_MULTITHREAD.
2019-05-26 03:20:37 -07:00
6a0638048a
Add Test
2019-05-24 17:21:44 -04:00
cb47871a0a
[dictBuilder] Be more specific than ERROR(generic) ( #1616 )
...
* Specify errors at a finer granularity than `ERROR(generic)`.
* Add tests for bad parameters in the dictionary builder.
2019-05-22 18:57:50 -07:00
bfbef51afb
Merge pull request #1610 from terrelln/pool-test
...
[test] Fix poolTests deadlock and flakiness
2019-05-02 15:13:50 -07:00
81f1f76479
[tests] Only sleep 10ms in poolTests
2019-05-02 14:35:25 -07:00
2953d45a0f
[test] Fix poolTests deadlock and flakiness
2019-05-02 13:23:41 -07:00
1d70bc3ba9
meson: Enable testing in CI ( #1609 )
...
* tests: Fix shellcheck warnings in playTests.sh
* tests: Do not use ../programs which is relative to tests dirs
This commit fixes error when running playTests.sh in Meson.
Mesonbuild runs out of tree, so ./datagen not in `zstd/tests` dir,
it lies in <mesonbuilddir>/tests. This leads to ../programs invalid.
* tests: Replace relative paths for zstd/tests dir
* playTests: Set shell options explicitly, not in shebang
* playTests: Replace echo -e with printf
* meson: Fix test-zstd
Use std=gnu99 to build and test just like `make test`.
* meson: Fix legacy test
* meson: Enable testing in CI
Run build under release mode for faster test time.
* meson: Increase timeout time for test-zstream
2019-05-02 12:35:37 -07:00
3e1e49d775
add test for zstdcat and zcat on symlink
2019-04-29 18:47:47 -04:00
785331acbd
[paramgrill] Fix mingw build errors
2019-04-18 15:06:56 -07:00
610a81ecf9
[fuzzer] Compile with legacy support
2019-04-18 12:44:55 -07:00
cc669006dc
[fuzzer] Size the decompression output buffer randomly
2019-04-18 12:44:21 -07:00
58bcc328a4
[fuzz] Add a seedcorpora target for oss-fuzz
2019-04-17 12:13:06 -07:00
09caa4d800
[fuzzer] Add a fuzzer for frame info functions
...
Add a fuzzer that fuzzes all helper functions that take compressed
input. This fuzzer caught one out of bounds read in
`ZSTD_decompressBound()`.
2019-04-17 11:29:42 -07:00
a48e32ca50
[fullbench] Fix measurements
2019-04-15 21:01:48 -07:00
6996804d05
[playTets] Don't pass status=none to dd
...
The OS X test machine on TravisCI doesn't recognize the parameter
https://travis-ci.org/facebook/zstd/jobs/520517933 .
2019-04-15 16:21:46 -07:00
a880ca239b
Spelling ( #1582 )
...
* spelling: accidentally
* spelling: across
* spelling: additionally
* spelling: addresses
* spelling: appropriate
* spelling: assumed
* spelling: available
* spelling: builder
* spelling: capacity
* spelling: compiler
* spelling: compressibility
* spelling: compressor
* spelling: compression
* spelling: contract
* spelling: convenience
* spelling: decompress
* spelling: description
* spelling: deflate
* spelling: deterministically
* spelling: dictionary
* spelling: display
* spelling: eliminate
* spelling: preemptively
* spelling: exclude
* spelling: failure
* spelling: independence
* spelling: independent
* spelling: intentionally
* spelling: matching
* spelling: maximum
* spelling: meaning
* spelling: mishandled
* spelling: memory
* spelling: occasionally
* spelling: occurrence
* spelling: official
* spelling: offsets
* spelling: original
* spelling: output
* spelling: overflow
* spelling: overridden
* spelling: parameter
* spelling: performance
* spelling: probability
* spelling: receives
* spelling: redundant
* spelling: recompression
* spelling: resources
* spelling: sanity
* spelling: segment
* spelling: series
* spelling: specified
* spelling: specify
* spelling: subtracted
* spelling: successful
* spelling: return
* spelling: translation
* spelling: update
* spelling: unrelated
* spelling: useless
* spelling: variables
* spelling: variety
* spelling: verbatim
* spelling: verification
* spelling: visited
* spelling: warming
* spelling: workers
* spelling: with
2019-04-12 11:18:11 -07:00
fb6901b2c0
tests: Add missing header timefn.h
2019-04-12 21:28:17 +07:00
8ac2831f3d
Merge pull request #1581 from facebook/benchfn
...
benchfn's reduced dependencies
2019-04-11 14:23:04 -07:00
058da605cb
fixed minor conversion warning
2019-04-11 12:25:27 -07:00
f8e9bec73a
fixed poolTests on Windows
...
must use ZSTD_ prefix in front of pthread types
so that they get properly translated for Windows.
2019-04-11 12:03:42 -07:00
8f56fa2f58
fixed poolTests
...
added poolTests to all
2019-04-11 09:50:39 -07:00
9703a59121
fixed minor conversion warning
2019-04-10 15:54:55 -07:00
50b9c41196
[libzstd] Fix decompression dictionary bugs and clean up initialization
...
Bugs:
* `ZSTD_DCtx_refPrefix()` didn't clear the dictionary after the first
use. Fix and add a test case.
* `ZSTD_DCtx_reset()` always cleared the dictionary. Fix and add a test
case.
* After calling `ZSTD_resetDStream()` you could no longer load a
dictionary, since the stage was set to `zdss_loadHeader`. Fix and add
a test case.
Cleanup:
* Make `ZSTD_initDStream*()` and `ZSTD_resetDStream()` wrap the new
advanced API, and add test cases.
* Document the equivalent of these functions in the advanced API and
document the unstable functions as deprecated.
2019-04-10 12:59:02 -07:00
59a7116cc2
benchfn dependencies reduced to only timefn
...
benchfn used to rely on mem.h, and util,
which in turn relied on platform.h.
Using benchfn outside of zstd required to bring all these dependencies.
Now, dependency is reduced to timefn only.
This required to create a separate timefn from util,
and rewrite benchfn and timefn to no longer need mem.h.
Separating timefn from util has a wide effect accross the code base,
as usage of time functions is widespread.
A lot of build scripts had to be updated to also include timefn.
2019-04-10 12:37:03 -07:00
094c000904
Merge branch 'dev' into benchfn
2019-04-10 11:57:05 -07:00
c45dec12c5
[fuzzer] Use ZSTD_DCtx_loadDictionary_advanced() half the time
2019-04-09 18:02:22 -07:00
10a3d4dca9
[fuzzer] Make the regression_driver work while fuzzers are active
2019-04-09 18:01:49 -07:00
824aaa695f
[libzstd] Fix ZSTD_decompressDCtx() with a dictionary
...
* `ZSTD_decompressDCtx()` did not use the dictionary loaded by
`ZSTD_DCtx_loadDictionary()`.
* Add a unit test.
* A stacked diff uses `ZSTD_decompressDCtx()` in the
`dictionary_round_trip` and `dictionary_decompress` fuzzers.
2019-04-09 17:59:27 -07:00
c5d70b7dbb
[fuzzer] Sometimes fuzz with one less output byte
...
Zstd compression sometimes does different stuff when it has at least
`ZSTD_compressBound()` output bytes, or not. Half of the time fuzz with
`ZSTD_compressBound() - 1` output bytes. Ensure that we have at least
one byte of overhead by disabling either the dictionary ID or checksum.
2019-04-09 16:47:59 -07:00
48a6427d22
[libzstd] Fix ZSTD_compress2() for multithreaded compression
...
`ZSTD_compress2()` wouldn't wait for multithreaded compression to
finish. We didn't find this because ZSTDMT will block when it can
compress all in one go, but it can't do that if it doesn't have enough
output space, or if `ZSTD_c_rsyncable` is enabled.
Since we will already sometimes block when using `ZSTD_e_end`, I've
changed `ZSTD_e_end` and `ZSTD_e_flush` to guarantee maximum forward
progress. This simplifies the API, and helps users avoid the easy bug
that was made in `ZSTD_compress2()`
* Found by the libfuzzer fuzzers.
* Added a test case that catches the problem.
* I will make the fuzzers sometimes allocate less than
`ZSTD_compressBound()` output space.
2019-04-09 16:24:17 -07:00
7a1fde2957
[fuzzer] Add dictionary fuzzers
2019-04-08 21:07:28 -07:00
462918560c
[fuzzer] Fix stream_round_trip for the new options
2019-04-08 21:06:19 -07:00
f871b5144e
[fuzz] Use the new advanced API
2019-04-08 20:01:38 -07:00
50c634b86e
[regression] Update results.csv for level 1 change
2019-04-05 10:46:22 -07:00
56682a7709
Fix ZSTD_estimateCStreamSize_usingCCtxParams()
...
It wasn't using the ZSTD_CCtx_params correctly. It must actualize
the compression parameters by calling ZSTD_getCParamsFromCCtxParams()
to get the real window log.
Tested by updating the streaming memory usage example in the next
commit. The CHECK() failed before this patch, and passes after.
I also added a unit test to zstreamtest.c that failed before this
patch, and passes after.
2019-04-01 18:02:52 -07:00
44f77b5c71
Add whitespace to test case
2019-03-24 03:42:11 -07:00
b25d7eacf2
Rename test
2019-03-24 03:40:03 -07:00
2b4491d81a
Add CLI test to validate error
2019-03-24 00:47:13 -07:00
6b053b9f60
[lib] Allow ZSTD_CCtx_loadDictionary() to be called before parameters are set
...
* After loading a dictionary only create the cdict once we've started the
compression job. This allows the user to pass the dictionary before they
set other settings, and is in line with the rest of the API.
* Add tests that mix the 3 dictionary loading APIs.
* Add extra tests for `ZSTD_CCtx_loadDictionary()`.
* The first 2 tests added fail before this patch.
* Run the regression test suite.
2019-03-21 16:13:53 -07:00
e55da9e963
Wrap the new advanced api completely
2019-03-21 10:54:40 -07:00
11e73576bb
[regression] Add more streaming tests
...
* Test all of the `ZSTD_initCStream*()` variants.
* Fix a typo in the zstdcli method.
2019-03-21 10:54:18 -07:00
19b75b6ecb
Test new ZSTD_findFrameCompressedSize and update documentation
2019-03-15 18:04:19 -07:00
60796e76b0
Add legacy support to decompressBound
2019-03-15 16:10:37 -07:00
f52a7d8faa
Merge pull request #1547 from shakeelrao/fix-error
...
Fix incorrect error code in ZSTD_errorFrameSizeInfo
2019-03-15 10:57:49 -07:00
4c0540da1c
Add static linking to legacy tests
2019-03-15 05:13:55 -07:00
91ffc8d256
Add test to validate patch
2019-03-15 03:59:03 -07:00
18fbcddd0c
[zstreamtest] Remove outdated test
2019-03-13 17:01:23 -07:00
787b76904a
[libzstd] Allow compression parameters to be set with a cdict
...
The order you set parameters in the advanced API is not supposed to matter.
However, once you call `ZSTD_CCtx_refCDict()` the compression parameters
cannot be changed. Remove that restriction, and document what parameters
are used when using a CDict.
If the CCtx is in dictionary mode, then the CDict's parameters are used.
If the CCtx is not in dictionary mode, then its requested parameters are
used.
2019-03-13 16:10:05 -07:00
18d3a97d43
Add unit test to validate the error case
2019-03-13 01:43:40 -07:00
13cc7c610e
Merge pull request #1543 from shakeelrao/dev
...
Provide a function to estimate decompressed size.
2019-03-04 08:00:47 -08:00
95dfd48143
update formatting
2019-03-01 23:11:15 -08:00
3da3dc2f45
add missing size content test
2019-03-01 21:27:30 -08:00
03026c3b1d
change compressedBound to ULL
2019-03-01 00:03:50 -08:00
8930c3c79b
implement API-level changes
2019-02-28 22:55:18 -08:00
d0a3f25697
change return type to ULL
2019-02-28 01:52:01 -08:00
c9d674b60d
Remove autogenerated test file
2019-02-28 01:29:04 -08:00
97d3d28dab
Fix decl-after-stmnt build error
2019-02-28 01:24:54 -08:00
820af1e078
Provide an API function to estimate decompressed size.
...
Introduces a new utility function `ZSTD_findFrameCompressedSize_internal` which
is equivalent to `ZSTD_findFrameCompressSize`, but accepts an additional output
parameter `bound` that computes an upper-bound for the compressed data in the frame.
The new API function is named `ZSTD_decompressBound` to be consistent with
`zstd_compressBound` (the inverse operation). Clients will now be able to compute an upper-bound for
their compressed payloads instead of guessing a large size.
Implements https://github.com/facebook/zstd/issues/1536 .
2019-02-28 00:42:49 -08:00
4b0024a97d
[fuzz] Add --enable-fuzzer for clang fuzzing
2019-02-27 17:15:52 -08:00
7ad7ba3178
[libzstd] Rename ZSTD_CCtxParam_* to ZSTD_CCtxParams_*
2019-02-19 17:44:52 -08:00
6efce7c9ca
[fuzzer] Add test cases
2019-02-19 13:22:44 -08:00
f4abba02ba
[libzstd] Clean up parameter code
...
* Move all ZSTDMT parameter setting code to ZSTD_CCtxParams_*Parameter().
ZSTDMT now calls these functions, so we can keep all the logic in the
same place.
* Clean up `ZSTD_CCtx_setParameter()` to only add extra checks where needed.
* Clean up `ZSTDMT_initJobCCtxParams()` by copying all parameters by default,
and then zeroing the ones that need to be zeroed. We've missed adding several
parameters here, and it makes more sense to only have to update it if you
change something in ZSTDMT.
* Add `ZSTDMT_cParam_clampBounds()` to clamp a parameter into its valid
range. Use this to keep backwards compatibility when setting ZSTDMT parameters,
which clamp into the valid range.
2019-02-19 13:22:37 -08:00
19ca3fbc03
[zstdcli] Respect --[no-]compress-literals in benchmark mode
2019-02-15 16:27:39 -08:00
0c53c5ad4a
[zstdcli] Add a flag to control literals compression
2019-02-15 15:00:45 -08:00
5261a288d1
Fix a bug in the compress cctx method
2019-02-15 14:58:19 -08:00
3d7377b874
[libzstd] Handle uncompressed literals
2019-02-15 14:58:11 -08:00
a96e67af6c
[regression] Test level 19 with uncompressed literals
2019-02-15 10:52:46 -08:00
dbaa7042d9
[regression] Test ZSTD_c_literalCompressionMode
...
Test a positive compression level with uncompressed literals,
and a negative compression level with compressed literals.
I double checked the `results.csv` and made sure that the compressed
sizes make sense.
2019-02-13 15:00:32 -08:00
Bimba Shrestha
Yann Collet
Yann Collet
Nick Terrell
Nick Terrell
Sen Huang
Felix Handte
Björn Ketelaars
Shashank Tavildar
Ahmed Abdellah
Dario Pavlovic
Dario Pavlovic
Bimba Shrestha
Nick Magerko
Ed Maste
Vivek Miglani
LeeYoung624
Ephraim Park
Tyler-Tran
Michael Forney
lzutao
cev
Josh Soref
shakeelrao