Commit Graph

1489 Commits (f69d8c027d0eb420d3c7129eecde8f5d1af027fe)

Author SHA1 Message Date
Nick Terrell 5717bd39ee [lib] Fix NULL pointer dereference
When the output buffer is `NULL` with size 0, but the frame content size
is non-zero, we will write to the NULL pointer because our bounds check
underflowed.

This was exposed by a recent PR that allowed an empty frame into the
single-pass shortcut in streaming mode.

* Fix the bug.
* Fix another NULL dereference in zstd-v1.
* Overflow checks in 32-bit mode.
* Add a dedicated test.
* Expose the bug in the dedicated simple_decompress fuzzer.
* Switch all mallocs in fuzzers to return NULL for size=0.
* Fix a new timeout in a fuzzer.

Neither clang nor gcc show a decompression speed regression on x86-64.
On x86-32 clang is slightly positive and gcc loses 2.5% of speed.

Credit to OSS-Fuzz.
2020-05-06 12:09:02 -07:00
Bimba Shrestha e7df0d41bb
Merge pull request #2095 from bimbashrestha/grep
[bugs] zstdgrep/grep inconsistencies
2020-05-06 11:18:15 -05:00
Bimba Shrestha 250184adf6 adding tests back 2020-05-05 16:51:06 -07:00
Felix Handte 7e9aabd652
Merge pull request #2099 from felixhandte/compile-under-pedantic
Compile Under `-pedantic -Werror` and `-std=c90`
2020-05-04 10:07:13 -07:00
Felix Handte 816ed80774
Merge pull request #1984 from MeghnaM/1636-Reduce-stack-usage-of-HUF_sort
Reduce stack usage of HUF_sort()
2020-05-04 08:15:31 -07:00
W. Felix Handte 2cf72d56a6 Try to Fix MSVC Error
It's complaining about the `memcpy`s, saying:

"warning C4090: 'function': different 'const' qualifiers"

Let's try explicitly casting to the argument types...
2020-05-04 10:59:15 -04:00
W. Felix Handte dacbcd2cc1 Fix Up Some Pointer Handling in Tests 2020-05-04 10:59:15 -04:00
Nick Terrell e103d7b4a6
Fix superblock mode (#2100)
Fixes:

Enable RLE blocks for superblock mode
Fix the limitation that the literals block must shrink. Instead, when we're within 200 bytes of the next header byte size, we will just use the next one up. That way we should (almost?) always have space for the table.
Remove the limitation that the first sub-block MUST have compressed literals and be compressed. Now one sub-block MUST be compressed (otherwise we fall back to raw block which is okay, since that is streamable). If no block has compressed literals that is okay, we will fix up the next Huffman table.
Handle the case where the last sub-block is uncompressed (maybe it is very small). Before it would skip superblock in this case, now we allow the last sub-block to be uncompressed. To do this we need to regenerate the correct repcodes.
Respect disableLiteralsCompression in superblock mode
Fix superblock mode to handle a block consisting of only compressed literals
Fix a off by 1 error in superblock mode that disabled it whenever there were last literals
Fix superblock mode with long literals/matches (> 0xFFFF)
Allow superblock mode to repeat Huffman tables
Respect ZSTD_minGain().
Tests:

Simple check for the condition in #2096.
When the simple_round_trip fuzzer enables superblock mode, it checks that the compressed size isn't expanded too much.
Remaining limitations:

O(targetCBlockSize^2) because we recompute statistics every sequence
Unable to split literals of length > targetCBlockSize into multiple sequences
Refuses to generate sub-blocks that don't shrink the compressed data, so we could end up with large sub-blocks. We should emit those sections as uncompressed blocks instead.
...
Fixes #2096
2020-05-01 16:11:47 -07:00
Meghna Malhotra cc7c29595d Fixed tests to use correct workspace size 2020-05-01 13:45:48 -07:00
Yann Collet da2748a855
Merge pull request #2097 from facebook/underlink
Fix underlinked libzstd
2020-04-30 10:16:24 -07:00
Yann Collet 7ea2ae6649 added test linking user program to multi-threaded libzstd 2020-04-28 21:18:29 -07:00
Nick Terrell 0ed07f6dfe
Merge pull request #2094 from terrelln/stable-dst
[lib] Add ZSTD_d_stableOutBuffer + fix single-pass mode for empty frames
2020-04-28 17:53:24 -07:00
Yann Collet f17ac423b2 new tests created new artifacts
they were not properly ignored
2020-04-28 15:58:22 -07:00
Nick Terrell 108a5572a5
Merge pull request #2048 from nocnokneo/ctest-support
Add CTest support
2020-04-28 11:01:13 -07:00
Nick Terrell 1343b815f8 [fuzz] Fuzz test ZSTD_d_stableOutBuffer 2020-04-27 20:04:04 -07:00
Nick Terrell f33de06c3e [lib] Fix single-pass mode for empty frames 2020-04-27 20:04:01 -07:00
Nick Terrell a4ff217baf [lib] Add ZSTD_d_stableOutBuffer 2020-04-27 18:09:44 -07:00
Bimba Shrestha 6b4a3e019f
Merge pull request #2088 from bimbashrestha/bug
[bug] Making compressStream2 fail when passing rawContent but claiming fullDict
2020-04-23 14:16:56 -05:00
Bimba Shrestha f7a7409a49 adding fail test when passing wrong fullDict using refPrefix 2020-04-21 22:26:48 -07:00
Bimba Shrestha dba02245bf bash to shell conversion 2020-04-21 20:31:11 -07:00
Bimba Shrestha 0b107188b8 adding test for long mode trigger 2020-04-21 21:09:49 -05:00
Bimba Shrestha 5b0a452cac
Adding --long support for --patch-from (#1959)
* adding long support for patch-from

* adding refPrefix to dictionary_decompress

* adding refPrefix to dictionary_loader

* conversion nit

* triggering log mode on chainLog < fileLog and removing old threshold

* adding refPrefix to dictionary_round_trip

* adding docs

* adding enableldm + forceWindow test for dict

* separate patch-from logic into FIO_adjustParamsForPatchFromMode

* moving memLimit adjustment to outside ifdefs (need for decomp)

* removing refPrefix gate on dictionary_round_trip

* rebase on top of dev refPrefix change

* making sure refPrefx + ldm is < 1% of srcSize

* combining notes for patch-from

* moving memlimit logic inside fileio.c

* adding display for optimal parser and long mode trigger

* conversion nit

* fuzzer found heap-overflow fix

* another conversion nit

* moving FIO_adjustMemLimitForPatchFromMode outside ifndef

* making params immutable

* moving memLimit update before createDictBuffer call

* making maxSrcSize unsigned long long

* making dictSize and maxSrcSize params unsigned long long

* error on files larger than 4gb

* extend refPrefix test to include round trip

* conversion to size_t

* making sure ldm is at least 10x better

* removing break

* including zstd_compress_internal and removing redundant macros

* exposing ZSTD_cycleLog()

* using cycleLog instead of chainLog

* add some more docs about user optimizations

* formatting
2020-04-17 15:58:53 -05:00
Yann Collet 458a1a1723 minor refactor
- fix a few comments
- reorder some parameters, to enforce "mutable references first"
- simplified fwriteSparse()
2020-04-13 14:09:57 -07:00
Bimba Shrestha 794f03459e adding refPrefix 2020-04-06 22:57:49 -07:00
Bimba Shrestha 31e76f1ed4 adding test for dctx size reduction 2020-04-04 08:49:24 -07:00
Nick Terrell 1665462573
Merge pull request #2054 from terrelln/license-fix
Standardize and fix copyright and licenses
2020-03-27 11:00:01 -07:00
Nick Terrell ef9e6fe227 [test] Fix playTests.sh with space in binary path
playTests.sh didn't work when `ZSTD_BIN` or `DATAGEN_BIN` had a space in
the path name. This happens for me because I split the cmake build
directory by compiler name, like "Clang 9.0.0".

The fix is to replace all instances of `$ZSTD` with the `zstd()`
function, and the replace `$DATAGEN` with `datagen()`. This will allow
us to change how we call zstd/datagen in the future without having to
change every callsite.
2020-03-26 19:52:19 -07:00
Nick Terrell 1f144351b7 [test] Add a test that checks for valid copyright and licenses
Tests all `.h`, `.c`, `.py`, and `Makefile` files for valid copyright
and license lines. Excludes a small number of exceptions (threading, and
divsufsort).

* Copyright does not contains `present`
* Copyright contains `Facebook, Inc`
* Copyright contains the current year
* License contains exactly the lines we expect
2020-03-26 17:02:09 -07:00
Nick Terrell ac58c8d720 Fix copyright and license lines
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
2020-03-26 17:02:06 -07:00
Taylor Braun-Jones 3cbc3d37e7 Add documentation for -T option 2020-03-23 17:49:04 -04:00
Bimba Shrestha fae64b3390 Adding test for --[no-]content-size 2020-03-09 14:44:38 -05:00
Nick Terrell dbd6439bb6 [zstdgrep] Add a simple test 2020-03-02 16:51:34 -08:00
Bimba Shrestha 6a4258a08a
Removing symbols already in unit tests and adding some new unit tests for missing symbols (#1985)
* Removing symbols that are not being tested

* Removing symbols used in zstdcli, fileio, dibio and benchzstd

* Removing symbols used in zbuff and add test-zbuff to travis

* Removing remaining symbols and adding unit tests instead

* Removing symbols test entirely
2020-02-05 16:55:00 -08:00
Bimba Shrestha 2f10019b92 Adding --show-default-cparams (show cparams before compressing 2020-01-30 14:12:03 -08:00
Bimba Shrestha 9b049836c9 Typo baseline_build -> baseline_label 2020-01-28 21:39:20 -08:00
Bimba Shrestha 8fe562a770 [automated_benchmarking] Make arguments optional and add --dict argument (#1968)
* Make arugments optional and add --dict argument

* Removing accidental print statement

* Change to more likely scenario for dictionary compression benchmark
2020-01-28 11:29:43 -08:00
Nick Terrell 009f388457 Fix playTests.sh for 32-bit mode 2020-01-17 14:20:44 -08:00
Nick Terrell a11a9271d6 Fix lowLimit underflow in overflow correction 2020-01-17 12:10:18 -08:00
Nick Terrell 3ed0f65158 [cmake] Add playTests.sh as a test 2020-01-13 14:16:15 -08:00
Nick Terrell 036b30b555
Fix super block compression and stream raw blocks in decompression (#1947)
Super blocks must never violate the zstd block bound of input_size + ZSTD_blockHeaderSize. The individual sub-blocks may, but not the super block. If the superblock violates the block bound we are liable to violate ZSTD_compressBound(), which we must not do. Whenever the super block violates the block bound we instead emit an uncompressed block.

This means we increase the latency because of the single uncompressed block. I fix this by enabling streaming an uncompressed block, so the latency of an uncompressed block is 1 byte. This doesn't reduce the latency of the buffer-less API, but I don't think we really care.

* I added a test case that verifies that the decompression has 1 byte latency.
* I rely on existing zstreamtest / fuzzer / libfuzzer regression tests for correctness. During development I had several correctness bugs, and they easily caught them.
* The added assert that the superblock doesn't violate the block bound will help us discover any missed conditions (though I think I got them all).

Credit to OSS-Fuzz.
2020-01-10 18:02:11 -08:00
Bimba Shrestha f25a6e9f8f Adding new cli endpoint --patch-from= (#1940)
* Adding new cli endpoint --diff-from=

* Appveyor conversion nit

* Using bool set trick instead of direct set

* Removing --diff-from and only leaving --diff-from=#

* Throwing error when both dictFileName vars are set

* Clean up syntax

* Renaming diff-from to patch-from

* Revering comma separated syntax clean up

* Updating playtests with patch-from

* Uncommenting accidentally commented

* Updating remaining docs and var names to be patch-from instead of diff-from

* Constifying

* Using existing log2 function and removing newly created one

* Argument order (moving prefs to end)

* Using comma separated syntax

* Moving to outside #ifndef
2020-01-10 14:25:24 -08:00
Nick Terrell d1cc9d2797
[fuzz] Allow zero sized buffers for streaming fuzzers (#1945)
* Allow zero sized buffers in `stream_decompress`. Ensure that we never have two
  zero sized buffers in a row so we guarantee forwards progress.
* Make case 4 in `stream_round_trip` do a zero sized buffers call followed by
  a full call to guarantee forwards progress.
* Fix `limitCopy()` in legacy decoders.
* Fix memcpy in `zstdmt_compress.c`.

Catches the bug fixed in PR #1939
2020-01-09 11:38:50 -08:00
Nick Terrell b77ad810c9
[fuzz] Fix regression_driver.c with directory input (#1944)
The `numFiles` variable wasn't updated, so the fuzzer didn't do anything.
I did two things to fix this:

1. Remove the `numFiles` variable entirely.
2. Error if we can't open a file and print the number of files tested.
2020-01-08 13:20:56 -08:00
Bimba Shrestha eb76f786bc [bench] Automated benchmarking script (#1906)
* Initial revised automated benchmarking script

* Updating nb_iterations and making loop infinite

* Allowing benchmarking params to be changed from cli

* Renaming old speed test

* Removing numpy dependency for cli

* Change filename and benchmakr on pr level

* Moving build outside loop and adding iterations param

* Moving benchmarking to seperate travis ci test

* Fixing typo and using unused variable

* Added mode labels and updated README accordingly

* Adding new mode 'current' that compraes facebook:dev against current hash

* Typo

* Reverting previous accidental diff

* Typo

* Adding frequency config variable to prevent github from blacklisting

* Added new argument for frequency of fetching new prs

* Updating documentation
2020-01-06 14:19:11 -08:00
Bimba Shrestha b1f53b1a10 [fuzz] Dividing by targetCBlockSize instead of blockSize for nbBlocks fit (#1936)
* Adding fail logging for superblock flow

* Dividing by targetCBlockSize instead of blockSize

* Adding new const and using more acurate formula for nbBlocks

* Only do dstCapacity check if using superblock

* Remvoing disabling logic

* Updating test to make it catch more extreme case of previou bug

* Also updating comment

* Only taking compressEnd shortcut on non-superblock
2020-01-03 16:53:51 -08:00
Felix Handte 6f4341c432 Fix playTests.sh Under QEMU (#1923) 2019-12-26 11:16:23 -08:00
Bimba Shrestha 56415efc76 Constifying, malloc check and naming nit 2019-12-17 17:16:51 -08:00
Bimba Shrestha 989ce13e19 One more type conversion 2019-12-13 16:50:21 -08:00
Bimba Shrestha 4399eed42e Adding explict cast to satisfy appveyor ci 2019-12-13 16:38:11 -08:00
Bimba Shrestha db5124ef6e More void* issues. Just replacing with BYTE* 2019-12-13 16:24:49 -08:00
Bimba Shrestha 49b2bf7106 'void* size issue' fix 2019-12-13 16:06:57 -08:00
Bimba Shrestha e3cd2785e2 Add test to catch too many noCompress superblocks on streaming 2019-12-13 15:31:29 -08:00
Yann Collet d73e2fb465
Merge pull request #1891 from bimbashrestha/oss
[fuzz] Superblock fuzz issues
2019-12-10 13:17:00 -08:00
Yann Collet 4b4de7493f refactoring : remove global variable g_displayOut
displaying into stderr or stdout is now an explicit operation,
no longer depending on some global state set elsewhere in the code.
2019-12-02 16:19:03 -08:00
Yann Collet a49417b5af fix recent issue combining -r with empty list of input files
This would resize the table of input filenames to zero,
delivering an empty table,
to which it was no longer possible to add stdin.
2019-12-02 14:28:18 -08:00
Yann Collet 46bdbb9c06 Merge branch 'dev' into ahmed_file 2019-11-26 14:33:04 -08:00
Yann Collet 49cacd858b fixed fifo test 2019-11-26 14:18:09 -08:00
Bimba Shrestha a3a3c62b81 [fuzz] Only set HUF_repeat_valid if loaded table has all non-zero weights (#1898)
Fixes a fuzz issue where dictionary_round_trip failed because the compressor was generating corrupt files thanks to zero weights in the table.

* Only setting loaded dict huf table to valid on non-zero

* Adding hasNoZeroWeights test to fse tables

* Forbiding nbBits != 0 when weight == 0

* Reverting the last commit

* Setting table log to 0 when weight == 0

* Small (invalid) zero weight dict test

* Small (valid) zero weight dict test

* Initializing repeatMode vars to check before zero check

* Removing FSE changes to seperate pr

* Reverting accidentally changed file

* Negating bool, using unsigned, optimization nit
2019-11-26 12:24:19 -08:00
Yann Collet c71bd45a3b Merge branch 'dev' into ahmed_file 2019-11-26 11:20:26 -08:00
Yann Collet 60db21a677 added decompression into /dev/null tests 2019-11-25 10:53:03 -08:00
Yann Collet 7aaac3f69c fix #1904
/dev/null permissions were modified when using sudo rights.
This fixes this bug during decompression.

More importantly, this patch  adds a test, triggered in TravisCI,
ensuring unaltered /dev/null permissions.
2019-11-25 10:35:36 -08:00
Bimba Shrestha 826b555463
Merge branch 'dev' into oss 2019-11-22 17:29:33 -08:00
Bimba Shrestha 707a12c419 Test enough room for checksum in superblock 2019-11-22 17:25:36 -08:00
Nick Terrell 385a447e52
Merge pull request #1903 from terrelln/null
Fix null pointer addition
2019-11-21 17:35:06 -08:00
Nick Terrell 659e9f05cf Fix null pointer addition 2019-11-20 18:36:04 -08:00
Nick Terrell e68db76b4b Update .gitignore 2019-11-20 16:36:40 -08:00
Nick Terrell a839d6852c
Merge pull request #1888 from senhuang42/superblocks_fixed
RLE test and re-enable RLE in main compression loop
2019-11-18 16:09:33 -08:00
Sen Huang bc3e21578d No margin on RLE test size check 2019-11-18 16:39:16 -05:00
Sen Huang db8efbfe7d Updated comment to reflect actual compression behavior 2019-11-15 16:11:14 -05:00
Sen Huang 75c34684c0 Modified existing RLE test to take compressed size into account 2019-11-15 12:26:48 -05:00
Yann Collet 4b1ac69f19
Merge pull request #1868 from senhuang42/superblocks_fixed
Superblocks rebased for merge
2019-11-14 13:31:34 -08:00
Yann Collet d67742bc5d
Merge pull request #1858 from senhuang42/dictionary_header_size
Method to get dictionary header size
2019-11-14 09:44:07 -08:00
Sen Huang 97b7f712f3 Change to heap allocation, remove implicit type conversion 2019-11-08 13:57:25 -05:00
Sen Huang e1edc554a3 Added 2 unit tests: one for sanity, one for correctnesson fixed dict 2019-11-08 13:57:25 -05:00
Nick Terrell 8c474f9845 Fix parameter selection and adjustment with srcSize == 0 2019-11-07 08:58:43 -08:00
Yann Collet aea2ff5d8d fixed wrong assert() in regression driver 2019-11-06 14:56:21 -08:00
Yann Collet a7e33e3e10 updated fuzz tests to use FileNamesTable* abstraction 2019-11-06 14:42:13 -08:00
Felix Handte 32f989c39a
Merge pull request #1869 from bket/openbsd
'head -c BYTES' is non-portable.
2019-11-06 16:54:03 -05:00
Björn Ketelaars e488cabf25 'date -r FILE' is non-portable.
date(1) is used to display the last modification time of a file, which
is not supported on OpenBSD, FreeBSD and Darwin. Instead use stat(1).

Tested on OpenBSD.
2019-11-06 13:55:59 +01:00
Björn Ketelaars 66b6688274 'head -c BYTES' is non-portable.
Similar to #1321: The '-c' option for 'head' is non-portable (not in
POSIX).
2019-11-06 11:39:59 +01:00
Yann Collet 81c51483e9 Merge branch 'dev' into ahmed_file 2019-11-05 14:44:12 -08:00
Sen Huang e21a8bbecd Fix FUZZ_rand32() bug 2019-11-05 16:43:24 -05:00
Sen Huang f2932fb5eb Fix more merge conflicts 2019-11-05 15:54:05 -05:00
Nick Terrell 60205fec02 Fix 2 bugs in dictionary loading
* Silently skip dictionaries less than 8 bytes, unless using `ZSTD_dct_fullDict`.
  This changes the compressor, which silently skips dictionaries <= 8 bytes.
* Allow repcodes that are equal to the dictionary content size, since it is in bounds.
2019-11-01 16:52:07 -07:00
Nick Terrell 75e7c0d107 [fuzz] Add dictionary_loader fuzzer
* Adds the fuzzer
* Adds an additional `InputType` for the fuzzer

I ran the fuzzer for about 10 minutes and it found 2 bugs:

* Catches the original bug without any help
* Catches an additional bug with 8-byte dictionaries
2019-11-01 15:54:24 -07:00
Yann Collet 47034cd6c3
Merge pull request #1850 from bimbashrestha/named_pipes
Gating named pipe support on windows
2019-10-30 12:18:58 -07:00
Shashank Tavildar 6d0b7bd6ce Changed extension comparision logic, added new test cases 2019-10-29 13:36:09 -07:00
Shashank Tavildar 9ab6a747d4 Created utility function to extract extension from filename, fixed tests 2019-10-29 12:27:54 -07:00
Shashank Tavildar 5e6dbad6c1 Deleting test directory before creation to pass travis-ci test 2019-10-29 09:54:54 -07:00
Shashank Tavildar 850ba66139 Minor fixes in test 2019-10-28 22:24:01 -07:00
Bimba Shrestha da73b11b14
Merge branch 'dev' into named_pipes 2019-10-29 01:17:13 -04:00
Shashank Tavildar 0f2bff2faf Addressing comments, removing cyclic dependency with header file, updating tests 2019-10-28 18:21:47 -07:00
Yann Collet 85df7a4da8 added test which can overflow internal list of filenames 2019-10-28 16:05:42 -07:00
Shashank Tavildar 0e9a37daeb Fixing tests and moving flag to advanced usage 2019-10-28 15:22:26 -07:00
Yann Collet 170982fbd9 added symlink test for --filelist=FILE 2019-10-28 15:15:26 -07:00
Shashank Tavildar f422914619 Resolving local merge conflicts 2019-10-28 15:15:13 -07:00
Yann Collet d9c634e13b return final `\0` directly from readLine() 2019-10-28 15:03:32 -07:00
Shashank Tavildar 02433e0b15 Addressing comments: -Created a list of extensions defined in fileio.h, -Updated test 2019-10-28 14:54:54 -07:00
Shashank Tavildar 48f856640e Added --exclude-compressed flag feature that skips compression of precompressed files 2019-10-28 13:57:40 -07:00
Yann Collet d7f258d845 change command name to `--filelist=`
updated documentation and man page
2019-10-28 13:54:36 -07:00
Yann Collet a78a8759a2 Merge branch 'dev' into ahmed_file 2019-10-28 13:43:28 -07:00
Yann Collet b4037b18ef
Merge pull request #1848 from felixhandte/tzst-ext-tmp
Support Decompressing Short Tar Extensions
2019-10-28 10:08:20 -07:00
Yann Collet 8e414b586d test resilience on garbage file
and clean traces
2019-10-25 17:01:26 -07:00
Yann Collet 1ead0c5d5a improved --file=FILE implementation
pass basic tests
2019-10-25 16:36:59 -07:00
Shashank Tavildar 55ee7d56e4 Added --exclude-compressed flag feature that skips compression of precompressed files 2019-10-25 15:49:11 -07:00
Bimba Shrestha 5e59d51991 Merge branch 'master' of https://github.com/bimbashrestha/zstd into named_pipes 2019-10-25 15:43:41 -07:00
Bimba Shrestha 5b45ff4f1c Gating named file support on windows 2019-10-25 11:32:38 -07:00
W. Felix Handte 4eccc82875 Minor Fixes 2019-10-25 13:59:19 -04:00
Yann Collet 14e9010bb5
Merge pull request #1844 from AhmedAbdellah19/adding_read_files_from_file_feature
Adding --file=FILE feature
2019-10-25 10:11:47 -07:00
Yann Collet c2140e9db0
Merge pull request #1845 from facebook/zbuff
improve deprecation warning macro
2019-10-25 09:59:00 -07:00
Ahmed Abdellah 1faeb222b2 adding some functional tests 2019-10-25 15:54:52 +01:00
W. Felix Handte 24499036ba Add Tests 2019-10-24 20:19:10 -04:00
Yann Collet 37f90930da
Merge pull request #1839 from bimbashrestha/named_pipes
Allowing named pipes to go through in zstdcli
2019-10-24 10:53:12 -07:00
Yann Collet 63e435dda1 improve deprecation warning macro
fix #1488

although, curiously enough, I was never able to reproduce the issue
(according to the bug report, it should be present while using gcc 4.8).
2019-10-23 11:59:32 -07:00
Nick Terrell 3c8ff35013
Merge pull request #1840 from terrelln/regression-fix
[regression] Fix the old streaming regression test
2019-10-22 17:58:31 -07:00
Bimba Shrestha 10f7147013 Adding shell test and removing bash test file 2019-10-22 17:09:44 -07:00
Yann Collet 09d0b52998
Merge pull request #1838 from facebook/fixAdvanced
fix initCStream_advanced() for fast strategies
2019-10-22 16:22:55 -07:00
Bimba Shrestha edae3c8f3c Adding bash test for named pipes 2019-10-22 16:15:59 -07:00
Nick Terrell cf21003995 [regression] Fix the old streaming regression test
* A copy-paste error made it so we weren't running the advanced/cdict
  streaming tests with the old API.
* Clean up the old streaming tests to skip incompatible configs.
* Update `results.csv`.

The tests now catch the bug in #1787.
2019-10-22 15:43:17 -07:00
Yann Collet 514f941a70
Merge pull request #1836 from facebook/dirFlat2
Added --output-dir-flat=
2019-10-22 15:19:04 -07:00
Yann Collet 5d5c895b18 fix initCStream_advanced() for fast strategies
Compression ratio of fast strategies (levels 1 & 2)
was seriously reduced, due to accidental disabling of Literals compression.

Credit to @QrczakMK, which perfectly described the issue, and implementation details,
making the fix straightforward.

Example : initCStream with level 1 on synthetic sample P50 :
Before : 5,273,976 bytes
After  : 3,154,678 bytes
ZSTD_compress (for comparison) : 3,154,550

Fix #1787.

To follow : refactor the test which was supposed to catch this issue (and failed)
2019-10-22 15:01:38 -07:00
Nick Terrell b1ec94e63c Fix ZSTD_f_zstd1_magicless for small data
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
  take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
  taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.

Fixes #1813.
2019-10-21 21:16:17 -07:00
Nick Terrell 919d1d8e93
Merge pull request #1831 from terrelln/zstdmt-bad-memset
[zstdmt] Don't memset the jobDescription
2019-10-21 15:53:57 -07:00
Yann Collet 989e0f0cee Added --output-dir-flat=
which is a preferred construction when applicable.

Also, added --output-dir-flat into man page.
2019-10-21 14:39:18 -07:00
Yann Collet 6cf04c0344
Merge pull request #1834 from facebook/winFix
Windows fixes
2019-10-21 13:45:17 -07:00
Yann Collet 2ad75bb574 validated 'make test' for mingw environment 2019-10-18 15:37:07 -07:00
Yann Collet 85a016ed92 made `make check` faster
for shorter feedback loop
2019-10-18 15:21:50 -07:00
Nick Terrell 243824551f [threading] Add debug utilities 2019-10-18 15:05:34 -07:00
Yann Collet 29e46ed0bd fix test on windows
isDirectory() doesn't work on Windows
if directory name is followed by '/'
2019-10-18 14:28:34 -07:00
Nick Terrell 8c11f089a1 [fuzz] Increase output buffer size of stream_round_trip
Fixes OSS-Fuzz crash.
Credit to OSS-Fuzz
2019-10-18 13:39:08 -07:00
Yann Collet ad86a5d0bc rewrite FIO_createFilename_fromOutDir() 2019-10-18 11:15:10 -07:00
Yann Collet ba7e2b6da7 tests: can override isTerminal with environment variable 2019-10-17 15:07:47 -07:00
Yann Collet 6323966e53 updated erroneous comments using ZSTD_dm_*
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822)
2019-10-16 16:14:04 -07:00
Yann Collet b3e92ad346
Merge pull request #1812 from senhuang42/output-dir-flat
Support for --output-dir-flat: output into a directory
2019-10-14 09:25:05 -07:00
Yann Collet fb77afc626
Merge pull request #1760 from bimbashrestha/extract_sequences_api
Adding api for extracting sequences from seqstore
2019-10-10 13:11:18 -07:00
Bimba Shrestha 36528b96c4 Manually moving instead of memcpy on decoder and using genBuffer() 2019-10-03 09:26:51 -07:00
Sen Huang f80437c586 Add support for --output-dir-flat
New flag to specify output directory destination for multiple files.
2019-10-02 11:08:20 -04:00
Bimba Shrestha b63a1e7ae5 Typo fix 2019-09-27 07:20:20 -07:00
Bimba Shrestha 91daee5c06 Fixing appveyor test 2019-09-26 16:21:57 -07:00
Bimba Shrestha 75b1286354 Fixing shortest failure 2019-09-26 16:07:34 -07:00
Bimba Shrestha bb27472afc Adding more realistic test for get sequences 2019-09-26 15:38:31 -07:00
Bimba Shrestha be0bebd24e Adding test and null check for malloc 2019-09-23 15:08:18 -07:00
Nick Terrell 7451c6578c
Merge pull request #1804 from terrelln/wild-and-fast
Optimize (de)compression and fix wildcopy overread
2019-09-21 17:04:36 -07:00
Felix Handte c047fcf7bf
Merge pull request #1806 from felixhandte/estimate-cctx-doc
Update Comment on `ZSTD_estimateCCtxSize()`
2019-09-20 15:36:00 -04:00
W. Felix Handte f7d9b36835 Update Comment on `ZSTD_estimateCCtxSize()` 2019-09-20 14:11:29 -04:00
Nick Terrell e068bd01df [tests] Fix decodecorpus 2019-09-20 01:09:47 -07:00
Nick Terrell 51990246c3 [test] Test the bounds of ZSTD_c_srcSizeHint 2019-09-18 11:05:08 -07:00
Bimba Shrestha 3cacc0a30b Casting void pointer to ZSTD_Sequence pointer 2019-09-17 17:44:08 -07:00
Bimba Shrestha 5b038f128f Merge branch 'extract_sequences_api' of https://github.com/bimbashrestha/zstd into extract_sequences_api 2019-09-16 13:35:49 -07:00
Bimba Shrestha 1f93be0f6d Handling memory leak and potential side effect 2019-09-16 13:35:45 -07:00
Bimba Shrestha a874435478
Merge branch 'dev' into extract_sequences_api 2019-09-16 13:29:59 -07:00
Felix Handte 2164a130f3
Merge pull request #1780 from felixhandte/workspace-efficiency-3
Avoid Clearing Tables Even When Changing CParams
2019-09-16 14:37:05 -04:00
Nick Terrell d721fcf3ee [fuzz] Fix leak in block_round_trip 2019-09-13 10:32:38 -07:00
Nick Terrell 7c4578160e [fuzz] Generate seed data up to 256KB 2019-09-12 15:02:01 -07:00
Dario Pavlovic 51e9d29a51 Merge branch 'improvDataGen' of github.com:darxsys/zstd into improvDataGen 2019-09-12 13:11:02 -07:00
Dario Pavlovic cd8588077e It's time for all of rng seed code to go. Goodbye 2019-09-12 13:10:34 -07:00
Dario Pavlovic 47bb4c6a23
Update tests/fuzz/fuzz_data_producer.h 2019-09-12 12:45:28 -07:00
Dario Pavlovic 92c58c4d5d Use range instead of the generic uint32 method to use less bytes when generating necessary numbers. 2019-09-12 12:40:12 -07:00
W. Felix Handte 194c542598 Fix Memory Leak in Test 2019-09-11 14:25:30 -04:00
W. Felix Handte ff67c62458 Fix Compilation Error (`uint32_t` -> `size_t`) 2019-09-11 13:59:09 -04:00
W. Felix Handte 5707c8a9d5 Speed Up Test a Little 2019-09-11 13:23:59 -04:00
W. Felix Handte ed4c2c60c3 Add Fuzzer Test Case for Index Reduction 2019-09-11 13:17:19 -04:00
Dario Pavlovic b5b24c2a0d Combining fuzz_data_producer restrict calls into a single function 2019-09-11 10:09:29 -07:00
Dario Pavlovic 23cc2d8510 All tests should give some portion of data to the producer and use the rest. 2019-09-10 16:52:38 -07:00
Dario Pavlovic 4dfc1bbf48 Merge branch 'dev' into improvDataGen 2019-09-10 16:20:27 -07:00
Dario Pavlovic 0630d084cb [Fuzz] Improve data generation #1723
Converting the rest of the tests to use the new data producer.
2019-09-10 16:14:43 -07:00
Nick Terrell 458b602bb9
Merge pull request #1777 from darxsys/improvDataGen
[Fuzz] Improve data generation #1723
2019-09-10 11:51:23 -07:00
Yann Collet 41416f0927
Merge pull request #1773 from bimbashrestha/rle_first_block_decompression_fix
Removing redundant condition in decompression, making first block rle…
2019-09-10 11:17:29 -07:00
Bimba Shrestha caaf43b258 Using a number instead of M prefix in head call 2019-09-10 09:30:37 -07:00
Bimba Shrestha 9e7bb55e14 Addressing comments 2019-09-09 20:04:46 -07:00
Dario Pavlovic ea1ad123da Addressing nits 2019-09-09 16:13:24 -07:00
Dario Pavlovic 3932fcfebc Fixing issues with double usage of data. 2019-09-09 15:39:04 -07:00
Bimba Shrestha e6be4cf4eb Changing test file directory names to be more descriptive 2019-09-09 12:08:33 -07:00
Bimba Shrestha 0b25ab2202 Testing first block decompression cli 2019-09-09 11:54:43 -07:00
Dario Pavlovic a71bbba7be [Fuzz] Improve data generation #1723 2019-09-09 08:43:22 -07:00
Bimba Shrestha 5f8b0f6890 Changing api to get sequences across all blocks 2019-08-30 09:18:44 -07:00
Nick Terrell 8ce9c1917e [regression] small improvement from RLE blocks 2019-08-29 18:05:58 -07:00
Yann Collet 5198347382
Merge pull request #1744 from bimbashrestha/dev
Generate RLE blocks in the encoder
2019-08-29 15:19:10 -07:00
bimbashrestha e5704bbfdf Added test for multiple blocks of zeros and fixed nit about comments 2019-08-28 08:32:34 -07:00
Nick Terrell 793c71d6ae [test][dictBuilder] Add multithreaded tests 2019-08-26 18:32:08 -07:00
Nick Terrell d0750a1c9c
Merge pull request #1733 from nmagerko/size-hint
Add --size-hint=# option
2019-08-23 10:16:10 -07:00
Nick Terrell e2030a2c40 [fuzz] Add a DEBUGLOG(3) statement to print file
Enable it by building with this command:

```
./fuzz.py build all --debug 3
```
2019-08-22 17:27:15 -07:00
Nick Magerko 493f95c7df Fix merge conflicts 2019-08-22 11:51:41 -07:00
Nick Terrell a505463710
Merge pull request #1726 from nmagerko/stream-size
Add --stream-size=# option
2019-08-22 11:31:15 -07:00
Nick Magerko b3540507f5 Remove bc from play tests 2019-08-21 10:27:54 -07:00
Nick Terrell 188311dd4a
Merge pull request #1736 from terrelln/fuzz-fix
[fuzz] Improve fuzzer build script and docs
2019-08-21 10:09:38 -07:00
Nick Terrell 3982935aef [fuzz] Improve fuzzer build script and docs
* Remove the `make libFuzzer` target since it is broken and obsoleted
  by `CC=clang CXX=clang++ ./fuzz.py build all --enable-fuzzer`. The
  new `-fsanitize=fuzzer` is much better because it works with MSAN
  by default.
* Improve the `./fuzz.py gen` command by making the input type explicit
  when creating a new target.
* Update the `README` for `--enable-fuzzer`.

Fixes #1727.
2019-08-20 16:44:50 -07:00
Nick Magerko c7a24d7a14 Define ZSTD_SRCSIZEHINT_MIN as 0 2019-08-20 13:06:15 -07:00
Nick Magerko f23402f1f5 Remove unnecessary test case 2019-08-19 17:20:46 -07:00
Nick Magerko 294f1e5cfe Fix typo in test 2019-08-19 16:53:02 -07:00
Nick Magerko f9af70ca8a Fix playTests and add additional cases 2019-08-19 16:48:35 -07:00
Nick Magerko ea9d35922c Add size-hint to fuzz tests 2019-08-19 15:12:29 -07:00
Nick Magerko dffbac5f89 Add --size-hint=# option 2019-08-19 11:38:49 -07:00
Nick Magerko 85d07c6c47 Tweak stdout, stderr redirection in new playTests 2019-08-16 12:49:21 -07:00
Nick Magerko af0c9501d1 Add --stream-size=# command 2019-08-16 11:34:39 -07:00
Ed Maste b81d7cc6a0 remove extraneous doubled ;s 2019-08-15 21:17:06 -04:00
Yann Collet 0b0b83e8f3 fix test 122
it's an unsupported scenario.
2019-08-03 16:51:26 +02:00
Yann Collet efe8496755 minor test refactoring
just for clarity, for the currently failing unit test
2019-08-02 19:31:19 +02:00
Yann Collet 387e20d4f0 fixed minor conversion warning in datagen 2019-08-02 18:02:54 +02:00
Yann Collet 37f47e51a8 fixed datagen
to produce same content on both 32 and 64-bit platforms
by removing floating from literal table determination.

also : added checksum trace in compression control test,
so that it's easier to determine if test fails
as a consequence of compressing a different sample.
2019-08-02 17:34:53 +02:00