1747 Commits

Author SHA1 Message Date
Nick Terrell
b77ad810c9
[fuzz] Fix regression_driver.c with directory input (#1944)
The `numFiles` variable wasn't updated, so the fuzzer didn't do anything.
I did two things to fix this:

1. Remove the `numFiles` variable entirely.
2. Error if we can't open a file and print the number of files tested.
2020-01-08 13:20:56 -08:00
Bimba Shrestha
eb76f786bc [bench] Automated benchmarking script (#1906)
* Initial revised automated benchmarking script

* Updating nb_iterations and making loop infinite

* Allowing benchmarking params to be changed from cli

* Renaming old speed test

* Removing numpy dependency for cli

* Change filename and benchmakr on pr level

* Moving build outside loop and adding iterations param

* Moving benchmarking to seperate travis ci test

* Fixing typo and using unused variable

* Added mode labels and updated README accordingly

* Adding new mode 'current' that compraes facebook:dev against current hash

* Typo

* Reverting previous accidental diff

* Typo

* Adding frequency config variable to prevent github from blacklisting

* Added new argument for frequency of fetching new prs

* Updating documentation
2020-01-06 14:19:11 -08:00
Bimba Shrestha
b1f53b1a10 [fuzz] Dividing by targetCBlockSize instead of blockSize for nbBlocks fit (#1936)
* Adding fail logging for superblock flow

* Dividing by targetCBlockSize instead of blockSize

* Adding new const and using more acurate formula for nbBlocks

* Only do dstCapacity check if using superblock

* Remvoing disabling logic

* Updating test to make it catch more extreme case of previou bug

* Also updating comment

* Only taking compressEnd shortcut on non-superblock
2020-01-03 16:53:51 -08:00
Felix Handte
6f4341c432 Fix playTests.sh Under QEMU (#1923) 2019-12-26 11:16:23 -08:00
Bimba Shrestha
56415efc76 Constifying, malloc check and naming nit 2019-12-17 17:16:51 -08:00
Bimba Shrestha
989ce13e19 One more type conversion 2019-12-13 16:50:21 -08:00
Bimba Shrestha
4399eed42e Adding explict cast to satisfy appveyor ci 2019-12-13 16:38:11 -08:00
Bimba Shrestha
db5124ef6e More void* issues. Just replacing with BYTE* 2019-12-13 16:24:49 -08:00
Bimba Shrestha
49b2bf7106 'void* size issue' fix 2019-12-13 16:06:57 -08:00
Bimba Shrestha
e3cd2785e2 Add test to catch too many noCompress superblocks on streaming 2019-12-13 15:31:29 -08:00
Yann Collet
d73e2fb465
Merge pull request #1891 from bimbashrestha/oss
[fuzz] Superblock fuzz issues
2019-12-10 13:17:00 -08:00
Yann Collet
4b4de7493f refactoring : remove global variable g_displayOut
displaying into stderr or stdout is now an explicit operation,
no longer depending on some global state set elsewhere in the code.
2019-12-02 16:19:03 -08:00
Yann Collet
a49417b5af fix recent issue combining -r with empty list of input files
This would resize the table of input filenames to zero,
delivering an empty table,
to which it was no longer possible to add stdin.
2019-12-02 14:28:18 -08:00
Yann Collet
46bdbb9c06 Merge branch 'dev' into ahmed_file 2019-11-26 14:33:04 -08:00
Yann Collet
49cacd858b fixed fifo test 2019-11-26 14:18:09 -08:00
Bimba Shrestha
a3a3c62b81 [fuzz] Only set HUF_repeat_valid if loaded table has all non-zero weights (#1898)
Fixes a fuzz issue where dictionary_round_trip failed because the compressor was generating corrupt files thanks to zero weights in the table.

* Only setting loaded dict huf table to valid on non-zero

* Adding hasNoZeroWeights test to fse tables

* Forbiding nbBits != 0 when weight == 0

* Reverting the last commit

* Setting table log to 0 when weight == 0

* Small (invalid) zero weight dict test

* Small (valid) zero weight dict test

* Initializing repeatMode vars to check before zero check

* Removing FSE changes to seperate pr

* Reverting accidentally changed file

* Negating bool, using unsigned, optimization nit
2019-11-26 12:24:19 -08:00
Yann Collet
c71bd45a3b Merge branch 'dev' into ahmed_file 2019-11-26 11:20:26 -08:00
Yann Collet
60db21a677 added decompression into /dev/null tests 2019-11-25 10:53:03 -08:00
Yann Collet
7aaac3f69c fix #1904
/dev/null permissions were modified when using sudo rights.
This fixes this bug during decompression.

More importantly, this patch  adds a test, triggered in TravisCI,
ensuring unaltered /dev/null permissions.
2019-11-25 10:35:36 -08:00
Bimba Shrestha
826b555463
Merge branch 'dev' into oss 2019-11-22 17:29:33 -08:00
Bimba Shrestha
707a12c419 Test enough room for checksum in superblock 2019-11-22 17:25:36 -08:00
Nick Terrell
385a447e52
Merge pull request #1903 from terrelln/null
Fix null pointer addition
2019-11-21 17:35:06 -08:00
Nick Terrell
659e9f05cf Fix null pointer addition 2019-11-20 18:36:04 -08:00
Nick Terrell
e68db76b4b Update .gitignore 2019-11-20 16:36:40 -08:00
Nick Terrell
a839d6852c
Merge pull request #1888 from senhuang42/superblocks_fixed
RLE test and re-enable RLE in main compression loop
2019-11-18 16:09:33 -08:00
Sen Huang
bc3e21578d No margin on RLE test size check 2019-11-18 16:39:16 -05:00
Sen Huang
db8efbfe7d Updated comment to reflect actual compression behavior 2019-11-15 16:11:14 -05:00
Sen Huang
75c34684c0 Modified existing RLE test to take compressed size into account 2019-11-15 12:26:48 -05:00
Yann Collet
4b1ac69f19
Merge pull request #1868 from senhuang42/superblocks_fixed
Superblocks rebased for merge
2019-11-14 13:31:34 -08:00
Yann Collet
d67742bc5d
Merge pull request #1858 from senhuang42/dictionary_header_size
Method to get dictionary header size
2019-11-14 09:44:07 -08:00
Sen Huang
97b7f712f3 Change to heap allocation, remove implicit type conversion 2019-11-08 13:57:25 -05:00
Sen Huang
e1edc554a3 Added 2 unit tests: one for sanity, one for correctnesson fixed dict 2019-11-08 13:57:25 -05:00
Nick Terrell
8c474f9845 Fix parameter selection and adjustment with srcSize == 0 2019-11-07 08:58:43 -08:00
Yann Collet
aea2ff5d8d fixed wrong assert() in regression driver 2019-11-06 14:56:21 -08:00
Yann Collet
a7e33e3e10 updated fuzz tests to use FileNamesTable* abstraction 2019-11-06 14:42:13 -08:00
Felix Handte
32f989c39a
Merge pull request #1869 from bket/openbsd
'head -c BYTES' is non-portable.
2019-11-06 16:54:03 -05:00
Björn Ketelaars
e488cabf25 'date -r FILE' is non-portable.
date(1) is used to display the last modification time of a file, which
is not supported on OpenBSD, FreeBSD and Darwin. Instead use stat(1).

Tested on OpenBSD.
2019-11-06 13:55:59 +01:00
Björn Ketelaars
66b6688274 'head -c BYTES' is non-portable.
Similar to #1321: The '-c' option for 'head' is non-portable (not in
POSIX).
2019-11-06 11:39:59 +01:00
Yann Collet
81c51483e9 Merge branch 'dev' into ahmed_file 2019-11-05 14:44:12 -08:00
Sen Huang
e21a8bbecd Fix FUZZ_rand32() bug 2019-11-05 16:43:24 -05:00
Sen Huang
f2932fb5eb Fix more merge conflicts 2019-11-05 15:54:05 -05:00
Nick Terrell
60205fec02 Fix 2 bugs in dictionary loading
* Silently skip dictionaries less than 8 bytes, unless using `ZSTD_dct_fullDict`.
  This changes the compressor, which silently skips dictionaries <= 8 bytes.
* Allow repcodes that are equal to the dictionary content size, since it is in bounds.
2019-11-01 16:52:07 -07:00
Nick Terrell
75e7c0d107 [fuzz] Add dictionary_loader fuzzer
* Adds the fuzzer
* Adds an additional `InputType` for the fuzzer

I ran the fuzzer for about 10 minutes and it found 2 bugs:

* Catches the original bug without any help
* Catches an additional bug with 8-byte dictionaries
2019-11-01 15:54:24 -07:00
Yann Collet
47034cd6c3
Merge pull request #1850 from bimbashrestha/named_pipes
Gating named pipe support on windows
2019-10-30 12:18:58 -07:00
Shashank Tavildar
6d0b7bd6ce Changed extension comparision logic, added new test cases 2019-10-29 13:36:09 -07:00
Shashank Tavildar
9ab6a747d4 Created utility function to extract extension from filename, fixed tests 2019-10-29 12:27:54 -07:00
Shashank Tavildar
5e6dbad6c1 Deleting test directory before creation to pass travis-ci test 2019-10-29 09:54:54 -07:00
Shashank Tavildar
850ba66139 Minor fixes in test 2019-10-28 22:24:01 -07:00
Bimba Shrestha
da73b11b14
Merge branch 'dev' into named_pipes 2019-10-29 01:17:13 -04:00
Shashank Tavildar
0f2bff2faf Addressing comments, removing cyclic dependency with header file, updating tests 2019-10-28 18:21:47 -07:00