Commit Graph

7331 Commits (f25a6e9f8f36375bce85bfefdf388c770057c27c)

Author SHA1 Message Date
Bimba Shrestha f25a6e9f8f Adding new cli endpoint --patch-from= (#1940)
* Adding new cli endpoint --diff-from=

* Appveyor conversion nit

* Using bool set trick instead of direct set

* Removing --diff-from and only leaving --diff-from=#

* Throwing error when both dictFileName vars are set

* Clean up syntax

* Renaming diff-from to patch-from

* Revering comma separated syntax clean up

* Updating playtests with patch-from

* Uncommenting accidentally commented

* Updating remaining docs and var names to be patch-from instead of diff-from

* Constifying

* Using existing log2 function and removing newly created one

* Argument order (moving prefs to end)

* Using comma separated syntax

* Moving to outside #ifndef
2020-01-10 14:25:24 -08:00
Nick Terrell d1cc9d2797
[fuzz] Allow zero sized buffers for streaming fuzzers (#1945)
* Allow zero sized buffers in `stream_decompress`. Ensure that we never have two
  zero sized buffers in a row so we guarantee forwards progress.
* Make case 4 in `stream_round_trip` do a zero sized buffers call followed by
  a full call to guarantee forwards progress.
* Fix `limitCopy()` in legacy decoders.
* Fix memcpy in `zstdmt_compress.c`.

Catches the bug fixed in PR #1939
2020-01-09 11:38:50 -08:00
Igor Sugak 03ffda7b88 fix UBSAN's invalid-null-argument error in zstd_decompress.c (#1939) 2020-01-08 16:17:42 -08:00
Nick Terrell b77ad810c9
[fuzz] Fix regression_driver.c with directory input (#1944)
The `numFiles` variable wasn't updated, so the fuzzer didn't do anything.
I did two things to fix this:

1. Remove the `numFiles` variable entirely.
2. Error if we can't open a file and print the number of files tested.
2020-01-08 13:20:56 -08:00
Christoph Reiter d0dcaf56c2 Make UTIL_countPhysicalCores() work under Cygwin (#1941)
Cygwin currently uses the fallback implementation which just returns 1 every time,
which leads to bad performance when zstd is called with -T0 for example.

Instead use the POSIX implementation used for the BSDs which works just fine under Cygwin.

Tested under Cygwin and MSYS2.
2020-01-07 15:48:26 -08:00
Yann Collet 3119256267 updated CHANGELOG
one missing item in v1.4.4
2020-01-07 09:51:22 -08:00
Bimba Shrestha eb76f786bc [bench] Automated benchmarking script (#1906)
* Initial revised automated benchmarking script

* Updating nb_iterations and making loop infinite

* Allowing benchmarking params to be changed from cli

* Renaming old speed test

* Removing numpy dependency for cli

* Change filename and benchmakr on pr level

* Moving build outside loop and adding iterations param

* Moving benchmarking to seperate travis ci test

* Fixing typo and using unused variable

* Added mode labels and updated README accordingly

* Adding new mode 'current' that compraes facebook:dev against current hash

* Typo

* Reverting previous accidental diff

* Typo

* Adding frequency config variable to prevent github from blacklisting

* Added new argument for frequency of fetching new prs

* Updating documentation
2020-01-06 14:19:11 -08:00
Bimba Shrestha b1f53b1a10 [fuzz] Dividing by targetCBlockSize instead of blockSize for nbBlocks fit (#1936)
* Adding fail logging for superblock flow

* Dividing by targetCBlockSize instead of blockSize

* Adding new const and using more acurate formula for nbBlocks

* Only do dstCapacity check if using superblock

* Remvoing disabling logic

* Updating test to make it catch more extreme case of previou bug

* Also updating comment

* Only taking compressEnd shortcut on non-superblock
2020-01-03 16:53:51 -08:00
Bimba Shrestha ef1684f29a [doc] Adding -M# decompression operation modifier documentation (#1938)
* Adding -M# operation modifier documentation

* Adding long name, typo and update desc
2020-01-03 16:36:47 -08:00
Felix Handte 68160372be
Merge pull request #1920 from felixhandte/fix-mtim-again
Use statbuf->st_mtim Again
2020-01-03 16:07:16 -05:00
Felix Handte 6f4341c432 Fix playTests.sh Under QEMU (#1923) 2019-12-26 11:16:23 -08:00
W. Felix Handte e2a99db2dd Also Define _ATFILE_SOURCE 2019-12-23 13:24:28 -05:00
W. Felix Handte 5666835ea7 Add Comment 2019-12-23 13:24:28 -05:00
W. Felix Handte 5af8cb7aea Use statbuf->st_mtim Again 2019-12-23 13:24:28 -05:00
W. Felix Handte 976c055a9a Add Travis Test that Builds ARM on Trusty 2019-12-23 13:24:28 -05:00
W. Felix Handte 5f46894184 Fix playTests.sh Under QEMU 2019-12-23 13:24:28 -05:00
Yann Collet 1c9ac4ccf4
Merge pull request #1928 from bimbashrestha/bimba-oss
[fuzz] Adding bool to check if there is enough room to emit noCompress Superblocks
2019-12-19 12:58:24 -08:00
Felix Handte 07ad866754
Merge pull request #1932 from felixhandte/diagnose-corruption-dicts
Add Support for Dictionaries in Corruption Diagnosis Tool
2019-12-19 12:08:28 -05:00
W. Felix Handte fe454c0a76 Add Support for Dictionaries 2019-12-18 11:54:39 -08:00
Yann Collet 4ae0f681b6
Merge pull request #1929 from facebook/travisTest
added ppc64le tests on travis
2019-12-18 11:49:26 -08:00
W. Felix Handte 18f03516b3 Print Summary of Errors Encountered 2019-12-18 11:29:53 -08:00
Bimba Shrestha 56415efc76 Constifying, malloc check and naming nit 2019-12-17 17:16:51 -08:00
W. Felix Handte 4584c88a99 Don't Rely on Implicit Return in Main 2019-12-17 11:26:18 -08:00
Yann Collet 3746693b71 added ppc64le tests on travis
on real hardware
2019-12-15 01:30:12 -08:00
Bimba Shrestha 989ce13e19 One more type conversion 2019-12-13 16:50:21 -08:00
Bimba Shrestha 4399eed42e Adding explict cast to satisfy appveyor ci 2019-12-13 16:38:11 -08:00
Bimba Shrestha db5124ef6e More void* issues. Just replacing with BYTE* 2019-12-13 16:24:49 -08:00
Bimba Shrestha 49b2bf7106 'void* size issue' fix 2019-12-13 16:06:57 -08:00
Bimba Shrestha 5225dcfc0f Adding bool to check if enough room left for noCompress superblocks 2019-12-13 15:47:28 -08:00
Bimba Shrestha e3cd2785e2 Add test to catch too many noCompress superblocks on streaming 2019-12-13 15:31:29 -08:00
Yann Collet d73e2fb465
Merge pull request #1891 from bimbashrestha/oss
[fuzz] Superblock fuzz issues
2019-12-10 13:17:00 -08:00
Yann Collet e839b31dc4
Merge pull request #1924 from ned14/dev
Make idiomatic cmake install so third party cmake installers work right.
2019-12-10 13:16:45 -08:00
Niall Douglas (s [underscore] sourceforge {at} nedprod [dot] com) 23e997439a
Make idiomatic cmake install so third party cmake installers work right. 2019-12-06 17:18:54 +00:00
Bimba Shrestha e1913dc87f Making const, removing unnecessary indent, changing parameter order 2019-12-04 15:51:17 -08:00
Bimba Shrestha 2ec556fec2 Moving init/end functions, moving compressSuperBlock inside body() 2019-12-04 15:23:13 -08:00
Bimba Shrestha ffb0463041 Refactor 2019-12-04 14:52:27 -08:00
Bimba Shrestha 49c6d49247 [fuzz] msan uninitialized unsigned value (#1908)
Fixes new fuzz issue

Credit to OSS-Fuzz

* Initializing unsigned value

* Initialilzing to 1 instead of 0 because its more conservative

* Unconditionoally setting to check first and then checking zero

* Moving bool to before block for c90

* Move check set before block
2019-12-04 10:02:17 -08:00
Yann Collet 5120883a9c bumped version number
so that potential issue report do not confuse `dev` with latest release
2019-12-03 17:06:42 -08:00
Yann Collet 4be9b4b9bb
Merge pull request #1893 from felixhandte/check-flipped-bits
Add Tool to Diagnose Whether Corrupt Blobs are Plausibly Bit-Flips
2019-12-03 15:35:07 -08:00
Yann Collet 901e545173
Merge pull request #1907 from selavy/check-fread-r1895
Check for fread failure
2019-12-03 15:31:11 -08:00
Bimba Shrestha 1fc9352f81 Using bss var instead of creating new bool 2019-12-02 21:39:06 -08:00
Yann Collet e8faa09bdc
Merge pull request #1918 from facebook/recurse_fix
small refactoring : remove global variable g_displayOut
2019-12-02 17:30:37 -08:00
Yann Collet 4b4de7493f refactoring : remove global variable g_displayOut
displaying into stderr or stdout is now an explicit operation,
no longer depending on some global state set elsewhere in the code.
2019-12-02 16:19:03 -08:00
Yann Collet cc3252acce
Merge pull request #1917 from facebook/recurse_fix
fix recent issue combining -r with empty list of input files
2019-12-02 15:38:14 -08:00
Yann Collet a49417b5af fix recent issue combining -r with empty list of input files
This would resize the table of input filenames to zero,
delivering an empty table,
to which it was no longer possible to add stdin.
2019-12-02 14:28:18 -08:00
Yann Collet d3ec368e13
Merge pull request #1910 from facebook/travisTest
mingw cross compilation test
2019-11-27 18:45:16 -08:00
Yann Collet 93ec5cfd05
Merge pull request #1883 from Ericson2314/fix-mingw-cross
Fix MinGW cross
2019-11-27 14:48:26 -08:00
Yann Collet 52dc6ff00e travis: updated mingw cross compilation test
- only compiles zstd CLI
- enforces zero warning tolerance policy

note : mingw compilation of the library complains (warning) about -fPIC flag.
It mostly ignores it, so it's not a big issue.
Still, it would need to be fixed if library cross-compilation becomes part of CI.
At this stage, it's unclear if this is a scenario.
2019-11-27 14:39:43 -08:00
Yann Collet d6e32afa0a added mingw cross compilation tests
based on a script in RocksDB project.

also : pushed qemu aarch64 tests to master/cron,
since there are already hardware aarch64 tests on dev.
2019-11-27 12:44:02 -08:00
Bimba Shrestha 1f681d8592 Merge branch 'oss' of https://github.com/bimbashrestha/zstd into oss 2019-11-27 10:56:54 -08:00