Commit Graph

608 Commits (7b75d772b1dc94c909d79f3c0b411d47f7a920ed)

Author SHA1 Message Date
Bimba Shrestha 0301ef5d04
[bench] Extending largeNbDicts to compression (#2089)
* adding cdict_collection_t

* adding shuffleCDictionaries()

* adding compressInstructions

* adding compress()

* integrating compression into bench()

* copy paste error fix

* static analyzer uninit value complaint fix

* changing to control

* removing assert

* changing to control

* moving memcpy to seperate function

* fixing static analyzer complaint

* another hacky solution attempt

* Copying createbuffer logic
2020-05-04 10:42:22 -07:00
Carl Woffenden 3eaa525138 Fixed clash when projects are already using xxHash
Undefing XXH_* macros allows the `.c` to build standalone without clashes. Removing `xxhash.c` and only including the header is the correct usage (according to `XXH_PRIVATE_API`).
2020-04-07 18:17:59 +02:00
Carl Woffenden d01e7c2fad Renamed directory 2020-04-07 13:34:19 +02:00
Carl Woffenden 59812afac7 Improved documentation 2020-04-07 13:19:52 +02:00
Carl Woffenden 7af7735fa3 Merge remote-tracking branch 'upstream/dev' into single-file-lib 2020-04-07 11:13:02 +02:00
Carl Woffenden edd9a07322 Code replicated in compression and decompression moved to shared headers
`CHECK_F` macro moved to `error_private.h` (shared between `fse_compress.c` and `fse_decompress.c`). `ZSTD_limitCopy()` moved to `zstd_internal.h` (shared between `zstd_compress.c` and `zstd_decompress.c`). Erroneous build artefact `zstd.h` removed from repo.
2020-04-07 11:02:06 +02:00
Bimba Shrestha 1d267dc5d6 returning on null check 2020-04-03 13:39:02 -07:00
Bimba Shrestha d598c88fb3 adding fclose before return 2020-04-03 13:31:47 -07:00
Carl Woffenden 88da79b42e Minor typo 2020-04-03 20:50:54 +02:00
Carl Woffenden 0442be5aa4 Added missing -pthread flag (was breaking on BSD) 2020-04-03 20:34:47 +02:00
Carl Woffenden 7c420344d2 Single-file decoder script can now (optionally) create an encoder
To complement the single-file decoder a new script was added to create an amalgamated single-file of all of the Zstd source, along with examples and (simple) tests.
2020-04-03 19:07:46 +02:00
Bimba Shrestha dbdeca1412 Edit distance match finder 2020-03-13 09:39:33 -05:00
W. Felix Handte fe454c0a76 Add Support for Dictionaries 2019-12-18 11:54:39 -08:00
W. Felix Handte 18f03516b3 Print Summary of Errors Encountered 2019-12-18 11:29:53 -08:00
W. Felix Handte 4584c88a99 Don't Rely on Implicit Return in Main 2019-12-17 11:26:18 -08:00
Yann Collet 4be9b4b9bb
Merge pull request #1893 from felixhandte/check-flipped-bits
Add Tool to Diagnose Whether Corrupt Blobs are Plausibly Bit-Flips
2019-12-03 15:35:07 -08:00
Yann Collet 9a3de0a535 changed name from createX to assembleX
shows that the resulting object just takes ownership of provided buffer.
2019-11-25 15:34:55 -08:00
W. Felix Handte b5fb2e7c12 Add Tool to Diagnose Whether Corrupt Blobs are Plausibly Bit-Flips
I spend an increasing amount of my time looking at "Corrupted block detected"
failures in decompression. Not infrequently, I suspect that it is the result
of hardware failure, and that the blob has become bit-flipped or otherwise
corrupted somewhere along the line.

For that reason I was motivated to write a little tool to inspect blobs that
fail to decompress, to try modifying them, and then check whether they
decompress successfully. This seems like potentially a generally useful tool,
so I figured it might be worth putting in `contrib/`.
2019-11-19 01:53:36 -05:00
Yann Collet 485fec5665 removed experimental_dict_builders
no longer required,
can still be accessed by going to back to an earlier commit or release (<= v1.4.4)
2019-11-06 14:43:14 -08:00
Yann Collet 31a0abbfda updated pzstd and largeNbDicts to use the new FileNamesTable* abstraction 2019-11-06 09:10:05 -08:00
Yann Collet 09b1844d9b
Merge pull request #1784 from bimbashrestha/fse_block_bound_err
Rearranging assert and allowing 4 extra for FSE_BLOCKBOUND()
2019-09-12 19:09:27 -07:00
Bimba Shrestha 43da5bf27e Rearranging assert and allowing 4 extra for FSE_BLOCKBOUND() 2019-09-12 14:43:50 -07:00
Carl Woffenden 88975e8c25 Minor: documented sizes smaller 2019-09-02 18:15:31 +02:00
Carl Woffenden 8ac29cc825 Correctness and tidy
Test compilation performed with warnings. Author and license added. Test for failing grep on ancient OSX versions. Replaced the test image with something less noisy (which compresses better).
2019-09-02 18:02:50 +02:00
Yann Collet 64102f08da Merge branch 'dev' into decTest 2019-08-29 09:48:12 -07:00
Carl Woffenden 72e51ac246 C99 and older GCC fixes 2019-08-29 11:16:57 +02:00
Yann Collet 4b3a8fe1c4 fix create_ script for sh 2019-08-28 13:23:48 -07:00
Yann Collet 9589e8e4bb
Merge pull request #1749 from facebook/rmadapt
removed adaptive-compression
2019-08-28 12:26:29 -07:00
Yann Collet 8af941d2d7 Merge branch 'dev' into decTest 2019-08-28 12:17:29 -07:00
Carl Woffenden cdf73e915e Rewrote the scripts to sh instead of bash 2019-08-28 19:20:42 +02:00
Yann Collet f61e8a231f minor script renaming, for clarity 2019-08-27 16:01:39 -07:00
Yann Collet 517aeb89dc changed contrib project name for clarity 2019-08-27 15:50:47 -07:00
Yann Collet 5ed1b1e11d removed adaptive-compression
the functionality is already integrated into `zstd` through `--adapt` command
2019-08-27 14:47:40 -07:00
Carl Woffenden 51868964ef Fixed test failure when Emscripten not present 2019-08-27 17:12:57 +02:00
Carl Woffenden 6213b7b3b4 Minor repetition 2019-08-27 16:57:23 +02:00
Carl Woffenden 59052d5fd8 Typo 2019-08-27 16:55:03 +02:00
Carl Woffenden ec12721538 Added clarification 2019-08-27 15:53:26 +02:00
Carl Woffenden 6712a644fa Added reasoning 2019-08-27 15:51:14 +02:00
Carl Woffenden 4f2a8b752a Typo 2019-08-27 15:38:34 +02:00
Carl Woffenden a57de4ac89 Added test script; tidied and documented
The test script combines the sources then builds and runs an example. A futher example is built if the Emscripten compiler is available on the system. Documentation covers building.
2019-08-27 15:36:06 +02:00
Carl Woffenden 7c6fa81579 Added Emscripten example, removed Buck, minor tidy
Work-in-progress. Added simple Emscripten WebGL example that adds 25kB when build with Zstd. Removed Buck (will replace). Minor correctness.
2019-08-26 21:28:19 +02:00
Carl Woffenden ea8f6d2a07 Able to test combine script; minor tidy 2019-08-26 07:48:57 +02:00
Carl Woffenden d760e35ebc Preparing to run tests
Combine script more robust and can output to a specified file. Initial buck files added (work in progress).
2019-08-25 22:49:01 +02:00
Carl Woffenden 36a59336da Minor fix for files with spaces. Typo. 2019-08-23 23:09:13 +02:00
Carl Woffenden 0a49353a46 Added generator script and simple test
The script will combine decompressor sources into a single file. The example shows this in use.
2019-08-23 18:43:29 +02:00
Felix Handte 2314906b68
Merge pull request #1699 from felixhandte/seekable-gitignore
Add New Seekable Compression Example to .gitignore
2019-07-24 19:07:55 -04:00
Yann Collet 0d38ee3c30
Merge pull request #1690 from piguin/dev
fix compiling errors with clang-8
2019-07-24 15:37:05 -07:00
W. Felix Handte 15da57820d Add New Seekable Compression Example to .gitignore 2019-07-24 18:22:20 -04:00
Sean Purcell 671d533ea7 Fix seekable decompression in-memory api 2019-07-21 23:22:25 -04:00
Qin Li 04a9d6b828 fix compiling errors with clang-8
Compiling with clang-8 fails with the following errors:

largeNbDicts.c:562:37: error: implicit conversion turns floating-point
number into integer: 'const double' to 'U64' (aka 'unsigned long')
[-Werror,-Wfloat-conversion]
        U64 const dTime_ns = result.nanoSecPerRun;
                  ~~~~~~~~   ~~~~~~~^~~~~~~~~~~~~

zstdcli.c:300:5: error: '@return' command used in a comment that is
not attached to a function or method declaration
[-Werror,-Wdocumentation]
 * @return 1 means that cover parameters were correct
   ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

zstdcli.c:301:5: error: '@return' command used in a comment that is
not attached to a function or method declaration
[-Werror,-Wdocumentation]
 * @return 0 in case of malformed parameters
   ~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2019-07-18 19:41:00 -07:00