Commit Graph

7550 Commits (7b75d772b1dc94c909d79f3c0b411d47f7a920ed)

Author SHA1 Message Date
W. Felix Handte 7b75d772b1 Remove Useless Assignment in Makefile 2020-05-04 15:20:26 -04:00
W. Felix Handte 7dcca6bc64 Also Move programs/ Directory to Relative Includes 2020-05-04 15:20:26 -04:00
W. Felix Handte 6028827fee Rewrite Include Paths to be Relative
Addresses #1998.
2020-05-04 15:20:26 -04:00
Bimba Shrestha 0301ef5d04
[bench] Extending largeNbDicts to compression (#2089)
* adding cdict_collection_t

* adding shuffleCDictionaries()

* adding compressInstructions

* adding compress()

* integrating compression into bench()

* copy paste error fix

* static analyzer uninit value complaint fix

* changing to control

* removing assert

* changing to control

* moving memcpy to seperate function

* fixing static analyzer complaint

* another hacky solution attempt

* Copying createbuffer logic
2020-05-04 10:42:22 -07:00
Felix Handte 7e9aabd652
Merge pull request #2099 from felixhandte/compile-under-pedantic
Compile Under `-pedantic -Werror` and `-std=c90`
2020-05-04 10:07:13 -07:00
Felix Handte 816ed80774
Merge pull request #1984 from MeghnaM/1636-Reduce-stack-usage-of-HUF_sort
Reduce stack usage of HUF_sort()
2020-05-04 08:15:31 -07:00
W. Felix Handte 2cf72d56a6 Try to Fix MSVC Error
It's complaining about the `memcpy`s, saying:

"warning C4090: 'function': different 'const' qualifiers"

Let's try explicitly casting to the argument types...
2020-05-04 10:59:15 -04:00
W. Felix Handte 6528fa088e Do a `-pedantic` Build in CircleCI 2020-05-04 10:59:15 -04:00
W. Felix Handte dacbcd2cc1 Fix Up Some Pointer Handling in Tests 2020-05-04 10:59:15 -04:00
W. Felix Handte 3764859060 Switch Helper Declaration to Not Force Inline
It was causing build issues in ANSI mode.
2020-05-04 10:59:15 -04:00
W. Felix Handte c7da66c9cf Purge C++-Style Comments (`// ...`), Make Compilation Succeed Under C90 2020-05-04 10:59:15 -04:00
W. Felix Handte 952427aebf Avoid inline Keyword in C90
Previously we would use it for all gcc-like compilations, even when a
restrictive mode that disallowed it had been selected.
2020-05-04 10:59:15 -04:00
W. Felix Handte baa4e2e36c Don't Evaluate Arguments to Dummy Function 2020-05-04 10:59:15 -04:00
W. Felix Handte 450542d3a7 Allow Empty Format Strings in Error Macro Invocations
`-Wall` implies `-Wformat-zero-length`, which will cause compilation to fail
under `-Werror` when an empty string is passed as the format string to a
`printf`-family function. This commit moves us back to prefixing the provided
format string, which successfully avoids that warning.

However, this removes the failure mode where that `RAWLOG` invocation would
fail to compile when no format string was provided at all (which was desirable
to avoid having code that would successfully compile normally but fail under
`-pedantic`, which *does* require that a non-zero number of args are provided).

So this commit also introduces a function which does nothing at all, but will
fail to compile if not provided with at least one argument, which is a string.
This successfully links the compilability of pedantic and non-pedantic builds.
2020-05-04 10:59:15 -04:00
W. Felix Handte 6696933b32 Make All Invocations Start With Literal Format String 2020-05-04 10:59:15 -04:00
W. Felix Handte 2745f7a7d5 Make Error Macro Invocation Without Info String Fail to Compile
Even without `-pedantic`, these macros will now fail to compile unless you
provide an info string argument. This will prevent us from regressing.
2020-05-04 10:59:15 -04:00
W. Felix Handte 5e5f262612 Add (Possibly Empty) Info Strings to All Variadic Error Handling Macro Invocations 2020-05-04 10:58:55 -04:00
Nick Terrell e103d7b4a6
Fix superblock mode (#2100)
Fixes:

Enable RLE blocks for superblock mode
Fix the limitation that the literals block must shrink. Instead, when we're within 200 bytes of the next header byte size, we will just use the next one up. That way we should (almost?) always have space for the table.
Remove the limitation that the first sub-block MUST have compressed literals and be compressed. Now one sub-block MUST be compressed (otherwise we fall back to raw block which is okay, since that is streamable). If no block has compressed literals that is okay, we will fix up the next Huffman table.
Handle the case where the last sub-block is uncompressed (maybe it is very small). Before it would skip superblock in this case, now we allow the last sub-block to be uncompressed. To do this we need to regenerate the correct repcodes.
Respect disableLiteralsCompression in superblock mode
Fix superblock mode to handle a block consisting of only compressed literals
Fix a off by 1 error in superblock mode that disabled it whenever there were last literals
Fix superblock mode with long literals/matches (> 0xFFFF)
Allow superblock mode to repeat Huffman tables
Respect ZSTD_minGain().
Tests:

Simple check for the condition in #2096.
When the simple_round_trip fuzzer enables superblock mode, it checks that the compressed size isn't expanded too much.
Remaining limitations:

O(targetCBlockSize^2) because we recompute statistics every sequence
Unable to split literals of length > targetCBlockSize into multiple sequences
Refuses to generate sub-blocks that don't shrink the compressed data, so we could end up with large sub-blocks. We should emit those sections as uncompressed blocks instead.
...
Fixes #2096
2020-05-01 16:11:47 -07:00
Meghna Malhotra 0adfc8dfce Fix broken CI; make changes in response to the comments 2020-05-01 13:45:48 -07:00
Meghna Malhotra 53d76dc20f Remove magic constant and made other changes addressing the comments 2020-05-01 13:45:48 -07:00
Meghna Malhotra cc7c29595d Fixed tests to use correct workspace size 2020-05-01 13:45:48 -07:00
Meghna Malhotra fe8402b522 WIP: Still getting an error 2020-05-01 13:45:48 -07:00
Meghna Malhotra a084d959bd WIP: Increased wksp size, but it's segfaulting 2020-05-01 13:45:48 -07:00
Meghna Malhotra fdb2780c47 Move rank table into HUF_buildCTable_wksp() 2020-05-01 13:45:48 -07:00
Bimba Shrestha 0eaaa4d3a7
Merge pull request #2101 from bimbashrestha/meson
[build] fix meson build issue
2020-05-01 12:52:37 -05:00
Bimba Shrestha 834a8f838a meson msvc build fix 2020-05-01 09:04:09 -05:00
Yann Collet da2748a855
Merge pull request #2097 from facebook/underlink
Fix underlinked libzstd
2020-04-30 10:16:24 -07:00
Yann Collet 6f62a9caaa fixed zstd-nolegacy target
when compiled as part of allVariants

(though I'm unsure why it was working before ...)
2020-04-29 11:56:21 -07:00
Yann Collet 7ea2ae6649 added test linking user program to multi-threaded libzstd 2020-04-28 21:18:29 -07:00
Yann Collet f77fd5ced0 generalized pattern rules 2020-04-28 18:43:55 -07:00
Yann Collet c6ae2e83bc fix libzstd-mt underlinking issue
fix #2045
When compiling `libzstd` in multithreading mode,
the `libzstd-mt` recipe would not include `-pthread`,
resulting in an underlinked dynamic library.

Added a test on Travis to check that the library is fully linked.

This makes it possible, in some future release,
to build a multi-threaded `libzstd` dynamic library by default
as it would no longer impact the build script of user programs.
2020-04-28 18:29:20 -07:00
Nick Terrell 0ed07f6dfe
Merge pull request #2094 from terrelln/stable-dst
[lib] Add ZSTD_d_stableOutBuffer + fix single-pass mode for empty frames
2020-04-28 17:53:24 -07:00
Yann Collet f17ac423b2 new tests created new artifacts
they were not properly ignored
2020-04-28 15:58:22 -07:00
Nick Terrell 55a57d46be Add extra warnings about not modifying the ZSTD_outBuffer 2020-04-28 12:07:42 -07:00
Nick Terrell 108a5572a5
Merge pull request #2048 from nocnokneo/ctest-support
Add CTest support
2020-04-28 11:01:13 -07:00
Nick Terrell 1343b815f8 [fuzz] Fuzz test ZSTD_d_stableOutBuffer 2020-04-27 20:04:04 -07:00
Nick Terrell 77a2945c43 Add some comments 2020-04-27 20:04:04 -07:00
Nick Terrell f33de06c3e [lib] Fix single-pass mode for empty frames 2020-04-27 20:04:01 -07:00
Nick Terrell a4ff217baf [lib] Add ZSTD_d_stableOutBuffer 2020-04-27 18:09:44 -07:00
Nick Terrell b104f8e3eb [zstd] Fix typo in ZSTD_dParameter 2020-04-27 12:12:28 -07:00
Bimba Shrestha 6b4a3e019f
Merge pull request #2088 from bimbashrestha/bug
[bug] Making compressStream2 fail when passing rawContent but claiming fullDict
2020-04-23 14:16:56 -05:00
Bimba Shrestha 861a73744d
Merge pull request #2086 from bimbashrestha/quick3
[bug] passing cParams.hashLog instead of comprParams->hashLog for ldm trigger
2020-04-23 13:02:57 -05:00
Bimba Shrestha 1875f616ce passing dictContentType instead of rawContent every time 2020-04-21 22:29:35 -07:00
Bimba Shrestha f7a7409a49 adding fail test when passing wrong fullDict using refPrefix 2020-04-21 22:26:48 -07:00
Bimba Shrestha dba02245bf bash to shell conversion 2020-04-21 20:31:11 -07:00
Bimba Shrestha 659ff85cf5 changing to cParams.hashLog 2020-04-21 21:12:50 -05:00
Bimba Shrestha 0b107188b8 adding test for long mode trigger 2020-04-21 21:09:49 -05:00
Bimba Shrestha 38a6d2a8f2
Merge pull request #2085 from bimbashrestha/quick2
[doc-bug] Updating the recommended max chainlog for patch-from
2020-04-20 14:09:23 -05:00
Bimba Shrestha b640802ff8 updating chainlog bound 2020-04-20 11:34:24 -07:00
Bimba Shrestha c6311584aa
Merge pull request #2083 from bimbashrestha/quick1
[nit] Adding missing newline at end of patch-from recommendations
2020-04-20 13:24:34 -05:00