facebook/zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,671 Commits 2 Branches 0 Tags
Commit Graph

1747 Commits

Author SHA1 Message Date
Yann Collet
bad7f82300
Merge pull request #2974 from facebook/fix2966_part3 
Lazy parameters adaptation (part 1 - ZSTD_c_stableInBuffer)
 2022-01-27 06:14:04 -08:00
brailovich
501a353b91
Update playTests.sh 2022-01-26 18:56:52 -08:00
Yann Collet
f2d9652ad8 more usage of new error code stabilityCondition_notRespected 
as suggested by @terrelln
 2022-01-26 18:30:55 -08:00
Nick Terrell
e60eba58bf Print zlib/lz4/lzma library versions in verbose version output 
Knowing the version of zlib/lz4/lzma we're linking against is very
useful for debugging issues with those libraries, so print it out in the
verbosity 4 version output.

Also print this information at the top of `playTests.sh`.
 2022-01-26 18:25:58 -08:00
brailovich
5e7523385b
Update playTests.sh 2022-01-26 16:53:11 -08:00
Yann Collet
dda4c10f07 added ZSTD_compressStream2() + ZSTD_c_stableInBuffer test 2022-01-26 13:33:04 -08:00
Yann Collet
af3d9c506e added streaming test starting from non-0 pos 2022-01-26 10:31:25 -08:00
Yann Collet
c1668a00d2 fix extended case combining stableInBuffer with continue() and flush() modes 2022-01-26 10:31:25 -08:00
Yann Collet
270f9bf005 better consistency in accessing @input 
as suggested by @terrelln.

Also : commented zstreamtest more
to ensure ZSTD_stableInBuffer is tested/
 2022-01-26 10:31:24 -08:00
Yann Collet
27d336b099 minor behavior refinements 
specifically, there is no obligation to start streaming compression with pos=0.
stableSrc mode is now compatible with this setup.
 2022-01-26 10:31:24 -08:00
Yann Collet
37b87add7a make stableSrc compatible with regular streaming API 
including flushStream().

Now the only condition is for `input.size` to continuously grow.
 2022-01-26 10:31:24 -08:00
Yann Collet
fc2ea97442 refactored fuzzer tests for sequence compression api 
add explicit delimiter mode to libfuzzer test
 2022-01-26 00:19:35 -08:00
Yann Collet
cc7d23bcec
Merge pull request #2965 from facebook/offbase 
Converge sumtype (offset | repcode) numeric representation towards offBase
 2022-01-24 15:47:42 -08:00
binhdvo
17017ac8db
Change zstdless behavior to align with zless (#2909) 
* Change zstdless behavior to align with zless
 2022-01-21 19:57:19 -05:00
Yonatan Komornik
1598e6c634
Async write for decompression (#2975) 
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
 2022-01-21 13:55:41 -08:00
Yann Collet
71921e596f
Merge pull request #2983 from facebook/minLitPricev2 
[opt] minor compression ratio improvement
 2022-01-20 16:02:31 -08:00
Nick Terrell
8ea3d57de4 [build][asm] Pass ASFLAGS to the assembler instead of CFLAGS 
* Add `-Wa,--noexecstack` to both `ASFLAGS` and `CFLAGS`
* Pass `ASFLAGS` to `.S` compilation instead of `CFLAGS`

Fixes #3006.
 2022-01-18 15:11:29 -08:00
Yann Collet
05e884f288
Merge pull request #2956 from sunwire/fix-tar-test-cases 
Fix tar test cases
 2022-01-11 10:16:34 -08:00
Yann Collet
5595aec629 updated regression results 2022-01-07 15:08:06 -08:00
Nick Terrell
c7b03c217c [license] Fix license header of huf_decompress_amd64.S 
* Add the license header for `huf_decompress_amd64.S`
* Add `.S` files to the `test-license.py` test
 2022-01-07 09:35:27 -08:00
Felix Handte
7e679511a8
Merge pull request #2964 from felixhandte/noexecstack-all-archs 
Mark Huffman Decoder Assembly `noexecstack` on All Architectures
 2022-01-05 16:52:39 -05:00
W. Felix Handte
ff5d1daf33 Clean Up Debugging Statements 2022-01-05 16:13:00 -05:00
W. Felix Handte
35208f702f Add Test Validating Stack is not Executable in playTests.sh 2022-01-04 16:57:35 -05:00
Yann Collet
a0b9520e38 fullbench: added compress_freshCCtx scenario 2022-01-02 08:52:33 -08:00
Yann Collet
213dc6110f fixed fullbench freshCCtx scenario 2022-01-01 23:46:49 -08:00
Yann Collet
03903f5701 fixed minor compression difference in btlazy2 
subtle dependency on sumtype numeric representation
 2021-12-29 18:51:03 -08:00
Yann Collet
7a18d709ae updated all names to offBase convention 2021-12-29 17:30:43 -08:00
Yann Collet
8da414231d found a few more places which were dependent on seqStore offcode sumtype numeric representation 2021-12-28 17:03:24 -08:00
Yann Collet
681c81f06c abstracted storeSeq() sumtype numeric representation from decodecorpus.c 2021-12-28 11:58:33 -08:00
Yann Collet
435f5a2e6d fixed regression test assert 
optLdm->offset might be == 0 in invalid case.
Only use STORE_OFFSET() after validating it's a correct case.
 2021-12-28 09:55:31 -08:00
Paweł Marciniak
666372c7bf Fix tar test cases 2021-12-24 15:05:26 +01:00
Yann Collet
1aed962216 introduce macros STORE_OFFSET() and STORE_REPCODE() 
this meant to abstract the sumtype representation required
to transfert `offcode` to `ZSTD_storeSeq()`.

Unfortunately, the sumtype numeric representation is currently a leaky abstraction
that has permeated many other parts of the code,
especially within `zstd_lazy.c` and also within `zstd_opt.c` and `zstd_compress.c`.

While this PR makes a good job a transfering a large nb of call sites
to using the new macros, there are still a few sites where this transformation is more complex,
or where the numeric representation itself it used "as is".

One of the problematics area is the decision to use the numeric format of the sumtype
within the match finders of `zstd_lazy`.

This commit doesn't change the behavior, it only introduces and employes the macros,
but eventually the resulting code remains identical.

At target, if the numeric representation of the sumtype can be completely abstracted
and no other part of the code depends on it,
it will be possible to move it towards something slightly more efficient.
 2021-12-23 22:03:30 -08:00
Yann Collet
aeff128331 change seqDef.offset into seqDef.offBase 
to better reflect the value stored in this field.
 2021-12-23 17:56:08 -08:00
Yann Collet
e145b58cfd changed seqDef.matchLength into seqDef.mlBase 
since this is effectively what is stored in this field (== matchLength - MINMATCH).
This makes it clearer what needs to be done when reading from / writing to this field.
 2021-12-23 13:39:46 -08:00
Yann Collet
b77fcac61f change ZSTD_storeSeq() interface to accept matchLength 
instead of mlBase.

This removes the need to do `- MINMATCH` at every call site.

The new interface contract is checked with an `assert()`.
 2021-12-23 12:03:33 -08:00
Yann Collet
a9e43b37d0
Revert "Limit ZSTD_maxCLevel to 21 for 32-bit binaries." 2021-12-20 11:43:14 -08:00
Yann Collet
01adddc3e0 update regression results 2021-12-16 20:43:23 -08:00
Yann Collet
7c7b9244d6 update regression results 2021-12-16 16:07:54 -08:00
Yann Collet
9a32492730 updated regression results.csv 2021-12-16 14:39:30 -08:00
Elliot Gorokhovsky
c5f1e826ca
Merge pull request #2925 from embg/dict_training_sample_limit_size 
Allow user to specify memory limit for dictionary training
 2021-12-15 15:58:17 -05:00
Elliot Gorokhovsky
71c0c07c19 Allow user to specify memory limit for dictionary training 2021-12-14 14:29:01 -05:00
Felix Handte
5e2fede604
Merge pull request #2921 from felixhandte/neg-lvl-stagger-step 
Stagger Stepping in Negative Levels
 2021-12-14 14:13:57 -05:00
W. Felix Handte
450fca9704 Update Regression Tests w/ New Sizes 2021-12-13 17:29:32 -05:00
Nick Terrell
3e2a70b6fb
Merge pull request #2905 from 15596858998/dev_1205 
add test case
 2021-12-13 13:45:23 -08:00
zx123123
c69d13eb99
Update playTests.sh 2021-12-13 08:58:42 +08:00
Felix Handte
0c26d98c0d
Merge pull request #2910 from felixhandte/reject-irregular-dicts 
Reject Irregular Dictionary Files
 2021-12-09 11:44:37 -05:00
W. Felix Handte
9985e10fda Reject Irregular Dictionary Files 
I hadn't seen #2890, so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.

Addresses #2874.
 2021-12-08 16:17:04 -05:00
Yann Collet
fb3522a3fe fixed very minor cast warning under cygwin 2021-12-08 09:48:56 -08:00
binhdvo
38dfc4699e
Imply -q when stderr is not a tty (#2884) 
* Imply -q when stderr is not a tty
 2021-12-07 16:56:19 -05:00
15596858998
ba7e8f1051 add test case 2021-12-05 19:12:52 +08:00