facebook/zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,929 Commits 2 Branches 0 Tags
Commit Graph

8929 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell
ba044bd6f1 [bug-fix] Fix a determinism bug with the DUBT 
The DUBT can be non-deterministic if an index is equal to
`ZSTD_DUBT_UNSORTED_MARK`. Ensure that never happens by starting the
indices at 2.

This bug was found by the OSS-Fuzz determinism fuzzer. With this change
the fuzzer test passes. And I've confirmed that this is the root cause,
not just hiding the problem.

Aside: This took me a long time to figure out, because I thought I had
tried this first thing. But, apparantly I messed it up, because when I
was going through it again with @felixhandte, I was pointing out that it
wasn't the case, but it turns out it was.

Credit to: OSS-Fuzz
 2021-07-15 13:02:49 -07:00
Yann Collet
b18febe52c
Merge pull request #2725 from makise-homura/mcst-lcc-support 
Add support for MCST LCC compiler
 2021-07-15 00:09:12 +02:00
makise-homura
3cd085cec3 Clarify no-tree-vectorize usage for ICC and LCC 2021-07-14 20:00:44 +03:00
makise-homura
a5f518ae27 Change zstdcli's main() declaration due to -Wmain on some compilers 2021-07-14 19:55:47 +03:00
makise-homura
d4ad02c721 Add support for MCST LCC compiler 2021-07-10 03:57:06 +03:00
binhdvo
b3e372c171
Merge pull request #2717 from binhdvo/bootcamp 
Proactively skip huffman compression based on sampling where non-comp…
 2021-07-01 10:39:58 -04:00
sen
d90bc0e0b6
Merge pull request #2720 from senhuang42/remove_folder 
Remove folder when done with test
 2021-06-30 14:37:21 -04:00
senhuang42
f5f6cc2e48 Remove folder when done with test 2021-06-30 13:09:00 -04:00
Binh Vo
dc5b693f1e Proactively skip huffman compression based on sampling where non-compressibility is suspected 2021-06-30 11:02:47 -04:00
Nick Terrell
7b43e69b45
Merge pull request #2655 from heitbaum/patch-1 
[trace] remove zstd_trace.c reference from freestanding
 2021-06-29 18:33:40 -07:00
Nick Terrell
609be382ac
Merge pull request #2719 from danlark1/danlark_iwyu 
Include what you use in zstd_ldm_geartab
 2021-06-29 16:53:10 -07:00
Nick Terrell
094b26081f
Merge pull request #2689 from danlark1/dev 
Optimize zstd decompression by another x%
 2021-06-29 11:34:36 -07:00
Danila Kutenin
e855b78be6 Include what you use in zstd_ldm_geartab 2021-06-29 17:57:53 +01:00
Danila Kutenin
2c2c9e7dfd Add possible improvements for gcc-11 2021-06-29 09:06:47 +01:00
Yann Collet
88f107b5b9
Merge pull request #2716 from facebook/fix2701 
remove invalid test
 2021-06-24 10:57:18 -07:00
sen
45d707e908
Merge pull request #2715 from senhuang42/sequence_api_3 
[RFC] Add internal API for converting ZSTD_Sequence into seqStore
 2021-06-24 13:02:11 -04:00
Yann Collet
139a2f1e0c remove invalid test 
`--mt` is no longer supported by `zstreamtest`
(relevant API entry point has been removed from `libzstd`).

fix #2701
 2021-06-24 09:07:55 -07:00
senhuang42
76466dfadf Add simple API for converting ZSTD_Sequence into seqStore 2021-06-23 12:10:48 -04:00
Felix Handte
628f65b79c
Merge pull request #2714 from luisdallos/build-macros-typos 
lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros
 2021-06-18 16:16:44 -04:00
Usuario
8bdce1ff97 lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros 2021-06-18 10:07:39 -04:00
binhdvo
b99d9f5ebd
Merge pull request #2710 from binhdvo/bootcamp 
Add option to use logical cores for default threads
 2021-06-17 09:06:18 -04:00
Binh Vo
6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
binhdvo
0152435ab0
Merge pull request #2708 from binhdvo/skippable 
Add API for fetching skippable frame content
 2021-06-14 19:00:31 -04:00
Binh Vo
9d9f7680f8 Add API for fetching skippable frame content 2021-06-14 16:01:28 -04:00
Nick Terrell
6917c4ecdf
Merge pull request #2709 from terrelln/bounds-check-fix 
[fix] Add missing bounds checks during compression
 2021-06-14 12:33:50 -07:00
Nick Terrell
05b6773fbc [fix] Add missing bounds checks during compression 
* The block splitter missed a bounds check, so when the buffer is too small it
  passes an erroneously large size to `ZSTD_entropyCompressSeqStore()`, which
  can then write the compressed data past the end of the buffer. This is a new
  regression in v1.5.0 when the block splitter is enabled. It is either enabled
  explicitly, or implicitly when using the optimal parser and `ZSTD_compress2()`
  or `ZSTD_compressStream*()`.
* `HUF_writeCTable_wksp()` omits a bounds check when calling
  `HUF_compressWeights()`. If it is called with `dstCapacity == 0` it will pass
  an erroneously large size to `HUF_compressWeights()`, which can then write
  past the end of the buffer. This bug has been present for ages. However, I
  believe that zstd cannot trigger the bug, because it never calls
  `HUF_compress*()` with `dstCapacity == 0` because of [this check][1].

Credit to: Oss-Fuzz

[1]: 89127e5ee2/lib/compress/zstd_compress_literals.c (L100)
 2021-06-14 11:35:33 -07:00
binhdvo
89127e5ee2
Merge pull request #2705 from binhdvo/bootcamp 
Add support for negative values in advanced flags
 2021-06-11 14:08:23 -04:00
Binh Vo
6fad35c6a1 Add support for negative levels in --adapt=min and --adapt=max" 2021-06-11 12:13:09 -04:00
Felix Handte
67a25968f0
Merge pull request #2702 from felixhandte/human_size_output 
Format File Sizes Human-Readable in the CLI
 2021-06-10 16:53:43 -04:00
W. Felix Handte
8c00807bbc Whitespace Fixes to Improve Cross-Line Alignment 2021-06-10 13:28:38 -04:00
W. Felix Handte
94cf57bb13 Update Tests to Reflect New Formatting 2021-06-10 13:14:18 -04:00
W. Felix Handte
87e94e3e39 Convert Other Size Displays to Use Human-Readable Formatting 2021-06-10 12:57:46 -04:00
W. Felix Handte
2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB") 
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
 2021-06-10 12:53:07 -04:00
W. Felix Handte
9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte
bc46b6efe4 Apply to Other Print Statement as Well 2021-06-10 12:53:07 -04:00
W. Felix Handte
7e0058848c Fix Whitespace 2021-06-10 12:53:07 -04:00
W. Felix Handte
93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte
464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte
9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte
bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting 
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
 2021-06-10 12:53:07 -04:00
Scott Baker
8e0a9695d7 Attempt to fix a failing test with help from @aqrit 2021-06-10 12:53:07 -04:00
Scott Baker
1eb852854b Some fixes to address things @felixhandte found 2021-06-10 12:53:07 -04:00
Scott Baker
376a2730a8 Try enabling the BIG strings now the unsigned long long is in effect 2021-06-10 12:53:07 -04:00
Scott Baker
20b9b00b41 Try unsigned long long 2021-06-10 12:53:07 -04:00
Scott Baker
64385ef7cb Update humanSize() to skip the big numbers (it requires 64 bit) 2021-06-10 12:53:07 -04:00
Scott Baker
1ef6f3d079 Use unsigned long instead to help with some tests 2021-06-10 12:53:07 -04:00
Scott Baker
e5fc830795 human_size() should use size_t 2021-06-10 12:53:07 -04:00
Scott Baker
35576e63ce Convert tabs to spaces 2021-06-10 12:53:07 -04:00
Scott Baker
77001f00fb Use human_size() on the "multiple files compressed" output also 2021-06-10 12:53:07 -04:00
Scott Baker
894698d3b6 Use human_size() in the benchmark output also 2021-06-10 12:53:07 -04:00